Re: [PATCH 1/6] docs: document bundle URI standard

From: Junio C Hamano <gitster@pobox.com>
Date: Jun 9, 2022, 17:56 UTC
Message-ID: <xmqqk09plnze.fsf@gitster.g>
In-Reply-To: <3d67b69b-fac8-3171-92dc-303ea672efbf@github.com>

Derrick Stolee <derrickstolee@github.com> writes:

Show 11 quoted lines

> Ævar's observation that bundles also add ref tips to the packfile is
> the key to breaking down this concern: these ref tips give us a way
> to negotiate the difference between what the client already has
> (including the bundles downloaded from a bundle provider) and what it
> wants from the origin Git server. This all happens without any change
> necessary to the origin Git server.
>
> And thus, this bundle URI design came about. It takes all of the best
> things about the GVFS Cache Server but then layers refs on top of the
> time-based prefetch packfiles so a normal Git client can do that
> "catch-up fetch" afterwards.

Yup. My observation was that (1) you would need ref tips in some way, (2) you are conveying not just "here are the set of bundle files", but "this bundle file has these associated attributes" (like .timestamp, and .uri to fetch it from), in the table-of-contents the clients are expected to obtain anyway, hence (3) you could, but you do not need to, use bundle as a way to convey "packfile contents plus refs" to the clients (iow, instead you can use packfile and then report these refs information in the table-of-contents as more "associated attributes" to the items listed in the table-of-contents).

> I could imagine updating GVFS Cache Servers to generate bundles
> instead (or also) and updating the VFS for Git clients to use the
> bundle URI feature to download the data.

I could, too, but I do not think that would buy us anything. If an existing system is happily working, I do not see a point in switching it to use bundle. What I was imagining is going the other way. A new thing being written, instead of basing it on bundles, can be based on packfiles, and that would allow you to share the on-disk packfiles between the two systems.

Show 6 quoted lines

> However, you seem to be hinting at "the GVFS Cache Servers seem to
> work just fine, so why do we need bundles?" but I think that the
> constraints of what is expected at the end of "git clone" or "git
> fetch" require us to not "catch up later" and instead complete the
> full download during the process. The refs in the bundles are critical
> to making that work.

The refs are critical.  They do not have to be recorded in a bundle.

Show 9 quoted lines

> I see two major issues with that:
>
> 1. We don't have a way to add that metadata directly into packfiles,
>    so we would need to update the file standard or update the
>    packfile-URI protocol to include that metadata.
>
> 2. The only source of packfile-URI listings come as a response to the
>    "git fetch" request to the origin Git server, so there is no way
>    to allow an independent server to provide that data.

It might be end up being the same thing at the end, but I was thinking about starting from the "bundle URI standard" document. I am not yet interested in discussing "packfile URI" feature that independently exists already in this discussion (but going this route we _might_ be able to share data and some code with it---but that was not where I am coming from).

Starting from "bundle URI standard" document at the beginning of the thread, if we replace all the mentions of "bundle file" with "packfile" in it, and then add .positiveRefs and .negativeRefs to each "packfile" (renamed from "bundle file") as additional "packfile.<id>.*" (renamed from "bundle.<id>.*") attributes, without changing anything else, the result would be feature equivalent to the original "bundle URI standard", I would think, but without having to wrap a packfile in a bundle file?

> I hope I am going in the right direction here, but I likely
> misunderstood some of your proposed alternatives.

I wasn't seriously "proposing" an alternative. It was just that it looked wasteful to go to a separate format (i.e. bundle) when packfiles should suffice, as you would be adding extra information that is not in bundles via the table-of-contents anyway, and what is given by a bundle that is missing in a packfile is only the refs information, which should be trivial to add to the table-of-contents.