Re: [GSoC Proposal] Implement promisor remote fetch ordering
- From
Christian Couder <christian.couder@gmail.com>
- Date
- Mar 14, 2026, 17:30 UTC
- Message-ID
- <CAP8UFD1=Ow6NNFKK6y5csmneVaS0J+e5z9pGjFmaVoJ2g1OPFg@mail.gmail.com>
- In-Reply-To
- <abBh__zmlWXY-yjI@lorenzo-VM>
On Tue, Mar 10, 2026 at 7:25 PM Lorenzo Pegorari <lorenzo.pegorari2002@gmail.com> wrote:
> > The following is my proposal for the GSoC'26 for the project "Implement > promisor remote fetch ordering".
Thank you for your interest in Git and this project.
> As soon as the the contributor application period begins, I will submit > the proposal in PDF format to the official GSoC website.
Good idea.
> I have dedicated a large section (about 40%) of the proposal to > explaining the current situation and the tests that I have done to gain a > lot of hands-on experience. I consider this section important, but if it > too long-winded, please let me know.
[...]
Show 15 quoted lines
> So far, I have made the following contributions to Git: > > * [GSoC PATCH v2] diff: improve scaling of filenames in diffstat to handle UTF-8 chars > * Link: https://lore.kernel.org/git/cover.1768520441.git.lorenzo.pegorari2002@gmail.com > * Description: The computation of column width made by `git diff --stat` > was confused when pathnames contained non-ASCII chars. > This issue was reported by a `NEEDSWORK` comment. > * Status: Merged to `master` > > * [GSoC PATCH v3] diff: handle ANSI escape codes in prefix when calculating diffstat width > * Link: https://lore.kernel.org/git/cover.1772226209.git.lorenzo.pegorari2002@gmail.com > * Description: Fixed `git log --graph --stat` not correctly counting > the display width of colored graph part of its own > output. This issue was reported by a `NEEDSWORK` comment. > * Status: Merged to `master`.
For the patches that are merged to master, it could help if you could give the object ID of the merge commit that merged your commits into master, or alternatively the object ID of all your commits.
> * [GSoC PATCH v3] doc: improve gitprotocol-pack > * Link: https://lore.kernel.org/git/cover.1772502209.git.lorenzo.pegorari2002@gmail.com > * Description: Improved the `gitprotocol-pack` documentation. > * Status: Will merge to `master`.
Yeah, this has been merged to master after your email.
[...]
Show 8 quoted lines
> Partial clones avoid this issue during `clone` and `fetch` operations by > passing all the objects to download through a `--filter=<filter-spec>` > specified by the user, which will limit the number of blobs and trees > that actually get downloaded. The `<filter-spec>`, can, for example, be: > * `blob:none`, which will filter out all blobs. > * `tree:0`, which will filter out all trees. > * `blob:limit=5k`, which will filter out all blobs whose size is greater > than $5$kB.
Why are there '$' signs above?
Show 29 quoted lines
> The filtered out objects will be lazily downloaded when the user runs a > command that requires those missing data. > > This mechanism works with the following steps: > * When the client wants to fetch some objects from the server using a > filter, the client, after sending a list of capabilities it wants to > be in effect, sends the `filter: <filter-spec>` capability, followed > by a request for the objects that the client wants to retrieve. The > following is an example of a request (extracted using > `GIT_TRACE_PACKET=1`) made by a client to a server to fetch 1 object > using the `<filter-spec>=blob:none`: > > ``` > [...] > pkt-line.c:85 packet: fetch< 0000 # "flush-pkt" > pkt-line.c:85 packet: fetch> command=fetch # Execute fetch > pkt-line.c:85 packet: fetch> agent=git/2.43.0 > pkt-line.c:85 packet: fetch> object-format=sha1 > pkt-line.c:85 packet: fetch> 0001 # "delim-pkt" > pkt-line.c:85 packet: fetch> thin-pack # Capability > pkt-line.c:85 packet: fetch> no-progress # Capability > pkt-line.c:85 packet: fetch> ofs-delta # Capability > pkt-line.c:85 packet: fetch> filter blob:none # Filter capability > # OID of the object the client wants to retrieve > pkt-line.c:85 packet: fetch> want 394ca7a7b5e75a57e736040480f685c8b71844eb > pkt-line.c:85 packet: fetch> done # End fetch > pkt-line.c:85 packet: fetch> 0000 # "flush-pkt" > [...] > ```
I think when lazy fetching like this, the filter is always blob:none. It's not really used anyway because the objects that the client wants are specified explicitly.
The filter is important when initially cloning or fetching from the server to specify which objects are initially excluded, even if some of these objects will be lazy fetched soon. For example the checkout part of a clone might need objects that were initially excluded, so it might lazy fetch some.
> * The server will apply the requested `<filter-spec>` as it creates the > "promisor packfile" of the requested objects.
This is important during an initial clone or fetch, not when lazy fetching.
Show 6 quoted lines
> A packfile is a binary > file that is used to compress many "loose objects", and it does so by > containing the most recent versions of the stored objects and deltas > of the previous versions of those objects. A promisor packfile is a > filtered packfile, where the unwanted objects are not present. The > promisor packfile is sent to the client.
Show 9 quoted lines
> I created a minimal example setup, mostly based on the test
> `t/t5710-promisor-remote-capability` added by `4602676` ("Add
> 'promisor-remote' capability to protocol v2", 2025-02-18), to experiment
> with multiple promisor remotes, in order to not simply rely on the
> documentation, but to actually get hands-on experience. The example setup
> creates a `server`, a 'lopm' ("Large Object Promisor medium") for blobs
> larger than 5kB, a `lopl` ("Large Object Promisor large") for blobs
> larger than 50kB, and a `client` that interfaces with all of these
> remotes. It is created in the following way:[...]
> Now, with this setup, by slightly tweaking the configurations of each > repository, it is possible to deeply test how multiple promisor remotes > are handled in various situations, and actually see what is described in > the documentation.
Yeah, it's quite complex to set up.
Show 11 quoted lines
> ## Testing Promisor Remotes Advertisement > > An important thing to test is the promisor remotes advertisement feature. > This feature is dependent on 2 main configuration options: the > server-side option `promisor.advertise`, which enables the server to > advertise the promisor remotes it is using to the client, and the > client-side option `promisor.acceptFromServer`, which describes how the > client should handle the promisor remotes advertised: > > * If `promisor.advertise=false`, when the `client` wants to fetch an > object that the `server` does not have,
I don't think it depends on the client fetching an object the server does not have. It depends on the client using a filter because the promisor-remote capability only makes sense in the case of partial clones (or fetches).
Show 11 quoted lines
> the `server` will not > advertise the `promisor-remote` capability, and so it has no other > choice than to first fetch the object from `lopl` and/or `lopm`, and > then give it to the `client`. This can be checked by doing `git -C > server rev-list --objects --all --missing=print`, and seeing that the > previously missing large blobs are now present inside the `server`, or > by directly looking into the `GIT_TRACE_PACKET` output, and seeing > that there is no reference to the `promisor-remote` capability. > > * If `promisor.advertise=true`, when the `client` wants to fetch an > object that the `server` does not have,
Same as above, it doesn't depend on the client fetching an object the server does not have. It depends on the client using a filter because the promisor-remote capability only makes sense in the case of partial clones (or fetches).
Show 11 quoted lines
> the `server` will advertise > its promisor remotes, as seen by the `GIT_TRACE_PACKET` output, which > will contain: > > ``` > [...] > packet: upload-pack> promisor-remote= \ > name=lopl,url=file://$(pwd)/lopl; \ # Adv lopl > name=lopm,url=file://$(pwd)/lopm # Adv lopm > [...] > ```
[...]
Show 8 quoted lines
> Recently, with the patch series "Implement `promisor.storeFields` and > `--filter=auto`" [5], the new client-side configuration variable > `promisor.storeFields` was added. It contains a list of field names > `partialCloneFilter` and/or `token`), and the values of these fields, > when transmitted by the server, will be stored in the local configuration > on the client. > > ## Testing Multiple Promisor Remotes Fetch Order
Yeah, I think this is the most relevant for the project.
Show 6 quoted lines
> Finally, the last mechanism that is fundamental to understand is the > fetch order when multiple promisor remotes are defined: > > * When multiple remotes are configured, they are tried one after the > other in the order in which they appear in the configuration, until > all objects are fetched.
Right, but there is the exception of a remote configured with `extensions.partialClone` that will be tried last. You mention it later though.
Show 22 quoted lines
> This can be easily seen from the output of > `GIT_TRACE`, which initially tries to fetch the objects from `lopl`, > and then from `lopm`: > > ``` > [...] > trace: built-in: git fetch lopl [...] --filter=blob:none [...] > [...] > trace: built-in: git fetch lopm [...] --filter=blob:none [...] > [...] > ``` > > While, if we make it so that we first define `lopm` in the `client` > configuration, then initially `lopm` will be used to fetch the > objects, and `lopl` will not be used at all (because `lopm` contains > all required objects: > > ``` > [...] > trace: built-in: git fetch lopm [...] --filter=blob:none [...] > [...] > ```
Yeah, when all the needed objects have been lazy fetched, there is no point in further fetching from any remote.
Show 23 quoted lines
> * If the configuration option `extensions.partialClone` is present, the > promisor remote that it specifies will always be the last one tried > when fetching objects. > > ------------------------------ > > # "Implement promisor remote fetch ordering" > > ## Project Goal > > This project aims to improve Git by implementing a fetch ordering > mechanism for multiple promisor remotes, that can be: > > * Configured locally by the client. > * Advertised by servers through the `promisor-remote` protocol. > > ## Approach > > The bulk of the project will be the creation of a system that allows to > define the order with which the promisor remotes will be tried when > fetching an object. > > The first goal will be the creation of a `remote.<name>.promisorPriority`
Yeah, or just `remote.<name>.priority`. The name is to be discussed.
> configuration option, which will hold a number between 1 and 'UCHAR_MAX',
UCHAR_MAX could be system dependent. It might be better to have configurations work in the same way on all machines though. So perhaps a fixed range like 1 to 100 would be better. Or are there other ranges of values used for similar things in Git or other well known software that could be reused?
Show 18 quoted lines
> and which defines the priority of that promisor remote in the fetch > order. This means that the order in which the promisor are tried will be > the following: > > * All promisor remotes that have a valid `remote.<name>.promisorPriority`, > starting from the one with higher priority (the lower `promisorPriority` > value). If 2 or more promisor remotes have the same priority, they will be > tried following the order in which they appear in the configuration file. > > * All promisor remotes that don't have or have an invalid > `remote.<name>.promisorPriority` configuration option. If 2 or more > promisor remotes don't define any priority, or have an invalid priority, > they will be tried following the order in which they appear in the > configuration file. > > * The promisor remote defined inside the `extensions.partialClone`, no > matter their priority (which will be ignored if present). This is > necessary for backward compatibility.
Yeah, I think something like what you describe makes sense.
> Having already taken a look at the code, I have a general idea of th
s/of th/of the/
> major steps to take to actually introduce the > `remote.<name>.promisorPriority` configuration option:
[...]
Show 11 quoted lines
> # Possible Issues > > From my understanding, the project as it is proposed will handle all > possible cases, except for one. Let's imagine the following situation: > > * `server1` and `server2` both use the promisor remotes `lop1` and `lop2`. > * `client` has both `server1` and `server2` as remotes. > > In this situation, the `client` has no way to specifically say that when > fetching from `server1`, it wants to first try `lop1` and then `lop2`, while > when fetching from `server2`, it wants to first try `lop2` and then `lop1`.
Right, but lazy fetching does not only happen as part of a clone or fetch from a server. It happens when for some reason (like a git show or a git blame for example) the user needs some objects it doesn't have locally, and when that happens, this is not related to a single server.
So global priorities are likely the most useful ones to have.
> One way to solve this very specific (and maybe unusual) issue is to > introduce a way to associate a `promisorPriority` to a specific remote.
Yeah, but I don't think it would be used a lot. We can perhaps think of some cases where it could be useful, but in practice it is likely that if there is an optimal order for one server, it will be optimal for all other servers too.
[...]
Thanks!