git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH RFC 0/3] cat-file: make remote-object-info request in batches

From
Pablo Sabater <pabloosabaterr@gmail.com>
Date
Oct 27, 2026, 20:47 UTC
Message-ID
<DMFWYZSCTM1C.ZM3OE2DGCMPC@gmail.com>
In-Reply-To
<xmqqld7jj6u5.fsf@gitster.g>
On Tue Oct 27, 2026 at 7:12 PM WET, Junio C Hamano wrote:
Show 25 quoted lines
> Pablo Sabater <pabloosabaterr@gmail.com> writes:
>
>> A noticeable difference can be seen from a 10k OID batch to a 50k batch.
>> After that the difference isn't as noticeable (the number of batches to
>> be requested isn't that big).
>>
>> I propose 50k OIDs as the default number of OIDs for the
>> remote-object-info batches. Coincidentally, git backfill's default batch
>> size is 50k as well though it's unrelated.
>
> Interesting.  We cannot share any code with backfill as we only
> retrieve info on objects while backfill does grab the objects
> themselves, but using the same number may make it easy to explain in
> the documentation.
>
>> I have barely done any benchmarking before, so I would appreciate
>> any feedback on the one in patch 2. Is there a better way to do it,
>> or other things I should take into account?
>>
>> Should the server also check for a limit on the number of OIDs it
>> accepts in a single object-info request?
>
> What I am more curious about is how this updated code interacts with
> existing code (i.e., updated "cat-file --batch" clients with deployed
> servers, and vice versa).

The code in this series only changes the client side, the server has no OID limit. Before this series, what prevented large requests was the client with its 10k hard limit, so the server is fine with any size without any change.

The server will see requests of at most 10k OIDs from old clients, and bigger ones from new clients.

The other change compared to an older client is that now multiple requests can be received on the same connection (instead of one per connection), but that should be fine, serve() keeps processing requests until EOF.

With this new option there's a risk (that was already there if someone modified their client) of setting a huge amount of OIDs in the same request, but I'm not sure how bad that is, a 1M OID request vs smaller ones on the same connection. Could you help me with this?

Anyway, the fix would be to add a cap on the server side. It can't be lower than 10k, or older clients would get an error they can't do anything about, and if it's lower than the client's batch size it should be advertised so newer clients can adapt.

IIRC clients ignore unknown capability keys, so advertising it shouldn't break older ones. In this case, it would be advertised on the handshake and the client would die() if the user sets a batch size bigger than what the server advertises. Ideally the default on the client (the proposed 50k) would be the min cap on the server, so someone that doesn't manually set the batch size and uses the default value doesn't get an error.

If we don't even want to die() we could set the batch size as:
	min(user-size, advertised-size)

Sorry for the brainstorm but I just realised that any client could modify the size and that a cap on the server side could be important.

Regards, Pablo

Previous: Junio C HamanoNext: Junio C Hamano
Message 11 of 12 in “cat-file: make remote-object-info request in batches”
  1. 0/3 cat-file: make remote-object-info request in batchesPablo Sabater, Oct 27, 2026
  2. 1/3 cat-file: extract the transport ownership from get_remote_info()Pablo Sabater, Oct 27, 2026
  3. Junio C HamanoOct 27, 2026
  4. Pablo SabaterOct 27, 2026
  5. Junio C HamanoOct 28, 2026
  6. Pablo SabaterOct 28, 2026
  7. Junio C HamanoOct 28, 2026
  8. 2/3 cat-file: make remote-object-info request in batchesPablo Sabater, Oct 27, 2026
  9. 3/3 cat-file: add --remote-max-batch optionPablo Sabater, Oct 27, 2026
  10. Junio C HamanoOct 27, 2026
  11. Pablo SabaterOct 27, 2026
  12. Junio C HamanoOct 27, 2026

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.