git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH RFC 0/3] cat-file: make remote-object-info request in batches

From
Pablo Sabater <pabloosabaterr@gmail.com>
Date
Oct 27, 2026, 03:16 UTC
Message-ID
<20261027-remote-object-info-batch-v1-0-ea7a1f0b1df1@gmail.com>

Currently the command remote-object-info of git cat-file has a hard limit of 10k OIDs to query in the same command. So, if you want to query 100k objects for example you would have to run the command 10 times.

This seemed strange to me because git backfill for example has a default limit of 50k objects and backfill fetches full objects, while remote-object-info only requests metadata (size, type, both or none) so it should be able to handle at least the same, if not more.

Even if we kept the 10k OID limit for remote-object-info, it is annoying having to run the command multiple times, so I propose to allow as many OIDs as wanted and batch them internally.

Regarding which number of objects should be the limit, I benchmarked different batch sizes over a simulated 47ms RTT (details in patch 2):

+------------+---------------------+
|            | objects requested   |
| batch size |   100k   |   400k   |
+------------+----------+----------+
| 10k        |  1.691s  |  5.862s  |
| 50k        |  0.981s  |  2.397s  |
| 100k       |  0.915s  |  2.058s  |
| 200k       |  0.924s  |  1.889s  |
| 400k       |  0.908s  |  1.820s  |
+------------+----------+----------+

A noticeable difference can be seen from a 10k OID batch to a 50k batch. After that the difference isn't as noticeable (the number of batches to be requested isn't that big).

I propose 50k OIDs as the default number of OIDs for the remote-object-info batches. Coincidentally, git backfill's default batch size is 50k as well though it's unrelated.

I have barely done any benchmarking before, so I would appreciate any feedback on the one in patch 2. Is there a better way to do it, or other things I should take into account?

Should the server also check for a limit on the number of OIDs it accepts in a single object-info request?

GitHub CI passed: https://github.com/pabloosabaterr/git/actions/runs/40424912154

Signed-off-by: Pablo Sabater <pabloosabaterr@gmail.com>
---
Pablo Sabater (3):
      cat-file: extract the transport ownership from get_remote_info()
      cat-file: make remote-object-info request in batches
      cat-file: add --remote-max-batch option
 Documentation/git-cat-file.adoc        |  16 ++-
 builtin/cat-file.c                     | 174 +++++++++++++++++++--------------
 t/t1017-cat-file-remote-object-info.sh |  68 +++++++++++++
 transport-helper.c                     |  13 ++-
 transport-internal.h                   |   6 +-
 transport.c                            |  33 +++----
 transport.h                            |   6 +-
 7 files changed, 208 insertions(+), 108 deletions(-)

--- base-commit: 37b3475fb530c14109211faca33716a173db61ed change-id: 20261015-remote-object-info-batch-c65f13fb5a95

Next: Pablo Sabater
Message 1 of 12 in “cat-file: make remote-object-info request in batches”
  1. 0/3 cat-file: make remote-object-info request in batchesPablo Sabater, Oct 27, 2026
  2. 1/3 cat-file: extract the transport ownership from get_remote_info()Pablo Sabater, Oct 27, 2026
  3. Junio C HamanoOct 27, 2026
  4. Pablo SabaterOct 27, 2026
  5. Junio C HamanoOct 28, 2026
  6. Pablo SabaterOct 28, 2026
  7. Junio C HamanoOct 28, 2026
  8. 2/3 cat-file: make remote-object-info request in batchesPablo Sabater, Oct 27, 2026
  9. 3/3 cat-file: add --remote-max-batch optionPablo Sabater, Oct 27, 2026
  10. Junio C HamanoOct 27, 2026
  11. Pablo SabaterOct 27, 2026
  12. Junio C HamanoOct 27, 2026

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.