[PATCH RFC 0/3] cat-file: make remote-object-info request in batches
- From
Pablo Sabater <pabloosabaterr@gmail.com>
- Date
- Oct 27, 2026, 03:16 UTC
- Message-ID
- <20261027-remote-object-info-batch-v1-0-ea7a1f0b1df1@gmail.com>
Currently the command remote-object-info of git cat-file has a hard limit of 10k OIDs to query in the same command. So, if you want to query 100k objects for example you would have to run the command 10 times.
This seemed strange to me because git backfill for example has a default limit of 50k objects and backfill fetches full objects, while remote-object-info only requests metadata (size, type, both or none) so it should be able to handle at least the same, if not more.
Even if we kept the 10k OID limit for remote-object-info, it is annoying having to run the command multiple times, so I propose to allow as many OIDs as wanted and batch them internally.
Regarding which number of objects should be the limit, I benchmarked different batch sizes over a simulated 47ms RTT (details in patch 2):
+------------+---------------------+ | | objects requested | | batch size | 100k | 400k | +------------+----------+----------+ | 10k | 1.691s | 5.862s | | 50k | 0.981s | 2.397s | | 100k | 0.915s | 2.058s | | 200k | 0.924s | 1.889s | | 400k | 0.908s | 1.820s | +------------+----------+----------+
A noticeable difference can be seen from a 10k OID batch to a 50k batch. After that the difference isn't as noticeable (the number of batches to be requested isn't that big).
I propose 50k OIDs as the default number of OIDs for the remote-object-info batches. Coincidentally, git backfill's default batch size is 50k as well though it's unrelated.
I have barely done any benchmarking before, so I would appreciate any feedback on the one in patch 2. Is there a better way to do it, or other things I should take into account?
Should the server also check for a limit on the number of OIDs it accepts in a single object-info request?
GitHub CI passed: https://github.com/pabloosabaterr/git/actions/runs/40424912154
Signed-off-by: Pablo Sabater <pabloosabaterr@gmail.com>
---
Pablo Sabater (3):
cat-file: extract the transport ownership from get_remote_info()
cat-file: make remote-object-info request in batches
cat-file: add --remote-max-batch optionDocumentation/git-cat-file.adoc | 16 ++- builtin/cat-file.c | 174 +++++++++++++++++++-------------- t/t1017-cat-file-remote-object-info.sh | 68 +++++++++++++ transport-helper.c | 13 ++- transport-internal.h | 6 +- transport.c | 33 +++---- transport.h | 6 +- 7 files changed, 208 insertions(+), 108 deletions(-)
--- base-commit: 37b3475fb530c14109211faca33716a173db61ed change-id: 20261015-remote-object-info-batch-c65f13fb5a95