threads / patch / 66843

patchhttp-backend: drop strbuf pre-sizing from get_info_packs()

Subject: [PATCH] http-backend: drop strbuf pre-sizing from get_info_packs()

## tl;dr

Yusuke Hayashi's cleanup removes a SHA-1-specific size estimate with unchecked arithmetic from the dumb-HTTP info/packs handler, and a second version explains why it does not just fix the formula. Read the story.

replies: 5people: 4as markdown or json

Yusuke Hayashi via B4 Relay· Nov 13, 2026, 02:17 UTC · lore
From: Yusuke Hayashi <yusuke8h@gmail.com>

get_info_packs() iterates over all packs in the repository to count local packs, calls `strbuf_grow(&buf, cnt * 53 + 2)`, and then iterates over the packs a second time to format each entry with `strbuf_addf()`.

This pre-sizing dates back to 2f4038ab33 (Git-aware CGI to provide dumb HTTP transport, 2009-10-30) and is no longer worth keeping: `53` assumes the length of a SHA-1 "P pack-<hex>.pack\n" line (under-counting the 77-byte lines in SHA-256 repositories), `cnt * 53 + 2` uses unchecked arithmetic, and `strbuf_addf()` / `strbuf_addch()` already grow the buffer automatically without needing an extra pass over the pack list. Similar pre-sizing was recently removed in a9ce8526dc (pretty: drop strbuf pre-sizing from add_rfc2047(), 2026-05-12).

Drop `cnt`, the counting loop, and the `strbuf_grow()` call.
Signed-off-by: Yusuke Hayashi <yusuke8h@gmail.com>
---
Inspired by https://lore.kernel.org/git/20260513185408.GA147423@coredump.intra.peff.net/
---
 http-backend.c | 7 -------
 1 file changed, 7 deletions(-)
Show changes to http-backend.c +0 −7
diff --git a/http-backend.c b/http-backend.c
index c7566b1d12..da9516a4c1 100644
--- a/http-backend.c
+++ b/http-backend.c
@@ -612,15 +612,8 @@ static void get_info_packs(struct strbuf *hdr, char *arg UNUSED)
 	size_t objdirlen = strlen(repo_get_object_directory(the_repository));
 	struct strbuf buf = STRBUF_INIT;
 	struct packed_git *p;
-	size_t cnt = 0;
 
 	select_getanyfile(hdr);
-	repo_for_each_pack(the_repository, p) {
-		if (p->pack_local)
-			cnt++;
-	}
-
-	strbuf_grow(&buf, cnt * 53 + 2);
 	repo_for_each_pack(the_repository, p) {
 		if (p->pack_local)
 			strbuf_addf(&buf, "P %s\n", p->pack_name + objdirlen + 6);

---
base-commit: 9354c5f04055932909b660a237905b360cba195a
change-id: 20261113-http-backend-drop-strbuf-presizing-6f496552d0ca
Patrick Steinhardt· Nov 13, 2026, 07:03 UTC · re: Yusuke Hayashi via B4 Relay · lore

Re: [PATCH] http-backend: drop strbuf pre-sizing from get_info_packs()

On Fri, Nov 13, 2026 at 02:17:58AM +0000, Yusuke Hayashi via B4 Relay wrote:
Show 14 quoted lines
> From: Yusuke Hayashi <yusuke8h@gmail.com>
> 
> get_info_packs() iterates over all packs in the repository to count
> local packs, calls `strbuf_grow(&buf, cnt * 53 + 2)`, and then iterates
> over the packs a second time to format each entry with `strbuf_addf()`.
> 
> This pre-sizing dates back to 2f4038ab33 (Git-aware CGI to provide dumb
> HTTP transport, 2009-10-30) and is no longer worth keeping: `53` assumes
> the length of a SHA-1 "P pack-<hex>.pack\n" line (under-counting the
> 77-byte lines in SHA-256 repositories), `cnt * 53 + 2` uses unchecked
> arithmetic, and `strbuf_addf()` / `strbuf_addch()` already grow the
> buffer automatically without needing an extra pass over the pack list.
> Similar pre-sizing was recently removed in a9ce8526dc (pretty: drop
> strbuf pre-sizing from add_rfc2047(), 2026-05-12).

So this explains that the behaviour is buggy, but that may lead the reader to wonder why we don't just fix the bug. And sure, we know to grow the buffer automatically, but the hint is intended as a performance improvement so that we know to allocate the buffer properly right from the start.

Note that I'm not saying that this change doesn't make sense. But I think the argument that "others do it" is rather weak. The more important question to answer is why it's fine to skip it in this particular function.

Thanks!
Patrick
Yusuke Hayashi· Nov 15, 2026, 09:58 UTC · re: Patrick Steinhardt · lore

Re: [PATCH] http-backend: drop strbuf pre-sizing from get_info_packs()

On Fri, Nov 13, 2026 at 08:03:52AM +0100, Patrick Steinhardt wrote:
Show 10 quoted lines
> So this explains that the behaviour is buggy, but that may lead the
> reader to wonder why we don't just fix the bug. And sure, we know to
> grow the buffer automatically, but the hint is intended as a performance
> improvement so that we know to allocate the buffer properly right from
> the start.
>
> Note that I'm not saying that this change doesn't make sense. But I
> think the argument that "others do it" is rather weak. The more
> important question to answer is why it's fine to skip it in this
> particular function.

Fair point -- pointing out the SHA-256 under-count and unchecked multiplication without explaining why we don't just fix the formula leaves the main question unanswered, and citing a9ce8526dc doesn't answer it for get_info_packs() either.

In this case, get_info_packs() is a one-shot CGI handler for dumb HTTP GET /objects/info/packs requests that formats a single response and exits. A typical repository only produces a few kilobytes of output, and even with thousands of packs strbuf's geometric growth only needs a couple dozen reallocations, which is negligible compared to CGI startup, scanning objects/pack to populate the pack list, and writing the HTTP response. Meanwhile, computing the hint up front requires an extra pass over the pack list.

I'll send a v2 with an updated commit message that explains this directly instead.

Thanks for the review!
Yusuke Hayashi via B4 Relay· Nov 15, 2026, 09:58 UTC · re: Yusuke Hayashi via B4 Relay · lore

[PATCH v2] http-backend: drop strbuf pre-sizing from get_info_packs()

From: Yusuke Hayashi <yusuke8h@gmail.com>

`get_info_packs()` iterates over all packs in the repository to count local packs, calls `strbuf_grow(&buf, cnt * 53 + 2)`, and then iterates over the packs a second time to format each entry with `strbuf_addf()`. This pre-sizing dates back to 2f4038ab33 (Git-aware CGI to provide dumb HTTP transport, 2009-10-30): `53` assumes the length of a SHA-1 "P pack-<hex>.pack\n" line (under-counting the 77-byte lines in SHA-256 repositories), and `cnt * 53 + 2` uses unchecked arithmetic.

Rather than fixing the size calculation with `the_hash_algo->hexsz` and `st_mult()` / `st_add()`, drop the pre-sizing altogether. `get_info_packs()` is a one-shot CGI handler for dumb HTTP `/objects/info/packs` requests that formats a single response and exits. A typical repository only has a few dozen packs (a few kilobytes of output), and even with thousands of packs `strbuf`'s geometric growth only performs a couple dozen reallocations. That cost is negligible next to CGI startup, scanning `objects/pack` to populate the pack list, and writing the HTTP response, whereas computing `cnt` up front requires an extra pass over the pack list.

Drop `cnt`, the counting loop, and the `strbuf_grow()` call, and let `strbuf_addf()` and `strbuf_addch()` grow the buffer as needed.

Signed-off-by: Yusuke Hayashi <yusuke8h@gmail.com>
---
Inspired by https://lore.kernel.org/git/20260513185408.GA147423@coredump.intra.peff.net/
---
Changes in v2:
- Rewrite the commit message to explain why skipping pre-sizing is fine
  in `get_info_packs()` specifically (one-shot CGI handler where
  `strbuf`'s geometric growth is negligible next to CGI startup, pack
  discovery, and HTTP I/O), rather than citing `a9ce8526dc` as precedent.
- No code changes.
- Link to v1: https://lore.kernel.org/git/20261113-http-backend-drop-strbuf-presizing-v1-1-7d66aae03b22@gmail.com/
---
Range-diff versus v1:
1:  9abfe3635b ! 1:  7561d02b83 http-backend: drop strbuf pre-sizing from get_info_packs()
    @@ Metadata
      ## Commit message ##
         http-backend: drop strbuf pre-sizing from get_info_packs()
     
    -    get_info_packs() iterates over all packs in the repository to count
    +    `get_info_packs()` iterates over all packs in the repository to count
         local packs, calls `strbuf_grow(&buf, cnt * 53 + 2)`, and then iterates
         over the packs a second time to format each entry with `strbuf_addf()`.
    -
         This pre-sizing dates back to 2f4038ab33 (Git-aware CGI to provide dumb
    -    HTTP transport, 2009-10-30) and is no longer worth keeping: `53` assumes
    -    the length of a SHA-1 "P pack-<hex>.pack\n" line (under-counting the
    -    77-byte lines in SHA-256 repositories), `cnt * 53 + 2` uses unchecked
    -    arithmetic, and `strbuf_addf()` / `strbuf_addch()` already grow the
    -    buffer automatically without needing an extra pass over the pack list.
    -    Similar pre-sizing was recently removed in a9ce8526dc (pretty: drop
    -    strbuf pre-sizing from add_rfc2047(), 2026-05-12).
    +    HTTP transport, 2009-10-30): `53` assumes the length of a SHA-1
    +    "P pack-<hex>.pack\n" line (under-counting the 77-byte lines in SHA-256
    +    repositories), and `cnt * 53 + 2` uses unchecked arithmetic.
    +
    +    Rather than fixing the size calculation with `the_hash_algo->hexsz` and
    +    `st_mult()` / `st_add()`, drop the pre-sizing altogether.
    +    `get_info_packs()` is a one-shot CGI handler for dumb HTTP
    +    `/objects/info/packs` requests that formats a single response and exits.
    +    A typical repository only has a few dozen packs (a few kilobytes of
    +    output), and even with thousands of packs `strbuf`'s geometric growth
    +    only performs a couple dozen reallocations. That cost is negligible next
    +    to CGI startup, scanning `objects/pack` to populate the pack list, and
    +    writing the HTTP response, whereas computing `cnt` up front requires an
    +    extra pass over the pack list.
     
    -    Drop `cnt`, the counting loop, and the `strbuf_grow()` call.
    +    Drop `cnt`, the counting loop, and the `strbuf_grow()` call, and let
    +    `strbuf_addf()` and `strbuf_addch()` grow the buffer as needed.
     
         Signed-off-by: Yusuke Hayashi <yusuke8h@gmail.com>
     
---
 http-backend.c | 7 -------
 1 file changed, 7 deletions(-)
Show changes to http-backend.c +0 −7
diff --git a/http-backend.c b/http-backend.c
index c7566b1d12..da9516a4c1 100644
--- a/http-backend.c
+++ b/http-backend.c
@@ -612,15 +612,8 @@ static void get_info_packs(struct strbuf *hdr, char *arg UNUSED)
 	size_t objdirlen = strlen(repo_get_object_directory(the_repository));
 	struct strbuf buf = STRBUF_INIT;
 	struct packed_git *p;
-	size_t cnt = 0;
 
 	select_getanyfile(hdr);
-	repo_for_each_pack(the_repository, p) {
-		if (p->pack_local)
-			cnt++;
-	}
-
-	strbuf_grow(&buf, cnt * 53 + 2);
 	repo_for_each_pack(the_repository, p) {
 		if (p->pack_local)
 			strbuf_addf(&buf, "P %s\n", p->pack_name + objdirlen + 6);

---
base-commit: 9354c5f04055932909b660a237905b360cba195a
change-id: 20261113-http-backend-drop-strbuf-presizing-6f496552d0ca
Patrick Steinhardt· Nov 16, 2026, 06:29 UTC · re: Yusuke Hayashi via B4 Relay · lore

Re: [PATCH v2] http-backend: drop strbuf pre-sizing from get_info_packs()

On Sun, Nov 15, 2026 at 09:58:47AM +0000, Yusuke Hayashi via B4 Relay wrote:
Show 20 quoted lines
> From: Yusuke Hayashi <yusuke8h@gmail.com>
> 
> `get_info_packs()` iterates over all packs in the repository to count
> local packs, calls `strbuf_grow(&buf, cnt * 53 + 2)`, and then iterates
> over the packs a second time to format each entry with `strbuf_addf()`.
> This pre-sizing dates back to 2f4038ab33 (Git-aware CGI to provide dumb
> HTTP transport, 2009-10-30): `53` assumes the length of a SHA-1
> "P pack-<hex>.pack\n" line (under-counting the 77-byte lines in SHA-256
> repositories), and `cnt * 53 + 2` uses unchecked arithmetic.
> 
> Rather than fixing the size calculation with `the_hash_algo->hexsz` and
> `st_mult()` / `st_add()`, drop the pre-sizing altogether.
> `get_info_packs()` is a one-shot CGI handler for dumb HTTP
> `/objects/info/packs` requests that formats a single response and exits.
> A typical repository only has a few dozen packs (a few kilobytes of
> output), and even with thousands of packs `strbuf`'s geometric growth
> only performs a couple dozen reallocations. That cost is negligible next
> to CGI startup, scanning `objects/pack` to populate the pack list, and
> writing the HTTP response, whereas computing `cnt` up front requires an
> extra pass over the pack list.

Yeah, that's a much more convincing message. The other angle that this is missing a bit is that giving a size hint ensures that we allocate exactly as many bytes as we need. But that shouldn't matter in the general case, and if you have to allocate gigabytes of memory to store the list of packfiles... well, then you really have bigger problems to worry about.

So I think this version is good enough, thanks!
Patrick
Junio C Hamano· Nov 16, 2026, 15:24 UTC · re: Patrick Steinhardt · lore

Re: [PATCH v2] http-backend: drop strbuf pre-sizing from get_info_packs()

Patrick Steinhardt <ps@pks.im> writes:
Show 11 quoted lines
>> This pre-sizing dates back to 2f4038ab33 (Git-aware CGI to provide dumb
>> HTTP transport, 2009-10-30): `53` assumes the length of a SHA-1
>> "P pack-<hex>.pack\n" line (under-counting the 77-byte lines in SHA-256
>> repositories), and `cnt * 53 + 2` uses unchecked arithmetic.
>>  ...
> Yeah, that's a much more convincing message. The other angle that this
> is missing a bit is that giving a size hint ensures that we allocate
> exactly as many bytes as we need. But that shouldn't matter in the
> general case, and if you have to allocate gigabytes of memory to store
> the list of packfiles... well, then you really have bigger problems to
> worry about.

Yeah, miscounting the preallocation is not a bug and just doing pointless work. I agree with you that for the purpose of sizing strbuf, you have a larger issue if you have to worry about overflowing arithmetic.

> So I think this version is good enough, thanks!
Thanks.

← back to recent threads