{"thread":{"id":"66843","subject":"[PATCH] http-backend: drop strbuf pre-sizing from get_info_packs()","startedAt":"2026-11-13T02:17:58Z","lastAt":"2026-11-16T15:24:35Z","messageCount":6,"participants":["Yusuke Hayashi via B4 Relay","Patrick Steinhardt","Yusuke Hayashi","Junio C Hamano"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"558365","messageId":"20261113-http-backend-drop-strbuf-presizing-v1-1-7d66aae03b22@gmail.com","threadId":"66843","inReplyTo":null,"subject":"[PATCH] http-backend: drop strbuf pre-sizing from get_info_packs()","fromName":"Yusuke Hayashi via B4 Relay","fromEmail":"devnull+yusuke8h.gmail.com@kernel.org","sentAt":"2026-11-13T02:17:58Z","receivedAt":"2026-11-13T02:17:58Z","isPatch":true,"sender":{"key":"name:Yusuke Hayashi","avatar":null},"body":"From: Yusuke Hayashi <yusuke8h@gmail.com>\n\nget_info_packs() iterates over all packs in the repository to count\nlocal packs, calls `strbuf_grow(&buf, cnt * 53 + 2)`, and then iterates\nover the packs a second time to format each entry with `strbuf_addf()`.\n\nThis pre-sizing dates back to 2f4038ab33 (Git-aware CGI to provide dumb\nHTTP transport, 2009-10-30) and is no longer worth keeping: `53` assumes\nthe length of a SHA-1 \"P pack-<hex>.pack\\n\" line (under-counting the\n77-byte lines in SHA-256 repositories), `cnt * 53 + 2` uses unchecked\narithmetic, and `strbuf_addf()` / `strbuf_addch()` already grow the\nbuffer automatically without needing an extra pass over the pack list.\nSimilar pre-sizing was recently removed in a9ce8526dc (pretty: drop\nstrbuf pre-sizing from add_rfc2047(), 2026-05-12).\n\nDrop `cnt`, the counting loop, and the `strbuf_grow()` call.\n\nSigned-off-by: Yusuke Hayashi <yusuke8h@gmail.com>\n---\nInspired by https://lore.kernel.org/git/20260513185408.GA147423@coredump.intra.peff.net/\n---\n http-backend.c | 7 -------\n 1 file changed, 7 deletions(-)\n\ndiff --git a/http-backend.c b/http-backend.c\nindex c7566b1d12..da9516a4c1 100644\n--- a/http-backend.c\n+++ b/http-backend.c\n@@ -612,15 +612,8 @@ static void get_info_packs(struct strbuf *hdr, char *arg UNUSED)\n \tsize_t objdirlen = strlen(repo_get_object_directory(the_repository));\n \tstruct strbuf buf = STRBUF_INIT;\n \tstruct packed_git *p;\n-\tsize_t cnt = 0;\n \n \tselect_getanyfile(hdr);\n-\trepo_for_each_pack(the_repository, p) {\n-\t\tif (p->pack_local)\n-\t\t\tcnt++;\n-\t}\n-\n-\tstrbuf_grow(&buf, cnt * 53 + 2);\n \trepo_for_each_pack(the_repository, p) {\n \t\tif (p->pack_local)\n \t\t\tstrbuf_addf(&buf, \"P %s\\n\", p->pack_name + objdirlen + 6);\n\n---\nbase-commit: 9354c5f04055932909b660a237905b360cba195a\nchange-id: 20261113-http-backend-drop-strbuf-presizing-6f496552d0ca\n\n\n\n"},{"id":"558368","messageId":"ava2WJkDag6-uthA@pks.im","threadId":"66843","inReplyTo":"20261113-http-backend-drop-strbuf-presizing-v1-1-7d66aae03b22@gmail.com","subject":"Re: [PATCH] http-backend: drop strbuf pre-sizing from get_info_packs()","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-11-13T07:03:52Z","receivedAt":"2026-11-13T07:03:52Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Fri, Nov 13, 2026 at 02:17:58AM +0000, Yusuke Hayashi via B4 Relay wrote:\n> From: Yusuke Hayashi <yusuke8h@gmail.com>\n> \n> get_info_packs() iterates over all packs in the repository to count\n> local packs, calls `strbuf_grow(&buf, cnt * 53 + 2)`, and then iterates\n> over the packs a second time to format each entry with `strbuf_addf()`.\n> \n> This pre-sizing dates back to 2f4038ab33 (Git-aware CGI to provide dumb\n> HTTP transport, 2009-10-30) and is no longer worth keeping: `53` assumes\n> the length of a SHA-1 \"P pack-<hex>.pack\\n\" line (under-counting the\n> 77-byte lines in SHA-256 repositories), `cnt * 53 + 2` uses unchecked\n> arithmetic, and `strbuf_addf()` / `strbuf_addch()` already grow the\n> buffer automatically without needing an extra pass over the pack list.\n> Similar pre-sizing was recently removed in a9ce8526dc (pretty: drop\n> strbuf pre-sizing from add_rfc2047(), 2026-05-12).\n\nSo this explains that the behaviour is buggy, but that may lead the\nreader to wonder why we don't just fix the bug. And sure, we know to\ngrow the buffer automatically, but the hint is intended as a performance\nimprovement so that we know to allocate the buffer properly right from\nthe start.\n\nNote that I'm not saying that this change doesn't make sense. But I\nthink the argument that \"others do it\" is rather weak. The more\nimportant question to answer is why it's fine to skip it in this\nparticular function.\n\nThanks!\n\nPatrick\n\n"},{"id":"558493","messageId":"CAB59oQ_K=AdVk+xXVPkodP_tPfGWuLO0fqvBwJAz2tvBAxrb7g@mail.gmail.com","threadId":"66843","inReplyTo":"ava2WJkDag6-uthA@pks.im","subject":"Re: [PATCH] http-backend: drop strbuf pre-sizing from get_info_packs()","fromName":"Yusuke Hayashi","fromEmail":"yusuke8h@gmail.com","sentAt":"2026-11-15T09:58:29Z","receivedAt":"2026-11-15T09:58:29Z","isPatch":true,"sender":{"key":"yusuke8h@gmail.com","avatar":null},"body":"On Fri, Nov 13, 2026 at 08:03:52AM +0100, Patrick Steinhardt wrote:\n> So this explains that the behaviour is buggy, but that may lead the\n> reader to wonder why we don't just fix the bug. And sure, we know to\n> grow the buffer automatically, but the hint is intended as a performance\n> improvement so that we know to allocate the buffer properly right from\n> the start.\n>\n> Note that I'm not saying that this change doesn't make sense. But I\n> think the argument that \"others do it\" is rather weak. The more\n> important question to answer is why it's fine to skip it in this\n> particular function.\n\nFair point -- pointing out the SHA-256 under-count and unchecked\nmultiplication without explaining why we don't just fix the formula\nleaves the main question unanswered, and citing a9ce8526dc doesn't\nanswer it for get_info_packs() either.\n\nIn this case, get_info_packs() is a one-shot CGI handler for dumb HTTP\nGET /objects/info/packs requests that formats a single response and\nexits. A typical repository only produces a few kilobytes of output, and\neven with thousands of packs strbuf's geometric growth only needs a\ncouple dozen reallocations, which is negligible compared to CGI startup,\nscanning objects/pack to populate the pack list, and writing the HTTP\nresponse. Meanwhile, computing the hint up front requires an extra pass\nover the pack list.\n\nI'll send a v2 with an updated commit message that explains this\ndirectly instead.\n\nThanks for the review!\n\n"},{"id":"558494","messageId":"20261115-http-backend-drop-strbuf-presizing-v2-1-b7ffe1be4482@gmail.com","threadId":"66843","inReplyTo":"20261113-http-backend-drop-strbuf-presizing-v1-1-7d66aae03b22@gmail.com","subject":"[PATCH v2] http-backend: drop strbuf pre-sizing from get_info_packs()","fromName":"Yusuke Hayashi via B4 Relay","fromEmail":"devnull+yusuke8h.gmail.com@kernel.org","sentAt":"2026-11-15T09:58:47Z","receivedAt":"2026-11-15T09:58:47Z","isPatch":true,"sender":{"key":"name:Yusuke Hayashi","avatar":null},"body":"From: Yusuke Hayashi <yusuke8h@gmail.com>\n\n`get_info_packs()` iterates over all packs in the repository to count\nlocal packs, calls `strbuf_grow(&buf, cnt * 53 + 2)`, and then iterates\nover the packs a second time to format each entry with `strbuf_addf()`.\nThis pre-sizing dates back to 2f4038ab33 (Git-aware CGI to provide dumb\nHTTP transport, 2009-10-30): `53` assumes the length of a SHA-1\n\"P pack-<hex>.pack\\n\" line (under-counting the 77-byte lines in SHA-256\nrepositories), and `cnt * 53 + 2` uses unchecked arithmetic.\n\nRather than fixing the size calculation with `the_hash_algo->hexsz` and\n`st_mult()` / `st_add()`, drop the pre-sizing altogether.\n`get_info_packs()` is a one-shot CGI handler for dumb HTTP\n`/objects/info/packs` requests that formats a single response and exits.\nA typical repository only has a few dozen packs (a few kilobytes of\noutput), and even with thousands of packs `strbuf`'s geometric growth\nonly performs a couple dozen reallocations. That cost is negligible next\nto CGI startup, scanning `objects/pack` to populate the pack list, and\nwriting the HTTP response, whereas computing `cnt` up front requires an\nextra pass over the pack list.\n\nDrop `cnt`, the counting loop, and the `strbuf_grow()` call, and let\n`strbuf_addf()` and `strbuf_addch()` grow the buffer as needed.\n\nSigned-off-by: Yusuke Hayashi <yusuke8h@gmail.com>\n---\nInspired by https://lore.kernel.org/git/20260513185408.GA147423@coredump.intra.peff.net/\n---\nChanges in v2:\n- Rewrite the commit message to explain why skipping pre-sizing is fine\n  in `get_info_packs()` specifically (one-shot CGI handler where\n  `strbuf`'s geometric growth is negligible next to CGI startup, pack\n  discovery, and HTTP I/O), rather than citing `a9ce8526dc` as precedent.\n- No code changes.\n- Link to v1: https://lore.kernel.org/git/20261113-http-backend-drop-strbuf-presizing-v1-1-7d66aae03b22@gmail.com/\n---\nRange-diff versus v1:\n\n1:  9abfe3635b ! 1:  7561d02b83 http-backend: drop strbuf pre-sizing from get_info_packs()\n    @@ Metadata\n      ## Commit message ##\n         http-backend: drop strbuf pre-sizing from get_info_packs()\n     \n    -    get_info_packs() iterates over all packs in the repository to count\n    +    `get_info_packs()` iterates over all packs in the repository to count\n         local packs, calls `strbuf_grow(&buf, cnt * 53 + 2)`, and then iterates\n         over the packs a second time to format each entry with `strbuf_addf()`.\n    -\n         This pre-sizing dates back to 2f4038ab33 (Git-aware CGI to provide dumb\n    -    HTTP transport, 2009-10-30) and is no longer worth keeping: `53` assumes\n    -    the length of a SHA-1 \"P pack-<hex>.pack\\n\" line (under-counting the\n    -    77-byte lines in SHA-256 repositories), `cnt * 53 + 2` uses unchecked\n    -    arithmetic, and `strbuf_addf()` / `strbuf_addch()` already grow the\n    -    buffer automatically without needing an extra pass over the pack list.\n    -    Similar pre-sizing was recently removed in a9ce8526dc (pretty: drop\n    -    strbuf pre-sizing from add_rfc2047(), 2026-05-12).\n    +    HTTP transport, 2009-10-30): `53` assumes the length of a SHA-1\n    +    \"P pack-<hex>.pack\\n\" line (under-counting the 77-byte lines in SHA-256\n    +    repositories), and `cnt * 53 + 2` uses unchecked arithmetic.\n    +\n    +    Rather than fixing the size calculation with `the_hash_algo->hexsz` and\n    +    `st_mult()` / `st_add()`, drop the pre-sizing altogether.\n    +    `get_info_packs()` is a one-shot CGI handler for dumb HTTP\n    +    `/objects/info/packs` requests that formats a single response and exits.\n    +    A typical repository only has a few dozen packs (a few kilobytes of\n    +    output), and even with thousands of packs `strbuf`'s geometric growth\n    +    only performs a couple dozen reallocations. That cost is negligible next\n    +    to CGI startup, scanning `objects/pack` to populate the pack list, and\n    +    writing the HTTP response, whereas computing `cnt` up front requires an\n    +    extra pass over the pack list.\n     \n    -    Drop `cnt`, the counting loop, and the `strbuf_grow()` call.\n    +    Drop `cnt`, the counting loop, and the `strbuf_grow()` call, and let\n    +    `strbuf_addf()` and `strbuf_addch()` grow the buffer as needed.\n     \n         Signed-off-by: Yusuke Hayashi <yusuke8h@gmail.com>\n     \n---\n http-backend.c | 7 -------\n 1 file changed, 7 deletions(-)\n\ndiff --git a/http-backend.c b/http-backend.c\nindex c7566b1d12..da9516a4c1 100644\n--- a/http-backend.c\n+++ b/http-backend.c\n@@ -612,15 +612,8 @@ static void get_info_packs(struct strbuf *hdr, char *arg UNUSED)\n \tsize_t objdirlen = strlen(repo_get_object_directory(the_repository));\n \tstruct strbuf buf = STRBUF_INIT;\n \tstruct packed_git *p;\n-\tsize_t cnt = 0;\n \n \tselect_getanyfile(hdr);\n-\trepo_for_each_pack(the_repository, p) {\n-\t\tif (p->pack_local)\n-\t\t\tcnt++;\n-\t}\n-\n-\tstrbuf_grow(&buf, cnt * 53 + 2);\n \trepo_for_each_pack(the_repository, p) {\n \t\tif (p->pack_local)\n \t\t\tstrbuf_addf(&buf, \"P %s\\n\", p->pack_name + objdirlen + 6);\n\n---\nbase-commit: 9354c5f04055932909b660a237905b360cba195a\nchange-id: 20261113-http-backend-drop-strbuf-presizing-6f496552d0ca\n\n\n\n"},{"id":"558511","messageId":"avqi2Jpd-QT5iNU9@pks.im","threadId":"66843","inReplyTo":"20261115-http-backend-drop-strbuf-presizing-v2-1-b7ffe1be4482@gmail.com","subject":"Re: [PATCH v2] http-backend: drop strbuf pre-sizing from get_info_packs()","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-11-16T06:29:44Z","receivedAt":"2026-11-16T06:29:44Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Sun, Nov 15, 2026 at 09:58:47AM +0000, Yusuke Hayashi via B4 Relay wrote:\n> From: Yusuke Hayashi <yusuke8h@gmail.com>\n> \n> `get_info_packs()` iterates over all packs in the repository to count\n> local packs, calls `strbuf_grow(&buf, cnt * 53 + 2)`, and then iterates\n> over the packs a second time to format each entry with `strbuf_addf()`.\n> This pre-sizing dates back to 2f4038ab33 (Git-aware CGI to provide dumb\n> HTTP transport, 2009-10-30): `53` assumes the length of a SHA-1\n> \"P pack-<hex>.pack\\n\" line (under-counting the 77-byte lines in SHA-256\n> repositories), and `cnt * 53 + 2` uses unchecked arithmetic.\n> \n> Rather than fixing the size calculation with `the_hash_algo->hexsz` and\n> `st_mult()` / `st_add()`, drop the pre-sizing altogether.\n> `get_info_packs()` is a one-shot CGI handler for dumb HTTP\n> `/objects/info/packs` requests that formats a single response and exits.\n> A typical repository only has a few dozen packs (a few kilobytes of\n> output), and even with thousands of packs `strbuf`'s geometric growth\n> only performs a couple dozen reallocations. That cost is negligible next\n> to CGI startup, scanning `objects/pack` to populate the pack list, and\n> writing the HTTP response, whereas computing `cnt` up front requires an\n> extra pass over the pack list.\n\nYeah, that's a much more convincing message. The other angle that this\nis missing a bit is that giving a size hint ensures that we allocate\nexactly as many bytes as we need. But that shouldn't matter in the\ngeneral case, and if you have to allocate gigabytes of memory to store\nthe list of packfiles... well, then you really have bigger problems to\nworry about.\n\nSo I think this version is good enough, thanks!\n\nPatrick\n\n"},{"id":"558551","messageId":"xmqqy0asn6j0.fsf@gitster.g","threadId":"66843","inReplyTo":"avqi2Jpd-QT5iNU9@pks.im","subject":"Re: [PATCH v2] http-backend: drop strbuf pre-sizing from get_info_packs()","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2026-11-16T15:24:35Z","receivedAt":"2026-11-16T15:24:35Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt <ps@pks.im> writes:\n\n>> This pre-sizing dates back to 2f4038ab33 (Git-aware CGI to provide dumb\n>> HTTP transport, 2009-10-30): `53` assumes the length of a SHA-1\n>> \"P pack-<hex>.pack\\n\" line (under-counting the 77-byte lines in SHA-256\n>> repositories), and `cnt * 53 + 2` uses unchecked arithmetic.\n>>  ...\n> Yeah, that's a much more convincing message. The other angle that this\n> is missing a bit is that giving a size hint ensures that we allocate\n> exactly as many bytes as we need. But that shouldn't matter in the\n> general case, and if you have to allocate gigabytes of memory to store\n> the list of packfiles... well, then you really have bigger problems to\n> worry about.\n\nYeah, miscounting the preallocation is not a bug and just doing\npointless work.  I agree with you that for the purpose of sizing\nstrbuf, you have a larger issue if you have to worry about\noverflowing arithmetic.\n\n> So I think this version is good enough, thanks!\n\nThanks.\n\n"}]}