{"thread":{"id":"66537","subject":"[PATCH] reftable/record: bound an object record's offset count by its length","startedAt":"2026-10-14T14:58:03Z","lastAt":"2026-10-21T21:03:35Z","messageCount":10,"participants":["Matthew Suozzo via GitGitGadget","Patrick Steinhardt","m@suozzo.co","Junio C Hamano","Aniket Gupta"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"555030","messageId":"pull.2255.git.1791989883899.gitgitgadget@gmail.com","threadId":"66537","inReplyTo":null,"subject":"[PATCH] reftable/record: bound an object record's offset count by its length","fromName":"Matthew Suozzo via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2026-10-14T14:58:03Z","receivedAt":"2026-10-14T14:58:03Z","isPatch":true,"sender":{"key":"name:Matthew Suozzo","avatar":null},"body":"From: Matthew Suozzo <m@suozzo.co>\n\nAn object record's count of offsets is a 64-bit varint from the table,\nand reftable_obj_record_decode() allocates the offsets array from it\nthrough REFTABLE_ALLOC_ARRAY(), which takes a size_t. Where size_t is\n32 bits, a count of 2^32 + 1 allocates room for one offset, and the\nloop, which runs to the full count, writes an offset from the table\nfor each varint left in the block past the end of the array. Where\nsize_t is 64 bits, a large count only makes a large allocation that\nthe decode then fails to fill.\n\nNothing in Git reaches this decoder from a repository: a block of a\ntype other than the one wanted is refused before its records are\ndecoded, and the object index, the only reader of object blocks, is\nused only by the unit tests. The decoder is still wrong.\n\nEach offset takes at least a byte, so refuse a count larger than what\nis left of the record.\n\nThe Makefile did not build the reftable record unit tests (the Meson\nbuild does). Add the suite there so that the new test runs in both.\n\nAssisted-by: Claude Opus 5.5, Claude Fable 5.1\nSigned-off-by: Matthew Suozzo <m@suozzo.co>\n---\n    reftable/record: bound an object record's offset count by its length\n\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-2255%2Fmsuozzo%2Freftable-obj-record-count-v1\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-2255/msuozzo/reftable-obj-record-count-v1\nPull-Request: https://github.com/gitgitgadget/git/pull/2255\n\n Makefile                         |  1 +\n reftable/record.c                |  4 ++++\n t/unit-tests/u-reftable-record.c | 35 ++++++++++++++++++++++++++++++++\n 3 files changed, 40 insertions(+)\n\ndiff --git a/Makefile b/Makefile\nindex a96be506b5..66b7270fc0 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1551,6 +1551,7 @@ CLAR_TEST_SUITES += u-reftable-block\n CLAR_TEST_SUITES += u-reftable-merged\n CLAR_TEST_SUITES += u-reftable-pq\n CLAR_TEST_SUITES += u-reftable-readwrite\n+CLAR_TEST_SUITES += u-reftable-record\n CLAR_TEST_SUITES += u-reftable-stack\n CLAR_TEST_SUITES += u-reftable-table\n CLAR_TEST_SUITES += u-reftable-tree\ndiff --git a/reftable/record.c b/reftable/record.c\nindex 1fce441930..96eceddca7 100644\n--- a/reftable/record.c\n+++ b/reftable/record.c\n@@ -589,6 +589,10 @@ static int reftable_obj_record_decode(void *rec, struct reftable_buf key,\n \tif (count == 0)\n \t\treturn start.len - in.len;\n \n+\t/* each offset takes at least a byte */\n+\tif (count > in.len)\n+\t\treturn REFTABLE_FORMAT_ERROR;\n+\n \tREFTABLE_ALLOC_ARRAY(r->offsets, count);\n \tif (!r->offsets)\n \t\treturn REFTABLE_OUT_OF_MEMORY_ERROR;\ndiff --git a/t/unit-tests/u-reftable-record.c b/t/unit-tests/u-reftable-record.c\nindex 9c95083ef4..0165b74bcb 100644\n--- a/t/unit-tests/u-reftable-record.c\n+++ b/t/unit-tests/u-reftable-record.c\n@@ -524,6 +524,41 @@ void test_reftable_record__obj_record_roundtrip(void)\n \treftable_buf_release(&scratch);\n }\n \n+void test_reftable_record__obj_record_decode_count_larger_than_record(void)\n+{\n+\tstruct reftable_buf key = REFTABLE_BUF_INIT, scratch = REFTABLE_BUF_INIT;\n+\tstruct reftable_record rec = { .type = REFTABLE_BLOCK_TYPE_OBJ };\n+\tuint8_t buffer[32];\n+\tstruct string_view dest = { .buf = buffer, .len = sizeof(buffer) }, in;\n+\tint n;\n+\n+\t/*\n+\t * A count of offsets that a 32-bit size_t truncates to one,\n+\t * followed by two offsets.\n+\t */\n+\tn = put_var_int(&dest, ((uint64_t)1 << 32) + 1);\n+\tcl_assert_gt_i(n, 0);\n+\tstring_view_consume(&dest, n);\n+\tn = put_var_int(&dest, 1);\n+\tcl_assert_gt_i(n, 0);\n+\tstring_view_consume(&dest, n);\n+\tn = put_var_int(&dest, 1);\n+\tcl_assert_gt_i(n, 0);\n+\tstring_view_consume(&dest, n);\n+\n+\tin.buf = buffer;\n+\tin.len = sizeof(buffer) - dest.len;\n+\n+\tcl_assert_equal_i(reftable_buf_addstr(&key, \"\\x01\\x02\"), 0);\n+\tcl_assert_equal_i(reftable_record_decode(&rec, key, 0, in,\n+\t\t\t\t\t\t REFTABLE_HASH_SIZE_SHA1, &scratch),\n+\t\t\t  REFTABLE_FORMAT_ERROR);\n+\n+\treftable_record_release(&rec);\n+\treftable_buf_release(&key);\n+\treftable_buf_release(&scratch);\n+}\n+\n void test_reftable_record__index_record_comparison(void)\n {\n \tstruct reftable_record in[3] = {\n\nbase-commit: 6de20f6092dcf9bdb1c8efe03db4b70c82b423dd\n-- \ngitgitgadget\n\n"},{"id":"555394","messageId":"atXK0ska_v2rFn8W@pks.im","threadId":"66537","inReplyTo":"pull.2255.git.1791989883899.gitgitgadget@gmail.com","subject":"Re: [PATCH] reftable/record: bound an object record's offset count by its length","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-10-19T07:46:26Z","receivedAt":"2026-10-19T07:46:26Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Wed, Oct 14, 2026 at 02:58:03PM +0000, Matthew Suozzo via GitGitGadget wrote:\n> From: Matthew Suozzo <m@suozzo.co>\n> \n> An object record's count of offsets is a 64-bit varint from the table,\n> and reftable_obj_record_decode() allocates the offsets array from it\n> through REFTABLE_ALLOC_ARRAY(), which takes a size_t. Where size_t is\n> 32 bits, a count of 2^32 + 1 allocates room for one offset, and the\n> loop, which runs to the full count, writes an offset from the table\n> for each varint left in the block past the end of the array. Where\n> size_t is 64 bits, a large count only makes a large allocation that\n> the decode then fails to fill.\n> \n> Nothing in Git reaches this decoder from a repository: a block of a\n> type other than the one wanted is refused before its records are\n> decoded, and the object index, the only reader of object blocks, is\n> used only by the unit tests. The decoder is still wrong.\n> \n> Each offset takes at least a byte, so refuse a count larger than what\n> is left of the record.\n\nSorry, but this whole commit message reads very AI-generated to me, as\nit is both overly verbose and hard to parse. I would welcome it if you\ncould maybe touch up the message for normal human beings.\n\n> diff --git a/Makefile b/Makefile\n> index a96be506b5..66b7270fc0 100644\n> --- a/Makefile\n> +++ b/Makefile\n> @@ -1551,6 +1551,7 @@ CLAR_TEST_SUITES += u-reftable-block\n>  CLAR_TEST_SUITES += u-reftable-merged\n>  CLAR_TEST_SUITES += u-reftable-pq\n>  CLAR_TEST_SUITES += u-reftable-readwrite\n> +CLAR_TEST_SUITES += u-reftable-record\n>  CLAR_TEST_SUITES += u-reftable-stack\n>  CLAR_TEST_SUITES += u-reftable-table\n>  CLAR_TEST_SUITES += u-reftable-tree\n\nMakes sense, even though this should probably have been in its own\npatch.\n\n> diff --git a/reftable/record.c b/reftable/record.c\n> index 1fce441930..96eceddca7 100644\n> --- a/reftable/record.c\n> +++ b/reftable/record.c\n> @@ -589,6 +589,10 @@ static int reftable_obj_record_decode(void *rec, struct reftable_buf key,\n>  \tif (count == 0)\n>  \t\treturn start.len - in.len;\n>  \n> +\t/* each offset takes at least a byte */\n> +\tif (count > in.len)\n> +\t\treturn REFTABLE_FORMAT_ERROR;\n\nThis check doesn't really seem to match what you're describing. What you\ndescribe is an integer overflow, but you only check for a very specific\nedge ecase here. I would have expected for example that...\n\n>  \tREFTABLE_ALLOC_ARRAY(r->offsets, count);\n\n... this line already detects the overflow and then refuses the\noperation. Because `REFTABLE_ALLOC_ARRAY()` itself already tries to\ncheck for overflows.\n\nThat being said, if you take a look at `reftable_alloc_size()` you can\nsee that the input is both of type `size_t`. And _that_ is indeed broken\non platforms where `size_t` is 32 bits, as we'd potentially wrap. But\nthat fact is not even mentioned as part of the commit message at all.\n\nSo overall I'm not convinced that this fix makes sense, we should rather\nfix the source of the problem, which is `reftable_alloc_size()`.\n\nThanks!\n\nPatrick\n\n"},{"id":"555442","messageId":"41edbeff828fcb4a2f3bc903802b14e3aa66f46b@suozzo.co","threadId":"66537","inReplyTo":"atXK0ska_v2rFn8W@pks.im","subject":"Re: [PATCH] reftable/record: bound an object record's offset count by its length","fromName":"","fromEmail":"m@suozzo.co","sentAt":"2026-10-19T17:14:08Z","receivedAt":"2026-10-19T17:14:08Z","isPatch":true,"sender":{"key":"m@suozzo.co","avatar":null},"body":"October 19, 2026 at 3:46 AM, \"Patrick Steinhardt\" <ps@pks.im mailto:ps@pks.im?to=%22Patrick%20Steinhardt%22%20%3Cps%40pks.im%3E > wrote:\n\n\n> \n> On Wed, Oct 14, 2026 at 02:58:03PM +0000, Matthew Suozzo via GitGitGadget wrote:\n> \n> > \n> > From: Matthew Suozzo <m@suozzo.co>\n> >  \n> >  An object record's count of offsets is a 64-bit varint from the table,\n> >  and reftable_obj_record_decode() allocates the offsets array from it\n> >  through REFTABLE_ALLOC_ARRAY(), which takes a size_t. Where size_t is\n> >  32 bits, a count of 2^32 + 1 allocates room for one offset, and the\n> >  loop, which runs to the full count, writes an offset from the table\n> >  for each varint left in the block past the end of the array. Where\n> >  size_t is 64 bits, a large count only makes a large allocation that\n> >  the decode then fails to fill.\n> >  \n> >  Nothing in Git reaches this decoder from a repository: a block of a\n> >  type other than the one wanted is refused before its records are\n> >  decoded, and the object index, the only reader of object blocks, is\n> >  used only by the unit tests. The decoder is still wrong.\n> >  \n> >  Each offset takes at least a byte, so refuse a count larger than what\n> >  is left of the record.\n> > \n> Sorry, but this whole commit message reads very AI-generated to me, as\n> it is both overly verbose and hard to parse. I would welcome it if you\n> could maybe touch up the message for normal human beings.\n\nyeah it was AI generated and i copy-edited it. it didn't feel too excessive since i wanted to be precise about the issue's mechanism, scope (32-bit only), severity (not really reachable), and describe the fix.\n\nany pointers on how you'd reword? remove the exposition?\n\n> \n> > \n> > diff --git a/Makefile b/Makefile\n> >  index a96be506b5..66b7270fc0 100644\n> >  --- a/Makefile\n> >  +++ b/Makefile\n> >  @@ -1551,6 +1551,7 @@ CLAR_TEST_SUITES += u-reftable-block\n> >  CLAR_TEST_SUITES += u-reftable-merged\n> >  CLAR_TEST_SUITES += u-reftable-pq\n> >  CLAR_TEST_SUITES += u-reftable-readwrite\n> >  +CLAR_TEST_SUITES += u-reftable-record\n> >  CLAR_TEST_SUITES += u-reftable-stack\n> >  CLAR_TEST_SUITES += u-reftable-table\n> >  CLAR_TEST_SUITES += u-reftable-tree\n> > \n> Makes sense, even though this should probably have been in its own\n> patch.\n\nhappy to move it to a separate patch.\n\n> \n> > \n> > diff --git a/reftable/record.c b/reftable/record.c\n> >  index 1fce441930..96eceddca7 100644\n> >  --- a/reftable/record.c\n> >  +++ b/reftable/record.c\n> >  @@ -589,6 +589,10 @@ static int reftable_obj_record_decode(void *rec, struct reftable_buf key,\n> >  if (count == 0)\n> >  return start.len - in.len;\n> >  \n> >  + /* each offset takes at least a byte */\n> >  + if (count > in.len)\n> >  + return REFTABLE_FORMAT_ERROR;\n> > \n> This check doesn't really seem to match what you're describing. What you\n> describe is an integer overflow, but you only check for a very specific\n> edge ecase here. I would have expected for example that...\n> \n> > \n> > REFTABLE_ALLOC_ARRAY(r->offsets, count);\n> > \n> ... this line already detects the overflow and then refuses the\n> operation. Because `REFTABLE_ALLOC_ARRAY()` itself already tries to\n> check for overflows.\n> \n> That being said, if you take a look at `reftable_alloc_size()` you can\n> see that the input is both of type `size_t`. And _that_ is indeed broken\n> on platforms where `size_t` is 32 bits, as we'd potentially wrap. But\n> that fact is not even mentioned as part of the commit message at all.\n\nisn't that \"REFTABLE_ALLOC_ARRAY(), which takes a size_t\" in the original message?\n\n> \n> So overall I'm not convinced that this fix makes sense, we should rather\n> fix the source of the problem, which is `reftable_alloc_size()`.\n\nthat felt like a pretty disruptive change i was hesitant to put up as my first patch but i'd be happy to make the change.\n\n> \n> Thanks!\n> \n> Patrick\n>\n\n"},{"id":"555499","messageId":"atcKOhm0Kd5XryAg@pks.im","threadId":"66537","inReplyTo":"41edbeff828fcb4a2f3bc903802b14e3aa66f46b@suozzo.co","subject":"Re: [PATCH] reftable/record: bound an object record's offset count by its length","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-10-20T06:29:14Z","receivedAt":"2026-10-20T06:29:14Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, Oct 19, 2026 at 05:14:08PM +0000, m@suozzo.co wrote:\n> October 19, 2026 at 3:46 AM, \"Patrick Steinhardt\" <ps@pks.im mailto:ps@pks.im?to=%22Patrick%20Steinhardt%22%20%3Cps%40pks.im%3E > wrote:\n> > On Wed, Oct 14, 2026 at 02:58:03PM +0000, Matthew Suozzo via GitGitGadget wrote:\n> > > From: Matthew Suozzo <m@suozzo.co>\n> > >  \n> > >  An object record's count of offsets is a 64-bit varint from the table,\n> > >  and reftable_obj_record_decode() allocates the offsets array from it\n> > >  through REFTABLE_ALLOC_ARRAY(), which takes a size_t. Where size_t is\n> > >  32 bits, a count of 2^32 + 1 allocates room for one offset, and the\n> > >  loop, which runs to the full count, writes an offset from the table\n> > >  for each varint left in the block past the end of the array. Where\n> > >  size_t is 64 bits, a large count only makes a large allocation that\n> > >  the decode then fails to fill.\n> > >  \n> > >  Nothing in Git reaches this decoder from a repository: a block of a\n> > >  type other than the one wanted is refused before its records are\n> > >  decoded, and the object index, the only reader of object blocks, is\n> > >  used only by the unit tests. The decoder is still wrong.\n> > >  \n> > >  Each offset takes at least a byte, so refuse a count larger than what\n> > >  is left of the record.\n> > > \n> > Sorry, but this whole commit message reads very AI-generated to me, as\n> > it is both overly verbose and hard to parse. I would welcome it if you\n> > could maybe touch up the message for normal human beings.\n> \n> yeah it was AI generated and i copy-edited it. it didn't feel too\n> excessive since i wanted to be precise about the issue's mechanism,\n> scope (32-bit only), severity (not really reachable), and describe the\n> fix.\n> \n> any pointers on how you'd reword? remove the exposition?\n\nIt's hard to say. With AI-generated commit messages you very often have\nthe feeling that you end up understanding everything and nothing. It\njust feels off in a way that is hard to describe, and consequently it's\nalso hard to say what exactly needs changing.\n\nMy typical suggestion is to not use AI to write your commit message at\nall and instead write it yourself. Commit messages have always been the\nplace where you can demonstrate that you understand what you are doing\nas part of the patch by properly explaining it in a way that another\nhuman being understands it. And I think that role is becoming even more\nimportant now: in a world where we cannot trust that the contributor has\nthe slightest understanding of what they are contributing, they need to\ngo the extra mile to prove it to us by having a well-written explanation\nthereof.\n\nI know that this is putting more of a burden on contributors, and I'm\nnot a fan of that. But I think it's becoming abundantly clear that AI\ndoes not really give open communities much of a choice, at least not if\nthey want to keep standards high.\n\n> > > diff --git a/reftable/record.c b/reftable/record.c\n> > >  index 1fce441930..96eceddca7 100644\n> > >  --- a/reftable/record.c\n> > >  +++ b/reftable/record.c\n> > >  @@ -589,6 +589,10 @@ static int reftable_obj_record_decode(void *rec, struct reftable_buf key,\n> > >  if (count == 0)\n> > >  return start.len - in.len;\n> > >  \n> > >  + /* each offset takes at least a byte */\n> > >  + if (count > in.len)\n> > >  + return REFTABLE_FORMAT_ERROR;\n> > > \n> > This check doesn't really seem to match what you're describing. What you\n> > describe is an integer overflow, but you only check for a very specific\n> > edge ecase here. I would have expected for example that...\n> > \n> > > \n> > > REFTABLE_ALLOC_ARRAY(r->offsets, count);\n> > > \n> > ... this line already detects the overflow and then refuses the\n> > operation. Because `REFTABLE_ALLOC_ARRAY()` itself already tries to\n> > check for overflows.\n> > \n> > That being said, if you take a look at `reftable_alloc_size()` you can\n> > see that the input is both of type `size_t`. And _that_ is indeed broken\n> > on platforms where `size_t` is 32 bits, as we'd potentially wrap. But\n> > that fact is not even mentioned as part of the commit message at all.\n> \n> isn't that \"REFTABLE_ALLOC_ARRAY(), which takes a size_t\" in the original message?\n\nUgh, no idea how I missed that. Probably because my brain was sucked out\nwhile reading it.\n\n> > So overall I'm not convinced that this fix makes sense, we should rather\n> > fix the source of the problem, which is `reftable_alloc_size()`.\n> \n> that felt like a pretty disruptive change i was hesitant to put up as\n> my first patch but i'd be happy to make the change.\n\nHm, I wouldn't call it disruptive but rather holistic. By fixing that\none function we also implicitly fix all the other callers that may\ncurrently be wrong. And in the end nothing really changes for callers\nanyway: I'd change the parameters to be of type `uintmax_t`, but the\nreturn value should probably remain a `size_t`. So we'd continue to work\njust fine in all cases where there is no overflow of `size_t`, but we'd\nproperly detect cases where the parameters themselves already overflow\nit.\n\nPatrick\n\n"},{"id":"555556","messageId":"xmqqqzhkxve6.fsf@gitster.g","threadId":"66537","inReplyTo":"atcKOhm0Kd5XryAg@pks.im","subject":"Re: [PATCH] reftable/record: bound an object record's offset count by its length","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2026-10-20T15:08:01Z","receivedAt":"2026-10-20T15:08:01Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt <ps@pks.im> writes:\n\n> It's hard to say. With AI-generated commit messages you very often have\n> the feeling that you end up understanding everything and nothing. It\n> just feels off in a way that is hard to describe, and consequently it's\n> also hard to say what exactly needs changing.\n>\n> My typical suggestion is to not use AI to write your commit message at\n> all and instead write it yourself. Commit messages have always been the\n> place where you can demonstrate that you understand what you are doing\n> as part of the patch by properly explaining it in a way that another\n> human being understands it. And I think that role is becoming even more\n> important now: in a world where we cannot trust that the contributor has\n> the slightest understanding of what they are contributing, they need to\n> go the extra mile to prove it to us by having a well-written explanation\n> thereof.\n>\n> I know that this is putting more of a burden on contributors, and I'm\n> not a fan of that. But I think it's becoming abundantly clear that AI\n> does not really give open communities much of a choice, at least not if\n> they want to keep standards high.\n\nPutting more burden on the user is fine, and the AI output, when\nused correctly, can be not so bad.\n\nHaving said that, the real problem is that nobody seems to guide\ngenerative AI machinery well enough to make intelligible output.  In\norder for a user to guide the machinery well enough, they must know\nwhat they want to discuss, and the output would be different from\nwhat a clueless and hands-off \"here is a bug report from somewhere I\nsaw, I do not understand the system well enough, solve it and\ndescribe the solution\" prompt would result in.\n\nWriting the first draft manually and having a machine check and\ncorrect typos and grammar is not a bad practice, as you remain in\ncontrol of the overall structure and contents of the final result.\n\nGoing in the other direction?  If you are not familiar with the area\nand are easily fooled by intelligent-sounding nonsense uttered by\ngenerative AI, correcting such nonsense into a usable shape would be\ndifficult, and would make those on the receiving end who have to\nread it miserable.\n\nBy the way, I cannot help but wonder what happened to the AI policy\ndiscussion that the contributors' summit promised we would have\nsoon.  Who is organizing it?\n\n"},{"id":"555559","messageId":"ateEgnW-0M0-O-UP@pks.im","threadId":"66537","inReplyTo":"xmqqqzhkxve6.fsf@gitster.g","subject":"Re: [PATCH] reftable/record: bound an object record's offset count by its length","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-10-20T15:10:58Z","receivedAt":"2026-10-20T15:10:58Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Oct 20, 2026 at 08:08:01AM -0700, Junio C Hamano wrote:\n> Patrick Steinhardt <ps@pks.im> writes:\n[snip]\n> By the way, I cannot help but wonder what happened to the AI policy\n> discussion that the contributors' summit promised we would have\n> soon.  Who is organizing it?\n\nI think Emily wanted to kick it off, if I'm not mistaken.\n\nPatrick\n\n"},{"id":"555583","messageId":"20261020165442.59954-1-aniketgupta3001@gmail.com","threadId":"66537","inReplyTo":"atcKOhm0Kd5XryAg@pks.im","subject":"Re: [PATCH] reftable/record: bound an object record's offset count by its length","fromName":"Aniket Gupta","fromEmail":"aniketgupta3001@gmail.com","sentAt":"2026-10-20T16:54:42Z","receivedAt":"2026-10-20T16:54:42Z","isPatch":true,"sender":{"key":"aniketgupta3001@gmail.com","avatar":null},"body":"Hi Matthew,\n\n> By fixing that one function we also implicitly fix all the other\n> callers that may currently be wrong.\n\nI tried Patrick's change on a 32 bit build and it works for\nREFTABLE_ALLOC_ARRAY. However REFTABLE_CALLOC_ARRAY goes straight to\nreftable_calloc() with a size_t and gets truncated. I think no one\npasses a 64 bit count to it so nothing breaks but maybe worth a\nmention in the commit message?\n\nAlso the new test passes because of the count > in.len check. If\nthat check goes away in v2 I think it might need a different test\nsince on 64 bit nothing truncates. Maybe we can test\nreftable_alloc_size() directly?\n\nAniket\n\n"},{"id":"555789","messageId":"pull.2255.v2.git.1792616615.gitgitgadget@gmail.com","threadId":"66537","inReplyTo":"pull.2255.git.1791989883899.gitgitgadget@gmail.com","subject":"[PATCH v2 0/2] reftable: widen allocation size arguments to uintmax_t","fromName":"Matthew Suozzo via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2026-10-21T21:03:33Z","receivedAt":"2026-10-21T21:03:33Z","isPatch":true,"sender":{"key":"name:Matthew Suozzo","avatar":null},"body":"Changes since v1:\n\n * Fix the truncation in reftable_alloc_size() and reftable_calloc() instead\n   of checking the count in reftable_obj_record_decode() (Patrick, Aniket).\n * Test reftable_alloc_size() directly instead of the decoder (Aniket).\n * Move the Makefile change into its own patch (Patrick).\n * Rewrite the commit messages (Patrick).\n\nMatthew Suozzo (2):\n  Makefile: build the reftable record unit tests\n  reftable/basics: widen allocation size arguments to uintmax_t\n\n Makefile                         |  1 +\n reftable/basics.c                |  9 +++++----\n reftable/basics.h                |  5 +++--\n t/unit-tests/u-reftable-basics.c | 18 ++++++++++++++++++\n 4 files changed, 27 insertions(+), 6 deletions(-)\n\n\nbase-commit: 6de20f6092dcf9bdb1c8efe03db4b70c82b423dd\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-2255%2Fmsuozzo%2Freftable-obj-record-count-v2\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-2255/msuozzo/reftable-obj-record-count-v2\nPull-Request: https://github.com/gitgitgadget/git/pull/2255\n\nRange-diff vs v1:\n\n 1:  5fb73ec0a8 ! 1:  0d083bc81f reftable/record: bound an object record's offset count by its length\n     @@ Metadata\n      Author: Matthew Suozzo <m@suozzo.co>\n      \n       ## Commit message ##\n     -    reftable/record: bound an object record's offset count by its length\n     +    Makefile: build the reftable record unit tests\n      \n     -    An object record's count of offsets is a 64-bit varint from the table,\n     -    and reftable_obj_record_decode() allocates the offsets array from it\n     -    through REFTABLE_ALLOC_ARRAY(), which takes a size_t. Where size_t is\n     -    32 bits, a count of 2^32 + 1 allocates room for one offset, and the\n     -    loop, which runs to the full count, writes an offset from the table\n     -    for each varint left in the block past the end of the array. Where\n     -    size_t is 64 bits, a large count only makes a large allocation that\n     -    the decode then fails to fill.\n     +    The Meson build includes the reftable record unit tests, but the\n     +    Makefile does not. The suite was dropped from the Makefile when the\n     +    test was converted to the clar framework in 2596bef5841f (2025-07-24).\n      \n     -    Nothing in Git reaches this decoder from a repository: a block of a\n     -    type other than the one wanted is refused before its records are\n     -    decoded, and the object index, the only reader of object blocks, is\n     -    used only by the unit tests. The decoder is still wrong.\n     +    Include the reftable record unit test suite in the Makefile test\n     +    target.\n      \n     -    Each offset takes at least a byte, so refuse a count larger than what\n     -    is left of the record.\n     -\n     -    The Makefile did not build the reftable record unit tests (the Meson\n     -    build does). Add the suite there so that the new test runs in both.\n     -\n     -    Assisted-by: Claude Opus 5.5, Claude Fable 5.1\n     +    Assisted-by: Claude Fable 5.1\n          Signed-off-by: Matthew Suozzo <m@suozzo.co>\n      \n       ## Makefile ##\n     @@ Makefile: CLAR_TEST_SUITES += u-reftable-block\n       CLAR_TEST_SUITES += u-reftable-stack\n       CLAR_TEST_SUITES += u-reftable-table\n       CLAR_TEST_SUITES += u-reftable-tree\n     -\n     - ## reftable/record.c ##\n     -@@ reftable/record.c: static int reftable_obj_record_decode(void *rec, struct reftable_buf key,\n     - \tif (count == 0)\n     - \t\treturn start.len - in.len;\n     - \n     -+\t/* each offset takes at least a byte */\n     -+\tif (count > in.len)\n     -+\t\treturn REFTABLE_FORMAT_ERROR;\n     -+\n     - \tREFTABLE_ALLOC_ARRAY(r->offsets, count);\n     - \tif (!r->offsets)\n     - \t\treturn REFTABLE_OUT_OF_MEMORY_ERROR;\n     -\n     - ## t/unit-tests/u-reftable-record.c ##\n     -@@ t/unit-tests/u-reftable-record.c: void test_reftable_record__obj_record_roundtrip(void)\n     - \treftable_buf_release(&scratch);\n     - }\n     - \n     -+void test_reftable_record__obj_record_decode_count_larger_than_record(void)\n     -+{\n     -+\tstruct reftable_buf key = REFTABLE_BUF_INIT, scratch = REFTABLE_BUF_INIT;\n     -+\tstruct reftable_record rec = { .type = REFTABLE_BLOCK_TYPE_OBJ };\n     -+\tuint8_t buffer[32];\n     -+\tstruct string_view dest = { .buf = buffer, .len = sizeof(buffer) }, in;\n     -+\tint n;\n     -+\n     -+\t/*\n     -+\t * A count of offsets that a 32-bit size_t truncates to one,\n     -+\t * followed by two offsets.\n     -+\t */\n     -+\tn = put_var_int(&dest, ((uint64_t)1 << 32) + 1);\n     -+\tcl_assert_gt_i(n, 0);\n     -+\tstring_view_consume(&dest, n);\n     -+\tn = put_var_int(&dest, 1);\n     -+\tcl_assert_gt_i(n, 0);\n     -+\tstring_view_consume(&dest, n);\n     -+\tn = put_var_int(&dest, 1);\n     -+\tcl_assert_gt_i(n, 0);\n     -+\tstring_view_consume(&dest, n);\n     -+\n     -+\tin.buf = buffer;\n     -+\tin.len = sizeof(buffer) - dest.len;\n     -+\n     -+\tcl_assert_equal_i(reftable_buf_addstr(&key, \"\\x01\\x02\"), 0);\n     -+\tcl_assert_equal_i(reftable_record_decode(&rec, key, 0, in,\n     -+\t\t\t\t\t\t REFTABLE_HASH_SIZE_SHA1, &scratch),\n     -+\t\t\t  REFTABLE_FORMAT_ERROR);\n     -+\n     -+\treftable_record_release(&rec);\n     -+\treftable_buf_release(&key);\n     -+\treftable_buf_release(&scratch);\n     -+}\n     -+\n     - void test_reftable_record__index_record_comparison(void)\n     - {\n     - \tstruct reftable_record in[3] = {\n -:  ---------- > 2:  8f4a8389d0 reftable/basics: widen allocation size arguments to uintmax_t\n\n-- \ngitgitgadget\n\n"},{"id":"555790","messageId":"0d083bc81fdb6665333b4095baedefd90b652473.1792616615.git.gitgitgadget@gmail.com","threadId":"66537","inReplyTo":"pull.2255.v2.git.1792616615.gitgitgadget@gmail.com","subject":"[PATCH v2 1/2] Makefile: build the reftable record unit tests","fromName":"Matthew Suozzo via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2026-10-21T21:03:34Z","receivedAt":"2026-10-21T21:03:34Z","isPatch":true,"sender":{"key":"name:Matthew Suozzo","avatar":null},"body":"From: Matthew Suozzo <m@suozzo.co>\n\nThe Meson build includes the reftable record unit tests, but the\nMakefile does not. The suite was dropped from the Makefile when the\ntest was converted to the clar framework in 2596bef5841f (2025-07-24).\n\nInclude the reftable record unit test suite in the Makefile test\ntarget.\n\nAssisted-by: Claude Fable 5.1\nSigned-off-by: Matthew Suozzo <m@suozzo.co>\n---\n Makefile | 1 +\n 1 file changed, 1 insertion(+)\n\ndiff --git a/Makefile b/Makefile\nindex a96be506b5..66b7270fc0 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1551,6 +1551,7 @@ CLAR_TEST_SUITES += u-reftable-block\n CLAR_TEST_SUITES += u-reftable-merged\n CLAR_TEST_SUITES += u-reftable-pq\n CLAR_TEST_SUITES += u-reftable-readwrite\n+CLAR_TEST_SUITES += u-reftable-record\n CLAR_TEST_SUITES += u-reftable-stack\n CLAR_TEST_SUITES += u-reftable-table\n CLAR_TEST_SUITES += u-reftable-tree\n-- \ngitgitgadget\n\n\n"},{"id":"555791","messageId":"8f4a8389d0c022de216d1086117fd23d2a8260a9.1792616615.git.gitgitgadget@gmail.com","threadId":"66537","inReplyTo":"pull.2255.v2.git.1792616615.gitgitgadget@gmail.com","subject":"[PATCH v2 2/2] reftable/basics: widen allocation size arguments to uintmax_t","fromName":"Matthew Suozzo via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2026-10-21T21:03:35Z","receivedAt":"2026-10-21T21:03:35Z","isPatch":true,"sender":{"key":"name:Matthew Suozzo","avatar":null},"body":"From: Matthew Suozzo <m@suozzo.co>\n\nreftable_alloc_size() takes its arguments as size_t, but\nreftable_obj_record_decode() passes it a uint64_t count. On 32-bit\nplatforms, a count wider than 32 bits is truncated before the overflow\ncheck, so the decoder can get a small allocation for a large count and\nwrite past its end. reftable_calloc() takes size_t arguments the same\nway.\n\nMake reftable_alloc_size() and reftable_calloc() take uintmax_t\narguments, so the check sees the full count.\n\nAssisted-by: Claude Opus 5.5\nHelped-by: Patrick Steinhardt <ps@pks.im>\nHelped-by: Aniket Gupta <aniketgupta3001@gmail.com>\nSigned-off-by: Matthew Suozzo <m@suozzo.co>\n---\n reftable/basics.c                |  9 +++++----\n reftable/basics.h                |  5 +++--\n t/unit-tests/u-reftable-basics.c | 18 ++++++++++++++++++\n 3 files changed, 26 insertions(+), 6 deletions(-)\n\ndiff --git a/reftable/basics.c b/reftable/basics.c\nindex f0442a46cf..6103b72b44 100644\n--- a/reftable/basics.c\n+++ b/reftable/basics.c\n@@ -44,18 +44,19 @@ void reftable_free(void *p)\n \t\tfree(p);\n }\n \n-void *reftable_calloc(size_t nelem, size_t elsize)\n+void *reftable_calloc(uintmax_t nelem, uintmax_t elsize)\n {\n+\tsize_t size;\n \tvoid *p;\n \n-\tif (nelem && elsize > SIZE_MAX / nelem)\n+\tif (reftable_alloc_size(nelem, elsize, &size) < 0)\n \t\treturn NULL;\n \n-\tp = reftable_malloc(nelem * elsize);\n+\tp = reftable_malloc(size);\n \tif (!p)\n \t\treturn NULL;\n \n-\tmemset(p, 0, nelem * elsize);\n+\tmemset(p, 0, size);\n \treturn p;\n }\n \ndiff --git a/reftable/basics.h b/reftable/basics.h\nindex e4b83b2b03..29d2b799ca 100644\n--- a/reftable/basics.h\n+++ b/reftable/basics.h\n@@ -184,10 +184,11 @@ size_t names_length(const char **names);\n void *reftable_malloc(size_t sz);\n void *reftable_realloc(void *p, size_t sz);\n void reftable_free(void *p);\n-void *reftable_calloc(size_t nelem, size_t elsize);\n+void *reftable_calloc(uintmax_t nelem, uintmax_t elsize);\n char *reftable_strdup(const char *str);\n \n-static inline int reftable_alloc_size(size_t nelem, size_t elsize, size_t *out)\n+static inline int reftable_alloc_size(uintmax_t nelem, uintmax_t elsize,\n+\t\t\t\t      size_t *out)\n {\n \tif (nelem && elsize > SIZE_MAX / nelem)\n \t\treturn -1;\ndiff --git a/t/unit-tests/u-reftable-basics.c b/t/unit-tests/u-reftable-basics.c\nindex c5d83b6714..37bf1ade5a 100644\n--- a/t/unit-tests/u-reftable-basics.c\n+++ b/t/unit-tests/u-reftable-basics.c\n@@ -200,6 +200,24 @@ void test_reftable_basics__put_get_be16(void)\n \tcl_assert_equal_i(in, out);\n }\n \n+void test_reftable_basics__alloc_size(void)\n+{\n+\tsize_t out;\n+\n+\tcl_assert_equal_i(reftable_alloc_size(3, 5, &out), 0);\n+\tcl_assert_equal_i(out, 15);\n+\tcl_assert(reftable_alloc_size(SIZE_MAX, 2, &out) < 0);\n+\n+\t/* Counts that do not fit in a size_t. */\n+\tif (SIZE_MAX < UINTMAX_MAX) {\n+\t\tcl_assert(reftable_alloc_size((uintmax_t)SIZE_MAX + 1,\n+\t\t\t\t\t      1, &out) < 0);\n+\t\tcl_assert(reftable_alloc_size(1, (uintmax_t)SIZE_MAX + 1,\n+\t\t\t\t\t      &out) < 0);\n+\t\tcl_assert(reftable_calloc((uintmax_t)SIZE_MAX + 1, 1) == NULL);\n+\t}\n+}\n+\n void test_reftable_basics__alloc_grow(void)\n {\n \tint *arr = NULL, *old_arr;\n-- \ngitgitgadget\n\n"}]}