{"thread":{"id":"66714","subject":"[PATCH 0/2] tree-walk: use size_t for tree sizes","startedAt":"2026-10-30T10:58:21Z","lastAt":"2026-11-18T03:47:32Z","messageCount":6,"participants":["Elia Pinto","Junio C Hamano"],"isPatch":true,"patchVersion":1,"patchTotal":2},"messages":[{"id":"556907","messageId":"cover.1793357680.git.gitter.spiros@gmail.com","threadId":"66714","inReplyTo":null,"subject":"[PATCH 0/2] tree-walk: use size_t for tree sizes","fromName":"Elia Pinto","fromEmail":"gitter.spiros@gmail.com","sentAt":"2026-10-30T10:58:21Z","receivedAt":"2026-10-30T10:58:21Z","isPatch":true,"sender":{"key":"gitter.spiros@gmail.com","avatar":"https://avatars.githubusercontent.com/u/158490?v=4"},"body":"struct tree_desc keeps the size of the tree buffer in an \"unsigned\nint\", so a tree object of 4GiB or more is truncated on all platforms,\nand struct tree adds another truncation on Windows, where \"long\" is\n32 bits. The walk then sees only a prefix of the tree: it never reads\npast the buffer, but when the cut falls between two entries the rest\nof the tree silently disappears, both for commands like \"git ls-tree\"\nand for the checks in fsck_tree().\n\nTrees that large are not something one creates by accident, but they\ncan be crafted, and fsck is what is supposed to look at every entry\nof them.\n\n 1/2 moves struct tree_desc and the functions that set it up to\n     size_t; this is the part that matters on all platforms.\n 2/2 does the same for struct tree and parse_tree_buffer().\n\nThere is no test, as it would need a tree larger than 4GiB in memory;\n1/2 describes how I checked the effect with a simulated 16-bit size.\n\nThis comes from the same search for truncated sizes as:\n\n  https://lore.kernel.org/git/cover.1793352943.git.gitter.spiros@gmail.com/\n  https://lore.kernel.org/git/cover.1793355390.git.gitter.spiros@gmail.com/\n\nIt is independent of both and merges cleanly with them. The series is\nbased on master (b9e5cff180).\n\nElia Pinto (2):\n  tree-walk: use size_t for the size of tree buffers\n  tree: use size_t for the size of parsed trees\n\n tree-walk.c | 14 +++++++-------\n tree-walk.h |  6 +++---\n tree.c      |  2 +-\n tree.h      |  4 ++--\n 4 files changed, 13 insertions(+), 13 deletions(-)\n\n\nbase-commit: b9e5cff1803ab24582201ef059516eb16014b9d9\n-- \n2.53.0\n\n\n"},{"id":"556908","messageId":"908bcf7a607195d7f5215f191a85f25ab0059e8a.1793357680.git.gitter.spiros@gmail.com","threadId":"66714","inReplyTo":"cover.1793357680.git.gitter.spiros@gmail.com","subject":"[PATCH 1/2] tree-walk: use size_t for the size of tree buffers","fromName":"Elia Pinto","fromEmail":"gitter.spiros@gmail.com","sentAt":"2026-10-30T10:58:22Z","receivedAt":"2026-10-30T10:58:22Z","isPatch":true,"sender":{"key":"gitter.spiros@gmail.com","avatar":"https://avatars.githubusercontent.com/u/158490?v=4"},"body":"struct tree_desc keeps the number of bytes left in the tree buffer in\nan \"unsigned int\", and init_tree_desc(), init_tree_desc_gently() and\ndecode_tree_entry() take the size as an \"unsigned long\". The size\ncomes from odb_read_object() as a size_t, so for a tree object of\n4GiB or more it is truncated. As the field is an \"unsigned int\", this\nhappens on all platforms, not only those where \"long\" is 32 bits.\n\nThe walk then sees only the first \"size mod 2^32\" bytes of the\nbuffer. decode_tree_entry() still checks that the NUL ending the last\nentry is within that size, so we never read past the buffer. But when\nthe cut falls between two entries, the rest of the tree silently\ndisappears: \"git ls-tree\" lists fewer entries and exits with 0, and\nfsck_tree() only checks the entries it sees, so its checks for \".git\",\n\".gitmodules\" and the like never look at the others. When the cut\nfalls inside an entry, we die with \"too-short tree\" instead.\n\nUse size_t for the size in struct tree_desc, in the functions that set\nit up and advance it, and in the struct dir_state used by\nget_tree_entry_follow_symlinks().\n\nThere is no test, as it would need a tree larger than 4GiB in memory.\nTruncating the size to 16 bits instead, in init_tree_desc_internal()\nand update_tree_entry_internal(), makes \"git ls-tree\" list 10 of the\n2058 entries of a 65856-byte tree created with \"git mktree\", without\nany error.\n\nSigned-off-by: Elia Pinto <gitter.spiros@gmail.com>\n---\n tree-walk.c | 14 +++++++-------\n tree-walk.h |  6 +++---\n 2 files changed, 10 insertions(+), 10 deletions(-)\n\ndiff --git a/tree-walk.c b/tree-walk.c\nindex a67f06b9eb..aaac1fd48d 100644\n--- a/tree-walk.c\n+++ b/tree-walk.c\n@@ -14,7 +14,7 @@\n #include \"environment.h\"\n #include \"read-cache-ll.h\"\n \n-static int decode_tree_entry(struct tree_desc *desc, const char *buf, unsigned long size, struct strbuf *err)\n+static int decode_tree_entry(struct tree_desc *desc, const char *buf, size_t size, struct strbuf *err)\n {\n \tconst char *path;\n \tunsigned int len;\n@@ -49,7 +49,7 @@ static int decode_tree_entry(struct tree_desc *desc, const char *buf, unsigned l\n \n static int init_tree_desc_internal(struct tree_desc *desc,\n \t\t\t\t   const struct object_id *oid,\n-\t\t\t\t   const void *buffer, unsigned long size,\n+\t\t\t\t   const void *buffer, size_t size,\n \t\t\t\t   struct strbuf *err,\n \t\t\t\t   enum tree_desc_flags flags)\n {\n@@ -63,7 +63,7 @@ static int init_tree_desc_internal(struct tree_desc *desc,\n }\n \n void init_tree_desc(struct tree_desc *desc, const struct object_id *tree_oid,\n-\t\t    const void *buffer, unsigned long size)\n+\t\t    const void *buffer, size_t size)\n {\n \tstruct strbuf err = STRBUF_INIT;\n \tif (init_tree_desc_internal(desc, tree_oid, buffer, size, &err, 0))\n@@ -72,7 +72,7 @@ void init_tree_desc(struct tree_desc *desc, const struct object_id *tree_oid,\n }\n \n int init_tree_desc_gently(struct tree_desc *desc, const struct object_id *oid,\n-\t\t\t  const void *buffer, unsigned long size,\n+\t\t\t  const void *buffer, size_t size,\n \t\t\t  enum tree_desc_flags flags)\n {\n \tstruct strbuf err = STRBUF_INIT;\n@@ -113,8 +113,8 @@ static int update_tree_entry_internal(struct tree_desc *desc, struct strbuf *err\n {\n \tconst void *buf = desc->buffer;\n \tconst unsigned char *end = (const unsigned char *)desc->entry.path + desc->entry.pathlen + 1 + desc->algo->rawsz;\n-\tunsigned long size = desc->size;\n-\tunsigned long len = end - (const unsigned char *)buf;\n+\tsize_t size = desc->size;\n+\tsize_t len = end - (const unsigned char *)buf;\n \n \tif (size < len)\n \t\tdie(_(\"too-short tree file\"));\n@@ -561,7 +561,7 @@ int traverse_trees(struct index_state *istate,\n \n struct dir_state {\n \tvoid *tree;\n-\tunsigned long size;\n+\tsize_t size;\n \tstruct object_id oid;\n };\n \ndiff --git a/tree-walk.h b/tree-walk.h\nindex 9646c47ac5..5ccd4a9361 100644\n--- a/tree-walk.h\n+++ b/tree-walk.h\n@@ -35,7 +35,7 @@ struct tree_desc {\n \tstruct name_entry entry;\n \n \t/* counts the number of bytes left in the `buffer`. */\n-\tunsigned int size;\n+\tsize_t size;\n \n \t/* option flags passed via init_tree_desc_gently() */\n \tenum tree_desc_flags {\n@@ -85,10 +85,10 @@ int update_tree_entry_gently(struct tree_desc *);\n  * members of `struct tree`.\n  */\n void init_tree_desc(struct tree_desc *desc, const struct object_id *tree_oid,\n-\t\t    const void *buf, unsigned long size);\n+\t\t    const void *buf, size_t size);\n \n int init_tree_desc_gently(struct tree_desc *desc, const struct object_id *oid,\n-\t\t\t  const void *buf, unsigned long size,\n+\t\t\t  const void *buf, size_t size,\n \t\t\t  enum tree_desc_flags flags);\n \n /*\n-- \n2.53.0\n\n\n"},{"id":"556909","messageId":"0b1d1c5668c89e3329e246d787ec80a161bd1dac.1793357680.git.gitter.spiros@gmail.com","threadId":"66714","inReplyTo":"cover.1793357680.git.gitter.spiros@gmail.com","subject":"[PATCH 2/2] tree: use size_t for the size of parsed trees","fromName":"Elia Pinto","fromEmail":"gitter.spiros@gmail.com","sentAt":"2026-10-30T10:58:23Z","receivedAt":"2026-10-30T10:58:23Z","isPatch":true,"sender":{"key":"gitter.spiros@gmail.com","avatar":"https://avatars.githubusercontent.com/u/158490?v=4"},"body":"struct tree stores the size of its buffer in an \"unsigned long\", which\nis only 32 bits wide on Windows, and parse_tree_buffer() takes it as\nsuch, although repo_parse_tree_gently() gets it from odb_read_object()\nas a size_t. Callers then pass tree->size to init_tree_desc(), so with\nthe previous commit a tree of 4GiB or more is still truncated on\nWindows before it gets there.\n\nUse size_t for both. parse_object_buffer() still hands an \"unsigned\nlong\" size to parse_tree_buffer(); it is shared with the other object\ntypes and is left for a later change.\n\nSigned-off-by: Elia Pinto <gitter.spiros@gmail.com>\n---\n tree.c | 2 +-\n tree.h | 4 ++--\n 2 files changed, 3 insertions(+), 3 deletions(-)\n\ndiff --git a/tree.c b/tree.c\nindex 53f7395e9f..d37b9bc7b1 100644\n--- a/tree.c\n+++ b/tree.c\n@@ -172,7 +172,7 @@ struct tree *lookup_tree(struct repository *r, const struct object_id *oid)\n \treturn object_as_type(obj, OBJ_TREE, 0);\n }\n \n-int parse_tree_buffer(struct tree *item, void *buffer, unsigned long size)\n+int parse_tree_buffer(struct tree *item, void *buffer, size_t size)\n {\n \tif (item->object.parsed)\n \t\treturn 0;\ndiff --git a/tree.h b/tree.h\nindex 677382eed8..50f0b15af4 100644\n--- a/tree.h\n+++ b/tree.h\n@@ -10,14 +10,14 @@ struct strbuf;\n struct tree {\n \tstruct object object;\n \tvoid *buffer;\n-\tunsigned long size;\n+\tsize_t size;\n };\n \n extern const char *tree_type;\n \n struct tree *lookup_tree(struct repository *r, const struct object_id *oid);\n \n-int parse_tree_buffer(struct tree *item, void *buffer, unsigned long size);\n+int parse_tree_buffer(struct tree *item, void *buffer, size_t size);\n \n #define parse_tree_gently(t, q) repo_parse_tree_gently(the_repository, t, q)\n int repo_parse_tree_gently(struct repository *r, struct tree *item,\n-- \n2.53.0\n\n\n"},{"id":"558651","messageId":"20261117214253.716770-1-gitter.spiros@gmail.com","threadId":"66714","inReplyTo":"cover.1793357680.git.gitter.spiros@gmail.com","subject":"Re: [PATCH 0/2] tree-walk: use size_t for tree sizes","fromName":"Elia Pinto","fromEmail":"gitter.spiros@gmail.com","sentAt":"2026-11-17T21:42:53Z","receivedAt":"2026-11-17T21:42:53Z","isPatch":true,"sender":{"key":"gitter.spiros@gmail.com","avatar":"https://avatars.githubusercontent.com/u/158490?v=4"},"body":"Hi,\n\nA gentle ping on this two-patch series, which has not had any comments\nso far. It still applies cleanly to current master (e1ec09b23c), and\nnothing has touched tree-walk.[ch] or tree.[ch] since it was sent.\n\nTo recap briefly: struct tree_desc keeps the size of the tree buffer in\nan \"unsigned int\", so on every platform a tree of 4GiB or more is seen\nonly up to \"size mod 2^32\". The walk never reads past the buffer, but\nwhen the cut falls between two entries the rest of the tree silently\ndisappears, for \"git ls-tree\" as well as for the checks in fsck_tree().\n2/2 does the same for struct tree, which matters only where \"long\" has\n32 bits while size_t has 64, as on Windows.\n\nI would appreciate an opinion on two questions:\n\n - Is this worth fixing at all, given that such a tree can only be\n   crafted on purpose?\n\n - If it is, is widening the types the right approach, or should a\n   tree too large for the walker rather be rejected, e.g. in\n   init_tree_desc_gently(), which fsck_tree() also uses?\n\nIf there is no interest, I am happy to drop the series.\n\nThanks,\nElia\n\n"},{"id":"558671","messageId":"xmqqmrr698dn.fsf@gitster.g","threadId":"66714","inReplyTo":"908bcf7a607195d7f5215f191a85f25ab0059e8a.1793357680.git.gitter.spiros@gmail.com","subject":"Re: [PATCH 1/2] tree-walk: use size_t for the size of tree buffers","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2026-11-18T02:32:52Z","receivedAt":"2026-11-18T02:32:52Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Elia Pinto <gitter.spiros@gmail.com> writes:\n\n> diff --git a/tree-walk.c b/tree-walk.c\n> index a67f06b9eb..aaac1fd48d 100644\n> --- a/tree-walk.c\n> +++ b/tree-walk.c\n> @@ -14,7 +14,7 @@\n>  #include \"environment.h\"\n>  #include \"read-cache-ll.h\"\n>  \n> -static int decode_tree_entry(struct tree_desc *desc, const char *buf, unsigned long size, struct strbuf *err)\n> +static int decode_tree_entry(struct tree_desc *desc, const char *buf, size_t size, struct strbuf *err)\n>  {\n>  \tconst char *path;\n>  \tunsigned int len;\n\nThis change (and all other changes) is not very interesting by\nitself.  Because the size of a tree object may not fit in unsigned\nlong but fits in size_t, it is obviously the right thing to use\nsize_t in place of unsigned long.  The only thing we may want to pay\nattention to is if arithmetic in the function that uses these\nquantities does not cause unsigned integer wraparound.  The type\nswap from unsigned long to size_t would affect the correctness (in\nother words, if the original was correct, then the move to size_t\nwould not make it incorrect), so if we assume that the original was\ncorrect, unsigned integer wraparound can be ignored.\n\nOther parts of this function, not the type of \"size\", are much more\ninteresting.\n\n\n        static int decode_tree_entry(struct tree_desc *desc, const char *buf, size_t size, struct strbuf *err)\n        {\n                const char *path;\n                unsigned int len;\n                uint16_t mode;\n                const unsigned hashsz = desc->algo->rawsz;\n\nOur buf[] starts at the beginning of a sequence of tree entries.\nEach tree entry is formatted as a variable-length record that\nlooks like this (see update_one() in 'cache-tree.c' that writes these):\n\n        <octal mode> SP <name> NUL <binary oid>\n\nThe primary caller of this function is update_tree_entry().  It\ncalls this function starting at the beginning of the inflated tree\nobject buffer in buf[] with its size in size.  This function carves\nout one tree entry of the above shape.  The caller then increments\nbuf[] by the size of the entry this function read, decrements size by\nthe same amount, and calls this function again.\n\nWith that in mind, let's see the initial \"sanity check\".\n\n                if (size < hashsz + 3 || buf[size - (hashsz + 1)]) {\n                        strbuf_addstr(err, _(\"too-short tree object\"));\n                        return -1;\n                }\n\nThe first half of the sanity check is very pessimistic.  Given that\nan octal mode must be at least one byte long (in practice, an entry\nthat records a tree object with mode 40000 has the shortest octal\nmode, which is five bytes long), SP and NUL take two bytes, and the\nname also has to be at least one byte long, hashsz (which is the\nlength of binary oid) + 3 is too short an estimate for the minimum\nlength of an entry.  It is correct to reject if size is shorter than\nthat, of course.\n\nThe latter half of the check is constant across calls.  It insists\nthat the very end of the sequence of tree entries contained in the\ninflated tree object buffer must end with NUL followed by a binary\noid.  As I explained earlier, because the caller increments buf[] by\nthe same byte count as it decrements size when it calls this\nfunction again, checking it only once (probably in init_tree_desc())\nshould be sufficient.  This does not affect correctness, but I see an\noptimization opportunity here.  #leftoverbits \n\n                path = parse_mode(buf, &mode);\n\nWe read the <octal mode> SP out of the buf, and point <name> with\nthis variable path.\n\n                if (!path) {\n                        strbuf_addstr(err, _(\"malformed mode in tree entry\"));\n                        return -1;\n                }\n                if (!*path) {\n                        strbuf_addstr(err, _(\"empty filename in tree entry\"));\n                        return -1;\n                }\n\nAfter sanity checkint the <name> part, we try to figure out its\nlength.\n\n                len = strlen(path) + 1;\n\nHere, we assume we can run strlen() on it, but have we checked that\nthere would be a NUL before we go beyond size?  Actually we did,\nsort of.  The redundantly repeated \"last tree entry that this tree\nobject records must end with NUL followed by <binary oid>\" check\nmakes sure that there is one NUL that would stop this strlen().\n\nThat does not invalidate my earlier comment on optimization\nopportunity.  As long as the caller does the right thing when\nincrementing buf[] and decrementing size, checking for that NUL just\nonce at the beginning is sufficient.\n\n                /* Initialize the descriptor entry */\n                desc->entry.path = path;\n                desc->entry.mode = (desc->flags & TREE_DESC_RAW_MODES) ? mode : canon_mode(mode);\n                desc->entry.pathlen = len - 1;\n                oidread(&desc->entry.oid, (const unsigned char *)path + len,\n                        desc->algo);\n\n                return 0;\n        }\n\nWhat the patch does looks good, by the way ;-)\n\n\n"},{"id":"558672","messageId":"xmqqh5he94x7.fsf@gitster.g","threadId":"66714","inReplyTo":"0b1d1c5668c89e3329e246d787ec80a161bd1dac.1793357680.git.gitter.spiros@gmail.com","subject":"Re: [PATCH 2/2] tree: use size_t for the size of parsed trees","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2026-11-18T03:47:32Z","receivedAt":"2026-11-18T03:47:32Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Elia Pinto <gitter.spiros@gmail.com> writes:\n\n> struct tree stores the size of its buffer in an \"unsigned long\", which\n> is only 32 bits wide on Windows, and parse_tree_buffer() takes it as\n> such, although repo_parse_tree_gently() gets it from odb_read_object()\n> as a size_t. Callers then pass tree->size to init_tree_desc(), so with\n> the previous commit a tree of 4GiB or more is still truncated on\n> Windows before it gets there.\n>\n> Use size_t for both. parse_object_buffer() still hands an \"unsigned\n> long\" size to parse_tree_buffer(); it is shared with the other object\n> types and is left for a later change.\n\nSounds good to me.\n\n> diff --git a/tree.c b/tree.c\n> index 53f7395e9f..d37b9bc7b1 100644\n> --- a/tree.c\n> +++ b/tree.c\n> @@ -172,7 +172,7 @@ struct tree *lookup_tree(struct repository *r, const struct object_id *oid)\n>  \treturn object_as_type(obj, OBJ_TREE, 0);\n>  }\n>  \n> -int parse_tree_buffer(struct tree *item, void *buffer, unsigned long size)\n> +int parse_tree_buffer(struct tree *item, void *buffer, size_t size)\n>  {\n>  \tif (item->object.parsed)\n>  \t\treturn 0;\n\nThe only caller of this function, object.c:parse_object_buffer(),\nhas its size as ulong.  Even though this alone would not help\nend-to-end, as a \"one thing at a time\" step, this patch may be OK.\n\n\n"}]}