{"thread":{"id":"66845","subject":"[PATCH 0/2] builtin/repo: improve counting of objects","startedAt":"2026-11-13T11:46:16Z","lastAt":"2026-11-16T17:32:00Z","messageCount":15,"participants":["Patrick Steinhardt","Kristofer Karlsson","Derrick Stolee","Junio C Hamano"],"isPatch":true,"patchVersion":1,"patchTotal":2},"messages":[{"id":"558380","messageId":"20261113-pks-repo-count-objects-speedup-v1-0-7ba4f29e74fb@pks.im","threadId":"66845","inReplyTo":null,"subject":"[PATCH 0/2] builtin/repo: improve counting of objects","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-11-13T11:46:16Z","receivedAt":"2026-11-13T11:46:16Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Hi,\n\nI was recently using `git repo structure` in a very large repository and\nwas eventually greeted with an OOM. So I had a bit of a look and found a\nway to improve the memory footprint and speed it up a bit.\n\nThe repository still doesn't produce results. As it turns out, walking\n33 billion tree entries takes a while, and everything rev-list based\nthat walks trees is kinda broken in there. Oh, well, a problem for\nanother day.\n\nThanks!\n\nPatrick\n\n---\nPatrick Steinhardt (2):\n      builtin/repo: only parse objects as needed\n      builtin/repo: don't fully parse trees\n\n builtin/repo.c | 34 +++++++++++++++++++---------------\n 1 file changed, 19 insertions(+), 15 deletions(-)\n\n\n---\nbase-commit: 1891de46f79dc444a47db1c87bd0c8c76b5463f2\nchange-id: 20261111-pks-repo-count-objects-speedup-528abf12ea23\n\n\n"},{"id":"558381","messageId":"20261113-pks-repo-count-objects-speedup-v1-1-7ba4f29e74fb@pks.im","threadId":"66845","inReplyTo":"20261113-pks-repo-count-objects-speedup-v1-0-7ba4f29e74fb@pks.im","subject":"[PATCH 1/2] builtin/repo: only parse objects as needed","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-11-13T11:46:17Z","receivedAt":"2026-11-13T11:46:17Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"To count objects we walk the repository's object graph. We then read the\nindividual objects that we encounter so that we can learn about their\nmetrics like their size.\n\nAs part of this, we also unconditionally read the object's content and\nparse it. But this is quite unnecessary, as we only really need parsed\ncommits and trees because we need their number of parents and tree\nentries, respectively. So the time we spend reading and parsing blobs\nand tags is ultimately pointless.\n\nImprove the status quo so that we only parse as needed. This allows us\nto do another small optimization: as we reach commits via the rev walk\nwe know that they have already been parsed anyway. Consequently, we\ndon't even need to read their data at all, and we can instead just look\nup the already-parsed commit.\n\nThis speeds up `git repo structure` a bit. The following benchmark was\nexecuted in \"git.git\":\n\n  Benchmark 1: ./git repo structure (commit = HEAD~) (10 runs)\n                  mean     ±       σ          min     …     max\n    Wall Time   10.891 s   ±   0.113 s     10.779 s   …  11.181 s\n    Memory         3.5 GiB ±     0.0 GiB      3.5 GiB …     3.5 GiB\n\n  Benchmark 2: ./git repo structure (commit = HEAD) (10 runs)\n                  mean     ±       σ          min     …     max\n    Wall Time    9.577 s   ±   0.064 s      9.485 s   …   9.673 s            -12.1%\n    Memory         3.4 GiB ±     0.0 GiB      3.4 GiB …     3.4 GiB           -4.1%\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n builtin/repo.c | 23 ++++++++++++++++-------\n 1 file changed, 16 insertions(+), 7 deletions(-)\n\ndiff --git a/builtin/repo.c b/builtin/repo.c\nindex 0399c558719..98be90891ce 100644\n--- a/builtin/repo.c\n+++ b/builtin/repo.c\n@@ -847,13 +847,14 @@ static int count_objects(const char *path UNUSED, struct oid_array *oids,\n \t\tsize_t inflated_st = 0;\n \t\tstruct commit *commit;\n \t\tstruct object *obj;\n-\t\tvoid *content;\n+\t\tvoid *content = NULL;\n \t\toff_t disk;\n-\t\tint eaten;\n+\t\tint eaten = 0;\n \n \t\toi.sizep = &inflated_st;\n \t\toi.disk_sizep = &disk;\n-\t\toi.contentp = &content;\n+\t\tif (type == OBJ_TREE)\n+\t\t\toi.contentp = &content;\n \n \t\tif (odb_read_object_info_extended(data->odb, &oids->oid[i], &oi,\n \t\t\t\t\t\t  OBJECT_INFO_SKIP_FETCH_OBJECT |\n@@ -861,9 +862,6 @@ static int count_objects(const char *path UNUSED, struct oid_array *oids,\n \t\t\tcontinue;\n \t\tinflated = cast_size_t_to_ulong(inflated_st);\n \n-\t\tobj = parse_object_buffer(the_repository, &oids->oid[i], type,\n-\t\t\t\t\t  inflated, content, &eaten);\n-\n \t\tswitch (type) {\n \t\tcase OBJ_TAG:\n \t\t\tstats->type_counts.tags++;\n@@ -873,7 +871,15 @@ static int count_objects(const char *path UNUSED, struct oid_array *oids,\n \t\t\t\t      inflated);\n \t\t\tbreak;\n \t\tcase OBJ_COMMIT:\n-\t\t\tcommit = object_as_type(obj, OBJ_COMMIT, 0);\n+\t\t\t/*\n+\t\t\t * Commits have already been parsed by the revision\n+\t\t\t * walk, so there is no need to read their contents\n+\t\t\t * again.\n+\t\t\t */\n+\t\t\tcommit = lookup_commit(data->odb->repo, &oids->oid[i]);\n+\t\t\tif (!commit || repo_parse_commit(data->odb->repo, commit))\n+\t\t\t\tcontinue;\n+\n \t\t\tstats->type_counts.commits++;\n \t\t\tstats->inflated_sizes.commits += inflated;\n \t\t\tstats->disk_sizes.commits += disk;\n@@ -883,6 +889,9 @@ static int count_objects(const char *path UNUSED, struct oid_array *oids,\n \t\t\t\t      commit_list_count(commit->parents));\n \t\t\tbreak;\n \t\tcase OBJ_TREE:\n+\t\t\tobj = parse_object_buffer(the_repository, &oids->oid[i], type,\n+\t\t\t\t\t\t  inflated, content, &eaten);\n+\n \t\t\tstats->type_counts.trees++;\n \t\t\tstats->inflated_sizes.trees += inflated;\n \t\t\tstats->disk_sizes.trees += disk;\n\n-- \n2.56.0.611.g1030f0b940.dirty\n\n\n"},{"id":"558382","messageId":"20261113-pks-repo-count-objects-speedup-v1-2-7ba4f29e74fb@pks.im","threadId":"66845","inReplyTo":"20261113-pks-repo-count-objects-speedup-v1-0-7ba4f29e74fb@pks.im","subject":"[PATCH 2/2] builtin/repo: don't fully parse trees","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-11-13T11:46:18Z","receivedAt":"2026-11-13T11:46:18Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"When the rev walk hits a tree object we read its content and then parse\nit via `parse_object_buffer()`. This requires us to perform some extra\nbookkeeping that is ultimately unnecessary as we keep those trees in\nmemory.\n\nThe only thing that we need the tree object's content for is so that we\ncan count its entries. But that can easily be done without having a\nparsed tree object, all we need is its content.\n\nRefactor the code accordingly, which gives us another significant\nspeedup when executing `git repo structure` in \"git.git\":\n\n  Benchmark 1: ./git repo structure (commit = HEAD~) (10 runs)\n                  mean     ±       σ          min     …     max\n    Wall Time    9.635 s   ±   0.072 s      9.534 s   …   9.799 s\n    Memory         3.4 GiB ±     0.0 GiB      3.4 GiB …     3.4 GiB\n\n  Benchmark 2: ./git repo structure (commit = HEAD) (10 runs)\n                  mean     ±       σ          min     …     max\n    Wall Time    7.676 s   ±   0.106 s      7.568 s   …   7.823 s            -20.3%\n    Memory         0.7 GiB ±     0.0 GiB      0.7 GiB …     0.7 GiB          -80.4%  (5.1x less)\n\nBut besides the speedup, the bigger win here is that we no longer keep\nthese parsed trees in memory at all, which translates in a significant\nreduction in peak memory usage.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n builtin/repo.c | 19 +++++++------------\n 1 file changed, 7 insertions(+), 12 deletions(-)\n\ndiff --git a/builtin/repo.c b/builtin/repo.c\nindex 98be90891ce..64fb5e788d1 100644\n--- a/builtin/repo.c\n+++ b/builtin/repo.c\n@@ -820,14 +820,14 @@ static void check_largest(struct object_data *data, struct object_id *oid,\n \t}\n }\n \n-static size_t count_tree_entries(struct object *obj)\n+static size_t count_tree_entries(const struct object_id *oid,\n+\t\t\t\t void *buffer, unsigned long size)\n {\n-\tstruct tree *t = object_as_type(obj, OBJ_TREE, 0);\n \tstruct name_entry entry;\n \tstruct tree_desc desc;\n \tsize_t count = 0;\n \n-\tinit_tree_desc(&desc, &t->object.oid, t->buffer, t->size);\n+\tinit_tree_desc(&desc, oid, buffer, size);\n \twhile (tree_entry(&desc, &entry))\n \t\tcount++;\n \n@@ -846,10 +846,8 @@ static int count_objects(const char *path UNUSED, struct oid_array *oids,\n \t\tunsigned long inflated;\n \t\tsize_t inflated_st = 0;\n \t\tstruct commit *commit;\n-\t\tstruct object *obj;\n \t\tvoid *content = NULL;\n \t\toff_t disk;\n-\t\tint eaten = 0;\n \n \t\toi.sizep = &inflated_st;\n \t\toi.disk_sizep = &disk;\n@@ -889,16 +887,16 @@ static int count_objects(const char *path UNUSED, struct oid_array *oids,\n \t\t\t\t      commit_list_count(commit->parents));\n \t\t\tbreak;\n \t\tcase OBJ_TREE:\n-\t\t\tobj = parse_object_buffer(the_repository, &oids->oid[i], type,\n-\t\t\t\t\t\t  inflated, content, &eaten);\n-\n \t\t\tstats->type_counts.trees++;\n \t\t\tstats->inflated_sizes.trees += inflated;\n \t\t\tstats->disk_sizes.trees += disk;\n \t\t\tcheck_largest(&stats->largest.tree_size, &oids->oid[i],\n \t\t\t\t      inflated);\n \t\t\tcheck_largest(&stats->largest.tree_entries, &oids->oid[i],\n-\t\t\t\t      count_tree_entries(obj));\n+\t\t\t\t      count_tree_entries(&oids->oid[i], content,\n+\t\t\t\t\t\t\t inflated));\n+\n+\t\t\tfree(content);\n \t\t\tbreak;\n \t\tcase OBJ_BLOB:\n \t\t\tstats->type_counts.blobs++;\n@@ -910,9 +908,6 @@ static int count_objects(const char *path UNUSED, struct oid_array *oids,\n \t\tdefault:\n \t\t\tBUG(\"invalid object type\");\n \t\t}\n-\n-\t\tif (!eaten)\n-\t\t\tfree(content);\n \t}\n \n \tobject_count = get_total_object_values(&stats->type_counts);\n\n-- \n2.56.0.611.g1030f0b940.dirty\n\n\n"},{"id":"558390","messageId":"CAL71e4NfnYLAXQT-wpb6ciJZwTrT51EZVsEs8VertN=xBmUD_g@mail.gmail.com","threadId":"66845","inReplyTo":"20261113-pks-repo-count-objects-speedup-v1-1-7ba4f29e74fb@pks.im","subject":"Re: [PATCH 1/2] builtin/repo: only parse objects as needed","fromName":"Kristofer Karlsson","fromEmail":"krka@spotify.com","sentAt":"2026-11-13T14:12:19Z","receivedAt":"2026-11-13T14:12:19Z","isPatch":true,"sender":{"key":"krka@spotify.com","avatar":"https://avatars.githubusercontent.com/u/5349266?v=4"},"body":"On Fri, 13 Nov 2026 at 13:50, Patrick Steinhardt <ps@pks.im> wrote:\n>\n> +                       /*\n> +                        * Commits have already been parsed by the revision\n> +                        * walk, so there is no need to read their contents\n> +                        * again.\n> +                        */\n> +                       commit = lookup_commit(data->odb->repo, &oids->oid[i]);\n> +                       if (!commit || repo_parse_commit(data->odb->repo, commit))\n> +                               continue;\n\nTiny nit: I understood the comment as saying that the code\ndepends on the commits being parsed already, but\nrepo_parse_commit() handles the case where they are not.\nSince path-walk hands us commits from get_revision(), I\nassume that is only a failsafe?  So this is just about the\nphrasing.  How about:\n\n/*\n* Commits should already have been parsed by the\n* revision walk, so we do not need to read their\n* contents again.\n*/\n\nOtherwise this looks good to me.  Only reading contents for\ntrees avoids inflating every blob just to throw it away, which\nis a nice easy win.\n\nThanks,\nKristofer\n\n"},{"id":"558391","messageId":"CAL71e4PscLozeuQbwNePkgn=QdDPhE1zqTtuGahnKWDN++B0Hw@mail.gmail.com","threadId":"66845","inReplyTo":"20261113-pks-repo-count-objects-speedup-v1-2-7ba4f29e74fb@pks.im","subject":"Re: [PATCH 2/2] builtin/repo: don't fully parse trees","fromName":"Kristofer Karlsson","fromEmail":"krka@spotify.com","sentAt":"2026-11-13T14:15:16Z","receivedAt":"2026-11-13T14:15:16Z","isPatch":true,"sender":{"key":"krka@spotify.com","avatar":"https://avatars.githubusercontent.com/u/5349266?v=4"},"body":"On Fri, 13 Nov 2026 at 14:06, Patrick Steinhardt <ps@pks.im> wrote:\n>\n>   Benchmark 1: ./git repo structure (commit = HEAD~) (10 runs)\n>                   mean     ±       σ          min     …     max\n>     Wall Time    9.635 s   ±   0.072 s      9.534 s   …   9.799 s\n>     Memory         3.4 GiB ±     0.0 GiB      3.4 GiB …     3.4 GiB\n>\n>   Benchmark 2: ./git repo structure (commit = HEAD) (10 runs)\n>                   mean     ±       σ          min     …     max\n>     Wall Time    7.676 s   ±   0.106 s      7.568 s   …   7.823 s            -20.3%\n>     Memory         0.7 GiB ±     0.0 GiB      0.7 GiB …     0.7 GiB          -80.4%  (5.1x less)\n\nThat's quite a nice memory reduction!\n\n> But besides the speedup, the bigger win here is that we no longer keep\n> these parsed trees in memory at all, which translates in a significant\n> reduction in peak memory usage.\n\nI took a look at this since it is very similar to what I\nrecently did for the incremental connectivity check, where\nthe tree verifier also reads raw tree buffers instead of\ngoing through parse_object_buffer(), for the same memory\nreasons (I only needed to track the oids, not the full objects).\n\nBoth patches look good to me.\n\nThanks,\nKristofer\n\n"},{"id":"558513","messageId":"avqk_DfxW1uKFXi7@pks.im","threadId":"66845","inReplyTo":"CAL71e4NfnYLAXQT-wpb6ciJZwTrT51EZVsEs8VertN=xBmUD_g@mail.gmail.com","subject":"Re: [PATCH 1/2] builtin/repo: only parse objects as needed","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-11-16T06:38:52Z","receivedAt":"2026-11-16T06:38:52Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Fri, Nov 13, 2026 at 03:12:19PM +0100, Kristofer Karlsson wrote:\n> On Fri, 13 Nov 2026 at 13:50, Patrick Steinhardt <ps@pks.im> wrote:\n> >\n> > +                       /*\n> > +                        * Commits have already been parsed by the revision\n> > +                        * walk, so there is no need to read their contents\n> > +                        * again.\n> > +                        */\n> > +                       commit = lookup_commit(data->odb->repo, &oids->oid[i]);\n> > +                       if (!commit || repo_parse_commit(data->odb->repo, commit))\n> > +                               continue;\n> \n> Tiny nit: I understood the comment as saying that the code\n> depends on the commits being parsed already, but\n> repo_parse_commit() handles the case where they are not.\n> Since path-walk hands us commits from get_revision(), I\n> assume that is only a failsafe?  So this is just about the\n> phrasing.  How about:\n\nYeah, it is. I was wondering whether I want to call `BUG()` here in the\ncase where the commit cannot be parsed, as that case really shouldn't\never happen. But that felt a bit heavy-handed to me.\n\n> /*\n> * Commits should already have been parsed by the\n> * revision walk, so we do not need to read their\n> * contents again.\n> */\n> \n> Otherwise this looks good to me.  Only reading contents for\n> trees avoids inflating every blob just to throw it away, which\n> is a nice easy win.\n\nYeah, that reads a bit better to me indeed. Will adapt, thanks!\n\nPatrick\n\n"},{"id":"558514","messageId":"avqlHHo36SDmupF6@pks.im","threadId":"66845","inReplyTo":"CAL71e4PscLozeuQbwNePkgn=QdDPhE1zqTtuGahnKWDN++B0Hw@mail.gmail.com","subject":"Re: [PATCH 2/2] builtin/repo: don't fully parse trees","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-11-16T06:39:24Z","receivedAt":"2026-11-16T06:39:24Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Fri, Nov 13, 2026 at 03:15:16PM +0100, Kristofer Karlsson wrote:\n> On Fri, 13 Nov 2026 at 14:06, Patrick Steinhardt <ps@pks.im> wrote:\n> >\n> >   Benchmark 1: ./git repo structure (commit = HEAD~) (10 runs)\n> >                   mean     ±       σ          min     …     max\n> >     Wall Time    9.635 s   ±   0.072 s      9.534 s   …   9.799 s\n> >     Memory         3.4 GiB ±     0.0 GiB      3.4 GiB …     3.4 GiB\n> >\n> >   Benchmark 2: ./git repo structure (commit = HEAD) (10 runs)\n> >                   mean     ±       σ          min     …     max\n> >     Wall Time    7.676 s   ±   0.106 s      7.568 s   …   7.823 s            -20.3%\n> >     Memory         0.7 GiB ±     0.0 GiB      0.7 GiB …     0.7 GiB          -80.4%  (5.1x less)\n> \n> That's quite a nice memory reduction!\n> \n> > But besides the speedup, the bigger win here is that we no longer keep\n> > these parsed trees in memory at all, which translates in a significant\n> > reduction in peak memory usage.\n> \n> I took a look at this since it is very similar to what I\n> recently did for the incremental connectivity check, where\n> the tree verifier also reads raw tree buffers instead of\n> going through parse_object_buffer(), for the same memory\n> reasons (I only needed to track the oids, not the full objects).\n> \n> Both patches look good to me.\n\nThanks for your review!\n\nPatrick\n\n"},{"id":"558524","messageId":"20261116-pks-repo-count-objects-speedup-v2-0-9c8be218eaff@pks.im","threadId":"66845","inReplyTo":"20261113-pks-repo-count-objects-speedup-v1-0-7ba4f29e74fb@pks.im","subject":"[PATCH v2 0/2] builtin/repo: improve counting of objects","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-11-16T08:01:33Z","receivedAt":"2026-11-16T08:01:33Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Hi,\n\nI was recently using `git repo structure` in a very large repository and\nwas eventually greeted with an OOM. So I had a bit of a look and found a\nway to improve the memory footprint and speed it up a bit.\n\nThe repository still doesn't produce results. As it turns out, walking\n33 billion tree entries takes a while, and everything rev-list based\nthat walks trees is kinda broken in there. Oh, well, a problem for\nanother day.\n\nChanges in v2:\n  - Reword a comment to point out that parsing the commit should in\n    theory not be required.\n  - Link to v1: https://patch.msgid.link/20261113-pks-repo-count-objects-speedup-v1-0-7ba4f29e74fb@pks.im\n\nThanks!\n\nPatrick\n\n---\nPatrick Steinhardt (2):\n      builtin/repo: only parse objects as needed\n      builtin/repo: don't fully parse trees\n\n builtin/repo.c | 34 +++++++++++++++++++---------------\n 1 file changed, 19 insertions(+), 15 deletions(-)\n\nRange-diff versus v1:\n\n1:  a7a1c1a8041 ! 1:  2780e491ead builtin/repo: only parse objects as needed\n    @@ builtin/repo.c: static int count_objects(const char *path UNUSED, struct oid_arr\n      \t\tcase OBJ_COMMIT:\n     -\t\t\tcommit = object_as_type(obj, OBJ_COMMIT, 0);\n     +\t\t\t/*\n    -+\t\t\t * Commits have already been parsed by the revision\n    -+\t\t\t * walk, so there is no need to read their contents\n    -+\t\t\t * again.\n    ++\t\t\t * Commits should already have been parsed by the\n    ++\t\t\t * revision walk, so in theory we do not need to read\n    ++\t\t\t * their contents again.\n     +\t\t\t */\n     +\t\t\tcommit = lookup_commit(data->odb->repo, &oids->oid[i]);\n     +\t\t\tif (!commit || repo_parse_commit(data->odb->repo, commit))\n2:  d9fd2366b84 = 2:  aa32dcb2961 builtin/repo: don't fully parse trees\n\n---\nbase-commit: 1891de46f79dc444a47db1c87bd0c8c76b5463f2\nchange-id: 20261111-pks-repo-count-objects-speedup-528abf12ea23\n\n\n"},{"id":"558525","messageId":"20261116-pks-repo-count-objects-speedup-v2-1-9c8be218eaff@pks.im","threadId":"66845","inReplyTo":"20261116-pks-repo-count-objects-speedup-v2-0-9c8be218eaff@pks.im","subject":"[PATCH v2 1/2] builtin/repo: only parse objects as needed","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-11-16T08:01:34Z","receivedAt":"2026-11-16T08:01:34Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"To count objects we walk the repository's object graph. We then read the\nindividual objects that we encounter so that we can learn about their\nmetrics like their size.\n\nAs part of this, we also unconditionally read the object's content and\nparse it. But this is quite unnecessary, as we only really need parsed\ncommits and trees because we need their number of parents and tree\nentries, respectively. So the time we spend reading and parsing blobs\nand tags is ultimately pointless.\n\nImprove the status quo so that we only parse as needed. This allows us\nto do another small optimization: as we reach commits via the rev walk\nwe know that they have already been parsed anyway. Consequently, we\ndon't even need to read their data at all, and we can instead just look\nup the already-parsed commit.\n\nThis speeds up `git repo structure` a bit. The following benchmark was\nexecuted in \"git.git\":\n\n  Benchmark 1: ./git repo structure (commit = HEAD~) (10 runs)\n                  mean     ±       σ          min     …     max\n    Wall Time   10.891 s   ±   0.113 s     10.779 s   …  11.181 s\n    Memory         3.5 GiB ±     0.0 GiB      3.5 GiB …     3.5 GiB\n\n  Benchmark 2: ./git repo structure (commit = HEAD) (10 runs)\n                  mean     ±       σ          min     …     max\n    Wall Time    9.577 s   ±   0.064 s      9.485 s   …   9.673 s            -12.1%\n    Memory         3.4 GiB ±     0.0 GiB      3.4 GiB …     3.4 GiB           -4.1%\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n builtin/repo.c | 23 ++++++++++++++++-------\n 1 file changed, 16 insertions(+), 7 deletions(-)\n\ndiff --git a/builtin/repo.c b/builtin/repo.c\nindex 0399c558719..45ac005c2f7 100644\n--- a/builtin/repo.c\n+++ b/builtin/repo.c\n@@ -847,13 +847,14 @@ static int count_objects(const char *path UNUSED, struct oid_array *oids,\n \t\tsize_t inflated_st = 0;\n \t\tstruct commit *commit;\n \t\tstruct object *obj;\n-\t\tvoid *content;\n+\t\tvoid *content = NULL;\n \t\toff_t disk;\n-\t\tint eaten;\n+\t\tint eaten = 0;\n \n \t\toi.sizep = &inflated_st;\n \t\toi.disk_sizep = &disk;\n-\t\toi.contentp = &content;\n+\t\tif (type == OBJ_TREE)\n+\t\t\toi.contentp = &content;\n \n \t\tif (odb_read_object_info_extended(data->odb, &oids->oid[i], &oi,\n \t\t\t\t\t\t  OBJECT_INFO_SKIP_FETCH_OBJECT |\n@@ -861,9 +862,6 @@ static int count_objects(const char *path UNUSED, struct oid_array *oids,\n \t\t\tcontinue;\n \t\tinflated = cast_size_t_to_ulong(inflated_st);\n \n-\t\tobj = parse_object_buffer(the_repository, &oids->oid[i], type,\n-\t\t\t\t\t  inflated, content, &eaten);\n-\n \t\tswitch (type) {\n \t\tcase OBJ_TAG:\n \t\t\tstats->type_counts.tags++;\n@@ -873,7 +871,15 @@ static int count_objects(const char *path UNUSED, struct oid_array *oids,\n \t\t\t\t      inflated);\n \t\t\tbreak;\n \t\tcase OBJ_COMMIT:\n-\t\t\tcommit = object_as_type(obj, OBJ_COMMIT, 0);\n+\t\t\t/*\n+\t\t\t * Commits should already have been parsed by the\n+\t\t\t * revision walk, so in theory we do not need to read\n+\t\t\t * their contents again.\n+\t\t\t */\n+\t\t\tcommit = lookup_commit(data->odb->repo, &oids->oid[i]);\n+\t\t\tif (!commit || repo_parse_commit(data->odb->repo, commit))\n+\t\t\t\tcontinue;\n+\n \t\t\tstats->type_counts.commits++;\n \t\t\tstats->inflated_sizes.commits += inflated;\n \t\t\tstats->disk_sizes.commits += disk;\n@@ -883,6 +889,9 @@ static int count_objects(const char *path UNUSED, struct oid_array *oids,\n \t\t\t\t      commit_list_count(commit->parents));\n \t\t\tbreak;\n \t\tcase OBJ_TREE:\n+\t\t\tobj = parse_object_buffer(the_repository, &oids->oid[i], type,\n+\t\t\t\t\t\t  inflated, content, &eaten);\n+\n \t\t\tstats->type_counts.trees++;\n \t\t\tstats->inflated_sizes.trees += inflated;\n \t\t\tstats->disk_sizes.trees += disk;\n\n-- \n2.56.1.763.g16177fe143.dirty\n\n\n"},{"id":"558526","messageId":"20261116-pks-repo-count-objects-speedup-v2-2-9c8be218eaff@pks.im","threadId":"66845","inReplyTo":"20261116-pks-repo-count-objects-speedup-v2-0-9c8be218eaff@pks.im","subject":"[PATCH v2 2/2] builtin/repo: don't fully parse trees","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-11-16T08:01:35Z","receivedAt":"2026-11-16T08:01:35Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"When the rev walk hits a tree object we read its content and then parse\nit via `parse_object_buffer()`. This requires us to perform some extra\nbookkeeping that is ultimately unnecessary as we keep those trees in\nmemory.\n\nThe only thing that we need the tree object's content for is so that we\ncan count its entries. But that can easily be done without having a\nparsed tree object, all we need is its content.\n\nRefactor the code accordingly, which gives us another significant\nspeedup when executing `git repo structure` in \"git.git\":\n\n  Benchmark 1: ./git repo structure (commit = HEAD~) (10 runs)\n                  mean     ±       σ          min     …     max\n    Wall Time    9.635 s   ±   0.072 s      9.534 s   …   9.799 s\n    Memory         3.4 GiB ±     0.0 GiB      3.4 GiB …     3.4 GiB\n\n  Benchmark 2: ./git repo structure (commit = HEAD) (10 runs)\n                  mean     ±       σ          min     …     max\n    Wall Time    7.676 s   ±   0.106 s      7.568 s   …   7.823 s            -20.3%\n    Memory         0.7 GiB ±     0.0 GiB      0.7 GiB …     0.7 GiB          -80.4%  (5.1x less)\n\nBut besides the speedup, the bigger win here is that we no longer keep\nthese parsed trees in memory at all, which translates in a significant\nreduction in peak memory usage.\n\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\n builtin/repo.c | 19 +++++++------------\n 1 file changed, 7 insertions(+), 12 deletions(-)\n\ndiff --git a/builtin/repo.c b/builtin/repo.c\nindex 45ac005c2f7..2c96a3ea577 100644\n--- a/builtin/repo.c\n+++ b/builtin/repo.c\n@@ -820,14 +820,14 @@ static void check_largest(struct object_data *data, struct object_id *oid,\n \t}\n }\n \n-static size_t count_tree_entries(struct object *obj)\n+static size_t count_tree_entries(const struct object_id *oid,\n+\t\t\t\t void *buffer, unsigned long size)\n {\n-\tstruct tree *t = object_as_type(obj, OBJ_TREE, 0);\n \tstruct name_entry entry;\n \tstruct tree_desc desc;\n \tsize_t count = 0;\n \n-\tinit_tree_desc(&desc, &t->object.oid, t->buffer, t->size);\n+\tinit_tree_desc(&desc, oid, buffer, size);\n \twhile (tree_entry(&desc, &entry))\n \t\tcount++;\n \n@@ -846,10 +846,8 @@ static int count_objects(const char *path UNUSED, struct oid_array *oids,\n \t\tunsigned long inflated;\n \t\tsize_t inflated_st = 0;\n \t\tstruct commit *commit;\n-\t\tstruct object *obj;\n \t\tvoid *content = NULL;\n \t\toff_t disk;\n-\t\tint eaten = 0;\n \n \t\toi.sizep = &inflated_st;\n \t\toi.disk_sizep = &disk;\n@@ -889,16 +887,16 @@ static int count_objects(const char *path UNUSED, struct oid_array *oids,\n \t\t\t\t      commit_list_count(commit->parents));\n \t\t\tbreak;\n \t\tcase OBJ_TREE:\n-\t\t\tobj = parse_object_buffer(the_repository, &oids->oid[i], type,\n-\t\t\t\t\t\t  inflated, content, &eaten);\n-\n \t\t\tstats->type_counts.trees++;\n \t\t\tstats->inflated_sizes.trees += inflated;\n \t\t\tstats->disk_sizes.trees += disk;\n \t\t\tcheck_largest(&stats->largest.tree_size, &oids->oid[i],\n \t\t\t\t      inflated);\n \t\t\tcheck_largest(&stats->largest.tree_entries, &oids->oid[i],\n-\t\t\t\t      count_tree_entries(obj));\n+\t\t\t\t      count_tree_entries(&oids->oid[i], content,\n+\t\t\t\t\t\t\t inflated));\n+\n+\t\t\tfree(content);\n \t\t\tbreak;\n \t\tcase OBJ_BLOB:\n \t\t\tstats->type_counts.blobs++;\n@@ -910,9 +908,6 @@ static int count_objects(const char *path UNUSED, struct oid_array *oids,\n \t\tdefault:\n \t\t\tBUG(\"invalid object type\");\n \t\t}\n-\n-\t\tif (!eaten)\n-\t\t\tfree(content);\n \t}\n \n \tobject_count = get_total_object_values(&stats->type_counts);\n\n-- \n2.56.1.763.g16177fe143.dirty\n\n\n"},{"id":"558529","messageId":"CAL71e4MKmEsP9xsZ7eg_LOFSCiwOjVU9H=rvOYkOFXV=3rMPxw@mail.gmail.com","threadId":"66845","inReplyTo":"avqk_DfxW1uKFXi7@pks.im","subject":"Re: [PATCH 1/2] builtin/repo: only parse objects as needed","fromName":"Kristofer Karlsson","fromEmail":"krka@spotify.com","sentAt":"2026-11-16T09:02:44Z","receivedAt":"2026-11-16T09:02:44Z","isPatch":true,"sender":{"key":"krka@spotify.com","avatar":"https://avatars.githubusercontent.com/u/5349266?v=4"},"body":"On Mon, 16 Nov 2026 at 07:38, Patrick Steinhardt <ps@pks.im> wrote:\n>\n> On Fri, Nov 13, 2026 at 03:12:19PM +0100, Kristofer Karlsson wrote:\n> > On Fri, 13 Nov 2026 at 13:50, Patrick Steinhardt <ps@pks.im> wrote:\n>\n> > Tiny nit: I understood the comment as saying that the code\n> > depends on the commits being parsed already, but\n> > repo_parse_commit() handles the case where they are not.\n> > Since path-walk hands us commits from get_revision(), I\n> > assume that is only a failsafe?  So this is just about the\n> > phrasing.  How about:\n>\n> Yeah, it is. I was wondering whether I want to call `BUG()` here in the\n> case where the commit cannot be parsed, as that case really shouldn't\n> ever happen. But that felt a bit heavy-handed to me.\n\nI think it would be a valid option, as long as their is enough\ntest coverage.  If there's ever a regression here that makes\nthis function be called for commits that have not been loaded,\nthe current implementation would mean surviving it, but at\nthe cost of extra unintentional work, whereas a BUG() assert\nwould make the regression more obvious.\n\nIt's not something I have a strong opinion on though, and I\nthink the updated comment will make future developers see\nwhat the intention is.\n\nThanks,\nKristofer\n\n"},{"id":"558530","messageId":"CAL71e4OcVOS4LPeNmLmF-92PMB_NC5NNZieAdGB15g4Yr7n01g@mail.gmail.com","threadId":"66845","inReplyTo":"20261116-pks-repo-count-objects-speedup-v2-0-9c8be218eaff@pks.im","subject":"Re: [PATCH v2 0/2] builtin/repo: improve counting of objects","fromName":"Kristofer Karlsson","fromEmail":"krka@spotify.com","sentAt":"2026-11-16T09:07:27Z","receivedAt":"2026-11-16T09:07:27Z","isPatch":true,"sender":{"key":"krka@spotify.com","avatar":"https://avatars.githubusercontent.com/u/5349266?v=4"},"body":"On Mon, 16 Nov 2026 at 09:01, Patrick Steinhardt <ps@pks.im> wrote:\n>\n> Range-diff versus v1:\n>\n> 1:  a7a1c1a8041 ! 1:  2780e491ead builtin/repo: only parse objects as needed\n>     @@ builtin/repo.c: static int count_objects(const char *path UNUSED, struct oid_arr\n>                 case OBJ_COMMIT:\n>      -                  commit = object_as_type(obj, OBJ_COMMIT, 0);\n>      +                  /*\n>     -+                   * Commits have already been parsed by the revision\n>     -+                   * walk, so there is no need to read their contents\n>     -+                   * again.\n>     ++                   * Commits should already have been parsed by the\n>     ++                   * revision walk, so in theory we do not need to read\n>     ++                   * their contents again.\n>      +                   */\n>      +                  commit = lookup_commit(data->odb->repo, &oids->oid[i]);\n>      +                  if (!commit || repo_parse_commit(data->odb->repo, commit))\n\nI think this range-diff addresses my only nit, so I am happy\nwith this version!\n\nThanks,\nKristofer\n\n"},{"id":"558549","messageId":"0b4f65e1-bebc-4dcd-961d-42da1fe02a77@gmail.com","threadId":"66845","inReplyTo":"20261113-pks-repo-count-objects-speedup-v1-2-7ba4f29e74fb@pks.im","subject":"Re: [PATCH 2/2] builtin/repo: don't fully parse trees","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2026-11-16T13:43:17Z","receivedAt":"2026-11-16T13:43:17Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 11/13/2026 6:46 AM, Patrick Steinhardt wrote:\n> When the rev walk hits a tree object we read its content and then parse\n> it via `parse_object_buffer()`. This requires us to perform some extra\n> bookkeeping that is ultimately unnecessary as we keep those trees in\n> memory.\n> \n> The only thing that we need the tree object's content for is so that we\n> can count its entries. But that can easily be done without having a\n> parsed tree object, all we need is its content.\n> \n> Refactor the code accordingly, which gives us another significant\n> speedup when executing `git repo structure` in \"git.git\":\n\n>     Memory         3.4 GiB ±     0.0 GiB      3.4 GiB …     3.4 GiB\n>     Memory         0.7 GiB ±     0.0 GiB      0.7 GiB …     0.7 GiB          -80.4%  (5.1x less)\n\nThis is an impressive memory reduction! Very nice.\n\n(I'd comment on v2, but my email host must have bounced those.0\n\nThanks,\n-Stolee\n\n\n"},{"id":"558555","messageId":"aacc4357-7f8c-4e5f-8cb2-1d2bdec767e4@gmail.com","threadId":"66845","inReplyTo":"20261116-pks-repo-count-objects-speedup-v2-0-9c8be218eaff@pks.im","subject":"Re: [PATCH v2 0/2] builtin/repo: improve counting of objects","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2026-11-16T16:02:30Z","receivedAt":"2026-11-16T16:02:30Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 11/16/2026 3:01 AM, Patrick Steinhardt wrote:\n> I was recently using `git repo structure` in a very large repository and\n> was eventually greeted with an OOM. So I had a bit of a look and found a\n> way to improve the memory footprint and speed it up a bit.\n> \n> The repository still doesn't produce results. As it turns out, walking\n> 33 billion tree entries takes a while, and everything rev-list based\n> that walks trees is kinda broken in there. Oh, well, a problem for\n> another day.\n> \n> Changes in v2:\n>   - Reword a comment to point out that parsing the commit should in\n>     theory not be required.\n>   - Link to v1: https://patch.msgid.link/20261113-pks-repo-count-objects-speedup-\nThanks for these changes. I found your v2 patches in my inbox somehow\ndisconnected from this cover letter. But I'm happy with v2.\n\nThanks,\n-Stolee\n\n\n"},{"id":"558563","messageId":"xmqqpkw4lm27.fsf@gitster.g","threadId":"66845","inReplyTo":"aacc4357-7f8c-4e5f-8cb2-1d2bdec767e4@gmail.com","subject":"Re: [PATCH v2 0/2] builtin/repo: improve counting of objects","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2026-11-16T17:32:00Z","receivedAt":"2026-11-16T17:32:00Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee <stolee@gmail.com> writes:\n\n> On 11/16/2026 3:01 AM, Patrick Steinhardt wrote:\n>> I was recently using `git repo structure` in a very large repository and\n>> was eventually greeted with an OOM. So I had a bit of a look and found a\n>> way to improve the memory footprint and speed it up a bit.\n>> \n>> The repository still doesn't produce results. As it turns out, walking\n>> 33 billion tree entries takes a while, and everything rev-list based\n>> that walks trees is kinda broken in there. Oh, well, a problem for\n>> another day.\n>> \n>> Changes in v2:\n>>   - Reword a comment to point out that parsing the commit should in\n>>     theory not be required.\n>>   - Link to v1: https://patch.msgid.link/20261113-pks-repo-count-objects-speedup-\n> Thanks for these changes. I found your v2 patches in my inbox somehow\n> disconnected from this cover letter. But I'm happy with v2.\n\nThanks for writing and reviewing.  Let me mark the topic as ready\nfor 'next'.\n\n"}]}