git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH 2/2] builtin/repo: don't fully parse trees

From
Patrick Steinhardt <ps@pks.im>
Date
Nov 13, 2026, 11:46 UTC
Message-ID
<20261113-pks-repo-count-objects-speedup-v1-2-7ba4f29e74fb@pks.im>
In-Reply-To
<20261113-pks-repo-count-objects-speedup-v1-0-7ba4f29e74fb@pks.im>

When the rev walk hits a tree object we read its content and then parse it via `parse_object_buffer()`. This requires us to perform some extra bookkeeping that is ultimately unnecessary as we keep those trees in memory.

The only thing that we need the tree object's content for is so that we can count its entries. But that can easily be done without having a parsed tree object, all we need is its content.

Refactor the code accordingly, which gives us another significant speedup when executing `git repo structure` in "git.git":

  Benchmark 1: ./git repo structure (commit = HEAD~) (10 runs)
                  mean     ±       σ          min     …     max
    Wall Time    9.635 s   ±   0.072 s      9.534 s   …   9.799 s
    Memory         3.4 GiB ±     0.0 GiB      3.4 GiB …     3.4 GiB
  Benchmark 2: ./git repo structure (commit = HEAD) (10 runs)
                  mean     ±       σ          min     …     max
    Wall Time    7.676 s   ±   0.106 s      7.568 s   …   7.823 s            -20.3%
    Memory         0.7 GiB ±     0.0 GiB      0.7 GiB …     0.7 GiB          -80.4%  (5.1x less)

But besides the speedup, the bigger win here is that we no longer keep these parsed trees in memory at all, which translates in a significant reduction in peak memory usage.

Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 builtin/repo.c | 19 +++++++------------
 1 file changed, 7 insertions(+), 12 deletions(-)
diff --git a/builtin/repo.c b/builtin/repo.c
index 98be90891ce..64fb5e788d1 100644
--- a/builtin/repo.c
+++ b/builtin/repo.c
@@ -820,14 +820,14 @@ static void check_largest(struct object_data *data, struct object_id *oid,
 	}
 }
 
-static size_t count_tree_entries(struct object *obj)
+static size_t count_tree_entries(const struct object_id *oid,
+				 void *buffer, unsigned long size)
 {
-	struct tree *t = object_as_type(obj, OBJ_TREE, 0);
 	struct name_entry entry;
 	struct tree_desc desc;
 	size_t count = 0;
 
-	init_tree_desc(&desc, &t->object.oid, t->buffer, t->size);
+	init_tree_desc(&desc, oid, buffer, size);
 	while (tree_entry(&desc, &entry))
 		count++;
 
@@ -846,10 +846,8 @@ static int count_objects(const char *path UNUSED, struct oid_array *oids,
 		unsigned long inflated;
 		size_t inflated_st = 0;
 		struct commit *commit;
-		struct object *obj;
 		void *content = NULL;
 		off_t disk;
-		int eaten = 0;
 
 		oi.sizep = &inflated_st;
 		oi.disk_sizep = &disk;
@@ -889,16 +887,16 @@ static int count_objects(const char *path UNUSED, struct oid_array *oids,
 				      commit_list_count(commit->parents));
 			break;
 		case OBJ_TREE:
-			obj = parse_object_buffer(the_repository, &oids->oid[i], type,
-						  inflated, content, &eaten);
-
 			stats->type_counts.trees++;
 			stats->inflated_sizes.trees += inflated;
 			stats->disk_sizes.trees += disk;
 			check_largest(&stats->largest.tree_size, &oids->oid[i],
 				      inflated);
 			check_largest(&stats->largest.tree_entries, &oids->oid[i],
-				      count_tree_entries(obj));
+				      count_tree_entries(&oids->oid[i], content,
+							 inflated));
+
+			free(content);
 			break;
 		case OBJ_BLOB:
 			stats->type_counts.blobs++;
@@ -910,9 +908,6 @@ static int count_objects(const char *path UNUSED, struct oid_array *oids,
 		default:
 			BUG("invalid object type");
 		}
-
-		if (!eaten)
-			free(content);
 	}
 
 	object_count = get_total_object_values(&stats->type_counts);
-- 
2.56.0.611.g1030f0b940.dirty
Previous: Kristofer KarlssonNext: Kristofer Karlsson
Message 6 of 15 in “builtin/repo: improve counting of objects”
  1. 0/2 builtin/repo: improve counting of objectsPatrick Steinhardt, Nov 13, 2026
  2. 1/2 builtin/repo: only parse objects as neededPatrick Steinhardt, Nov 13, 2026
  3. Kristofer KarlssonNov 13, 2026
  4. Patrick SteinhardtNov 16, 2026
  5. Kristofer KarlssonNov 16, 2026
  6. 2/2 builtin/repo: don't fully parse treesPatrick Steinhardt, Nov 13, 2026
  7. Kristofer KarlssonNov 13, 2026
  8. Patrick SteinhardtNov 16, 2026
  9. Derrick StoleeNov 16, 2026
  10. 0/2 builtin/repo: improve counting of objectsPatrick Steinhardt, Nov 16, 2026
  11. 1/2 builtin/repo: only parse objects as neededPatrick Steinhardt, Nov 16, 2026
  12. 2/2 builtin/repo: don't fully parse treesPatrick Steinhardt, Nov 16, 2026
  13. Kristofer KarlssonNov 16, 2026
  14. Derrick StoleeNov 16, 2026
  15. Junio C HamanoNov 16, 2026

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.