git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH v2 1/2] builtin/repo: only parse objects as needed

From
Patrick Steinhardt <ps@pks.im>
Date
Nov 16, 2026, 08:01 UTC
Message-ID
<20261116-pks-repo-count-objects-speedup-v2-1-9c8be218eaff@pks.im>
In-Reply-To
<20261116-pks-repo-count-objects-speedup-v2-0-9c8be218eaff@pks.im>

To count objects we walk the repository's object graph. We then read the individual objects that we encounter so that we can learn about their metrics like their size.

As part of this, we also unconditionally read the object's content and parse it. But this is quite unnecessary, as we only really need parsed commits and trees because we need their number of parents and tree entries, respectively. So the time we spend reading and parsing blobs and tags is ultimately pointless.

Improve the status quo so that we only parse as needed. This allows us to do another small optimization: as we reach commits via the rev walk we know that they have already been parsed anyway. Consequently, we don't even need to read their data at all, and we can instead just look up the already-parsed commit.

This speeds up `git repo structure` a bit. The following benchmark was executed in "git.git":

  Benchmark 1: ./git repo structure (commit = HEAD~) (10 runs)
                  mean     ±       σ          min     …     max
    Wall Time   10.891 s   ±   0.113 s     10.779 s   …  11.181 s
    Memory         3.5 GiB ±     0.0 GiB      3.5 GiB …     3.5 GiB
  Benchmark 2: ./git repo structure (commit = HEAD) (10 runs)
                  mean     ±       σ          min     …     max
    Wall Time    9.577 s   ±   0.064 s      9.485 s   …   9.673 s            -12.1%
    Memory         3.4 GiB ±     0.0 GiB      3.4 GiB …     3.4 GiB           -4.1%
Signed-off-by: Patrick Steinhardt <ps@pks.im>
---
 builtin/repo.c | 23 ++++++++++++++++-------
 1 file changed, 16 insertions(+), 7 deletions(-)
diff --git a/builtin/repo.c b/builtin/repo.c
index 0399c558719..45ac005c2f7 100644
--- a/builtin/repo.c
+++ b/builtin/repo.c
@@ -847,13 +847,14 @@ static int count_objects(const char *path UNUSED, struct oid_array *oids,
 		size_t inflated_st = 0;
 		struct commit *commit;
 		struct object *obj;
-		void *content;
+		void *content = NULL;
 		off_t disk;
-		int eaten;
+		int eaten = 0;
 
 		oi.sizep = &inflated_st;
 		oi.disk_sizep = &disk;
-		oi.contentp = &content;
+		if (type == OBJ_TREE)
+			oi.contentp = &content;
 
 		if (odb_read_object_info_extended(data->odb, &oids->oid[i], &oi,
 						  OBJECT_INFO_SKIP_FETCH_OBJECT |
@@ -861,9 +862,6 @@ static int count_objects(const char *path UNUSED, struct oid_array *oids,
 			continue;
 		inflated = cast_size_t_to_ulong(inflated_st);
 
-		obj = parse_object_buffer(the_repository, &oids->oid[i], type,
-					  inflated, content, &eaten);
-
 		switch (type) {
 		case OBJ_TAG:
 			stats->type_counts.tags++;
@@ -873,7 +871,15 @@ static int count_objects(const char *path UNUSED, struct oid_array *oids,
 				      inflated);
 			break;
 		case OBJ_COMMIT:
-			commit = object_as_type(obj, OBJ_COMMIT, 0);
+			/*
+			 * Commits should already have been parsed by the
+			 * revision walk, so in theory we do not need to read
+			 * their contents again.
+			 */
+			commit = lookup_commit(data->odb->repo, &oids->oid[i]);
+			if (!commit || repo_parse_commit(data->odb->repo, commit))
+				continue;
+
 			stats->type_counts.commits++;
 			stats->inflated_sizes.commits += inflated;
 			stats->disk_sizes.commits += disk;
@@ -883,6 +889,9 @@ static int count_objects(const char *path UNUSED, struct oid_array *oids,
 				      commit_list_count(commit->parents));
 			break;
 		case OBJ_TREE:
+			obj = parse_object_buffer(the_repository, &oids->oid[i], type,
+						  inflated, content, &eaten);
+
 			stats->type_counts.trees++;
 			stats->inflated_sizes.trees += inflated;
 			stats->disk_sizes.trees += disk;
-- 
2.56.1.763.g16177fe143.dirty
Previous: Patrick SteinhardtNext: Patrick Steinhardt
Message 11 of 15 in “builtin/repo: improve counting of objects”
  1. 0/2 builtin/repo: improve counting of objectsPatrick Steinhardt, Nov 13, 2026
  2. 1/2 builtin/repo: only parse objects as neededPatrick Steinhardt, Nov 13, 2026
  3. Kristofer KarlssonNov 13, 2026
  4. Patrick SteinhardtNov 16, 2026
  5. Kristofer KarlssonNov 16, 2026
  6. 2/2 builtin/repo: don't fully parse treesPatrick Steinhardt, Nov 13, 2026
  7. Kristofer KarlssonNov 13, 2026
  8. Patrick SteinhardtNov 16, 2026
  9. Derrick StoleeNov 16, 2026
  10. 0/2 builtin/repo: improve counting of objectsPatrick Steinhardt, Nov 16, 2026
  11. 1/2 builtin/repo: only parse objects as neededPatrick Steinhardt, Nov 16, 2026
  12. 2/2 builtin/repo: don't fully parse treesPatrick Steinhardt, Nov 16, 2026
  13. Kristofer KarlssonNov 16, 2026
  14. Derrick StoleeNov 16, 2026
  15. Junio C HamanoNov 16, 2026

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.