git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH v2 0/2] builtin/repo: improve counting of objects

From
Patrick Steinhardt <ps@pks.im>
Date
Nov 16, 2026, 08:01 UTC
Message-ID
<20261116-pks-repo-count-objects-speedup-v2-0-9c8be218eaff@pks.im>
In-Reply-To
<20261113-pks-repo-count-objects-speedup-v1-0-7ba4f29e74fb@pks.im>
Hi,

I was recently using `git repo structure` in a very large repository and was eventually greeted with an OOM. So I had a bit of a look and found a way to improve the memory footprint and speed it up a bit.

The repository still doesn't produce results. As it turns out, walking 33 billion tree entries takes a while, and everything rev-list based that walks trees is kinda broken in there. Oh, well, a problem for another day.

Changes in v2:
  - Reword a comment to point out that parsing the commit should in
    theory not be required.
  - Link to v1: https://patch.msgid.link/20261113-pks-repo-count-objects-speedup-v1-0-7ba4f29e74fb@pks.im
Thanks!
Patrick
---
Patrick Steinhardt (2):
      builtin/repo: only parse objects as needed
      builtin/repo: don't fully parse trees
 builtin/repo.c | 34 +++++++++++++++++++---------------
 1 file changed, 19 insertions(+), 15 deletions(-)
Range-diff versus v1:
1:  a7a1c1a8041 ! 1:  2780e491ead builtin/repo: only parse objects as needed
    @@ builtin/repo.c: static int count_objects(const char *path UNUSED, struct oid_arr
      		case OBJ_COMMIT:
     -			commit = object_as_type(obj, OBJ_COMMIT, 0);
     +			/*
    -+			 * Commits have already been parsed by the revision
    -+			 * walk, so there is no need to read their contents
    -+			 * again.
    ++			 * Commits should already have been parsed by the
    ++			 * revision walk, so in theory we do not need to read
    ++			 * their contents again.
     +			 */
     +			commit = lookup_commit(data->odb->repo, &oids->oid[i]);
     +			if (!commit || repo_parse_commit(data->odb->repo, commit))
2:  d9fd2366b84 = 2:  aa32dcb2961 builtin/repo: don't fully parse trees

--- base-commit: 1891de46f79dc444a47db1c87bd0c8c76b5463f2 change-id: 20261111-pks-repo-count-objects-speedup-528abf12ea23

Previous: Derrick StoleeNext: Patrick Steinhardt
Message 10 of 15 in “builtin/repo: improve counting of objects”
  1. 0/2 builtin/repo: improve counting of objectsPatrick Steinhardt, Nov 13, 2026
  2. 1/2 builtin/repo: only parse objects as neededPatrick Steinhardt, Nov 13, 2026
  3. Kristofer KarlssonNov 13, 2026
  4. Patrick SteinhardtNov 16, 2026
  5. Kristofer KarlssonNov 16, 2026
  6. 2/2 builtin/repo: don't fully parse treesPatrick Steinhardt, Nov 13, 2026
  7. Kristofer KarlssonNov 13, 2026
  8. Patrick SteinhardtNov 16, 2026
  9. Derrick StoleeNov 16, 2026
  10. 0/2 builtin/repo: improve counting of objectsPatrick Steinhardt, Nov 16, 2026
  11. 1/2 builtin/repo: only parse objects as neededPatrick Steinhardt, Nov 16, 2026
  12. 2/2 builtin/repo: don't fully parse treesPatrick Steinhardt, Nov 16, 2026
  13. Kristofer KarlssonNov 16, 2026
  14. Derrick StoleeNov 16, 2026
  15. Junio C HamanoNov 16, 2026

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.