git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH 0/3] Lazy-load trees when reading commit-graph

From
Jeff King <peff@peff.net>
Date
Apr 3, 2018, 20:20 UTC
Message-ID
<20180403202030.GA15157@sigill.intra.peff.net>
In-Reply-To
<007c630a-0df8-803c-b637-fb5ccefa7b7c@gmail.com>
On Tue, Apr 03, 2018 at 09:14:50AM -0400, Derrick Stolee wrote:
Show 19 quoted lines
> > I'm not sure what the exact solution would be, but I imagine something
> > like variable-sized "struct commit"s with the parent pointers embedded,
> > with some kind of flag to indicate the number of parents (and probably
> > some fallback to break out to a linked list for extreme cases of more
> > than 2 parents).  It may end up pretty invasive, though, as there's a
> > lot of open-coded traversals of that parent list.
> > 
> > Anyway, not anything to do with this patch, but food for thought as you
> > micro-optimize these traversals.
> 
> One other thing that I've been thinking about is that 'struct commit' is so
> much bigger than the other structs in 'union any_object'. This means that
> the object cache, which I think creates blocks of 'union any_object' for
> memory-alignment reasons, is overly bloated. This would be especially true
> when walking many more trees than commits.
> 
> Perhaps there are other memory concerns in that case (such as cached
> buffers) that the 'union any_object' is not a concern, but it is worth
> thinking about as we brainstorm how to reduce the parent-list memory.

It definitely bloats any_object, but I don't think we typically allocate too many of those. Those should only come from lookup_unknown_object(), but typically we'll come across objects by traversing the graph, in which case we have an expectation of the type (and use the appropriate lookup_foo() function, which uses the type-specific block allocators).

-Peff
Previous: Derrick StoleeNext: Derrick Stolee
Message 11 of 26 in “Lazy-load trees when reading commit-graph”
  1. 0/3 Lazy-load trees when reading commit-graphDerrick Stolee, Apr 3, 2018
  2. 1/3 commit: create get_commit_tree() methodDerrick Stolee, Apr 3, 2018
  3. 3/3 commit-graph: lazy-load treesDerrick Stolee, Apr 3, 2018
  4. Stefan BellerApr 3, 2018
  5. Derrick StoleeApr 3, 2018
  6. Stefan BellerApr 3, 2018
  7. 2/3 treewide: use get_commit_tree() for tree accessDerrick Stolee, Apr 3, 2018
  8. Derrick StoleeApr 3, 2018
  9. Jeff KingApr 3, 2018
  10. Derrick StoleeApr 3, 2018
  11. Jeff KingApr 3, 2018
  12. Derrick StoleeApr 4, 2018
  13. 0/4 Lazy-load trees when reading commit-graphDerrick Stolee, Apr 6, 2018
  14. 2/4 commit: create get_commit_tree() methodDerrick Stolee, Apr 6, 2018
  15. 1/4 treewide: rename tree to maybe_treeDerrick Stolee, Apr 6, 2018
  16. 3/4 treewide: replace maybe_tree with accessor methodsDerrick Stolee, Apr 6, 2018
  17. 4/4 commit-graph: lazy-load trees for commitsDerrick Stolee, Apr 6, 2018
  18. Jeff KingApr 6, 2018
  19. Derrick StoleeApr 6, 2018
  20. Stefan BellerApr 6, 2018
  21. Junio C HamanoApr 8, 2018
  22. Derrick StoleeApr 9, 2018
  23. Stefan BellerApr 9, 2018
  24. Jakub NarebskiApr 7, 2018
  25. Derrick StoleeApr 8, 2018
  26. Jakub NarebskiApr 11, 2018

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.