git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH v2 0/8] ref-filter: ahead/behind counting, faster --merged option

From
Junio C Hamano <gitster@pobox.com>
Date
Mar 10, 2023, 19:16 UTC
Message-ID
<xmqqedpw5se6.fsf@gitster.g>
In-Reply-To
<pull.1489.v2.git.1678468863.gitgitgadget@gmail.com>
"Derrick Stolee via GitGitGadget" <gitgitgadget@gmail.com> writes:
Show 5 quoted lines
> I was
> initially concerned about the overhead of 'git for-each-ref' and its
> generality and sorting, but I was not able to measure any important
> difference between this implementation and our internal 'git ahead-behind'
> implementation.
That certainly is nice to know.
Show 12 quoted lines
> However, for our specific uses, we like to batch a list of exact references
> that could be very long. We introduce a new --stdin option here.
>
> To keep things close to the v1 outline, I replaced the existing patches with
> closely-related ones, when possible.
>
> Patch 1 adds the --stdin option to 'git for-each-ref'. (This is similar to
> the boilerplate patch from v1.)
>
> Patch 2 adds a test to explicitly check that 'git for-each-ref' will still
> succeed when all input refs are missing. (This is similar to the
> --ignore-missing patch from v1.)
Sensible.
Show 16 quoted lines
> Patches 3-5 introduce a new method: ensure_generations_valid(). Patch 3 does
> some refactoring of the existing generation number computations to make it
> more generic, and patch 4 updates the definition of
> commit_graph_generation() slightly, making way for patch 5 to implement the
> method. With an existing commit-graph file, the commits that are not present
> in the file are considered as having generation number "infinity". This is
> useful for most of our reachability queries to this point, since those
> commits are "above" the ones tracked by the commit-graph. When these commits
> are low in number, then there is very little performance cost and zero
> correctness cost. (These patches match v1 exactly.)
>
> However, we will see that the ahead/behind computation requires accurate
> generation numbers to avoid overcounting. Thus, ensure_generations_valid()
> is a way to specify a list of commits that need generation numbers computed
> before continuing. It's a no-op if all of those commits are in the
> commit-graph file. It's expensive if the commit-graph doesn't exist.
Reasonable.
Show 5 quoted lines
> However, '%(ahead-behind:)' computations are likely to be slow no matter
> what without a commit-graph, so assuming an existing commit-graph file is
> reasonable. If we find sufficient desire to have an implementation that does
> not have this requirement, we could create a second implementation and
> toggle to it when generation_numbers_enabled() returns false.

At that point, it might make sense to find a way to make the work ensure_generations_valid() had to spend cycles on not to go to waste. Something like "ah, you do not have commit-graph at all, so let's try to create one if you can write into the repository" at the beginning of the function, or something? Just thinking aloud.

Show 5 quoted lines
> Patch 6 implements the ahead-behind algorithm, but it is not connected to a
> builtin. It's a long commit message, so hopefully it explains the algorithm
> sufficiently. (The difference from v1 is that it no longer integrates with a
> builtin and there are no new tests. It also uses 'unsigned int' and is
> correctly co-authored by Taylor.)
Nice.
Show 9 quoted lines
> Patch 7 integrates the ahead-behind algorithm with the ref-filter code,
> including parsing the "ahead-behind" token. This finally adds tests that
> check both ahead_behind() and ensure_generations_valid() via
> t6600-test-reach.sh. (This patch is essentially completely new in v2.)
>
> Patch 8 implements the tips_reachable_from_base() method, and uses it within
> the ref-filter code to speed up 'git for-each-ref --merged' and 'git branch
> --merged'. (The interface is slightly different than v1, due to the needs of
> the new caller.)
Very nice.

Having read all the patches, I am very impressed and pleased, but are we losing anything by having the feature inside for-each-ref compared to a new command ahead-behind? As far as I can tell, the new "for-each-ref --stdin" would still want to match refs and work only on refs, but there shouldn't be any reason for ahead-behind computation to limit to tips that are at the tip of a ref, so that may be one downside in this updated design. For the intended use case of "let's find which branches are stale", that downside does not matter in practice, but for other use cases people will think of in the future, the limitation might matter (at which time we can easily resurrect the other subcommand, using the internal machinery we have here, so it is not a huge deal, I presume).

Thanks.
Previous: Derrick StoleeNext: Derrick Stolee
Message 56 of 90 in “ahead-behind: new builtin for counting multiple commit ranges”
  1. 0/8 ahead-behind: new builtin for counting multiple commit rangesDerrick Stolee via GitGitGadget, Mar 6, 2023
  2. 1/8 ahead-behind: create empty builtinDerrick Stolee via GitGitGadget, Mar 6, 2023
  3. Junio C HamanoMar 6, 2023
  4. Taylor BlauMar 7, 2023
  5. Derrick StoleeMar 8, 2023
  6. Junio C HamanoMar 8, 2023
  7. 2/8 ahead-behind: parse tip referencesDerrick Stolee via GitGitGadget, Mar 6, 2023
  8. Taylor BlauMar 7, 2023
  9. 3/8 ahead-behind: implement --ignore-missing optionDerrick Stolee via GitGitGadget, Mar 6, 2023
  10. Taylor BlauMar 7, 2023
  11. 4/8 commit-graph: combine generation computationsDerrick Stolee via GitGitGadget, Mar 6, 2023
  12. 5/8 commit-graph: return generation from memoryDerrick Stolee via GitGitGadget, Mar 6, 2023
  13. 6/8 commit-graph: introduce `ensure_generations_valid()`Taylor Blau via GitGitGadget, Mar 6, 2023
  14. Junio C HamanoMar 6, 2023
  15. Taylor BlauMar 7, 2023
  16. 7/8 ahead-behind: implement ahead_behind() logicDerrick Stolee via GitGitGadget, Mar 6, 2023
  17. Taylor BlauMar 7, 2023
  18. Derrick StoleeMar 9, 2023
  19. 8/8 ahead-behind: add --contains modeDerrick Stolee via GitGitGadget, Mar 6, 2023
  20. Junio C HamanoMar 6, 2023
  21. Derrick StoleeMar 6, 2023
  22. Junio C HamanoMar 6, 2023
  23. Taylor BlauMar 7, 2023
  24. Jeff KingMar 9, 2023
  25. Junio C HamanoMar 9, 2023
  26. Taylor BlauMar 7, 2023
  27. 0/8 ref-filter: ahead/behind counting, faster --merged optionDerrick Stolee via GitGitGadget, Mar 10, 2023
  28. 1/8 for-each-ref: add --stdin optionDerrick Stolee via GitGitGadget, Mar 10, 2023
  29. Junio C HamanoMar 10, 2023
  30. Phillip WoodMar 13, 2023
  31. Derrick StoleeMar 13, 2023
  32. Taylor BlauMar 13, 2023
  33. Ævar Arnfjörð BjarmasonMar 15, 2023
  34. Jeff KingMar 15, 2023
  35. Jeff KingMar 15, 2023
  36. Junio C HamanoMar 15, 2023
  37. Jeff KingMar 15, 2023
  38. 2/8 for-each-ref: explicitly test no matchesDerrick Stolee via GitGitGadget, Mar 10, 2023
  39. 3/8 commit-graph: combine generation computationsDerrick Stolee via GitGitGadget, Mar 10, 2023
  40. 4/8 commit-graph: return generation from memoryDerrick Stolee via GitGitGadget, Mar 10, 2023
  41. 5/8 commit-graph: introduce `ensure_generations_valid()`Taylor Blau via GitGitGadget, Mar 10, 2023
  42. 6/8 commit-reach: implement ahead_behind() logicDerrick Stolee via GitGitGadget, Mar 10, 2023
  43. Ævar Arnfjörð BjarmasonMar 15, 2023
  44. Junio C HamanoMar 15, 2023
  45. Derrick StoleeMar 15, 2023
  46. 8/8 commit-reach: add tips_reachable_from_bases()Derrick Stolee via GitGitGadget, Mar 10, 2023
  47. Ævar Arnfjörð BjarmasonMar 15, 2023
  48. Derrick StoleeMar 15, 2023
  49. Derrick StoleeMar 15, 2023
  50. 7/8 for-each-ref: add ahead-behind format atomDerrick Stolee via GitGitGadget, Mar 10, 2023
  51. Junio C HamanoMar 10, 2023
  52. Ævar Arnfjörð BjarmasonMar 15, 2023
  53. Junio C HamanoMar 15, 2023
  54. Derrick StoleeMar 15, 2023
  55. Derrick StoleeMar 15, 2023
  56. Junio C HamanoMar 10, 2023
  57. Derrick StoleeMar 10, 2023
  58. Jeff KingMar 15, 2023
  59. Derrick StoleeMar 15, 2023
  60. Junio C HamanoMar 15, 2023
  61. Ævar Arnfjörð BjarmasonMar 15, 2023
  62. Derrick StoleeMar 15, 2023
  63. 0/8 ref-filter: ahead/behind counting, faster --merged optionDerrick Stolee via GitGitGadget, Mar 15, 2023
  64. 1/8 for-each-ref: add --stdin optionDerrick Stolee via GitGitGadget, Mar 15, 2023
  65. Jeff KingMar 15, 2023
  66. Junio C HamanoMar 15, 2023
  67. Jonathan TanMar 15, 2023
  68. 2/8 for-each-ref: explicitly test no matchesDerrick Stolee via GitGitGadget, Mar 15, 2023
  69. 5/8 commit-graph: introduce `ensure_generations_valid()`Taylor Blau via GitGitGadget, Mar 15, 2023
  70. 3/8 commit-graph: combine generation computationsDerrick Stolee via GitGitGadget, Mar 15, 2023
  71. Jonathan TanMar 15, 2023
  72. Derrick StoleeMar 17, 2023
  73. 6/8 commit-reach: implement ahead_behind() logicDerrick Stolee via GitGitGadget, Mar 15, 2023
  74. Jonathan TanMar 15, 2023
  75. Derrick StoleeMar 17, 2023
  76. 4/8 commit-graph: return generation from memoryDerrick Stolee via GitGitGadget, Mar 15, 2023
  77. Jonathan TanMar 15, 2023
  78. 8/8 commit-reach: add tips_reachable_from_bases()Derrick Stolee via GitGitGadget, Mar 15, 2023
  79. 7/8 for-each-ref: add ahead-behind format atomDerrick Stolee via GitGitGadget, Mar 15, 2023
  80. 0/9 ref-filter: ahead/behind counting, faster --merged optionDerrick Stolee via GitGitGadget, Mar 20, 2023
  81. 1/9 for-each-ref: add --stdin optionDerrick Stolee via GitGitGadget, Mar 20, 2023
  82. 2/9 for-each-ref: explicitly test no matchesDerrick Stolee via GitGitGadget, Mar 20, 2023
  83. 4/9 commit-graph: simplify compute_generation_numbers()Derrick Stolee via GitGitGadget, Mar 20, 2023
  84. 3/9 commit-graph: refactor compute_topological_levels()Derrick Stolee via GitGitGadget, Mar 20, 2023
  85. 5/9 commit-graph: return generation from memoryDerrick Stolee via GitGitGadget, Mar 20, 2023
  86. 6/9 commit-graph: introduce `ensure_generations_valid()`Taylor Blau via GitGitGadget, Mar 20, 2023
  87. 7/9 commit-reach: implement ahead_behind() logicDerrick Stolee via GitGitGadget, Mar 20, 2023
  88. Jonathan TanMar 20, 2023
  89. 9/9 commit-reach: add tips_reachable_from_bases()Derrick Stolee via GitGitGadget, Mar 20, 2023
  90. 8/9 for-each-ref: add ahead-behind format atomDerrick Stolee via GitGitGadget, Mar 20, 2023

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.