git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [RFC PATCH] We should add a "git gc --auto" after "git clone" due to commit graph

From
Derrick Stolee <stolee@gmail.com>
Date
Oct 5, 2018, 13:45 UTC
Message-ID
<05fefab0-4012-4ef0-5df1-2430eea2bf4d@gmail.com>
In-Reply-To
<87ftxkh7bf.fsf@evledraar.gmail.com>
On 10/5/2018 9:05 AM, Ævar Arnfjörð Bjarmason wrote:
Show 45 quoted lines
> On Fri, Oct 05 2018, Derrick Stolee wrote:
>
>> On 10/4/2018 5:42 PM, Ævar Arnfjörð Bjarmason wrote:
>>> I don't have time to polish this up for submission now, but here's a WIP
>>> patch that implements this, highlights:
>>>
>>>    * There's a gc.clone.autoDetach=false default setting which overrides
>>>      gc.autoDetach if 'git gc --auto' is run via git-clone (we just pass a
>>>      --cloning option to indicate this).
>> I'll repeat that it could make sense to do the same thing on clone
>> _and_ fetch. Perhaps a "--post-fetch" flag would be good here to
>> communicate that we just downloaded a pack from a remote.
> I don't think that makes sense, but let's talk about why, because maybe
> I've missed something, you're certainly more familiar with the
> commit-graph than I am.
>
> The reason to do it on clone as a special-case or when the file is
> missing, is because we know the file is desired (via the GC config), and
> presumably is expected to help performance, and we have 0% of it. So by
> going from 0% to 100% on clone we'll get fast --contains and other
> goodies the graph helps with.
>
> But when we're doing a fetch, or really anything else that runs "git gc
> --auto" we can safely assume that we have a recent enough graph, because
> it will have been run whenever auto-gc kicked in.
>
> I.e.:
>
>      # Slow, if we assume background forked commit-graph generation
>      # (which I'm avoiding)
>      git clone x && cd x && git tag --contains
>      # Fast enough, since we have an existing commit-graph
>      cd x && git fetch && git tag --contains
>
> I *do* think it might make sense to in general split off parts of "gc
> --auto" that we'd like to be more aggressive about, simply because the
> ratio of how long it takes to do, and how much it helps with performance
> makes more sense than a full repack, which is what the current heuristic
> is based on.
>
> And maybe when we run in that mode we should run in the foreground, but
> I don't see why git-fetch should be a special case there, and in this
> regard, the gc.clone.autoDetach=false setting I've made doesn't make
> much sence. I.e. maybe we should also skip forking to the background in
> such a mode when we trigger such a "mini gc" via git-commit or whatever.

My misunderstanding was that your proposed change to gc computes the commit-graph in either of these two cases:

(1) The auto-GC threshold is met.
(2) There is no commit-graph file.
And what I hope to have instead of (2) is (3):
(3) The commit-graph file is "sufficiently behind" the tip refs.

This condition is intentionally vague at the moment. It could be that we hint that (3) holds by saying "--post-fetch" (i.e. "We just downloaded a pack, and it probably contains a lot of new commits") or we could create some more complicated condition based on counting reachable commits with infinite generation number (the number of commits not in the commit-graph file).

I like that you are moving forward to make the commit-graph be written more frequently, but I'm trying to push us in a direction of writing it even more often than your proposed strategy. We should avoid creating too many orthogonal conditions that trigger the commit-graph write, which is why I'm pushing on your design here.

Anyone else have thoughts on this direction?
Thanks,
-Stolee
Previous: Ævar Arnfjörð BjarmasonNext: Ævar Arnfjörð Bjarmason
Message 70 of 78 in “We should add a "git gc --auto" after "git clone" due to commit graph”
  1. Ævar Arnfjörð BjarmasonOct 3, 2018
  2. SZEDER GáborOct 3, 2018
  3. Derrick StoleeOct 3, 2018
  4. Ævar Arnfjörð BjarmasonOct 3, 2018
  5. Ævar Arnfjörð BjarmasonOct 3, 2018
  6. SZEDER GáborOct 3, 2018
  7. Ævar Arnfjörð BjarmasonOct 3, 2018
  8. SZEDER GáborOct 3, 2018
  9. Ævar Arnfjörð BjarmasonOct 3, 2018
  10. SZEDER GáborOct 3, 2018
  11. Junio C HamanoOct 5, 2018
  12. SZEDER GáborOct 10, 2018
  13. Ævar Arnfjörð BjarmasonOct 10, 2018
  14. Stefan BellerOct 3, 2018
  15. Jeff KingOct 3, 2018
  16. Ævar Arnfjörð BjarmasonOct 3, 2018
  17. Stefan BellerOct 3, 2018
  18. Ævar Arnfjörð BjarmasonOct 3, 2018
  19. Jeff KingOct 3, 2018
  20. Derrick StoleeOct 3, 2018
  21. Jeff KingOct 3, 2018
  22. SZEDER GáborOct 8, 2018
  23. Derrick StoleeOct 8, 2018
  24. SZEDER GáborOct 8, 2018
  25. Derrick StoleeOct 8, 2018
  26. Jeff KingOct 9, 2018
  27. Bloom Filters (was Re: We should add a "git gc --auto" after "git clone" due to commit graph)Derrick Stolee, Oct 9, 2018
  28. Ævar Arnfjörð BjarmasonOct 9, 2018
  29. Jeff KingOct 9, 2018
  30. Derrick StoleeOct 9, 2018
  31. Jeff KingOct 9, 2018
  32. Jeff KingOct 9, 2018
  33. 1/3 initial tree-bitmap proof of conceptJeff King, Oct 9, 2018
  34. 2/3 test-tree-bitmap: add "dump" modeJeff King, Oct 9, 2018
  35. Junio C HamanoOct 10, 2018
  36. Jeff KingOct 11, 2018
  37. 3/3 test-tree-bitmap: replace ewah with custom rle encodingJeff King, Oct 9, 2018
  38. Junio C HamanoOct 10, 2018
  39. Jeff KingOct 11, 2018
  40. Derrick StoleeOct 11, 2018
  41. Jeff KingOct 11, 2018
  42. SZEDER GáborOct 9, 2018
  43. 0/4 Bloom filter experimentSZEDER Gábor, Oct 9, 2018
  44. 1/4 Add a (very) barebones Bloom filter implementationSZEDER Gábor, Oct 9, 2018
  45. 2/4 commit-graph: write a Bloom filter containing changed paths for each commitSZEDER Gábor, Oct 9, 2018
  46. Jeff KingOct 9, 2018
  47. SZEDER GáborOct 9, 2018
  48. 3/4 revision.c: use the Bloom filter to speed up path-limited revision walksSZEDER Gábor, Oct 9, 2018
  49. 4/4 revision.c: add GIT_TRACE_BLOOM_FILTER for a bit of statisticsSZEDER Gábor, Oct 9, 2018
  50. Derrick StoleeOct 9, 2018
  51. 0/2 Per-commit filter proof of conceptJonathan Tan, Oct 11, 2018
  52. 1/2 One filter per commitJonathan Tan, Oct 11, 2018
  53. Derrick StoleeOct 11, 2018
  54. Per-commit and per-parent filters for 2 parentsJonathan Tan, Oct 11, 2018
  55. 2/2 Only make bloom filter for first parentJonathan Tan, Oct 11, 2018
  56. Ævar Arnfjörð BjarmasonOct 11, 2018
  57. Derrick StoleeOct 15, 2018
  58. Junio C HamanoOct 16, 2018
  59. Derrick StoleeOct 16, 2018
  60. Ævar Arnfjörð BjarmasonOct 16, 2018
  61. Derrick StoleeOct 16, 2018
  62. Junio C HamanoOct 18, 2018
  63. Jonathan TanOct 16, 2018
  64. Junio C HamanoOct 8, 2018
  65. Duy NguyenOct 3, 2018
  66. Duy NguyenOct 3, 2018
  67. We should add a "git gc --auto" after "git clone" due to commit graphÆvar Arnfjörð Bjarmason, Oct 4, 2018
  68. Derrick StoleeOct 5, 2018
  69. Ævar Arnfjörð BjarmasonOct 5, 2018
  70. Derrick StoleeOct 5, 2018
  71. Ævar Arnfjörð BjarmasonOct 5, 2018
  72. Jeff KingOct 5, 2018
  73. Derrick StoleeOct 5, 2018
  74. Jeff KingOct 5, 2018
  75. Derrick StoleeOct 5, 2018
  76. Jeff KingOct 5, 2018
  77. Ævar Arnfjörð BjarmasonOct 5, 2018
  78. Jeff KingOct 5, 2018

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.