git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: We should add a "git gc --auto" after "git clone" due to commit graph

From
Junio C Hamano <gitster@pobox.com>
Date
Oct 5, 2018, 06:09 UTC
Message-ID
<xmqqpnwo3ow9.fsf@gitster-ct.c.googlers.com>
In-Reply-To
<20181003165926.GR23446@localhost>
SZEDER Gábor <szeder.dev@gmail.com> writes:
Show 8 quoted lines
>>     git-gc - Cleanup unnecessary files and optimize the local repository
>> 
>> Creating these indexes like the commit-graph falls under "optimize the
>> local repository",
>
> But it doesn't fall under "cleanup unnecessary files", which the
> commit-graph file is, since, strictly speaking, it's purely
> optimization.

I won't be actively engaged in this discussion soon, but I must say that "git gc" doing "garbage collection" is merely an implementation detail of optimizing the repository for further use. And from that point of view, what needs to be updated is the synopsis of the git-gc doc. It states "X and Y" above, but it actually is "Y by doing X and other things".

I understand your "by definition there is no garbage immediately after clone" position, and also I would understand if you find it (perhaps philosophically) disturbing that "git clone" may give users a suboptimal repository that immediately needs optimizing [*1*].

But that bridge was crossed long time ago ever since pack transfer was invented. The data source sends only the pack data stream, and the receiving end is responsible for spending cycles to build .idx file. Theoretically, .pack should be all that is needed---you should be able to locate any necessary object by parsing the .pack file every time you open it, and .idx is mere optimization. You can think of the .midx and graph files the same way.

I'd consider it a growing pain that these two recent inventions were and are still built as a totally optional and separate features, requiring completely separate full enumeration of objects in the repository that needs to happen anyway when we build .idx out of the received .pack.

I would not be surprised by a future in which the initial index-pack that is responsible for receiving the incoming pack stream and storing that in .pack file(s) while creating corresponding .idx file(s) becomes also responsible for building .midx and graph files in the same pass, or at least smaller number of passes. Once we gain experience and confidence with these new auxiliary files, that ought to happen naturally. And at that point, we won't be having this discussion---we'd all happily run index-pack to receive the pack data, because that is pretty much the fundamental requirement to make use of the data.

[Footnote]
*1* Even without considering these recent invention of auxiliary
    files, cloning from a sloppily packed server whose primary focus
    is to avoid spending cycles by not computing better deltas will
    give the cloner a suboptimal repository.  If we truly want to
    have an optimized repository ready to be used after cloning, we
    should run an equivalent of "repack -a -d -f" immediately after
    "git clone".
Previous: SZEDER GáborNext: SZEDER Gábor
Message 11 of 78 in “We should add a "git gc --auto" after "git clone" due to commit graph”
  1. Ævar Arnfjörð BjarmasonOct 3, 2018
  2. SZEDER GáborOct 3, 2018
  3. Derrick StoleeOct 3, 2018
  4. Ævar Arnfjörð BjarmasonOct 3, 2018
  5. Ævar Arnfjörð BjarmasonOct 3, 2018
  6. SZEDER GáborOct 3, 2018
  7. Ævar Arnfjörð BjarmasonOct 3, 2018
  8. SZEDER GáborOct 3, 2018
  9. Ævar Arnfjörð BjarmasonOct 3, 2018
  10. SZEDER GáborOct 3, 2018
  11. Junio C HamanoOct 5, 2018
  12. SZEDER GáborOct 10, 2018
  13. Ævar Arnfjörð BjarmasonOct 10, 2018
  14. Stefan BellerOct 3, 2018
  15. Jeff KingOct 3, 2018
  16. Ævar Arnfjörð BjarmasonOct 3, 2018
  17. Stefan BellerOct 3, 2018
  18. Ævar Arnfjörð BjarmasonOct 3, 2018
  19. Jeff KingOct 3, 2018
  20. Derrick StoleeOct 3, 2018
  21. Jeff KingOct 3, 2018
  22. SZEDER GáborOct 8, 2018
  23. Derrick StoleeOct 8, 2018
  24. SZEDER GáborOct 8, 2018
  25. Derrick StoleeOct 8, 2018
  26. Jeff KingOct 9, 2018
  27. Bloom Filters (was Re: We should add a "git gc --auto" after "git clone" due to commit graph)Derrick Stolee, Oct 9, 2018
  28. Ævar Arnfjörð BjarmasonOct 9, 2018
  29. Jeff KingOct 9, 2018
  30. Derrick StoleeOct 9, 2018
  31. Jeff KingOct 9, 2018
  32. Jeff KingOct 9, 2018
  33. 1/3 initial tree-bitmap proof of conceptJeff King, Oct 9, 2018
  34. 2/3 test-tree-bitmap: add "dump" modeJeff King, Oct 9, 2018
  35. Junio C HamanoOct 10, 2018
  36. Jeff KingOct 11, 2018
  37. 3/3 test-tree-bitmap: replace ewah with custom rle encodingJeff King, Oct 9, 2018
  38. Junio C HamanoOct 10, 2018
  39. Jeff KingOct 11, 2018
  40. Derrick StoleeOct 11, 2018
  41. Jeff KingOct 11, 2018
  42. SZEDER GáborOct 9, 2018
  43. 0/4 Bloom filter experimentSZEDER Gábor, Oct 9, 2018
  44. 1/4 Add a (very) barebones Bloom filter implementationSZEDER Gábor, Oct 9, 2018
  45. 2/4 commit-graph: write a Bloom filter containing changed paths for each commitSZEDER Gábor, Oct 9, 2018
  46. Jeff KingOct 9, 2018
  47. SZEDER GáborOct 9, 2018
  48. 3/4 revision.c: use the Bloom filter to speed up path-limited revision walksSZEDER Gábor, Oct 9, 2018
  49. 4/4 revision.c: add GIT_TRACE_BLOOM_FILTER for a bit of statisticsSZEDER Gábor, Oct 9, 2018
  50. Derrick StoleeOct 9, 2018
  51. 0/2 Per-commit filter proof of conceptJonathan Tan, Oct 11, 2018
  52. 1/2 One filter per commitJonathan Tan, Oct 11, 2018
  53. Derrick StoleeOct 11, 2018
  54. Per-commit and per-parent filters for 2 parentsJonathan Tan, Oct 11, 2018
  55. 2/2 Only make bloom filter for first parentJonathan Tan, Oct 11, 2018
  56. Ævar Arnfjörð BjarmasonOct 11, 2018
  57. Derrick StoleeOct 15, 2018
  58. Junio C HamanoOct 16, 2018
  59. Derrick StoleeOct 16, 2018
  60. Ævar Arnfjörð BjarmasonOct 16, 2018
  61. Derrick StoleeOct 16, 2018
  62. Junio C HamanoOct 18, 2018
  63. Jonathan TanOct 16, 2018
  64. Junio C HamanoOct 8, 2018
  65. Duy NguyenOct 3, 2018
  66. Duy NguyenOct 3, 2018
  67. We should add a "git gc --auto" after "git clone" due to commit graphÆvar Arnfjörð Bjarmason, Oct 4, 2018
  68. Derrick StoleeOct 5, 2018
  69. Ævar Arnfjörð BjarmasonOct 5, 2018
  70. Derrick StoleeOct 5, 2018
  71. Ævar Arnfjörð BjarmasonOct 5, 2018
  72. Jeff KingOct 5, 2018
  73. Derrick StoleeOct 5, 2018
  74. Jeff KingOct 5, 2018
  75. Derrick StoleeOct 5, 2018
  76. Jeff KingOct 5, 2018
  77. Ævar Arnfjörð BjarmasonOct 5, 2018
  78. Jeff KingOct 5, 2018

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.