git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH v6 00/14] Serialized Git Commit Graph

From
Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Date
Mar 16, 2018, 15:06 UTC
Message-ID
<878tasdpqo.fsf@evledraar.gmail.com>
In-Reply-To
<20180314192736.70602-1-dstolee@microsoft.com>
On Wed, Mar 14 2018, Derrick Stolee jotted:
> Hopefully this version is ready to merge. I have several follow-up topics
> in mind to submit soon after, including:

I've been doing some preliminary testing of this internally, all good so far, on a relatively small repo (~100k commits) I was using for testing:

    - git -c core.commitGraph=true -C <repo> rev-list --all:
        -      /mnt/ext4_graph => min:273    mean:279    max:389    -- (273 274 275 276 277 279 282 282 345 389)
        -            /mnt/ext4 => min:1087   mean:1123   max:1175   -- (1087 1092 1092 1104 1117 1123 1126 1136 1143 1175)

This is on a fresh clone with one giant pack and where the commit graph data was generated afterwards with "git commit-graph write" for the *_graph repo, so it contains all the commits.

So less than 25% of the mean time it spent before. Nice. Those are times in milliseconds over 10 runs, for this particular one I didn't get much of an improvement in --graph, but still ~10%:

    - git -c core.commitGraph=true -C <repo> log --oneline --graph:
        -      /mnt/ext4_graph => min:1420   mean:1449   max:1586   -- (1420 1423 1428 1434 1449 1449 1490 1548 1567 1586)
        -            /mnt/ext4 => min:1547   mean:1616   max:2136   -- (1547 1557 1581 1585 1598 1616 1621 1740 1964 2136)

I noticed that it takes a *long* time to generate the graph, on a bigger repo I have it takes 20 minutes, and this is a repo where repack -A -d itself takes 5-8 minutes, probably on the upper end of that with the bitmap, but once you do that it's relatively snappy with --stdin-commits --additive when I feed it the new commits.

I don't have any need really to make this run in 10m instead of 20m, just something I found interesting, i.e. how it compares to the repack itself.

Previous: Derrick StoleeNext: SZEDER Gábor
Message 53 of 110 in “Serialized Git Commit Graph”
  1. 00/13 Serialized Git Commit GraphDerrick Stolee, Feb 27, 2018
  2. 01/13 commit-graph: add format documentDerrick Stolee, Feb 27, 2018
  3. 02/13 graph: add commit graph design documentDerrick Stolee, Feb 27, 2018
  4. 04/13 csum-file: add CSUM_KEEP_OPEN flagDerrick Stolee, Feb 27, 2018
  5. Derrick StoleeMar 12, 2018
  6. Junio C HamanoMar 13, 2018
  7. Derrick StoleeMar 14, 2018
  8. Junio C HamanoMar 14, 2018
  9. 06/13 commit-graph: implement 'git-commit-graph write'Derrick Stolee, Feb 27, 2018
  10. 05/13 commit-graph: implement write_commit_graph()Derrick Stolee, Feb 27, 2018
  11. 03/13 commit-graph: create git-commit-graph builtinDerrick Stolee, Feb 27, 2018
  12. 13/13 commit-graph: implement "--additive" optionDerrick Stolee, Feb 27, 2018
  13. 12/13 commit-graph: build graph from starting commitsDerrick Stolee, Feb 27, 2018
  14. 11/13 commit-graph: read only from specific pack-indexesDerrick Stolee, Feb 27, 2018
  15. Stefan BellerFeb 27, 2018
  16. 10/13 commit: integrate commit graph with commit parsingDerrick Stolee, Feb 27, 2018
  17. 07/13 commit-graph: implement git commit-graph readDerrick Stolee, Feb 27, 2018
  18. 08/13 commit-graph: add core.commitGraph settingDerrick Stolee, Feb 27, 2018
  19. 09/13 commit-graph: close under reachabilityDerrick Stolee, Feb 27, 2018
  20. Stefan BellerFeb 27, 2018
  21. 00/14 Serialized Git Commit GraphDerrick Stolee, Mar 14, 2018
  22. 02/14 csum-file: refactor finalize_hashfile() methodDerrick Stolee, Mar 14, 2018
  23. 03/14 commit-graph: add format documentDerrick Stolee, Mar 14, 2018
  24. 04/14 graph: add commit graph design documentDerrick Stolee, Mar 14, 2018
  25. 08/14 commit-graph: implement git commit-graph readDerrick Stolee, Mar 14, 2018
  26. 06/14 commit-graph: implement write_commit_graph()Derrick Stolee, Mar 14, 2018
  27. 10/14 commit-graph: close under reachabilityDerrick Stolee, Mar 14, 2018
  28. 12/14 commit-graph: read only from specific pack-indexesDerrick Stolee, Mar 14, 2018
  29. SZEDER GáborMar 15, 2018
  30. Derrick StoleeMar 19, 2018
  31. 09/14 commit-graph: add core.commitGraph settingDerrick Stolee, Mar 14, 2018
  32. 11/14 commit: integrate commit graph with commit parsingDerrick Stolee, Mar 14, 2018
  33. 13/14 commit-graph: build graph from starting commitsDerrick Stolee, Mar 14, 2018
  34. 14/14 commit-graph: implement "--additive" optionDerrick Stolee, Mar 14, 2018
  35. 05/14 commit-graph: create git-commit-graph builtinDerrick Stolee, Mar 14, 2018
  36. 07/14 commit-graph: implement 'git-commit-graph write'Derrick Stolee, Mar 14, 2018
  37. Ævar Arnfjörð BjarmasonMar 18, 2018
  38. Derrick StoleeMar 19, 2018
  39. Ævar Arnfjörð BjarmasonMar 19, 2018
  40. Derrick StoleeMar 19, 2018
  41. Ævar Arnfjörð BjarmasonMar 19, 2018
  42. 01/14 csum-file: rename hashclose() to finalize_hashfile()Derrick Stolee, Mar 14, 2018
  43. Ramsay JonesMar 14, 2018
  44. Junio C HamanoMar 14, 2018
  45. Johannes SchindelinMar 15, 2018
  46. Junio C HamanoMar 15, 2018
  47. Ramsay JonesMar 15, 2018
  48. Johannes SchindelinMar 16, 2018
  49. Junio C HamanoMar 16, 2018
  50. Johannes SchindelinMar 19, 2018
  51. Lars SchneiderMar 16, 2018
  52. Derrick StoleeMar 19, 2018
  53. Ævar Arnfjörð BjarmasonMar 16, 2018
  54. SZEDER GáborMar 16, 2018
  55. Junio C HamanoMar 16, 2018
  56. SZEDER GáborMar 16, 2018
  57. Jeff KingMar 16, 2018
  58. Jeff KingMar 16, 2018
  59. Derrick StoleeMar 19, 2018
  60. Derrick StoleeMar 20, 2018
  61. Jeff KingMar 16, 2018
  62. 00/14 Serialized Git Commit GraphDerrick Stolee, Apr 2, 2018
  63. 01/14 csum-file: rename hashclose() to finalize_hashfile()Derrick Stolee, Apr 2, 2018
  64. 02/14 csum-file: refactor finalize_hashfile() methodDerrick Stolee, Apr 2, 2018
  65. Jakub NarebskiApr 7, 2018
  66. 03/14 commit-graph: add format documentDerrick Stolee, Apr 2, 2018
  67. Jakub NarebskiApr 7, 2018
  68. 04/14 graph: add commit graph design documentDerrick Stolee, Apr 2, 2018
  69. Jakub NarebskiApr 8, 2018
  70. 06/14 commit-graph: implement write_commit_graph()Derrick Stolee, Apr 2, 2018
  71. 11/14 commit: integrate commit graph with commit parsingDerrick Stolee, Apr 2, 2018
  72. 13/14 commit-graph: build graph from starting commitsDerrick Stolee, Apr 2, 2018
  73. Jakub NarebskiApr 8, 2018
  74. 14/14 commit-graph: implement "--additive" optionDerrick Stolee, Apr 2, 2018
  75. SZEDER GáborApr 5, 2018
  76. 10/14 commit-graph: close under reachabilityDerrick Stolee, Apr 2, 2018
  77. 12/14 commit-graph: read only from specific pack-indexesDerrick Stolee, Apr 2, 2018
  78. 09/14 commit-graph: add core.commitGraph settingDerrick Stolee, Apr 2, 2018
  79. Jakub NarebskiApr 8, 2018
  80. 05/14 commit-graph: create git-commit-graph builtinDerrick Stolee, Apr 2, 2018
  81. 07/14 commit-graph: implement git-commit-graph writeDerrick Stolee, Apr 2, 2018
  82. Jakub NarebskiApr 8, 2018
  83. 08/14 commit-graph: implement git commit-graph readDerrick Stolee, Apr 2, 2018
  84. Junio C HamanoApr 2, 2018
  85. Derrick StoleeApr 3, 2018
  86. Jakub NarebskiApr 8, 2018
  87. 00/14 Serialized Git Commit GraphDerrick Stolee, Apr 10, 2018
  88. 01/14 csum-file: rename hashclose() to finalize_hashfile()Derrick Stolee, Apr 10, 2018
  89. 02/14 csum-file: refactor finalize_hashfile() methodDerrick Stolee, Apr 10, 2018
  90. 03/14 commit-graph: add format documentDerrick Stolee, Apr 10, 2018
  91. Stefan BellerApr 10, 2018
  92. Derrick StoleeApr 10, 2018
  93. Jakub NarebskiApr 11, 2018
  94. Derrick StoleeApr 12, 2018
  95. Jakub NarebskiApr 13, 2018
  96. 04/14 graph: add commit graph design documentDerrick Stolee, Apr 10, 2018
  97. Jakub NarebskiApr 15, 2018
  98. 05/14 commit-graph: create git-commit-graph builtinDerrick Stolee, Apr 10, 2018
  99. 07/14 commit-graph: implement git-commit-graph writeDerrick Stolee, Apr 10, 2018
  100. 08/14 commit-graph: implement git commit-graph readDerrick Stolee, Apr 10, 2018
  101. Jakub NarebskiApr 14, 2018
  102. Eric SunshineApr 15, 2018
  103. 09/14 commit-graph: add core.commitGraph settingDerrick Stolee, Apr 10, 2018
  104. Jakub NarebskiApr 14, 2018
  105. 06/14 commit-graph: implement write_commit_graph()Derrick Stolee, Apr 10, 2018
  106. 10/14 commit-graph: close under reachabilityDerrick Stolee, Apr 10, 2018
  107. 11/14 commit: integrate commit graph with commit parsingDerrick Stolee, Apr 10, 2018
  108. 12/14 commit-graph: read only from specific pack-indexesDerrick Stolee, Apr 10, 2018
  109. 13/14 commit-graph: build graph from starting commitsDerrick Stolee, Apr 10, 2018
  110. 14/14 commit-graph: implement "--append" optionDerrick Stolee, Apr 10, 2018

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.