git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH v8 03/14] commit-graph: add format document

From
Derrick Stolee <stolee@gmail.com>
Date
Apr 10, 2018, 19:18 UTC
Message-ID
<f08076ac-2c89-561e-b4ae-192dd9c50dc2@gmail.com>
In-Reply-To
<CAGZ79kYDXTGdS8sx-iW_JNdk7icpwdd3Opwbjw1cL8gMs_47dw@mail.gmail.com>
On 4/10/2018 3:10 PM, Stefan Beller wrote:
Show 25 quoted lines
> Hi Derrick,
>
> On Tue, Apr 10, 2018 at 5:55 AM, Derrick Stolee <stolee@gmail.com> wrote:
>
>> +  OID Fanout (ID: {'O', 'I', 'D', 'F'}) (256 * 4 bytes)
>> +      The ith entry, F[i], stores the number of OIDs with first
>> +      byte at most i. Thus F[255] stores the total
>> +      number of commits (N).
> I was about to give this series one last read not expecting any questions
> to come up (this series has had a lot of feedback already!)
> Although I just did.
>
> What were your design considerations for the fanout table?
> Did you include it as the pack index has one or did you come up with
> them from first principles?
> Have you measured the performance impact of the fanout table
> (maybe even depending on the size of the fanout) ?
>
> context:
> https://public-inbox.org/git/CAJo=hJsto1ik=GTC8c3+2_jBuUqcAPL0UWp-1uoYYMpgbLB+qg@mail.gmail.com/
> (side note: searching the web for fanout makes it seem
> as if it is git-lingo, apparently the term is not widely used)
>
> I don't think we want to restart the design discussion,
> I am just curious.

I knew that I wanted some amount of a fanout table, and the 256-entry one was used for IDX files (and in my MIDX RFC). With the recent addition of "packfile: refactor hash search with fanout table" [1] it is probably best to keep the 256-entry table to reduce code clones.

As for speed, we have the notion of 'graph_pos' which gives random access into the commit-graph after a commit is loaded as a parent of a commit from the commit-graph file. Thus, we are spending time in the binary search only for commits that do not exist in the commit-graph file and those that are first found in the file. Thus, running profilers on long commit-graph walks do not show any measurable time spent in 'bsearch_graph()'.

Thanks, -Stolee

[1] https://github.com/gitster/git/commit/b4e00f7306a160639f047b3421985e8f3d0c6fb1

Previous: Stefan BellerNext: Jakub Narebski
Message 92 of 110 in “Serialized Git Commit Graph”
  1. 00/13 Serialized Git Commit GraphDerrick Stolee, Feb 27, 2018
  2. 01/13 commit-graph: add format documentDerrick Stolee, Feb 27, 2018
  3. 02/13 graph: add commit graph design documentDerrick Stolee, Feb 27, 2018
  4. 04/13 csum-file: add CSUM_KEEP_OPEN flagDerrick Stolee, Feb 27, 2018
  5. Derrick StoleeMar 12, 2018
  6. Junio C HamanoMar 13, 2018
  7. Derrick StoleeMar 14, 2018
  8. Junio C HamanoMar 14, 2018
  9. 06/13 commit-graph: implement 'git-commit-graph write'Derrick Stolee, Feb 27, 2018
  10. 05/13 commit-graph: implement write_commit_graph()Derrick Stolee, Feb 27, 2018
  11. 03/13 commit-graph: create git-commit-graph builtinDerrick Stolee, Feb 27, 2018
  12. 13/13 commit-graph: implement "--additive" optionDerrick Stolee, Feb 27, 2018
  13. 12/13 commit-graph: build graph from starting commitsDerrick Stolee, Feb 27, 2018
  14. 11/13 commit-graph: read only from specific pack-indexesDerrick Stolee, Feb 27, 2018
  15. Stefan BellerFeb 27, 2018
  16. 10/13 commit: integrate commit graph with commit parsingDerrick Stolee, Feb 27, 2018
  17. 07/13 commit-graph: implement git commit-graph readDerrick Stolee, Feb 27, 2018
  18. 08/13 commit-graph: add core.commitGraph settingDerrick Stolee, Feb 27, 2018
  19. 09/13 commit-graph: close under reachabilityDerrick Stolee, Feb 27, 2018
  20. Stefan BellerFeb 27, 2018
  21. 00/14 Serialized Git Commit GraphDerrick Stolee, Mar 14, 2018
  22. 02/14 csum-file: refactor finalize_hashfile() methodDerrick Stolee, Mar 14, 2018
  23. 03/14 commit-graph: add format documentDerrick Stolee, Mar 14, 2018
  24. 04/14 graph: add commit graph design documentDerrick Stolee, Mar 14, 2018
  25. 08/14 commit-graph: implement git commit-graph readDerrick Stolee, Mar 14, 2018
  26. 06/14 commit-graph: implement write_commit_graph()Derrick Stolee, Mar 14, 2018
  27. 10/14 commit-graph: close under reachabilityDerrick Stolee, Mar 14, 2018
  28. 12/14 commit-graph: read only from specific pack-indexesDerrick Stolee, Mar 14, 2018
  29. SZEDER GáborMar 15, 2018
  30. Derrick StoleeMar 19, 2018
  31. 09/14 commit-graph: add core.commitGraph settingDerrick Stolee, Mar 14, 2018
  32. 11/14 commit: integrate commit graph with commit parsingDerrick Stolee, Mar 14, 2018
  33. 13/14 commit-graph: build graph from starting commitsDerrick Stolee, Mar 14, 2018
  34. 14/14 commit-graph: implement "--additive" optionDerrick Stolee, Mar 14, 2018
  35. 05/14 commit-graph: create git-commit-graph builtinDerrick Stolee, Mar 14, 2018
  36. 07/14 commit-graph: implement 'git-commit-graph write'Derrick Stolee, Mar 14, 2018
  37. Ævar Arnfjörð BjarmasonMar 18, 2018
  38. Derrick StoleeMar 19, 2018
  39. Ævar Arnfjörð BjarmasonMar 19, 2018
  40. Derrick StoleeMar 19, 2018
  41. Ævar Arnfjörð BjarmasonMar 19, 2018
  42. 01/14 csum-file: rename hashclose() to finalize_hashfile()Derrick Stolee, Mar 14, 2018
  43. Ramsay JonesMar 14, 2018
  44. Junio C HamanoMar 14, 2018
  45. Johannes SchindelinMar 15, 2018
  46. Junio C HamanoMar 15, 2018
  47. Ramsay JonesMar 15, 2018
  48. Johannes SchindelinMar 16, 2018
  49. Junio C HamanoMar 16, 2018
  50. Johannes SchindelinMar 19, 2018
  51. Lars SchneiderMar 16, 2018
  52. Derrick StoleeMar 19, 2018
  53. Ævar Arnfjörð BjarmasonMar 16, 2018
  54. SZEDER GáborMar 16, 2018
  55. Junio C HamanoMar 16, 2018
  56. SZEDER GáborMar 16, 2018
  57. Jeff KingMar 16, 2018
  58. Jeff KingMar 16, 2018
  59. Derrick StoleeMar 19, 2018
  60. Derrick StoleeMar 20, 2018
  61. Jeff KingMar 16, 2018
  62. 00/14 Serialized Git Commit GraphDerrick Stolee, Apr 2, 2018
  63. 01/14 csum-file: rename hashclose() to finalize_hashfile()Derrick Stolee, Apr 2, 2018
  64. 02/14 csum-file: refactor finalize_hashfile() methodDerrick Stolee, Apr 2, 2018
  65. Jakub NarebskiApr 7, 2018
  66. 03/14 commit-graph: add format documentDerrick Stolee, Apr 2, 2018
  67. Jakub NarebskiApr 7, 2018
  68. 04/14 graph: add commit graph design documentDerrick Stolee, Apr 2, 2018
  69. Jakub NarebskiApr 8, 2018
  70. 06/14 commit-graph: implement write_commit_graph()Derrick Stolee, Apr 2, 2018
  71. 11/14 commit: integrate commit graph with commit parsingDerrick Stolee, Apr 2, 2018
  72. 13/14 commit-graph: build graph from starting commitsDerrick Stolee, Apr 2, 2018
  73. Jakub NarebskiApr 8, 2018
  74. 14/14 commit-graph: implement "--additive" optionDerrick Stolee, Apr 2, 2018
  75. SZEDER GáborApr 5, 2018
  76. 10/14 commit-graph: close under reachabilityDerrick Stolee, Apr 2, 2018
  77. 12/14 commit-graph: read only from specific pack-indexesDerrick Stolee, Apr 2, 2018
  78. 09/14 commit-graph: add core.commitGraph settingDerrick Stolee, Apr 2, 2018
  79. Jakub NarebskiApr 8, 2018
  80. 05/14 commit-graph: create git-commit-graph builtinDerrick Stolee, Apr 2, 2018
  81. 07/14 commit-graph: implement git-commit-graph writeDerrick Stolee, Apr 2, 2018
  82. Jakub NarebskiApr 8, 2018
  83. 08/14 commit-graph: implement git commit-graph readDerrick Stolee, Apr 2, 2018
  84. Junio C HamanoApr 2, 2018
  85. Derrick StoleeApr 3, 2018
  86. Jakub NarebskiApr 8, 2018
  87. 00/14 Serialized Git Commit GraphDerrick Stolee, Apr 10, 2018
  88. 01/14 csum-file: rename hashclose() to finalize_hashfile()Derrick Stolee, Apr 10, 2018
  89. 02/14 csum-file: refactor finalize_hashfile() methodDerrick Stolee, Apr 10, 2018
  90. 03/14 commit-graph: add format documentDerrick Stolee, Apr 10, 2018
  91. Stefan BellerApr 10, 2018
  92. Derrick StoleeApr 10, 2018
  93. Jakub NarebskiApr 11, 2018
  94. Derrick StoleeApr 12, 2018
  95. Jakub NarebskiApr 13, 2018
  96. 04/14 graph: add commit graph design documentDerrick Stolee, Apr 10, 2018
  97. Jakub NarebskiApr 15, 2018
  98. 05/14 commit-graph: create git-commit-graph builtinDerrick Stolee, Apr 10, 2018
  99. 07/14 commit-graph: implement git-commit-graph writeDerrick Stolee, Apr 10, 2018
  100. 08/14 commit-graph: implement git commit-graph readDerrick Stolee, Apr 10, 2018
  101. Jakub NarebskiApr 14, 2018
  102. Eric SunshineApr 15, 2018
  103. 09/14 commit-graph: add core.commitGraph settingDerrick Stolee, Apr 10, 2018
  104. Jakub NarebskiApr 14, 2018
  105. 06/14 commit-graph: implement write_commit_graph()Derrick Stolee, Apr 10, 2018
  106. 10/14 commit-graph: close under reachabilityDerrick Stolee, Apr 10, 2018
  107. 11/14 commit: integrate commit graph with commit parsingDerrick Stolee, Apr 10, 2018
  108. 12/14 commit-graph: read only from specific pack-indexesDerrick Stolee, Apr 10, 2018
  109. 13/14 commit-graph: build graph from starting commitsDerrick Stolee, Apr 10, 2018
  110. 14/14 commit-graph: implement "--append" optionDerrick Stolee, Apr 10, 2018

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.