git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH v7 03/14] commit-graph: add format document

From
Jakub Narebski <jnareb@gmail.com>
Date
Apr 7, 2018, 23:49 UTC
Message-ID
<86efjqefww.fsf@gmail.com>
In-Reply-To
<20180402203427.170177-4-dstolee@microsoft.com>
Derrick Stolee <stolee@gmail.com> writes:
Show 18 quoted lines
> diff --git a/Documentation/technical/commit-graph-format.txt
> b/Documentation/technical/commit-graph-format.txt
> new file mode 100644
> index 0000000000..ad6af8105c
> --- /dev/null
> +++ b/Documentation/technical/commit-graph-format.txt
> @@ -0,0 +1,97 @@
> +Git commit graph format
> +=======================
> +
> +The Git commit graph stores a list of commit OIDs and some associated
> +metadata, including:
> +
> +- The generation number of the commit. Commits with no parents have
> +  generation number 1; commits with parents have generation number
> +  one more than the maximum generation number of its parents. We
> +  reserve zero as special, and can be used to mark a generation
> +  number invalid or as "not computed".

Actually we will be reserving "all bits 1" as special, though I don't think it is worth mentioning here, and at this time.

Show 26 quoted lines
> +
> +- The root tree OID.
> +
> +- The commit date.
> +
> +- The parents of the commit, stored using positional references within
> +  the graph file.
> +
> +These positional references are stored as unsigned 32-bit integers
> +corresponding to the array position withing the list of commit OIDs. We
> +use the most-significant bit for special purposes, so we can store at most
> +(1 << 31) - 1 (around 2 billion) commits.
> +
> +== Commit graph files have the following format:
> +
> +In order to allow extensions that add extra data to the graph, we organize
> +the body into "chunks" and provide a binary lookup table at the beginning
> +of the body. The header includes certain values, such as number of chunks
> +and hash type.
> +
> +All 4-byte numbers are in network order.
> +
> +HEADER:
> +
> +  4-byte signature:
> +      The signature is: {'C', 'G', 'P', 'H'}
The mnemonics: CGPH = Commit GraPH
Show 11 quoted lines
> +
> +  1-byte version number:
> +      Currently, the only valid version is 1.
> +
> +  1-byte Hash Version (1 = SHA-1)
> +      We infer the hash length (H) from this value.
> +
> +  1-byte number (C) of "chunks"
> +
> +  1-byte (reserved for later use)
> +     Current clients should ignore this value.

All right, with this reserved byte that makes header word-aligned (be it 32-bit or 64-bit)

Show 5 quoted lines
> +
> +CHUNK LOOKUP:
> +
> +  (C + 1) * 12 bytes listing the table of contents for the chunks:
> +      First 4 bytes describe the chunk id. Value 0 is a terminating label.

As I understand it, it is value 0 as 4-byte integer, that is 4 x byte 0. This may need clarification (or may need not).

Show 8 quoted lines
> +      Other 8 bytes provide the byte-offset in current file for chunk to
> +      start. (Chunks are ordered contiguously in the file, so you can infer
> +      the length using the next chunk position if necessary.) Each chunk
> +      ID appears at most once.
> +
> +  The remaining data in the body is described one chunk at a time, and
> +  these chunks may be given in any order. Chunks are required unless
> +  otherwise specified.

Perhaps there should be here list of all required chunks, and list of all optional chunks.

> +
> +CHUNK DATA:
> +
> +  OID Fanout (ID: {'O', 'I', 'D', 'F'}) (256 * 4 bytes)
The mnemonics: OIDF = CID Fanout
> +      The ith entry, F[i], stores the number of OIDs with first
> +      byte at most i. Thus F[255] stores the total
> +      number of commits (N).

So in other words this is cumulative histogram? Just ensuring that I understand it correctly.

> +
> +  OID Lookup (ID: {'O', 'I', 'D', 'L'}) (N * H bytes)
The mnemonics: OIDL = OID Lookup
> +      The OIDs for all commits in the graph, sorted in ascending order.
> +
> +  Commit Data (ID: {'C', 'G', 'E', 'T' }) (N * (H + 16) bytes)
The mnemonics: CGET = ???
It is not CDAT, CGDT, DATA, etc.
Show 6 quoted lines
> +    * The first H bytes are for the OID of the root tree.
> +    * The next 8 bytes are for the positions of the first two parents
> +      of the ith commit. Stores value 0xffffffff if no parent in that
> +      position. If there are more than two parents, the second value
> +      has its most-significant bit on and the other bits store an array
> +      position into the Large Edge List chunk.
Possibly better:
  +      position into the Large Edge List chunk (EDGE chunk).
> +    * The next 8 bytes store the generation number of the commit and
> +      the commit time in seconds since EPOCH.

The commit time is committer date (without timezone info), that is the date that the commit object was created, isn't it?

Show 7 quoted lines
>                                                 The generation number
> +      uses the higher 30 bits of the first 4 bytes, while the commit
> +      time uses the 32 bits of the second 4 bytes, along with the lowest
> +      2 bits of the lowest byte, storing the 33rd and 34th bit of the
> +      commit time.
> +
> +  Large Edge List (ID: {'E', 'D', 'G', 'E'}) [Optional]

Well, it is optional in the sense that it may not be here if the project doesn't have any octopus merges. It is not optional in the sense that Git can ignore this chunk if it doesn't know its type.

Sidenote: PNG has critical and ancillary chunks [1]. Critical chunks
contain information that is necessary to read the file. If a decoder
encounters a critical chunk it does not recognize, it must abort reading
the file or supply the user with an appropriate warning.  If the first
letter of chunk name is uppercase, the chunk is critical; if not, the
chunk is ancillary.
[1]: https://en.wikipedia.org/wiki/Portable_Network_Graphics#%22Chunks%22_within_the_file
Show 6 quoted lines
> +      This list of 4-byte values store the second through nth parents for
> +      all octopus merges. The second parent value in the commit data stores
> +      an array position within this list along with the most-significant bit
> +      on. Starting at that array position, iterate through this list of commit
> +      positions for the parents until reaching a value with the most-significant
> +      bit on. The other bits correspond to the position of the last parent.
The second sentence in the above paragraph is not entirely clean to me.
> +
> +TRAILER:
> +
> +	H-byte HASH-checksum of all of the above.
Best,
-- 
Jakub Narębski
Previous: Derrick StoleeNext: Derrick Stolee
Message 67 of 110 in “Serialized Git Commit Graph”
  1. 00/13 Serialized Git Commit GraphDerrick Stolee, Feb 27, 2018
  2. 01/13 commit-graph: add format documentDerrick Stolee, Feb 27, 2018
  3. 02/13 graph: add commit graph design documentDerrick Stolee, Feb 27, 2018
  4. 04/13 csum-file: add CSUM_KEEP_OPEN flagDerrick Stolee, Feb 27, 2018
  5. Derrick StoleeMar 12, 2018
  6. Junio C HamanoMar 13, 2018
  7. Derrick StoleeMar 14, 2018
  8. Junio C HamanoMar 14, 2018
  9. 06/13 commit-graph: implement 'git-commit-graph write'Derrick Stolee, Feb 27, 2018
  10. 05/13 commit-graph: implement write_commit_graph()Derrick Stolee, Feb 27, 2018
  11. 03/13 commit-graph: create git-commit-graph builtinDerrick Stolee, Feb 27, 2018
  12. 13/13 commit-graph: implement "--additive" optionDerrick Stolee, Feb 27, 2018
  13. 12/13 commit-graph: build graph from starting commitsDerrick Stolee, Feb 27, 2018
  14. 11/13 commit-graph: read only from specific pack-indexesDerrick Stolee, Feb 27, 2018
  15. Stefan BellerFeb 27, 2018
  16. 10/13 commit: integrate commit graph with commit parsingDerrick Stolee, Feb 27, 2018
  17. 07/13 commit-graph: implement git commit-graph readDerrick Stolee, Feb 27, 2018
  18. 08/13 commit-graph: add core.commitGraph settingDerrick Stolee, Feb 27, 2018
  19. 09/13 commit-graph: close under reachabilityDerrick Stolee, Feb 27, 2018
  20. Stefan BellerFeb 27, 2018
  21. 00/14 Serialized Git Commit GraphDerrick Stolee, Mar 14, 2018
  22. 02/14 csum-file: refactor finalize_hashfile() methodDerrick Stolee, Mar 14, 2018
  23. 03/14 commit-graph: add format documentDerrick Stolee, Mar 14, 2018
  24. 04/14 graph: add commit graph design documentDerrick Stolee, Mar 14, 2018
  25. 08/14 commit-graph: implement git commit-graph readDerrick Stolee, Mar 14, 2018
  26. 06/14 commit-graph: implement write_commit_graph()Derrick Stolee, Mar 14, 2018
  27. 10/14 commit-graph: close under reachabilityDerrick Stolee, Mar 14, 2018
  28. 12/14 commit-graph: read only from specific pack-indexesDerrick Stolee, Mar 14, 2018
  29. SZEDER GáborMar 15, 2018
  30. Derrick StoleeMar 19, 2018
  31. 09/14 commit-graph: add core.commitGraph settingDerrick Stolee, Mar 14, 2018
  32. 11/14 commit: integrate commit graph with commit parsingDerrick Stolee, Mar 14, 2018
  33. 13/14 commit-graph: build graph from starting commitsDerrick Stolee, Mar 14, 2018
  34. 14/14 commit-graph: implement "--additive" optionDerrick Stolee, Mar 14, 2018
  35. 05/14 commit-graph: create git-commit-graph builtinDerrick Stolee, Mar 14, 2018
  36. 07/14 commit-graph: implement 'git-commit-graph write'Derrick Stolee, Mar 14, 2018
  37. Ævar Arnfjörð BjarmasonMar 18, 2018
  38. Derrick StoleeMar 19, 2018
  39. Ævar Arnfjörð BjarmasonMar 19, 2018
  40. Derrick StoleeMar 19, 2018
  41. Ævar Arnfjörð BjarmasonMar 19, 2018
  42. 01/14 csum-file: rename hashclose() to finalize_hashfile()Derrick Stolee, Mar 14, 2018
  43. Ramsay JonesMar 14, 2018
  44. Junio C HamanoMar 14, 2018
  45. Johannes SchindelinMar 15, 2018
  46. Junio C HamanoMar 15, 2018
  47. Ramsay JonesMar 15, 2018
  48. Johannes SchindelinMar 16, 2018
  49. Junio C HamanoMar 16, 2018
  50. Johannes SchindelinMar 19, 2018
  51. Lars SchneiderMar 16, 2018
  52. Derrick StoleeMar 19, 2018
  53. Ævar Arnfjörð BjarmasonMar 16, 2018
  54. SZEDER GáborMar 16, 2018
  55. Junio C HamanoMar 16, 2018
  56. SZEDER GáborMar 16, 2018
  57. Jeff KingMar 16, 2018
  58. Jeff KingMar 16, 2018
  59. Derrick StoleeMar 19, 2018
  60. Derrick StoleeMar 20, 2018
  61. Jeff KingMar 16, 2018
  62. 00/14 Serialized Git Commit GraphDerrick Stolee, Apr 2, 2018
  63. 01/14 csum-file: rename hashclose() to finalize_hashfile()Derrick Stolee, Apr 2, 2018
  64. 02/14 csum-file: refactor finalize_hashfile() methodDerrick Stolee, Apr 2, 2018
  65. Jakub NarebskiApr 7, 2018
  66. 03/14 commit-graph: add format documentDerrick Stolee, Apr 2, 2018
  67. Jakub NarebskiApr 7, 2018
  68. 04/14 graph: add commit graph design documentDerrick Stolee, Apr 2, 2018
  69. Jakub NarebskiApr 8, 2018
  70. 06/14 commit-graph: implement write_commit_graph()Derrick Stolee, Apr 2, 2018
  71. 11/14 commit: integrate commit graph with commit parsingDerrick Stolee, Apr 2, 2018
  72. 13/14 commit-graph: build graph from starting commitsDerrick Stolee, Apr 2, 2018
  73. Jakub NarebskiApr 8, 2018
  74. 14/14 commit-graph: implement "--additive" optionDerrick Stolee, Apr 2, 2018
  75. SZEDER GáborApr 5, 2018
  76. 10/14 commit-graph: close under reachabilityDerrick Stolee, Apr 2, 2018
  77. 12/14 commit-graph: read only from specific pack-indexesDerrick Stolee, Apr 2, 2018
  78. 09/14 commit-graph: add core.commitGraph settingDerrick Stolee, Apr 2, 2018
  79. Jakub NarebskiApr 8, 2018
  80. 05/14 commit-graph: create git-commit-graph builtinDerrick Stolee, Apr 2, 2018
  81. 07/14 commit-graph: implement git-commit-graph writeDerrick Stolee, Apr 2, 2018
  82. Jakub NarebskiApr 8, 2018
  83. 08/14 commit-graph: implement git commit-graph readDerrick Stolee, Apr 2, 2018
  84. Junio C HamanoApr 2, 2018
  85. Derrick StoleeApr 3, 2018
  86. Jakub NarebskiApr 8, 2018
  87. 00/14 Serialized Git Commit GraphDerrick Stolee, Apr 10, 2018
  88. 01/14 csum-file: rename hashclose() to finalize_hashfile()Derrick Stolee, Apr 10, 2018
  89. 02/14 csum-file: refactor finalize_hashfile() methodDerrick Stolee, Apr 10, 2018
  90. 03/14 commit-graph: add format documentDerrick Stolee, Apr 10, 2018
  91. Stefan BellerApr 10, 2018
  92. Derrick StoleeApr 10, 2018
  93. Jakub NarebskiApr 11, 2018
  94. Derrick StoleeApr 12, 2018
  95. Jakub NarebskiApr 13, 2018
  96. 04/14 graph: add commit graph design documentDerrick Stolee, Apr 10, 2018
  97. Jakub NarebskiApr 15, 2018
  98. 05/14 commit-graph: create git-commit-graph builtinDerrick Stolee, Apr 10, 2018
  99. 07/14 commit-graph: implement git-commit-graph writeDerrick Stolee, Apr 10, 2018
  100. 08/14 commit-graph: implement git commit-graph readDerrick Stolee, Apr 10, 2018
  101. Jakub NarebskiApr 14, 2018
  102. Eric SunshineApr 15, 2018
  103. 09/14 commit-graph: add core.commitGraph settingDerrick Stolee, Apr 10, 2018
  104. Jakub NarebskiApr 14, 2018
  105. 06/14 commit-graph: implement write_commit_graph()Derrick Stolee, Apr 10, 2018
  106. 10/14 commit-graph: close under reachabilityDerrick Stolee, Apr 10, 2018
  107. 11/14 commit: integrate commit graph with commit parsingDerrick Stolee, Apr 10, 2018
  108. 12/14 commit-graph: read only from specific pack-indexesDerrick Stolee, Apr 10, 2018
  109. 13/14 commit-graph: build graph from starting commitsDerrick Stolee, Apr 10, 2018
  110. 14/14 commit-graph: implement "--append" optionDerrick Stolee, Apr 10, 2018

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.