git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH 0/3] Convert index writes to use hashfile API

From
Derrick Stolee via GitGitGadget <gitgitgadget@gmail.com>
Date
Mar 26, 2021, 19:12 UTC
Message-ID
<pull.916.git.1616785928.gitgitgadget@gmail.com>

As I prepare some ideas on index v5, one thing that strikes me as an interesting direction to try is to use the chunk-format API. This would make our extension model extremely simple (they become optional chunks, easily identified by the table of contents).

But there is a huge hurdle to even starting that investigation: the index uses its own hashing methods, separate from the hashfile API in csum-file.c!

The internals of the algorithms are mostly identical. The only possible change is that the buffer sizes are different: 8KB for hashfile and 128KB in read-cache.c. I was unable to find a performance difference in these two implementations, despite testing on several repo sizes.

There is a subtle point about how the EOIE extension works in that it needs a hash of just the previous extension data. This is solved by adding a new "nested hashfile" mechanism that computes the hash at one level and then passes the data below to another hashfile. (The good news is that this extension will not need to exist at all if we use the chunk-format API to manage extensions.)

Thanks, -Stolee
Derrick Stolee (3):
  csum-file: add nested_hashfile()
  read-cache: use hashfile instead of git_hash_ctx
  read-cache: delete unused hashing methods
 csum-file.c  |  22 +++++++
 csum-file.h  |   9 +++
 read-cache.c | 182 ++++++++++++++++-----------------------------------
 3 files changed, 89 insertions(+), 124 deletions(-)
base-commit: 142430338477d9d1bb25be66267225fb58498d92
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-916%2Fderrickstolee%2Findex-hashfile-v1
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-916/derrickstolee/index-hashfile-v1
Pull-Request: https://github.com/gitgitgadget/git/pull/916
-- 
gitgitgadget
Next: Derrick Stolee via GitGitGadget
Message 1 of 27 in “Convert index writes to use hashfile API”
  1. 0/3 Convert index writes to use hashfile APIDerrick Stolee via GitGitGadget, Mar 26, 2021
  2. 3/3 read-cache: delete unused hashing methodsDerrick Stolee via GitGitGadget, Mar 26, 2021
  3. 1/3 csum-file: add nested_hashfile()Derrick Stolee via GitGitGadget, Mar 26, 2021
  4. 2/3 read-cache: use hashfile instead of git_hash_ctxDerrick Stolee via GitGitGadget, Mar 26, 2021
  5. Derrick StoleeMar 29, 2021
  6. Derrick StoleeMar 29, 2021
  7. Derrick StoleeMar 26, 2021
  8. 0/4 Convert index writes to use hashfile APIDerrick Stolee via GitGitGadget, May 17, 2021
  9. 1/4 hashfile: use write_in_full()Derrick Stolee via GitGitGadget, May 17, 2021
  10. 3/4 read-cache: use hashfile instead of git_hash_ctxDerrick Stolee via GitGitGadget, May 17, 2021
  11. Junio C HamanoMay 17, 2021
  12. Derrick StoleeMay 18, 2021
  13. 2/4 csum-file.h: increase hashfile buffer sizeDerrick Stolee via GitGitGadget, May 17, 2021
  14. Junio C HamanoMay 17, 2021
  15. Jeff KingMay 18, 2021
  16. Derrick StoleeMay 18, 2021
  17. Jeff KingMay 18, 2021
  18. Jeff KingMay 18, 2021
  19. 4/4 read-cache: delete unused hashing methodsDerrick Stolee via GitGitGadget, May 17, 2021
  20. 0/4 Convert index writes to use hashfile APIDerrick Stolee via GitGitGadget, May 18, 2021
  21. 2/4 csum-file.h: increase hashfile buffer sizeDerrick Stolee via GitGitGadget, May 18, 2021
  22. t4216-log-bloom.sh fails with -v (but not --verbose-log)Ævar Arnfjörð Bjarmason, Nov 25, 2021
  23. Jeff KingNov 26, 2021
  24. Derrick StoleeNov 29, 2021
  25. 1/4 hashfile: use write_in_full()Derrick Stolee via GitGitGadget, May 18, 2021
  26. 3/4 read-cache: use hashfile instead of git_hash_ctxDerrick Stolee via GitGitGadget, May 18, 2021
  27. 4/4 read-cache: delete unused hashing methodsDerrick Stolee via GitGitGadget, May 18, 2021

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.