git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH 0/4] hash.h: support choosing a separate SHA-1 for non-cryptographic uses

From
Taylor Blau <me@ttaylorr.com>
Date
Sep 3, 2024, 19:48 UTC
Message-ID
<ZtdoD/slbc+u7Aql@nand.local>
In-Reply-To
<xmqq8qwaoh73.fsf@gitster.g>
On Sun, Sep 01, 2024 at 08:41:36PM -0700, Junio C Hamano wrote:
Show 15 quoted lines
> Taylor Blau <me@ttaylorr.com> writes:
>
> > After some profiling, I noticed that we spend a significant amount of
> > time in hashwrite(), which is not all that surprising. But much of that
> > time is wasted in GitHub's infrastructure, since we are using the same
> > collision-detecting SHA-1 implementation to produce a trailing checksum
> > for the pack which does not need to be cryptographically secure.
>
> Cute.
>
> I wish we can upgrade the file formats so that a writer can choose
> the hash algorithm independently from whatever the payload uses.
> Most of our use of the tail sum are for files that are consumed
> locally in the same repository so nobody shouldn't need to know that
> you are using xxHash for the tail sum instead.

Yeah. I would actually like to get here in the long-term, but of course that change is much larger than this one (the protocol would have to be adjusted to learn a new "tailsum" capability for callers to negotiate which checksumming hash function they want to use, etc.).

Show 5 quoted lines
> Except that the story is not so simple for packfiles, which is named
> after the file's tail sum, so you cannot use a hash algorithm of
> your choice independently without affecting other folks.  All other
> csum-file protected file types are lot smaller than the pack files
> to matter, sadly.
Right.

Thanks, Taylor

Previous: Junio C HamanoNext: Junio C Hamano
Message 22 of 99 in “hash.h: support choosing a separate SHA-1 for non-cryptographic uses”
  1. 0/4 hash.h: support choosing a separate SHA-1 for non-cryptographic usesTaylor Blau, Sep 1, 2024
  2. 1/4 sha1: do not redefine `platform_SHA_CTX` and friendsTaylor Blau, Sep 1, 2024
  3. Patrick SteinhardtSep 2, 2024
  4. Taylor BlauSep 3, 2024
  5. 2/4 hash.h: scaffolding for _fast hashing variantsTaylor Blau, Sep 1, 2024
  6. Patrick SteinhardtSep 2, 2024
  7. Junio C HamanoSep 3, 2024
  8. Taylor BlauSep 3, 2024
  9. Junio C HamanoSep 3, 2024
  10. Taylor BlauSep 3, 2024
  11. Patrick SteinhardtSep 4, 2024
  12. Junio C HamanoSep 4, 2024
  13. Taylor BlauSep 3, 2024
  14. 3/4 Makefile: allow specifying a SHA-1 for non-cryptographic usesTaylor Blau, Sep 1, 2024
  15. Patrick SteinhardtSep 2, 2024
  16. Taylor BlauSep 3, 2024
  17. 4/4 csum-file.c: use fast SHA-1 implementation when availableTaylor Blau, Sep 1, 2024
  18. Patrick SteinhardtSep 2, 2024
  19. brian m. carlsonSep 3, 2024
  20. Taylor BlauSep 3, 2024
  21. Junio C HamanoSep 2, 2024
  22. Taylor BlauSep 3, 2024
  23. Junio C HamanoSep 3, 2024
  24. brian m. carlsonSep 2, 2024
  25. Taylor BlauSep 3, 2024
  26. Junio C HamanoSep 3, 2024
  27. brian m. carlsonSep 4, 2024
  28. Jeff KingSep 5, 2024
  29. Junio C HamanoSep 5, 2024
  30. Taylor BlauSep 5, 2024
  31. Junio C HamanoSep 5, 2024
  32. Taylor BlauSep 5, 2024
  33. Taylor BlauSep 5, 2024
  34. Taylor BlauSep 5, 2024
  35. Jeff KingSep 5, 2024
  36. Junio C HamanoSep 5, 2024
  37. 0/4 hash.h: support choosing a separate SHA-1 for non-cryptographic usesTaylor Blau, Sep 5, 2024
  38. 1/4 sha1: do not redefine `platform_SHA_CTX` and friendsTaylor Blau, Sep 5, 2024
  39. 2/4 hash.h: scaffolding for _fast hashing variantsTaylor Blau, Sep 5, 2024
  40. 3/4 Makefile: allow specifying a SHA-1 for non-cryptographic usesTaylor Blau, Sep 5, 2024
  41. 4/4 csum-file.c: use fast SHA-1 implementation when availableTaylor Blau, Sep 5, 2024
  42. 0/9 hash.h: support choosing a separate SHA-1 for non-cryptographic usesTaylor Blau, Sep 6, 2024
  43. 1/9 finalize_object_file(): check for name collision before renamingTaylor Blau, Sep 6, 2024
  44. 2/9 finalize_object_file(): refactor unlink_or_warn() placementTaylor Blau, Sep 6, 2024
  45. 3/9 finalize_object_file(): implement collision checkTaylor Blau, Sep 6, 2024
  46. Junio C HamanoSep 6, 2024
  47. Chris TorekSep 6, 2024
  48. Jeff KingSep 10, 2024
  49. Junio C HamanoSep 10, 2024
  50. Patrick SteinhardtSep 16, 2024
  51. Taylor BlauSep 16, 2024
  52. Taylor BlauSep 16, 2024
  53. Junio C HamanoSep 17, 2024
  54. 4/9 pack-objects: use finalize_object_file() to rename pack/idx/etcTaylor Blau, Sep 6, 2024
  55. 5/9 i5500-git-daemon.sh: use compile-able version of Git without OpenSSLTaylor Blau, Sep 6, 2024
  56. Jeff KingSep 11, 2024
  57. Jeff KingSep 11, 2024
  58. Junio C HamanoSep 12, 2024
  59. Junio C HamanoSep 11, 2024
  60. Jeff KingSep 11, 2024
  61. 6/9 sha1: do not redefine `platform_SHA_CTX` and friendsTaylor Blau, Sep 6, 2024
  62. 7/9 hash.h: scaffolding for _fast hashing variantsTaylor Blau, Sep 6, 2024
  63. 8/9 Makefile: allow specifying a SHA-1 for non-cryptographic usesTaylor Blau, Sep 6, 2024
  64. 9/9 csum-file.c: use fast SHA-1 implementation when availableTaylor Blau, Sep 6, 2024
  65. Junio C HamanoSep 6, 2024
  66. 0/8 hash.h: support choosing a separate SHA-1 for non-cryptographic usesTaylor Blau, Sep 24, 2024
  67. 1/8 finalize_object_file(): check for name collision before renamingTaylor Blau, Sep 24, 2024
  68. Junio C HamanoSep 25, 2024
  69. 2/8 finalize_object_file(): refactor unlink_or_warn() placementTaylor Blau, Sep 24, 2024
  70. 3/8 finalize_object_file(): implement collision checkTaylor Blau, Sep 24, 2024
  71. Jeff KingSep 24, 2024
  72. Taylor BlauSep 24, 2024
  73. Jeff KingSep 24, 2024
  74. Taylor BlauSep 25, 2024
  75. Junio C HamanoSep 24, 2024
  76. Taylor BlauSep 24, 2024
  77. 4/8 pack-objects: use finalize_object_file() to rename pack/idx/etcTaylor Blau, Sep 24, 2024
  78. Junio C HamanoSep 24, 2024
  79. 5/8 sha1: do not redefine `platform_SHA_CTX` and friendsTaylor Blau, Sep 24, 2024
  80. 6/8 hash.h: scaffolding for _unsafe hashing variantsTaylor Blau, Sep 24, 2024
  81. 7/8 Makefile: allow specifying a SHA-1 for non-cryptographic usesTaylor Blau, Sep 24, 2024
  82. 8/8 csum-file.c: use unsafe SHA-1 implementation when availableTaylor Blau, Sep 24, 2024
  83. Jeff KingSep 24, 2024
  84. Elijah NewrenSep 25, 2024
  85. Junio C HamanoSep 25, 2024
  86. Taylor BlauSep 25, 2024
  87. Taylor BlauSep 25, 2024
  88. 0/8 hash.h: support choosing a separate SHA-1 for non-cryptographic usesTaylor Blau, Sep 26, 2024
  89. 1/8 finalize_object_file(): check for name collision before renamingTaylor Blau, Sep 26, 2024
  90. 2/8 finalize_object_file(): refactor unlink_or_warn() placementTaylor Blau, Sep 26, 2024
  91. 3/8 finalize_object_file(): implement collision checkTaylor Blau, Sep 26, 2024
  92. 4/8 pack-objects: use finalize_object_file() to rename pack/idx/etcTaylor Blau, Sep 26, 2024
  93. 5/8 sha1: do not redefine `platform_SHA_CTX` and friendsTaylor Blau, Sep 26, 2024
  94. 6/8 hash.h: scaffolding for _unsafe hashing variantsTaylor Blau, Sep 26, 2024
  95. 7/8 Makefile: allow specifying a SHA-1 for non-cryptographic usesTaylor Blau, Sep 26, 2024
  96. 8/8 csum-file.c: use unsafe SHA-1 implementation when availableTaylor Blau, Sep 26, 2024
  97. Elijah NewrenSep 26, 2024
  98. Junio C HamanoSep 27, 2024
  99. Jeff KingSep 27, 2024

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.