git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH 18/41] index-pack: abstract away hash function constant

From
Duy Nguyen <pclouds@gmail.com>
Date
Apr 26, 2018, 15:46 UTC
Message-ID
<CACsJy8DUsFLDb786FmsR+eTriXaWGXEE+ZG8kCjq7JoipN1Phg@mail.gmail.com>
In-Reply-To
<CAN0heSqpj9JfTrnMFRbquraxve9iTwoowgWRUhcD-gXHMg3V=g@mail.gmail.com>
On Wed, Apr 25, 2018 at 8:49 PM, Martin Ågren <martin.agren@gmail.com> wrote:
Show 23 quoted lines
>> I agree that pack v2 is not going to have anything but SHA-1.  However,
>> writing all the code such that it's algorithm agnostic means that we can
>> do testing of new algorithms by wholesale replacing the algorithm with a
>> new one, which simplifies things considerably.
>
> Ok. I do sort of wonder if a "successful" test run after globally
> substituting Hash-Foo for SHA-1 (regardless of whether the size changes
> or not) hints at a problem. That is, nowhere do we test that this code
> uses 20-byte SHA-1s, regardless of what other hash functions are
> available and configured. Of course, until soon, that did not really
> have to be tested since there was only one hash function available to
> choose from. As for identifying all the places that matter ... no idea.
>
> Of course I can see how this helps get things to a point where Git does
> not crash and burn because the hash has a different size, and where the
> test suite doesn't spew failures because the initial chaining value of
> "SHA-1" is changed.
>
> Once that is accomplished, I sort of suspect that this code will want to
> be updated to not always blindly use the_hash_algo, but to always work
> with SHA-1 sizes. Or rather, this would turn into more generic code to
> handle both "v2 with SHA-1" and "v3 with some hash function(s)". This
> commit might be a good first step in that direction.

I also have an uneasy feeling when things this close to on-disk file format get hash-agnostic treatment. I think we would need to start adding new file formats soon, from bottom up with simple things like loose object files (cat-file and hash-object should be enough to test blobs...), then moving up to pack files and more. This is when we can really decide where to use the new hash and whether we should keep some hashes as sha-1.

For trailing hashes for example, there's no need to move to a new hash which only costs us more cycles. We just use it as a fancy checksum to avoid bit flips. But then my assumption about cost may be completely wrong without experimenting.

> Long rambling short, yeah, I see your point.

So yeah. It may be ok to move everything to "new hash" now. But we need a closer look soon.

-- 
Duy
Previous: Martin ÅgrenNext: brian m. carlson
Message 61 of 76 in “object_id part 13”
  1. 00/41 object_id part 13brian m. carlson, Apr 23, 2018
  2. 01/41 cache: add a function to read an object ID from a bufferbrian m. carlson, Apr 23, 2018
  3. Martin ÅgrenApr 24, 2018
  4. Duy NguyenMay 1, 2018
  5. brian m. carlsonMay 1, 2018
  6. 02/41 server-info: remove unused members from struct pack_infobrian m. carlson, Apr 23, 2018
  7. Martin ÅgrenApr 24, 2018
  8. Duy NguyenMay 1, 2018
  9. 03/41 Remove unused member in struct object_contextbrian m. carlson, Apr 23, 2018
  10. Duy NguyenMay 1, 2018
  11. 04/41 packfile: remove unused member from struct pack_entrybrian m. carlson, Apr 23, 2018
  12. Duy NguyenMay 1, 2018
  13. 06/41 sha1_file: convert freshen functions to object_idbrian m. carlson, Apr 23, 2018
  14. 07/41 packfile: convert find_pack_entry to object_idbrian m. carlson, Apr 23, 2018
  15. 09/41 pack-objects: abstract away hash algorithmbrian m. carlson, Apr 23, 2018
  16. Duy NguyenMay 1, 2018
  17. 17/41 pack-redundant: convert linked lists to use struct object_idbrian m. carlson, Apr 23, 2018
  18. 16/41 Update struct index_state to use struct object_idbrian m. carlson, Apr 23, 2018
  19. 22/41 revision: replace use of hard-coded constantsbrian m. carlson, Apr 23, 2018
  20. 21/41 http: eliminate hard-coded constantsbrian m. carlson, Apr 23, 2018
  21. Martin ÅgrenApr 24, 2018
  22. Junio C HamanoApr 24, 2018
  23. brian m. carlsonApr 25, 2018
  24. 20/41 dir: convert struct untracked_cache_dir to object_idbrian m. carlson, Apr 23, 2018
  25. 25/41 builtin/receive-pack: avoid hard-coded constants for push certsbrian m. carlson, Apr 23, 2018
  26. Martin ÅgrenApr 24, 2018
  27. brian m. carlsonApr 25, 2018
  28. Martin ÅgrenApr 25, 2018
  29. 28/41 merge: convert empty tree constant to the_hash_algobrian m. carlson, Apr 23, 2018
  30. 36/41 sequencer: use the_hash_algo for empty tree object IDbrian m. carlson, Apr 23, 2018
  31. 34/41 sha1_file: convert cached object code to struct object_idbrian m. carlson, Apr 23, 2018
  32. 30/41 submodule: convert several uses of EMPTY_TREE_SHA1_HEXbrian m. carlson, Apr 23, 2018
  33. 40/41 add--interactive: compute the empty tree valuebrian m. carlson, Apr 23, 2018
  34. 39/41 Update shell scripts to compute empty tree object IDbrian m. carlson, Apr 23, 2018
  35. Duy NguyenMay 1, 2018
  36. brian m. carlsonMay 4, 2018
  37. 38/41 sha1_file: only expose empty object constants through git_hash_algobrian m. carlson, Apr 23, 2018
  38. 37/41 dir: use the_hash_algo for empty blob object IDbrian m. carlson, Apr 23, 2018
  39. 41/41 merge-one-file: compute empty blob object IDbrian m. carlson, Apr 23, 2018
  40. SZEDER GáborApr 24, 2018
  41. brian m. carlsonApr 24, 2018
  42. 35/41 cache-tree: use is_empty_tree_oidbrian m. carlson, Apr 23, 2018
  43. 33/41 builtin/reset: convert use of EMPTY_TREE_SHA1_BINbrian m. carlson, Apr 23, 2018
  44. 31/41 wt-status: convert two uses of EMPTY_TREE_SHA1_HEXbrian m. carlson, Apr 23, 2018
  45. Martin ÅgrenApr 24, 2018
  46. brian m. carlsonMay 1, 2018
  47. 32/41 builtin/receive-pack: convert one use of EMPTY_TREE_SHA1_HEXbrian m. carlson, Apr 23, 2018
  48. 29/41 sequencer: convert one use of EMPTY_TREE_SHA1_HEXbrian m. carlson, Apr 23, 2018
  49. 27/41 builtin/merge: switch tree functions to use object_idbrian m. carlson, Apr 23, 2018
  50. 23/41 upload-pack: replace use of several hard-coded constantsbrian m. carlson, Apr 23, 2018
  51. Simon RuderichApr 24, 2018
  52. 26/41 builtin/am: convert uses of EMPTY_TREE_SHA1_BIN to the_hash_algobrian m. carlson, Apr 23, 2018
  53. 13/41 fsck: convert static functions to struct object_idbrian m. carlson, Apr 23, 2018
  54. 24/41 diff: specify abbreviation size in terms of the_hash_algobrian m. carlson, Apr 23, 2018
  55. 19/41 commit: convert uses of get_sha1_hex to get_oid_hexbrian m. carlson, Apr 23, 2018
  56. 05/41 packfile: convert has_sha1_pack to object_idbrian m. carlson, Apr 23, 2018
  57. 18/41 index-pack: abstract away hash function constantbrian m. carlson, Apr 23, 2018
  58. Martin ÅgrenApr 24, 2018
  59. brian m. carlsonApr 24, 2018
  60. Martin ÅgrenApr 25, 2018
  61. Duy NguyenApr 26, 2018
  62. brian m. carlsonApr 27, 2018
  63. Duy NguyenApr 28, 2018
  64. 15/41 split-index: convert struct split_index to object_idbrian m. carlson, Apr 23, 2018
  65. 14/41 submodule-config: convert structures to object_idbrian m. carlson, Apr 23, 2018
  66. 12/41 tree-walk: convert get_tree_entry_follow_symlinks to object_idbrian m. carlson, Apr 23, 2018
  67. 11/41 tree-walk: avoid hard-coded 20 constantbrian m. carlson, Apr 23, 2018
  68. 10/41 pack-redundant: abstract away hash algorithmbrian m. carlson, Apr 23, 2018
  69. 08/41 packfile: abstract away hash constant valuesbrian m. carlson, Apr 23, 2018
  70. Duy NguyenMay 1, 2018
  71. brian m. carlsonMay 2, 2018
  72. Duy NguyenMay 2, 2018
  73. brian m. carlsonMay 2, 2018
  74. Duy NguyenApr 30, 2018
  75. brian m. carlsonApr 30, 2018
  76. Duy NguyenMay 1, 2018

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.