{"thread":{"id":"62522","subject":"[PATCH 0/6] hash: introduce unsafe_hash_algo(), drop unsafe_ variants","startedAt":"2024-11-20T19:13:44Z","lastAt":"2025-01-23T18:50:55Z","messageCount":60,"participants":["Taylor Blau","Jeff King","brian m. carlson","Junio C Hamano","Patrick Steinhardt"],"isPatch":true,"patchVersion":1,"patchTotal":6},"messages":[{"id":"507773","messageId":"cover.1732130001.git.me@ttaylorr.com","threadId":"62522","inReplyTo":null,"subject":"[PATCH 0/6] hash: introduce unsafe_hash_algo(), drop unsafe_ variants","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-11-20T19:13:41Z","receivedAt":"2024-11-20T19:13:44Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"(This series is based on my 'tb/sha1-unsafe-helper', which I sent to the\nlist here[1].)\n\nThis series implements an idea discussed in [2] which suggests that we\nintroduce a way to access a wrapped version of a 'struct git_hash_algo'\nwhich represents the unsafe variant of that algorithm, rather than\nhaving individual unsafe_ functions (like unsafe_init_fn() versus\ninit_fn(), etc.).\n\nThis approach is relatively straightforward to implement, and removes a\nsignificant deficiency in the original implementation of\nunsafe/non-cryptographic hash functions by making it impossible to\nswitch between safe- and unsafe variants of hash functions. It also\ncleans up the sha1-unsafe test helper's implementation by removing a\nlarge number of \"if (unsafe)\"-style conditionals.\n\nThe series is laid out as follows:\n\n  * The first two patches prepare the hashfile API for the upcoming\n    change:\n\n      csum-file: store the hash algorithm as a struct field\n      csum-file.c: extract algop from hashfile_checksum_valid()\n\n  * The next patch implements the new 'unsafe_hash_algo()' function at\n    the heart of this series' approach:\n\n      hash.h: introduce `unsafe_hash_algo()`\n\n  * The next two patches convert existing callers to use the new\n    'unsafe_hash_algo()' function, instead of switching between safe and\n    unsafe_ variants of individual functions:\n\n      csum-file.c: use unsafe_hash_algo()\n      t/helper/test-hash.c: use unsafe_hash_algo()\n\n  * The final patch drops the unsafe_ function variants following all\n    callers being converted to use the new pattern:\n\n      hash.h: drop unsafe_ function variants\n\nThanks in advance for your review!\n\n[1]: https://lore.kernel.org/git/cover.1730833506.git.me@ttaylorr.com/\n[2]: https://lore.kernel.org/git/20241107013915.GA961214@coredump.intra.peff.net/\n\nTaylor Blau (6):\n  csum-file: store the hash algorithm as a struct field\n  csum-file.c: extract algop from hashfile_checksum_valid()\n  hash.h: introduce `unsafe_hash_algo()`\n  csum-file.c: use unsafe_hash_algo()\n  t/helper/test-hash.c: use unsafe_hash_algo()\n  hash.h: drop unsafe_ function variants\n\n csum-file.c          | 33 ++++++++++++++++++---------------\n csum-file.h          |  1 +\n hash.h               | 20 +++++---------------\n object-file.c        | 41 ++++++++++++++++++++++++++---------------\n t/helper/test-hash.c | 17 +++++------------\n 5 files changed, 55 insertions(+), 57 deletions(-)\n\n\nbase-commit: d8c1fc78b57e02a140b5c363caaa14c2dc2bb274\n-- \n2.47.0.237.gc601277f4c4\n"},{"id":"507774","messageId":"380133a11421fe4ce8428d11be479d78923bc5cc.1732130001.git.me@ttaylorr.com","threadId":"62522","inReplyTo":"cover.1732130001.git.me@ttaylorr.com","subject":"[PATCH 1/6] csum-file: store the hash algorithm as a struct field","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-11-20T19:13:44Z","receivedAt":"2024-11-20T19:13:47Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Throughout the hashfile API, we rely on a reference to 'the_hash_algo',\nand call its _usnafe function variants directly.\n\nPrepare for a future change where we may use a different 'git_hash_algo'\npointer (instead of just relying on 'the_hash_algo' throughout) by\nmaking the 'git_hash_algo' pointer a member of the 'hashfile' structure\nitself.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n csum-file.c | 20 +++++++++++---------\n csum-file.h |  1 +\n 2 files changed, 12 insertions(+), 9 deletions(-)\n\ndiff --git a/csum-file.c b/csum-file.c\nindex c203ebf11b3..62f4965f094 100644\n--- a/csum-file.c\n+++ b/csum-file.c\n@@ -50,7 +50,7 @@ void hashflush(struct hashfile *f)\n \n \tif (offset) {\n \t\tif (!f->skip_hash)\n-\t\t\tthe_hash_algo->unsafe_update_fn(&f->ctx, f->buffer, offset);\n+\t\t\tf->algop->unsafe_update_fn(&f->ctx, f->buffer, offset);\n \t\tflush(f, f->buffer, offset);\n \t\tf->offset = 0;\n \t}\n@@ -71,14 +71,14 @@ int finalize_hashfile(struct hashfile *f, unsigned char *result,\n \thashflush(f);\n \n \tif (f->skip_hash)\n-\t\thashclr(f->buffer, the_repository->hash_algo);\n+\t\thashclr(f->buffer, f->algop);\n \telse\n-\t\tthe_hash_algo->unsafe_final_fn(f->buffer, &f->ctx);\n+\t\tf->algop->unsafe_final_fn(f->buffer, &f->ctx);\n \n \tif (result)\n-\t\thashcpy(result, f->buffer, the_repository->hash_algo);\n+\t\thashcpy(result, f->buffer, f->algop);\n \tif (flags & CSUM_HASH_IN_STREAM)\n-\t\tflush(f, f->buffer, the_hash_algo->rawsz);\n+\t\tflush(f, f->buffer, f->algop->rawsz);\n \tif (flags & CSUM_FSYNC)\n \t\tfsync_component_or_die(component, f->fd, f->name);\n \tif (flags & CSUM_CLOSE) {\n@@ -128,7 +128,7 @@ void hashwrite(struct hashfile *f, const void *buf, unsigned int count)\n \t\t\t * f->offset is necessarily zero.\n \t\t\t */\n \t\t\tif (!f->skip_hash)\n-\t\t\t\tthe_hash_algo->unsafe_update_fn(&f->ctx, buf, nr);\n+\t\t\t\tf->algop->unsafe_update_fn(&f->ctx, buf, nr);\n \t\t\tflush(f, buf, nr);\n \t\t} else {\n \t\t\t/*\n@@ -174,7 +174,9 @@ static struct hashfile *hashfd_internal(int fd, const char *name,\n \tf->name = name;\n \tf->do_crc = 0;\n \tf->skip_hash = 0;\n-\tthe_hash_algo->unsafe_init_fn(&f->ctx);\n+\n+\tf->algop = the_hash_algo;\n+\tf->algop->unsafe_init_fn(&f->ctx);\n \n \tf->buffer_len = buffer_len;\n \tf->buffer = xmalloc(buffer_len);\n@@ -208,7 +210,7 @@ void hashfile_checkpoint(struct hashfile *f, struct hashfile_checkpoint *checkpo\n {\n \thashflush(f);\n \tcheckpoint->offset = f->total;\n-\tthe_hash_algo->unsafe_clone_fn(&checkpoint->ctx, &f->ctx);\n+\tf->algop->unsafe_clone_fn(&checkpoint->ctx, &f->ctx);\n }\n \n int hashfile_truncate(struct hashfile *f, struct hashfile_checkpoint *checkpoint)\n@@ -219,7 +221,7 @@ int hashfile_truncate(struct hashfile *f, struct hashfile_checkpoint *checkpoint\n \t    lseek(f->fd, offset, SEEK_SET) != offset)\n \t\treturn -1;\n \tf->total = offset;\n-\tthe_hash_algo->unsafe_clone_fn(&f->ctx, &checkpoint->ctx);\n+\tf->algop->unsafe_clone_fn(&f->ctx, &checkpoint->ctx);\n \tf->offset = 0; /* hashflush() was called in checkpoint */\n \treturn 0;\n }\ndiff --git a/csum-file.h b/csum-file.h\nindex 7c73da0a40a..2b45f4673a2 100644\n--- a/csum-file.h\n+++ b/csum-file.h\n@@ -20,6 +20,7 @@ struct hashfile {\n \tsize_t buffer_len;\n \tunsigned char *buffer;\n \tunsigned char *check_buffer;\n+\tconst struct git_hash_algo *algop;\n \n \t/**\n \t * If non-zero, skip_hash indicates that we should\n-- \n2.47.0.237.gc601277f4c4\n\n"},{"id":"507775","messageId":"e5076f003bf5a53fc63bb673213988328c50c98e.1732130001.git.me@ttaylorr.com","threadId":"62522","inReplyTo":"cover.1732130001.git.me@ttaylorr.com","subject":"[PATCH 2/6] csum-file.c: extract algop from hashfile_checksum_valid()","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-11-20T19:13:47Z","receivedAt":"2024-11-20T19:13:50Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Perform a similar transformation as in the previous commit, but focused\ninstead on hashfile_checksum_valid(). This function does not work with a\nhashfile structure itself, and instead validates the raw contents of a\nfile written using the hashfile API.\n\nWe'll want to be prepared for a similar change to this function in the\nfuture, so prepare ourselves for that by extracting 'the_hash_algo' into\nits own field for use within this function.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n csum-file.c | 13 +++++++------\n 1 file changed, 7 insertions(+), 6 deletions(-)\n\ndiff --git a/csum-file.c b/csum-file.c\nindex 62f4965f094..107bc4c96f9 100644\n--- a/csum-file.c\n+++ b/csum-file.c\n@@ -242,14 +242,15 @@ int hashfile_checksum_valid(const unsigned char *data, size_t total_len)\n {\n \tunsigned char got[GIT_MAX_RAWSZ];\n \tgit_hash_ctx ctx;\n-\tsize_t data_len = total_len - the_hash_algo->rawsz;\n+\tconst struct git_hash_algo *algop = the_hash_algo;\n+\tsize_t data_len = total_len - algop->rawsz;\n \n-\tif (total_len < the_hash_algo->rawsz)\n+\tif (total_len < algop->rawsz)\n \t\treturn 0; /* say \"too short\"? */\n \n-\tthe_hash_algo->unsafe_init_fn(&ctx);\n-\tthe_hash_algo->unsafe_update_fn(&ctx, data, data_len);\n-\tthe_hash_algo->unsafe_final_fn(got, &ctx);\n+\talgop->unsafe_init_fn(&ctx);\n+\talgop->unsafe_update_fn(&ctx, data, data_len);\n+\talgop->unsafe_final_fn(got, &ctx);\n \n-\treturn hasheq(got, data + data_len, the_repository->hash_algo);\n+\treturn hasheq(got, data + data_len, algop);\n }\n-- \n2.47.0.237.gc601277f4c4\n\n"},{"id":"507776","messageId":"17f92dba34bee235177c8100daab49068fe37254.1732130001.git.me@ttaylorr.com","threadId":"62522","inReplyTo":"cover.1732130001.git.me@ttaylorr.com","subject":"[PATCH 3/6] hash.h: introduce `unsafe_hash_algo()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-11-20T19:13:50Z","receivedAt":"2024-11-20T19:13:53Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In 253ed9ecff (hash.h: scaffolding for _unsafe hashing variants,\n2024-09-26), we introduced \"unsafe\" variants of the SHA-1 hashing\nfunctions by introducing new functions like \"unsafe_init_fn()\" and so\non.\n\nThis approach has a major shortcoming that callers must remember to\nconsistently use one variant or the other. Failing to consistently use\n(or not use) the unsafe variants can lead to crashes at best, or subtle\nmemory corruption issues at worst.\n\nIn the hashfile API, this isn't difficult to achieve, but verifying that\nall callers consistently use the unsafe variants is somewhat of a chore\ngiven how spread out all of the callers are. In the sha1 and sha1-unsafe\ntest helpers, all of the calls to various hash functions are guarded by\nan \"if (unsafe)\" conditional, which is repetitive and cumbersome.\n\nAddress these issues by introducing a new pattern whereby one\n'git_hash_algo' can return a pointer to another 'git_hash_algo' that\nrepresents the unsafe version of itself. So instead of having something\nlike:\n\n    if (unsafe)\n      the_hash_algo->init_fn(...);\n      the_hash_algo->update_fn(...);\n      the_hash_algo->final_fn(...);\n    else\n      the_hash_algo->unsafe_init_fn(...);\n      the_hash_algo->unsafe_update_fn(...);\n      the_hash_algo->unsafe_final_fn(...);\n\nwe can instead write:\n\n    struct git_hash_algo *algop = the_hash_algo;\n    if (unsafe)\n      algop = unsafe_hash_algo(algop);\n\n    the_hash_algo->init_fn(...);\n    the_hash_algo->update_fn(...);\n    the_hash_algo->final_fn(...);\n\nThis removes the existing shortcoming by no longer forcing the caller to\n\"remember\" which variant of the hash functions it wants to call, only to\nhold onto a 'struct git_hash_algo' pointer that is initialized once.\n\nSimilarly, while there currently is still a way to \"mix\" safe and unsafe\nfunctions, this too will go away after subsequent commits remove all\ndirect calls to the unsafe_ variants.\n\nSuggested-by: Jeff King <peff@peff.net>\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n hash.h        |  5 +++++\n object-file.c | 26 ++++++++++++++++++++++++++\n 2 files changed, 31 insertions(+)\n\ndiff --git a/hash.h b/hash.h\nindex 756166ce5e8..23cf6876e50 100644\n--- a/hash.h\n+++ b/hash.h\n@@ -305,6 +305,9 @@ struct git_hash_algo {\n \n \t/* The all-zeros OID. */\n \tconst struct object_id *null_oid;\n+\n+\t/* The unsafe variant of this hash function, if one exists. */\n+\tconst struct git_hash_algo *unsafe;\n };\n extern const struct git_hash_algo hash_algos[GIT_HASH_NALGOS];\n \n@@ -323,6 +326,8 @@ static inline int hash_algo_by_ptr(const struct git_hash_algo *p)\n \treturn p - hash_algos;\n }\n \n+const struct git_hash_algo *unsafe_hash_algo(const struct git_hash_algo *algop);\n+\n const struct object_id *null_oid(void);\n \n static inline int hashcmp(const unsigned char *sha1, const unsigned char *sha2, const struct git_hash_algo *algop)\ndiff --git a/object-file.c b/object-file.c\nindex b1a3463852c..fddcdbe9ba6 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -204,6 +204,22 @@ static void git_hash_unknown_final_oid(struct object_id *oid UNUSED,\n \tBUG(\"trying to finalize unknown hash\");\n }\n \n+static const struct git_hash_algo sha1_unsafe_algo = {\n+\t.name = \"sha1\",\n+\t.format_id = GIT_SHA1_FORMAT_ID,\n+\t.rawsz = GIT_SHA1_RAWSZ,\n+\t.hexsz = GIT_SHA1_HEXSZ,\n+\t.blksz = GIT_SHA1_BLKSZ,\n+\t.init_fn = git_hash_sha1_init_unsafe,\n+\t.clone_fn = git_hash_sha1_clone_unsafe,\n+\t.update_fn = git_hash_sha1_update_unsafe,\n+\t.final_fn = git_hash_sha1_final_unsafe,\n+\t.final_oid_fn = git_hash_sha1_final_oid_unsafe,\n+\t.empty_tree = &empty_tree_oid,\n+\t.empty_blob = &empty_blob_oid,\n+\t.null_oid = &null_oid_sha1,\n+};\n+\n const struct git_hash_algo hash_algos[GIT_HASH_NALGOS] = {\n \t{\n \t\t.name = NULL,\n@@ -241,6 +257,7 @@ const struct git_hash_algo hash_algos[GIT_HASH_NALGOS] = {\n \t\t.unsafe_update_fn = git_hash_sha1_update_unsafe,\n \t\t.unsafe_final_fn = git_hash_sha1_final_unsafe,\n \t\t.unsafe_final_oid_fn = git_hash_sha1_final_oid_unsafe,\n+\t\t.unsafe = &sha1_unsafe_algo,\n \t\t.empty_tree = &empty_tree_oid,\n \t\t.empty_blob = &empty_blob_oid,\n \t\t.null_oid = &null_oid_sha1,\n@@ -307,6 +324,15 @@ int hash_algo_by_length(int len)\n \treturn GIT_HASH_UNKNOWN;\n }\n \n+const struct git_hash_algo *unsafe_hash_algo(const struct git_hash_algo *algop)\n+{\n+\t/* If we have a faster \"unsafe\" implementation, use that. */\n+\tif (algop->unsafe)\n+\t\treturn algop->unsafe;\n+\t/* Otherwise use the default one. */\n+\treturn algop;\n+}\n+\n /*\n  * This is meant to hold a *small* number of objects that you would\n  * want repo_read_object_file() to be able to return, but yet you do not want\n-- \n2.47.0.237.gc601277f4c4\n\n"},{"id":"507777","messageId":"0d8e12599e2e20ee9b5d1065856a6ddf7aa5f14a.1732130001.git.me@ttaylorr.com","threadId":"62522","inReplyTo":"cover.1732130001.git.me@ttaylorr.com","subject":"[PATCH 4/6] csum-file.c: use unsafe_hash_algo()","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-11-20T19:13:53Z","receivedAt":"2024-11-20T19:13:56Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Instead of calling the unsafe_ hash function variants directly, make use\nof the shared 'algop' pointer by initializing it to:\n\n    f->algop = unsafe_hash_algo(the_hash_algo);\n\n, thus making all calls use the unsafe variants directly.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n csum-file.c | 22 +++++++++++-----------\n 1 file changed, 11 insertions(+), 11 deletions(-)\n\ndiff --git a/csum-file.c b/csum-file.c\nindex 107bc4c96f9..fd51f5e432e 100644\n--- a/csum-file.c\n+++ b/csum-file.c\n@@ -50,7 +50,7 @@ void hashflush(struct hashfile *f)\n \n \tif (offset) {\n \t\tif (!f->skip_hash)\n-\t\t\tf->algop->unsafe_update_fn(&f->ctx, f->buffer, offset);\n+\t\t\tf->algop->update_fn(&f->ctx, f->buffer, offset);\n \t\tflush(f, f->buffer, offset);\n \t\tf->offset = 0;\n \t}\n@@ -73,7 +73,7 @@ int finalize_hashfile(struct hashfile *f, unsigned char *result,\n \tif (f->skip_hash)\n \t\thashclr(f->buffer, f->algop);\n \telse\n-\t\tf->algop->unsafe_final_fn(f->buffer, &f->ctx);\n+\t\tf->algop->final_fn(f->buffer, &f->ctx);\n \n \tif (result)\n \t\thashcpy(result, f->buffer, f->algop);\n@@ -128,7 +128,7 @@ void hashwrite(struct hashfile *f, const void *buf, unsigned int count)\n \t\t\t * f->offset is necessarily zero.\n \t\t\t */\n \t\t\tif (!f->skip_hash)\n-\t\t\t\tf->algop->unsafe_update_fn(&f->ctx, buf, nr);\n+\t\t\t\tf->algop->update_fn(&f->ctx, buf, nr);\n \t\t\tflush(f, buf, nr);\n \t\t} else {\n \t\t\t/*\n@@ -175,8 +175,8 @@ static struct hashfile *hashfd_internal(int fd, const char *name,\n \tf->do_crc = 0;\n \tf->skip_hash = 0;\n \n-\tf->algop = the_hash_algo;\n-\tf->algop->unsafe_init_fn(&f->ctx);\n+\tf->algop = unsafe_hash_algo(the_hash_algo);\n+\tf->algop->init_fn(&f->ctx);\n \n \tf->buffer_len = buffer_len;\n \tf->buffer = xmalloc(buffer_len);\n@@ -210,7 +210,7 @@ void hashfile_checkpoint(struct hashfile *f, struct hashfile_checkpoint *checkpo\n {\n \thashflush(f);\n \tcheckpoint->offset = f->total;\n-\tf->algop->unsafe_clone_fn(&checkpoint->ctx, &f->ctx);\n+\tf->algop->clone_fn(&checkpoint->ctx, &f->ctx);\n }\n \n int hashfile_truncate(struct hashfile *f, struct hashfile_checkpoint *checkpoint)\n@@ -221,7 +221,7 @@ int hashfile_truncate(struct hashfile *f, struct hashfile_checkpoint *checkpoint\n \t    lseek(f->fd, offset, SEEK_SET) != offset)\n \t\treturn -1;\n \tf->total = offset;\n-\tf->algop->unsafe_clone_fn(&f->ctx, &checkpoint->ctx);\n+\tf->algop->clone_fn(&f->ctx, &checkpoint->ctx);\n \tf->offset = 0; /* hashflush() was called in checkpoint */\n \treturn 0;\n }\n@@ -242,15 +242,15 @@ int hashfile_checksum_valid(const unsigned char *data, size_t total_len)\n {\n \tunsigned char got[GIT_MAX_RAWSZ];\n \tgit_hash_ctx ctx;\n-\tconst struct git_hash_algo *algop = the_hash_algo;\n+\tconst struct git_hash_algo *algop = unsafe_hash_algo(the_hash_algo);\n \tsize_t data_len = total_len - algop->rawsz;\n \n \tif (total_len < algop->rawsz)\n \t\treturn 0; /* say \"too short\"? */\n \n-\talgop->unsafe_init_fn(&ctx);\n-\talgop->unsafe_update_fn(&ctx, data, data_len);\n-\talgop->unsafe_final_fn(got, &ctx);\n+\talgop->init_fn(&ctx);\n+\talgop->update_fn(&ctx, data, data_len);\n+\talgop->final_fn(got, &ctx);\n \n \treturn hasheq(got, data + data_len, algop);\n }\n-- \n2.47.0.237.gc601277f4c4\n\n"},{"id":"507778","messageId":"a49a41703e22f0c5d57fb59f6a00d95bc16bd245.1732130001.git.me@ttaylorr.com","threadId":"62522","inReplyTo":"cover.1732130001.git.me@ttaylorr.com","subject":"[PATCH 5/6] t/helper/test-hash.c: use unsafe_hash_algo()","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-11-20T19:13:56Z","receivedAt":"2024-11-20T19:13:59Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Remove a series of conditionals within the shared cmd_hash_impl() helper\nthat powers the 'sha1' and 'sha1-unsafe' helpers.\n\nInstead, replace them with a single conditional that transforms the\nspecified hash algorithm into its unsafe variant. Then all subsequent\ncalls can directly use whatever function it wants to call without having\nto decide between the safe and unsafe variants.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n t/helper/test-hash.c | 17 +++++------------\n 1 file changed, 5 insertions(+), 12 deletions(-)\n\ndiff --git a/t/helper/test-hash.c b/t/helper/test-hash.c\nindex d0ee668df95..aa82638c621 100644\n--- a/t/helper/test-hash.c\n+++ b/t/helper/test-hash.c\n@@ -9,6 +9,8 @@ int cmd_hash_impl(int ac, const char **av, int algo, int unsafe)\n \tint binary = 0;\n \tchar *buffer;\n \tconst struct git_hash_algo *algop = &hash_algos[algo];\n+\tif (unsafe)\n+\t\talgop = unsafe_hash_algo(algop);\n \n \tif (ac == 2) {\n \t\tif (!strcmp(av[1], \"-b\"))\n@@ -27,10 +29,7 @@ int cmd_hash_impl(int ac, const char **av, int algo, int unsafe)\n \t\t\tdie(\"OOPS\");\n \t}\n \n-\tif (unsafe)\n-\t\talgop->unsafe_init_fn(&ctx);\n-\telse\n-\t\talgop->init_fn(&ctx);\n+\talgop->init_fn(&ctx);\n \n \twhile (1) {\n \t\tssize_t sz, this_sz;\n@@ -49,15 +48,9 @@ int cmd_hash_impl(int ac, const char **av, int algo, int unsafe)\n \t\t}\n \t\tif (this_sz == 0)\n \t\t\tbreak;\n-\t\tif (unsafe)\n-\t\t\talgop->unsafe_update_fn(&ctx, buffer, this_sz);\n-\t\telse\n-\t\t\talgop->update_fn(&ctx, buffer, this_sz);\n+\t\talgop->update_fn(&ctx, buffer, this_sz);\n \t}\n-\tif (unsafe)\n-\t\talgop->unsafe_final_fn(hash, &ctx);\n-\telse\n-\t\talgop->final_fn(hash, &ctx);\n+\talgop->final_fn(hash, &ctx);\n \n \tif (binary)\n \t\tfwrite(hash, 1, algop->rawsz, stdout);\n-- \n2.47.0.237.gc601277f4c4\n\n"},{"id":"507779","messageId":"4081ad08549895a1b05c45fbc6f229683fec3963.1732130001.git.me@ttaylorr.com","threadId":"62522","inReplyTo":"cover.1732130001.git.me@ttaylorr.com","subject":"[PATCH 6/6] hash.h: drop unsafe_ function variants","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2024-11-20T19:13:59Z","receivedAt":"2024-11-20T19:14:02Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Now that all callers have been converted from:\n\n    the_hash_algo->unsafe_init_fn();\n\nto\n\n    unsafe_hash_algo(the_hash_algo)->unsafe_init_fn();\n\nand similar, we can remove the scaffolding for the unsafe_ function\nvariants and force callers to use the new unsafe_hash_algo() mechanic\ninstead.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n hash.h        | 15 ---------------\n object-file.c | 15 ---------------\n 2 files changed, 30 deletions(-)\n\ndiff --git a/hash.h b/hash.h\nindex 23cf6876e50..6dcbf6ab835 100644\n--- a/hash.h\n+++ b/hash.h\n@@ -282,21 +282,6 @@ struct git_hash_algo {\n \t/* The hash finalization function for object IDs. */\n \tgit_hash_final_oid_fn final_oid_fn;\n \n-\t/* The non-cryptographic hash initialization function. */\n-\tgit_hash_init_fn unsafe_init_fn;\n-\n-\t/* The non-cryptographic hash context cloning function. */\n-\tgit_hash_clone_fn unsafe_clone_fn;\n-\n-\t/* The non-cryptographic hash update function. */\n-\tgit_hash_update_fn unsafe_update_fn;\n-\n-\t/* The non-cryptographic hash finalization function. */\n-\tgit_hash_final_fn unsafe_final_fn;\n-\n-\t/* The non-cryptographic hash finalization function. */\n-\tgit_hash_final_oid_fn unsafe_final_oid_fn;\n-\n \t/* The OID of the empty tree. */\n \tconst struct object_id *empty_tree;\n \ndiff --git a/object-file.c b/object-file.c\nindex fddcdbe9ba6..1040a5408f2 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -232,11 +232,6 @@ const struct git_hash_algo hash_algos[GIT_HASH_NALGOS] = {\n \t\t.update_fn = git_hash_unknown_update,\n \t\t.final_fn = git_hash_unknown_final,\n \t\t.final_oid_fn = git_hash_unknown_final_oid,\n-\t\t.unsafe_init_fn = git_hash_unknown_init,\n-\t\t.unsafe_clone_fn = git_hash_unknown_clone,\n-\t\t.unsafe_update_fn = git_hash_unknown_update,\n-\t\t.unsafe_final_fn = git_hash_unknown_final,\n-\t\t.unsafe_final_oid_fn = git_hash_unknown_final_oid,\n \t\t.empty_tree = NULL,\n \t\t.empty_blob = NULL,\n \t\t.null_oid = NULL,\n@@ -252,11 +247,6 @@ const struct git_hash_algo hash_algos[GIT_HASH_NALGOS] = {\n \t\t.update_fn = git_hash_sha1_update,\n \t\t.final_fn = git_hash_sha1_final,\n \t\t.final_oid_fn = git_hash_sha1_final_oid,\n-\t\t.unsafe_init_fn = git_hash_sha1_init_unsafe,\n-\t\t.unsafe_clone_fn = git_hash_sha1_clone_unsafe,\n-\t\t.unsafe_update_fn = git_hash_sha1_update_unsafe,\n-\t\t.unsafe_final_fn = git_hash_sha1_final_unsafe,\n-\t\t.unsafe_final_oid_fn = git_hash_sha1_final_oid_unsafe,\n \t\t.unsafe = &sha1_unsafe_algo,\n \t\t.empty_tree = &empty_tree_oid,\n \t\t.empty_blob = &empty_blob_oid,\n@@ -273,11 +263,6 @@ const struct git_hash_algo hash_algos[GIT_HASH_NALGOS] = {\n \t\t.update_fn = git_hash_sha256_update,\n \t\t.final_fn = git_hash_sha256_final,\n \t\t.final_oid_fn = git_hash_sha256_final_oid,\n-\t\t.unsafe_init_fn = git_hash_sha256_init,\n-\t\t.unsafe_clone_fn = git_hash_sha256_clone,\n-\t\t.unsafe_update_fn = git_hash_sha256_update,\n-\t\t.unsafe_final_fn = git_hash_sha256_final,\n-\t\t.unsafe_final_oid_fn = git_hash_sha256_final_oid,\n \t\t.empty_tree = &empty_tree_oid_sha256,\n \t\t.empty_blob = &empty_blob_oid_sha256,\n \t\t.null_oid = &null_oid_sha256,\n-- \n2.47.0.237.gc601277f4c4\n"},{"id":"507812","messageId":"20241121091832.GC602681@coredump.intra.peff.net","threadId":"62522","inReplyTo":"380133a11421fe4ce8428d11be479d78923bc5cc.1732130001.git.me@ttaylorr.com","subject":"Re: [PATCH 1/6] csum-file: store the hash algorithm as a struct field","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2024-11-21T09:18:32Z","receivedAt":"2024-11-21T09:18:33Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Nov 20, 2024 at 02:13:44PM -0500, Taylor Blau wrote:\n\n> Throughout the hashfile API, we rely on a reference to 'the_hash_algo',\n> and call its _usnafe function variants directly.\n\ns/usnafe/unsafe/\n\n> Prepare for a future change where we may use a different 'git_hash_algo'\n> pointer (instead of just relying on 'the_hash_algo' throughout) by\n> making the 'git_hash_algo' pointer a member of the 'hashfile' structure\n> itself.\n\nThis makes sense in the context of your series, which will eventually\njust hold the unsafe algo struct.\n\nBut I also think it is a good step in general, because it means that\nrather than referring to the_hash_algo all over the place, we only do it\nwhen initializing the hashfile struct. In the long run, that should take\na repository struct or a git_hash_algo pointer directly, and we'd want\nto pass it just once to hashfd(), etc.\n\n-Peff\n"},{"id":"507813","messageId":"20241121093731.GD602681@coredump.intra.peff.net","threadId":"62522","inReplyTo":"17f92dba34bee235177c8100daab49068fe37254.1732130001.git.me@ttaylorr.com","subject":"Re: [PATCH 3/6] hash.h: introduce `unsafe_hash_algo()`","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2024-11-21T09:37:31Z","receivedAt":"2024-11-21T09:37:32Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Nov 20, 2024 at 02:13:50PM -0500, Taylor Blau wrote:\n\n> +static const struct git_hash_algo sha1_unsafe_algo = {\n> +\t.name = \"sha1\",\n> +\t.format_id = GIT_SHA1_FORMAT_ID,\n> +\t.rawsz = GIT_SHA1_RAWSZ,\n> +\t.hexsz = GIT_SHA1_HEXSZ,\n> +\t.blksz = GIT_SHA1_BLKSZ,\n> +\t.init_fn = git_hash_sha1_init_unsafe,\n> +\t.clone_fn = git_hash_sha1_clone_unsafe,\n> +\t.update_fn = git_hash_sha1_update_unsafe,\n> +\t.final_fn = git_hash_sha1_final_unsafe,\n> +\t.final_oid_fn = git_hash_sha1_final_oid_unsafe,\n> +\t.empty_tree = &empty_tree_oid,\n> +\t.empty_blob = &empty_blob_oid,\n> +\t.null_oid = &null_oid_sha1,\n> +};\n\nAll of the non-function fields here naturally must match the ones in the\nparent algo struct, or chaos ensues. That's a little fragile, but it's\nnot like we're adding new algorithm variants a lot. The biggest risk, I\nguess, would be adding a new field to git_hash_algo which defaults to\nzero-initialization here. But again, there are only three total and they\nare defined near each other here, so I don't think it's a big risk\noverall.\n\nI think this struct is a potential landmine for hash_algo_by_ptr():\n\n  static inline int hash_algo_by_ptr(const struct git_hash_algo *p)\n  {\n          return p - hash_algos;\n  }\n\nIt's undefined behavior to pass in sha1_unsafe_algo to this function\n(but the compiler would not complain since the types are the same). I\ndon't find it incredibly likely that somebody would want to do that on\nan unsafe variant, but I'm not thrilled about leaving that wart for\nsomebody to find.\n\nIf we don't care about the speed of this function, then an\nimplementation like:\n\n  for (i = 0; i < GIT_HASH_NALGOS; i++) {\n\tif (p == &hash_algos[i] || p == hash_algos[i]->unsafe)\n\t\treturn i;\n  }\n  return GIT_HASH_UNKNOWN;\n\nwould work. I'm not sure if that would be measurable. I was surprised at\nthe number of places that hash_algo_by_ptr() is called. Many low-level\noid functions need it because we store the integer id there rather than\na direct pointer (so oidread(), oidclr(), oid_object_info_extended(),\nand so on). But I'd also expect the loop above to be pretty fast. So I\ndunno.\n\n>  const struct git_hash_algo hash_algos[GIT_HASH_NALGOS] = {\n>  \t{\n>  \t\t.name = NULL,\n> @@ -241,6 +257,7 @@ const struct git_hash_algo hash_algos[GIT_HASH_NALGOS] = {\n>  \t\t.unsafe_update_fn = git_hash_sha1_update_unsafe,\n>  \t\t.unsafe_final_fn = git_hash_sha1_final_unsafe,\n>  \t\t.unsafe_final_oid_fn = git_hash_sha1_final_oid_unsafe,\n> +\t\t.unsafe = &sha1_unsafe_algo,\n>  \t\t.empty_tree = &empty_tree_oid,\n>  \t\t.empty_blob = &empty_blob_oid,\n>  \t\t.null_oid = &null_oid_sha1,\n\nOK, and we can leave the sha256 pointer zero-initialized, since the\nfunction handles that at runtime. Good.\n\n-Peff\n"},{"id":"507814","messageId":"20241121094144.GE602681@coredump.intra.peff.net","threadId":"62522","inReplyTo":"4081ad08549895a1b05c45fbc6f229683fec3963.1732130001.git.me@ttaylorr.com","subject":"Re: [PATCH 6/6] hash.h: drop unsafe_ function variants","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2024-11-21T09:41:44Z","receivedAt":"2024-11-21T09:41:45Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Nov 20, 2024 at 02:13:59PM -0500, Taylor Blau wrote:\n\n> Now that all callers have been converted from:\n> \n>     the_hash_algo->unsafe_init_fn();\n> \n> to\n> \n>     unsafe_hash_algo(the_hash_algo)->unsafe_init_fn();\n> \n> and similar, we can remove the scaffolding for the unsafe_ function\n> variants and force callers to use the new unsafe_hash_algo() mechanic\n> instead.\n\nNice. Especially for sha256, which does not even need to care about this\nunsafe thing at all (so in 2099, when we finally remove sha1 support,\nthis whole system can go away!).\n\nI think this also opens up alternatives for how we conditionally compile\nthings. E.g., if you have no *_SHA1_UNSAFE macro defined, we could avoid\ndefining sha1_unsafe_algo at all, and just leave it as NULL. I can't\nthink of a significant enough advantage to merit the work in converting\nto that, though, so it's probably not worth doing unless we later decide\nit would make things simpler for some reason.\n\n-Peff\n"},{"id":"507869","messageId":"Zz_SwuEw-KYM9xJl@tapette.crustytoothpaste.net","threadId":"62522","inReplyTo":"20241121093731.GD602681@coredump.intra.peff.net","subject":"Re: [PATCH 3/6] hash.h: introduce `unsafe_hash_algo()`","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2024-11-22T00:39:30Z","receivedAt":"2024-11-22T00:39:38Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On 2024-11-21 at 09:37:31, Jeff King wrote:\n> On Wed, Nov 20, 2024 at 02:13:50PM -0500, Taylor Blau wrote:\n> \n> > +static const struct git_hash_algo sha1_unsafe_algo = {\n> > +\t.name = \"sha1\",\n> > +\t.format_id = GIT_SHA1_FORMAT_ID,\n> > +\t.rawsz = GIT_SHA1_RAWSZ,\n> > +\t.hexsz = GIT_SHA1_HEXSZ,\n> > +\t.blksz = GIT_SHA1_BLKSZ,\n> > +\t.init_fn = git_hash_sha1_init_unsafe,\n> > +\t.clone_fn = git_hash_sha1_clone_unsafe,\n> > +\t.update_fn = git_hash_sha1_update_unsafe,\n> > +\t.final_fn = git_hash_sha1_final_unsafe,\n> > +\t.final_oid_fn = git_hash_sha1_final_oid_unsafe,\n> > +\t.empty_tree = &empty_tree_oid,\n> > +\t.empty_blob = &empty_blob_oid,\n> > +\t.null_oid = &null_oid_sha1,\n> > +};\n> \n> All of the non-function fields here naturally must match the ones in the\n> parent algo struct, or chaos ensues. That's a little fragile, but it's\n> not like we're adding new algorithm variants a lot. The biggest risk, I\n> guess, would be adding a new field to git_hash_algo which defaults to\n> zero-initialization here. But again, there are only three total and they\n> are defined near each other here, so I don't think it's a big risk\n> overall.\n> \n> I think this struct is a potential landmine for hash_algo_by_ptr():\n> \n>   static inline int hash_algo_by_ptr(const struct git_hash_algo *p)\n>   {\n>           return p - hash_algos;\n>   }\n> \n> It's undefined behavior to pass in sha1_unsafe_algo to this function\n> (but the compiler would not complain since the types are the same). I\n> don't find it incredibly likely that somebody would want to do that on\n> an unsafe variant, but I'm not thrilled about leaving that wart for\n> somebody to find.\n> \n> If we don't care about the speed of this function, then an\n> implementation like:\n> \n>   for (i = 0; i < GIT_HASH_NALGOS; i++) {\n> \tif (p == &hash_algos[i] || p == hash_algos[i]->unsafe)\n> \t\treturn i;\n>   }\n>   return GIT_HASH_UNKNOWN;\n> \n> would work. I'm not sure if that would be measurable. I was surprised at\n> the number of places that hash_algo_by_ptr() is called. Many low-level\n> oid functions need it because we store the integer id there rather than\n> a direct pointer (so oidread(), oidclr(), oid_object_info_extended(),\n> and so on). But I'd also expect the loop above to be pretty fast. So I\n> dunno.\n\nYeah, I'm also a little nervous about this change with hash_algo_by_ptr.\nI think we had discussed in the other thread about doing something like\nthis:\n\nstruct git_hash_algo_fns {\n\t/* The hash initialization function. */\n\tgit_hash_init_fn init_fn;\n\n\t/* The hash context cloning function. */\n\tgit_hash_clone_fn clone_fn;\n\n\t/* The hash update function. */\n\tgit_hash_update_fn update_fn;\n\n\t/* The hash finalization function. */\n\tgit_hash_final_fn final_fn;\n\n\t/* The hash finalization function for object IDs. */\n\tgit_hash_final_oid_fn final_oid_fn;\n};\n\nand then doing this:\n\nstruct git_hash_algo {\n\t[...]\n\n  struct git_hash_algo_fns fns;\n  struct git_hash_algo_fns unsafe_fns;\n\n  [...]\n};\n\nThat would mean that we'd have to deal with two pointers in your first\npatch, but I don't really think that's too terrible.  And, since this\napproach doesn't result in an extra struct git_hash_algo, it's\nimpossible to misuse hash_algo_by_ptr.\n\nI'm not a hard no on the current approach, but I think the above would\nprobably be a little safer and harder to misuse.  It would require extra\npatches to convert existing callers, though, but they aren't hugely\nnumerous, and the conversion could probably be mostly automated.\n-- \nbrian m. carlson (they/them or he/him)\nToronto, Ontario, CA\n"},{"id":"507886","messageId":"20241122082523.GB1179306@coredump.intra.peff.net","threadId":"62522","inReplyTo":"Zz_SwuEw-KYM9xJl@tapette.crustytoothpaste.net","subject":"Re: [PATCH 3/6] hash.h: introduce `unsafe_hash_algo()`","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2024-11-22T08:25:23Z","receivedAt":"2024-11-22T08:25:31Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Nov 22, 2024 at 12:39:30AM +0000, brian m. carlson wrote:\n\n> Yeah, I'm also a little nervous about this change with hash_algo_by_ptr.\n> I think we had discussed in the other thread about doing something like\n> this:\n> \n> struct git_hash_algo_fns {\n> [...]\n>\n> I'm not a hard no on the current approach, but I think the above would\n> probably be a little safer and harder to misuse.  It would require extra\n> patches to convert existing callers, though, but they aren't hugely\n> numerous, and the conversion could probably be mostly automated.\n\nYeah, I was worried about the fallout to callers. But it might be worth\nseeing how extensive it is.\n\n...\n\nOK, here's a quick and dirty conversion that applies on top of Taylor's\npatches 1 and 2 (and then patches 3 and on would need to adjust to the\nnew world order).\n\nThere are quite a lot of spots that needed to be adjusted, but the most\npart it was just s/->\\([a-z]\\)_fn/->fn.\\1/.  (I'll leave adjusting the\nunsafe_ variants as an exercise for the reader).\n\nThe naming convention was just what I made up, but one nice effect is\nthat it replaces \"_fn\" with \"fn.\", so the line lengths remain the same.\n:) It does mean there's a function called \"final\", which is a keyword in\nC++. I don't know if that matters to us or not (I feel like we were\ntrying to avoid those, but it appears as a function elsewhere, too).\n\nSo I dunno. It is a one-time pain, but I think the result harmonizes the\ntype system with the concept better.\n\n---\n builtin/fast-import.c      |  18 +++---\n builtin/index-pack.c       |  18 +++---\n builtin/patch-id.c         |  10 ++--\n builtin/receive-pack.c     |  22 +++----\n builtin/unpack-objects.c   |  12 ++--\n bulk-checkin.c             |  10 ++--\n csum-file.c                |  18 +++---\n diff.c                     |  28 ++++-----\n hash.h                     |  37 ++++--------\n http-push.c                |   6 +-\n http.c                     |   8 +--\n object-file.c              | 112 +++++++++++++++++++----------------\n pack-check.c               |   6 +-\n pack-write.c               |  20 +++----\n read-cache.c               |  20 +++----\n rerere.c                   |   8 +--\n t/helper/test-hash-speed.c |   6 +-\n t/helper/test-hash.c       |   6 +-\n t/unit-tests/t-hash.c      |   6 +-\n trace2/tr2_sid.c           |   6 +-\n 20 files changed, 189 insertions(+), 188 deletions(-)\n\ndiff --git a/builtin/fast-import.c b/builtin/fast-import.c\nindex 76d5c20f14..b7870d1f1e 100644\n--- a/builtin/fast-import.c\n+++ b/builtin/fast-import.c\n@@ -953,10 +953,10 @@ static int store_object(\n \n \thdrlen = format_object_header((char *)hdr, sizeof(hdr), type,\n \t\t\t\t      dat->len);\n-\tthe_hash_algo->init_fn(&c);\n-\tthe_hash_algo->update_fn(&c, hdr, hdrlen);\n-\tthe_hash_algo->update_fn(&c, dat->buf, dat->len);\n-\tthe_hash_algo->final_oid_fn(&oid, &c);\n+\tthe_hash_algo->fn.init(&c);\n+\tthe_hash_algo->fn.update(&c, hdr, hdrlen);\n+\tthe_hash_algo->fn.update(&c, dat->buf, dat->len);\n+\tthe_hash_algo->fn.final_oid(&oid, &c);\n \tif (oidout)\n \t\toidcpy(oidout, &oid);\n \n@@ -1101,14 +1101,14 @@ static void stream_blob(uintmax_t len, struct object_id *oidout, uintmax_t mark)\n \t\t|| (pack_size + PACK_SIZE_THRESHOLD + len) < pack_size)\n \t\tcycle_packfile();\n \n-\tthe_hash_algo->init_fn(&checkpoint.ctx);\n+\tthe_hash_algo->fn.init(&checkpoint.ctx);\n \thashfile_checkpoint(pack_file, &checkpoint);\n \toffset = checkpoint.offset;\n \n \thdrlen = format_object_header((char *)out_buf, out_sz, OBJ_BLOB, len);\n \n-\tthe_hash_algo->init_fn(&c);\n-\tthe_hash_algo->update_fn(&c, out_buf, hdrlen);\n+\tthe_hash_algo->fn.init(&c);\n+\tthe_hash_algo->fn.update(&c, out_buf, hdrlen);\n \n \tcrc32_begin(pack_file);\n \n@@ -1126,7 +1126,7 @@ static void stream_blob(uintmax_t len, struct object_id *oidout, uintmax_t mark)\n \t\t\tif (!n && feof(stdin))\n \t\t\t\tdie(\"EOF in data (%\" PRIuMAX \" bytes remaining)\", len);\n \n-\t\t\tthe_hash_algo->update_fn(&c, in_buf, n);\n+\t\t\tthe_hash_algo->fn.update(&c, in_buf, n);\n \t\t\ts.next_in = in_buf;\n \t\t\ts.avail_in = n;\n \t\t\tlen -= n;\n@@ -1152,7 +1152,7 @@ static void stream_blob(uintmax_t len, struct object_id *oidout, uintmax_t mark)\n \t\t}\n \t}\n \tgit_deflate_end(&s);\n-\tthe_hash_algo->final_oid_fn(&oid, &c);\n+\tthe_hash_algo->fn.final_oid(&oid, &c);\n \n \tif (oidout)\n \t\toidcpy(oidout, &oid);\ndiff --git a/builtin/index-pack.c b/builtin/index-pack.c\nindex 08b340552f..c5b5573edf 100644\n--- a/builtin/index-pack.c\n+++ b/builtin/index-pack.c\n@@ -298,7 +298,7 @@ static void flush(void)\n \tif (input_offset) {\n \t\tif (output_fd >= 0)\n \t\t\twrite_or_die(output_fd, input_buffer, input_offset);\n-\t\tthe_hash_algo->update_fn(&input_ctx, input_buffer, input_offset);\n+\t\tthe_hash_algo->fn.update(&input_ctx, input_buffer, input_offset);\n \t\tmemmove(input_buffer, input_buffer + input_offset, input_len);\n \t\tinput_offset = 0;\n \t}\n@@ -371,7 +371,7 @@ static const char *open_pack_file(const char *pack_name)\n \t\toutput_fd = -1;\n \t\tnothread_data.pack_fd = input_fd;\n \t}\n-\tthe_hash_algo->init_fn(&input_ctx);\n+\tthe_hash_algo->fn.init(&input_ctx);\n \treturn pack_name;\n }\n \n@@ -476,8 +476,8 @@ static void *unpack_entry_data(off_t offset, unsigned long size,\n \n \tif (!is_delta_type(type)) {\n \t\thdrlen = format_object_header(hdr, sizeof(hdr), type, size);\n-\t\tthe_hash_algo->init_fn(&c);\n-\t\tthe_hash_algo->update_fn(&c, hdr, hdrlen);\n+\t\tthe_hash_algo->fn.init(&c);\n+\t\tthe_hash_algo->fn.update(&c, hdr, hdrlen);\n \t} else\n \t\toid = NULL;\n \tif (type == OBJ_BLOB && size > big_file_threshold)\n@@ -497,7 +497,7 @@ static void *unpack_entry_data(off_t offset, unsigned long size,\n \t\tstatus = git_inflate(&stream, 0);\n \t\tuse(input_len - stream.avail_in);\n \t\tif (oid)\n-\t\t\tthe_hash_algo->update_fn(&c, last_out, stream.next_out - last_out);\n+\t\t\tthe_hash_algo->fn.update(&c, last_out, stream.next_out - last_out);\n \t\tif (buf == fixed_buf) {\n \t\t\tstream.next_out = buf;\n \t\t\tstream.avail_out = sizeof(fixed_buf);\n@@ -507,7 +507,7 @@ static void *unpack_entry_data(off_t offset, unsigned long size,\n \t\tbad_object(offset, _(\"inflate returned %d\"), status);\n \tgit_inflate_end(&stream);\n \tif (oid)\n-\t\tthe_hash_algo->final_oid_fn(oid, &c);\n+\t\tthe_hash_algo->fn.final_oid(oid, &c);\n \treturn buf == fixed_buf ? NULL : buf;\n }\n \n@@ -1256,9 +1256,9 @@ static void parse_pack_objects(unsigned char *hash)\n \n \t/* Check pack integrity */\n \tflush();\n-\tthe_hash_algo->init_fn(&tmp_ctx);\n-\tthe_hash_algo->clone_fn(&tmp_ctx, &input_ctx);\n-\tthe_hash_algo->final_fn(hash, &tmp_ctx);\n+\tthe_hash_algo->fn.init(&tmp_ctx);\n+\tthe_hash_algo->fn.clone(&tmp_ctx, &input_ctx);\n+\tthe_hash_algo->fn.final(hash, &tmp_ctx);\n \tif (!hasheq(fill(the_hash_algo->rawsz), hash, the_repository->hash_algo))\n \t\tdie(_(\"pack is corrupted (SHA1 mismatch)\"));\n \tuse(the_hash_algo->rawsz);\ndiff --git a/builtin/patch-id.c b/builtin/patch-id.c\nindex 93b398e391..3bf1367341 100644\n--- a/builtin/patch-id.c\n+++ b/builtin/patch-id.c\n@@ -70,7 +70,7 @@ static int get_one_patchid(struct object_id *next_oid, struct object_id *result,\n \tchar pre_oid_str[GIT_MAX_HEXSZ + 1], post_oid_str[GIT_MAX_HEXSZ + 1];\n \tgit_hash_ctx ctx;\n \n-\tthe_hash_algo->init_fn(&ctx);\n+\tthe_hash_algo->fn.init(&ctx);\n \toidclr(result, the_repository->hash_algo);\n \n \twhile (strbuf_getwholeline(line_buf, stdin, '\\n') != EOF) {\n@@ -83,7 +83,7 @@ static int get_one_patchid(struct object_id *next_oid, struct object_id *result,\n \t\t    !skip_prefix(line, \"From \", &p) &&\n \t\t    starts_with(line, \"\\\\ \") && 12 < strlen(line)) {\n \t\t\tif (verbatim)\n-\t\t\t\tthe_hash_algo->update_fn(&ctx, line, strlen(line));\n+\t\t\t\tthe_hash_algo->fn.update(&ctx, line, strlen(line));\n \t\t\tcontinue;\n \t\t}\n \n@@ -102,9 +102,9 @@ static int get_one_patchid(struct object_id *next_oid, struct object_id *result,\n \t\t\t    starts_with(line, \"Binary files\")) {\n \t\t\t\tdiff_is_binary = 1;\n \t\t\t\tbefore = 0;\n-\t\t\t\tthe_hash_algo->update_fn(&ctx, pre_oid_str,\n+\t\t\t\tthe_hash_algo->fn.update(&ctx, pre_oid_str,\n \t\t\t\t\t\t\t strlen(pre_oid_str));\n-\t\t\t\tthe_hash_algo->update_fn(&ctx, post_oid_str,\n+\t\t\t\tthe_hash_algo->fn.update(&ctx, post_oid_str,\n \t\t\t\t\t\t\t strlen(post_oid_str));\n \t\t\t\tif (stable)\n \t\t\t\t\tflush_one_hunk(result, &ctx);\n@@ -163,7 +163,7 @@ static int get_one_patchid(struct object_id *next_oid, struct object_id *result,\n \t\t/* Add line to hash algo (possibly removing whitespace) */\n \t\tlen = verbatim ? strlen(line) : remove_space(line);\n \t\tpatchlen += len;\n-\t\tthe_hash_algo->update_fn(&ctx, line, len);\n+\t\tthe_hash_algo->fn.update(&ctx, line, len);\n \t}\n \n \tif (!found_next)\ndiff --git a/builtin/receive-pack.c b/builtin/receive-pack.c\nindex ab5b20e39c..b7aa6cc556 100644\n--- a/builtin/receive-pack.c\n+++ b/builtin/receive-pack.c\n@@ -569,9 +569,9 @@ static void hmac_hash(unsigned char *out,\n \t/* RFC 2104 2. (1) */\n \tmemset(key, '\\0', GIT_MAX_BLKSZ);\n \tif (the_hash_algo->blksz < key_len) {\n-\t\tthe_hash_algo->init_fn(&ctx);\n-\t\tthe_hash_algo->update_fn(&ctx, key_in, key_len);\n-\t\tthe_hash_algo->final_fn(key, &ctx);\n+\t\tthe_hash_algo->fn.init(&ctx);\n+\t\tthe_hash_algo->fn.update(&ctx, key_in, key_len);\n+\t\tthe_hash_algo->fn.final(key, &ctx);\n \t} else {\n \t\tmemcpy(key, key_in, key_len);\n \t}\n@@ -583,16 +583,16 @@ static void hmac_hash(unsigned char *out,\n \t}\n \n \t/* RFC 2104 2. (3) & (4) */\n-\tthe_hash_algo->init_fn(&ctx);\n-\tthe_hash_algo->update_fn(&ctx, k_ipad, sizeof(k_ipad));\n-\tthe_hash_algo->update_fn(&ctx, text, text_len);\n-\tthe_hash_algo->final_fn(out, &ctx);\n+\tthe_hash_algo->fn.init(&ctx);\n+\tthe_hash_algo->fn.update(&ctx, k_ipad, sizeof(k_ipad));\n+\tthe_hash_algo->fn.update(&ctx, text, text_len);\n+\tthe_hash_algo->fn.final(out, &ctx);\n \n \t/* RFC 2104 2. (6) & (7) */\n-\tthe_hash_algo->init_fn(&ctx);\n-\tthe_hash_algo->update_fn(&ctx, k_opad, sizeof(k_opad));\n-\tthe_hash_algo->update_fn(&ctx, out, the_hash_algo->rawsz);\n-\tthe_hash_algo->final_fn(out, &ctx);\n+\tthe_hash_algo->fn.init(&ctx);\n+\tthe_hash_algo->fn.update(&ctx, k_opad, sizeof(k_opad));\n+\tthe_hash_algo->fn.update(&ctx, out, the_hash_algo->rawsz);\n+\tthe_hash_algo->fn.final(out, &ctx);\n }\n \n static char *prepare_push_cert_nonce(const char *path, timestamp_t stamp)\ndiff --git a/builtin/unpack-objects.c b/builtin/unpack-objects.c\nindex 02b8d02f63..c263343531 100644\n--- a/builtin/unpack-objects.c\n+++ b/builtin/unpack-objects.c\n@@ -67,7 +67,7 @@ static void *fill(int min)\n \tif (min > sizeof(buffer))\n \t\tdie(\"cannot fill %d bytes\", min);\n \tif (offset) {\n-\t\tthe_hash_algo->update_fn(&ctx, buffer, offset);\n+\t\tthe_hash_algo->fn.update(&ctx, buffer, offset);\n \t\tmemmove(buffer, buffer + offset, len);\n \t\toffset = 0;\n \t}\n@@ -667,12 +667,12 @@ int cmd_unpack_objects(int argc,\n \t\t/* We don't take any non-flag arguments now.. Maybe some day */\n \t\tusage(unpack_usage);\n \t}\n-\tthe_hash_algo->init_fn(&ctx);\n+\tthe_hash_algo->fn.init(&ctx);\n \tunpack_all();\n-\tthe_hash_algo->update_fn(&ctx, buffer, offset);\n-\tthe_hash_algo->init_fn(&tmp_ctx);\n-\tthe_hash_algo->clone_fn(&tmp_ctx, &ctx);\n-\tthe_hash_algo->final_oid_fn(&oid, &tmp_ctx);\n+\tthe_hash_algo->fn.update(&ctx, buffer, offset);\n+\tthe_hash_algo->fn.init(&tmp_ctx);\n+\tthe_hash_algo->fn.clone(&tmp_ctx, &ctx);\n+\tthe_hash_algo->fn.final_oid(&oid, &tmp_ctx);\n \tif (strict) {\n \t\twrite_rest();\n \t\tif (fsck_finish(&fsck_options))\ndiff --git a/bulk-checkin.c b/bulk-checkin.c\nindex 2753d5bbe4..7fc545703a 100644\n--- a/bulk-checkin.c\n+++ b/bulk-checkin.c\n@@ -193,7 +193,7 @@ static int stream_blob_to_pack(struct bulk_checkin_packfile *state,\n \t\t\t\tif (rsize < hsize)\n \t\t\t\t\thsize = rsize;\n \t\t\t\tif (hsize)\n-\t\t\t\t\tthe_hash_algo->update_fn(ctx, ibuf, hsize);\n+\t\t\t\t\tthe_hash_algo->fn.update(ctx, ibuf, hsize);\n \t\t\t\t*already_hashed_to = offset;\n \t\t\t}\n \t\t\ts.next_in = ibuf;\n@@ -269,9 +269,9 @@ static int deflate_blob_to_pack(struct bulk_checkin_packfile *state,\n \n \theader_len = format_object_header((char *)obuf, sizeof(obuf),\n \t\t\t\t\t  OBJ_BLOB, size);\n-\tthe_hash_algo->init_fn(&ctx);\n-\tthe_hash_algo->update_fn(&ctx, obuf, header_len);\n-\tthe_hash_algo->init_fn(&checkpoint.ctx);\n+\tthe_hash_algo->fn.init(&ctx);\n+\tthe_hash_algo->fn.update(&ctx, obuf, header_len);\n+\tthe_hash_algo->fn.init(&checkpoint.ctx);\n \n \t/* Note: idx is non-NULL when we are writing */\n \tif ((flags & HASH_WRITE_OBJECT) != 0)\n@@ -302,7 +302,7 @@ static int deflate_blob_to_pack(struct bulk_checkin_packfile *state,\n \t\tif (lseek(fd, seekback, SEEK_SET) == (off_t) -1)\n \t\t\treturn error(\"cannot seek back\");\n \t}\n-\tthe_hash_algo->final_oid_fn(result_oid, &ctx);\n+\tthe_hash_algo->fn.final_oid(result_oid, &ctx);\n \tif (!idx)\n \t\treturn 0;\n \ndiff --git a/csum-file.c b/csum-file.c\nindex 107bc4c96f..05e20de441 100644\n--- a/csum-file.c\n+++ b/csum-file.c\n@@ -50,7 +50,7 @@ void hashflush(struct hashfile *f)\n \n \tif (offset) {\n \t\tif (!f->skip_hash)\n-\t\t\tf->algop->unsafe_update_fn(&f->ctx, f->buffer, offset);\n+\t\t\tf->algop->unsafe_fn.update(&f->ctx, f->buffer, offset);\n \t\tflush(f, f->buffer, offset);\n \t\tf->offset = 0;\n \t}\n@@ -73,7 +73,7 @@ int finalize_hashfile(struct hashfile *f, unsigned char *result,\n \tif (f->skip_hash)\n \t\thashclr(f->buffer, f->algop);\n \telse\n-\t\tf->algop->unsafe_final_fn(f->buffer, &f->ctx);\n+\t\tf->algop->unsafe_fn.final(f->buffer, &f->ctx);\n \n \tif (result)\n \t\thashcpy(result, f->buffer, f->algop);\n@@ -128,7 +128,7 @@ void hashwrite(struct hashfile *f, const void *buf, unsigned int count)\n \t\t\t * f->offset is necessarily zero.\n \t\t\t */\n \t\t\tif (!f->skip_hash)\n-\t\t\t\tf->algop->unsafe_update_fn(&f->ctx, buf, nr);\n+\t\t\t\tf->algop->unsafe_fn.update(&f->ctx, buf, nr);\n \t\t\tflush(f, buf, nr);\n \t\t} else {\n \t\t\t/*\n@@ -176,7 +176,7 @@ static struct hashfile *hashfd_internal(int fd, const char *name,\n \tf->skip_hash = 0;\n \n \tf->algop = the_hash_algo;\n-\tf->algop->unsafe_init_fn(&f->ctx);\n+\tf->algop->unsafe_fn.init(&f->ctx);\n \n \tf->buffer_len = buffer_len;\n \tf->buffer = xmalloc(buffer_len);\n@@ -210,7 +210,7 @@ void hashfile_checkpoint(struct hashfile *f, struct hashfile_checkpoint *checkpo\n {\n \thashflush(f);\n \tcheckpoint->offset = f->total;\n-\tf->algop->unsafe_clone_fn(&checkpoint->ctx, &f->ctx);\n+\tf->algop->unsafe_fn.clone(&checkpoint->ctx, &f->ctx);\n }\n \n int hashfile_truncate(struct hashfile *f, struct hashfile_checkpoint *checkpoint)\n@@ -221,7 +221,7 @@ int hashfile_truncate(struct hashfile *f, struct hashfile_checkpoint *checkpoint\n \t    lseek(f->fd, offset, SEEK_SET) != offset)\n \t\treturn -1;\n \tf->total = offset;\n-\tf->algop->unsafe_clone_fn(&f->ctx, &checkpoint->ctx);\n+\tf->algop->unsafe_fn.clone(&f->ctx, &checkpoint->ctx);\n \tf->offset = 0; /* hashflush() was called in checkpoint */\n \treturn 0;\n }\n@@ -248,9 +248,9 @@ int hashfile_checksum_valid(const unsigned char *data, size_t total_len)\n \tif (total_len < algop->rawsz)\n \t\treturn 0; /* say \"too short\"? */\n \n-\talgop->unsafe_init_fn(&ctx);\n-\talgop->unsafe_update_fn(&ctx, data, data_len);\n-\talgop->unsafe_final_fn(got, &ctx);\n+\talgop->unsafe_fn.init(&ctx);\n+\talgop->unsafe_fn.update(&ctx, data, data_len);\n+\talgop->unsafe_fn.final(got, &ctx);\n \n \treturn hasheq(got, data + data_len, algop);\n }\ndiff --git a/diff.c b/diff.c\nindex dceac20d18..85d776fa37 100644\n--- a/diff.c\n+++ b/diff.c\n@@ -6413,8 +6413,8 @@ void flush_one_hunk(struct object_id *result, git_hash_ctx *ctx)\n \tunsigned short carry = 0;\n \tint i;\n \n-\tthe_hash_algo->final_fn(hash, ctx);\n-\tthe_hash_algo->init_fn(ctx);\n+\tthe_hash_algo->fn.final(hash, ctx);\n+\tthe_hash_algo->fn.init(ctx);\n \t/* 20-byte sum, with carry */\n \tfor (i = 0; i < the_hash_algo->rawsz; ++i) {\n \t\tcarry += result->hash[i] + hash[i];\n@@ -6432,22 +6432,22 @@ static int patch_id_consume(void *priv, char *line, unsigned long len)\n \t\treturn 0;\n \tnew_len = remove_space(line, len);\n \n-\tthe_hash_algo->update_fn(data->ctx, line, new_len);\n+\tthe_hash_algo->fn.update(data->ctx, line, new_len);\n \tdata->patchlen += new_len;\n \treturn 0;\n }\n \n static void patch_id_add_string(git_hash_ctx *ctx, const char *str)\n {\n-\tthe_hash_algo->update_fn(ctx, str, strlen(str));\n+\tthe_hash_algo->fn.update(ctx, str, strlen(str));\n }\n \n static void patch_id_add_mode(git_hash_ctx *ctx, unsigned mode)\n {\n \t/* large enough for 2^32 in octal */\n \tchar buf[12];\n \tint len = xsnprintf(buf, sizeof(buf), \"%06o\", mode);\n-\tthe_hash_algo->update_fn(ctx, buf, len);\n+\tthe_hash_algo->fn.update(ctx, buf, len);\n }\n \n /* returns 0 upon success, and writes result into oid */\n@@ -6458,7 +6458,7 @@ static int diff_get_patch_id(struct diff_options *options, struct object_id *oid\n \tgit_hash_ctx ctx;\n \tstruct patch_id_t data;\n \n-\tthe_hash_algo->init_fn(&ctx);\n+\tthe_hash_algo->fn.init(&ctx);\n \tmemset(&data, 0, sizeof(struct patch_id_t));\n \tdata.ctx = &ctx;\n \toidclr(oid, the_repository->hash_algo);\n@@ -6491,9 +6491,9 @@ static int diff_get_patch_id(struct diff_options *options, struct object_id *oid\n \t\tlen2 = remove_space(p->two->path, strlen(p->two->path));\n \t\tpatch_id_add_string(&ctx, \"diff--git\");\n \t\tpatch_id_add_string(&ctx, \"a/\");\n-\t\tthe_hash_algo->update_fn(&ctx, p->one->path, len1);\n+\t\tthe_hash_algo->fn.update(&ctx, p->one->path, len1);\n \t\tpatch_id_add_string(&ctx, \"b/\");\n-\t\tthe_hash_algo->update_fn(&ctx, p->two->path, len2);\n+\t\tthe_hash_algo->fn.update(&ctx, p->two->path, len2);\n \n \t\tif (p->one->mode == 0) {\n \t\t\tpatch_id_add_string(&ctx, \"newfilemode\");\n@@ -6512,24 +6512,24 @@ static int diff_get_patch_id(struct diff_options *options, struct object_id *oid\n \t\t\t/* don't do anything since we're only populating header info */\n \t\t} else if (diff_filespec_is_binary(options->repo, p->one) ||\n \t\t    diff_filespec_is_binary(options->repo, p->two)) {\n-\t\t\tthe_hash_algo->update_fn(&ctx, oid_to_hex(&p->one->oid),\n+\t\t\tthe_hash_algo->fn.update(&ctx, oid_to_hex(&p->one->oid),\n \t\t\t\t\tthe_hash_algo->hexsz);\n-\t\t\tthe_hash_algo->update_fn(&ctx, oid_to_hex(&p->two->oid),\n+\t\t\tthe_hash_algo->fn.update(&ctx, oid_to_hex(&p->two->oid),\n \t\t\t\t\tthe_hash_algo->hexsz);\n \t\t} else {\n \t\t\tif (p->one->mode == 0) {\n \t\t\t\tpatch_id_add_string(&ctx, \"---/dev/null\");\n \t\t\t\tpatch_id_add_string(&ctx, \"+++b/\");\n-\t\t\t\tthe_hash_algo->update_fn(&ctx, p->two->path, len2);\n+\t\t\t\tthe_hash_algo->fn.update(&ctx, p->two->path, len2);\n \t\t\t} else if (p->two->mode == 0) {\n \t\t\t\tpatch_id_add_string(&ctx, \"---a/\");\n-\t\t\t\tthe_hash_algo->update_fn(&ctx, p->one->path, len1);\n+\t\t\t\tthe_hash_algo->fn.update(&ctx, p->one->path, len1);\n \t\t\t\tpatch_id_add_string(&ctx, \"+++/dev/null\");\n \t\t\t} else {\n \t\t\t\tpatch_id_add_string(&ctx, \"---a/\");\n-\t\t\t\tthe_hash_algo->update_fn(&ctx, p->one->path, len1);\n+\t\t\t\tthe_hash_algo->fn.update(&ctx, p->one->path, len1);\n \t\t\t\tpatch_id_add_string(&ctx, \"+++b/\");\n-\t\t\t\tthe_hash_algo->update_fn(&ctx, p->two->path, len2);\n+\t\t\t\tthe_hash_algo->fn.update(&ctx, p->two->path, len2);\n \t\t\t}\n \n \t\t\tif (fill_mmfile(options->repo, &mf1, p->one) < 0 ||\ndiff --git a/hash.h b/hash.h\nindex 756166ce5e..6107d5b47a 100644\n--- a/hash.h\n+++ b/hash.h\n@@ -267,35 +267,24 @@ struct git_hash_algo {\n \t/* The block size of the hash. */\n \tsize_t blksz;\n \n-\t/* The hash initialization function. */\n-\tgit_hash_init_fn init_fn;\n+\tstruct git_hash_algo_fns {\n+\t\t/* The hash initialization function. */\n+\t\tgit_hash_init_fn init;\n \n-\t/* The hash context cloning function. */\n-\tgit_hash_clone_fn clone_fn;\n+\t\t/* The hash context cloning function. */\n+\t\tgit_hash_clone_fn clone;\n \n-\t/* The hash update function. */\n-\tgit_hash_update_fn update_fn;\n+\t\t/* The hash update function. */\n+\t\tgit_hash_update_fn update;\n \n-\t/* The hash finalization function. */\n-\tgit_hash_final_fn final_fn;\n+\t\t/* The hash finalization function. */\n+\t\tgit_hash_final_fn final;\n \n-\t/* The hash finalization function for object IDs. */\n-\tgit_hash_final_oid_fn final_oid_fn;\n+\t\t/* The hash finalization function for object IDs. */\n+\t\tgit_hash_final_oid_fn final_oid;\n+\t} fn;\n \n-\t/* The non-cryptographic hash initialization function. */\n-\tgit_hash_init_fn unsafe_init_fn;\n-\n-\t/* The non-cryptographic hash context cloning function. */\n-\tgit_hash_clone_fn unsafe_clone_fn;\n-\n-\t/* The non-cryptographic hash update function. */\n-\tgit_hash_update_fn unsafe_update_fn;\n-\n-\t/* The non-cryptographic hash finalization function. */\n-\tgit_hash_final_fn unsafe_final_fn;\n-\n-\t/* The non-cryptographic hash finalization function. */\n-\tgit_hash_final_oid_fn unsafe_final_oid_fn;\n+\tstruct git_hash_algo_fns unsafe_fn;\n \n \t/* The OID of the empty tree. */\n \tconst struct object_id *empty_tree;\ndiff --git a/http-push.c b/http-push.c\nindex 4d24e6b8d4..9659e777d1 100644\n--- a/http-push.c\n+++ b/http-push.c\n@@ -773,9 +773,9 @@ static void handle_new_lock_ctx(struct xml_ctx *ctx, int tag_closed)\n \t\t} else if (!strcmp(ctx->name, DAV_ACTIVELOCK_TOKEN)) {\n \t\t\tlock->token = xstrdup(ctx->cdata);\n \n-\t\t\tthe_hash_algo->init_fn(&hash_ctx);\n-\t\t\tthe_hash_algo->update_fn(&hash_ctx, lock->token, strlen(lock->token));\n-\t\t\tthe_hash_algo->final_fn(lock_token_hash, &hash_ctx);\n+\t\t\tthe_hash_algo->fn.init(&hash_ctx);\n+\t\t\tthe_hash_algo->fn.update(&hash_ctx, lock->token, strlen(lock->token));\n+\t\t\tthe_hash_algo->fn.final(lock_token_hash, &hash_ctx);\n \n \t\t\tlock->tmpfile_suffix[0] = '_';\n \t\t\tmemcpy(lock->tmpfile_suffix + 1, hash_to_hex(lock_token_hash), the_hash_algo->hexsz);\ndiff --git a/http.c b/http.c\nindex 58242b9d2d..b3f31b52e6 100644\n--- a/http.c\n+++ b/http.c\n@@ -2654,7 +2654,7 @@ static size_t fwrite_sha1_file(char *ptr, size_t eltsize, size_t nmemb,\n \t\tfreq->stream.next_out = expn;\n \t\tfreq->stream.avail_out = sizeof(expn);\n \t\tfreq->zret = git_inflate(&freq->stream, Z_SYNC_FLUSH);\n-\t\tthe_hash_algo->update_fn(&freq->c, expn,\n+\t\tthe_hash_algo->fn.update(&freq->c, expn,\n \t\t\t\t\t sizeof(expn) - freq->stream.avail_out);\n \t} while (freq->stream.avail_in && freq->zret == Z_OK);\n \treturn nmemb;\n@@ -2713,7 +2713,7 @@ struct http_object_request *new_http_object_request(const char *base_url,\n \n \tgit_inflate_init(&freq->stream);\n \n-\tthe_hash_algo->init_fn(&freq->c);\n+\tthe_hash_algo->fn.init(&freq->c);\n \n \tfreq->url = get_remote_object_url(base_url, hex, 0);\n \n@@ -2749,7 +2749,7 @@ struct http_object_request *new_http_object_request(const char *base_url,\n \t\tgit_inflate_end(&freq->stream);\n \t\tmemset(&freq->stream, 0, sizeof(freq->stream));\n \t\tgit_inflate_init(&freq->stream);\n-\t\tthe_hash_algo->init_fn(&freq->c);\n+\t\tthe_hash_algo->fn.init(&freq->c);\n \t\tif (prev_posn>0) {\n \t\t\tprev_posn = 0;\n \t\t\tlseek(freq->localfile, 0, SEEK_SET);\n@@ -2820,7 +2820,7 @@ int finish_http_object_request(struct http_object_request *freq)\n \t\treturn -1;\n \t}\n \n-\tthe_hash_algo->final_oid_fn(&freq->real_oid, &freq->c);\n+\tthe_hash_algo->fn.final_oid(&freq->real_oid, &freq->c);\n \tif (freq->zret != Z_STREAM_END) {\n \t\tunlink_or_warn(freq->tmpfile.buf);\n \t\treturn -1;\ndiff --git a/object-file.c b/object-file.c\nindex b1a3463852..c8399745d8 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -211,16 +211,20 @@ const struct git_hash_algo hash_algos[GIT_HASH_NALGOS] = {\n \t\t.rawsz = 0,\n \t\t.hexsz = 0,\n \t\t.blksz = 0,\n-\t\t.init_fn = git_hash_unknown_init,\n-\t\t.clone_fn = git_hash_unknown_clone,\n-\t\t.update_fn = git_hash_unknown_update,\n-\t\t.final_fn = git_hash_unknown_final,\n-\t\t.final_oid_fn = git_hash_unknown_final_oid,\n-\t\t.unsafe_init_fn = git_hash_unknown_init,\n-\t\t.unsafe_clone_fn = git_hash_unknown_clone,\n-\t\t.unsafe_update_fn = git_hash_unknown_update,\n-\t\t.unsafe_final_fn = git_hash_unknown_final,\n-\t\t.unsafe_final_oid_fn = git_hash_unknown_final_oid,\n+\t\t.fn = {\n+\t\t\t.init = git_hash_unknown_init,\n+\t\t\t.clone = git_hash_unknown_clone,\n+\t\t\t.update = git_hash_unknown_update,\n+\t\t\t.final = git_hash_unknown_final,\n+\t\t\t.final_oid = git_hash_unknown_final_oid,\n+\t\t},\n+\t\t.unsafe_fn = {\n+\t\t\t.init = git_hash_unknown_init,\n+\t\t\t.clone = git_hash_unknown_clone,\n+\t\t\t.update = git_hash_unknown_update,\n+\t\t\t.final = git_hash_unknown_final,\n+\t\t\t.final_oid = git_hash_unknown_final_oid,\n+\t\t},\n \t\t.empty_tree = NULL,\n \t\t.empty_blob = NULL,\n \t\t.null_oid = NULL,\n@@ -231,16 +235,20 @@ const struct git_hash_algo hash_algos[GIT_HASH_NALGOS] = {\n \t\t.rawsz = GIT_SHA1_RAWSZ,\n \t\t.hexsz = GIT_SHA1_HEXSZ,\n \t\t.blksz = GIT_SHA1_BLKSZ,\n-\t\t.init_fn = git_hash_sha1_init,\n-\t\t.clone_fn = git_hash_sha1_clone,\n-\t\t.update_fn = git_hash_sha1_update,\n-\t\t.final_fn = git_hash_sha1_final,\n-\t\t.final_oid_fn = git_hash_sha1_final_oid,\n-\t\t.unsafe_init_fn = git_hash_sha1_init_unsafe,\n-\t\t.unsafe_clone_fn = git_hash_sha1_clone_unsafe,\n-\t\t.unsafe_update_fn = git_hash_sha1_update_unsafe,\n-\t\t.unsafe_final_fn = git_hash_sha1_final_unsafe,\n-\t\t.unsafe_final_oid_fn = git_hash_sha1_final_oid_unsafe,\n+\t\t.fn = {\n+\t\t\t.init = git_hash_sha1_init,\n+\t\t\t.clone = git_hash_sha1_clone,\n+\t\t\t.update = git_hash_sha1_update,\n+\t\t\t.final = git_hash_sha1_final,\n+\t\t\t.final_oid = git_hash_sha1_final_oid,\n+\t\t},\n+\t\t.unsafe_fn = {\n+\t\t\t.init = git_hash_sha1_init_unsafe,\n+\t\t\t.clone = git_hash_sha1_clone_unsafe,\n+\t\t\t.update = git_hash_sha1_update_unsafe,\n+\t\t\t.final = git_hash_sha1_final_unsafe,\n+\t\t\t.final_oid = git_hash_sha1_final_oid_unsafe,\n+\t\t},\n \t\t.empty_tree = &empty_tree_oid,\n \t\t.empty_blob = &empty_blob_oid,\n \t\t.null_oid = &null_oid_sha1,\n@@ -251,16 +259,20 @@ const struct git_hash_algo hash_algos[GIT_HASH_NALGOS] = {\n \t\t.rawsz = GIT_SHA256_RAWSZ,\n \t\t.hexsz = GIT_SHA256_HEXSZ,\n \t\t.blksz = GIT_SHA256_BLKSZ,\n-\t\t.init_fn = git_hash_sha256_init,\n-\t\t.clone_fn = git_hash_sha256_clone,\n-\t\t.update_fn = git_hash_sha256_update,\n-\t\t.final_fn = git_hash_sha256_final,\n-\t\t.final_oid_fn = git_hash_sha256_final_oid,\n-\t\t.unsafe_init_fn = git_hash_sha256_init,\n-\t\t.unsafe_clone_fn = git_hash_sha256_clone,\n-\t\t.unsafe_update_fn = git_hash_sha256_update,\n-\t\t.unsafe_final_fn = git_hash_sha256_final,\n-\t\t.unsafe_final_oid_fn = git_hash_sha256_final_oid,\n+\t\t.fn = {\n+\t\t\t.init = git_hash_sha256_init,\n+\t\t\t.clone = git_hash_sha256_clone,\n+\t\t\t.update = git_hash_sha256_update,\n+\t\t\t.final = git_hash_sha256_final,\n+\t\t\t.final_oid = git_hash_sha256_final_oid,\n+\t\t},\n+\t\t.unsafe_fn = {\n+\t\t\t.init = git_hash_sha256_init,\n+\t\t\t.clone = git_hash_sha256_clone,\n+\t\t\t.update = git_hash_sha256_update,\n+\t\t\t.final = git_hash_sha256_final,\n+\t\t\t.final_oid = git_hash_sha256_final_oid,\n+\t\t},\n \t\t.empty_tree = &empty_tree_oid_sha256,\n \t\t.empty_blob = &empty_blob_oid_sha256,\n \t\t.null_oid = &null_oid_sha256,\n@@ -1185,8 +1197,8 @@ int stream_object_signature(struct repository *r, const struct object_id *oid)\n \thdrlen = format_object_header(hdr, sizeof(hdr), obj_type, size);\n \n \t/* Sha1.. */\n-\tr->hash_algo->init_fn(&c);\n-\tr->hash_algo->update_fn(&c, hdr, hdrlen);\n+\tr->hash_algo->fn.init(&c);\n+\tr->hash_algo->fn.update(&c, hdr, hdrlen);\n \tfor (;;) {\n \t\tchar buf[1024 * 16];\n \t\tssize_t readlen = read_istream(st, buf, sizeof(buf));\n@@ -1197,9 +1209,9 @@ int stream_object_signature(struct repository *r, const struct object_id *oid)\n \t\t}\n \t\tif (!readlen)\n \t\t\tbreak;\n-\t\tr->hash_algo->update_fn(&c, buf, readlen);\n+\t\tr->hash_algo->fn.update(&c, buf, readlen);\n \t}\n-\tr->hash_algo->final_oid_fn(&real_oid, &c);\n+\tr->hash_algo->fn.final_oid(&real_oid, &c);\n \tclose_istream(st);\n \treturn !oideq(oid, &real_oid) ? -1 : 0;\n }\n@@ -1943,10 +1955,10 @@ static void hash_object_body(const struct git_hash_algo *algo, git_hash_ctx *c,\n \t\t\t     struct object_id *oid,\n \t\t\t     char *hdr, int *hdrlen)\n {\n-\talgo->init_fn(c);\n-\talgo->update_fn(c, hdr, *hdrlen);\n-\talgo->update_fn(c, buf, len);\n-\talgo->final_oid_fn(oid, c);\n+\talgo->fn.init(c);\n+\talgo->fn.update(c, hdr, *hdrlen);\n+\talgo->fn.update(c, buf, len);\n+\talgo->fn.final_oid(oid, c);\n }\n \n static void write_object_file_prepare(const struct git_hash_algo *algo,\n@@ -2206,18 +2218,18 @@ static int start_loose_object_common(struct strbuf *tmp_file,\n \tgit_deflate_init(stream, zlib_compression_level);\n \tstream->next_out = buf;\n \tstream->avail_out = buflen;\n-\talgo->init_fn(c);\n+\talgo->fn.init(c);\n \tif (compat && compat_c)\n-\t\tcompat->init_fn(compat_c);\n+\t\tcompat->fn.init(compat_c);\n \n \t/*  Start to feed header to zlib stream */\n \tstream->next_in = (unsigned char *)hdr;\n \tstream->avail_in = hdrlen;\n \twhile (git_deflate(stream, 0) == Z_OK)\n \t\t; /* nothing */\n-\talgo->update_fn(c, hdr, hdrlen);\n+\talgo->fn.update(c, hdr, hdrlen);\n \tif (compat && compat_c)\n-\t\tcompat->update_fn(compat_c, hdr, hdrlen);\n+\t\tcompat->fn.update(compat_c, hdr, hdrlen);\n \n \treturn fd;\n }\n@@ -2238,9 +2250,9 @@ static int write_loose_object_common(git_hash_ctx *c, git_hash_ctx *compat_c,\n \tint ret;\n \n \tret = git_deflate(stream, flush ? Z_FINISH : 0);\n-\talgo->update_fn(c, in0, stream->next_in - in0);\n+\talgo->fn.update(c, in0, stream->next_in - in0);\n \tif (compat && compat_c)\n-\t\tcompat->update_fn(compat_c, in0, stream->next_in - in0);\n+\t\tcompat->fn.update(compat_c, in0, stream->next_in - in0);\n \tif (write_in_full(fd, compressed, stream->next_out - compressed) < 0)\n \t\tdie_errno(_(\"unable to write loose object file\"));\n \tstream->next_out = compressed;\n@@ -2267,9 +2279,9 @@ static int end_loose_object_common(git_hash_ctx *c, git_hash_ctx *compat_c,\n \tret = git_deflate_end_gently(stream);\n \tif (ret != Z_OK)\n \t\treturn ret;\n-\talgo->final_oid_fn(oid, c);\n+\talgo->fn.final_oid(oid, c);\n \tif (compat && compat_c)\n-\t\tcompat->final_oid_fn(compat_oid, compat_c);\n+\t\tcompat->fn.final_oid(compat_oid, compat_c);\n \n \treturn Z_OK;\n }\n@@ -3027,8 +3039,8 @@ static int check_stream_oid(git_zstream *stream,\n \tunsigned long total_read;\n \tint status = Z_OK;\n \n-\tthe_hash_algo->init_fn(&c);\n-\tthe_hash_algo->update_fn(&c, hdr, stream->total_out);\n+\tthe_hash_algo->fn.init(&c);\n+\tthe_hash_algo->fn.update(&c, hdr, stream->total_out);\n \n \t/*\n \t * We already read some bytes into hdr, but the ones up to the NUL\n@@ -3048,7 +3060,7 @@ static int check_stream_oid(git_zstream *stream,\n \t\tif (size - total_read < stream->avail_out)\n \t\t\tstream->avail_out = size - total_read;\n \t\tstatus = git_inflate(stream, Z_FINISH);\n-\t\tthe_hash_algo->update_fn(&c, buf, stream->next_out - buf);\n+\t\tthe_hash_algo->fn.update(&c, buf, stream->next_out - buf);\n \t\ttotal_read += stream->next_out - buf;\n \t}\n \tgit_inflate_end(stream);\n@@ -3063,7 +3075,7 @@ static int check_stream_oid(git_zstream *stream,\n \t\treturn -1;\n \t}\n \n-\tthe_hash_algo->final_oid_fn(&real_oid, &c);\n+\tthe_hash_algo->fn.final_oid(&real_oid, &c);\n \tif (!oideq(expected_oid, &real_oid)) {\n \t\terror(_(\"hash mismatch for %s (expected %s)\"), path,\n \t\t      oid_to_hex(expected_oid));\ndiff --git a/pack-check.c b/pack-check.c\nindex e883dae3f2..2c1c2c54fa 100644\n--- a/pack-check.c\n+++ b/pack-check.c\n@@ -67,7 +67,7 @@ static int verify_packfile(struct repository *r,\n \tif (!is_pack_valid(p))\n \t\treturn error(\"packfile %s cannot be accessed\", p->pack_name);\n \n-\tr->hash_algo->init_fn(&ctx);\n+\tr->hash_algo->fn.init(&ctx);\n \tdo {\n \t\tunsigned long remaining;\n \t\tunsigned char *in = use_pack(p, w_curs, offset, &remaining);\n@@ -76,9 +76,9 @@ static int verify_packfile(struct repository *r,\n \t\t\tpack_sig_ofs = p->pack_size - r->hash_algo->rawsz;\n \t\tif (offset > pack_sig_ofs)\n \t\t\tremaining -= (unsigned int)(offset - pack_sig_ofs);\n-\t\tr->hash_algo->update_fn(&ctx, in, remaining);\n+\t\tr->hash_algo->fn.update(&ctx, in, remaining);\n \t} while (offset < pack_sig_ofs);\n-\tr->hash_algo->final_fn(hash, &ctx);\n+\tr->hash_algo->fn.final(hash, &ctx);\n \tpack_sig = use_pack(p, w_curs, pack_sig_ofs, NULL);\n \tif (!hasheq(hash, pack_sig, the_repository->hash_algo))\n \t\terr = error(\"%s pack checksum mismatch\",\ndiff --git a/pack-write.c b/pack-write.c\nindex 8c7dfddc5a..d202c6c9a4 100644\n--- a/pack-write.c\n+++ b/pack-write.c\n@@ -392,8 +392,8 @@ void fixup_pack_header_footer(int pack_fd,\n \tchar *buf;\n \tssize_t read_result;\n \n-\tthe_hash_algo->init_fn(&old_hash_ctx);\n-\tthe_hash_algo->init_fn(&new_hash_ctx);\n+\tthe_hash_algo->fn.init(&old_hash_ctx);\n+\tthe_hash_algo->fn.init(&new_hash_ctx);\n \n \tif (lseek(pack_fd, 0, SEEK_SET) != 0)\n \t\tdie_errno(\"Failed seeking to start of '%s'\", pack_name);\n@@ -405,9 +405,9 @@ void fixup_pack_header_footer(int pack_fd,\n \t\t\t  pack_name);\n \tif (lseek(pack_fd, 0, SEEK_SET) != 0)\n \t\tdie_errno(\"Failed seeking to start of '%s'\", pack_name);\n-\tthe_hash_algo->update_fn(&old_hash_ctx, &hdr, sizeof(hdr));\n+\tthe_hash_algo->fn.update(&old_hash_ctx, &hdr, sizeof(hdr));\n \thdr.hdr_entries = htonl(object_count);\n-\tthe_hash_algo->update_fn(&new_hash_ctx, &hdr, sizeof(hdr));\n+\tthe_hash_algo->fn.update(&new_hash_ctx, &hdr, sizeof(hdr));\n \twrite_or_die(pack_fd, &hdr, sizeof(hdr));\n \tpartial_pack_offset -= sizeof(hdr);\n \n@@ -422,7 +422,7 @@ void fixup_pack_header_footer(int pack_fd,\n \t\t\tbreak;\n \t\tif (n < 0)\n \t\t\tdie_errno(\"Failed to checksum '%s'\", pack_name);\n-\t\tthe_hash_algo->update_fn(&new_hash_ctx, buf, n);\n+\t\tthe_hash_algo->fn.update(&new_hash_ctx, buf, n);\n \n \t\taligned_sz -= n;\n \t\tif (!aligned_sz)\n@@ -431,11 +431,11 @@ void fixup_pack_header_footer(int pack_fd,\n \t\tif (!partial_pack_hash)\n \t\t\tcontinue;\n \n-\t\tthe_hash_algo->update_fn(&old_hash_ctx, buf, n);\n+\t\tthe_hash_algo->fn.update(&old_hash_ctx, buf, n);\n \t\tpartial_pack_offset -= n;\n \t\tif (partial_pack_offset == 0) {\n \t\t\tunsigned char hash[GIT_MAX_RAWSZ];\n-\t\t\tthe_hash_algo->final_fn(hash, &old_hash_ctx);\n+\t\t\tthe_hash_algo->fn.final(hash, &old_hash_ctx);\n \t\t\tif (!hasheq(hash, partial_pack_hash,\n \t\t\t\t    the_repository->hash_algo))\n \t\t\t\tdie(\"Unexpected checksum for %s \"\n@@ -445,16 +445,16 @@ void fixup_pack_header_footer(int pack_fd,\n \t\t\t * pack, which also means making partial_pack_offset\n \t\t\t * big enough not to matter anymore.\n \t\t\t */\n-\t\t\tthe_hash_algo->init_fn(&old_hash_ctx);\n+\t\t\tthe_hash_algo->fn.init(&old_hash_ctx);\n \t\t\tpartial_pack_offset = ~partial_pack_offset;\n \t\t\tpartial_pack_offset -= MSB(partial_pack_offset, 1);\n \t\t}\n \t}\n \tfree(buf);\n \n \tif (partial_pack_hash)\n-\t\tthe_hash_algo->final_fn(partial_pack_hash, &old_hash_ctx);\n-\tthe_hash_algo->final_fn(new_pack_hash, &new_hash_ctx);\n+\t\tthe_hash_algo->fn.final(partial_pack_hash, &old_hash_ctx);\n+\tthe_hash_algo->fn.final(new_pack_hash, &new_hash_ctx);\n \twrite_or_die(pack_fd, new_pack_hash, the_hash_algo->rawsz);\n \tfsync_component_or_die(FSYNC_COMPONENT_PACK, pack_fd, pack_name);\n }\ndiff --git a/read-cache.c b/read-cache.c\nindex 01d0b3ad22..5170b07a33 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -1736,9 +1736,9 @@ static int verify_hdr(const struct cache_header *hdr, unsigned long size)\n \tif (oideq(&oid, null_oid()))\n \t\treturn 0;\n \n-\tthe_hash_algo->init_fn(&c);\n-\tthe_hash_algo->update_fn(&c, hdr, size - the_hash_algo->rawsz);\n-\tthe_hash_algo->final_fn(hash, &c);\n+\tthe_hash_algo->fn.init(&c);\n+\tthe_hash_algo->fn.update(&c, hdr, size - the_hash_algo->rawsz);\n+\tthe_hash_algo->fn.final(hash, &c);\n \tif (!hasheq(hash, start, the_repository->hash_algo))\n \t\treturn error(_(\"bad index file sha1 signature\"));\n \treturn 0;\n@@ -2574,8 +2574,8 @@ static int write_index_ext_header(struct hashfile *f,\n \tif (eoie_f) {\n \t\text = htonl(ext);\n \t\tsz = htonl(sz);\n-\t\tthe_hash_algo->update_fn(eoie_f, &ext, sizeof(ext));\n-\t\tthe_hash_algo->update_fn(eoie_f, &sz, sizeof(sz));\n+\t\tthe_hash_algo->fn.update(eoie_f, &ext, sizeof(ext));\n+\t\tthe_hash_algo->fn.update(eoie_f, &sz, sizeof(sz));\n \t}\n \treturn 0;\n }\n@@ -2977,7 +2977,7 @@ static int do_write_index(struct index_state *istate, struct tempfile *tempfile,\n \t */\n \tif (offset && record_eoie()) {\n \t\tCALLOC_ARRAY(eoie_c, 1);\n-\t\tthe_hash_algo->init_fn(eoie_c);\n+\t\tthe_hash_algo->fn.init(eoie_c);\n \t}\n \n \t/*\n@@ -3616,7 +3616,7 @@ static size_t read_eoie_extension(const char *mmap, size_t mmap_size)\n \t *\t \"REUC\" + <binary representation of M>)\n \t */\n \tsrc_offset = offset;\n-\tthe_hash_algo->init_fn(&c);\n+\tthe_hash_algo->fn.init(&c);\n \twhile (src_offset < mmap_size - the_hash_algo->rawsz - EOIE_SIZE_WITH_HEADER) {\n \t\t/* After an array of active_nr index entries,\n \t\t * there can be arbitrary number of extended\n@@ -3632,12 +3632,12 @@ static size_t read_eoie_extension(const char *mmap, size_t mmap_size)\n \t\tif (src_offset + 8 + extsize < src_offset)\n \t\t\treturn 0;\n \n-\t\tthe_hash_algo->update_fn(&c, mmap + src_offset, 8);\n+\t\tthe_hash_algo->fn.update(&c, mmap + src_offset, 8);\n \n \t\tsrc_offset += 8;\n \t\tsrc_offset += extsize;\n \t}\n-\tthe_hash_algo->final_fn(hash, &c);\n+\tthe_hash_algo->fn.final(hash, &c);\n \tif (!hasheq(hash, (const unsigned char *)index, the_repository->hash_algo))\n \t\treturn 0;\n \n@@ -3658,7 +3658,7 @@ static void write_eoie_extension(struct strbuf *sb, git_hash_ctx *eoie_context,\n \tstrbuf_add(sb, &buffer, sizeof(uint32_t));\n \n \t/* hash */\n-\tthe_hash_algo->final_fn(hash, eoie_context);\n+\tthe_hash_algo->fn.final(hash, eoie_context);\n \tstrbuf_add(sb, hash, the_hash_algo->rawsz);\n }\n \ndiff --git a/rerere.c b/rerere.c\nindex d01e98bf65..ef3b58b5c7 100644\n--- a/rerere.c\n+++ b/rerere.c\n@@ -395,10 +395,10 @@ static int handle_conflict(struct strbuf *out, struct rerere_io *io,\n \t\t\tstrbuf_addbuf(out, &two);\n \t\t\trerere_strbuf_putconflict(out, '>', marker_size);\n \t\t\tif (ctx) {\n-\t\t\t\tthe_hash_algo->update_fn(ctx, one.buf ?\n+\t\t\t\tthe_hash_algo->fn.update(ctx, one.buf ?\n \t\t\t\t\t\t\t one.buf : \"\",\n \t\t\t\t\t\t\t one.len + 1);\n-\t\t\t\tthe_hash_algo->update_fn(ctx, two.buf ?\n+\t\t\t\tthe_hash_algo->fn.update(ctx, two.buf ?\n \t\t\t\t\t\t\t two.buf : \"\",\n \t\t\t\t\t\t\t two.len + 1);\n \t\t\t}\n@@ -435,7 +435,7 @@ static int handle_path(unsigned char *hash, struct rerere_io *io, int marker_siz\n \tstruct strbuf buf = STRBUF_INIT, out = STRBUF_INIT;\n \tint has_conflicts = 0;\n \tif (hash)\n-\t\tthe_hash_algo->init_fn(&ctx);\n+\t\tthe_hash_algo->fn.init(&ctx);\n \n \twhile (!io->getline(&buf, io)) {\n \t\tif (is_cmarker(buf.buf, '<', marker_size)) {\n@@ -452,7 +452,7 @@ static int handle_path(unsigned char *hash, struct rerere_io *io, int marker_siz\n \tstrbuf_release(&out);\n \n \tif (hash)\n-\t\tthe_hash_algo->final_fn(hash, &ctx);\n+\t\tthe_hash_algo->fn.final(hash, &ctx);\n \n \treturn has_conflicts;\n }\ndiff --git a/t/helper/test-hash-speed.c b/t/helper/test-hash-speed.c\nindex 7de822af51..1fd8cdac79 100644\n--- a/t/helper/test-hash-speed.c\n+++ b/t/helper/test-hash-speed.c\n@@ -5,9 +5,9 @@\n \n static inline void compute_hash(const struct git_hash_algo *algo, git_hash_ctx *ctx, uint8_t *final, const void *p, size_t len)\n {\n-\talgo->init_fn(ctx);\n-\talgo->update_fn(ctx, p, len);\n-\talgo->final_fn(final, ctx);\n+\talgo->fn.init(ctx);\n+\talgo->fn.update(ctx, p, len);\n+\talgo->fn.final(final, ctx);\n }\n \n int cmd__hash_speed(int ac, const char **av)\ndiff --git a/t/helper/test-hash.c b/t/helper/test-hash.c\nindex 45d829c908..3e740bc4fb 100644\n--- a/t/helper/test-hash.c\n+++ b/t/helper/test-hash.c\n@@ -27,7 +27,7 @@ int cmd_hash_impl(int ac, const char **av, int algo)\n \t\t\tdie(\"OOPS\");\n \t}\n \n-\talgop->init_fn(&ctx);\n+\talgop->fn.init(&ctx);\n \n \twhile (1) {\n \t\tssize_t sz, this_sz;\n@@ -46,9 +46,9 @@ int cmd_hash_impl(int ac, const char **av, int algo)\n \t\t}\n \t\tif (this_sz == 0)\n \t\t\tbreak;\n-\t\talgop->update_fn(&ctx, buffer, this_sz);\n+\t\talgop->fn.update(&ctx, buffer, this_sz);\n \t}\n-\talgop->final_fn(hash, &ctx);\n+\talgop->fn.final(hash, &ctx);\n \n \tif (binary)\n \t\tfwrite(hash, 1, algop->rawsz, stdout);\ndiff --git a/t/unit-tests/t-hash.c b/t/unit-tests/t-hash.c\nindex e62647019b..9c1c9bf70b 100644\n--- a/t/unit-tests/t-hash.c\n+++ b/t/unit-tests/t-hash.c\n@@ -15,9 +15,9 @@ static void check_hash_data(const void *data, size_t data_length,\n \t\tunsigned char hash[GIT_MAX_HEXSZ];\n \t\tconst struct git_hash_algo *algop = &hash_algos[i];\n \n-\t\talgop->init_fn(&ctx);\n-\t\talgop->update_fn(&ctx, data, data_length);\n-\t\talgop->final_fn(hash, &ctx);\n+\t\talgop->fn.init(&ctx);\n+\t\talgop->fn.update(&ctx, data, data_length);\n+\t\talgop->fn.final(hash, &ctx);\n \n \t\tif (!check_str(hash_to_hex_algop(hash, algop), expected_hashes[i - 1]))\n \t\t\ttest_msg(\"result does not match with the expected for %s\\n\", hash_algos[i].name);\ndiff --git a/trace2/tr2_sid.c b/trace2/tr2_sid.c\nindex 09c4ef0d17..a7b17abe2b 100644\n--- a/trace2/tr2_sid.c\n+++ b/trace2/tr2_sid.c\n@@ -45,9 +45,9 @@ static void tr2_sid_append_my_sid_component(void)\n \tif (xgethostname(hostname, sizeof(hostname)))\n \t\tstrbuf_add(&tr2sid_buf, \"Localhost\", 9);\n \telse {\n-\t\talgo->init_fn(&ctx);\n-\t\talgo->update_fn(&ctx, hostname, strlen(hostname));\n-\t\talgo->final_fn(hash, &ctx);\n+\t\talgo->fn.init(&ctx);\n+\t\talgo->fn.update(&ctx, hostname, strlen(hostname));\n+\t\talgo->fn.final(hash, &ctx);\n \t\thash_to_hex_algop_r(hex, hash, algo);\n \t\tstrbuf_addch(&tr2sid_buf, 'H');\n \t\tstrbuf_add(&tr2sid_buf, hex, 8);\n"},{"id":"507967","messageId":"Z0DrfjZ_ktEwwE1w@tapette.crustytoothpaste.net","threadId":"62522","inReplyTo":"20241122082523.GB1179306@coredump.intra.peff.net","subject":"Re: [PATCH 3/6] hash.h: introduce `unsafe_hash_algo()`","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2024-11-22T20:37:18Z","receivedAt":"2024-11-22T20:37:21Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On 2024-11-22 at 08:25:23, Jeff King wrote:\n> OK, here's a quick and dirty conversion that applies on top of Taylor's\n> patches 1 and 2 (and then patches 3 and on would need to adjust to the\n> new world order).\n> \n> There are quite a lot of spots that needed to be adjusted, but the most\n> part it was just s/->\\([a-z]\\)_fn/->fn.\\1/.  (I'll leave adjusting the\n> unsafe_ variants as an exercise for the reader).\n\nYeah, I saw that there were about 50 callers of the hash functions,\nwhich on one hand isn't too many to change, but on the other hand also\nsurprised me.  I don't remember there being that many callers when I did\nthe SHA-256 work, but it could be that I'm remembering poorly.\n\n> The naming convention was just what I made up, but one nice effect is\n> that it replaces \"_fn\" with \"fn.\", so the line lengths remain the same.\n> :) It does mean there's a function called \"final\", which is a keyword in\n> C++. I don't know if that matters to us or not (I feel like we were\n> trying to avoid those, but it appears as a function elsewhere, too).\n\nIt doesn't.  I think we had one contributor trying to make the code\ncompile as C++ at one point, but it definitely doesn't now.  A quick\n`git grep` show that we have lots of variables named `new` as well.  (In\naddition, the implicit cast of `void *`, such as from `malloc`, to a\npointer of any type is not allowed without a cast in C++, so that would\nneed to be handled, too.)\n\n> So I dunno. It is a one-time pain, but I think the result harmonizes the\n> type system with the concept better.\n\nYeah, I agree.  It definitely looks like it could mostly be done with a\nPerl or Ruby script, even if slightly inconvenient.  And if there is a\nconflict, it should be easy to fix up.\n-- \nbrian m. carlson (they/them or he/him)\nToronto, Ontario, CA\n"},{"id":"510189","messageId":"cover.1736363652.git.me@ttaylorr.com","threadId":"62522","inReplyTo":"cover.1732130001.git.me@ttaylorr.com","subject":"[PATCH v2 0/8] hash: introduce unsafe_hash_algo(), drop unsafe_ variants","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-01-08T19:14:29Z","receivedAt":"2025-01-08T19:14:32Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"(This series is rebased on 'master', which is 14650065b7\n(RelNotes/2.48.0: fix typos etc., 2025-01-07) at the time of writing).\n\nThe bulk of this series is unchanged since last time, but a new seventh\npatch that further hardens the hashfile_checkpoint callers on top of\nPatrick's recent series[1].\n\nThe original cover letter is as follows:\n\n------------\n\nThis series implements an idea discussed in [2] which suggests that we\nintroduce a way to access a wrapped version of a 'struct git_hash_algo'\nwhich represents the unsafe variant of that algorithm, rather than\nhaving individual unsafe_ functions (like unsafe_init_fn() versus\ninit_fn(), etc.).\n\nThis approach is relatively straightforward to implement, and removes a\nsignificant deficiency in the original implementation of\nunsafe/non-cryptographic hash functions by making it impossible to\nswitch between safe- and unsafe variants of hash functions. It also\ncleans up the sha1-unsafe test helper's implementation by removing a\nlarge number of \"if (unsafe)\"-style conditionals.\n\nThe series is laid out as follows:\n\n  * The first two patches prepare the hashfile API for the upcoming\n    change:\n\n      csum-file: store the hash algorithm as a struct field\n      csum-file.c: extract algop from hashfile_checksum_valid()\n\n  * The next patch implements the new 'unsafe_hash_algo()' function at\n    the heart of this series' approach:\n\n      hash.h: introduce `unsafe_hash_algo()`\n\n  * The next two patches convert existing callers to use the new\n    'unsafe_hash_algo()' function, instead of switching between safe and\n    unsafe_ variants of individual functions:\n\n      csum-file.c: use unsafe_hash_algo()\n      t/helper/test-hash.c: use unsafe_hash_algo()\n\n  * The final patch drops the unsafe_ function variants following all\n    callers being converted to use the new pattern:\n\n      hash.h: drop unsafe_ function variants\n\nThanks in advance for your review!\n\n[1]: https://lore.kernel.org/git/20241230-pks-meson-sha1-unsafe-v1-0-efb276e171f5@pks.im/\n[2]: https://lore.kernel.org/git/20241107013915.GA961214@coredump.intra.peff.net/\n\nTaylor Blau (8):\n  t/helper/test-tool: implement sha1-unsafe helper\n  csum-file: store the hash algorithm as a struct field\n  csum-file.c: extract algop from hashfile_checksum_valid()\n  hash.h: introduce `unsafe_hash_algo()`\n  csum-file.c: use unsafe_hash_algo()\n  t/helper/test-hash.c: use unsafe_hash_algo()\n  csum-file: introduce hashfile_checkpoint_init()\n  hash.h: drop unsafe_ function variants\n\n builtin/fast-import.c  |  2 +-\n bulk-checkin.c         |  9 ++++++---\n csum-file.c            | 40 +++++++++++++++++++++++++---------------\n csum-file.h            |  2 ++\n hash.h                 | 20 +++++---------------\n object-file.c          | 41 ++++++++++++++++++++++++++---------------\n t/helper/test-hash.c   |  4 +++-\n t/helper/test-sha1.c   |  7 ++++++-\n t/helper/test-sha1.sh  | 38 ++++++++++++++++++++++----------------\n t/helper/test-sha256.c |  2 +-\n t/helper/test-tool.c   |  1 +\n t/helper/test-tool.h   |  3 ++-\n 12 files changed, 100 insertions(+), 69 deletions(-)\n\nRange-diff against v1:\n2:  d8c1fc78b57 ! 1:  4c1523a04f1 t/helper/test-tool: implement sha1-unsafe helper\n    @@ Metadata\n      ## Commit message ##\n         t/helper/test-tool: implement sha1-unsafe helper\n     \n    -    Add a new helper similar to 't/helper/test-tool sha1' called instead\n    -    \"sha1-unsafe\" which uses the unsafe variant of Git's SHA-1 wrappers.\n    +    With the new \"unsafe\" SHA-1 build knob, it is convenient to have a\n    +    test-tool that can exercise Git's unsafe SHA-1 wrappers for testing,\n    +    similar to 't/helper/test-tool sha1'.\n     \n    -    While we're at it, modify the test-sha1.sh script to exercise both\n    -    the sha1 and sha1-unsafe test tools to ensure that both produce the\n    -    expected hash values.\n    +    Implement that helper by altering the implementation of that test-tool\n    +    (in cmd_hash_impl(), which is generic and parameterized over different\n    +    hash functions) to conditionally run the unsafe variants of the chosen\n    +    hash function, and expose the new behavior via a new 'sha1-unsafe' test\n    +    helper.\n     \n         Signed-off-by: Taylor Blau <me@ttaylorr.com>\n     \n    + ## t/helper/test-hash.c ##\n    +@@\n    + #include \"test-tool.h\"\n    + #include \"hex.h\"\n    + \n    +-int cmd_hash_impl(int ac, const char **av, int algo)\n    ++int cmd_hash_impl(int ac, const char **av, int algo, int unsafe)\n    + {\n    + \tgit_hash_ctx ctx;\n    + \tunsigned char hash[GIT_MAX_HEXSZ];\n    +@@ t/helper/test-hash.c: int cmd_hash_impl(int ac, const char **av, int algo)\n    + \t\t\tdie(\"OOPS\");\n    + \t}\n    + \n    +-\talgop->init_fn(&ctx);\n    ++\tif (unsafe)\n    ++\t\talgop->unsafe_init_fn(&ctx);\n    ++\telse\n    ++\t\talgop->init_fn(&ctx);\n    + \n    + \twhile (1) {\n    + \t\tssize_t sz, this_sz;\n    +@@ t/helper/test-hash.c: int cmd_hash_impl(int ac, const char **av, int algo)\n    + \t\t}\n    + \t\tif (this_sz == 0)\n    + \t\t\tbreak;\n    +-\t\talgop->update_fn(&ctx, buffer, this_sz);\n    ++\t\tif (unsafe)\n    ++\t\t\talgop->unsafe_update_fn(&ctx, buffer, this_sz);\n    ++\t\telse\n    ++\t\t\talgop->update_fn(&ctx, buffer, this_sz);\n    + \t}\n    +-\talgop->final_fn(hash, &ctx);\n    ++\tif (unsafe)\n    ++\t\talgop->unsafe_final_fn(hash, &ctx);\n    ++\telse\n    ++\t\talgop->final_fn(hash, &ctx);\n    + \n    + \tif (binary)\n    + \t\tfwrite(hash, 1, algop->rawsz, stdout);\n    +\n      ## t/helper/test-sha1.c ##\n    +@@\n    + \n    + int cmd__sha1(int ac, const char **av)\n    + {\n    +-\treturn cmd_hash_impl(ac, av, GIT_HASH_SHA1);\n    ++\treturn cmd_hash_impl(ac, av, GIT_HASH_SHA1, 0);\n    + }\n    + \n    + int cmd__sha1_is_sha1dc(int argc UNUSED, const char **argv UNUSED)\n     @@ t/helper/test-sha1.c: int cmd__sha1_is_sha1dc(int argc UNUSED, const char **argv UNUSED)\n      #endif\n      \treturn 1;\n    @@ t/helper/test-sha1.sh\n      da39a3ee5e6b4b0d3255bfef95601890afd80709 0\n      3f786850e387550fdab836ed7e6dc881de23001b 0 a\n     \n    + ## t/helper/test-sha256.c ##\n    +@@\n    + \n    + int cmd__sha256(int ac, const char **av)\n    + {\n    +-\treturn cmd_hash_impl(ac, av, GIT_HASH_SHA256);\n    ++\treturn cmd_hash_impl(ac, av, GIT_HASH_SHA256, 0);\n    + }\n    +\n      ## t/helper/test-tool.c ##\n     @@ t/helper/test-tool.c: static struct test_cmd cmds[] = {\n      \t{ \"serve-v2\", cmd__serve_v2 },\n    @@ t/helper/test-tool.h: int cmd__scrap_cache_tree(int argc, const char **argv);\n      int cmd__sha256(int argc, const char **argv);\n      int cmd__sigchain(int argc, const char **argv);\n      int cmd__simple_ipc(int argc, const char **argv);\n    +@@ t/helper/test-tool.h: int cmd__windows_named_pipe(int argc, const char **argv);\n    + #endif\n    + int cmd__write_cache(int argc, const char **argv);\n    + \n    +-int cmd_hash_impl(int ac, const char **av, int algo);\n    ++int cmd_hash_impl(int ac, const char **av, int algo, int unsafe);\n    + \n    + #endif\n3:  380133a1142 = 2:  99cc44895b5 csum-file: store the hash algorithm as a struct field\n4:  e5076f003bf = 3:  1ffab2f8289 csum-file.c: extract algop from hashfile_checksum_valid()\n5:  17f92dba34b = 4:  99dcbe2e716 hash.h: introduce `unsafe_hash_algo()`\n6:  0d8e12599e2 = 5:  2dcc2aa6803 csum-file.c: use unsafe_hash_algo()\n7:  a49a41703e2 = 6:  a2b9ef03080 t/helper/test-hash.c: use unsafe_hash_algo()\n1:  0e2fcee6894 ! 7:  94c07fd8a55 t/helper/test-sha1: prepare for an unsafe mode\n    @@ Metadata\n     Author: Taylor Blau <me@ttaylorr.com>\n     \n      ## Commit message ##\n    -    t/helper/test-sha1: prepare for an unsafe mode\n    +    csum-file: introduce hashfile_checkpoint_init()\n     \n    -    With the new \"unsafe\" SHA-1 build knob, it would be convenient to have\n    -    a test-tool that can exercise Git's unsafe SHA-1 wrappers for testing,\n    -    similar to 't/helper/test-tool sha1'.\n    +    In 106140a99f (builtin/fast-import: fix segfault with unsafe SHA1\n    +    backend, 2024-12-30) and 9218c0bfe1 (bulk-checkin: fix segfault with\n    +    unsafe SHA1 backend, 2024-12-30), we observed the effects of failing to\n    +    initialize a hashfile_checkpoint with the same hash function\n    +    implementation as is used by the hashfile it is used to checkpoint.\n     \n    -    Prepare for such a helper by altering the implementation of that\n    -    test-tool (in cmd_hash_impl(), which is generic and parameterized over\n    -    different hash functions) to conditionally run the unsafe variants of\n    -    the chosen hash function.\n    +    While both 106140a99f and 9218c0bfe1 work around the immediate crash,\n    +    changing the hash function implementation within the hashfile API to,\n    +    for example, the non-unsafe variant would re-introduce the crash. This\n    +    is a result of the tight coupling between initializing hashfiles and\n    +    hashfile_checkpoints.\n     \n    -    The following commit will add a new test-tool which makes use of this\n    -    new parameter.\n    +    Introduce and use a new function which ensures that both parts of a\n    +    hashfile and hashfile_checkpoint pair use the same hash function\n    +    implementation to avoid such crashes.\n     \n    +    A few things worth noting:\n    +\n    +      - In the change to builtin/fast-import.c::stream_blob(), we can see\n    +        that by removing the explicit reference to\n    +        'the_hash_algo->unsafe_init_fn()', we are hardened against the\n    +        hashfile API changing away from the_hash_algo (or its unsafe\n    +        variant) in the future.\n    +\n    +      - The bulk-checkin code no longer needs to explicitly zero-initialize\n    +        the hashfile_checkpoint, since it is now done as a result of calling\n    +        'hashfile_checkpoint_init()'.\n    +\n    +      - Also in the bulk-checkin code, we add an additional call to\n    +        prepare_to_stream() outside of the main loop in order to initialize\n    +        'state->f' so we know which hash function implementation to use when\n    +        calling 'hashfile_checkpoint_init()'.\n    +\n    +        This is OK, since subsequent 'prepare_to_stream()' calls are noops.\n    +        However, we only need to call 'prepare_to_stream()' when we have the\n    +        HASH_WRITE_OBJECT bit set in our flags. Without that bit, calling\n    +        'prepare_to_stream()' does not assign 'state->f', so we have nothing\n    +        to initialize.\n    +\n    +      - Other uses of the 'checkpoint' in 'deflate_blob_to_pack()' are\n    +        appropriately guarded.\n    +\n    +    Helped-by: Patrick Steinhardt <ps@pks.im>\n         Signed-off-by: Taylor Blau <me@ttaylorr.com>\n     \n    - ## t/helper/test-hash.c ##\n    -@@\n    - #include \"test-tool.h\"\n    - #include \"hex.h\"\n    + ## builtin/fast-import.c ##\n    +@@ builtin/fast-import.c: static void stream_blob(uintmax_t len, struct object_id *oidout, uintmax_t mark)\n    + \t\t|| (pack_size + PACK_SIZE_THRESHOLD + len) < pack_size)\n    + \t\tcycle_packfile();\n      \n    --int cmd_hash_impl(int ac, const char **av, int algo)\n    -+int cmd_hash_impl(int ac, const char **av, int algo, int unsafe)\n    - {\n    +-\tthe_hash_algo->unsafe_init_fn(&checkpoint.ctx);\n    ++\thashfile_checkpoint_init(pack_file, &checkpoint);\n    + \thashfile_checkpoint(pack_file, &checkpoint);\n    + \toffset = checkpoint.offset;\n    + \n    +\n    + ## bulk-checkin.c ##\n    +@@ bulk-checkin.c: static int deflate_blob_to_pack(struct bulk_checkin_packfile *state,\n      \tgit_hash_ctx ctx;\n    - \tunsigned char hash[GIT_MAX_HEXSZ];\n    -@@ t/helper/test-hash.c: int cmd_hash_impl(int ac, const char **av, int algo)\n    - \t\t\tdie(\"OOPS\");\n    - \t}\n    + \tunsigned char obuf[16384];\n    + \tunsigned header_len;\n    +-\tstruct hashfile_checkpoint checkpoint = {0};\n    ++\tstruct hashfile_checkpoint checkpoint;\n    + \tstruct pack_idx_entry *idx = NULL;\n      \n    --\talgop->init_fn(&ctx);\n    -+\tif (unsafe)\n    -+\t\talgop->unsafe_init_fn(&ctx);\n    -+\telse\n    -+\t\talgop->init_fn(&ctx);\n    + \tseekback = lseek(fd, 0, SEEK_CUR);\n    +@@ bulk-checkin.c: static int deflate_blob_to_pack(struct bulk_checkin_packfile *state,\n    + \t\t\t\t\t  OBJ_BLOB, size);\n    + \tthe_hash_algo->init_fn(&ctx);\n    + \tthe_hash_algo->update_fn(&ctx, obuf, header_len);\n    +-\tthe_hash_algo->unsafe_init_fn(&checkpoint.ctx);\n    + \n    + \t/* Note: idx is non-NULL when we are writing */\n    +-\tif ((flags & HASH_WRITE_OBJECT) != 0)\n    ++\tif ((flags & HASH_WRITE_OBJECT) != 0) {\n    + \t\tCALLOC_ARRAY(idx, 1);\n    + \n    ++\t\tprepare_to_stream(state, flags);\n    ++\t\thashfile_checkpoint_init(state->f, &checkpoint);\n    ++\t}\n    ++\n    + \talready_hashed_to = 0;\n      \n      \twhile (1) {\n    - \t\tssize_t sz, this_sz;\n    -@@ t/helper/test-hash.c: int cmd_hash_impl(int ac, const char **av, int algo)\n    - \t\t}\n    - \t\tif (this_sz == 0)\n    - \t\t\tbreak;\n    --\t\talgop->update_fn(&ctx, buffer, this_sz);\n    -+\t\tif (unsafe)\n    -+\t\t\talgop->unsafe_update_fn(&ctx, buffer, this_sz);\n    -+\t\telse\n    -+\t\t\talgop->update_fn(&ctx, buffer, this_sz);\n    - \t}\n    --\talgop->final_fn(hash, &ctx);\n    -+\tif (unsafe)\n    -+\t\talgop->unsafe_final_fn(hash, &ctx);\n    -+\telse\n    -+\t\talgop->final_fn(hash, &ctx);\n    - \n    - \tif (binary)\n    - \t\tfwrite(hash, 1, algop->rawsz, stdout);\n     \n    - ## t/helper/test-sha1.c ##\n    -@@\n    - \n    - int cmd__sha1(int ac, const char **av)\n    - {\n    --\treturn cmd_hash_impl(ac, av, GIT_HASH_SHA1);\n    -+\treturn cmd_hash_impl(ac, av, GIT_HASH_SHA1, 0);\n    + ## csum-file.c ##\n    +@@ csum-file.c: struct hashfile *hashfd_throughput(int fd, const char *name, struct progress *tp\n    + \treturn hashfd_internal(fd, name, tp, 8 * 1024);\n      }\n      \n    - int cmd__sha1_is_sha1dc(int argc UNUSED, const char **argv UNUSED)\n    -\n    - ## t/helper/test-sha256.c ##\n    -@@\n    - \n    - int cmd__sha256(int ac, const char **av)\n    ++void hashfile_checkpoint_init(struct hashfile *f,\n    ++\t\t\t      struct hashfile_checkpoint *checkpoint)\n    ++{\n    ++\tmemset(checkpoint, 0, sizeof(*checkpoint));\n    ++\tf->algop->init_fn(&checkpoint->ctx);\n    ++}\n    ++\n    + void hashfile_checkpoint(struct hashfile *f, struct hashfile_checkpoint *checkpoint)\n      {\n    --\treturn cmd_hash_impl(ac, av, GIT_HASH_SHA256);\n    -+\treturn cmd_hash_impl(ac, av, GIT_HASH_SHA256, 0);\n    - }\n    + \thashflush(f);\n     \n    - ## t/helper/test-tool.h ##\n    -@@ t/helper/test-tool.h: int cmd__windows_named_pipe(int argc, const char **argv);\n    - #endif\n    - int cmd__write_cache(int argc, const char **argv);\n    + ## csum-file.h ##\n    +@@ csum-file.h: struct hashfile_checkpoint {\n    + \tgit_hash_ctx ctx;\n    + };\n      \n    --int cmd_hash_impl(int ac, const char **av, int algo);\n    -+int cmd_hash_impl(int ac, const char **av, int algo, int unsafe);\n    ++void hashfile_checkpoint_init(struct hashfile *, struct hashfile_checkpoint *);\n    + void hashfile_checkpoint(struct hashfile *, struct hashfile_checkpoint *);\n    + int hashfile_truncate(struct hashfile *, struct hashfile_checkpoint *);\n      \n    - #endif\n8:  4081ad08549 = 8:  f5579883816 hash.h: drop unsafe_ function variants\n\nbase-commit: 14650065b76b28d3cfa9453356ac5669b19e706e\n-- \n2.48.0.rc2.33.gaab3d23ed4c\n"},{"id":"510190","messageId":"4c1523a04f1b4750da38f5e2149109753d84022b.1736363652.git.me@ttaylorr.com","threadId":"62522","inReplyTo":"cover.1736363652.git.me@ttaylorr.com","subject":"[PATCH v2 1/8] t/helper/test-tool: implement sha1-unsafe helper","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-01-08T19:14:32Z","receivedAt":"2025-01-08T19:14:35Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"With the new \"unsafe\" SHA-1 build knob, it is convenient to have a\ntest-tool that can exercise Git's unsafe SHA-1 wrappers for testing,\nsimilar to 't/helper/test-tool sha1'.\n\nImplement that helper by altering the implementation of that test-tool\n(in cmd_hash_impl(), which is generic and parameterized over different\nhash functions) to conditionally run the unsafe variants of the chosen\nhash function, and expose the new behavior via a new 'sha1-unsafe' test\nhelper.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n t/helper/test-hash.c   | 17 +++++++++++++----\n t/helper/test-sha1.c   |  7 ++++++-\n t/helper/test-sha1.sh  | 38 ++++++++++++++++++++++----------------\n t/helper/test-sha256.c |  2 +-\n t/helper/test-tool.c   |  1 +\n t/helper/test-tool.h   |  3 ++-\n 6 files changed, 45 insertions(+), 23 deletions(-)\n\ndiff --git a/t/helper/test-hash.c b/t/helper/test-hash.c\nindex 45d829c908f..d0ee668df95 100644\n--- a/t/helper/test-hash.c\n+++ b/t/helper/test-hash.c\n@@ -1,7 +1,7 @@\n #include \"test-tool.h\"\n #include \"hex.h\"\n \n-int cmd_hash_impl(int ac, const char **av, int algo)\n+int cmd_hash_impl(int ac, const char **av, int algo, int unsafe)\n {\n \tgit_hash_ctx ctx;\n \tunsigned char hash[GIT_MAX_HEXSZ];\n@@ -27,7 +27,10 @@ int cmd_hash_impl(int ac, const char **av, int algo)\n \t\t\tdie(\"OOPS\");\n \t}\n \n-\talgop->init_fn(&ctx);\n+\tif (unsafe)\n+\t\talgop->unsafe_init_fn(&ctx);\n+\telse\n+\t\talgop->init_fn(&ctx);\n \n \twhile (1) {\n \t\tssize_t sz, this_sz;\n@@ -46,9 +49,15 @@ int cmd_hash_impl(int ac, const char **av, int algo)\n \t\t}\n \t\tif (this_sz == 0)\n \t\t\tbreak;\n-\t\talgop->update_fn(&ctx, buffer, this_sz);\n+\t\tif (unsafe)\n+\t\t\talgop->unsafe_update_fn(&ctx, buffer, this_sz);\n+\t\telse\n+\t\t\talgop->update_fn(&ctx, buffer, this_sz);\n \t}\n-\talgop->final_fn(hash, &ctx);\n+\tif (unsafe)\n+\t\talgop->unsafe_final_fn(hash, &ctx);\n+\telse\n+\t\talgop->final_fn(hash, &ctx);\n \n \tif (binary)\n \t\tfwrite(hash, 1, algop->rawsz, stdout);\ndiff --git a/t/helper/test-sha1.c b/t/helper/test-sha1.c\nindex e60d000c039..349540c4df8 100644\n--- a/t/helper/test-sha1.c\n+++ b/t/helper/test-sha1.c\n@@ -3,7 +3,7 @@\n \n int cmd__sha1(int ac, const char **av)\n {\n-\treturn cmd_hash_impl(ac, av, GIT_HASH_SHA1);\n+\treturn cmd_hash_impl(ac, av, GIT_HASH_SHA1, 0);\n }\n \n int cmd__sha1_is_sha1dc(int argc UNUSED, const char **argv UNUSED)\n@@ -13,3 +13,8 @@ int cmd__sha1_is_sha1dc(int argc UNUSED, const char **argv UNUSED)\n #endif\n \treturn 1;\n }\n+\n+int cmd__sha1_unsafe(int ac, const char **av)\n+{\n+\treturn cmd_hash_impl(ac, av, GIT_HASH_SHA1, 1);\n+}\ndiff --git a/t/helper/test-sha1.sh b/t/helper/test-sha1.sh\nindex 84594885c70..bf387d3db14 100755\n--- a/t/helper/test-sha1.sh\n+++ b/t/helper/test-sha1.sh\n@@ -3,25 +3,31 @@\n dd if=/dev/zero bs=1048576 count=100 2>/dev/null |\n /usr/bin/time t/helper/test-tool sha1 >/dev/null\n \n+dd if=/dev/zero bs=1048576 count=100 2>/dev/null |\n+/usr/bin/time t/helper/test-tool sha1-unsafe >/dev/null\n+\n while read expect cnt pfx\n do\n \tcase \"$expect\" in '#'*) continue ;; esac\n-\tactual=$(\n-\t\t{\n-\t\t\ttest -z \"$pfx\" || echo \"$pfx\"\n-\t\t\tdd if=/dev/zero bs=1048576 count=$cnt 2>/dev/null |\n-\t\t\tperl -pe 'y/\\000/g/'\n-\t\t} | ./t/helper/test-tool sha1 $cnt\n-\t)\n-\tif test \"$expect\" = \"$actual\"\n-\tthen\n-\t\techo \"OK: $expect $cnt $pfx\"\n-\telse\n-\t\techo >&2 \"OOPS: $cnt\"\n-\t\techo >&2 \"expect: $expect\"\n-\t\techo >&2 \"actual: $actual\"\n-\t\texit 1\n-\tfi\n+\tfor sha1 in sha1 sha1-unsafe\n+\tdo\n+\t\tactual=$(\n+\t\t\t{\n+\t\t\t\ttest -z \"$pfx\" || echo \"$pfx\"\n+\t\t\t\tdd if=/dev/zero bs=1048576 count=$cnt 2>/dev/null |\n+\t\t\t\tperl -pe 'y/\\000/g/'\n+\t\t\t} | ./t/helper/test-tool $sha1 $cnt\n+\t\t)\n+\t\tif test \"$expect\" = \"$actual\"\n+\t\tthen\n+\t\t\techo \"OK ($sha1): $expect $cnt $pfx\"\n+\t\telse\n+\t\t\techo >&2 \"OOPS ($sha1): $cnt\"\n+\t\t\techo >&2 \"expect ($sha1): $expect\"\n+\t\t\techo >&2 \"actual ($sha1): $actual\"\n+\t\t\texit 1\n+\t\tfi\n+\tdone\n done <<EOF\n da39a3ee5e6b4b0d3255bfef95601890afd80709 0\n 3f786850e387550fdab836ed7e6dc881de23001b 0 a\ndiff --git a/t/helper/test-sha256.c b/t/helper/test-sha256.c\nindex 2fb20438f3c..7fd0aa1fcd3 100644\n--- a/t/helper/test-sha256.c\n+++ b/t/helper/test-sha256.c\n@@ -3,5 +3,5 @@\n \n int cmd__sha256(int ac, const char **av)\n {\n-\treturn cmd_hash_impl(ac, av, GIT_HASH_SHA256);\n+\treturn cmd_hash_impl(ac, av, GIT_HASH_SHA256, 0);\n }\ndiff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\nindex 4a7aa993ba9..958452ef12e 100644\n--- a/t/helper/test-tool.c\n+++ b/t/helper/test-tool.c\n@@ -70,6 +70,7 @@ static struct test_cmd cmds[] = {\n \t{ \"serve-v2\", cmd__serve_v2 },\n \t{ \"sha1\", cmd__sha1 },\n \t{ \"sha1-is-sha1dc\", cmd__sha1_is_sha1dc },\n+\t{ \"sha1-unsafe\", cmd__sha1_unsafe },\n \t{ \"sha256\", cmd__sha256 },\n \t{ \"sigchain\", cmd__sigchain },\n \t{ \"simple-ipc\", cmd__simple_ipc },\ndiff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\nindex 21802ac27da..24149edd414 100644\n--- a/t/helper/test-tool.h\n+++ b/t/helper/test-tool.h\n@@ -63,6 +63,7 @@ int cmd__scrap_cache_tree(int argc, const char **argv);\n int cmd__serve_v2(int argc, const char **argv);\n int cmd__sha1(int argc, const char **argv);\n int cmd__sha1_is_sha1dc(int argc, const char **argv);\n+int cmd__sha1_unsafe(int argc, const char **argv);\n int cmd__sha256(int argc, const char **argv);\n int cmd__sigchain(int argc, const char **argv);\n int cmd__simple_ipc(int argc, const char **argv);\n@@ -81,6 +82,6 @@ int cmd__windows_named_pipe(int argc, const char **argv);\n #endif\n int cmd__write_cache(int argc, const char **argv);\n \n-int cmd_hash_impl(int ac, const char **av, int algo);\n+int cmd_hash_impl(int ac, const char **av, int algo, int unsafe);\n \n #endif\n-- \n2.48.0.rc2.33.gaab3d23ed4c\n\n"},{"id":"510191","messageId":"99cc44895b57cc75fc8f447000817b3595368e4c.1736363652.git.me@ttaylorr.com","threadId":"62522","inReplyTo":"cover.1736363652.git.me@ttaylorr.com","subject":"[PATCH v2 2/8] csum-file: store the hash algorithm as a struct field","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-01-08T19:14:35Z","receivedAt":"2025-01-08T19:14:38Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Throughout the hashfile API, we rely on a reference to 'the_hash_algo',\nand call its _usnafe function variants directly.\n\nPrepare for a future change where we may use a different 'git_hash_algo'\npointer (instead of just relying on 'the_hash_algo' throughout) by\nmaking the 'git_hash_algo' pointer a member of the 'hashfile' structure\nitself.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n csum-file.c | 20 +++++++++++---------\n csum-file.h |  1 +\n 2 files changed, 12 insertions(+), 9 deletions(-)\n\ndiff --git a/csum-file.c b/csum-file.c\nindex 5716016e12e..b28cd047e3f 100644\n--- a/csum-file.c\n+++ b/csum-file.c\n@@ -50,7 +50,7 @@ void hashflush(struct hashfile *f)\n \n \tif (offset) {\n \t\tif (!f->skip_hash)\n-\t\t\tthe_hash_algo->unsafe_update_fn(&f->ctx, f->buffer, offset);\n+\t\t\tf->algop->unsafe_update_fn(&f->ctx, f->buffer, offset);\n \t\tflush(f, f->buffer, offset);\n \t\tf->offset = 0;\n \t}\n@@ -71,14 +71,14 @@ int finalize_hashfile(struct hashfile *f, unsigned char *result,\n \thashflush(f);\n \n \tif (f->skip_hash)\n-\t\thashclr(f->buffer, the_repository->hash_algo);\n+\t\thashclr(f->buffer, f->algop);\n \telse\n-\t\tthe_hash_algo->unsafe_final_fn(f->buffer, &f->ctx);\n+\t\tf->algop->unsafe_final_fn(f->buffer, &f->ctx);\n \n \tif (result)\n-\t\thashcpy(result, f->buffer, the_repository->hash_algo);\n+\t\thashcpy(result, f->buffer, f->algop);\n \tif (flags & CSUM_HASH_IN_STREAM)\n-\t\tflush(f, f->buffer, the_hash_algo->rawsz);\n+\t\tflush(f, f->buffer, f->algop->rawsz);\n \tif (flags & CSUM_FSYNC)\n \t\tfsync_component_or_die(component, f->fd, f->name);\n \tif (flags & CSUM_CLOSE) {\n@@ -128,7 +128,7 @@ void hashwrite(struct hashfile *f, const void *buf, unsigned int count)\n \t\t\t * f->offset is necessarily zero.\n \t\t\t */\n \t\t\tif (!f->skip_hash)\n-\t\t\t\tthe_hash_algo->unsafe_update_fn(&f->ctx, buf, nr);\n+\t\t\t\tf->algop->unsafe_update_fn(&f->ctx, buf, nr);\n \t\t\tflush(f, buf, nr);\n \t\t} else {\n \t\t\t/*\n@@ -174,7 +174,9 @@ static struct hashfile *hashfd_internal(int fd, const char *name,\n \tf->name = name;\n \tf->do_crc = 0;\n \tf->skip_hash = 0;\n-\tthe_hash_algo->unsafe_init_fn(&f->ctx);\n+\n+\tf->algop = the_hash_algo;\n+\tf->algop->unsafe_init_fn(&f->ctx);\n \n \tf->buffer_len = buffer_len;\n \tf->buffer = xmalloc(buffer_len);\n@@ -208,7 +210,7 @@ void hashfile_checkpoint(struct hashfile *f, struct hashfile_checkpoint *checkpo\n {\n \thashflush(f);\n \tcheckpoint->offset = f->total;\n-\tthe_hash_algo->unsafe_clone_fn(&checkpoint->ctx, &f->ctx);\n+\tf->algop->unsafe_clone_fn(&checkpoint->ctx, &f->ctx);\n }\n \n int hashfile_truncate(struct hashfile *f, struct hashfile_checkpoint *checkpoint)\n@@ -219,7 +221,7 @@ int hashfile_truncate(struct hashfile *f, struct hashfile_checkpoint *checkpoint\n \t    lseek(f->fd, offset, SEEK_SET) != offset)\n \t\treturn -1;\n \tf->total = offset;\n-\tthe_hash_algo->unsafe_clone_fn(&f->ctx, &checkpoint->ctx);\n+\tf->algop->unsafe_clone_fn(&f->ctx, &checkpoint->ctx);\n \tf->offset = 0; /* hashflush() was called in checkpoint */\n \treturn 0;\n }\ndiff --git a/csum-file.h b/csum-file.h\nindex 7c73da0a40a..2b45f4673a2 100644\n--- a/csum-file.h\n+++ b/csum-file.h\n@@ -20,6 +20,7 @@ struct hashfile {\n \tsize_t buffer_len;\n \tunsigned char *buffer;\n \tunsigned char *check_buffer;\n+\tconst struct git_hash_algo *algop;\n \n \t/**\n \t * If non-zero, skip_hash indicates that we should\n-- \n2.48.0.rc2.33.gaab3d23ed4c\n\n"},{"id":"510192","messageId":"1ffab2f82891af8d8644443d75bd6f29bcbd6906.1736363652.git.me@ttaylorr.com","threadId":"62522","inReplyTo":"cover.1736363652.git.me@ttaylorr.com","subject":"[PATCH v2 3/8] csum-file.c: extract algop from hashfile_checksum_valid()","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-01-08T19:14:39Z","receivedAt":"2025-01-08T19:14:41Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Perform a similar transformation as in the previous commit, but focused\ninstead on hashfile_checksum_valid(). This function does not work with a\nhashfile structure itself, and instead validates the raw contents of a\nfile written using the hashfile API.\n\nWe'll want to be prepared for a similar change to this function in the\nfuture, so prepare ourselves for that by extracting 'the_hash_algo' into\nits own field for use within this function.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n csum-file.c | 13 +++++++------\n 1 file changed, 7 insertions(+), 6 deletions(-)\n\ndiff --git a/csum-file.c b/csum-file.c\nindex b28cd047e3f..7a71121e340 100644\n--- a/csum-file.c\n+++ b/csum-file.c\n@@ -242,14 +242,15 @@ int hashfile_checksum_valid(const unsigned char *data, size_t total_len)\n {\n \tunsigned char got[GIT_MAX_RAWSZ];\n \tgit_hash_ctx ctx;\n-\tsize_t data_len = total_len - the_hash_algo->rawsz;\n+\tconst struct git_hash_algo *algop = the_hash_algo;\n+\tsize_t data_len = total_len - algop->rawsz;\n \n-\tif (total_len < the_hash_algo->rawsz)\n+\tif (total_len < algop->rawsz)\n \t\treturn 0; /* say \"too short\"? */\n \n-\tthe_hash_algo->unsafe_init_fn(&ctx);\n-\tthe_hash_algo->unsafe_update_fn(&ctx, data, data_len);\n-\tthe_hash_algo->unsafe_final_fn(got, &ctx);\n+\talgop->unsafe_init_fn(&ctx);\n+\talgop->unsafe_update_fn(&ctx, data, data_len);\n+\talgop->unsafe_final_fn(got, &ctx);\n \n-\treturn hasheq(got, data + data_len, the_repository->hash_algo);\n+\treturn hasheq(got, data + data_len, algop);\n }\n-- \n2.48.0.rc2.33.gaab3d23ed4c\n\n"},{"id":"510193","messageId":"99dcbe2e7165d96e3a9c025540995a75f74b2489.1736363652.git.me@ttaylorr.com","threadId":"62522","inReplyTo":"cover.1736363652.git.me@ttaylorr.com","subject":"[PATCH v2 4/8] hash.h: introduce `unsafe_hash_algo()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-01-08T19:14:42Z","receivedAt":"2025-01-08T19:14:44Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In 253ed9ecff (hash.h: scaffolding for _unsafe hashing variants,\n2024-09-26), we introduced \"unsafe\" variants of the SHA-1 hashing\nfunctions by introducing new functions like \"unsafe_init_fn()\" and so\non.\n\nThis approach has a major shortcoming that callers must remember to\nconsistently use one variant or the other. Failing to consistently use\n(or not use) the unsafe variants can lead to crashes at best, or subtle\nmemory corruption issues at worst.\n\nIn the hashfile API, this isn't difficult to achieve, but verifying that\nall callers consistently use the unsafe variants is somewhat of a chore\ngiven how spread out all of the callers are. In the sha1 and sha1-unsafe\ntest helpers, all of the calls to various hash functions are guarded by\nan \"if (unsafe)\" conditional, which is repetitive and cumbersome.\n\nAddress these issues by introducing a new pattern whereby one\n'git_hash_algo' can return a pointer to another 'git_hash_algo' that\nrepresents the unsafe version of itself. So instead of having something\nlike:\n\n    if (unsafe)\n      the_hash_algo->init_fn(...);\n      the_hash_algo->update_fn(...);\n      the_hash_algo->final_fn(...);\n    else\n      the_hash_algo->unsafe_init_fn(...);\n      the_hash_algo->unsafe_update_fn(...);\n      the_hash_algo->unsafe_final_fn(...);\n\nwe can instead write:\n\n    struct git_hash_algo *algop = the_hash_algo;\n    if (unsafe)\n      algop = unsafe_hash_algo(algop);\n\n    the_hash_algo->init_fn(...);\n    the_hash_algo->update_fn(...);\n    the_hash_algo->final_fn(...);\n\nThis removes the existing shortcoming by no longer forcing the caller to\n\"remember\" which variant of the hash functions it wants to call, only to\nhold onto a 'struct git_hash_algo' pointer that is initialized once.\n\nSimilarly, while there currently is still a way to \"mix\" safe and unsafe\nfunctions, this too will go away after subsequent commits remove all\ndirect calls to the unsafe_ variants.\n\nSuggested-by: Jeff King <peff@peff.net>\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n hash.h        |  5 +++++\n object-file.c | 26 ++++++++++++++++++++++++++\n 2 files changed, 31 insertions(+)\n\ndiff --git a/hash.h b/hash.h\nindex 756166ce5e8..23cf6876e50 100644\n--- a/hash.h\n+++ b/hash.h\n@@ -305,6 +305,9 @@ struct git_hash_algo {\n \n \t/* The all-zeros OID. */\n \tconst struct object_id *null_oid;\n+\n+\t/* The unsafe variant of this hash function, if one exists. */\n+\tconst struct git_hash_algo *unsafe;\n };\n extern const struct git_hash_algo hash_algos[GIT_HASH_NALGOS];\n \n@@ -323,6 +326,8 @@ static inline int hash_algo_by_ptr(const struct git_hash_algo *p)\n \treturn p - hash_algos;\n }\n \n+const struct git_hash_algo *unsafe_hash_algo(const struct git_hash_algo *algop);\n+\n const struct object_id *null_oid(void);\n \n static inline int hashcmp(const unsigned char *sha1, const unsigned char *sha2, const struct git_hash_algo *algop)\ndiff --git a/object-file.c b/object-file.c\nindex 5b792b3dd42..43efa4ca361 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -202,6 +202,22 @@ static void git_hash_unknown_final_oid(struct object_id *oid UNUSED,\n \tBUG(\"trying to finalize unknown hash\");\n }\n \n+static const struct git_hash_algo sha1_unsafe_algo = {\n+\t.name = \"sha1\",\n+\t.format_id = GIT_SHA1_FORMAT_ID,\n+\t.rawsz = GIT_SHA1_RAWSZ,\n+\t.hexsz = GIT_SHA1_HEXSZ,\n+\t.blksz = GIT_SHA1_BLKSZ,\n+\t.init_fn = git_hash_sha1_init_unsafe,\n+\t.clone_fn = git_hash_sha1_clone_unsafe,\n+\t.update_fn = git_hash_sha1_update_unsafe,\n+\t.final_fn = git_hash_sha1_final_unsafe,\n+\t.final_oid_fn = git_hash_sha1_final_oid_unsafe,\n+\t.empty_tree = &empty_tree_oid,\n+\t.empty_blob = &empty_blob_oid,\n+\t.null_oid = &null_oid_sha1,\n+};\n+\n const struct git_hash_algo hash_algos[GIT_HASH_NALGOS] = {\n \t{\n \t\t.name = NULL,\n@@ -239,6 +255,7 @@ const struct git_hash_algo hash_algos[GIT_HASH_NALGOS] = {\n \t\t.unsafe_update_fn = git_hash_sha1_update_unsafe,\n \t\t.unsafe_final_fn = git_hash_sha1_final_unsafe,\n \t\t.unsafe_final_oid_fn = git_hash_sha1_final_oid_unsafe,\n+\t\t.unsafe = &sha1_unsafe_algo,\n \t\t.empty_tree = &empty_tree_oid,\n \t\t.empty_blob = &empty_blob_oid,\n \t\t.null_oid = &null_oid_sha1,\n@@ -305,6 +322,15 @@ int hash_algo_by_length(int len)\n \treturn GIT_HASH_UNKNOWN;\n }\n \n+const struct git_hash_algo *unsafe_hash_algo(const struct git_hash_algo *algop)\n+{\n+\t/* If we have a faster \"unsafe\" implementation, use that. */\n+\tif (algop->unsafe)\n+\t\treturn algop->unsafe;\n+\t/* Otherwise use the default one. */\n+\treturn algop;\n+}\n+\n /*\n  * This is meant to hold a *small* number of objects that you would\n  * want repo_read_object_file() to be able to return, but yet you do not want\n-- \n2.48.0.rc2.33.gaab3d23ed4c\n\n"},{"id":"510194","messageId":"2dcc2aa6803cdaad762ec82cd31fc14dd6aca414.1736363652.git.me@ttaylorr.com","threadId":"62522","inReplyTo":"cover.1736363652.git.me@ttaylorr.com","subject":"[PATCH v2 5/8] csum-file.c: use unsafe_hash_algo()","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-01-08T19:14:45Z","receivedAt":"2025-01-08T19:14:48Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Instead of calling the unsafe_ hash function variants directly, make use\nof the shared 'algop' pointer by initializing it to:\n\n    f->algop = unsafe_hash_algo(the_hash_algo);\n\n, thus making all calls use the unsafe variants directly.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n csum-file.c | 22 +++++++++++-----------\n 1 file changed, 11 insertions(+), 11 deletions(-)\n\ndiff --git a/csum-file.c b/csum-file.c\nindex 7a71121e340..ebffc80ef71 100644\n--- a/csum-file.c\n+++ b/csum-file.c\n@@ -50,7 +50,7 @@ void hashflush(struct hashfile *f)\n \n \tif (offset) {\n \t\tif (!f->skip_hash)\n-\t\t\tf->algop->unsafe_update_fn(&f->ctx, f->buffer, offset);\n+\t\t\tf->algop->update_fn(&f->ctx, f->buffer, offset);\n \t\tflush(f, f->buffer, offset);\n \t\tf->offset = 0;\n \t}\n@@ -73,7 +73,7 @@ int finalize_hashfile(struct hashfile *f, unsigned char *result,\n \tif (f->skip_hash)\n \t\thashclr(f->buffer, f->algop);\n \telse\n-\t\tf->algop->unsafe_final_fn(f->buffer, &f->ctx);\n+\t\tf->algop->final_fn(f->buffer, &f->ctx);\n \n \tif (result)\n \t\thashcpy(result, f->buffer, f->algop);\n@@ -128,7 +128,7 @@ void hashwrite(struct hashfile *f, const void *buf, unsigned int count)\n \t\t\t * f->offset is necessarily zero.\n \t\t\t */\n \t\t\tif (!f->skip_hash)\n-\t\t\t\tf->algop->unsafe_update_fn(&f->ctx, buf, nr);\n+\t\t\t\tf->algop->update_fn(&f->ctx, buf, nr);\n \t\t\tflush(f, buf, nr);\n \t\t} else {\n \t\t\t/*\n@@ -175,8 +175,8 @@ static struct hashfile *hashfd_internal(int fd, const char *name,\n \tf->do_crc = 0;\n \tf->skip_hash = 0;\n \n-\tf->algop = the_hash_algo;\n-\tf->algop->unsafe_init_fn(&f->ctx);\n+\tf->algop = unsafe_hash_algo(the_hash_algo);\n+\tf->algop->init_fn(&f->ctx);\n \n \tf->buffer_len = buffer_len;\n \tf->buffer = xmalloc(buffer_len);\n@@ -210,7 +210,7 @@ void hashfile_checkpoint(struct hashfile *f, struct hashfile_checkpoint *checkpo\n {\n \thashflush(f);\n \tcheckpoint->offset = f->total;\n-\tf->algop->unsafe_clone_fn(&checkpoint->ctx, &f->ctx);\n+\tf->algop->clone_fn(&checkpoint->ctx, &f->ctx);\n }\n \n int hashfile_truncate(struct hashfile *f, struct hashfile_checkpoint *checkpoint)\n@@ -221,7 +221,7 @@ int hashfile_truncate(struct hashfile *f, struct hashfile_checkpoint *checkpoint\n \t    lseek(f->fd, offset, SEEK_SET) != offset)\n \t\treturn -1;\n \tf->total = offset;\n-\tf->algop->unsafe_clone_fn(&f->ctx, &checkpoint->ctx);\n+\tf->algop->clone_fn(&f->ctx, &checkpoint->ctx);\n \tf->offset = 0; /* hashflush() was called in checkpoint */\n \treturn 0;\n }\n@@ -242,15 +242,15 @@ int hashfile_checksum_valid(const unsigned char *data, size_t total_len)\n {\n \tunsigned char got[GIT_MAX_RAWSZ];\n \tgit_hash_ctx ctx;\n-\tconst struct git_hash_algo *algop = the_hash_algo;\n+\tconst struct git_hash_algo *algop = unsafe_hash_algo(the_hash_algo);\n \tsize_t data_len = total_len - algop->rawsz;\n \n \tif (total_len < algop->rawsz)\n \t\treturn 0; /* say \"too short\"? */\n \n-\talgop->unsafe_init_fn(&ctx);\n-\talgop->unsafe_update_fn(&ctx, data, data_len);\n-\talgop->unsafe_final_fn(got, &ctx);\n+\talgop->init_fn(&ctx);\n+\talgop->update_fn(&ctx, data, data_len);\n+\talgop->final_fn(got, &ctx);\n \n \treturn hasheq(got, data + data_len, algop);\n }\n-- \n2.48.0.rc2.33.gaab3d23ed4c\n\n"},{"id":"510195","messageId":"a2b9ef03080020ca45f2a956128b602d0dca9e3b.1736363652.git.me@ttaylorr.com","threadId":"62522","inReplyTo":"cover.1736363652.git.me@ttaylorr.com","subject":"[PATCH v2 6/8] t/helper/test-hash.c: use unsafe_hash_algo()","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-01-08T19:14:48Z","receivedAt":"2025-01-08T19:14:50Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Remove a series of conditionals within the shared cmd_hash_impl() helper\nthat powers the 'sha1' and 'sha1-unsafe' helpers.\n\nInstead, replace them with a single conditional that transforms the\nspecified hash algorithm into its unsafe variant. Then all subsequent\ncalls can directly use whatever function it wants to call without having\nto decide between the safe and unsafe variants.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n t/helper/test-hash.c | 17 +++++------------\n 1 file changed, 5 insertions(+), 12 deletions(-)\n\ndiff --git a/t/helper/test-hash.c b/t/helper/test-hash.c\nindex d0ee668df95..aa82638c621 100644\n--- a/t/helper/test-hash.c\n+++ b/t/helper/test-hash.c\n@@ -9,6 +9,8 @@ int cmd_hash_impl(int ac, const char **av, int algo, int unsafe)\n \tint binary = 0;\n \tchar *buffer;\n \tconst struct git_hash_algo *algop = &hash_algos[algo];\n+\tif (unsafe)\n+\t\talgop = unsafe_hash_algo(algop);\n \n \tif (ac == 2) {\n \t\tif (!strcmp(av[1], \"-b\"))\n@@ -27,10 +29,7 @@ int cmd_hash_impl(int ac, const char **av, int algo, int unsafe)\n \t\t\tdie(\"OOPS\");\n \t}\n \n-\tif (unsafe)\n-\t\talgop->unsafe_init_fn(&ctx);\n-\telse\n-\t\talgop->init_fn(&ctx);\n+\talgop->init_fn(&ctx);\n \n \twhile (1) {\n \t\tssize_t sz, this_sz;\n@@ -49,15 +48,9 @@ int cmd_hash_impl(int ac, const char **av, int algo, int unsafe)\n \t\t}\n \t\tif (this_sz == 0)\n \t\t\tbreak;\n-\t\tif (unsafe)\n-\t\t\talgop->unsafe_update_fn(&ctx, buffer, this_sz);\n-\t\telse\n-\t\t\talgop->update_fn(&ctx, buffer, this_sz);\n+\t\talgop->update_fn(&ctx, buffer, this_sz);\n \t}\n-\tif (unsafe)\n-\t\talgop->unsafe_final_fn(hash, &ctx);\n-\telse\n-\t\talgop->final_fn(hash, &ctx);\n+\talgop->final_fn(hash, &ctx);\n \n \tif (binary)\n \t\tfwrite(hash, 1, algop->rawsz, stdout);\n-- \n2.48.0.rc2.33.gaab3d23ed4c\n\n"},{"id":"510196","messageId":"94c07fd8a557c569fdc83015d5f3902094f21994.1736363652.git.me@ttaylorr.com","threadId":"62522","inReplyTo":"cover.1736363652.git.me@ttaylorr.com","subject":"[PATCH v2 7/8] csum-file: introduce hashfile_checkpoint_init()","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-01-08T19:14:51Z","receivedAt":"2025-01-08T19:14:54Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In 106140a99f (builtin/fast-import: fix segfault with unsafe SHA1\nbackend, 2024-12-30) and 9218c0bfe1 (bulk-checkin: fix segfault with\nunsafe SHA1 backend, 2024-12-30), we observed the effects of failing to\ninitialize a hashfile_checkpoint with the same hash function\nimplementation as is used by the hashfile it is used to checkpoint.\n\nWhile both 106140a99f and 9218c0bfe1 work around the immediate crash,\nchanging the hash function implementation within the hashfile API to,\nfor example, the non-unsafe variant would re-introduce the crash. This\nis a result of the tight coupling between initializing hashfiles and\nhashfile_checkpoints.\n\nIntroduce and use a new function which ensures that both parts of a\nhashfile and hashfile_checkpoint pair use the same hash function\nimplementation to avoid such crashes.\n\nA few things worth noting:\n\n  - In the change to builtin/fast-import.c::stream_blob(), we can see\n    that by removing the explicit reference to\n    'the_hash_algo->unsafe_init_fn()', we are hardened against the\n    hashfile API changing away from the_hash_algo (or its unsafe\n    variant) in the future.\n\n  - The bulk-checkin code no longer needs to explicitly zero-initialize\n    the hashfile_checkpoint, since it is now done as a result of calling\n    'hashfile_checkpoint_init()'.\n\n  - Also in the bulk-checkin code, we add an additional call to\n    prepare_to_stream() outside of the main loop in order to initialize\n    'state->f' so we know which hash function implementation to use when\n    calling 'hashfile_checkpoint_init()'.\n\n    This is OK, since subsequent 'prepare_to_stream()' calls are noops.\n    However, we only need to call 'prepare_to_stream()' when we have the\n    HASH_WRITE_OBJECT bit set in our flags. Without that bit, calling\n    'prepare_to_stream()' does not assign 'state->f', so we have nothing\n    to initialize.\n\n  - Other uses of the 'checkpoint' in 'deflate_blob_to_pack()' are\n    appropriately guarded.\n\nHelped-by: Patrick Steinhardt <ps@pks.im>\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n builtin/fast-import.c | 2 +-\n bulk-checkin.c        | 9 ++++++---\n csum-file.c           | 7 +++++++\n csum-file.h           | 1 +\n 4 files changed, 15 insertions(+), 4 deletions(-)\n\ndiff --git a/builtin/fast-import.c b/builtin/fast-import.c\nindex 0f86392761a..4a6c7ab52ac 100644\n--- a/builtin/fast-import.c\n+++ b/builtin/fast-import.c\n@@ -1106,7 +1106,7 @@ static void stream_blob(uintmax_t len, struct object_id *oidout, uintmax_t mark)\n \t\t|| (pack_size + PACK_SIZE_THRESHOLD + len) < pack_size)\n \t\tcycle_packfile();\n \n-\tthe_hash_algo->unsafe_init_fn(&checkpoint.ctx);\n+\thashfile_checkpoint_init(pack_file, &checkpoint);\n \thashfile_checkpoint(pack_file, &checkpoint);\n \toffset = checkpoint.offset;\n \ndiff --git a/bulk-checkin.c b/bulk-checkin.c\nindex 433070a3bda..892176d23d2 100644\n--- a/bulk-checkin.c\n+++ b/bulk-checkin.c\n@@ -261,7 +261,7 @@ static int deflate_blob_to_pack(struct bulk_checkin_packfile *state,\n \tgit_hash_ctx ctx;\n \tunsigned char obuf[16384];\n \tunsigned header_len;\n-\tstruct hashfile_checkpoint checkpoint = {0};\n+\tstruct hashfile_checkpoint checkpoint;\n \tstruct pack_idx_entry *idx = NULL;\n \n \tseekback = lseek(fd, 0, SEEK_CUR);\n@@ -272,12 +272,15 @@ static int deflate_blob_to_pack(struct bulk_checkin_packfile *state,\n \t\t\t\t\t  OBJ_BLOB, size);\n \tthe_hash_algo->init_fn(&ctx);\n \tthe_hash_algo->update_fn(&ctx, obuf, header_len);\n-\tthe_hash_algo->unsafe_init_fn(&checkpoint.ctx);\n \n \t/* Note: idx is non-NULL when we are writing */\n-\tif ((flags & HASH_WRITE_OBJECT) != 0)\n+\tif ((flags & HASH_WRITE_OBJECT) != 0) {\n \t\tCALLOC_ARRAY(idx, 1);\n \n+\t\tprepare_to_stream(state, flags);\n+\t\thashfile_checkpoint_init(state->f, &checkpoint);\n+\t}\n+\n \talready_hashed_to = 0;\n \n \twhile (1) {\ndiff --git a/csum-file.c b/csum-file.c\nindex ebffc80ef71..232121f415f 100644\n--- a/csum-file.c\n+++ b/csum-file.c\n@@ -206,6 +206,13 @@ struct hashfile *hashfd_throughput(int fd, const char *name, struct progress *tp\n \treturn hashfd_internal(fd, name, tp, 8 * 1024);\n }\n \n+void hashfile_checkpoint_init(struct hashfile *f,\n+\t\t\t      struct hashfile_checkpoint *checkpoint)\n+{\n+\tmemset(checkpoint, 0, sizeof(*checkpoint));\n+\tf->algop->init_fn(&checkpoint->ctx);\n+}\n+\n void hashfile_checkpoint(struct hashfile *f, struct hashfile_checkpoint *checkpoint)\n {\n \thashflush(f);\ndiff --git a/csum-file.h b/csum-file.h\nindex 2b45f4673a2..b7475f16c20 100644\n--- a/csum-file.h\n+++ b/csum-file.h\n@@ -36,6 +36,7 @@ struct hashfile_checkpoint {\n \tgit_hash_ctx ctx;\n };\n \n+void hashfile_checkpoint_init(struct hashfile *, struct hashfile_checkpoint *);\n void hashfile_checkpoint(struct hashfile *, struct hashfile_checkpoint *);\n int hashfile_truncate(struct hashfile *, struct hashfile_checkpoint *);\n \n-- \n2.48.0.rc2.33.gaab3d23ed4c\n\n"},{"id":"510197","messageId":"f5579883816b8b5ce68d8a8ff27da4f8bb84573b.1736363652.git.me@ttaylorr.com","threadId":"62522","inReplyTo":"cover.1736363652.git.me@ttaylorr.com","subject":"[PATCH v2 8/8] hash.h: drop unsafe_ function variants","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-01-08T19:14:54Z","receivedAt":"2025-01-08T19:14:57Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Now that all callers have been converted from:\n\n    the_hash_algo->unsafe_init_fn();\n\nto\n\n    unsafe_hash_algo(the_hash_algo)->unsafe_init_fn();\n\nand similar, we can remove the scaffolding for the unsafe_ function\nvariants and force callers to use the new unsafe_hash_algo() mechanic\ninstead.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n hash.h        | 15 ---------------\n object-file.c | 15 ---------------\n 2 files changed, 30 deletions(-)\n\ndiff --git a/hash.h b/hash.h\nindex 23cf6876e50..6dcbf6ab835 100644\n--- a/hash.h\n+++ b/hash.h\n@@ -282,21 +282,6 @@ struct git_hash_algo {\n \t/* The hash finalization function for object IDs. */\n \tgit_hash_final_oid_fn final_oid_fn;\n \n-\t/* The non-cryptographic hash initialization function. */\n-\tgit_hash_init_fn unsafe_init_fn;\n-\n-\t/* The non-cryptographic hash context cloning function. */\n-\tgit_hash_clone_fn unsafe_clone_fn;\n-\n-\t/* The non-cryptographic hash update function. */\n-\tgit_hash_update_fn unsafe_update_fn;\n-\n-\t/* The non-cryptographic hash finalization function. */\n-\tgit_hash_final_fn unsafe_final_fn;\n-\n-\t/* The non-cryptographic hash finalization function. */\n-\tgit_hash_final_oid_fn unsafe_final_oid_fn;\n-\n \t/* The OID of the empty tree. */\n \tconst struct object_id *empty_tree;\n \ndiff --git a/object-file.c b/object-file.c\nindex 43efa4ca361..c4b42dd4be9 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -230,11 +230,6 @@ const struct git_hash_algo hash_algos[GIT_HASH_NALGOS] = {\n \t\t.update_fn = git_hash_unknown_update,\n \t\t.final_fn = git_hash_unknown_final,\n \t\t.final_oid_fn = git_hash_unknown_final_oid,\n-\t\t.unsafe_init_fn = git_hash_unknown_init,\n-\t\t.unsafe_clone_fn = git_hash_unknown_clone,\n-\t\t.unsafe_update_fn = git_hash_unknown_update,\n-\t\t.unsafe_final_fn = git_hash_unknown_final,\n-\t\t.unsafe_final_oid_fn = git_hash_unknown_final_oid,\n \t\t.empty_tree = NULL,\n \t\t.empty_blob = NULL,\n \t\t.null_oid = NULL,\n@@ -250,11 +245,6 @@ const struct git_hash_algo hash_algos[GIT_HASH_NALGOS] = {\n \t\t.update_fn = git_hash_sha1_update,\n \t\t.final_fn = git_hash_sha1_final,\n \t\t.final_oid_fn = git_hash_sha1_final_oid,\n-\t\t.unsafe_init_fn = git_hash_sha1_init_unsafe,\n-\t\t.unsafe_clone_fn = git_hash_sha1_clone_unsafe,\n-\t\t.unsafe_update_fn = git_hash_sha1_update_unsafe,\n-\t\t.unsafe_final_fn = git_hash_sha1_final_unsafe,\n-\t\t.unsafe_final_oid_fn = git_hash_sha1_final_oid_unsafe,\n \t\t.unsafe = &sha1_unsafe_algo,\n \t\t.empty_tree = &empty_tree_oid,\n \t\t.empty_blob = &empty_blob_oid,\n@@ -271,11 +261,6 @@ const struct git_hash_algo hash_algos[GIT_HASH_NALGOS] = {\n \t\t.update_fn = git_hash_sha256_update,\n \t\t.final_fn = git_hash_sha256_final,\n \t\t.final_oid_fn = git_hash_sha256_final_oid,\n-\t\t.unsafe_init_fn = git_hash_sha256_init,\n-\t\t.unsafe_clone_fn = git_hash_sha256_clone,\n-\t\t.unsafe_update_fn = git_hash_sha256_update,\n-\t\t.unsafe_final_fn = git_hash_sha256_final,\n-\t\t.unsafe_final_oid_fn = git_hash_sha256_final_oid,\n \t\t.empty_tree = &empty_tree_oid_sha256,\n \t\t.empty_blob = &empty_blob_oid_sha256,\n \t\t.null_oid = &null_oid_sha256,\n-- \n2.48.0.rc2.33.gaab3d23ed4c\n"},{"id":"510287","messageId":"20250110103756.GA1014709@coredump.intra.peff.net","threadId":"62522","inReplyTo":"94c07fd8a557c569fdc83015d5f3902094f21994.1736363652.git.me@ttaylorr.com","subject":"Re: [PATCH v2 7/8] csum-file: introduce hashfile_checkpoint_init()","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2025-01-10T10:37:56Z","receivedAt":"2025-01-10T10:38:05Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Jan 08, 2025 at 02:14:51PM -0500, Taylor Blau wrote:\n\n> Introduce and use a new function which ensures that both parts of a\n> hashfile and hashfile_checkpoint pair use the same hash function\n> implementation to avoid such crashes.\n\nThat makes sense. This should have been encapsulated all along, just\nlike the actual hash initialization happens inside hashfile_init().\n\nA hashfile_checkpoint is sort of inherently tied to a hashfile, right? I\nmean, it is recording an offset that only makes sense in the context of\nthe parent hashfile.\n\nAnd that is only more true after the unsafe-hash patches, because now it\nneeds to use the \"algop\" pointer from the parent hashfile (though for\nnow we expect all hashfiles to use the same unsafe-algop, in theory we\ncould use different checksums for each file).\n\nSo in the new constructor:\n\n> +void hashfile_checkpoint_init(struct hashfile *f,\n> +\t\t\t      struct hashfile_checkpoint *checkpoint)\n> +{\n> +\tmemset(checkpoint, 0, sizeof(*checkpoint));\n> +\tf->algop->init_fn(&checkpoint->ctx);\n> +}\n\n...should we actually record \"f\" itself? And then in the existing\nfunctions:\n\n>  void hashfile_checkpoint(struct hashfile *f, struct hashfile_checkpoint *checkpoint)\n\n...they'd no longer need to take the extra parameter.\n\nIt creates a lifetime dependency of the checkpoint struct on the \"f\" it\nis checkpointing, but I think that is naturally modeling the domain.\n\nA semi-related thing I wondered about: do we need a destructor/release\nfunction of some kind? Long ago when this checkpoint code was added, a\nmemcpy() of the sha_ctx struct was sufficient. But these days we use\nclone_fn(), which may call openssl_SHA1_Clone(), which does\nEVP_MD_CTX_copy_ex() under the hood. Do we have any promise that this\ndoesn't allocate any resources that might need a call to _Final() to\nrelease (or I guess the more efficient way is directly EVP_MD_CTX_free()\nunder the hood).\n\nMy reading of the openssl manpages suggests that we should be doing\nthat, or we may see leaks. But it may also be the case that it doesn't\nhappen to trigger for their implementation.\n\nAt any rate, we do not seem to have such a cleanup function. So it is\ncertainly an orthogonal issue to your series. I wondered about it here\nbecause if we did have one, it would be necessary to clean up checkpoint\nbefore the hashfile due to the lifetime dependency I mentioned above.\n\n-Peff\n"},{"id":"510288","messageId":"20250110104106.GB1014709@coredump.intra.peff.net","threadId":"62522","inReplyTo":"cover.1736363652.git.me@ttaylorr.com","subject":"Re: [PATCH v2 0/8] hash: introduce unsafe_hash_algo(), drop unsafe_ variants","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2025-01-10T10:41:06Z","receivedAt":"2025-01-10T10:41:08Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Jan 08, 2025 at 02:14:29PM -0500, Taylor Blau wrote:\n\n> (This series is rebased on 'master', which is 14650065b7\n> (RelNotes/2.48.0: fix typos etc., 2025-01-07) at the time of writing).\n> \n> The bulk of this series is unchanged since last time, but a new seventh\n> patch that further hardens the hashfile_checkpoint callers on top of\n> Patrick's recent series[1].\n\nI think that new patch is a definite improvement, though I left some\ncomments there.\n\nThe changes in patch 1 look fine to me (I still think a generic\n\"test-tool hash\" would make more sense, but I'm OK punting on that for\nnow).\n\nI didn't see any response to the review in round 1 about the pointer\ndangers in patch 3. What do you think of using a separate\ngit_hash_algo_fns struct, with the one-time conversion I showed in the\nsubthread of:\n\n  https://lore.kernel.org/git/20241121093731.GD602681@coredump.intra.peff.net/\n\n?\n\n-Peff\n"},{"id":"510345","messageId":"Z4GRQvAIsSIeQuYd@nand.local","threadId":"62522","inReplyTo":"20250110104106.GB1014709@coredump.intra.peff.net","subject":"Re: [PATCH v2 0/8] hash: introduce unsafe_hash_algo(), drop unsafe_ variants","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-01-10T21:29:38Z","receivedAt":"2025-01-10T21:29:43Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Fri, Jan 10, 2025 at 05:41:06AM -0500, Jeff King wrote:\n> On Wed, Jan 08, 2025 at 02:14:29PM -0500, Taylor Blau wrote:\n>\n> > (This series is rebased on 'master', which is 14650065b7\n> > (RelNotes/2.48.0: fix typos etc., 2025-01-07) at the time of writing).\n> >\n> > The bulk of this series is unchanged since last time, but a new seventh\n> > patch that further hardens the hashfile_checkpoint callers on top of\n> > Patrick's recent series[1].\n>\n> I think that new patch is a definite improvement, though I left some\n> comments there.\n>\n> The changes in patch 1 look fine to me (I still think a generic\n> \"test-tool hash\" would make more sense, but I'm OK punting on that for\n> now).\n>\n> I didn't see any response to the review in round 1 about the pointer\n> dangers in patch 3. What do you think of using a separate\n> git_hash_algo_fns struct, with the one-time conversion I showed in the\n> subthread of:\n>\n>   https://lore.kernel.org/git/20241121093731.GD602681@coredump.intra.peff.net/\n>\n> ?\n\nOops. I must have missed those messages; and sure enough when focusing\nmy inbox on this thread they are indeed unread :-).\n\nThat said, I am not sure that that direction is one that I'd want to go\nin. Part of the goal of this series is to make it possible to mix safe\nand unsafe function calls on the same hash function. So doing something\nlike:\n\n    struct git_hash_algo *algop;\n\n    algop->init_fn(&ctx);\n\nin one part of the code, and then (using the same algop) calling:\n\n    algop->unsafe_final_fn(...);\n\nshould be impossible to do to. The benefit of having only a single set\nof functions implemented on the git_hash_algo type is that it is\nimpossible to mix the two: you'd have to use a different git_hash_algo\naltogether!\n\nSo porting the above example to your and brian's git_hash_algo_fns\nstruct, you'd still be able to do:\n\n    algop->fn.init(&ctx);\n\nin one part of the code and algop->unsafe_fn.final(...) in another part,\nwhich doesn't appear to me to be safer than the current situation that\nthis series aims to solve.\n\nBut if I am misunderstanding something about your/brian's discussion\nearlier in this thread, please feel free to correct me.\n\nThanks,\nTaylor\n"},{"id":"510347","messageId":"Z4GTcKTVFOFmSsgO@nand.local","threadId":"62522","inReplyTo":"20241121093731.GD602681@coredump.intra.peff.net","subject":"Re: [PATCH 3/6] hash.h: introduce `unsafe_hash_algo()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-01-10T21:38:56Z","receivedAt":"2025-01-10T21:38:59Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Nov 21, 2024 at 04:37:31AM -0500, Jeff King wrote:\n> If we don't care about the speed of this function, then an\n> implementation like:\n>\n>   for (i = 0; i < GIT_HASH_NALGOS; i++) {\n> \tif (p == &hash_algos[i] || p == hash_algos[i]->unsafe)\n> \t\treturn i;\n>   }\n>   return GIT_HASH_UNKNOWN;\n>\n> would work. I'm not sure if that would be measurable. I was surprised at\n> the number of places that hash_algo_by_ptr() is called. Many low-level\n> oid functions need it because we store the integer id there rather than\n> a direct pointer (so oidread(), oidclr(), oid_object_info_extended(),\n> and so on). But I'd also expect the loop above to be pretty fast. So I\n> dunno.\n\nConcerns about the speed aside (I agree that the for loop is likely to\nbe very fast, and will probably get unrolled by modern compilers), this\nlooks good to me with one small tweak.\n\nWe can't use `hash_algos[i]->unsafe` directly it might be NULL, so the\nfunction as above would change the behavior of hash_algo_by_ptr()\nslightly when provided NULL (it would return SHA-256 instead of\nUNKNOWN).\n\nSo I think you'd want to write the loop like:\n\n    size_t i;\n\n    for (i = 0; i < GIT_HASH_NALGOS) {\n        const struct git_hash_algo *algop = &hash_algos[i];\n\n        if (p == algop || (algop->unsafe && p == algop->unsafe))\n            return i;\n    }\n\n    return GIT_HASH_UNKNOWN;\n\nThanks,\nTaylor\n"},{"id":"510348","messageId":"Z4GWIZkJOUa278VA@nand.local","threadId":"62522","inReplyTo":"20250110103756.GA1014709@coredump.intra.peff.net","subject":"Re: [PATCH v2 7/8] csum-file: introduce hashfile_checkpoint_init()","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-01-10T21:50:25Z","receivedAt":"2025-01-10T21:50:28Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Fri, Jan 10, 2025 at 05:37:56AM -0500, Jeff King wrote:\n> So in the new constructor:\n>\n> > +void hashfile_checkpoint_init(struct hashfile *f,\n> > +\t\t\t      struct hashfile_checkpoint *checkpoint)\n> > +{\n> > +\tmemset(checkpoint, 0, sizeof(*checkpoint));\n> > +\tf->algop->init_fn(&checkpoint->ctx);\n> > +}\n>\n> ...should we actually record \"f\" itself? And then in the existing\n> functions:\n>\n> >  void hashfile_checkpoint(struct hashfile *f, struct hashfile_checkpoint *checkpoint)\n>\n> ...they'd no longer need to take the extra parameter.\n>\n> It creates a lifetime dependency of the checkpoint struct on the \"f\" it\n> is checkpointing, but I think that is naturally modeling the domain.\n\nThanks, I really like these suggestions. I adjusted the series\naccordingly to do this cleanup in two patches (one for\nhashfile_checkpoint(), another for hashfile_truncate()) after the patch\nintroducing hashfile_checkpoint_init().\n\n> A semi-related thing I wondered about: do we need a destructor/release\n> function of some kind? Long ago when this checkpoint code was added, a\n> memcpy() of the sha_ctx struct was sufficient. But these days we use\n> clone_fn(), which may call openssl_SHA1_Clone(), which does\n> EVP_MD_CTX_copy_ex() under the hood. Do we have any promise that this\n> doesn't allocate any resources that might need a call to _Final() to\n> release (or I guess the more efficient way is directly EVP_MD_CTX_free()\n> under the hood).\n>\n> My reading of the openssl manpages suggests that we should be doing\n> that, or we may see leaks. But it may also be the case that it doesn't\n> happen to trigger for their implementation.\n>\n> At any rate, we do not seem to have such a cleanup function. So it is\n> certainly an orthogonal issue to your series. I wondered about it here\n> because if we did have one, it would be necessary to clean up checkpoint\n> before the hashfile due to the lifetime dependency I mentioned above.\n\nI like the idea of a cleanup function, but let's do so in a separate\nseries.\n\nThanks,\nTaylor\n"},{"id":"510359","messageId":"xmqq4j26cikd.fsf@gitster.g","threadId":"62522","inReplyTo":"cover.1736363652.git.me@ttaylorr.com","subject":"Re: [PATCH v2 0/8] hash: introduce unsafe_hash_algo(), drop unsafe_ variants","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-01-11T00:14:58Z","receivedAt":"2025-01-11T00:15:02Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau <me@ttaylorr.com> writes:\n\n> (This series is rebased on 'master', which is 14650065b7\n> (RelNotes/2.48.0: fix typos etc., 2025-01-07) at the time of writing).\n\nThe previous round was based on\n<cover.1730833506.git.me@ttaylorr.com> which became\n'tb/unsafe-hash-test', but this round is based on a recent 'master'\nthat does not yet contain it?  Does it mean that the 2-patch series\nthe previous round of this series was based on is no longer needed?\n\nThanks.\n"},{"id":"510363","messageId":"20250111024203.GA590176@coredump.intra.peff.net","threadId":"62522","inReplyTo":"Z4GRQvAIsSIeQuYd@nand.local","subject":"Re: [PATCH v2 0/8] hash: introduce unsafe_hash_algo(), drop unsafe_ variants","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2025-01-11T02:42:03Z","receivedAt":"2025-01-11T02:42:05Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Jan 10, 2025 at 04:29:38PM -0500, Taylor Blau wrote:\n\n> > I didn't see any response to the review in round 1 about the pointer\n> > dangers in patch 3. What do you think of using a separate\n> > git_hash_algo_fns struct, with the one-time conversion I showed in the\n> > subthread of:\n> >\n> >   https://lore.kernel.org/git/20241121093731.GD602681@coredump.intra.peff.net/\n> >\n> > ?\n> \n> Oops. I must have missed those messages; and sure enough when focusing\n> my inbox on this thread they are indeed unread :-).\n> \n> That said, I am not sure that that direction is one that I'd want to go\n> in. Part of the goal of this series is to make it possible to mix safe\n> and unsafe function calls on the same hash function. So doing something\n> like:\n> \n>     struct git_hash_algo *algop;\n> \n>     algop->init_fn(&ctx);\n> \n> in one part of the code, and then (using the same algop) calling:\n> \n>     algop->unsafe_final_fn(...);\n> \n> should be impossible to do to. The benefit of having only a single set\n> of functions implemented on the git_hash_algo type is that it is\n> impossible to mix the two: you'd have to use a different git_hash_algo\n> altogether!\n> \n> So porting the above example to your and brian's git_hash_algo_fns\n> struct, you'd still be able to do:\n> \n>     algop->fn.init(&ctx);\n> \n> in one part of the code and algop->unsafe_fn.final(...) in another part,\n> which doesn't appear to me to be safer than the current situation that\n> this series aims to solve.\n\nI think what that proposal is doing is orthogonal to the goal of your\nseries. You'd still have an unsafe_hash_algo() function, but it would\nreturn a git_hash_algo_fns struct, and that's what struct hashfile would\nstore. So your patches would still remain.\n\nThe advantage is mostly that you can't confuse it with a regular\ngit_hash_algo struct, so it avoids the pointer and hash-id issues.\n\nI do think there is one gotcha, though, which is that the hashfile code\nprobably still needs the outer algop pointer for things like\nalgop->raw_size. So you'd have to store both.\n\nIt's _possible_ to still confuse the two, but the idea is that you'd\nhave to explicitly call algop->fn, to get the wrong one there. If we\nwanted to make that harder to get wrong, we could start making it a\nhabit to never use algo->fn directly, but to ask for the safe/unsafe\ngit_hash_algo_fns struct. But that would be even more churn in the\nsurrounding code. I think just doing it consistently within hashfile\n(which is the only unsafe user) would be sufficient.\n\n-Peff\n"},{"id":"510364","messageId":"20250111024502.GB590176@coredump.intra.peff.net","threadId":"62522","inReplyTo":"Z4GTcKTVFOFmSsgO@nand.local","subject":"Re: [PATCH 3/6] hash.h: introduce `unsafe_hash_algo()`","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2025-01-11T02:45:02Z","receivedAt":"2025-01-11T02:45:03Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Jan 10, 2025 at 04:38:56PM -0500, Taylor Blau wrote:\n\n> On Thu, Nov 21, 2024 at 04:37:31AM -0500, Jeff King wrote:\n> > If we don't care about the speed of this function, then an\n> > implementation like:\n> >\n> >   for (i = 0; i < GIT_HASH_NALGOS; i++) {\n> > \tif (p == &hash_algos[i] || p == hash_algos[i]->unsafe)\n> > \t\treturn i;\n> >   }\n> >   return GIT_HASH_UNKNOWN;\n> >\n> > would work. I'm not sure if that would be measurable. I was surprised at\n> > the number of places that hash_algo_by_ptr() is called. Many low-level\n> > oid functions need it because we store the integer id there rather than\n> > a direct pointer (so oidread(), oidclr(), oid_object_info_extended(),\n> > and so on). But I'd also expect the loop above to be pretty fast. So I\n> > dunno.\n> \n> Concerns about the speed aside (I agree that the for loop is likely to\n> be very fast, and will probably get unrolled by modern compilers), this\n> looks good to me with one small tweak.\n\nIt should definitely be unrolled. The big change against the existing\ncode is that it has branches. But again, I'm not sure how much the\nperformance matters here (I would have naively said not at all, but it\ndoes get called in lots of low-level spots).\n\n> We can't use `hash_algos[i]->unsafe` directly it might be NULL, so the\n> function as above would change the behavior of hash_algo_by_ptr()\n> slightly when provided NULL (it would return SHA-256 instead of\n> UNKNOWN).\n> \n> So I think you'd want to write the loop like:\n> \n>     size_t i;\n> \n>     for (i = 0; i < GIT_HASH_NALGOS) {\n>         const struct git_hash_algo *algop = &hash_algos[i];\n> \n>         if (p == algop || (algop->unsafe && p == algop->unsafe))\n>             return i;\n>     }\n> \n>     return GIT_HASH_UNKNOWN;\n\nYes, that works. Or maybe even just:\n\n  if (!algop)\n\treturn GIT_HASH_UNKNOWN;\n\nat the top to cover the special case.\n\n-Peff\n"},{"id":"510371","messageId":"Z4Km9QaGflwAI7tA@nand.local","threadId":"62522","inReplyTo":"xmqq4j26cikd.fsf@gitster.g","subject":"Re: [PATCH v2 0/8] hash: introduce unsafe_hash_algo(), drop unsafe_ variants","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-01-11T17:14:29Z","receivedAt":"2025-01-11T17:14:36Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Fri, Jan 10, 2025 at 04:14:58PM -0800, Junio C Hamano wrote:\n> Taylor Blau <me@ttaylorr.com> writes:\n>\n> > (This series is rebased on 'master', which is 14650065b7\n> > (RelNotes/2.48.0: fix typos etc., 2025-01-07) at the time of writing).\n>\n> The previous round was based on\n> <cover.1730833506.git.me@ttaylorr.com> which became\n> 'tb/unsafe-hash-test', but this round is based on a recent 'master'\n> that does not yet contain it?  Does it mean that the 2-patch series\n> the previous round of this series was based on is no longer needed?\n\nThose two patches got squashed together and became the first patch of\nthis series, so 'tb/unsafe-hash-test' is safe to discard. Thank you for\nshuffling the patches around as always :-).\n\nThanks,\nTaylor\n"},{"id":"510691","messageId":"Z4jyH4yIXuD0vuqQ@pks.im","threadId":"62522","inReplyTo":"99cc44895b57cc75fc8f447000817b3595368e4c.1736363652.git.me@ttaylorr.com","subject":"Re: [PATCH v2 2/8] csum-file: store the hash algorithm as a struct field","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-01-16T11:48:58Z","receivedAt":"2025-01-16T11:49:03Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Wed, Jan 08, 2025 at 02:14:35PM -0500, Taylor Blau wrote:\n> Throughout the hashfile API, we rely on a reference to 'the_hash_algo',\n> and call its _usnafe function variants directly.\n\ns/usnafe/unsafe/\n\n> Prepare for a future change where we may use a different 'git_hash_algo'\n> pointer (instead of just relying on 'the_hash_algo' throughout) by\n> making the 'git_hash_algo' pointer a member of the 'hashfile' structure\n> itself.\n\nMakes sense, and it's also a good step for libification. I wonder: does\nit mean that we can also get rid of `USE_THE_REPOSITORY_VARIABLE`, or do\nwe still depend on it in this file? The answer is yes, as we only reduce\nthe sites where we use `the_hash_algo`, but don't remove it altogether.\nThat would require the caller to provide the hash algo to us.\n\nPatrick\n"},{"id":"510692","messageId":"Z4jyZCAwqOjZ-u2U@pks.im","threadId":"62522","inReplyTo":"99dcbe2e7165d96e3a9c025540995a75f74b2489.1736363652.git.me@ttaylorr.com","subject":"Re: [PATCH v2 4/8] hash.h: introduce `unsafe_hash_algo()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-01-16T11:49:56Z","receivedAt":"2025-01-16T11:50:00Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Wed, Jan 08, 2025 at 02:14:42PM -0500, Taylor Blau wrote:\n> Address these issues by introducing a new pattern whereby one\n> 'git_hash_algo' can return a pointer to another 'git_hash_algo' that\n> represents the unsafe version of itself. So instead of having something\n> like:\n> \n>     if (unsafe)\n>       the_hash_algo->init_fn(...);\n>       the_hash_algo->update_fn(...);\n>       the_hash_algo->final_fn(...);\n>     else\n>       the_hash_algo->unsafe_init_fn(...);\n>       the_hash_algo->unsafe_update_fn(...);\n>       the_hash_algo->unsafe_final_fn(...);\n> \n> we can instead write:\n> \n>     struct git_hash_algo *algop = the_hash_algo;\n>     if (unsafe)\n>       algop = unsafe_hash_algo(algop);\n> \n>     the_hash_algo->init_fn(...);\n>     the_hash_algo->update_fn(...);\n>     the_hash_algo->final_fn(...);\n\nThis should all be `algop->init_fn(...)` and so on, right?\n\n> This removes the existing shortcoming by no longer forcing the caller to\n> \"remember\" which variant of the hash functions it wants to call, only to\n> hold onto a 'struct git_hash_algo' pointer that is initialized once.\n\nWe still have to remember something, but now we only have to remember\nthe pointer to the hash algorithm itself, which definitely is less\ntedious.\n\nFor what it's worth, this prompted me to have another look at my\nproposal to stop having to use the hash algo altogether for `update()`\net al, and now have a version that works. It builds on top of your patch\nseries, which still is a step into the right direction. So I'll send it\nonce your series is being merged down.\n\nThanks!\n\nPatrick\n"},{"id":"510826","messageId":"Z4rI2r/voKHzu6Xh@nand.local","threadId":"62522","inReplyTo":"Z4jyH4yIXuD0vuqQ@pks.im","subject":"Re: [PATCH v2 2/8] csum-file: store the hash algorithm as a struct field","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-01-17T21:17:14Z","receivedAt":"2025-01-17T21:17:22Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Jan 16, 2025 at 12:48:58PM +0100, Patrick Steinhardt wrote:\n> On Wed, Jan 08, 2025 at 02:14:35PM -0500, Taylor Blau wrote:\n> > Throughout the hashfile API, we rely on a reference to 'the_hash_algo',\n> > and call its _usnafe function variants directly.\n>\n> s/usnafe/unsafe/\n\nOops. Thanks for spotting!\n\n> > Prepare for a future change where we may use a different 'git_hash_algo'\n> > pointer (instead of just relying on 'the_hash_algo' throughout) by\n> > making the 'git_hash_algo' pointer a member of the 'hashfile' structure\n> > itself.\n>\n> Makes sense, and it's also a good step for libification. I wonder: does\n> it mean that we can also get rid of `USE_THE_REPOSITORY_VARIABLE`, or do\n> we still depend on it in this file? The answer is yes, as we only reduce\n> the sites where we use `the_hash_algo`, but don't remove it altogether.\n> That would require the caller to provide the hash algo to us.\n\nYeah, I agree that we're not quite there yet, but likewise that this\npatch is a step in that direction.\n\nThanks,\nTaylor\n"},{"id":"510828","messageId":"Z4rJMhEkbNfhQ4ua@nand.local","threadId":"62522","inReplyTo":"Z4jyZCAwqOjZ-u2U@pks.im","subject":"Re: [PATCH v2 4/8] hash.h: introduce `unsafe_hash_algo()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-01-17T21:18:42Z","receivedAt":"2025-01-17T21:18:44Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Jan 16, 2025 at 12:49:56PM +0100, Patrick Steinhardt wrote:\n> > we can instead write:\n> >\n> >     struct git_hash_algo *algop = the_hash_algo;\n> >     if (unsafe)\n> >       algop = unsafe_hash_algo(algop);\n> >\n> >     the_hash_algo->init_fn(...);\n> >     the_hash_algo->update_fn(...);\n> >     the_hash_algo->final_fn(...);\n>\n> This should all be `algop->init_fn(...)` and so on, right?\n\nMost definitely :-).\n\n> For what it's worth, this prompted me to have another look at my\n> proposal to stop having to use the hash algo altogether for `update()`\n> et al, and now have a version that works. It builds on top of your patch\n> series, which still is a step into the right direction. So I'll send it\n> once your series is being merged down.\n\nExciting! I'm looking forward to reviewing it.\n\nThanks,\nTaylor\n"},{"id":"510830","messageId":"Z4rL3TbEeR8EiUOi@nand.local","threadId":"62522","inReplyTo":"Z4GWIZkJOUa278VA@nand.local","subject":"Re: [PATCH v2 7/8] csum-file: introduce hashfile_checkpoint_init()","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-01-17T21:30:05Z","receivedAt":"2025-01-17T21:30:08Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Fri, Jan 10, 2025 at 04:50:25PM -0500, Taylor Blau wrote:\n> On Fri, Jan 10, 2025 at 05:37:56AM -0500, Jeff King wrote:\n> > So in the new constructor:\n> >\n> > > +void hashfile_checkpoint_init(struct hashfile *f,\n> > > +\t\t\t      struct hashfile_checkpoint *checkpoint)\n> > > +{\n> > > +\tmemset(checkpoint, 0, sizeof(*checkpoint));\n> > > +\tf->algop->init_fn(&checkpoint->ctx);\n> > > +}\n> >\n> > ...should we actually record \"f\" itself? And then in the existing\n> > functions:\n> >\n> > >  void hashfile_checkpoint(struct hashfile *f, struct hashfile_checkpoint *checkpoint)\n> >\n> > ...they'd no longer need to take the extra parameter.\n> >\n> > It creates a lifetime dependency of the checkpoint struct on the \"f\" it\n> > is checkpointing, but I think that is naturally modeling the domain.\n>\n> Thanks, I really like these suggestions. I adjusted the series\n> accordingly to do this cleanup in two patches (one for\n> hashfile_checkpoint(), another for hashfile_truncate()) after the patch\n> introducing hashfile_checkpoint_init().\n\nHmm. I'm not sure that I like this as much as I thought I did.\n\nI agree with you that ultimately the hashfile_checkpoint is (or should\nbe) tied to the lifetime of the hashfile that it is checkpointing\nunderneath. But in practice things are a little funky.\n\nLet's suppose I did something like the following:\n\n--- 8< ---\ndiff --git a/csum-file.c b/csum-file.c\nindex ebffc80ef7..47b8317a1f 100644\n--- a/csum-file.c\n+++ b/csum-file.c\n@@ -206,6 +206,15 @@ struct hashfile *hashfd_throughput(int fd, const char *name, struct progress *tp\n \treturn hashfd_internal(fd, name, tp, 8 * 1024);\n }\n\n+void hashfile_checkpoint_init(struct hashfile *f,\n+\t\t\t      struct hashfile_checkpoint *checkpoint)\n+{\n+\tmemset(checkpoint, 0, sizeof(*checkpoint));\n+\n+\tcheckpoint->f = f;\n+\tcheckpoint->f->algop->init_fn(&checkpoint->ctx);\n+}\n+\n void hashfile_checkpoint(struct hashfile *f, struct hashfile_checkpoint *checkpoint)\n {\n \thashflush(f);\ndiff --git a/csum-file.h b/csum-file.h\nindex 2b45f4673a..8016509c71 100644\n--- a/csum-file.h\n+++ b/csum-file.h\n@@ -34,8 +34,10 @@ struct hashfile {\n struct hashfile_checkpoint {\n \toff_t offset;\n \tgit_hash_ctx ctx;\n+\tstruct hashfile *f;\n };\n\n+void hashfile_checkpoint_init(struct hashfile *, struct hashfile_checkpoint *);\n void hashfile_checkpoint(struct hashfile *, struct hashfile_checkpoint *);\n int hashfile_truncate(struct hashfile *, struct hashfile_checkpoint *);\n--- >8 ---\n\n, where I'm eliding the trivial changes necessary to make this work at\nthe two callers. Let's look a little closer at the bulk-checkin caller.\nIf I do this on top:\n\n--- 8< ---\ndiff --git a/bulk-checkin.c b/bulk-checkin.c\nindex 433070a3bd..892176d23d 100644\n--- a/bulk-checkin.c\n+++ b/bulk-checkin.c\n@@ -261,7 +261,7 @@ static int deflate_blob_to_pack(struct bulk_checkin_packfile *state,\n \tgit_hash_ctx ctx;\n \tunsigned char obuf[16384];\n \tunsigned header_len;\n-\tstruct hashfile_checkpoint checkpoint = {0};\n+\tstruct hashfile_checkpoint checkpoint;\n \tstruct pack_idx_entry *idx = NULL;\n\n \tseekback = lseek(fd, 0, SEEK_CUR);\n@@ -272,12 +272,15 @@ static int deflate_blob_to_pack(struct bulk_checkin_packfile *state,\n \t\t\t\t\t  OBJ_BLOB, size);\n \tthe_hash_algo->init_fn(&ctx);\n \tthe_hash_algo->update_fn(&ctx, obuf, header_len);\n-\tthe_hash_algo->unsafe_init_fn(&checkpoint.ctx);\n\n \t/* Note: idx is non-NULL when we are writing */\n-\tif ((flags & HASH_WRITE_OBJECT) != 0)\n+\tif ((flags & HASH_WRITE_OBJECT) != 0) {\n \t\tCALLOC_ARRAY(idx, 1);\n\n+\t\tprepare_to_stream(state, flags);\n+\t\thashfile_checkpoint_init(state->f, &checkpoint);\n+\t}\n+\n \talready_hashed_to = 0;\n\n \twhile (1) {\n--- >8 ---\n\nthen test 14 in t1050-large.sh fails because of a segfault in 'git add'.\nIf we compile with SANITIZE=address, we can see that there's a\nuse-after-free in hashflush(), which is called by hashfile_checkpoint().\n\nThat is a result of the max pack-size code. So we could try something\nlike:\n\n--- 8< ---\ndiff --git a/bulk-checkin.c b/bulk-checkin.c\nindex 7b8a6eb2df..9dc114d132 100644\n--- a/bulk-checkin.c\n+++ b/bulk-checkin.c\n@@ -261,7 +261,7 @@ static int deflate_blob_to_pack(struct bulk_checkin_packfile *state,\n \tgit_hash_ctx ctx;\n \tunsigned char obuf[16384];\n \tunsigned header_len;\n-\tstruct hashfile_checkpoint checkpoint;\n+\tstruct hashfile_checkpoint checkpoint = { 0 };\n \tstruct pack_idx_entry *idx = NULL;\n\n \tseekback = lseek(fd, 0, SEEK_CUR);\n@@ -274,17 +274,14 @@ static int deflate_blob_to_pack(struct bulk_checkin_packfile *state,\n \tthe_hash_algo->update_fn(&ctx, obuf, header_len);\n\n \t/* Note: idx is non-NULL when we are writing */\n-\tif ((flags & HASH_WRITE_OBJECT) != 0) {\n+\tif ((flags & HASH_WRITE_OBJECT) != 0)\n \t\tCALLOC_ARRAY(idx, 1);\n-\n-\t\tprepare_to_stream(state, flags);\n-\t\thashfile_checkpoint_init(state->f, &checkpoint);\n-\t}\n-\n \talready_hashed_to = 0;\n\n \twhile (1) {\n \t\tprepare_to_stream(state, flags);\n+\t\tif (checkpoint.f != state->f)\n+\t\t\thashfile_checkpoint_init(state->f, &checkpoint);\n \t\tif (idx) {\n \t\t\thashfile_checkpoint(&checkpoint);\n \t\t\tidx->offset = state->offset;\n--- >8 ---\n\nwhich would do the trick, but it feels awfully hacky to have the \"if\n(checkpoint.f != state->f)\" check in there, since that feels too\nintimately tied to the implementation of the hashfile_checkpoint API for\nmy comfort.\n\nIt would be nice if we could make the checkpoint only declared within\nthe loop body itself, but we can't because we need to call\nhashfile_truncate() outside of the loop.\n\nAnyway, that's all to say that I think that while this is probably\ndoable in theory, in practice it's kind of a mess, at least currently.\nI would rather see if there are other ways to clean up the\ndeflate_blob_to_pack() function first in a way that made this change\nless awkward.\n\nI think the most reasonable course here would be to pursue a minimal\nchange like the one presented here and then think about further clean up\nas a separate step.\n\nThanks,\nTaylor\n"},{"id":"510840","messageId":"cover.1737151386.git.me@ttaylorr.com","threadId":"62522","inReplyTo":"cover.1732130001.git.me@ttaylorr.com","subject":"[PATCH v3 0/8] hash: introduce unsafe_hash_algo(), drop unsafe_ variants","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-01-17T22:03:07Z","receivedAt":"2025-01-17T22:03:09Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"(This series is based on 14650065b7 (RelNotes/2.48.0: fix typos etc.,\n2025-01-07)).\n\nThe bulk of this series is unchanged since last time, save for a couple\nof typofixes on spots noticed by Peff and Patrick Steinhardt. More\nimportantly, it fixes hash_algo_by_ptr() when passing the unsafe SHA-1\nvariant.\n\nThere were a couple of other ideas floated around, but I don't think\nthey panned out as we had hoped in practice, so I think that this\nversion should be good to go.\n\nThe original cover letter is as follows:\n\n------------\n\nThis series implements an idea discussed in [2] which suggests that we\nintroduce a way to access a wrapped version of a 'struct git_hash_algo'\nwhich represents the unsafe variant of that algorithm, rather than\nhaving individual unsafe_ functions (like unsafe_init_fn() versus\ninit_fn(), etc.).\n\nThis approach is relatively straightforward to implement, and removes a\nsignificant deficiency in the original implementation of\nunsafe/non-cryptographic hash functions by making it impossible to\nswitch between safe- and unsafe variants of hash functions. It also\ncleans up the sha1-unsafe test helper's implementation by removing a\nlarge number of \"if (unsafe)\"-style conditionals.\n\nThe series is laid out as follows:\n\n  * The first two patches prepare the hashfile API for the upcoming\n    change:\n\n      csum-file: store the hash algorithm as a struct field\n      csum-file.c: extract algop from hashfile_checksum_valid()\n\n  * The next patch implements the new 'unsafe_hash_algo()' function at\n    the heart of this series' approach:\n\n      hash.h: introduce `unsafe_hash_algo()`\n\n  * The next two patches convert existing callers to use the new\n    'unsafe_hash_algo()' function, instead of switching between safe and\n    unsafe_ variants of individual functions:\n\n      csum-file.c: use unsafe_hash_algo()\n      t/helper/test-hash.c: use unsafe_hash_algo()\n\n  * The final patch drops the unsafe_ function variants following all\n    callers being converted to use the new pattern:\n\n      hash.h: drop unsafe_ function variants\n\nThanks in advance for your review!\n\n[1]: https://lore.kernel.org/git/20241230-pks-meson-sha1-unsafe-v1-0-efb276e171f5@pks.im/\n[2]: https://lore.kernel.org/git/20241107013915.GA961214@coredump.intra.peff.net/\n\nTaylor Blau (8):\n  t/helper/test-tool: implement sha1-unsafe helper\n  csum-file: store the hash algorithm as a struct field\n  csum-file.c: extract algop from hashfile_checksum_valid()\n  hash.h: introduce `unsafe_hash_algo()`\n  csum-file.c: use unsafe_hash_algo()\n  t/helper/test-hash.c: use unsafe_hash_algo()\n  csum-file: introduce hashfile_checkpoint_init()\n  hash.h: drop unsafe_ function variants\n\n builtin/fast-import.c  |  2 +-\n bulk-checkin.c         |  9 ++++++---\n csum-file.c            | 40 +++++++++++++++++++++++++---------------\n csum-file.h            |  2 ++\n hash.h                 | 28 ++++++++++++----------------\n object-file.c          | 41 ++++++++++++++++++++++++++---------------\n t/helper/test-hash.c   |  4 +++-\n t/helper/test-sha1.c   |  7 ++++++-\n t/helper/test-sha1.sh  | 38 ++++++++++++++++++++++----------------\n t/helper/test-sha256.c |  2 +-\n t/helper/test-tool.c   |  1 +\n t/helper/test-tool.h   |  3 ++-\n 12 files changed, 107 insertions(+), 70 deletions(-)\n\nRange-diff against v2:\n1:  4c1523a04f1 = 1:  ae6b8c75294 t/helper/test-tool: implement sha1-unsafe helper\n2:  99cc44895b5 ! 2:  2b79c76e471 csum-file: store the hash algorithm as a struct field\n    @@ Commit message\n         csum-file: store the hash algorithm as a struct field\n     \n         Throughout the hashfile API, we rely on a reference to 'the_hash_algo',\n    -    and call its _usnafe function variants directly.\n    +    and call its _unsafe function variants directly.\n     \n         Prepare for a future change where we may use a different 'git_hash_algo'\n         pointer (instead of just relying on 'the_hash_algo' throughout) by\n3:  1ffab2f8289 = 3:  d7deb3f338e csum-file.c: extract algop from hashfile_checksum_valid()\n4:  99dcbe2e716 ! 4:  b6b2bb2714f hash.h: introduce `unsafe_hash_algo()`\n    @@ Commit message\n             if (unsafe)\n               algop = unsafe_hash_algo(algop);\n     \n    -        the_hash_algo->init_fn(...);\n    -        the_hash_algo->update_fn(...);\n    -        the_hash_algo->final_fn(...);\n    +        algop->init_fn(...);\n    +        algop->update_fn(...);\n    +        algop->final_fn(...);\n     \n         This removes the existing shortcoming by no longer forcing the caller to\n         \"remember\" which variant of the hash functions it wants to call, only to\n    @@ Commit message\n         functions, this too will go away after subsequent commits remove all\n         direct calls to the unsafe_ variants.\n     \n    +    Note that hash_algo_by_ptr() needs an adjustment to allow passing in the\n    +    unsafe variant of a hash function. All other query functions on the\n    +    hash_algos array will continue to return the safe variants of any\n    +    function.\n    +\n         Suggested-by: Jeff King <peff@peff.net>\n         Signed-off-by: Taylor Blau <me@ttaylorr.com>\n     \n    @@ hash.h: struct git_hash_algo {\n      };\n      extern const struct git_hash_algo hash_algos[GIT_HASH_NALGOS];\n      \n    -@@ hash.h: static inline int hash_algo_by_ptr(const struct git_hash_algo *p)\n    - \treturn p - hash_algos;\n    +@@ hash.h: int hash_algo_by_length(int len);\n    + /* Identical, except for a pointer to struct git_hash_algo. */\n    + static inline int hash_algo_by_ptr(const struct git_hash_algo *p)\n    + {\n    +-\treturn p - hash_algos;\n    ++\tsize_t i;\n    ++\tfor (i = 0; i < GIT_HASH_NALGOS; i++) {\n    ++\t\tconst struct git_hash_algo *algop = &hash_algos[i];\n    ++\t\tif (p == algop || (algop->unsafe && p == algop->unsafe))\n    ++\t\t\treturn i;\n    ++\t}\n    ++\treturn GIT_HASH_UNKNOWN;\n      }\n      \n     +const struct git_hash_algo *unsafe_hash_algo(const struct git_hash_algo *algop);\n5:  2dcc2aa6803 = 5:  ca67de80971 csum-file.c: use unsafe_hash_algo()\n6:  a2b9ef03080 = 6:  21b175b07ff t/helper/test-hash.c: use unsafe_hash_algo()\n7:  94c07fd8a55 = 7:  850d4f407db csum-file: introduce hashfile_checkpoint_init()\n8:  f5579883816 ! 8:  0c4d006e6e8 hash.h: drop unsafe_ function variants\n    @@ Commit message\n     \n         to\n     \n    -        unsafe_hash_algo(the_hash_algo)->unsafe_init_fn();\n    +        unsafe_hash_algo(the_hash_algo)->init_fn();\n     \n         and similar, we can remove the scaffolding for the unsafe_ function\n         variants and force callers to use the new unsafe_hash_algo() mechanic\n\nbase-commit: 14650065b76b28d3cfa9453356ac5669b19e706e\n-- \n2.48.0.rc2.35.g0c4d006e6e8\n"},{"id":"510841","messageId":"ae6b8c75294933fbf1f9112a6d22b8730a454ccc.1737151386.git.me@ttaylorr.com","threadId":"62522","inReplyTo":"cover.1737151386.git.me@ttaylorr.com","subject":"[PATCH v3 1/8] t/helper/test-tool: implement sha1-unsafe helper","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-01-17T22:03:10Z","receivedAt":"2025-01-17T22:03:13Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"With the new \"unsafe\" SHA-1 build knob, it is convenient to have a\ntest-tool that can exercise Git's unsafe SHA-1 wrappers for testing,\nsimilar to 't/helper/test-tool sha1'.\n\nImplement that helper by altering the implementation of that test-tool\n(in cmd_hash_impl(), which is generic and parameterized over different\nhash functions) to conditionally run the unsafe variants of the chosen\nhash function, and expose the new behavior via a new 'sha1-unsafe' test\nhelper.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n t/helper/test-hash.c   | 17 +++++++++++++----\n t/helper/test-sha1.c   |  7 ++++++-\n t/helper/test-sha1.sh  | 38 ++++++++++++++++++++++----------------\n t/helper/test-sha256.c |  2 +-\n t/helper/test-tool.c   |  1 +\n t/helper/test-tool.h   |  3 ++-\n 6 files changed, 45 insertions(+), 23 deletions(-)\n\ndiff --git a/t/helper/test-hash.c b/t/helper/test-hash.c\nindex 45d829c908f..d0ee668df95 100644\n--- a/t/helper/test-hash.c\n+++ b/t/helper/test-hash.c\n@@ -1,7 +1,7 @@\n #include \"test-tool.h\"\n #include \"hex.h\"\n \n-int cmd_hash_impl(int ac, const char **av, int algo)\n+int cmd_hash_impl(int ac, const char **av, int algo, int unsafe)\n {\n \tgit_hash_ctx ctx;\n \tunsigned char hash[GIT_MAX_HEXSZ];\n@@ -27,7 +27,10 @@ int cmd_hash_impl(int ac, const char **av, int algo)\n \t\t\tdie(\"OOPS\");\n \t}\n \n-\talgop->init_fn(&ctx);\n+\tif (unsafe)\n+\t\talgop->unsafe_init_fn(&ctx);\n+\telse\n+\t\talgop->init_fn(&ctx);\n \n \twhile (1) {\n \t\tssize_t sz, this_sz;\n@@ -46,9 +49,15 @@ int cmd_hash_impl(int ac, const char **av, int algo)\n \t\t}\n \t\tif (this_sz == 0)\n \t\t\tbreak;\n-\t\talgop->update_fn(&ctx, buffer, this_sz);\n+\t\tif (unsafe)\n+\t\t\talgop->unsafe_update_fn(&ctx, buffer, this_sz);\n+\t\telse\n+\t\t\talgop->update_fn(&ctx, buffer, this_sz);\n \t}\n-\talgop->final_fn(hash, &ctx);\n+\tif (unsafe)\n+\t\talgop->unsafe_final_fn(hash, &ctx);\n+\telse\n+\t\talgop->final_fn(hash, &ctx);\n \n \tif (binary)\n \t\tfwrite(hash, 1, algop->rawsz, stdout);\ndiff --git a/t/helper/test-sha1.c b/t/helper/test-sha1.c\nindex e60d000c039..349540c4df8 100644\n--- a/t/helper/test-sha1.c\n+++ b/t/helper/test-sha1.c\n@@ -3,7 +3,7 @@\n \n int cmd__sha1(int ac, const char **av)\n {\n-\treturn cmd_hash_impl(ac, av, GIT_HASH_SHA1);\n+\treturn cmd_hash_impl(ac, av, GIT_HASH_SHA1, 0);\n }\n \n int cmd__sha1_is_sha1dc(int argc UNUSED, const char **argv UNUSED)\n@@ -13,3 +13,8 @@ int cmd__sha1_is_sha1dc(int argc UNUSED, const char **argv UNUSED)\n #endif\n \treturn 1;\n }\n+\n+int cmd__sha1_unsafe(int ac, const char **av)\n+{\n+\treturn cmd_hash_impl(ac, av, GIT_HASH_SHA1, 1);\n+}\ndiff --git a/t/helper/test-sha1.sh b/t/helper/test-sha1.sh\nindex 84594885c70..bf387d3db14 100755\n--- a/t/helper/test-sha1.sh\n+++ b/t/helper/test-sha1.sh\n@@ -3,25 +3,31 @@\n dd if=/dev/zero bs=1048576 count=100 2>/dev/null |\n /usr/bin/time t/helper/test-tool sha1 >/dev/null\n \n+dd if=/dev/zero bs=1048576 count=100 2>/dev/null |\n+/usr/bin/time t/helper/test-tool sha1-unsafe >/dev/null\n+\n while read expect cnt pfx\n do\n \tcase \"$expect\" in '#'*) continue ;; esac\n-\tactual=$(\n-\t\t{\n-\t\t\ttest -z \"$pfx\" || echo \"$pfx\"\n-\t\t\tdd if=/dev/zero bs=1048576 count=$cnt 2>/dev/null |\n-\t\t\tperl -pe 'y/\\000/g/'\n-\t\t} | ./t/helper/test-tool sha1 $cnt\n-\t)\n-\tif test \"$expect\" = \"$actual\"\n-\tthen\n-\t\techo \"OK: $expect $cnt $pfx\"\n-\telse\n-\t\techo >&2 \"OOPS: $cnt\"\n-\t\techo >&2 \"expect: $expect\"\n-\t\techo >&2 \"actual: $actual\"\n-\t\texit 1\n-\tfi\n+\tfor sha1 in sha1 sha1-unsafe\n+\tdo\n+\t\tactual=$(\n+\t\t\t{\n+\t\t\t\ttest -z \"$pfx\" || echo \"$pfx\"\n+\t\t\t\tdd if=/dev/zero bs=1048576 count=$cnt 2>/dev/null |\n+\t\t\t\tperl -pe 'y/\\000/g/'\n+\t\t\t} | ./t/helper/test-tool $sha1 $cnt\n+\t\t)\n+\t\tif test \"$expect\" = \"$actual\"\n+\t\tthen\n+\t\t\techo \"OK ($sha1): $expect $cnt $pfx\"\n+\t\telse\n+\t\t\techo >&2 \"OOPS ($sha1): $cnt\"\n+\t\t\techo >&2 \"expect ($sha1): $expect\"\n+\t\t\techo >&2 \"actual ($sha1): $actual\"\n+\t\t\texit 1\n+\t\tfi\n+\tdone\n done <<EOF\n da39a3ee5e6b4b0d3255bfef95601890afd80709 0\n 3f786850e387550fdab836ed7e6dc881de23001b 0 a\ndiff --git a/t/helper/test-sha256.c b/t/helper/test-sha256.c\nindex 2fb20438f3c..7fd0aa1fcd3 100644\n--- a/t/helper/test-sha256.c\n+++ b/t/helper/test-sha256.c\n@@ -3,5 +3,5 @@\n \n int cmd__sha256(int ac, const char **av)\n {\n-\treturn cmd_hash_impl(ac, av, GIT_HASH_SHA256);\n+\treturn cmd_hash_impl(ac, av, GIT_HASH_SHA256, 0);\n }\ndiff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\nindex 4a7aa993ba9..958452ef12e 100644\n--- a/t/helper/test-tool.c\n+++ b/t/helper/test-tool.c\n@@ -70,6 +70,7 @@ static struct test_cmd cmds[] = {\n \t{ \"serve-v2\", cmd__serve_v2 },\n \t{ \"sha1\", cmd__sha1 },\n \t{ \"sha1-is-sha1dc\", cmd__sha1_is_sha1dc },\n+\t{ \"sha1-unsafe\", cmd__sha1_unsafe },\n \t{ \"sha256\", cmd__sha256 },\n \t{ \"sigchain\", cmd__sigchain },\n \t{ \"simple-ipc\", cmd__simple_ipc },\ndiff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\nindex 21802ac27da..24149edd414 100644\n--- a/t/helper/test-tool.h\n+++ b/t/helper/test-tool.h\n@@ -63,6 +63,7 @@ int cmd__scrap_cache_tree(int argc, const char **argv);\n int cmd__serve_v2(int argc, const char **argv);\n int cmd__sha1(int argc, const char **argv);\n int cmd__sha1_is_sha1dc(int argc, const char **argv);\n+int cmd__sha1_unsafe(int argc, const char **argv);\n int cmd__sha256(int argc, const char **argv);\n int cmd__sigchain(int argc, const char **argv);\n int cmd__simple_ipc(int argc, const char **argv);\n@@ -81,6 +82,6 @@ int cmd__windows_named_pipe(int argc, const char **argv);\n #endif\n int cmd__write_cache(int argc, const char **argv);\n \n-int cmd_hash_impl(int ac, const char **av, int algo);\n+int cmd_hash_impl(int ac, const char **av, int algo, int unsafe);\n \n #endif\n-- \n2.48.0.rc2.35.g0c4d006e6e8\n\n"},{"id":"510842","messageId":"2b79c76e4714145057a526876b9ad581c497af0f.1737151386.git.me@ttaylorr.com","threadId":"62522","inReplyTo":"cover.1737151386.git.me@ttaylorr.com","subject":"[PATCH v3 2/8] csum-file: store the hash algorithm as a struct field","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-01-17T22:03:13Z","receivedAt":"2025-01-17T22:03:16Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Throughout the hashfile API, we rely on a reference to 'the_hash_algo',\nand call its _unsafe function variants directly.\n\nPrepare for a future change where we may use a different 'git_hash_algo'\npointer (instead of just relying on 'the_hash_algo' throughout) by\nmaking the 'git_hash_algo' pointer a member of the 'hashfile' structure\nitself.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n csum-file.c | 20 +++++++++++---------\n csum-file.h |  1 +\n 2 files changed, 12 insertions(+), 9 deletions(-)\n\ndiff --git a/csum-file.c b/csum-file.c\nindex 5716016e12e..b28cd047e3f 100644\n--- a/csum-file.c\n+++ b/csum-file.c\n@@ -50,7 +50,7 @@ void hashflush(struct hashfile *f)\n \n \tif (offset) {\n \t\tif (!f->skip_hash)\n-\t\t\tthe_hash_algo->unsafe_update_fn(&f->ctx, f->buffer, offset);\n+\t\t\tf->algop->unsafe_update_fn(&f->ctx, f->buffer, offset);\n \t\tflush(f, f->buffer, offset);\n \t\tf->offset = 0;\n \t}\n@@ -71,14 +71,14 @@ int finalize_hashfile(struct hashfile *f, unsigned char *result,\n \thashflush(f);\n \n \tif (f->skip_hash)\n-\t\thashclr(f->buffer, the_repository->hash_algo);\n+\t\thashclr(f->buffer, f->algop);\n \telse\n-\t\tthe_hash_algo->unsafe_final_fn(f->buffer, &f->ctx);\n+\t\tf->algop->unsafe_final_fn(f->buffer, &f->ctx);\n \n \tif (result)\n-\t\thashcpy(result, f->buffer, the_repository->hash_algo);\n+\t\thashcpy(result, f->buffer, f->algop);\n \tif (flags & CSUM_HASH_IN_STREAM)\n-\t\tflush(f, f->buffer, the_hash_algo->rawsz);\n+\t\tflush(f, f->buffer, f->algop->rawsz);\n \tif (flags & CSUM_FSYNC)\n \t\tfsync_component_or_die(component, f->fd, f->name);\n \tif (flags & CSUM_CLOSE) {\n@@ -128,7 +128,7 @@ void hashwrite(struct hashfile *f, const void *buf, unsigned int count)\n \t\t\t * f->offset is necessarily zero.\n \t\t\t */\n \t\t\tif (!f->skip_hash)\n-\t\t\t\tthe_hash_algo->unsafe_update_fn(&f->ctx, buf, nr);\n+\t\t\t\tf->algop->unsafe_update_fn(&f->ctx, buf, nr);\n \t\t\tflush(f, buf, nr);\n \t\t} else {\n \t\t\t/*\n@@ -174,7 +174,9 @@ static struct hashfile *hashfd_internal(int fd, const char *name,\n \tf->name = name;\n \tf->do_crc = 0;\n \tf->skip_hash = 0;\n-\tthe_hash_algo->unsafe_init_fn(&f->ctx);\n+\n+\tf->algop = the_hash_algo;\n+\tf->algop->unsafe_init_fn(&f->ctx);\n \n \tf->buffer_len = buffer_len;\n \tf->buffer = xmalloc(buffer_len);\n@@ -208,7 +210,7 @@ void hashfile_checkpoint(struct hashfile *f, struct hashfile_checkpoint *checkpo\n {\n \thashflush(f);\n \tcheckpoint->offset = f->total;\n-\tthe_hash_algo->unsafe_clone_fn(&checkpoint->ctx, &f->ctx);\n+\tf->algop->unsafe_clone_fn(&checkpoint->ctx, &f->ctx);\n }\n \n int hashfile_truncate(struct hashfile *f, struct hashfile_checkpoint *checkpoint)\n@@ -219,7 +221,7 @@ int hashfile_truncate(struct hashfile *f, struct hashfile_checkpoint *checkpoint\n \t    lseek(f->fd, offset, SEEK_SET) != offset)\n \t\treturn -1;\n \tf->total = offset;\n-\tthe_hash_algo->unsafe_clone_fn(&f->ctx, &checkpoint->ctx);\n+\tf->algop->unsafe_clone_fn(&f->ctx, &checkpoint->ctx);\n \tf->offset = 0; /* hashflush() was called in checkpoint */\n \treturn 0;\n }\ndiff --git a/csum-file.h b/csum-file.h\nindex 7c73da0a40a..2b45f4673a2 100644\n--- a/csum-file.h\n+++ b/csum-file.h\n@@ -20,6 +20,7 @@ struct hashfile {\n \tsize_t buffer_len;\n \tunsigned char *buffer;\n \tunsigned char *check_buffer;\n+\tconst struct git_hash_algo *algop;\n \n \t/**\n \t * If non-zero, skip_hash indicates that we should\n-- \n2.48.0.rc2.35.g0c4d006e6e8\n\n"},{"id":"510843","messageId":"d7deb3f338eaa09e39d205d028704b8d5ef47f16.1737151386.git.me@ttaylorr.com","threadId":"62522","inReplyTo":"cover.1737151386.git.me@ttaylorr.com","subject":"[PATCH v3 3/8] csum-file.c: extract algop from hashfile_checksum_valid()","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-01-17T22:03:16Z","receivedAt":"2025-01-17T22:03:19Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Perform a similar transformation as in the previous commit, but focused\ninstead on hashfile_checksum_valid(). This function does not work with a\nhashfile structure itself, and instead validates the raw contents of a\nfile written using the hashfile API.\n\nWe'll want to be prepared for a similar change to this function in the\nfuture, so prepare ourselves for that by extracting 'the_hash_algo' into\nits own field for use within this function.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n csum-file.c | 13 +++++++------\n 1 file changed, 7 insertions(+), 6 deletions(-)\n\ndiff --git a/csum-file.c b/csum-file.c\nindex b28cd047e3f..7a71121e340 100644\n--- a/csum-file.c\n+++ b/csum-file.c\n@@ -242,14 +242,15 @@ int hashfile_checksum_valid(const unsigned char *data, size_t total_len)\n {\n \tunsigned char got[GIT_MAX_RAWSZ];\n \tgit_hash_ctx ctx;\n-\tsize_t data_len = total_len - the_hash_algo->rawsz;\n+\tconst struct git_hash_algo *algop = the_hash_algo;\n+\tsize_t data_len = total_len - algop->rawsz;\n \n-\tif (total_len < the_hash_algo->rawsz)\n+\tif (total_len < algop->rawsz)\n \t\treturn 0; /* say \"too short\"? */\n \n-\tthe_hash_algo->unsafe_init_fn(&ctx);\n-\tthe_hash_algo->unsafe_update_fn(&ctx, data, data_len);\n-\tthe_hash_algo->unsafe_final_fn(got, &ctx);\n+\talgop->unsafe_init_fn(&ctx);\n+\talgop->unsafe_update_fn(&ctx, data, data_len);\n+\talgop->unsafe_final_fn(got, &ctx);\n \n-\treturn hasheq(got, data + data_len, the_repository->hash_algo);\n+\treturn hasheq(got, data + data_len, algop);\n }\n-- \n2.48.0.rc2.35.g0c4d006e6e8\n\n"},{"id":"510844","messageId":"b6b2bb2714f8ba9b483b04a2b734a5cd36230eb7.1737151386.git.me@ttaylorr.com","threadId":"62522","inReplyTo":"cover.1737151386.git.me@ttaylorr.com","subject":"[PATCH v3 4/8] hash.h: introduce `unsafe_hash_algo()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-01-17T22:03:19Z","receivedAt":"2025-01-17T22:03:22Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In 253ed9ecff (hash.h: scaffolding for _unsafe hashing variants,\n2024-09-26), we introduced \"unsafe\" variants of the SHA-1 hashing\nfunctions by introducing new functions like \"unsafe_init_fn()\" and so\non.\n\nThis approach has a major shortcoming that callers must remember to\nconsistently use one variant or the other. Failing to consistently use\n(or not use) the unsafe variants can lead to crashes at best, or subtle\nmemory corruption issues at worst.\n\nIn the hashfile API, this isn't difficult to achieve, but verifying that\nall callers consistently use the unsafe variants is somewhat of a chore\ngiven how spread out all of the callers are. In the sha1 and sha1-unsafe\ntest helpers, all of the calls to various hash functions are guarded by\nan \"if (unsafe)\" conditional, which is repetitive and cumbersome.\n\nAddress these issues by introducing a new pattern whereby one\n'git_hash_algo' can return a pointer to another 'git_hash_algo' that\nrepresents the unsafe version of itself. So instead of having something\nlike:\n\n    if (unsafe)\n      the_hash_algo->init_fn(...);\n      the_hash_algo->update_fn(...);\n      the_hash_algo->final_fn(...);\n    else\n      the_hash_algo->unsafe_init_fn(...);\n      the_hash_algo->unsafe_update_fn(...);\n      the_hash_algo->unsafe_final_fn(...);\n\nwe can instead write:\n\n    struct git_hash_algo *algop = the_hash_algo;\n    if (unsafe)\n      algop = unsafe_hash_algo(algop);\n\n    algop->init_fn(...);\n    algop->update_fn(...);\n    algop->final_fn(...);\n\nThis removes the existing shortcoming by no longer forcing the caller to\n\"remember\" which variant of the hash functions it wants to call, only to\nhold onto a 'struct git_hash_algo' pointer that is initialized once.\n\nSimilarly, while there currently is still a way to \"mix\" safe and unsafe\nfunctions, this too will go away after subsequent commits remove all\ndirect calls to the unsafe_ variants.\n\nNote that hash_algo_by_ptr() needs an adjustment to allow passing in the\nunsafe variant of a hash function. All other query functions on the\nhash_algos array will continue to return the safe variants of any\nfunction.\n\nSuggested-by: Jeff King <peff@peff.net>\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n hash.h        | 13 ++++++++++++-\n object-file.c | 26 ++++++++++++++++++++++++++\n 2 files changed, 38 insertions(+), 1 deletion(-)\n\ndiff --git a/hash.h b/hash.h\nindex 756166ce5e8..a68a2b6a161 100644\n--- a/hash.h\n+++ b/hash.h\n@@ -305,6 +305,9 @@ struct git_hash_algo {\n \n \t/* The all-zeros OID. */\n \tconst struct object_id *null_oid;\n+\n+\t/* The unsafe variant of this hash function, if one exists. */\n+\tconst struct git_hash_algo *unsafe;\n };\n extern const struct git_hash_algo hash_algos[GIT_HASH_NALGOS];\n \n@@ -320,9 +323,17 @@ int hash_algo_by_length(int len);\n /* Identical, except for a pointer to struct git_hash_algo. */\n static inline int hash_algo_by_ptr(const struct git_hash_algo *p)\n {\n-\treturn p - hash_algos;\n+\tsize_t i;\n+\tfor (i = 0; i < GIT_HASH_NALGOS; i++) {\n+\t\tconst struct git_hash_algo *algop = &hash_algos[i];\n+\t\tif (p == algop || (algop->unsafe && p == algop->unsafe))\n+\t\t\treturn i;\n+\t}\n+\treturn GIT_HASH_UNKNOWN;\n }\n \n+const struct git_hash_algo *unsafe_hash_algo(const struct git_hash_algo *algop);\n+\n const struct object_id *null_oid(void);\n \n static inline int hashcmp(const unsigned char *sha1, const unsigned char *sha2, const struct git_hash_algo *algop)\ndiff --git a/object-file.c b/object-file.c\nindex 5b792b3dd42..43efa4ca361 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -202,6 +202,22 @@ static void git_hash_unknown_final_oid(struct object_id *oid UNUSED,\n \tBUG(\"trying to finalize unknown hash\");\n }\n \n+static const struct git_hash_algo sha1_unsafe_algo = {\n+\t.name = \"sha1\",\n+\t.format_id = GIT_SHA1_FORMAT_ID,\n+\t.rawsz = GIT_SHA1_RAWSZ,\n+\t.hexsz = GIT_SHA1_HEXSZ,\n+\t.blksz = GIT_SHA1_BLKSZ,\n+\t.init_fn = git_hash_sha1_init_unsafe,\n+\t.clone_fn = git_hash_sha1_clone_unsafe,\n+\t.update_fn = git_hash_sha1_update_unsafe,\n+\t.final_fn = git_hash_sha1_final_unsafe,\n+\t.final_oid_fn = git_hash_sha1_final_oid_unsafe,\n+\t.empty_tree = &empty_tree_oid,\n+\t.empty_blob = &empty_blob_oid,\n+\t.null_oid = &null_oid_sha1,\n+};\n+\n const struct git_hash_algo hash_algos[GIT_HASH_NALGOS] = {\n \t{\n \t\t.name = NULL,\n@@ -239,6 +255,7 @@ const struct git_hash_algo hash_algos[GIT_HASH_NALGOS] = {\n \t\t.unsafe_update_fn = git_hash_sha1_update_unsafe,\n \t\t.unsafe_final_fn = git_hash_sha1_final_unsafe,\n \t\t.unsafe_final_oid_fn = git_hash_sha1_final_oid_unsafe,\n+\t\t.unsafe = &sha1_unsafe_algo,\n \t\t.empty_tree = &empty_tree_oid,\n \t\t.empty_blob = &empty_blob_oid,\n \t\t.null_oid = &null_oid_sha1,\n@@ -305,6 +322,15 @@ int hash_algo_by_length(int len)\n \treturn GIT_HASH_UNKNOWN;\n }\n \n+const struct git_hash_algo *unsafe_hash_algo(const struct git_hash_algo *algop)\n+{\n+\t/* If we have a faster \"unsafe\" implementation, use that. */\n+\tif (algop->unsafe)\n+\t\treturn algop->unsafe;\n+\t/* Otherwise use the default one. */\n+\treturn algop;\n+}\n+\n /*\n  * This is meant to hold a *small* number of objects that you would\n  * want repo_read_object_file() to be able to return, but yet you do not want\n-- \n2.48.0.rc2.35.g0c4d006e6e8\n\n"},{"id":"510845","messageId":"ca67de80971fc1b8b77241a647e246e1cd4e58cf.1737151386.git.me@ttaylorr.com","threadId":"62522","inReplyTo":"cover.1737151386.git.me@ttaylorr.com","subject":"[PATCH v3 5/8] csum-file.c: use unsafe_hash_algo()","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-01-17T22:03:23Z","receivedAt":"2025-01-17T22:03:25Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Instead of calling the unsafe_ hash function variants directly, make use\nof the shared 'algop' pointer by initializing it to:\n\n    f->algop = unsafe_hash_algo(the_hash_algo);\n\n, thus making all calls use the unsafe variants directly.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n csum-file.c | 22 +++++++++++-----------\n 1 file changed, 11 insertions(+), 11 deletions(-)\n\ndiff --git a/csum-file.c b/csum-file.c\nindex 7a71121e340..ebffc80ef71 100644\n--- a/csum-file.c\n+++ b/csum-file.c\n@@ -50,7 +50,7 @@ void hashflush(struct hashfile *f)\n \n \tif (offset) {\n \t\tif (!f->skip_hash)\n-\t\t\tf->algop->unsafe_update_fn(&f->ctx, f->buffer, offset);\n+\t\t\tf->algop->update_fn(&f->ctx, f->buffer, offset);\n \t\tflush(f, f->buffer, offset);\n \t\tf->offset = 0;\n \t}\n@@ -73,7 +73,7 @@ int finalize_hashfile(struct hashfile *f, unsigned char *result,\n \tif (f->skip_hash)\n \t\thashclr(f->buffer, f->algop);\n \telse\n-\t\tf->algop->unsafe_final_fn(f->buffer, &f->ctx);\n+\t\tf->algop->final_fn(f->buffer, &f->ctx);\n \n \tif (result)\n \t\thashcpy(result, f->buffer, f->algop);\n@@ -128,7 +128,7 @@ void hashwrite(struct hashfile *f, const void *buf, unsigned int count)\n \t\t\t * f->offset is necessarily zero.\n \t\t\t */\n \t\t\tif (!f->skip_hash)\n-\t\t\t\tf->algop->unsafe_update_fn(&f->ctx, buf, nr);\n+\t\t\t\tf->algop->update_fn(&f->ctx, buf, nr);\n \t\t\tflush(f, buf, nr);\n \t\t} else {\n \t\t\t/*\n@@ -175,8 +175,8 @@ static struct hashfile *hashfd_internal(int fd, const char *name,\n \tf->do_crc = 0;\n \tf->skip_hash = 0;\n \n-\tf->algop = the_hash_algo;\n-\tf->algop->unsafe_init_fn(&f->ctx);\n+\tf->algop = unsafe_hash_algo(the_hash_algo);\n+\tf->algop->init_fn(&f->ctx);\n \n \tf->buffer_len = buffer_len;\n \tf->buffer = xmalloc(buffer_len);\n@@ -210,7 +210,7 @@ void hashfile_checkpoint(struct hashfile *f, struct hashfile_checkpoint *checkpo\n {\n \thashflush(f);\n \tcheckpoint->offset = f->total;\n-\tf->algop->unsafe_clone_fn(&checkpoint->ctx, &f->ctx);\n+\tf->algop->clone_fn(&checkpoint->ctx, &f->ctx);\n }\n \n int hashfile_truncate(struct hashfile *f, struct hashfile_checkpoint *checkpoint)\n@@ -221,7 +221,7 @@ int hashfile_truncate(struct hashfile *f, struct hashfile_checkpoint *checkpoint\n \t    lseek(f->fd, offset, SEEK_SET) != offset)\n \t\treturn -1;\n \tf->total = offset;\n-\tf->algop->unsafe_clone_fn(&f->ctx, &checkpoint->ctx);\n+\tf->algop->clone_fn(&f->ctx, &checkpoint->ctx);\n \tf->offset = 0; /* hashflush() was called in checkpoint */\n \treturn 0;\n }\n@@ -242,15 +242,15 @@ int hashfile_checksum_valid(const unsigned char *data, size_t total_len)\n {\n \tunsigned char got[GIT_MAX_RAWSZ];\n \tgit_hash_ctx ctx;\n-\tconst struct git_hash_algo *algop = the_hash_algo;\n+\tconst struct git_hash_algo *algop = unsafe_hash_algo(the_hash_algo);\n \tsize_t data_len = total_len - algop->rawsz;\n \n \tif (total_len < algop->rawsz)\n \t\treturn 0; /* say \"too short\"? */\n \n-\talgop->unsafe_init_fn(&ctx);\n-\talgop->unsafe_update_fn(&ctx, data, data_len);\n-\talgop->unsafe_final_fn(got, &ctx);\n+\talgop->init_fn(&ctx);\n+\talgop->update_fn(&ctx, data, data_len);\n+\talgop->final_fn(got, &ctx);\n \n \treturn hasheq(got, data + data_len, algop);\n }\n-- \n2.48.0.rc2.35.g0c4d006e6e8\n\n"},{"id":"510846","messageId":"21b175b07ff46471acdaa1afe5fa8b0c2749248b.1737151386.git.me@ttaylorr.com","threadId":"62522","inReplyTo":"cover.1737151386.git.me@ttaylorr.com","subject":"[PATCH v3 6/8] t/helper/test-hash.c: use unsafe_hash_algo()","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-01-17T22:03:26Z","receivedAt":"2025-01-17T22:03:28Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Remove a series of conditionals within the shared cmd_hash_impl() helper\nthat powers the 'sha1' and 'sha1-unsafe' helpers.\n\nInstead, replace them with a single conditional that transforms the\nspecified hash algorithm into its unsafe variant. Then all subsequent\ncalls can directly use whatever function it wants to call without having\nto decide between the safe and unsafe variants.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n t/helper/test-hash.c | 17 +++++------------\n 1 file changed, 5 insertions(+), 12 deletions(-)\n\ndiff --git a/t/helper/test-hash.c b/t/helper/test-hash.c\nindex d0ee668df95..aa82638c621 100644\n--- a/t/helper/test-hash.c\n+++ b/t/helper/test-hash.c\n@@ -9,6 +9,8 @@ int cmd_hash_impl(int ac, const char **av, int algo, int unsafe)\n \tint binary = 0;\n \tchar *buffer;\n \tconst struct git_hash_algo *algop = &hash_algos[algo];\n+\tif (unsafe)\n+\t\talgop = unsafe_hash_algo(algop);\n \n \tif (ac == 2) {\n \t\tif (!strcmp(av[1], \"-b\"))\n@@ -27,10 +29,7 @@ int cmd_hash_impl(int ac, const char **av, int algo, int unsafe)\n \t\t\tdie(\"OOPS\");\n \t}\n \n-\tif (unsafe)\n-\t\talgop->unsafe_init_fn(&ctx);\n-\telse\n-\t\talgop->init_fn(&ctx);\n+\talgop->init_fn(&ctx);\n \n \twhile (1) {\n \t\tssize_t sz, this_sz;\n@@ -49,15 +48,9 @@ int cmd_hash_impl(int ac, const char **av, int algo, int unsafe)\n \t\t}\n \t\tif (this_sz == 0)\n \t\t\tbreak;\n-\t\tif (unsafe)\n-\t\t\talgop->unsafe_update_fn(&ctx, buffer, this_sz);\n-\t\telse\n-\t\t\talgop->update_fn(&ctx, buffer, this_sz);\n+\t\talgop->update_fn(&ctx, buffer, this_sz);\n \t}\n-\tif (unsafe)\n-\t\talgop->unsafe_final_fn(hash, &ctx);\n-\telse\n-\t\talgop->final_fn(hash, &ctx);\n+\talgop->final_fn(hash, &ctx);\n \n \tif (binary)\n \t\tfwrite(hash, 1, algop->rawsz, stdout);\n-- \n2.48.0.rc2.35.g0c4d006e6e8\n\n"},{"id":"510847","messageId":"850d4f407dbb3999e194bf094d0fd7b95a9c29d8.1737151386.git.me@ttaylorr.com","threadId":"62522","inReplyTo":"cover.1737151386.git.me@ttaylorr.com","subject":"[PATCH v3 7/8] csum-file: introduce hashfile_checkpoint_init()","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-01-17T22:03:29Z","receivedAt":"2025-01-17T22:03:32Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In 106140a99f (builtin/fast-import: fix segfault with unsafe SHA1\nbackend, 2024-12-30) and 9218c0bfe1 (bulk-checkin: fix segfault with\nunsafe SHA1 backend, 2024-12-30), we observed the effects of failing to\ninitialize a hashfile_checkpoint with the same hash function\nimplementation as is used by the hashfile it is used to checkpoint.\n\nWhile both 106140a99f and 9218c0bfe1 work around the immediate crash,\nchanging the hash function implementation within the hashfile API to,\nfor example, the non-unsafe variant would re-introduce the crash. This\nis a result of the tight coupling between initializing hashfiles and\nhashfile_checkpoints.\n\nIntroduce and use a new function which ensures that both parts of a\nhashfile and hashfile_checkpoint pair use the same hash function\nimplementation to avoid such crashes.\n\nA few things worth noting:\n\n  - In the change to builtin/fast-import.c::stream_blob(), we can see\n    that by removing the explicit reference to\n    'the_hash_algo->unsafe_init_fn()', we are hardened against the\n    hashfile API changing away from the_hash_algo (or its unsafe\n    variant) in the future.\n\n  - The bulk-checkin code no longer needs to explicitly zero-initialize\n    the hashfile_checkpoint, since it is now done as a result of calling\n    'hashfile_checkpoint_init()'.\n\n  - Also in the bulk-checkin code, we add an additional call to\n    prepare_to_stream() outside of the main loop in order to initialize\n    'state->f' so we know which hash function implementation to use when\n    calling 'hashfile_checkpoint_init()'.\n\n    This is OK, since subsequent 'prepare_to_stream()' calls are noops.\n    However, we only need to call 'prepare_to_stream()' when we have the\n    HASH_WRITE_OBJECT bit set in our flags. Without that bit, calling\n    'prepare_to_stream()' does not assign 'state->f', so we have nothing\n    to initialize.\n\n  - Other uses of the 'checkpoint' in 'deflate_blob_to_pack()' are\n    appropriately guarded.\n\nHelped-by: Patrick Steinhardt <ps@pks.im>\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n builtin/fast-import.c | 2 +-\n bulk-checkin.c        | 9 ++++++---\n csum-file.c           | 7 +++++++\n csum-file.h           | 1 +\n 4 files changed, 15 insertions(+), 4 deletions(-)\n\ndiff --git a/builtin/fast-import.c b/builtin/fast-import.c\nindex 0f86392761a..4a6c7ab52ac 100644\n--- a/builtin/fast-import.c\n+++ b/builtin/fast-import.c\n@@ -1106,7 +1106,7 @@ static void stream_blob(uintmax_t len, struct object_id *oidout, uintmax_t mark)\n \t\t|| (pack_size + PACK_SIZE_THRESHOLD + len) < pack_size)\n \t\tcycle_packfile();\n \n-\tthe_hash_algo->unsafe_init_fn(&checkpoint.ctx);\n+\thashfile_checkpoint_init(pack_file, &checkpoint);\n \thashfile_checkpoint(pack_file, &checkpoint);\n \toffset = checkpoint.offset;\n \ndiff --git a/bulk-checkin.c b/bulk-checkin.c\nindex 433070a3bda..892176d23d2 100644\n--- a/bulk-checkin.c\n+++ b/bulk-checkin.c\n@@ -261,7 +261,7 @@ static int deflate_blob_to_pack(struct bulk_checkin_packfile *state,\n \tgit_hash_ctx ctx;\n \tunsigned char obuf[16384];\n \tunsigned header_len;\n-\tstruct hashfile_checkpoint checkpoint = {0};\n+\tstruct hashfile_checkpoint checkpoint;\n \tstruct pack_idx_entry *idx = NULL;\n \n \tseekback = lseek(fd, 0, SEEK_CUR);\n@@ -272,12 +272,15 @@ static int deflate_blob_to_pack(struct bulk_checkin_packfile *state,\n \t\t\t\t\t  OBJ_BLOB, size);\n \tthe_hash_algo->init_fn(&ctx);\n \tthe_hash_algo->update_fn(&ctx, obuf, header_len);\n-\tthe_hash_algo->unsafe_init_fn(&checkpoint.ctx);\n \n \t/* Note: idx is non-NULL when we are writing */\n-\tif ((flags & HASH_WRITE_OBJECT) != 0)\n+\tif ((flags & HASH_WRITE_OBJECT) != 0) {\n \t\tCALLOC_ARRAY(idx, 1);\n \n+\t\tprepare_to_stream(state, flags);\n+\t\thashfile_checkpoint_init(state->f, &checkpoint);\n+\t}\n+\n \talready_hashed_to = 0;\n \n \twhile (1) {\ndiff --git a/csum-file.c b/csum-file.c\nindex ebffc80ef71..232121f415f 100644\n--- a/csum-file.c\n+++ b/csum-file.c\n@@ -206,6 +206,13 @@ struct hashfile *hashfd_throughput(int fd, const char *name, struct progress *tp\n \treturn hashfd_internal(fd, name, tp, 8 * 1024);\n }\n \n+void hashfile_checkpoint_init(struct hashfile *f,\n+\t\t\t      struct hashfile_checkpoint *checkpoint)\n+{\n+\tmemset(checkpoint, 0, sizeof(*checkpoint));\n+\tf->algop->init_fn(&checkpoint->ctx);\n+}\n+\n void hashfile_checkpoint(struct hashfile *f, struct hashfile_checkpoint *checkpoint)\n {\n \thashflush(f);\ndiff --git a/csum-file.h b/csum-file.h\nindex 2b45f4673a2..b7475f16c20 100644\n--- a/csum-file.h\n+++ b/csum-file.h\n@@ -36,6 +36,7 @@ struct hashfile_checkpoint {\n \tgit_hash_ctx ctx;\n };\n \n+void hashfile_checkpoint_init(struct hashfile *, struct hashfile_checkpoint *);\n void hashfile_checkpoint(struct hashfile *, struct hashfile_checkpoint *);\n int hashfile_truncate(struct hashfile *, struct hashfile_checkpoint *);\n \n-- \n2.48.0.rc2.35.g0c4d006e6e8\n\n"},{"id":"510848","messageId":"0c4d006e6e8005c10387d10fe0224aec5c5613e5.1737151386.git.me@ttaylorr.com","threadId":"62522","inReplyTo":"cover.1737151386.git.me@ttaylorr.com","subject":"[PATCH v3 8/8] hash.h: drop unsafe_ function variants","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-01-17T22:03:32Z","receivedAt":"2025-01-17T22:03:35Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Now that all callers have been converted from:\n\n    the_hash_algo->unsafe_init_fn();\n\nto\n\n    unsafe_hash_algo(the_hash_algo)->init_fn();\n\nand similar, we can remove the scaffolding for the unsafe_ function\nvariants and force callers to use the new unsafe_hash_algo() mechanic\ninstead.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n hash.h        | 15 ---------------\n object-file.c | 15 ---------------\n 2 files changed, 30 deletions(-)\n\ndiff --git a/hash.h b/hash.h\nindex a68a2b6a161..68d4292e6da 100644\n--- a/hash.h\n+++ b/hash.h\n@@ -282,21 +282,6 @@ struct git_hash_algo {\n \t/* The hash finalization function for object IDs. */\n \tgit_hash_final_oid_fn final_oid_fn;\n \n-\t/* The non-cryptographic hash initialization function. */\n-\tgit_hash_init_fn unsafe_init_fn;\n-\n-\t/* The non-cryptographic hash context cloning function. */\n-\tgit_hash_clone_fn unsafe_clone_fn;\n-\n-\t/* The non-cryptographic hash update function. */\n-\tgit_hash_update_fn unsafe_update_fn;\n-\n-\t/* The non-cryptographic hash finalization function. */\n-\tgit_hash_final_fn unsafe_final_fn;\n-\n-\t/* The non-cryptographic hash finalization function. */\n-\tgit_hash_final_oid_fn unsafe_final_oid_fn;\n-\n \t/* The OID of the empty tree. */\n \tconst struct object_id *empty_tree;\n \ndiff --git a/object-file.c b/object-file.c\nindex 43efa4ca361..c4b42dd4be9 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -230,11 +230,6 @@ const struct git_hash_algo hash_algos[GIT_HASH_NALGOS] = {\n \t\t.update_fn = git_hash_unknown_update,\n \t\t.final_fn = git_hash_unknown_final,\n \t\t.final_oid_fn = git_hash_unknown_final_oid,\n-\t\t.unsafe_init_fn = git_hash_unknown_init,\n-\t\t.unsafe_clone_fn = git_hash_unknown_clone,\n-\t\t.unsafe_update_fn = git_hash_unknown_update,\n-\t\t.unsafe_final_fn = git_hash_unknown_final,\n-\t\t.unsafe_final_oid_fn = git_hash_unknown_final_oid,\n \t\t.empty_tree = NULL,\n \t\t.empty_blob = NULL,\n \t\t.null_oid = NULL,\n@@ -250,11 +245,6 @@ const struct git_hash_algo hash_algos[GIT_HASH_NALGOS] = {\n \t\t.update_fn = git_hash_sha1_update,\n \t\t.final_fn = git_hash_sha1_final,\n \t\t.final_oid_fn = git_hash_sha1_final_oid,\n-\t\t.unsafe_init_fn = git_hash_sha1_init_unsafe,\n-\t\t.unsafe_clone_fn = git_hash_sha1_clone_unsafe,\n-\t\t.unsafe_update_fn = git_hash_sha1_update_unsafe,\n-\t\t.unsafe_final_fn = git_hash_sha1_final_unsafe,\n-\t\t.unsafe_final_oid_fn = git_hash_sha1_final_oid_unsafe,\n \t\t.unsafe = &sha1_unsafe_algo,\n \t\t.empty_tree = &empty_tree_oid,\n \t\t.empty_blob = &empty_blob_oid,\n@@ -271,11 +261,6 @@ const struct git_hash_algo hash_algos[GIT_HASH_NALGOS] = {\n \t\t.update_fn = git_hash_sha256_update,\n \t\t.final_fn = git_hash_sha256_final,\n \t\t.final_oid_fn = git_hash_sha256_final_oid,\n-\t\t.unsafe_init_fn = git_hash_sha256_init,\n-\t\t.unsafe_clone_fn = git_hash_sha256_clone,\n-\t\t.unsafe_update_fn = git_hash_sha256_update,\n-\t\t.unsafe_final_fn = git_hash_sha256_final,\n-\t\t.unsafe_final_oid_fn = git_hash_sha256_final_oid,\n \t\t.empty_tree = &empty_tree_oid_sha256,\n \t\t.empty_blob = &empty_blob_oid_sha256,\n \t\t.null_oid = &null_oid_sha256,\n-- \n2.48.0.rc2.35.g0c4d006e6e8\n"},{"id":"510869","messageId":"20250118121514.GB3510721@coredump.intra.peff.net","threadId":"62522","inReplyTo":"Z4rL3TbEeR8EiUOi@nand.local","subject":"Re: [PATCH v2 7/8] csum-file: introduce hashfile_checkpoint_init()","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2025-01-18T12:15:14Z","receivedAt":"2025-01-18T12:15:16Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Jan 17, 2025 at 04:30:05PM -0500, Taylor Blau wrote:\n\n> diff --git a/bulk-checkin.c b/bulk-checkin.c\n> index 433070a3bd..892176d23d 100644\n> --- a/bulk-checkin.c\n> +++ b/bulk-checkin.c\n> @@ -261,7 +261,7 @@ static int deflate_blob_to_pack(struct bulk_checkin_packfile *state,\n>  \tgit_hash_ctx ctx;\n>  \tunsigned char obuf[16384];\n>  \tunsigned header_len;\n> -\tstruct hashfile_checkpoint checkpoint = {0};\n> +\tstruct hashfile_checkpoint checkpoint;\n>  \tstruct pack_idx_entry *idx = NULL;\n> \n>  \tseekback = lseek(fd, 0, SEEK_CUR);\n> @@ -272,12 +272,15 @@ static int deflate_blob_to_pack(struct bulk_checkin_packfile *state,\n>  \t\t\t\t\t  OBJ_BLOB, size);\n>  \tthe_hash_algo->init_fn(&ctx);\n>  \tthe_hash_algo->update_fn(&ctx, obuf, header_len);\n> -\tthe_hash_algo->unsafe_init_fn(&checkpoint.ctx);\n> \n>  \t/* Note: idx is non-NULL when we are writing */\n> -\tif ((flags & HASH_WRITE_OBJECT) != 0)\n> +\tif ((flags & HASH_WRITE_OBJECT) != 0) {\n>  \t\tCALLOC_ARRAY(idx, 1);\n> \n> +\t\tprepare_to_stream(state, flags);\n> +\t\thashfile_checkpoint_init(state->f, &checkpoint);\n> +\t}\n> +\n>  \talready_hashed_to = 0;\n> \n>  \twhile (1) {\n\nYeah, that's ugly. We are potentially throwing away the hashfile that\nthe checkpoint was created for. That makes my instinct to push the\ncheckpoint down into the loop where we we might restart a new pack, like\nthis (and like you suggested below):\n\ndiff --git a/bulk-checkin.c b/bulk-checkin.c\nindex 433070a3bd..efa59074fb 100644\n--- a/bulk-checkin.c\n+++ b/bulk-checkin.c\n@@ -261,7 +261,6 @@ static int deflate_blob_to_pack(struct bulk_checkin_packfile *state,\n \tgit_hash_ctx ctx;\n \tunsigned char obuf[16384];\n \tunsigned header_len;\n-\tstruct hashfile_checkpoint checkpoint = {0};\n \tstruct pack_idx_entry *idx = NULL;\n \n \tseekback = lseek(fd, 0, SEEK_CUR);\n@@ -281,8 +280,10 @@ static int deflate_blob_to_pack(struct bulk_checkin_packfile *state,\n \talready_hashed_to = 0;\n \n \twhile (1) {\n+\t\tstruct hashfile_checkpoint checkpoint = {0};\n \t\tprepare_to_stream(state, flags);\n \t\tif (idx) {\n+\t\t\thashfile_checkpoint_init(state->f, &checkpoint);\n \t\t\thashfile_checkpoint(state->f, &checkpoint);\n \t\t\tidx->offset = state->offset;\n \t\t\tcrc32_begin(state->f);\n\nbut that doesn't work, because the checkpoint is also used later for the\nalready_written() check:\n\n        if (already_written(state, result_oid)) {\n                hashfile_truncate(state->f, &checkpoint);\n                state->offset = checkpoint.offset;\n                free(idx);\n\t} else\n\nThat made me wonder if there is a bug lurking there. What if we found\nthe pack was too big, truncated to our checkpoint, and then opened a new\npack? Then the original checkpoint would now be bogus! It would mention\nan offset in the original packfile which doesn't make any sense with\nwhat we have open. But I think this is OK, because we can only leave the\nloop when stream_blob_to_pack() returns, and we always establish a new\ncheckpoint before then.\n\nSo I do think that moving the initialization of the checkpoint into the\nloop, but _not_ moving the variable would work the same way it does now\n(i.e., what you suggested below).\n\nBut I admit that the way this loop works kind of makes my head spin. It\ncan really only ever run twice, but it is hard to see: we break out if\nstream_blob_to_pack() returns success. And it will only return error if\nwe would bust the packsize limit (all other errors cause us to die()).\nAnd only if we would bust the limit _and_ we are not the only object in\nthe pack. And since we start a new pack if we loop, that will never be\ntrue on the second iteration; we'll always either die() or return\nsuccess.\n\nI do think it would be much easier to read with a single explicit retry:\n\n  if (checkpoint_and_try_to_stream() < 0) {\n\t/* we busted the limit; make a new pack and try again */\n\thashfile_truncate();\n\tetc...\n        if (checkpoint_and_try_to_stream() < 0)\n\t\tBUG(\"yikes, we should not fail a second time!\");\n  }\n\nwhere checkpoint_and_try_to_stream() is the first half of the loop, up\nto the stream_blob_to_pack() call.\n\nAnyway, that is all outside of your patch, and relevant only because\n_if_ we untangled it a bit more, it might make the checkpoint lifetime a\nbit more obvious and less scary to refactor.\n\nBut it does imply to me that the data dependency introduced by my\nsuggestion is not always so straight-forward as I thought it would be,\nand we should probably punt on it for your series.\n\n> which would do the trick, but it feels awfully hacky to have the \"if\n> (checkpoint.f != state->f)\" check in there, since that feels too\n> intimately tied to the implementation of the hashfile_checkpoint API for\n> my comfort.\n\nI think you could unconditionally checkpoint at that part; we're about\nto do a write, so we want to store the state before the write in case we\nneed to roll back.\n\n> Anyway, that's all to say that I think that while this is probably\n> doable in theory, in practice it's kind of a mess, at least currently.\n> I would rather see if there are other ways to clean up the\n> deflate_blob_to_pack() function first in a way that made this change\n> less awkward.\n\nYeah, I actually wrote what I wrote above before reading this far down\nin your email, but we arrived at the exact same conclusion. ;) Hopefully\nwhat I wrote might give some pointers if somebody wants to refactor\nlater.\n\n> I think the most reasonable course here would be to pursue a minimal\n> change like the one presented here and then think about further clean up\n> as a separate step.\n\nYep. Thanks for looking into it.\n\n-Peff\n"},{"id":"510870","messageId":"20250118122831.GC3510721@coredump.intra.peff.net","threadId":"62522","inReplyTo":"cover.1737151386.git.me@ttaylorr.com","subject":"Re: [PATCH v3 0/8] hash: introduce unsafe_hash_algo(), drop unsafe_ variants","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2025-01-18T12:28:31Z","receivedAt":"2025-01-18T12:28:32Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Jan 17, 2025 at 05:03:07PM -0500, Taylor Blau wrote:\n\n>     +    Note that hash_algo_by_ptr() needs an adjustment to allow passing in the\n>     +    unsafe variant of a hash function. All other query functions on the\n>     +    hash_algos array will continue to return the safe variants of any\n>     +    function.\n>     +\n>          Suggested-by: Jeff King <peff@peff.net>\n>          Signed-off-by: Taylor Blau <me@ttaylorr.com>\n>      \n>     @@ hash.h: struct git_hash_algo {\n>       };\n>       extern const struct git_hash_algo hash_algos[GIT_HASH_NALGOS];\n>       \n>     -@@ hash.h: static inline int hash_algo_by_ptr(const struct git_hash_algo *p)\n>     - \treturn p - hash_algos;\n>     +@@ hash.h: int hash_algo_by_length(int len);\n>     + /* Identical, except for a pointer to struct git_hash_algo. */\n>     + static inline int hash_algo_by_ptr(const struct git_hash_algo *p)\n>     + {\n>     +-\treturn p - hash_algos;\n>     ++\tsize_t i;\n>     ++\tfor (i = 0; i < GIT_HASH_NALGOS; i++) {\n>     ++\t\tconst struct git_hash_algo *algop = &hash_algos[i];\n>     ++\t\tif (p == algop || (algop->unsafe && p == algop->unsafe))\n>     ++\t\t\treturn i;\n>     ++\t}\n>     ++\treturn GIT_HASH_UNKNOWN;\n>       }\n\nOK, so this version introduces the loop we discussed earlier. I think we\ncan probably dismiss any performance loss as theoretical unless somebody\ncan think of a good way to measure. It seems like worrying about it is\nprobably a premature micro-optimization.\n\nIt is a little quirky that it loses the transitive nature of\nhash_algo_by_ptr() and hash_algo_by_id(). So this is unsafe:\n\n  /* start with unsafe variant */\n  algo = unsafe_hash_algo(the_hash_algo);\n  algo->init_fn(...);\n\n  /* returns GIT_HASH_SHA1, even for the unsafe variant */\n  id = hash_algo_by_ptr(algo);\n\n  /* returns the safe variant */\n  algo = hash_algo_by_id(id);\n\n  /* oops, this is mix-and-matching */\n  alog->update_fn(...);\n\nNow obviously that sort of round-trip is a silly thing to do in a single\nfunction. Could it happen across a larger call-chain, where the id is\nstored in a struct and passed somewhere far away? I guess so, but it\nalso seems kind of unlikely.\n\nIt does make me wonder if hash_algo_by_ptr() should just return UNKNOWN\nfor the unsafe variant. Then we'd at least get a loud error from the\ncaller (as opposed to the code before your patch, which is undefined\nbehavior). I dunno.\n\n-Peff\n"},{"id":"510871","messageId":"20250118124343.GA3828177@coredump.intra.peff.net","threadId":"62522","inReplyTo":"20250118122831.GC3510721@coredump.intra.peff.net","subject":"Re: [PATCH v3 0/8] hash: introduce unsafe_hash_algo(), drop unsafe_ variants","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2025-01-18T12:43:43Z","receivedAt":"2025-01-18T12:43:45Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sat, Jan 18, 2025 at 07:28:31AM -0500, Jeff King wrote:\n\n> It does make me wonder if hash_algo_by_ptr() should just return UNKNOWN\n> for the unsafe variant. Then we'd at least get a loud error from the\n> caller (as opposed to the code before your patch, which is undefined\n> behavior). I dunno.\n\nDoing this:\n\ndiff --git a/hash.h b/hash.h\nindex 68d4292e6d..ad2c919991 100644\n--- a/hash.h\n+++ b/hash.h\n@@ -311,7 +311,7 @@ static inline int hash_algo_by_ptr(const struct git_hash_algo *p)\n \tsize_t i;\n \tfor (i = 0; i < GIT_HASH_NALGOS; i++) {\n \t\tconst struct git_hash_algo *algop = &hash_algos[i];\n-\t\tif (p == algop || (algop->unsafe && p == algop->unsafe))\n+\t\tif (p == algop)\n \t\t\treturn i;\n \t}\n \treturn GIT_HASH_UNKNOWN;\n\non top of your series doesn't trigger any issues in the test suite.\nWhich I think shows that we would never have triggered the UB from the\noriginal implementation, either[1]. Still, I think I prefer your loop to\nbeing one errant function call away from undefined behavior. But maybe\nreturning UNKNOWN (as above) is a safer bet than losing the inverse\nnature of the by_ptr() and by_id() functions, which could otherwise\ncause hard-to-find interactions?\n\n-Peff\n\n[1] One thing that still puzzles me. If you further add a BUG() to that\n    function when we'd return UNKNOWN, you can see that we trigger a few\n    cases in the test suite where we pass in NULL (e.g., because we're\n    not in a repo). I think these are already UB with the existing\n    \"p - hash_algos\" implementation! We'll return what is effectively\n    \"p\" cast to an int.\n\n    E.g. if I do this:\n\n    diff --git a/hash.h b/hash.h\n    index 756166ce5e..ff31156416 100644\n    --- a/hash.h\n    +++ b/hash.h\n    @@ -320,6 +320,8 @@ int hash_algo_by_length(int len);\n     /* Identical, except for a pointer to struct git_hash_algo. */\n     static inline int hash_algo_by_ptr(const struct git_hash_algo *p)\n     {\n    +\tif (!p)\n    +\t\tBUG(\"null hash, return is %d\", (int)(p - hash_algos));\n     \treturn p - hash_algos;\n     }\n\n     then t1517 shows grep dying with:\n\n       BUG: hash.h:324: null hash, return is -1646754892\n\n     I think it \"works\" because the backtrace is:\n\n      [...]\n      #5  0x0000556d436f6b71 in BUG_fl (file=0x556d43790e8b \"hash.h\", line=324,\n          fmt=0x556d43790e73 \"null hash, return is %d\") at usage.c:335\n      #6  0x0000556d4357c2f9 in hash_algo_by_ptr (p=0x0) at /home/peff/compile/git/hash.h:324\n      #7  0x0000556d4357c437 in oidclr (oid=0x7ffdf5810ea4, algop=0x0) at /home/peff/compile/git/hash.h:392\n      #8  0x0000556d435801c7 in prep_exclude (dir=0x7ffdf5811190, istate=0x556d608959c0, base=0x556d6089da50 \"nums\",\n          baselen=0) at dir.c:1699\n      [...]\n      #16 0x0000556d4344dd4a in cmd_grep (argc=0, argv=0x556d60895ee8, prefix=0x0, repo=0x0) at builtin/grep.c:1257\n\n     So we probably write a totally garbage algo field into the\n     object_id, but nobody ever ends up looking at it. Probably\n     something we should clean up, but way out of scope for your series.\n     But I do think it reinforces that returning UNKNOWN is an\n     improvement; it avoids undefined behavior and anybody who tried to\n     use it should get a BUG() from calling git_hash_unknown_*()\n     functions.\n\n"},{"id":"511087","messageId":"xmqqwmemv8ky.fsf@gitster.g","threadId":"62522","inReplyTo":"20250118124343.GA3828177@coredump.intra.peff.net","subject":"Re: [PATCH v3 0/8] hash: introduce unsafe_hash_algo(), drop unsafe_ variants","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-01-22T21:31:57Z","receivedAt":"2025-01-22T21:32:01Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King <peff@peff.net> writes:\n\n> .... But maybe\n> returning UNKNOWN (as above) is a safer bet than losing the inverse\n> nature of the by_ptr() and by_id() functions, which could otherwise\n> cause hard-to-find interactions?\n\nYeah, that sounds sensible.\n\n>      I think it \"works\" because the backtrace is:\n>\n>       [...]\n>       #5  0x0000556d436f6b71 in BUG_fl (file=0x556d43790e8b \"hash.h\", line=324,\n>           fmt=0x556d43790e73 \"null hash, return is %d\") at usage.c:335\n>       #6  0x0000556d4357c2f9 in hash_algo_by_ptr (p=0x0) at /home/peff/compile/git/hash.h:324\n>       #7  0x0000556d4357c437 in oidclr (oid=0x7ffdf5810ea4, algop=0x0) at /home/peff/compile/git/hash.h:392\n>       #8  0x0000556d435801c7 in prep_exclude (dir=0x7ffdf5811190, istate=0x556d608959c0, base=0x556d6089da50 \"nums\",\n>           baselen=0) at dir.c:1699\n>       [...]\n>       #16 0x0000556d4344dd4a in cmd_grep (argc=0, argv=0x556d60895ee8, prefix=0x0, repo=0x0) at builtin/grep.c:1257\n>\n>      So we probably write a totally garbage algo field into the\n>      object_id, but nobody ever ends up looking at it. Probably\n>      something we should clean up, but way out of scope for your series.\n\nYeah, that looks bad, but I think it is fine to leave it outside the\nseries to fix that.\n\n>      But I do think it reinforces that returning UNKNOWN is an\n>      improvement; it avoids undefined behavior and anybody who tried to\n>      use it should get a BUG() from calling git_hash_unknown_*()\n>      functions.\n\nSounds like a sensitive direction to go in.\n\nThanks.\n\n"},{"id":"511113","messageId":"cover.1737653640.git.me@ttaylorr.com","threadId":"62522","inReplyTo":"cover.1732130001.git.me@ttaylorr.com","subject":"[PATCH v4 0/8] hash: introduce unsafe_hash_algo(), drop unsafe_ variants","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-01-23T17:34:12Z","receivedAt":"2025-01-23T17:34:19Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"(This series is based on 14650065b7 (RelNotes/2.48.0: fix typos etc.,\n2025-01-07)).\n\nHere is a hopefully final version of my series to harden the unsafe hash\nalgorithm changes added in v2.47.0. The only difference from last time\nis that hash_algo_by_ptr() now returns GIT_HASH_UNKNOWN for NULL and\nunsafe variants, which is a strict improvement from both v3 of this\nseries and the status-quo on master.\n\nAs usual, a (small) range-diff is included below for convenience, and\nthe original cover letter is as follows:\n\n------------\n\nThis series implements an idea discussed in [2] which suggests that we\nintroduce a way to access a wrapped version of a 'struct git_hash_algo'\nwhich represents the unsafe variant of that algorithm, rather than\nhaving individual unsafe_ functions (like unsafe_init_fn() versus\ninit_fn(), etc.).\n\nThis approach is relatively straightforward to implement, and removes a\nsignificant deficiency in the original implementation of\nunsafe/non-cryptographic hash functions by making it impossible to\nswitch between safe- and unsafe variants of hash functions. It also\ncleans up the sha1-unsafe test helper's implementation by removing a\nlarge number of \"if (unsafe)\"-style conditionals.\n\nThe series is laid out as follows:\n\n  * The first two patches prepare the hashfile API for the upcoming\n    change:\n\n      csum-file: store the hash algorithm as a struct field\n      csum-file.c: extract algop from hashfile_checksum_valid()\n\n  * The next patch implements the new 'unsafe_hash_algo()' function at\n    the heart of this series' approach:\n\n      hash.h: introduce `unsafe_hash_algo()`\n\n  * The next two patches convert existing callers to use the new\n    'unsafe_hash_algo()' function, instead of switching between safe and\n    unsafe_ variants of individual functions:\n\n      csum-file.c: use unsafe_hash_algo()\n      t/helper/test-hash.c: use unsafe_hash_algo()\n\n  * The final patch drops the unsafe_ function variants following all\n    callers being converted to use the new pattern:\n\n      hash.h: drop unsafe_ function variants\n\nThanks in advance for your review!\n\n[1]: https://lore.kernel.org/git/20241230-pks-meson-sha1-unsafe-v1-0-efb276e171f5@pks.im/\n[2]: https://lore.kernel.org/git/20241107013915.GA961214@coredump.intra.peff.net/\n\nTaylor Blau (8):\n  t/helper/test-tool: implement sha1-unsafe helper\n  csum-file: store the hash algorithm as a struct field\n  csum-file.c: extract algop from hashfile_checksum_valid()\n  hash.h: introduce `unsafe_hash_algo()`\n  csum-file.c: use unsafe_hash_algo()\n  t/helper/test-hash.c: use unsafe_hash_algo()\n  csum-file: introduce hashfile_checkpoint_init()\n  hash.h: drop unsafe_ function variants\n\n builtin/fast-import.c  |  2 +-\n bulk-checkin.c         |  9 ++++++---\n csum-file.c            | 40 +++++++++++++++++++++++++---------------\n csum-file.h            |  2 ++\n hash.h                 | 28 ++++++++++++----------------\n object-file.c          | 41 ++++++++++++++++++++++++++---------------\n t/helper/test-hash.c   |  4 +++-\n t/helper/test-sha1.c   |  7 ++++++-\n t/helper/test-sha1.sh  | 38 ++++++++++++++++++++++----------------\n t/helper/test-sha256.c |  2 +-\n t/helper/test-tool.c   |  1 +\n t/helper/test-tool.h   |  3 ++-\n 12 files changed, 107 insertions(+), 70 deletions(-)\n\nRange-diff against v3:\n1:  ae6b8c75294 = 1:  b64ae238248 t/helper/test-tool: implement sha1-unsafe helper\n2:  2b79c76e471 = 2:  d03f503682f csum-file: store the hash algorithm as a struct field\n3:  d7deb3f338e = 3:  73554c3b881 csum-file.c: extract algop from hashfile_checksum_valid()\n4:  b6b2bb2714f ! 4:  ae01f1f4274 hash.h: introduce `unsafe_hash_algo()`\n    @@ hash.h: int hash_algo_by_length(int len);\n     +\tsize_t i;\n     +\tfor (i = 0; i < GIT_HASH_NALGOS; i++) {\n     +\t\tconst struct git_hash_algo *algop = &hash_algos[i];\n    -+\t\tif (p == algop || (algop->unsafe && p == algop->unsafe))\n    ++\t\tif (p == algop)\n     +\t\t\treturn i;\n     +\t}\n     +\treturn GIT_HASH_UNKNOWN;\n5:  ca67de80971 = 5:  64a850c77ae csum-file.c: use unsafe_hash_algo()\n6:  21b175b07ff = 6:  3dcccccf752 t/helper/test-hash.c: use unsafe_hash_algo()\n7:  850d4f407db = 7:  da97157c4a1 csum-file: introduce hashfile_checkpoint_init()\n8:  0c4d006e6e8 = 8:  0ba27182b5e hash.h: drop unsafe_ function variants\n\nbase-commit: 14650065b76b28d3cfa9453356ac5669b19e706e\n-- \n2.48.0.rc2.35.gd215225db14\n"},{"id":"511114","messageId":"b64ae2382483b0518fd71df23433e40cd1cb986b.1737653640.git.me@ttaylorr.com","threadId":"62522","inReplyTo":"cover.1737653640.git.me@ttaylorr.com","subject":"[PATCH v4 1/8] t/helper/test-tool: implement sha1-unsafe helper","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-01-23T17:34:19Z","receivedAt":"2025-01-23T17:34:22Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"With the new \"unsafe\" SHA-1 build knob, it is convenient to have a\ntest-tool that can exercise Git's unsafe SHA-1 wrappers for testing,\nsimilar to 't/helper/test-tool sha1'.\n\nImplement that helper by altering the implementation of that test-tool\n(in cmd_hash_impl(), which is generic and parameterized over different\nhash functions) to conditionally run the unsafe variants of the chosen\nhash function, and expose the new behavior via a new 'sha1-unsafe' test\nhelper.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n t/helper/test-hash.c   | 17 +++++++++++++----\n t/helper/test-sha1.c   |  7 ++++++-\n t/helper/test-sha1.sh  | 38 ++++++++++++++++++++++----------------\n t/helper/test-sha256.c |  2 +-\n t/helper/test-tool.c   |  1 +\n t/helper/test-tool.h   |  3 ++-\n 6 files changed, 45 insertions(+), 23 deletions(-)\n\ndiff --git a/t/helper/test-hash.c b/t/helper/test-hash.c\nindex 45d829c908f..d0ee668df95 100644\n--- a/t/helper/test-hash.c\n+++ b/t/helper/test-hash.c\n@@ -1,7 +1,7 @@\n #include \"test-tool.h\"\n #include \"hex.h\"\n \n-int cmd_hash_impl(int ac, const char **av, int algo)\n+int cmd_hash_impl(int ac, const char **av, int algo, int unsafe)\n {\n \tgit_hash_ctx ctx;\n \tunsigned char hash[GIT_MAX_HEXSZ];\n@@ -27,7 +27,10 @@ int cmd_hash_impl(int ac, const char **av, int algo)\n \t\t\tdie(\"OOPS\");\n \t}\n \n-\talgop->init_fn(&ctx);\n+\tif (unsafe)\n+\t\talgop->unsafe_init_fn(&ctx);\n+\telse\n+\t\talgop->init_fn(&ctx);\n \n \twhile (1) {\n \t\tssize_t sz, this_sz;\n@@ -46,9 +49,15 @@ int cmd_hash_impl(int ac, const char **av, int algo)\n \t\t}\n \t\tif (this_sz == 0)\n \t\t\tbreak;\n-\t\talgop->update_fn(&ctx, buffer, this_sz);\n+\t\tif (unsafe)\n+\t\t\talgop->unsafe_update_fn(&ctx, buffer, this_sz);\n+\t\telse\n+\t\t\talgop->update_fn(&ctx, buffer, this_sz);\n \t}\n-\talgop->final_fn(hash, &ctx);\n+\tif (unsafe)\n+\t\talgop->unsafe_final_fn(hash, &ctx);\n+\telse\n+\t\talgop->final_fn(hash, &ctx);\n \n \tif (binary)\n \t\tfwrite(hash, 1, algop->rawsz, stdout);\ndiff --git a/t/helper/test-sha1.c b/t/helper/test-sha1.c\nindex e60d000c039..349540c4df8 100644\n--- a/t/helper/test-sha1.c\n+++ b/t/helper/test-sha1.c\n@@ -3,7 +3,7 @@\n \n int cmd__sha1(int ac, const char **av)\n {\n-\treturn cmd_hash_impl(ac, av, GIT_HASH_SHA1);\n+\treturn cmd_hash_impl(ac, av, GIT_HASH_SHA1, 0);\n }\n \n int cmd__sha1_is_sha1dc(int argc UNUSED, const char **argv UNUSED)\n@@ -13,3 +13,8 @@ int cmd__sha1_is_sha1dc(int argc UNUSED, const char **argv UNUSED)\n #endif\n \treturn 1;\n }\n+\n+int cmd__sha1_unsafe(int ac, const char **av)\n+{\n+\treturn cmd_hash_impl(ac, av, GIT_HASH_SHA1, 1);\n+}\ndiff --git a/t/helper/test-sha1.sh b/t/helper/test-sha1.sh\nindex 84594885c70..bf387d3db14 100755\n--- a/t/helper/test-sha1.sh\n+++ b/t/helper/test-sha1.sh\n@@ -3,25 +3,31 @@\n dd if=/dev/zero bs=1048576 count=100 2>/dev/null |\n /usr/bin/time t/helper/test-tool sha1 >/dev/null\n \n+dd if=/dev/zero bs=1048576 count=100 2>/dev/null |\n+/usr/bin/time t/helper/test-tool sha1-unsafe >/dev/null\n+\n while read expect cnt pfx\n do\n \tcase \"$expect\" in '#'*) continue ;; esac\n-\tactual=$(\n-\t\t{\n-\t\t\ttest -z \"$pfx\" || echo \"$pfx\"\n-\t\t\tdd if=/dev/zero bs=1048576 count=$cnt 2>/dev/null |\n-\t\t\tperl -pe 'y/\\000/g/'\n-\t\t} | ./t/helper/test-tool sha1 $cnt\n-\t)\n-\tif test \"$expect\" = \"$actual\"\n-\tthen\n-\t\techo \"OK: $expect $cnt $pfx\"\n-\telse\n-\t\techo >&2 \"OOPS: $cnt\"\n-\t\techo >&2 \"expect: $expect\"\n-\t\techo >&2 \"actual: $actual\"\n-\t\texit 1\n-\tfi\n+\tfor sha1 in sha1 sha1-unsafe\n+\tdo\n+\t\tactual=$(\n+\t\t\t{\n+\t\t\t\ttest -z \"$pfx\" || echo \"$pfx\"\n+\t\t\t\tdd if=/dev/zero bs=1048576 count=$cnt 2>/dev/null |\n+\t\t\t\tperl -pe 'y/\\000/g/'\n+\t\t\t} | ./t/helper/test-tool $sha1 $cnt\n+\t\t)\n+\t\tif test \"$expect\" = \"$actual\"\n+\t\tthen\n+\t\t\techo \"OK ($sha1): $expect $cnt $pfx\"\n+\t\telse\n+\t\t\techo >&2 \"OOPS ($sha1): $cnt\"\n+\t\t\techo >&2 \"expect ($sha1): $expect\"\n+\t\t\techo >&2 \"actual ($sha1): $actual\"\n+\t\t\texit 1\n+\t\tfi\n+\tdone\n done <<EOF\n da39a3ee5e6b4b0d3255bfef95601890afd80709 0\n 3f786850e387550fdab836ed7e6dc881de23001b 0 a\ndiff --git a/t/helper/test-sha256.c b/t/helper/test-sha256.c\nindex 2fb20438f3c..7fd0aa1fcd3 100644\n--- a/t/helper/test-sha256.c\n+++ b/t/helper/test-sha256.c\n@@ -3,5 +3,5 @@\n \n int cmd__sha256(int ac, const char **av)\n {\n-\treturn cmd_hash_impl(ac, av, GIT_HASH_SHA256);\n+\treturn cmd_hash_impl(ac, av, GIT_HASH_SHA256, 0);\n }\ndiff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\nindex 4a7aa993ba9..958452ef12e 100644\n--- a/t/helper/test-tool.c\n+++ b/t/helper/test-tool.c\n@@ -70,6 +70,7 @@ static struct test_cmd cmds[] = {\n \t{ \"serve-v2\", cmd__serve_v2 },\n \t{ \"sha1\", cmd__sha1 },\n \t{ \"sha1-is-sha1dc\", cmd__sha1_is_sha1dc },\n+\t{ \"sha1-unsafe\", cmd__sha1_unsafe },\n \t{ \"sha256\", cmd__sha256 },\n \t{ \"sigchain\", cmd__sigchain },\n \t{ \"simple-ipc\", cmd__simple_ipc },\ndiff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\nindex 21802ac27da..24149edd414 100644\n--- a/t/helper/test-tool.h\n+++ b/t/helper/test-tool.h\n@@ -63,6 +63,7 @@ int cmd__scrap_cache_tree(int argc, const char **argv);\n int cmd__serve_v2(int argc, const char **argv);\n int cmd__sha1(int argc, const char **argv);\n int cmd__sha1_is_sha1dc(int argc, const char **argv);\n+int cmd__sha1_unsafe(int argc, const char **argv);\n int cmd__sha256(int argc, const char **argv);\n int cmd__sigchain(int argc, const char **argv);\n int cmd__simple_ipc(int argc, const char **argv);\n@@ -81,6 +82,6 @@ int cmd__windows_named_pipe(int argc, const char **argv);\n #endif\n int cmd__write_cache(int argc, const char **argv);\n \n-int cmd_hash_impl(int ac, const char **av, int algo);\n+int cmd_hash_impl(int ac, const char **av, int algo, int unsafe);\n \n #endif\n-- \n2.48.0.rc2.35.gd215225db14\n\n"},{"id":"511115","messageId":"d03f503682f017eefcf8d296cb09da2ef4f7200b.1737653640.git.me@ttaylorr.com","threadId":"62522","inReplyTo":"cover.1737653640.git.me@ttaylorr.com","subject":"[PATCH v4 2/8] csum-file: store the hash algorithm as a struct field","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-01-23T17:34:23Z","receivedAt":"2025-01-23T17:34:25Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Throughout the hashfile API, we rely on a reference to 'the_hash_algo',\nand call its _unsafe function variants directly.\n\nPrepare for a future change where we may use a different 'git_hash_algo'\npointer (instead of just relying on 'the_hash_algo' throughout) by\nmaking the 'git_hash_algo' pointer a member of the 'hashfile' structure\nitself.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n csum-file.c | 20 +++++++++++---------\n csum-file.h |  1 +\n 2 files changed, 12 insertions(+), 9 deletions(-)\n\ndiff --git a/csum-file.c b/csum-file.c\nindex 5716016e12e..b28cd047e3f 100644\n--- a/csum-file.c\n+++ b/csum-file.c\n@@ -50,7 +50,7 @@ void hashflush(struct hashfile *f)\n \n \tif (offset) {\n \t\tif (!f->skip_hash)\n-\t\t\tthe_hash_algo->unsafe_update_fn(&f->ctx, f->buffer, offset);\n+\t\t\tf->algop->unsafe_update_fn(&f->ctx, f->buffer, offset);\n \t\tflush(f, f->buffer, offset);\n \t\tf->offset = 0;\n \t}\n@@ -71,14 +71,14 @@ int finalize_hashfile(struct hashfile *f, unsigned char *result,\n \thashflush(f);\n \n \tif (f->skip_hash)\n-\t\thashclr(f->buffer, the_repository->hash_algo);\n+\t\thashclr(f->buffer, f->algop);\n \telse\n-\t\tthe_hash_algo->unsafe_final_fn(f->buffer, &f->ctx);\n+\t\tf->algop->unsafe_final_fn(f->buffer, &f->ctx);\n \n \tif (result)\n-\t\thashcpy(result, f->buffer, the_repository->hash_algo);\n+\t\thashcpy(result, f->buffer, f->algop);\n \tif (flags & CSUM_HASH_IN_STREAM)\n-\t\tflush(f, f->buffer, the_hash_algo->rawsz);\n+\t\tflush(f, f->buffer, f->algop->rawsz);\n \tif (flags & CSUM_FSYNC)\n \t\tfsync_component_or_die(component, f->fd, f->name);\n \tif (flags & CSUM_CLOSE) {\n@@ -128,7 +128,7 @@ void hashwrite(struct hashfile *f, const void *buf, unsigned int count)\n \t\t\t * f->offset is necessarily zero.\n \t\t\t */\n \t\t\tif (!f->skip_hash)\n-\t\t\t\tthe_hash_algo->unsafe_update_fn(&f->ctx, buf, nr);\n+\t\t\t\tf->algop->unsafe_update_fn(&f->ctx, buf, nr);\n \t\t\tflush(f, buf, nr);\n \t\t} else {\n \t\t\t/*\n@@ -174,7 +174,9 @@ static struct hashfile *hashfd_internal(int fd, const char *name,\n \tf->name = name;\n \tf->do_crc = 0;\n \tf->skip_hash = 0;\n-\tthe_hash_algo->unsafe_init_fn(&f->ctx);\n+\n+\tf->algop = the_hash_algo;\n+\tf->algop->unsafe_init_fn(&f->ctx);\n \n \tf->buffer_len = buffer_len;\n \tf->buffer = xmalloc(buffer_len);\n@@ -208,7 +210,7 @@ void hashfile_checkpoint(struct hashfile *f, struct hashfile_checkpoint *checkpo\n {\n \thashflush(f);\n \tcheckpoint->offset = f->total;\n-\tthe_hash_algo->unsafe_clone_fn(&checkpoint->ctx, &f->ctx);\n+\tf->algop->unsafe_clone_fn(&checkpoint->ctx, &f->ctx);\n }\n \n int hashfile_truncate(struct hashfile *f, struct hashfile_checkpoint *checkpoint)\n@@ -219,7 +221,7 @@ int hashfile_truncate(struct hashfile *f, struct hashfile_checkpoint *checkpoint\n \t    lseek(f->fd, offset, SEEK_SET) != offset)\n \t\treturn -1;\n \tf->total = offset;\n-\tthe_hash_algo->unsafe_clone_fn(&f->ctx, &checkpoint->ctx);\n+\tf->algop->unsafe_clone_fn(&f->ctx, &checkpoint->ctx);\n \tf->offset = 0; /* hashflush() was called in checkpoint */\n \treturn 0;\n }\ndiff --git a/csum-file.h b/csum-file.h\nindex 7c73da0a40a..2b45f4673a2 100644\n--- a/csum-file.h\n+++ b/csum-file.h\n@@ -20,6 +20,7 @@ struct hashfile {\n \tsize_t buffer_len;\n \tunsigned char *buffer;\n \tunsigned char *check_buffer;\n+\tconst struct git_hash_algo *algop;\n \n \t/**\n \t * If non-zero, skip_hash indicates that we should\n-- \n2.48.0.rc2.35.gd215225db14\n\n"},{"id":"511116","messageId":"73554c3b8814b4864a53469e6e20c2170a9f58ec.1737653640.git.me@ttaylorr.com","threadId":"62522","inReplyTo":"cover.1737653640.git.me@ttaylorr.com","subject":"[PATCH v4 3/8] csum-file.c: extract algop from hashfile_checksum_valid()","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-01-23T17:34:26Z","receivedAt":"2025-01-23T17:34:29Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Perform a similar transformation as in the previous commit, but focused\ninstead on hashfile_checksum_valid(). This function does not work with a\nhashfile structure itself, and instead validates the raw contents of a\nfile written using the hashfile API.\n\nWe'll want to be prepared for a similar change to this function in the\nfuture, so prepare ourselves for that by extracting 'the_hash_algo' into\nits own field for use within this function.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n csum-file.c | 13 +++++++------\n 1 file changed, 7 insertions(+), 6 deletions(-)\n\ndiff --git a/csum-file.c b/csum-file.c\nindex b28cd047e3f..7a71121e340 100644\n--- a/csum-file.c\n+++ b/csum-file.c\n@@ -242,14 +242,15 @@ int hashfile_checksum_valid(const unsigned char *data, size_t total_len)\n {\n \tunsigned char got[GIT_MAX_RAWSZ];\n \tgit_hash_ctx ctx;\n-\tsize_t data_len = total_len - the_hash_algo->rawsz;\n+\tconst struct git_hash_algo *algop = the_hash_algo;\n+\tsize_t data_len = total_len - algop->rawsz;\n \n-\tif (total_len < the_hash_algo->rawsz)\n+\tif (total_len < algop->rawsz)\n \t\treturn 0; /* say \"too short\"? */\n \n-\tthe_hash_algo->unsafe_init_fn(&ctx);\n-\tthe_hash_algo->unsafe_update_fn(&ctx, data, data_len);\n-\tthe_hash_algo->unsafe_final_fn(got, &ctx);\n+\talgop->unsafe_init_fn(&ctx);\n+\talgop->unsafe_update_fn(&ctx, data, data_len);\n+\talgop->unsafe_final_fn(got, &ctx);\n \n-\treturn hasheq(got, data + data_len, the_repository->hash_algo);\n+\treturn hasheq(got, data + data_len, algop);\n }\n-- \n2.48.0.rc2.35.gd215225db14\n\n"},{"id":"511117","messageId":"ae01f1f4274baa81d632f1075c236a6676a9476d.1737653640.git.me@ttaylorr.com","threadId":"62522","inReplyTo":"cover.1737653640.git.me@ttaylorr.com","subject":"[PATCH v4 4/8] hash.h: introduce `unsafe_hash_algo()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-01-23T17:34:29Z","receivedAt":"2025-01-23T17:34:32Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In 253ed9ecff (hash.h: scaffolding for _unsafe hashing variants,\n2024-09-26), we introduced \"unsafe\" variants of the SHA-1 hashing\nfunctions by introducing new functions like \"unsafe_init_fn()\" and so\non.\n\nThis approach has a major shortcoming that callers must remember to\nconsistently use one variant or the other. Failing to consistently use\n(or not use) the unsafe variants can lead to crashes at best, or subtle\nmemory corruption issues at worst.\n\nIn the hashfile API, this isn't difficult to achieve, but verifying that\nall callers consistently use the unsafe variants is somewhat of a chore\ngiven how spread out all of the callers are. In the sha1 and sha1-unsafe\ntest helpers, all of the calls to various hash functions are guarded by\nan \"if (unsafe)\" conditional, which is repetitive and cumbersome.\n\nAddress these issues by introducing a new pattern whereby one\n'git_hash_algo' can return a pointer to another 'git_hash_algo' that\nrepresents the unsafe version of itself. So instead of having something\nlike:\n\n    if (unsafe)\n      the_hash_algo->init_fn(...);\n      the_hash_algo->update_fn(...);\n      the_hash_algo->final_fn(...);\n    else\n      the_hash_algo->unsafe_init_fn(...);\n      the_hash_algo->unsafe_update_fn(...);\n      the_hash_algo->unsafe_final_fn(...);\n\nwe can instead write:\n\n    struct git_hash_algo *algop = the_hash_algo;\n    if (unsafe)\n      algop = unsafe_hash_algo(algop);\n\n    algop->init_fn(...);\n    algop->update_fn(...);\n    algop->final_fn(...);\n\nThis removes the existing shortcoming by no longer forcing the caller to\n\"remember\" which variant of the hash functions it wants to call, only to\nhold onto a 'struct git_hash_algo' pointer that is initialized once.\n\nSimilarly, while there currently is still a way to \"mix\" safe and unsafe\nfunctions, this too will go away after subsequent commits remove all\ndirect calls to the unsafe_ variants.\n\nNote that hash_algo_by_ptr() needs an adjustment to allow passing in the\nunsafe variant of a hash function. All other query functions on the\nhash_algos array will continue to return the safe variants of any\nfunction.\n\nSuggested-by: Jeff King <peff@peff.net>\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n hash.h        | 13 ++++++++++++-\n object-file.c | 26 ++++++++++++++++++++++++++\n 2 files changed, 38 insertions(+), 1 deletion(-)\n\ndiff --git a/hash.h b/hash.h\nindex 756166ce5e8..0bf63cedfa4 100644\n--- a/hash.h\n+++ b/hash.h\n@@ -305,6 +305,9 @@ struct git_hash_algo {\n \n \t/* The all-zeros OID. */\n \tconst struct object_id *null_oid;\n+\n+\t/* The unsafe variant of this hash function, if one exists. */\n+\tconst struct git_hash_algo *unsafe;\n };\n extern const struct git_hash_algo hash_algos[GIT_HASH_NALGOS];\n \n@@ -320,9 +323,17 @@ int hash_algo_by_length(int len);\n /* Identical, except for a pointer to struct git_hash_algo. */\n static inline int hash_algo_by_ptr(const struct git_hash_algo *p)\n {\n-\treturn p - hash_algos;\n+\tsize_t i;\n+\tfor (i = 0; i < GIT_HASH_NALGOS; i++) {\n+\t\tconst struct git_hash_algo *algop = &hash_algos[i];\n+\t\tif (p == algop)\n+\t\t\treturn i;\n+\t}\n+\treturn GIT_HASH_UNKNOWN;\n }\n \n+const struct git_hash_algo *unsafe_hash_algo(const struct git_hash_algo *algop);\n+\n const struct object_id *null_oid(void);\n \n static inline int hashcmp(const unsigned char *sha1, const unsigned char *sha2, const struct git_hash_algo *algop)\ndiff --git a/object-file.c b/object-file.c\nindex 5b792b3dd42..43efa4ca361 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -202,6 +202,22 @@ static void git_hash_unknown_final_oid(struct object_id *oid UNUSED,\n \tBUG(\"trying to finalize unknown hash\");\n }\n \n+static const struct git_hash_algo sha1_unsafe_algo = {\n+\t.name = \"sha1\",\n+\t.format_id = GIT_SHA1_FORMAT_ID,\n+\t.rawsz = GIT_SHA1_RAWSZ,\n+\t.hexsz = GIT_SHA1_HEXSZ,\n+\t.blksz = GIT_SHA1_BLKSZ,\n+\t.init_fn = git_hash_sha1_init_unsafe,\n+\t.clone_fn = git_hash_sha1_clone_unsafe,\n+\t.update_fn = git_hash_sha1_update_unsafe,\n+\t.final_fn = git_hash_sha1_final_unsafe,\n+\t.final_oid_fn = git_hash_sha1_final_oid_unsafe,\n+\t.empty_tree = &empty_tree_oid,\n+\t.empty_blob = &empty_blob_oid,\n+\t.null_oid = &null_oid_sha1,\n+};\n+\n const struct git_hash_algo hash_algos[GIT_HASH_NALGOS] = {\n \t{\n \t\t.name = NULL,\n@@ -239,6 +255,7 @@ const struct git_hash_algo hash_algos[GIT_HASH_NALGOS] = {\n \t\t.unsafe_update_fn = git_hash_sha1_update_unsafe,\n \t\t.unsafe_final_fn = git_hash_sha1_final_unsafe,\n \t\t.unsafe_final_oid_fn = git_hash_sha1_final_oid_unsafe,\n+\t\t.unsafe = &sha1_unsafe_algo,\n \t\t.empty_tree = &empty_tree_oid,\n \t\t.empty_blob = &empty_blob_oid,\n \t\t.null_oid = &null_oid_sha1,\n@@ -305,6 +322,15 @@ int hash_algo_by_length(int len)\n \treturn GIT_HASH_UNKNOWN;\n }\n \n+const struct git_hash_algo *unsafe_hash_algo(const struct git_hash_algo *algop)\n+{\n+\t/* If we have a faster \"unsafe\" implementation, use that. */\n+\tif (algop->unsafe)\n+\t\treturn algop->unsafe;\n+\t/* Otherwise use the default one. */\n+\treturn algop;\n+}\n+\n /*\n  * This is meant to hold a *small* number of objects that you would\n  * want repo_read_object_file() to be able to return, but yet you do not want\n-- \n2.48.0.rc2.35.gd215225db14\n\n"},{"id":"511118","messageId":"64a850c77ae23b92698acf89761f0e7513b6d35e.1737653640.git.me@ttaylorr.com","threadId":"62522","inReplyTo":"cover.1737653640.git.me@ttaylorr.com","subject":"[PATCH v4 5/8] csum-file.c: use unsafe_hash_algo()","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-01-23T17:34:33Z","receivedAt":"2025-01-23T17:34:35Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Instead of calling the unsafe_ hash function variants directly, make use\nof the shared 'algop' pointer by initializing it to:\n\n    f->algop = unsafe_hash_algo(the_hash_algo);\n\n, thus making all calls use the unsafe variants directly.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n csum-file.c | 22 +++++++++++-----------\n 1 file changed, 11 insertions(+), 11 deletions(-)\n\ndiff --git a/csum-file.c b/csum-file.c\nindex 7a71121e340..ebffc80ef71 100644\n--- a/csum-file.c\n+++ b/csum-file.c\n@@ -50,7 +50,7 @@ void hashflush(struct hashfile *f)\n \n \tif (offset) {\n \t\tif (!f->skip_hash)\n-\t\t\tf->algop->unsafe_update_fn(&f->ctx, f->buffer, offset);\n+\t\t\tf->algop->update_fn(&f->ctx, f->buffer, offset);\n \t\tflush(f, f->buffer, offset);\n \t\tf->offset = 0;\n \t}\n@@ -73,7 +73,7 @@ int finalize_hashfile(struct hashfile *f, unsigned char *result,\n \tif (f->skip_hash)\n \t\thashclr(f->buffer, f->algop);\n \telse\n-\t\tf->algop->unsafe_final_fn(f->buffer, &f->ctx);\n+\t\tf->algop->final_fn(f->buffer, &f->ctx);\n \n \tif (result)\n \t\thashcpy(result, f->buffer, f->algop);\n@@ -128,7 +128,7 @@ void hashwrite(struct hashfile *f, const void *buf, unsigned int count)\n \t\t\t * f->offset is necessarily zero.\n \t\t\t */\n \t\t\tif (!f->skip_hash)\n-\t\t\t\tf->algop->unsafe_update_fn(&f->ctx, buf, nr);\n+\t\t\t\tf->algop->update_fn(&f->ctx, buf, nr);\n \t\t\tflush(f, buf, nr);\n \t\t} else {\n \t\t\t/*\n@@ -175,8 +175,8 @@ static struct hashfile *hashfd_internal(int fd, const char *name,\n \tf->do_crc = 0;\n \tf->skip_hash = 0;\n \n-\tf->algop = the_hash_algo;\n-\tf->algop->unsafe_init_fn(&f->ctx);\n+\tf->algop = unsafe_hash_algo(the_hash_algo);\n+\tf->algop->init_fn(&f->ctx);\n \n \tf->buffer_len = buffer_len;\n \tf->buffer = xmalloc(buffer_len);\n@@ -210,7 +210,7 @@ void hashfile_checkpoint(struct hashfile *f, struct hashfile_checkpoint *checkpo\n {\n \thashflush(f);\n \tcheckpoint->offset = f->total;\n-\tf->algop->unsafe_clone_fn(&checkpoint->ctx, &f->ctx);\n+\tf->algop->clone_fn(&checkpoint->ctx, &f->ctx);\n }\n \n int hashfile_truncate(struct hashfile *f, struct hashfile_checkpoint *checkpoint)\n@@ -221,7 +221,7 @@ int hashfile_truncate(struct hashfile *f, struct hashfile_checkpoint *checkpoint\n \t    lseek(f->fd, offset, SEEK_SET) != offset)\n \t\treturn -1;\n \tf->total = offset;\n-\tf->algop->unsafe_clone_fn(&f->ctx, &checkpoint->ctx);\n+\tf->algop->clone_fn(&f->ctx, &checkpoint->ctx);\n \tf->offset = 0; /* hashflush() was called in checkpoint */\n \treturn 0;\n }\n@@ -242,15 +242,15 @@ int hashfile_checksum_valid(const unsigned char *data, size_t total_len)\n {\n \tunsigned char got[GIT_MAX_RAWSZ];\n \tgit_hash_ctx ctx;\n-\tconst struct git_hash_algo *algop = the_hash_algo;\n+\tconst struct git_hash_algo *algop = unsafe_hash_algo(the_hash_algo);\n \tsize_t data_len = total_len - algop->rawsz;\n \n \tif (total_len < algop->rawsz)\n \t\treturn 0; /* say \"too short\"? */\n \n-\talgop->unsafe_init_fn(&ctx);\n-\talgop->unsafe_update_fn(&ctx, data, data_len);\n-\talgop->unsafe_final_fn(got, &ctx);\n+\talgop->init_fn(&ctx);\n+\talgop->update_fn(&ctx, data, data_len);\n+\talgop->final_fn(got, &ctx);\n \n \treturn hasheq(got, data + data_len, algop);\n }\n-- \n2.48.0.rc2.35.gd215225db14\n\n"},{"id":"511119","messageId":"3dcccccf75275d62d533773db4e027ef051d5388.1737653640.git.me@ttaylorr.com","threadId":"62522","inReplyTo":"cover.1737653640.git.me@ttaylorr.com","subject":"[PATCH v4 6/8] t/helper/test-hash.c: use unsafe_hash_algo()","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-01-23T17:34:36Z","receivedAt":"2025-01-23T17:34:38Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Remove a series of conditionals within the shared cmd_hash_impl() helper\nthat powers the 'sha1' and 'sha1-unsafe' helpers.\n\nInstead, replace them with a single conditional that transforms the\nspecified hash algorithm into its unsafe variant. Then all subsequent\ncalls can directly use whatever function it wants to call without having\nto decide between the safe and unsafe variants.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n t/helper/test-hash.c | 17 +++++------------\n 1 file changed, 5 insertions(+), 12 deletions(-)\n\ndiff --git a/t/helper/test-hash.c b/t/helper/test-hash.c\nindex d0ee668df95..aa82638c621 100644\n--- a/t/helper/test-hash.c\n+++ b/t/helper/test-hash.c\n@@ -9,6 +9,8 @@ int cmd_hash_impl(int ac, const char **av, int algo, int unsafe)\n \tint binary = 0;\n \tchar *buffer;\n \tconst struct git_hash_algo *algop = &hash_algos[algo];\n+\tif (unsafe)\n+\t\talgop = unsafe_hash_algo(algop);\n \n \tif (ac == 2) {\n \t\tif (!strcmp(av[1], \"-b\"))\n@@ -27,10 +29,7 @@ int cmd_hash_impl(int ac, const char **av, int algo, int unsafe)\n \t\t\tdie(\"OOPS\");\n \t}\n \n-\tif (unsafe)\n-\t\talgop->unsafe_init_fn(&ctx);\n-\telse\n-\t\talgop->init_fn(&ctx);\n+\talgop->init_fn(&ctx);\n \n \twhile (1) {\n \t\tssize_t sz, this_sz;\n@@ -49,15 +48,9 @@ int cmd_hash_impl(int ac, const char **av, int algo, int unsafe)\n \t\t}\n \t\tif (this_sz == 0)\n \t\t\tbreak;\n-\t\tif (unsafe)\n-\t\t\talgop->unsafe_update_fn(&ctx, buffer, this_sz);\n-\t\telse\n-\t\t\talgop->update_fn(&ctx, buffer, this_sz);\n+\t\talgop->update_fn(&ctx, buffer, this_sz);\n \t}\n-\tif (unsafe)\n-\t\talgop->unsafe_final_fn(hash, &ctx);\n-\telse\n-\t\talgop->final_fn(hash, &ctx);\n+\talgop->final_fn(hash, &ctx);\n \n \tif (binary)\n \t\tfwrite(hash, 1, algop->rawsz, stdout);\n-- \n2.48.0.rc2.35.gd215225db14\n\n"},{"id":"511120","messageId":"da97157c4a1fde62a62dbb258d54a91320712fd7.1737653640.git.me@ttaylorr.com","threadId":"62522","inReplyTo":"cover.1737653640.git.me@ttaylorr.com","subject":"[PATCH v4 7/8] csum-file: introduce hashfile_checkpoint_init()","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-01-23T17:34:39Z","receivedAt":"2025-01-23T17:34:42Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In 106140a99f (builtin/fast-import: fix segfault with unsafe SHA1\nbackend, 2024-12-30) and 9218c0bfe1 (bulk-checkin: fix segfault with\nunsafe SHA1 backend, 2024-12-30), we observed the effects of failing to\ninitialize a hashfile_checkpoint with the same hash function\nimplementation as is used by the hashfile it is used to checkpoint.\n\nWhile both 106140a99f and 9218c0bfe1 work around the immediate crash,\nchanging the hash function implementation within the hashfile API to,\nfor example, the non-unsafe variant would re-introduce the crash. This\nis a result of the tight coupling between initializing hashfiles and\nhashfile_checkpoints.\n\nIntroduce and use a new function which ensures that both parts of a\nhashfile and hashfile_checkpoint pair use the same hash function\nimplementation to avoid such crashes.\n\nA few things worth noting:\n\n  - In the change to builtin/fast-import.c::stream_blob(), we can see\n    that by removing the explicit reference to\n    'the_hash_algo->unsafe_init_fn()', we are hardened against the\n    hashfile API changing away from the_hash_algo (or its unsafe\n    variant) in the future.\n\n  - The bulk-checkin code no longer needs to explicitly zero-initialize\n    the hashfile_checkpoint, since it is now done as a result of calling\n    'hashfile_checkpoint_init()'.\n\n  - Also in the bulk-checkin code, we add an additional call to\n    prepare_to_stream() outside of the main loop in order to initialize\n    'state->f' so we know which hash function implementation to use when\n    calling 'hashfile_checkpoint_init()'.\n\n    This is OK, since subsequent 'prepare_to_stream()' calls are noops.\n    However, we only need to call 'prepare_to_stream()' when we have the\n    HASH_WRITE_OBJECT bit set in our flags. Without that bit, calling\n    'prepare_to_stream()' does not assign 'state->f', so we have nothing\n    to initialize.\n\n  - Other uses of the 'checkpoint' in 'deflate_blob_to_pack()' are\n    appropriately guarded.\n\nHelped-by: Patrick Steinhardt <ps@pks.im>\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n builtin/fast-import.c | 2 +-\n bulk-checkin.c        | 9 ++++++---\n csum-file.c           | 7 +++++++\n csum-file.h           | 1 +\n 4 files changed, 15 insertions(+), 4 deletions(-)\n\ndiff --git a/builtin/fast-import.c b/builtin/fast-import.c\nindex 0f86392761a..4a6c7ab52ac 100644\n--- a/builtin/fast-import.c\n+++ b/builtin/fast-import.c\n@@ -1106,7 +1106,7 @@ static void stream_blob(uintmax_t len, struct object_id *oidout, uintmax_t mark)\n \t\t|| (pack_size + PACK_SIZE_THRESHOLD + len) < pack_size)\n \t\tcycle_packfile();\n \n-\tthe_hash_algo->unsafe_init_fn(&checkpoint.ctx);\n+\thashfile_checkpoint_init(pack_file, &checkpoint);\n \thashfile_checkpoint(pack_file, &checkpoint);\n \toffset = checkpoint.offset;\n \ndiff --git a/bulk-checkin.c b/bulk-checkin.c\nindex 433070a3bda..892176d23d2 100644\n--- a/bulk-checkin.c\n+++ b/bulk-checkin.c\n@@ -261,7 +261,7 @@ static int deflate_blob_to_pack(struct bulk_checkin_packfile *state,\n \tgit_hash_ctx ctx;\n \tunsigned char obuf[16384];\n \tunsigned header_len;\n-\tstruct hashfile_checkpoint checkpoint = {0};\n+\tstruct hashfile_checkpoint checkpoint;\n \tstruct pack_idx_entry *idx = NULL;\n \n \tseekback = lseek(fd, 0, SEEK_CUR);\n@@ -272,12 +272,15 @@ static int deflate_blob_to_pack(struct bulk_checkin_packfile *state,\n \t\t\t\t\t  OBJ_BLOB, size);\n \tthe_hash_algo->init_fn(&ctx);\n \tthe_hash_algo->update_fn(&ctx, obuf, header_len);\n-\tthe_hash_algo->unsafe_init_fn(&checkpoint.ctx);\n \n \t/* Note: idx is non-NULL when we are writing */\n-\tif ((flags & HASH_WRITE_OBJECT) != 0)\n+\tif ((flags & HASH_WRITE_OBJECT) != 0) {\n \t\tCALLOC_ARRAY(idx, 1);\n \n+\t\tprepare_to_stream(state, flags);\n+\t\thashfile_checkpoint_init(state->f, &checkpoint);\n+\t}\n+\n \talready_hashed_to = 0;\n \n \twhile (1) {\ndiff --git a/csum-file.c b/csum-file.c\nindex ebffc80ef71..232121f415f 100644\n--- a/csum-file.c\n+++ b/csum-file.c\n@@ -206,6 +206,13 @@ struct hashfile *hashfd_throughput(int fd, const char *name, struct progress *tp\n \treturn hashfd_internal(fd, name, tp, 8 * 1024);\n }\n \n+void hashfile_checkpoint_init(struct hashfile *f,\n+\t\t\t      struct hashfile_checkpoint *checkpoint)\n+{\n+\tmemset(checkpoint, 0, sizeof(*checkpoint));\n+\tf->algop->init_fn(&checkpoint->ctx);\n+}\n+\n void hashfile_checkpoint(struct hashfile *f, struct hashfile_checkpoint *checkpoint)\n {\n \thashflush(f);\ndiff --git a/csum-file.h b/csum-file.h\nindex 2b45f4673a2..b7475f16c20 100644\n--- a/csum-file.h\n+++ b/csum-file.h\n@@ -36,6 +36,7 @@ struct hashfile_checkpoint {\n \tgit_hash_ctx ctx;\n };\n \n+void hashfile_checkpoint_init(struct hashfile *, struct hashfile_checkpoint *);\n void hashfile_checkpoint(struct hashfile *, struct hashfile_checkpoint *);\n int hashfile_truncate(struct hashfile *, struct hashfile_checkpoint *);\n \n-- \n2.48.0.rc2.35.gd215225db14\n\n"},{"id":"511121","messageId":"0ba27182b5ed666cb9f49c674f2597eec7572b90.1737653640.git.me@ttaylorr.com","threadId":"62522","inReplyTo":"cover.1737653640.git.me@ttaylorr.com","subject":"[PATCH v4 8/8] hash.h: drop unsafe_ function variants","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2025-01-23T17:34:42Z","receivedAt":"2025-01-23T17:34:45Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Now that all callers have been converted from:\n\n    the_hash_algo->unsafe_init_fn();\n\nto\n\n    unsafe_hash_algo(the_hash_algo)->init_fn();\n\nand similar, we can remove the scaffolding for the unsafe_ function\nvariants and force callers to use the new unsafe_hash_algo() mechanic\ninstead.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n hash.h        | 15 ---------------\n object-file.c | 15 ---------------\n 2 files changed, 30 deletions(-)\n\ndiff --git a/hash.h b/hash.h\nindex 0bf63cedfa4..ad2c919991c 100644\n--- a/hash.h\n+++ b/hash.h\n@@ -282,21 +282,6 @@ struct git_hash_algo {\n \t/* The hash finalization function for object IDs. */\n \tgit_hash_final_oid_fn final_oid_fn;\n \n-\t/* The non-cryptographic hash initialization function. */\n-\tgit_hash_init_fn unsafe_init_fn;\n-\n-\t/* The non-cryptographic hash context cloning function. */\n-\tgit_hash_clone_fn unsafe_clone_fn;\n-\n-\t/* The non-cryptographic hash update function. */\n-\tgit_hash_update_fn unsafe_update_fn;\n-\n-\t/* The non-cryptographic hash finalization function. */\n-\tgit_hash_final_fn unsafe_final_fn;\n-\n-\t/* The non-cryptographic hash finalization function. */\n-\tgit_hash_final_oid_fn unsafe_final_oid_fn;\n-\n \t/* The OID of the empty tree. */\n \tconst struct object_id *empty_tree;\n \ndiff --git a/object-file.c b/object-file.c\nindex 43efa4ca361..c4b42dd4be9 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -230,11 +230,6 @@ const struct git_hash_algo hash_algos[GIT_HASH_NALGOS] = {\n \t\t.update_fn = git_hash_unknown_update,\n \t\t.final_fn = git_hash_unknown_final,\n \t\t.final_oid_fn = git_hash_unknown_final_oid,\n-\t\t.unsafe_init_fn = git_hash_unknown_init,\n-\t\t.unsafe_clone_fn = git_hash_unknown_clone,\n-\t\t.unsafe_update_fn = git_hash_unknown_update,\n-\t\t.unsafe_final_fn = git_hash_unknown_final,\n-\t\t.unsafe_final_oid_fn = git_hash_unknown_final_oid,\n \t\t.empty_tree = NULL,\n \t\t.empty_blob = NULL,\n \t\t.null_oid = NULL,\n@@ -250,11 +245,6 @@ const struct git_hash_algo hash_algos[GIT_HASH_NALGOS] = {\n \t\t.update_fn = git_hash_sha1_update,\n \t\t.final_fn = git_hash_sha1_final,\n \t\t.final_oid_fn = git_hash_sha1_final_oid,\n-\t\t.unsafe_init_fn = git_hash_sha1_init_unsafe,\n-\t\t.unsafe_clone_fn = git_hash_sha1_clone_unsafe,\n-\t\t.unsafe_update_fn = git_hash_sha1_update_unsafe,\n-\t\t.unsafe_final_fn = git_hash_sha1_final_unsafe,\n-\t\t.unsafe_final_oid_fn = git_hash_sha1_final_oid_unsafe,\n \t\t.unsafe = &sha1_unsafe_algo,\n \t\t.empty_tree = &empty_tree_oid,\n \t\t.empty_blob = &empty_blob_oid,\n@@ -271,11 +261,6 @@ const struct git_hash_algo hash_algos[GIT_HASH_NALGOS] = {\n \t\t.update_fn = git_hash_sha256_update,\n \t\t.final_fn = git_hash_sha256_final,\n \t\t.final_oid_fn = git_hash_sha256_final_oid,\n-\t\t.unsafe_init_fn = git_hash_sha256_init,\n-\t\t.unsafe_clone_fn = git_hash_sha256_clone,\n-\t\t.unsafe_update_fn = git_hash_sha256_update,\n-\t\t.unsafe_final_fn = git_hash_sha256_final,\n-\t\t.unsafe_final_oid_fn = git_hash_sha256_final_oid,\n \t\t.empty_tree = &empty_tree_oid_sha256,\n \t\t.empty_blob = &empty_blob_oid_sha256,\n \t\t.null_oid = &null_oid_sha256,\n-- \n2.48.0.rc2.35.gd215225db14\n"},{"id":"511130","messageId":"xmqqwmelqt75.fsf@gitster.g","threadId":"62522","inReplyTo":"cover.1737653640.git.me@ttaylorr.com","subject":"Re: [PATCH v4 0/8] hash: introduce unsafe_hash_algo(), drop unsafe_ variants","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-01-23T18:30:06Z","receivedAt":"2025-01-23T18:30:09Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau <me@ttaylorr.com> writes:\n\n> (This series is based on 14650065b7 (RelNotes/2.48.0: fix typos etc.,\n> 2025-01-07)).\n>\n> Here is a hopefully final version of my series to harden the unsafe hash\n> algorithm changes added in v2.47.0. The only difference from last time\n> is that hash_algo_by_ptr() now returns GIT_HASH_UNKNOWN for NULL and\n> unsafe variants, which is a strict improvement from both v3 of this\n> series and the status-quo on master.\n>\n> As usual, a (small) range-diff is included below for convenience, and\n> the original cover letter is as follows:\n\nThanks.  I'll mark it for 'next', unless there are any further\ncomments.\n\n"},{"id":"511131","messageId":"20250123185053.GA3928656@coredump.intra.peff.net","threadId":"62522","inReplyTo":"xmqqwmelqt75.fsf@gitster.g","subject":"Re: [PATCH v4 0/8] hash: introduce unsafe_hash_algo(), drop unsafe_ variants","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2025-01-23T18:50:53Z","receivedAt":"2025-01-23T18:50:55Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Jan 23, 2025 at 10:30:06AM -0800, Junio C Hamano wrote:\n\n> Taylor Blau <me@ttaylorr.com> writes:\n> \n> > (This series is based on 14650065b7 (RelNotes/2.48.0: fix typos etc.,\n> > 2025-01-07)).\n> >\n> > Here is a hopefully final version of my series to harden the unsafe hash\n> > algorithm changes added in v2.47.0. The only difference from last time\n> > is that hash_algo_by_ptr() now returns GIT_HASH_UNKNOWN for NULL and\n> > unsafe variants, which is a strict improvement from both v3 of this\n> > series and the status-quo on master.\n> >\n> > As usual, a (small) range-diff is included below for convenience, and\n> > the original cover letter is as follows:\n> \n> Thanks.  I'll mark it for 'next', unless there are any further\n> comments.\n\nNope, it looks good to me.\n\n-Peff\n"}]}