{"thread":{"id":"66408","subject":"[PATCH 0/4] Add a compile-time option to use the new, very fast sha1dc Rust crate","startedAt":"2026-09-28T15:51:33Z","lastAt":"2026-10-03T21:01:08Z","messageCount":9,"participants":["Johannes Schindelin via GitGitGadget","Johannes Schindelin","Junio C Hamano","brian m. carlson"],"isPatch":true,"patchVersion":1,"patchTotal":4},"messages":[{"id":"553495","messageId":"pull.2240.git.1790610691.gitgitgadget@gmail.com","threadId":"66408","inReplyTo":null,"subject":"[PATCH 0/4] Add a compile-time option to use the new, very fast sha1dc Rust crate","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2026-09-28T15:51:27Z","receivedAt":"2026-09-28T15:51:33Z","isPatch":true,"body":"I stumbled across this new Rust crate last week. Its performance numbers are\nquite impressive. Naturally, I want to make use of this and get for Windows,\nwhich is used on many monorepos where this makes a real difference: In a\npretty fast and loose test, I verified that a git index-pack runs roughly\nthree times faster solely due to using those SIMD-based optimizations!\n\nAs a safety precaution, because this sha1dc crate is quite new, I wanted to\nintroduce an escape hatch: core.sha1dcBackend=c, but turn it on by default,\nwhich is the reason for the three additional patches. Should these patches\nbe undesirable for the Git project? I would not be mad at all if they were\nsimply dropped.\n\nJohannes Schindelin (4):\n  libgitcore: add `sha1dc` as an optional feature\n  sha1dc: allow selecting the C backend without rebuilding\n  pthread: provide `pthread_once()` shims for Windows and for\n    NO_PTHREADS\n  sha1dc: make `sha1dc_init()` thread-safe\n\n Cargo.toml                     |   4 ++\n Documentation/config/core.adoc |   5 ++\n Makefile                       |  29 +++++++++\n compat/win32/pthread.c         |  16 +++++\n compat/win32/pthread.h         |   5 ++\n hash.h                         |   5 ++\n sha1dc_git.c                   | 109 +++++++++++++++++++++++++++++++--\n sha1dc_git.h                   |   5 +-\n sha1dc_rs.h                    |  37 +++++++++++\n src/lib.rs                     |   2 +\n src/sha1dc_rs.rs               |  78 +++++++++++++++++++++++\n t/helper/test-sha1.c           |  19 +++++-\n t/t0013-sha1dc.sh              |  21 ++++++-\n thread-utils.h                 |  16 +++++\n 14 files changed, 343 insertions(+), 8 deletions(-)\n create mode 100644 sha1dc_rs.h\n create mode 100644 src/sha1dc_rs.rs\n\n\nbase-commit: 3bc0341126508f78f5869cbfc0005e987efdf0c7\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-2240%2Fdscho%2Foptionally-use-sha1dc-rs-v1\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-2240/dscho/optionally-use-sha1dc-rs-v1\nPull-Request: https://github.com/gitgitgadget/git/pull/2240\n-- \ngitgitgadget\n"},{"id":"553496","messageId":"b1f30a6a05673c4094d59fda80c695472850d671.1790610691.git.gitgitgadget@gmail.com","threadId":"66408","inReplyTo":"pull.2240.git.1790610691.gitgitgadget@gmail.com","subject":"[PATCH 1/4] libgitcore: add `sha1dc` as an optional feature","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2026-09-28T15:51:28Z","receivedAt":"2026-09-28T15:51:35Z","isPatch":true,"body":"From: Johannes Schindelin <johannes.schindelin@gmx.de>\n\nThe new Rust crate `sha1dc` (https://crates.io/crates/sha1dc) promises\nnot only type safety but also much better performance compared to the\ncollision-detecting SHA-1 library Git uses at the moment. This\nperformance comes mostly from a SIMD-centric design that relies on\nfeatures provided by many x86_64 and aarch64 CPUs.\n\nLet's optionally use this crate, toggled by the build option\n`DC_SHA1_RS`.\n\nTo avoid requiring a shim around the `sha1dc_rs_final()` function just\nto call `die()` upon a detected collision, pass a pointer to that\nfunction to Rust and let it call it directly. This is safe: `die()` is a\nvariadic function, but the Rust code calls it with a simple string\nwithout any interpolation required.\n\nIn a pretty unscientific test on a moderately busy Windows Ryzen 7\nmachine (UCRT64 GCC 16.2), Rust-backed `git-index-pack.exe` using\n`sha1dc` 0.1.3 was over three times faster than the C backend by median\nwall time. Here are the results running five iterations of the `sha1dc`\nC, Rust v0.1.2, and Rust v0.1.3 backends in randomized, balanced order\nwith `--verify --no-rev-index --threads=1 --object-format=sha1`:\n\nSHA1DC backend\tMedian\tBest of five\n--------------\t------\t------------\nC (default)\t32.9s\t32.1s\nRust 0.1.2\t10.9s\t10.4s\nRust 0.1.3\t10.7s\t10.5s\n\nVersion 0.1.3 finished first in four of five triplets, but its median\nadvantage over 0.1.2 was only about 2.4%, and the best 0.1.2 run was\nfaster. In other words, the difference is mostly in the noise.\n\nOn the same machine, using WSL (\"Windows Subsystem for Linux\") with the\nsame packfile copied to Linux' ext4 filesystem:\n\nSHA1DC backend\tMedian\tBest of five\n--------------\t------\t------------\nC (default)\t23.971s\t23.435s\nRust 0.1.2\t9.123s\t8.492s\nRust 0.1.3\t9.020s\t8.491s\n\nThis is overall faster because of the ext4 vs NTFS performance\ncharacteristics, but the same finding holds true: the Rust version of\n`sha1dc` is dramatically faster.\n\nStudying the runs with the Linux perf tools reveals that with the C\nbackend, over 70% of the total time is spent in `git_hash_update()`,\nwith either version of the Rust backend it is around 30%.\n\nA comparable test on an M4 Mac yields these results:\n\nSHA1DC backend \tMedian \tBest of five\n-------------- \t------ \t------------\nC (default) \t8.035s \t8.009s\nRust 0.1.2 \t4.359s \t4.338s\nRust 0.1.3 \t4.361s \t4.268s\n\nNote that the `sha1dc` crate still requires a significantly newer Rust\nversion than Git's existing Rust support requires: 1.87 instead of\n1.63 (https://crates.io/api/v1/crates/sha1dc/0.1.3).\n\nAssisted-by: GPT-6 Sol\nSigned-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>\n---\n Cargo.toml       |  4 +++\n Makefile         | 29 +++++++++++++++++++\n hash.h           |  5 ++++\n sha1dc_rs.h      | 23 ++++++++++++++++\n src/lib.rs       |  2 ++\n src/sha1dc_rs.rs | 72 ++++++++++++++++++++++++++++++++++++++++++++++++\n 6 files changed, 135 insertions(+)\n create mode 100644 sha1dc_rs.h\n create mode 100644 src/sha1dc_rs.rs\n\ndiff --git a/Cargo.toml b/Cargo.toml\nindex 2f51bf5d5f..0a953dd481 100644\n--- a/Cargo.toml\n+++ b/Cargo.toml\n@@ -8,3 +8,7 @@ rust-version = \"1.49.0\"\n crate-type = [\"staticlib\"]\n \n [dependencies]\n+sha1dc = { version = \"0.1.3\", optional = true }\n+\n+[features]\n+sha1dc-rs = [\"sha1dc\"]\ndiff --git a/Makefile b/Makefile\nindex c649c93c51..0c321494cf 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -567,6 +567,9 @@ include shared.mak\n # by the git project to migrate to using sha1collisiondetection as a\n # submodule.\n #\n+# Define DC_SHA1_RS to use the sha1dc Rust crate instead of the default\n+# C implementation. This requires Rust 1.87 or newer.\n+#\n # === SHA-256 backend ===\n #\n # ==== Security ====\n@@ -2137,6 +2140,23 @@ ifdef PPC_SHA1\n $(error the PPC_SHA1 flag has been removed along with the PowerPC-specific SHA-1 implementation.)\n endif\n \n+ifdef DC_SHA1_RS\n+ifdef NO_RUST\n+$(error DC_SHA1_RS requires Rust support)\n+endif\n+ifneq ($(strip $(OPENSSL_SHA1)$(BLK_SHA1)$(APPLE_COMMON_CRYPTO_SHA1)),)\n+$(error DC_SHA1_RS cannot be combined with another SHA-1 backend)\n+endif\n+ifdef DC_SHA1_EXTERNAL\n+$(error Only set DC_SHA1_RS or DC_SHA1_EXTERNAL, not both)\n+endif\n+ifdef DC_SHA1_SUBMODULE\n+ifneq ($(DC_SHA1_SUBMODULE),auto)\n+$(error Only set DC_SHA1_RS or DC_SHA1_SUBMODULE, not both)\n+endif\n+endif\n+endif\n+\n ifdef OPENSSL_SHA1\n \tEXTLIBS += $(LIB_4_CRYPTO)\n \tBASIC_CFLAGS += -DSHA1_OPENSSL\n@@ -2150,6 +2170,14 @@ ifdef APPLE_COMMON_CRYPTO_SHA1\n \tBASIC_CFLAGS += -DSHA1_APPLE\n else\n \tBASIC_CFLAGS += -DSHA1_DC\n+ifdef DC_SHA1_RS\n+\tBASIC_CFLAGS += -DDC_SHA1_RS\n+\tCARGO_ARGS += --features sha1dc-rs\n+\tRUST_SOURCES += src/sha1dc_rs.rs\n+ifeq ($(uname_S),MINGW)\n+\tEXTLIBS += -luserenv\n+endif\n+else\n \tLIB_OBJS += sha1dc_git.o\n ifdef DC_SHA1_EXTERNAL\n         ifdef DC_SHA1_SUBMODULE\n@@ -2177,6 +2205,7 @@ endif\n endif\n endif\n endif\n+endif\n \n ifdef OPENSSL_SHA1_UNSAFE\n ifndef OPENSSL_SHA1\ndiff --git a/hash.h b/hash.h\nindex cf94ad5700..dd2e66e1c9 100644\n--- a/hash.h\n+++ b/hash.h\n@@ -12,8 +12,13 @@\n #    include \"sha1/openssl.h\"\n #  endif\n #elif defined(SHA1_DC)\n+#ifdef DC_SHA1_RS\n+#define SHA1_BACKEND \"SHA1_DC-rs\"\n+#include \"sha1dc_rs.h\"\n+#else\n #define SHA1_BACKEND \"SHA1_DC\"\n #include \"sha1dc_git.h\"\n+#endif\n #else /* SHA1_BLK */\n #define SHA1_BACKEND \"SHA1_BLK (No collision detection)\"\n #include \"block-sha1/sha1.h\"\ndiff --git a/sha1dc_rs.h b/sha1dc_rs.h\nnew file mode 100644\nindex 0000000000..35e3865d72\n--- /dev/null\n+++ b/sha1dc_rs.h\n@@ -0,0 +1,23 @@\n+#ifndef SHA1DC_RS_H\n+#define SHA1DC_RS_H\n+\n+#define platform_SHA_IS_SHA1DC /* used by \"test-tool sha1-is-sha1dc\" */\n+\n+typedef struct sha1dc_rs_hasher *SHA1_CTX;\n+\n+void sha1dc_rs_init(SHA1_CTX *);\n+void sha1dc_rs_clone(SHA1_CTX *, const SHA1_CTX *);\n+void sha1dc_rs_update(SHA1_CTX *, const void *, size_t);\n+void sha1dc_rs_final(unsigned char [20], SHA1_CTX *,\n+\t\t     void (*die_fn)(const char *, ...));\n+void sha1dc_rs_discard(SHA1_CTX *);\n+\n+#define platform_SHA_CTX SHA1_CTX\n+#define platform_SHA1_Init sha1dc_rs_init\n+#define platform_SHA1_Update sha1dc_rs_update\n+#define platform_SHA1_Final(hash, ctx) sha1dc_rs_final((hash), (ctx), die)\n+#define SHA1_NEEDS_CLONE_HELPER\n+#define platform_SHA1_Clone sha1dc_rs_clone\n+#define platform_SHA1_Discard sha1dc_rs_discard\n+\n+#endif\ndiff --git a/src/lib.rs b/src/lib.rs\nindex 0c598298b1..a34f4d489c 100644\n--- a/src/lib.rs\n+++ b/src/lib.rs\n@@ -1,4 +1,6 @@\n pub mod csum_file;\n pub mod hash;\n pub mod loose;\n+#[cfg(feature = \"sha1dc-rs\")]\n+mod sha1dc_rs;\n pub mod varint;\ndiff --git a/src/sha1dc_rs.rs b/src/sha1dc_rs.rs\nnew file mode 100644\nindex 0000000000..df075a5d83\n--- /dev/null\n+++ b/src/sha1dc_rs.rs\n@@ -0,0 +1,72 @@\n+use sha1dc::Hasher;\n+use std::ffi::CString;\n+use std::os::raw::c_char;\n+use std::{ptr, slice};\n+\n+/// Initialize a collision-detecting SHA-1 context.\n+///\n+/// # Safety\n+/// `ctx` must point to an uninitialized SHA-1 context.\n+#[no_mangle]\n+pub unsafe extern \"C\" fn sha1dc_rs_init(ctx: *mut *mut Hasher) {\n+    *ctx = Box::into_raw(Box::new(Hasher::new()));\n+}\n+\n+/// Replace a SHA-1 context with a clone of another.\n+///\n+/// # Safety\n+/// Both contexts must be initialized.\n+#[no_mangle]\n+pub unsafe extern \"C\" fn sha1dc_rs_clone(dst: *mut *mut Hasher, src: *const *mut Hasher) {\n+    let hasher = Box::new((**src).clone());\n+    drop(Box::from_raw(*dst));\n+    *dst = Box::into_raw(hasher);\n+}\n+\n+/// Update the SHA-1 hasher with the given bytes.\n+///\n+/// # Safety\n+/// `ctx` must be initialized and `data` must point to `len` bytes unless\n+/// `len` is zero.\n+#[no_mangle]\n+pub unsafe extern \"C\" fn sha1dc_rs_update(ctx: *mut *mut Hasher, data: *const c_char, len: usize) {\n+    if len != 0 {\n+        (**ctx).update(slice::from_raw_parts(data.cast::<u8>(), len));\n+    }\n+}\n+\n+/// Finalize SHA-1, reporting detected collisions through `die`.\n+///\n+/// # Safety\n+/// `ctx` must be initialized, `hash` must point to at least 20 bytes, and\n+/// `die` must be a non-returning C variadic function.\n+#[no_mangle]\n+pub unsafe extern \"C\" fn sha1dc_rs_final(\n+    hash: *mut u8,\n+    ctx: *mut *mut Hasher,\n+    die: unsafe extern \"C\" fn(*const c_char, ...) -> !,\n+) {\n+    let hasher = *Box::from_raw(*ctx);\n+    *ctx = ptr::null_mut();\n+    match hasher.finalize() {\n+        Ok(digest) => ptr::copy_nonoverlapping(digest.as_bytes().as_ptr(), hash, 20),\n+        Err(collision) => {\n+            let message = CString::new(format!(\n+                \"SHA-1 appears to be part of a collision attack: {}\",\n+                collision.digest()\n+            ))\n+            .expect(\"collision message contains no NUL\");\n+            die(message.as_ptr());\n+        }\n+    }\n+}\n+\n+/// Discard a SHA-1 context without producing a digest.\n+///\n+/// # Safety\n+/// `ctx` must be initialized.\n+#[no_mangle]\n+pub unsafe extern \"C\" fn sha1dc_rs_discard(ctx: *mut *mut Hasher) {\n+    drop(Box::from_raw(*ctx));\n+    *ctx = ptr::null_mut();\n+}\n-- \ngitgitgadget\n\n"},{"id":"553497","messageId":"5a414a4babf9cb755b4cf43eb42d1f06c8b45d6c.1790610691.git.gitgitgadget@gmail.com","threadId":"66408","inReplyTo":"pull.2240.git.1790610691.gitgitgadget@gmail.com","subject":"[PATCH 2/4] sha1dc: allow selecting the C backend without rebuilding","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2026-09-28T15:51:29Z","receivedAt":"2026-09-28T15:51:36Z","isPatch":true,"body":"From: Johannes Schindelin <johannes.schindelin@gmx.de>\n\nThe Rust `sha1dc` create is really new. While it produced only correct\nhashes in my hands, before unleashing this to the masses, we need to\nprovide an \"escape hatch\" in case it doesn't do the right thing.\n\nTherefore, when building with `DC_SHA1_RS`, use the Rust `sha1dc` by\ndefault, yet also offer to use the C version of `sha1dc` via\n`core.sha1dcBackend=c` (and `core.sha1dcBackend=rust` to select Rust\nexplicitly).\n\nThis is made possible by a set of function pointers that are initialized\nupon the first call to the `git_hash_init()` function.\n\nNote that the order in which `hex.h` and `sha1dc_git.h` are included in\n`sha1dc_git.c` now have to be turned the other way round, to avoid\nredefining the `platform_SHA*` constants.\n\nAssisted-by: GPT-6 Sol\nSigned-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>\n---\n Documentation/config/core.adoc |  5 +++\n Makefile                       | 10 +++---\n sha1dc_git.c                   | 60 +++++++++++++++++++++++++++++++---\n sha1dc_git.h                   |  5 +--\n sha1dc_rs.h                    | 30 ++++++++++++-----\n src/sha1dc_rs.rs               | 18 ++++++----\n t/helper/test-sha1.c           | 19 ++++++++++-\n t/t0013-sha1dc.sh              | 21 +++++++++++-\n 8 files changed, 141 insertions(+), 27 deletions(-)\n\ndiff --git a/Documentation/config/core.adoc b/Documentation/config/core.adoc\nindex 340329edc3..2ef56a107e 100644\n--- a/Documentation/config/core.adoc\n+++ b/Documentation/config/core.adoc\n@@ -382,6 +382,11 @@ core.repositoryFormatVersion::\n \tInternal variable identifying the repository format and layout\n \tversion. See linkgit:gitrepository-layout[5].\n \n+core.sha1dcBackend::\n+\tSelect the collision-detecting SHA-1 implementation when Git is built\n+\twith `DC_SHA1_RS`: valid values are `rust` (the default) and `c` (the C\n+\tfallback). This setting has no effect with other SHA-1 backends.\n+\n core.sharedRepository::\n \tWhen 'group' (or 'true'), the repository is made shareable between\n \tseveral users in a group (making sure all the files and objects are\ndiff --git a/Makefile b/Makefile\nindex 0c321494cf..31afa7d917 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -567,8 +567,9 @@ include shared.mak\n # by the git project to migrate to using sha1collisiondetection as a\n # submodule.\n #\n-# Define DC_SHA1_RS to use the sha1dc Rust crate instead of the default\n-# C implementation. This requires Rust 1.87 or newer.\n+# Define DC_SHA1_RS to use the sha1dc Rust crate by default, with the C\n+# implementation available via core.sha1dcBackend=c. This requires Rust\n+# 1.87 or newer.\n #\n # === SHA-256 backend ===\n #\n@@ -2170,6 +2171,7 @@ ifdef APPLE_COMMON_CRYPTO_SHA1\n \tBASIC_CFLAGS += -DSHA1_APPLE\n else\n \tBASIC_CFLAGS += -DSHA1_DC\n+\tLIB_OBJS += sha1dc_git.o\n ifdef DC_SHA1_RS\n \tBASIC_CFLAGS += -DDC_SHA1_RS\n \tCARGO_ARGS += --features sha1dc-rs\n@@ -2177,8 +2179,7 @@ ifdef DC_SHA1_RS\n ifeq ($(uname_S),MINGW)\n \tEXTLIBS += -luserenv\n endif\n-else\n-\tLIB_OBJS += sha1dc_git.o\n+endif\n ifdef DC_SHA1_EXTERNAL\n         ifdef DC_SHA1_SUBMODULE\n                 ifneq ($(DC_SHA1_SUBMODULE),auto)\n@@ -2205,7 +2206,6 @@ endif\n endif\n endif\n endif\n-endif\n \n ifdef OPENSSL_SHA1_UNSAFE\n ifndef OPENSSL_SHA1\ndiff --git a/sha1dc_git.c b/sha1dc_git.c\nindex fe58d7962a..dcc5c1ca8e 100644\n--- a/sha1dc_git.c\n+++ b/sha1dc_git.c\n@@ -1,6 +1,13 @@\n+#ifdef DC_SHA1_RS\n+#define USE_THE_REPOSITORY_VARIABLE\n+#endif\n #include \"git-compat-util.h\"\n-#include \"sha1dc_git.h\"\n #include \"hex.h\"\n+#include \"sha1dc_git.h\"\n+#ifdef DC_SHA1_RS\n+#include \"config.h\"\n+#include \"repository.h\"\n+#endif\n \n #ifdef DC_SHA1_EXTERNAL\n /*\n@@ -16,12 +23,13 @@ void git_SHA1DCInit(SHA1_CTX *ctx)\n /*\n  * Same as SHA1DCFinal, but convert collision attack case into a verbose die().\n  */\n-void git_SHA1DCFinal(unsigned char hash[20], SHA1_CTX *ctx)\n+void git_SHA1DCFinal(unsigned char hash[20], SHA1_CTX *ctx,\n+\t\t     void (*die_fn)(const char *, ...))\n {\n \tif (!SHA1DCFinal(hash, ctx))\n \t\treturn;\n-\tdie(\"SHA-1 appears to be part of a collision attack: %s\",\n-\t    hash_to_hex_algop(hash, &hash_algos[GIT_HASH_SHA1]));\n+\tdie_fn(\"SHA-1 appears to be part of a collision attack: %s\",\n+\t       hash_to_hex_algop(hash, &hash_algos[GIT_HASH_SHA1]));\n }\n \n /*\n@@ -37,3 +45,47 @@ void git_SHA1DCUpdate(SHA1_CTX *ctx, const void *vdata, size_t len)\n \t}\n \tSHA1DCUpdate(ctx, data, len);\n }\n+\n+#ifdef DC_SHA1_RS\n+static void sha1dc_c_clone(SHA1_CTX *dst, const SHA1_CTX *src)\n+{\n+\t*dst = *src;\n+}\n+\n+static void sha1dc_c_discard(SHA1_CTX *ctx UNUSED)\n+{\n+\t/* The C context owns no resources. */\n+}\n+\n+/* The first SHA-1 initialization must precede concurrent hashing. */\n+static void sha1dc_choose(SHA1_CTX *ctx);\n+\n+void (*sha1dc_init)(SHA1_CTX *) = sha1dc_choose;\n+void (*sha1dc_clone)(SHA1_CTX *, const SHA1_CTX *);\n+void (*sha1dc_update)(SHA1_CTX *, const void *, size_t);\n+void (*sha1dc_final)(unsigned char [20], SHA1_CTX *,\n+\t\t     void (*die_fn)(const char *, ...));\n+void (*sha1dc_discard)(SHA1_CTX *);\n+\n+static void sha1dc_choose(SHA1_CTX *ctx)\n+{\n+\tconst char *backend;\n+\tint use_c = 0;\n+\n+\tif (!repo_config_get_string_tmp(the_repository, \"core.sha1dcbackend\",\n+\t\t\t\t\t&backend)) {\n+\t\tif (!strcasecmp(backend, \"c\"))\n+\t\t\tuse_c = 1;\n+\t\telse if (strcasecmp(backend, \"rust\"))\n+\t\t\tdie(\"invalid value for core.sha1dcBackend: '%s'\",\n+\t\t\t    backend);\n+\t}\n+\n+\tsha1dc_clone = use_c ? sha1dc_c_clone : sha1dc_rs_clone;\n+\tsha1dc_update = use_c ? git_SHA1DCUpdate : sha1dc_rs_update;\n+\tsha1dc_final = use_c ? git_SHA1DCFinal : sha1dc_rs_final;\n+\tsha1dc_discard = use_c ? sha1dc_c_discard : sha1dc_rs_discard;\n+\tsha1dc_init = use_c ? git_SHA1DCInit : sha1dc_rs_init;\n+\tsha1dc_init(ctx);\n+}\n+#endif\ndiff --git a/sha1dc_git.h b/sha1dc_git.h\nindex 0bcf1aa84b..4c4aefe3e8 100644\n--- a/sha1dc_git.h\n+++ b/sha1dc_git.h\n@@ -14,7 +14,8 @@ void git_SHA1DCInit(SHA1_CTX *);\n #define git_SHA1DCInit\tSHA1DCInit\n #endif\n \n-void git_SHA1DCFinal(unsigned char [20], SHA1_CTX *);\n+void git_SHA1DCFinal(unsigned char [20], SHA1_CTX *,\n+\t\t    void (*die_fn)(const char *, ...));\n void git_SHA1DCUpdate(SHA1_CTX *ctx, const void *data, size_t len);\n \n #define platform_SHA_IS_SHA1DC /* used by \"test-tool sha1-is-sha1dc\" */\n@@ -23,5 +24,5 @@ void git_SHA1DCUpdate(SHA1_CTX *ctx, const void *data, size_t len);\n #define platform_SHA_CTX SHA1_CTX\n #define platform_SHA1_Init git_SHA1DCInit\n #define platform_SHA1_Update git_SHA1DCUpdate\n-#define platform_SHA1_Final git_SHA1DCFinal\n+#define platform_SHA1_Final(hash, ctx) git_SHA1DCFinal((hash), (ctx), die)\n #endif\ndiff --git a/sha1dc_rs.h b/sha1dc_rs.h\nindex 35e3865d72..e5d6345363 100644\n--- a/sha1dc_rs.h\n+++ b/sha1dc_rs.h\n@@ -1,9 +1,15 @@\n #ifndef SHA1DC_RS_H\n #define SHA1DC_RS_H\n \n-#define platform_SHA_IS_SHA1DC /* used by \"test-tool sha1-is-sha1dc\" */\n+#define platform_SHA_CTX union sha1dc_ctx\n+#include \"sha1dc_git.h\"\n \n-typedef struct sha1dc_rs_hasher *SHA1_CTX;\n+typedef struct sha1dc_rs_hasher *sha1dc_rs_ctx;\n+\n+union sha1dc_ctx {\n+\tSHA1_CTX c;\n+\tsha1dc_rs_ctx rs;\n+};\n \n void sha1dc_rs_init(SHA1_CTX *);\n void sha1dc_rs_clone(SHA1_CTX *, const SHA1_CTX *);\n@@ -12,12 +18,20 @@ void sha1dc_rs_final(unsigned char [20], SHA1_CTX *,\n \t\t     void (*die_fn)(const char *, ...));\n void sha1dc_rs_discard(SHA1_CTX *);\n \n-#define platform_SHA_CTX SHA1_CTX\n-#define platform_SHA1_Init sha1dc_rs_init\n-#define platform_SHA1_Update sha1dc_rs_update\n-#define platform_SHA1_Final(hash, ctx) sha1dc_rs_final((hash), (ctx), die)\n+extern void (*sha1dc_init)(SHA1_CTX *);\n+extern void (*sha1dc_clone)(SHA1_CTX *, const SHA1_CTX *);\n+extern void (*sha1dc_update)(SHA1_CTX *, const void *, size_t);\n+extern void (*sha1dc_final)(unsigned char [20], SHA1_CTX *,\n+\t\t\t    void (*die_fn)(const char *, ...));\n+extern void (*sha1dc_discard)(SHA1_CTX *);\n+\n+#define platform_SHA1_Init(ctx) sha1dc_init(&(ctx)->c)\n+#define platform_SHA1_Update(ctx, data, len) \\\n+\tsha1dc_update(&(ctx)->c, (data), (len))\n+#define platform_SHA1_Final(hash, ctx) \\\n+\tsha1dc_final((hash), &(ctx)->c, die)\n #define SHA1_NEEDS_CLONE_HELPER\n-#define platform_SHA1_Clone sha1dc_rs_clone\n-#define platform_SHA1_Discard sha1dc_rs_discard\n+#define platform_SHA1_Clone(dst, src) sha1dc_clone(&(dst)->c, &(src)->c)\n+#define platform_SHA1_Discard(ctx) sha1dc_discard(&(ctx)->c)\n \n #endif\ndiff --git a/src/sha1dc_rs.rs b/src/sha1dc_rs.rs\nindex df075a5d83..b9c430d0dc 100644\n--- a/src/sha1dc_rs.rs\n+++ b/src/sha1dc_rs.rs\n@@ -1,5 +1,5 @@\n use sha1dc::Hasher;\n-use std::ffi::CString;\n+use std::ffi::{c_void, CString};\n use std::os::raw::c_char;\n use std::{ptr, slice};\n \n@@ -8,7 +8,8 @@ use std::{ptr, slice};\n /// # Safety\n /// `ctx` must point to an uninitialized SHA-1 context.\n #[no_mangle]\n-pub unsafe extern \"C\" fn sha1dc_rs_init(ctx: *mut *mut Hasher) {\n+pub unsafe extern \"C\" fn sha1dc_rs_init(ctx: *mut c_void) {\n+    let ctx = ctx.cast::<*mut Hasher>();\n     *ctx = Box::into_raw(Box::new(Hasher::new()));\n }\n \n@@ -17,7 +18,9 @@ pub unsafe extern \"C\" fn sha1dc_rs_init(ctx: *mut *mut Hasher) {\n /// # Safety\n /// Both contexts must be initialized.\n #[no_mangle]\n-pub unsafe extern \"C\" fn sha1dc_rs_clone(dst: *mut *mut Hasher, src: *const *mut Hasher) {\n+pub unsafe extern \"C\" fn sha1dc_rs_clone(dst: *mut c_void, src: *const c_void) {\n+    let dst = dst.cast::<*mut Hasher>();\n+    let src = src.cast::<*mut Hasher>();\n     let hasher = Box::new((**src).clone());\n     drop(Box::from_raw(*dst));\n     *dst = Box::into_raw(hasher);\n@@ -29,7 +32,8 @@ pub unsafe extern \"C\" fn sha1dc_rs_clone(dst: *mut *mut Hasher, src: *const *mut\n /// `ctx` must be initialized and `data` must point to `len` bytes unless\n /// `len` is zero.\n #[no_mangle]\n-pub unsafe extern \"C\" fn sha1dc_rs_update(ctx: *mut *mut Hasher, data: *const c_char, len: usize) {\n+pub unsafe extern \"C\" fn sha1dc_rs_update(ctx: *mut c_void, data: *const c_void, len: usize) {\n+    let ctx = ctx.cast::<*mut Hasher>();\n     if len != 0 {\n         (**ctx).update(slice::from_raw_parts(data.cast::<u8>(), len));\n     }\n@@ -43,9 +47,10 @@ pub unsafe extern \"C\" fn sha1dc_rs_update(ctx: *mut *mut Hasher, data: *const c_\n #[no_mangle]\n pub unsafe extern \"C\" fn sha1dc_rs_final(\n     hash: *mut u8,\n-    ctx: *mut *mut Hasher,\n+    ctx: *mut c_void,\n     die: unsafe extern \"C\" fn(*const c_char, ...) -> !,\n ) {\n+    let ctx = ctx.cast::<*mut Hasher>();\n     let hasher = *Box::from_raw(*ctx);\n     *ctx = ptr::null_mut();\n     match hasher.finalize() {\n@@ -66,7 +71,8 @@ pub unsafe extern \"C\" fn sha1dc_rs_final(\n /// # Safety\n /// `ctx` must be initialized.\n #[no_mangle]\n-pub unsafe extern \"C\" fn sha1dc_rs_discard(ctx: *mut *mut Hasher) {\n+pub unsafe extern \"C\" fn sha1dc_rs_discard(ctx: *mut c_void) {\n+    let ctx = ctx.cast::<*mut Hasher>();\n     drop(Box::from_raw(*ctx));\n     *ctx = ptr::null_mut();\n }\ndiff --git a/t/helper/test-sha1.c b/t/helper/test-sha1.c\nindex 349540c4df..827fd2d6d5 100644\n--- a/t/helper/test-sha1.c\n+++ b/t/helper/test-sha1.c\n@@ -1,13 +1,30 @@\n+#define USE_THE_REPOSITORY_VARIABLE\n #include \"test-tool.h\"\n #include \"hash.h\"\n+#include \"setup.h\"\n \n int cmd__sha1(int ac, const char **av)\n {\n \treturn cmd_hash_impl(ac, av, GIT_HASH_SHA1, 0);\n }\n \n-int cmd__sha1_is_sha1dc(int argc UNUSED, const char **argv UNUSED)\n+int cmd__sha1_is_sha1dc(int argc, const char **argv)\n {\n+#ifdef DC_SHA1_RS\n+\tif (argc == 2 && !strcmp(argv[1], \"--backend\")) {\n+\t\tgit_SHA_CTX ctx;\n+\t\tint nongit;\n+\n+\t\tsetup_git_directory_gently(the_repository, &nongit);\n+\t\tgit_SHA1_Init(&ctx);\n+\t\tputs(sha1dc_init == git_SHA1DCInit ? \"c\" : \"rust\");\n+\t\tgit_SHA1_Discard(&ctx);\n+\t\treturn 0;\n+\t}\n+#else\n+\tif (argc == 2 && !strcmp(argv[1], \"--backend\"))\n+\t\treturn 1;\n+#endif\n #ifdef platform_SHA_IS_SHA1DC\n \treturn 0;\n #endif\ndiff --git a/t/t0013-sha1dc.sh b/t/t0013-sha1dc.sh\nindex 3ea3169d92..9f6b72f8ef 100755\n--- a/t/t0013-sha1dc.sh\n+++ b/t/t0013-sha1dc.sh\n@@ -13,10 +13,29 @@ then\n \ttest_done\n fi\n \n+test_lazy_prereq SHA1DC_RS '\n+\ttest rust = \"$(GIT_CONFIG_PARAMETERS=\"${SQ}core.sha1dcBackend=rust${SQ}\" \\\n+\t\ttest-tool sha1-is-sha1dc --backend)\"\n+'\n+\n test_expect_success 'test-sha1 detects shattered pdf' '\n \ttest_must_fail test-tool sha1 <\"$TEST_DATA/shattered-1.pdf\" 2>err &&\n \ttest_grep collision err &&\n-\ttest_grep 38762cf7f55934b34d179ae6a4c80cadccbb7f0a err\n+\ttest_grep 38762cf7f55934b34d179ae6a4c80cadccbb7f0a err &&\n+\tif test_have_prereq SHA1DC_RS\n+\tthen\n+\t\ttest_must_fail env \\\n+\t\t\tGIT_CONFIG_PARAMETERS=\"${SQ}core.sha1dcBackend=c${SQ}\" \\\n+\t\t\ttest-tool sha1 <\"$TEST_DATA/shattered-1.pdf\" 2>err &&\n+\t\ttest_grep collision err &&\n+\t\ttest_grep 38762cf7f55934b34d179ae6a4c80cadccbb7f0a err\n+\tfi\n+'\n+\n+test_expect_success SHA1DC_RS 'select SHA1DC backend via config' '\n+\ttest rust = \"$(test-tool sha1-is-sha1dc --backend)\" &&\n+\ttest_config core.sha1dcBackend c &&\n+\ttest c = \"$(test-tool sha1-is-sha1dc --backend)\"\n '\n \n test_done\n-- \ngitgitgadget\n\n"},{"id":"553498","messageId":"2f3077b60a7777b8d4c21551f9a8d02034a49cee.1790610691.git.gitgitgadget@gmail.com","threadId":"66408","inReplyTo":"pull.2240.git.1790610691.gitgitgadget@gmail.com","subject":"[PATCH 3/4] pthread: provide `pthread_once()` shims for Windows and for NO_PTHREADS","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2026-09-28T15:51:30Z","receivedAt":"2026-09-28T15:51:38Z","isPatch":true,"body":"From: Johannes Schindelin <johannes.schindelin@gmx.de>\n\nI am about to introduce logic that needs to perform some initialization\nonce, and once only, even if called concurrently.\n\nThis is a perfect job for `pthread_once()`, but Git's source code\ncurrently lacks a Win32 shim. So let's add one!\n\nAlso provide a trivial shim for `NO_PTHREAD` builds.\n\nAssisted-by: GPT-6 Sol\nSigned-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>\n---\n compat/win32/pthread.c | 16 ++++++++++++++++\n compat/win32/pthread.h |  5 +++++\n thread-utils.h         | 16 ++++++++++++++++\n 3 files changed, 37 insertions(+)\n\ndiff --git a/compat/win32/pthread.c b/compat/win32/pthread.c\nindex 398caa9602..5af95edd3b 100644\n--- a/compat/win32/pthread.c\n+++ b/compat/win32/pthread.c\n@@ -60,6 +60,22 @@ pthread_t pthread_self(void)\n \treturn t;\n }\n \n+static BOOL CALLBACK win32_pthread_once(PINIT_ONCE once UNUSED,\n+\t\t\t\t\tPVOID parameter,\n+\t\t\t\t\tPVOID *context UNUSED)\n+{\n+\t(*(void (**)(void))parameter)();\n+\treturn TRUE;\n+}\n+\n+int pthread_once(pthread_once_t *once_control, void (*init_routine)(void))\n+{\n+\tif (!InitOnceExecuteOnce(once_control, win32_pthread_once,\n+\t\t\t\t &init_routine, NULL))\n+\t\treturn err_win_to_posix(GetLastError());\n+\treturn 0;\n+}\n+\n int pthread_cond_wait(pthread_cond_t *cond, pthread_mutex_t *mutex)\n {\n \tif (SleepConditionVariableCS(cond, mutex, INFINITE) == 0)\ndiff --git a/compat/win32/pthread.h b/compat/win32/pthread.h\nindex d80df8d12a..79a6bd9680 100644\n--- a/compat/win32/pthread.h\n+++ b/compat/win32/pthread.h\n@@ -26,6 +26,11 @@ static inline int return_0(int i UNUSED) {\n #define pthread_mutex_lock EnterCriticalSection\n #define pthread_mutex_unlock LeaveCriticalSection\n \n+typedef INIT_ONCE pthread_once_t;\n+#define PTHREAD_ONCE_INIT INIT_ONCE_STATIC_INIT\n+\n+int pthread_once(pthread_once_t *once_control, void (*init_routine)(void));\n+\n typedef int pthread_mutexattr_t;\n #define pthread_mutexattr_init(a) (*(a) = 0)\n #define pthread_mutexattr_destroy(a) do {} while (0)\ndiff --git a/thread-utils.h b/thread-utils.h\nindex 4961487ed9..98b574eff4 100644\n--- a/thread-utils.h\n+++ b/thread-utils.h\n@@ -19,6 +19,22 @@\n #define pthread_mutex_t int\n #define pthread_cond_t int\n #define pthread_key_t int\n+#define pthread_once_t int\n+#undef PTHREAD_ONCE_INIT\n+#define PTHREAD_ONCE_INIT 0\n+\n+static inline int dummy_pthread_once(pthread_once_t *once_control,\n+\t\t\t\t      void (*init_routine)(void))\n+{\n+\tif (!*once_control) {\n+\t\tinit_routine();\n+\t\t*once_control = 1;\n+\t}\n+\treturn 0;\n+}\n+\n+#define pthread_once(once_control, init_routine) \\\n+\tdummy_pthread_once((once_control), (init_routine))\n \n #define pthread_mutex_init(mutex, attr) dummy_pthread_init(mutex)\n #define pthread_mutex_lock(mutex)\n-- \ngitgitgadget\n\n"},{"id":"553499","messageId":"aac6a83a8ebd91f33cbe2074245b6d45cb264105.1790610691.git.gitgitgadget@gmail.com","threadId":"66408","inReplyTo":"pull.2240.git.1790610691.gitgitgadget@gmail.com","subject":"[PATCH 4/4] sha1dc: make `sha1dc_init()` thread-safe","fromName":"Johannes Schindelin via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2026-09-28T15:51:31Z","receivedAt":"2026-09-28T15:51:39Z","isPatch":true,"body":"From: Johannes Schindelin <johannes.schindelin@gmx.de>\n\nThe `sha1dc_init` function pointer initially points to a function that\ndetermines which sha1dc backend to use. Naturally, this initialization\nshould only run once.\n\nTo allow for that function to be called concurrently in multiple\nthreads, we need to use a pthread primitive to ensure that the\n`sha1dc_*()` function pointers are initialized exactly once.\n\nUnfortunately, this requires quite a bit of non-DRY code to prevent data\nraces when different threads run `initial_init()` concurrently (see\nhttps://en.cppreference.com/c/language/memory_model#Threads_and_data_races).\n\nSigned-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>\n---\n sha1dc_git.c | 63 ++++++++++++++++++++++++++++++++++++++++++++++------\n 1 file changed, 56 insertions(+), 7 deletions(-)\n\ndiff --git a/sha1dc_git.c b/sha1dc_git.c\nindex dcc5c1ca8e..eba62b12ab 100644\n--- a/sha1dc_git.c\n+++ b/sha1dc_git.c\n@@ -7,6 +7,7 @@\n #ifdef DC_SHA1_RS\n #include \"config.h\"\n #include \"repository.h\"\n+#include \"thread-utils.h\"\n #endif\n \n #ifdef DC_SHA1_EXTERNAL\n@@ -58,16 +59,21 @@ static void sha1dc_c_discard(SHA1_CTX *ctx UNUSED)\n }\n \n /* The first SHA-1 initialization must precede concurrent hashing. */\n-static void sha1dc_choose(SHA1_CTX *ctx);\n+static void initial_init(SHA1_CTX *);\n+static void initial_clone(SHA1_CTX *, const SHA1_CTX *);\n+static void initial_update(SHA1_CTX *, const void *, size_t);\n+static void initial_final(unsigned char [20], SHA1_CTX *,\n+\t\t\t  void (*die_fn)(const char *, ...));\n+static void initial_discard(SHA1_CTX *ctx);\n \n-void (*sha1dc_init)(SHA1_CTX *) = sha1dc_choose;\n-void (*sha1dc_clone)(SHA1_CTX *, const SHA1_CTX *);\n-void (*sha1dc_update)(SHA1_CTX *, const void *, size_t);\n+void (*sha1dc_init)(SHA1_CTX *) = initial_init;\n+void (*sha1dc_clone)(SHA1_CTX *, const SHA1_CTX *) = initial_clone;\n+void (*sha1dc_update)(SHA1_CTX *, const void *, size_t) = initial_update;\n void (*sha1dc_final)(unsigned char [20], SHA1_CTX *,\n-\t\t     void (*die_fn)(const char *, ...));\n-void (*sha1dc_discard)(SHA1_CTX *);\n+\t\t     void (*die_fn)(const char *, ...)) = initial_final;\n+void (*sha1dc_discard)(SHA1_CTX *) = initial_discard;\n \n-static void sha1dc_choose(SHA1_CTX *ctx)\n+static void sha1dc_choose(void)\n {\n \tconst char *backend;\n \tint use_c = 0;\n@@ -86,6 +92,49 @@ static void sha1dc_choose(SHA1_CTX *ctx)\n \tsha1dc_final = use_c ? git_SHA1DCFinal : sha1dc_rs_final;\n \tsha1dc_discard = use_c ? sha1dc_c_discard : sha1dc_rs_discard;\n \tsha1dc_init = use_c ? git_SHA1DCInit : sha1dc_rs_init;\n+}\n+\n+static pthread_once_t once = PTHREAD_ONCE_INIT;\n+\n+static void initial_init(SHA1_CTX *ctx)\n+{\n+\tint ret = pthread_once(&once, sha1dc_choose);\n+\tif (ret)\n+\t\tdie(\"cannot initialize SHA-1 backend: %s\", strerror(ret));\n \tsha1dc_init(ctx);\n }\n+\n+static void initial_clone(SHA1_CTX *dst, const SHA1_CTX *src)\n+{\n+\tint ret = pthread_once(&once, sha1dc_choose);\n+\tif (ret)\n+\t\tdie(\"cannot initialize SHA-1 backend: %s\", strerror(ret));\n+\tsha1dc_clone(dst, src);\n+}\n+\n+static void initial_update(SHA1_CTX *ctx, const void *buf, size_t len)\n+{\n+\tint ret = pthread_once(&once, sha1dc_choose);\n+\tif (ret)\n+\t\tdie(\"cannot initialize SHA-1 backend: %s\", strerror(ret));\n+\tsha1dc_update(ctx, buf, len);\n+}\n+\n+static void initial_final(unsigned char hash[20], SHA1_CTX *ctx,\n+\t\t\t  void (*die_fn)(const char *, ...))\n+{\n+\tint ret = pthread_once(&once, sha1dc_choose);\n+\tif (ret)\n+\t\tdie(\"cannot initialize SHA-1 backend: %s\", strerror(ret));\n+\tsha1dc_final(hash, ctx, die_fn);\n+}\n+\n+static void initial_discard(SHA1_CTX *ctx)\n+{\n+\tint ret = pthread_once(&once, sha1dc_choose);\n+\tif (ret)\n+\t\tdie(\"cannot initialize SHA-1 backend: %s\", strerror(ret));\n+\tsha1dc_discard(ctx);\n+}\n+\n #endif\n-- \ngitgitgadget\n"},{"id":"553505","messageId":"2f7509d7-1166-1303-87b4-58974702c73d@gmx.de","threadId":"66408","inReplyTo":"pull.2240.git.1790610691.gitgitgadget@gmail.com","subject":"Re: [PATCH 0/4] Add a compile-time option to use the new, very fast sha1dc Rust crate","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2026-09-28T16:20:50Z","receivedAt":"2026-09-28T16:20:54Z","isPatch":true,"body":"Hi,\n\nOn Mon, 28 Sep 2026, Johannes Schindelin via GitGitGadget wrote:\n\n> I stumbled across this new Rust crate last week. Its performance numbers are\n> quite impressive. Naturally, I want to make use of this and get for Windows,\n\n                                                          ^^^^^^^^^^^^^^^^^^^\n\t\t\t\t\t\t\t  in Git for Windows\n\nMy sincerest apologies; I am using Cohere Transcribe to compensate for my\ninadequate typing speed, which typically works very, very well for me, yet\nmissed this typo. Likewise:\n\n> which is used on many monorepos where this makes a real difference: In a\n> pretty fast and loose test, I verified that a git index-pack runs roughly\n> three times faster solely due to using those SIMD-based optimizations!\n> \n> As a safety precaution, because this sha1dc crate is quite new, I wanted to\n> introduce an escape hatch: core.sha1dcBackend=c, but turn it on by default,\n> which is the reason for the three additional patches. Should these patches\n> be undesirable for the Git project? I would not be mad at all if they were\n\n                                    ^\n\t\t\t\t    , and\n\nSorry about that,\nJohannes\n\n> simply dropped.\n> \n> Johannes Schindelin (4):\n>   libgitcore: add `sha1dc` as an optional feature\n>   sha1dc: allow selecting the C backend without rebuilding\n>   pthread: provide `pthread_once()` shims for Windows and for\n>     NO_PTHREADS\n>   sha1dc: make `sha1dc_init()` thread-safe\n> \n>  Cargo.toml                     |   4 ++\n>  Documentation/config/core.adoc |   5 ++\n>  Makefile                       |  29 +++++++++\n>  compat/win32/pthread.c         |  16 +++++\n>  compat/win32/pthread.h         |   5 ++\n>  hash.h                         |   5 ++\n>  sha1dc_git.c                   | 109 +++++++++++++++++++++++++++++++--\n>  sha1dc_git.h                   |   5 +-\n>  sha1dc_rs.h                    |  37 +++++++++++\n>  src/lib.rs                     |   2 +\n>  src/sha1dc_rs.rs               |  78 +++++++++++++++++++++++\n>  t/helper/test-sha1.c           |  19 +++++-\n>  t/t0013-sha1dc.sh              |  21 ++++++-\n>  thread-utils.h                 |  16 +++++\n>  14 files changed, 343 insertions(+), 8 deletions(-)\n>  create mode 100644 sha1dc_rs.h\n>  create mode 100644 src/sha1dc_rs.rs\n> \n> \n> base-commit: 3bc0341126508f78f5869cbfc0005e987efdf0c7\n> Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-2240%2Fdscho%2Foptionally-use-sha1dc-rs-v1\n> Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-2240/dscho/optionally-use-sha1dc-rs-v1\n> Pull-Request: https://github.com/gitgitgadget/git/pull/2240\n> -- \n> gitgitgadget\n> \n"},{"id":"553553","messageId":"xmqqa4p0jz0d.fsf@gitster.g","threadId":"66408","inReplyTo":"pull.2240.git.1790610691.gitgitgadget@gmail.com","subject":"Re: [PATCH 0/4] Add a compile-time option to use the new, very fast sha1dc Rust crate","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2026-09-29T07:30:10Z","receivedAt":"2026-09-29T07:30:13Z","isPatch":true,"body":"\"Johannes Schindelin via GitGitGadget\" <gitgitgadget@gmail.com>\nwrites:\n\n> I stumbled across this new Rust crate last week. Its performance numbers are\n> quite impressive. Naturally, I want to make use of this and get for Windows,\n> which is used on many monorepos where this makes a real difference: In a\n> pretty fast and loose test, I verified that a git index-pack runs roughly\n> three times faster solely due to using those SIMD-based optimizations!\n>\n> As a safety precaution, because this sha1dc crate is quite new, I wanted to\n> introduce an escape hatch: core.sha1dcBackend=c, but turn it on by default,\n> which is the reason for the three additional patches. Should these patches\n> be undesirable for the Git project? I would not be mad at all if they were\n> simply dropped.\n>\n> Johannes Schindelin (4):\n>   libgitcore: add `sha1dc` as an optional feature\n>   sha1dc: allow selecting the C backend without rebuilding\n>   pthread: provide `pthread_once()` shims for Windows and for\n>     NO_PTHREADS\n>   sha1dc: make `sha1dc_init()` thread-safe\n\nThe feature sha1dc_choose() means that you can between Rust and C\nimplementations of sha1dc pick at runtime and I was confused by the\n\"compile-time\" in the topic title, which is misleading.  From the\nend-user's point of view, being able to choose between the two at\nruntime gives them a lot bigger value, even though from the point of\nview of the developer who added the feature to allow users to do so,\nthat feature being a compile-time choice might matter more.\n\nHow close are these two implementations?  Do they implement the same\nidea but the details may differ?  Do they both faithfully implement\nwhat the same paper wrote and given the same fudged input they will\nalways detect the attempted attack the same way?\n\n"},{"id":"554052","messageId":"09b76ee1-2bac-4c33-b09d-d414717d3ced@gmx.de","threadId":"66408","inReplyTo":"xmqqa4p0jz0d.fsf@gitster.g","subject":"Re: [PATCH 0/4] Add a compile-time option to use the new, very fast sha1dc Rust crate","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2026-10-03T10:23:52Z","receivedAt":"2026-10-03T10:23:55Z","isPatch":true,"body":"Hi Junio,\n\nOn Tue, 29 Sep 2026, Junio C Hamano wrote:\n\n> \"Johannes Schindelin via GitGitGadget\" <gitgitgadget@gmail.com>\n> writes:\n> \n> > I stumbled across this new Rust crate last week. Its performance\n> > numbers are quite impressive. Naturally, I want to make use of this\n> > and get for Windows, which is used on many monorepos where this makes\n> > a real difference: In a pretty fast and loose test, I verified that a\n> > git index-pack runs roughly three times faster solely due to using\n> > those SIMD-based optimizations!\n> >\n> > As a safety precaution, because this sha1dc crate is quite new, I\n> > wanted to introduce an escape hatch: core.sha1dcBackend=c, but turn it\n> > on by default, which is the reason for the three additional patches.\n> > Should these patches be undesirable for the Git project? I would not\n> > be mad at all if they were simply dropped.\n> >\n> > Johannes Schindelin (4):\n> >   libgitcore: add `sha1dc` as an optional feature\n> >   sha1dc: allow selecting the C backend without rebuilding\n> >   pthread: provide `pthread_once()` shims for Windows and for\n> >     NO_PTHREADS\n> >   sha1dc: make `sha1dc_init()` thread-safe\n> \n> The feature sha1dc_choose() means that you can between Rust and C\n> implementations of sha1dc pick at runtime and I was confused by the\n> \"compile-time\" in the topic title, which is misleading.\n\nRight. I was almost certain that you'd reject the runtime flag, which is\nreally only interesting for binary-first distribution vectors such as Git\nfor Windows but not source-code-only releases such as core Git's.\n\n> From the end-user's point of view, being able to choose between the two\n> at runtime gives them a lot bigger value, even though from the point of\n> view of the developer who added the feature to allow users to do so,\n> that feature being a compile-time choice might matter more.\n> \n> How close are these two implementations?  Do they implement the same\n> idea but the details may differ?  Do they both faithfully implement\n> what the same paper wrote and given the same fudged input they will\n> always detect the attempted attack the same way?\n\nThose two implementations are quite different. As the author of the Rust\ncrate detailed in https://sam.dev/blog/faster-sha1-collision-detection,\nthey first tried to accelerate the quite faithful Rust port of the library\nthat is used by Git, and while there were some gains to be made, a more\nfundamental approach proved to offer way bigger wins.\n\nWhile I would have _loved_ to have the time to dig into this myself, armed\nwith pencil and paper only, and doing maths again for once, I simply could\nnot afford the time to assess the validity of the Rust `sha1dc`\nimplementation without AI assistance. With that disclaimer out of the way\n(which should actually _increase_ your confidence, because I haven't been\nin the math business in a very, very long time, so the double-teaming with\nGPT-5.5 Sol and Opus 5.5 probably increased the soundness of my analysis),\nhere are my findings:\n\n- The Rust implementation chooses a different approach from the C\n  implementation. The idea is the same, though: to dismiss as quickly as\n  possible as many of the DV vectors (each check can cover several of\n  those at once). It's just that with SIMD, the technique differs, and\n  that informs about the order and the grouping of those checks.\n\n  (In more technical terms: The UBC filter is different, not the overall\n  collision-detection algorithm. Both Rust and C implementation cover the\n  same per-DV affine solution space over GF(2), albeit with different\n  equations).\n\n- While the approach is different, exploiting SIMD-specific advantages to\n  great speed-wise effects, the covered DV vectors are exactly the same,\n  and the filtering and recompression checks are equivalent; therefore\n  both C and Rust implementation detect the very same class of collisions\n  under the paper's assumptions.\n\n- The Rust implementation is robust and correct. I performed a light\n  (well, for me, not so much for the AI models) static analysis, and then\n  I ran some substantial tests. My plan was to exercise Git's entire test\n  suite (with a patched-in mode that would exercise both C and Rust and\n  validate that they compute the same SHA-1), but I haven't managed to\n  kick off _those_ particular AI-assisted sessions yet (I would want to\n  exercise both x86_64 and aarch64, of course).\n\n- The reason why I posted this before I finished _all_ the tests? To allow\n  other Git contributors an early look, and to inspire (see e.g. Scott's\n  alternative), to invite collaboration on this patch series.\n\nCiao,\nJohannes\n"},{"id":"554082","messageId":"asFtDF6NBc1ofnIl@fruit.crustytoothpaste.net","threadId":"66408","inReplyTo":"b1f30a6a05673c4094d59fda80c695472850d671.1790610691.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 1/4] libgitcore: add `sha1dc` as an optional feature","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2026-10-03T21:01:01Z","receivedAt":"2026-10-03T21:01:08Z","isPatch":true,"body":"On 2026-09-28 at 15:51:28, Johannes Schindelin via GitGitGadget wrote:\n> Note that the `sha1dc` crate still requires a significantly newer Rust\n> version than Git's existing Rust support requires: 1.87 instead of\n> 1.63 (https://crates.io/api/v1/crates/sha1dc/0.1.3).\n\nI think this is going to be a problem.  Yes, this is optional, but we\ndeclare compatibility with Rust 1.49.0 in Cargo.toml and we want\neverything to work there.\n\nThe goal was to have everything work with gccrs, but I think we're\nnearing Git 3.0 and gccrs has not made enough progress for it to be\nviable.  This is not a surprise to me, but that was our goal.\n\nThe approach I've been advocating is that we support the version in\nDebian stable, plus the version in Debian oldstable for a year after the\nnew stable comes out.  That would get us to Rust 1.85.1, since Debian\n13 (trixie) came out over a year ago, but not to Rust 1.87.\n\nIn any event, if we want to raise the version of Rust, we should\nprobably discuss that in a separate series that adds or updates a policy\ndocument and bumps the version in Cargo.toml.\n-- \nbrian m. carlson (they/them)\nToronto, Ontario, CA\n"}]}