{"thread":{"id":"64069","subject":"[PATCH 0/2] refs/files: fix issues with git-fetch on case-insensitive FS","startedAt":"2025-09-02T08:34:43Z","lastAt":"2025-09-17T20:40:51Z","messageCount":58,"participants":["Karthik Nayak","Chris Torek","Patrick Steinhardt","Junio C Hamano","Justin Tobler","Toon Claes"],"isPatch":true,"patchVersion":1,"patchTotal":2},"messages":[{"id":"525298","messageId":"20250902-587-git-fetch-1-fails-fetches-on-case-insensitive-repositories-v1-0-35e69bbb507d@gmail.com","threadId":"64069","inReplyTo":null,"subject":"[PATCH 0/2] refs/files: fix issues with git-fetch on case-insensitive FS","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-09-02T08:34:24Z","receivedAt":"2025-09-02T08:34:43Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Hello!\n\nWith Git 2.51 we moved 'git-fetch(1)' and 'git-receive-pack(1)' to use\nbatched updates while doing reference updates. This provided a nice perf\nboost since both commands will now use a single transaction for\nreference updates. This removes the overhead of using individual\ntransaction per reference update and also avoids unnecessary\nauto-compaction between reference updates in the reftable backend.\n\nHowever, in the files-backend it does introduce two bugs when used on a\ncase-insensitive filesystem [1]:\n\n1. When fetching references such as:\n\n - refs/heads/foo\n - refs/heads/Foo\n\nEarlier we would simply overwrite the first reference with the second\nand continue. Since Git 2.51 we simply abort stating a conflict.\n\nThis is resolved in the first commit by marking such conflicts as\nREF_TRANSACTION_ERROR_CREATE_EXISTS. This allows batched updates to\nreject the particular update, while applying the rest.\n\n2. When fetching references such as:\n\n - refs/heads/foo\n - refs/heads/Foo/bar\n\nEarlier we would apply the first, while the second would fail due to\nconflict. Since Git 2.51, the lock files for both would be created, but\nthe 'commit' phase would abruptly end leaving the lock files.\n\nThe second commit fixes this by ensuring that on case-insensitive\nfilesystems we lowercase the refnames for availability check to ensure\nF/D are caught and reported to the user.\n\nSorry for the long wait on this, with the OSSE and Git mini summit last\nweek, I didn't get as much computer time as I needed to test this.\n\n- Karthik\n\n[1]: https://lore.kernel.org/all/YQXPR01MB3046197EF39296549EE6DD669A33A@YQXPR01MB3046.CANPRD01.PROD.OUTLOOK.COM/\n\nSigned-off-by: Karthik Nayak \n---\n builtin/fetch.c | 21 ++++++++++++++++++---\n refs/files-backend.c | 21 +++++++++++++++++++--\n t/t1400-update-ref.sh | 24 ++++++++++++++++++++++++\n t/t5510-fetch.sh | 42 +++++++++++++++++++++++++++++++++++++++++-\n 4 files changed, 102 insertions(+), 6 deletions(-)\n\nKarthik Nayak (2):\n refs/files: use correct error type when locking fails\n refs/files: handle F/D conflicts in case-insensitive FS\n\n\n\nbase-commit: c44beea485f0f2feaf460e2ac87fdd5608d63cf0\nchange-id: 20250822-587-git-fetch-1-fails-fetches-on-case-insensitive-repositories-0a187c7ac41e\n\nThanks\n- Karthik\n\n"},{"id":"525299","messageId":"20250902-587-git-fetch-1-fails-fetches-on-case-insensitive-repositories-v1-1-35e69bbb507d@gmail.com","threadId":"64069","inReplyTo":"20250902-587-git-fetch-1-fails-fetches-on-case-insensitive-repositories-v1-0-35e69bbb507d@gmail.com","subject":"[PATCH 1/2] refs/files: use correct error type when locking fails","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-09-02T08:34:25Z","receivedAt":"2025-09-02T08:34:44Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"During the 'prepare' phase of reference transaction in the files\nbackend, we create the lock files for references to be created. When\nusing batched updates on case-insensitive filesystems, the transactions\nwould be aborted if there are conflicting names such as:\n\n refs/heads/Foo\n refs/heads/foo\n\nThis affects all commands which were migrated to use batched updates in\nGit 2.51, including 'git-fetch(1)' and 'git-receive-pack(1)'. Before\nthat, references updates would be applied serially with one transaction\nused per update. When users fetched multiple references on\ncase-insensitive systems, subsequent references would simply overwrite\nany earlier references. So when fetching:\n\n refs/heads/foo: 5f34ec0bfeac225b1c854340257a65b106f70ea6\n refs/heads/Foo: ec3053b0977e83d9b67fc32c4527a117953994f3\n refs/heads/sample: 2eefd1150e06d8fca1ddfa684dec016f36bf4e56\n\nThe user would simply end up with:\n\n refs/heads/foo: ec3053b0977e83d9b67fc32c4527a117953994f3\n refs/heads/sample: 2eefd1150e06d8fca1ddfa684dec016f36bf4e56\n\nThis is buggy behavior since the user is never intimated about the\noverrides performed and missing references. Nevertheless, the user is\nleft with a working repository with a subset of the references. Since\nGit 2.51, in such situations fetches would simply fail without applying\nany references. Which is also buggy behavior and worse off since the\nuser is left without any references.\n\nThe error is triggered in `lock_raw_ref()` where the files backend\nattempts to create a lock file. When a lock file already exists the\nfunction returns a 'REF_TRANSACTION_ERROR_GENERIC'. Change this to return\n'REF_TRANSACTION_ERROR_CREATE_EXISTS' instead to aid the batched update\nmechanism to simply reject such errors.\n\nThis bubbles the error type up to `files_transaction_prepare()` which\ntries to lock each reference update. So if the locking fails, we check\nif the rejection type can be ignored, which is done by calling\n`ref_transaction_maybe_set_rejected()`.\n\nAs the error type is now 'REF_TRANSACTION_ERROR_CREATE_EXISTS', the\nspecific reference update would simply be rejected, while other updates\nin the transaction would continue to be applied. This allows partial\napplication of references in case-insensitive filesystems when fetching\ncolliding references.\n\nWhile the earlier implementation allowed the last reference to be\napplied overriding the initial references, this change would allow the\nfirst reference to be applied while rejecting consequent collisions.\nThis should be an OKAY compromise since with the files backend, there is\nno scenario possible where we would retain all colliding references.\n\nThe change only affects batched updates since batched updates will\nreject individual updates with non-generic errors. So specifically this\nwould only affect:\n\n 1. git fetch\n 2. git receive-pack\n 3. git update-ref --batch-updates\n\nLet's also be more pro-active and notify users on case-insensitive\nfilesystems about such problems by providing a brief about the issue\nwhile also recommending using the reftable backend, which doesn't have\nthe same issue.\n\nReported-by: Joe Drew \nSigned-off-by: Karthik Nayak \n---\n builtin/fetch.c | 21 ++++++++++++++++++---\n refs/files-backend.c | 2 ++\n t/t1400-update-ref.sh | 24 ++++++++++++++++++++++++\n t/t5510-fetch.sh | 22 +++++++++++++++++++++-\n 4 files changed, 65 insertions(+), 4 deletions(-)\n\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex 24645c4653..9563abbe12 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -1643,7 +1643,8 @@ static int set_head(const struct ref *remote_refs, struct remote *remote)\n \n struct ref_rejection_data {\n \tint *retcode;\n-\tint conflict_msg_shown;\n+\tbool conflict_msg_shown;\n+\tbool case_sensitive_msg_shown;\n \tconst char *remote_name;\n };\n \n@@ -1657,11 +1658,25 @@ static void ref_transaction_rejection_handler(const char *refname,\n {\n \tstruct ref_rejection_data *data = cb_data;\n \n-\tif (err == REF_TRANSACTION_ERROR_NAME_CONFLICT && !data->conflict_msg_shown) {\n+\tif (err == REF_TRANSACTION_ERROR_CREATE_EXISTS && ignore_case &&\n+\t !data->case_sensitive_msg_shown) {\n+\t\terror(_(\"You're on a case-insensitive filesystem, and the remote you are\\n\"\n+\t\t\t\"trying to fetch from has references that only differ in casing. It\\n\"\n+\t\t\t\"is impossible to store such references with the 'files' backend. You\\n\"\n+\t\t\t\"can either accept this as-is, in which case you won't be able to\\n\"\n+\t\t\t\"store all remote references on disk. Or you can alternatively\\n\"\n+\t\t\t\"migrate your repository to use the 'reftable' backend with the\\n\"\n+\t\t\t\"following command:\\n\\n git refs migrate --ref-format=reftable\\n\\n\"\n+\t\t\t\"Please keep in mind that not all implementations of Git support this\\n\"\n+\t\t\t\"new format yet. So if you use tools other than Git to access this\\n\"\n+\t\t\t\"repository it may not be an option to migrate to reftables.\\n\"));\n+\t\tdata->case_sensitive_msg_shown = true;\n+\t} else if (err == REF_TRANSACTION_ERROR_NAME_CONFLICT &&\n+\t\t !data->conflict_msg_shown) {\n \t\terror(_(\"some local refs could not be updated; try running\\n\"\n \t\t\t\" 'git remote prune %s' to remove any old, conflicting \"\n \t\t\t\"branches\"), data->remote_name);\n-\t\tdata->conflict_msg_shown = 1;\n+\t\tdata->conflict_msg_shown = true;\n \t} else {\n \t\tconst char *reason = ref_transaction_error_msg(err);\n \ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 088b52c740..9f58ea4858 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -776,6 +776,8 @@ static enum ref_transaction_error lock_raw_ref(struct files_ref_store *refs,\n \t\t\tgoto retry;\n \t\t} else {\n \t\t\tunable_to_lock_message(ref_file.buf, myerr, err);\n+\t\t\tif (myerr == EEXIST)\n+\t\t\t\tret = REF_TRANSACTION_ERROR_CREATE_EXISTS;\n \t\t\tgoto error_return;\n \t\t}\n \t}\ndiff --git a/t/t1400-update-ref.sh b/t/t1400-update-ref.sh\nindex 96648a6e5d..e37a5d83e8 100755\n--- a/t/t1400-update-ref.sh\n+++ b/t/t1400-update-ref.sh\n@@ -2294,6 +2294,30 @@ do\n \t\t)\n \t'\n \n+\ttest_expect_success CASE_INSENSITIVE_FS,REFFILES \"stdin $type batch-updates existing reference\" '\n+\t\tgit init repo &&\n+\t\ttest_when_finished \"rm -fr repo\" &&\n+\t\t(\n+\t\t\tcd repo &&\n+\t\t\ttest_commit one &&\n+\t\t\told_head=$(git rev-parse HEAD) &&\n+\t\t\ttest_commit two &&\n+\t\t\thead=$(git rev-parse HEAD) &&\n+\n+\t\t\tformat_command $type \"create refs/heads/foo\" \"$head\" >stdin &&\n+\t\t\tformat_command $type \"create refs/heads/ref\" \"$old_head\" >>stdin &&\n+\t\t\tformat_command $type \"create refs/heads/Foo\" \"$old_head\" >>stdin &&\n+\t\t\tgit update-ref $type --stdin --batch-updates stdout &&\n+\n+\t\t\techo $head >expect &&\n+\t\t\tgit rev-parse refs/heads/foo >actual &&\n+\t\t\techo $old_head >expect &&\n+\t\t\tgit rev-parse refs/heads/ref >actual &&\n+\t\t\ttest_cmp expect actual &&\n+\t\t\ttest_grep -q \"reference already exists\" stdout\n+\t\t)\n+\t'\n+\n \ttest_expect_success \"stdin $type batch-updates delete incorrect symbolic ref\" '\n \t\tgit init repo &&\n \t\ttest_when_finished \"rm -fr repo\" &&\ndiff --git a/t/t5510-fetch.sh b/t/t5510-fetch.sh\nindex ebc696546b..57f60da81b 100755\n--- a/t/t5510-fetch.sh\n+++ b/t/t5510-fetch.sh\n@@ -47,7 +47,13 @@ test_expect_success \"clone and setup child repos\" '\n \t\tgit config set branch.main.merge refs/heads/one\n \t) &&\n \tgit clone . bundle &&\n-\tgit clone . seven\n+\tgit clone . seven &&\n+\tgit clone --ref-format=reftable . case_sensitive &&\n+\t(\n+\t\tcd case_sensitive &&\n+\t\tgit branch branch1 &&\n+\t\tgit branch bRanch1\n+\t)\n '\n \n test_expect_success \"fetch test\" '\n@@ -1526,6 +1532,20 @@ test_expect_success SYMLINKS 'clone does not get confused by a D/F conflict' '\n \ttest_path_is_missing whoops\n '\n \n+test_expect_success CASE_INSENSITIVE_FS,REFFILES 'existing references in a case insensitive filesystem' '\n+\ttest_when_finished rm -rf case_insensitive &&\n+\t(\n+\t\tgit init --bare case_insensitive &&\n+\t\tcd case_insensitive &&\n+\t\tgit remote add origin -- ../case_sensitive &&\n+\t\ttest_must_fail git fetch -f origin \"refs/heads/*:refs/heads/*\" 2>err &&\n+\t\ttest_grep \"You${SQ}re on a case-insensitive filesystem\" err &&\n+\t\tgit rev-parse refs/heads/main >expect &&\n+\t\tgit rev-parse refs/heads/branch1 >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n . \"$TEST_DIRECTORY\"/lib-httpd.sh\n start_httpd\n \n\n-- \n2.50.1\n\n"},{"id":"525300","messageId":"20250902-587-git-fetch-1-fails-fetches-on-case-insensitive-repositories-v1-2-35e69bbb507d@gmail.com","threadId":"64069","inReplyTo":"20250902-587-git-fetch-1-fails-fetches-on-case-insensitive-repositories-v1-0-35e69bbb507d@gmail.com","subject":"[PATCH 2/2] refs/files: handle F/D conflicts in case-insensitive FS","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-09-02T08:34:26Z","receivedAt":"2025-09-02T08:34:45Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Similar to the previous commit, when using the files-backend on\ncase-insensitive filesystems, there is possibility of hitting F/D\nconflicts when creating references within a single transaction, such as:\n\n - 'refs/heads/foo'\n - 'refs/heads/Foo/bar'\n\nIdeally such conflicts are caught in `refs_verify_refnames_available()`\nwhich is responsible for checking F/D conflicts within a given\ntransaction. This utility function is shared across the reference\nbackends. As such, it doesn't consider the issues of using a\ncase-insensitive, which only affects the files-backend.\n\nWhile one solution would be to make the function aware of such issues.\nThis feels like leaking implementation details of file-backend specific\nissues into the utility function. So opt for the more simpler option, of\nlowercasing all references sent to this function when on a\ncase-insensitive filesystem and operating on the files-backend.\n\nTo do this, simply use a `struct strbuf` to convert the refname to a\nlower case and append it to the list of refnames to be checked. Since we\nuse a `struct strbuf` and the memory is cleared right after, make sure\nthat the string list duplicates all provided string.\n\nWithout this change, the user would simply be left with a repository\nwith '.lock' files which were created in the 'prepare' phase of the\ntransaction, as the 'commit' phase would simply abort and not do the\nnecessary cleanup.\n\nReported-by: Junio C Hamano \nSigned-off-by: Karthik Nayak \n---\n refs/files-backend.c | 19 +++++++++++++++++--\n t/t5510-fetch.sh | 20 ++++++++++++++++++++\n 2 files changed, 37 insertions(+), 2 deletions(-)\n\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 9f58ea4858..466cdfe121 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -869,8 +869,23 @@ static enum ref_transaction_error lock_raw_ref(struct files_ref_store *refs,\n \t\t * If the ref did not exist and we are creating it, we have to\n \t\t * make sure there is no existing packed ref that conflicts\n \t\t * with refname. This check is deferred so that we can batch it.\n+\t\t *\n+\t\t * For case-insensitive filesystems, we should also check for F/D\n+\t\t * conflicts between 'foo' and 'Foo/bar'. So let's lowercase\n+\t\t * the refname.\n \t\t */\n-\t\titem = string_list_append(refnames_to_check, refname);\n+\t\tif (ignore_case) {\n+\t\t\tstruct strbuf lower = STRBUF_INIT;\n+\n+\t\t\tstrbuf_addstr(&lower, refname);\n+\t\t\tstrbuf_tolower(&lower);\n+\n+\t\t\titem = string_list_append(refnames_to_check, lower.buf);\n+\t\t\tstrbuf_release(&lower);\n+\t\t} else {\n+\t\t\titem = string_list_append(refnames_to_check, refname);\n+\t\t}\n+\n \t\titem->util = xmalloc(sizeof(update_idx));\n \t\tmemcpy(item->util, &update_idx, sizeof(update_idx));\n \t}\n@@ -2796,7 +2811,7 @@ static int files_transaction_prepare(struct ref_store *ref_store,\n \t\t\t \"ref_transaction_prepare\");\n \tsize_t i;\n \tint ret = 0;\n-\tstruct string_list refnames_to_check = STRING_LIST_INIT_NODUP;\n+\tstruct string_list refnames_to_check = STRING_LIST_INIT_DUP;\n \tchar *head_ref = NULL;\n \tint head_type;\n \tstruct files_transaction_backend_data *backend_data;\ndiff --git a/t/t5510-fetch.sh b/t/t5510-fetch.sh\nindex 57f60da81b..84dc68e5f3 100755\n--- a/t/t5510-fetch.sh\n+++ b/t/t5510-fetch.sh\n@@ -53,6 +53,12 @@ test_expect_success \"clone and setup child repos\" '\n \t\tcd case_sensitive &&\n \t\tgit branch branch1 &&\n \t\tgit branch bRanch1\n+\t) &&\n+\tgit clone --ref-format=reftable . case_sensitive_fd &&\n+\t(\n+\t\tcd case_sensitive_fd &&\n+\t\tgit branch foo/bar &&\n+\t\tgit branch Foo\n \t)\n '\n \n@@ -1546,6 +1552,20 @@ test_expect_success CASE_INSENSITIVE_FS,REFFILES 'existing references in a case\n \t)\n '\n \n+test_expect_success CASE_INSENSITIVE_FS,REFFILES 'F/D conflict on case insensitive filesystem' '\n+\ttest_when_finished rm -rf case_insensitive &&\n+\t(\n+\t\tgit init --bare case_insensitive &&\n+\t\tcd case_insensitive &&\n+\t\tgit remote add origin -- ../case_sensitive_fd &&\n+\t\ttest_must_fail git fetch -f origin \"refs/heads/*:refs/heads/*\" 2>err &&\n+\t\ttest_grep \"failed: refname conflict\" err &&\n+\t\tgit rev-parse refs/heads/main >expect &&\n+\t\tgit rev-parse refs/heads/foo/bar >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n . \"$TEST_DIRECTORY\"/lib-httpd.sh\n start_httpd\n \n\n-- \n2.50.1\n\n"},{"id":"525378","messageId":"CAPx1GvdA3dhWN8b6wORdL1MktdVoJ3E2s9sgoAyUDwFK+7WdzA@mail.gmail.com","threadId":"64069","inReplyTo":"20250902-587-git-fetch-1-fails-fetches-on-case-insensitive-repositories-v1-1-35e69bbb507d@gmail.com","subject":"Re: [PATCH 1/2] refs/files: use correct error type when locking fails","fromName":"Chris Torek","fromEmail":"chris.torek@gmail.com","sentAt":"2025-09-02T21:22:51Z","receivedAt":"2025-09-02T21:23:05Z","isPatch":true,"sender":{"key":"chris.torek@gmail.com","avatar":"https://avatars.githubusercontent.com/u/16826774?v=4"},"body":"Minor:\n\nOn Tue, Sep 2, 2025 at 1:40 AM Karthik Nayak wrote:\n> During the 'prepare' phase of reference transaction in the files\n> backend, we create the lock files for references to be created. When\n[mass snippage for space]\n>\n> This is buggy behavior since the user is never intimated about the\n> overrides performed and missing references. Nevertheless, the user is\n\n\"Intimated\" is the wrong word (also this is a rare form, \"intimate\"\nas a verb that is, at least in US and British English). I'd suggest\n\"informed\" here, or some alternate phrasing.\n\nChris\n"},{"id":"525405","messageId":"aLfw8xiys53A-azC@pks.im","threadId":"64069","inReplyTo":"20250902-587-git-fetch-1-fails-fetches-on-case-insensitive-repositories-v1-1-35e69bbb507d@gmail.com","subject":"Re: [PATCH 1/2] refs/files: use correct error type when locking fails","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-09-03T07:40:35Z","receivedAt":"2025-09-03T07:40:42Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Sep 02, 2025 at 10:34:25AM +0200, Karthik Nayak wrote:\n> During the 'prepare' phase of reference transaction in the files\n> backend, we create the lock files for references to be created. When\n> using batched updates on case-insensitive filesystems, the transactions\n> would be aborted if there are conflicting names such as:\n> \n> refs/heads/Foo\n> refs/heads/foo\n> \n> This affects all commands which were migrated to use batched updates in\n> Git 2.51, including 'git-fetch(1)' and 'git-receive-pack(1)'. Before\n> that, references updates would be applied serially with one transaction\n\ns/references/reference/\n\n> used per update. When users fetched multiple references on\n> case-insensitive systems, subsequent references would simply overwrite\n> any earlier references. So when fetching:\n> \n> refs/heads/foo: 5f34ec0bfeac225b1c854340257a65b106f70ea6\n> refs/heads/Foo: ec3053b0977e83d9b67fc32c4527a117953994f3\n> refs/heads/sample: 2eefd1150e06d8fca1ddfa684dec016f36bf4e56\n> \n> The user would simply end up with:\n> \n> refs/heads/foo: ec3053b0977e83d9b67fc32c4527a117953994f3\n> refs/heads/sample: 2eefd1150e06d8fca1ddfa684dec016f36bf4e56\n> \n> This is buggy behavior since the user is never intimated about the\n> overrides performed and missing references. Nevertheless, the user is\n> left with a working repository with a subset of the references. Since\n> Git 2.51, in such situations fetches would simply fail without applying\n> any references. Which is also buggy behavior and worse off since the\n> user is left without any references.\n\nYup, agreed.\n\n> The error is triggered in `lock_raw_ref()` where the files backend\n> attempts to create a lock file. When a lock file already exists the\n> function returns a 'REF_TRANSACTION_ERROR_GENERIC'. Change this to return\n> 'REF_TRANSACTION_ERROR_CREATE_EXISTS' instead to aid the batched update\n> mechanism to simply reject such errors.\n> \n> This bubbles the error type up to `files_transaction_prepare()` which\n> tries to lock each reference update. So if the locking fails, we check\n> if the rejection type can be ignored, which is done by calling\n> `ref_transaction_maybe_set_rejected()`.\n> \n> As the error type is now 'REF_TRANSACTION_ERROR_CREATE_EXISTS', the\n> specific reference update would simply be rejected, while other updates\n> in the transaction would continue to be applied. This allows partial\n> application of references in case-insensitive filesystems when fetching\n> colliding references.\n\nOkay. Does that mean that both git-fetch(1) and git-receive-pack(1) are\nalready told to evict unsuccessful updates? If so, this bit of info\nshould probably be added to the commit message to say that it was\nalready the intent, but that it didn't work out because of the\nunexpected error type.\n\n> While the earlier implementation allowed the last reference to be\n> applied overriding the initial references, this change would allow the\n> first reference to be applied while rejecting consequent collisions.\n> This should be an OKAY compromise since with the files backend, there is\n\nI don't quite get why we're shouting :) In any case I think the\ncompromise is acceptable, but we very much should warn the user about\nthis error. Ideally, we'd even guide them towards the reftable backend.\nBut let's read on, maybe you already do that.\n\n> no scenario possible where we would retain all colliding references.\n> \n> The change only affects batched updates since batched updates will\n> reject individual updates with non-generic errors. So specifically this\n> would only affect:\n> \n> 1. git fetch\n> 2. git receive-pack\n> 3. git update-ref --batch-updates\n\nOkay, here you mention that we already use batched updates for those\ncommands. I think it would help the reader if this was explained before\ngoing into the individual error codes.\n\n> Let's also be more pro-active and notify users on case-insensitive\n> filesystems about such problems by providing a brief about the issue\n> while also recommending using the reftable backend, which doesn't have\n> the same issue.\n\nAnd yup, you already do exactly what I was proposing. Nice!\n\n> diff --git a/builtin/fetch.c b/builtin/fetch.c\n> index 24645c4653..9563abbe12 100644\n> --- a/builtin/fetch.c\n> +++ b/builtin/fetch.c\n> @@ -1643,7 +1643,8 @@ static int set_head(const struct ref *remote_refs, struct remote *remote)\n> \n> struct ref_rejection_data {\n> \tint *retcode;\n> -\tint conflict_msg_shown;\n> +\tbool conflict_msg_shown;\n> +\tbool case_sensitive_msg_shown;\n> \tconst char *remote_name;\n> };\n> \n> @@ -1657,11 +1658,25 @@ static void ref_transaction_rejection_handler(const char *refname,\n> {\n> \tstruct ref_rejection_data *data = cb_data;\n> \n> -\tif (err == REF_TRANSACTION_ERROR_NAME_CONFLICT && !data->conflict_msg_shown) {\n> +\tif (err == REF_TRANSACTION_ERROR_CREATE_EXISTS && ignore_case &&\n> +\t !data->case_sensitive_msg_shown) {\n> +\t\terror(_(\"You're on a case-insensitive filesystem, and the remote you are\\n\"\n> +\t\t\t\"trying to fetch from has references that only differ in casing. It\\n\"\n> +\t\t\t\"is impossible to store such references with the 'files' backend. You\\n\"\n> +\t\t\t\"can either accept this as-is, in which case you won't be able to\\n\"\n> +\t\t\t\"store all remote references on disk. Or you can alternatively\\n\"\n> +\t\t\t\"migrate your repository to use the 'reftable' backend with the\\n\"\n> +\t\t\t\"following command:\\n\\n git refs migrate --ref-format=reftable\\n\\n\"\n> +\t\t\t\"Please keep in mind that not all implementations of Git support this\\n\"\n> +\t\t\t\"new format yet. So if you use tools other than Git to access this\\n\"\n> +\t\t\t\"repository it may not be an option to migrate to reftables.\\n\"));\n\nThis reads familiar :)\n\n> +\t\tdata->case_sensitive_msg_shown = true;\n> +\t} else if (err == REF_TRANSACTION_ERROR_NAME_CONFLICT &&\n> +\t\t !data->conflict_msg_shown) {\n> \t\terror(_(\"some local refs could not be updated; try running\\n\"\n> \t\t\t\" 'git remote prune %s' to remove any old, conflicting \"\n> \t\t\t\"branches\"), data->remote_name);\n> -\t\tdata->conflict_msg_shown = 1;\n> +\t\tdata->conflict_msg_shown = true;\n> \t} else {\n> \t\tconst char *reason = ref_transaction_error_msg(err);\n> \n> diff --git a/refs/files-backend.c b/refs/files-backend.c\n> index 088b52c740..9f58ea4858 100644\n> --- a/refs/files-backend.c\n> +++ b/refs/files-backend.c\n> @@ -776,6 +776,8 @@ static enum ref_transaction_error lock_raw_ref(struct files_ref_store *refs,\n> \t\t\tgoto retry;\n> \t\t} else {\n> \t\t\tunable_to_lock_message(ref_file.buf, myerr, err);\n> +\t\t\tif (myerr == EEXIST)\n> +\t\t\t\tret = REF_TRANSACTION_ERROR_CREATE_EXISTS;\n> \t\t\tgoto error_return;\n> \t\t}\n> \t}\n\nThis here is the actual bug fix that makes us treat the error\ngracefully.\n\n> diff --git a/t/t1400-update-ref.sh b/t/t1400-update-ref.sh\n> index 96648a6e5d..e37a5d83e8 100755\n> --- a/t/t1400-update-ref.sh\n> +++ b/t/t1400-update-ref.sh\n> @@ -2294,6 +2294,30 @@ do\n> \t\t)\n> \t'\n> \n> +\ttest_expect_success CASE_INSENSITIVE_FS,REFFILES \"stdin $type batch-updates existing reference\" '\n> +\t\tgit init repo &&\n> +\t\ttest_when_finished \"rm -fr repo\" &&\n> +\t\t(\n> +\t\t\tcd repo &&\n> +\t\t\ttest_commit one &&\n> +\t\t\told_head=$(git rev-parse HEAD) &&\n> +\t\t\ttest_commit two &&\n> +\t\t\thead=$(git rev-parse HEAD) &&\n> +\n> +\t\t\tformat_command $type \"create refs/heads/foo\" \"$head\" >stdin &&\n> +\t\t\tformat_command $type \"create refs/heads/ref\" \"$old_head\" >>stdin &&\n> +\t\t\tformat_command $type \"create refs/heads/Foo\" \"$old_head\" >>stdin &&\n\nThese could be written as:\n\n {\n format_command $type \"create refs/heads/foo\" \"$head\" &&\n format_command $type \"create refs/heads/ref\" \"$old_head\" &&\n format_command $type \"create refs/heads/Foo\" \"$old_head\"\n } >stdin\n\n> +\t\t\tgit update-ref $type --stdin --batch-updates stdout &&\n> +\n> +\t\t\techo $head >expect &&\n> +\t\t\tgit rev-parse refs/heads/foo >actual &&\n> +\t\t\techo $old_head >expect &&\n> +\t\t\tgit rev-parse refs/heads/ref >actual &&\n> +\t\t\ttest_cmp expect actual &&\n> +\t\t\ttest_grep -q \"reference already exists\" stdout\n> +\t\t)\n> +\t'\n> +\n> \ttest_expect_success \"stdin $type batch-updates delete incorrect symbolic ref\" '\n> \t\tgit init repo &&\n> \t\ttest_when_finished \"rm -fr repo\" &&\n\nWe could think about making these tests not depend on the REFFILES\nprerequisite and then verify that with the reftable backend things work\nas expected.\n\nPatrick\n"},{"id":"525406","messageId":"aLfw-peLY8NEKSZd@pks.im","threadId":"64069","inReplyTo":"20250902-587-git-fetch-1-fails-fetches-on-case-insensitive-repositories-v1-2-35e69bbb507d@gmail.com","subject":"Re: [PATCH 2/2] refs/files: handle F/D conflicts in case-insensitive FS","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-09-03T07:40:42Z","receivedAt":"2025-09-03T07:40:48Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Sep 02, 2025 at 10:34:26AM +0200, Karthik Nayak wrote:\n> Similar to the previous commit, when using the files-backend on\n> case-insensitive filesystems, there is possibility of hitting F/D\n> conflicts when creating references within a single transaction, such as:\n> \n> - 'refs/heads/foo'\n> - 'refs/heads/Foo/bar'\n\nGreat, I wanted to ask about this scenario.\n\n> Ideally such conflicts are caught in `refs_verify_refnames_available()`\n> which is responsible for checking F/D conflicts within a given\n> transaction. This utility function is shared across the reference\n> backends. As such, it doesn't consider the issues of using a\n> case-insensitive, which only affects the files-backend.\n> \n> While one solution would be to make the function aware of such issues.\n> This feels like leaking implementation details of file-backend specific\n> issues into the utility function. So opt for the more simpler option, of\n> lowercasing all references sent to this function when on a\n> case-insensitive filesystem and operating on the files-backend.\n> \n> To do this, simply use a `struct strbuf` to convert the refname to a\n> lower case and append it to the list of refnames to be checked. Since we\n> use a `struct strbuf` and the memory is cleared right after, make sure\n> that the string list duplicates all provided string.\n> \n> Without this change, the user would simply be left with a repository\n> with '.lock' files which were created in the 'prepare' phase of the\n> transaction, as the 'commit' phase would simply abort and not do the\n> necessary cleanup.\n\nOh, that's a clever hack. Does this also work for the case where we have\npreexisting refs already that differ only in casing? I guess it should\ngiven that the lookups we perform should yield those refs regardless of\ntheir casing.\n\nIn any case, if we don't already have such a test it would be great to\nalso verify that this works as expected.\n\n> diff --git a/refs/files-backend.c b/refs/files-backend.c\n> index 9f58ea4858..466cdfe121 100644\n> --- a/refs/files-backend.c\n> +++ b/refs/files-backend.c\n> @@ -869,8 +869,23 @@ static enum ref_transaction_error lock_raw_ref(struct files_ref_store *refs,\n> \t\t * If the ref did not exist and we are creating it, we have to\n> \t\t * make sure there is no existing packed ref that conflicts\n> \t\t * with refname. This check is deferred so that we can batch it.\n> +\t\t *\n> +\t\t * For case-insensitive filesystems, we should also check for F/D\n> +\t\t * conflicts between 'foo' and 'Foo/bar'. So let's lowercase\n> +\t\t * the refname.\n> \t\t */\n> -\t\titem = string_list_append(refnames_to_check, refname);\n> +\t\tif (ignore_case) {\n> +\t\t\tstruct strbuf lower = STRBUF_INIT;\n> +\n> +\t\t\tstrbuf_addstr(&lower, refname);\n> +\t\t\tstrbuf_tolower(&lower);\n> +\n> +\t\t\titem = string_list_append(refnames_to_check, lower.buf);\n> +\t\t\tstrbuf_release(&lower);\n\nCan we use `string_list_append_nodup()` together with `strbuf_detach()`\nhere to avoid one memory allocation?\n\n> +\t\t} else {\n> +\t\t\titem = string_list_append(refnames_to_check, refname);\n> +\t\t}\n> +\n> \t\titem->util = xmalloc(sizeof(update_idx));\n> \t\tmemcpy(item->util, &update_idx, sizeof(update_idx));\n> \t}\n> diff --git a/t/t5510-fetch.sh b/t/t5510-fetch.sh\n> index 57f60da81b..84dc68e5f3 100755\n> --- a/t/t5510-fetch.sh\n> +++ b/t/t5510-fetch.sh\n> @@ -53,6 +53,12 @@ test_expect_success \"clone and setup child repos\" '\n> \t\tcd case_sensitive &&\n> \t\tgit branch branch1 &&\n> \t\tgit branch bRanch1\n> +\t) &&\n> +\tgit clone --ref-format=reftable . case_sensitive_fd &&\n> +\t(\n> +\t\tcd case_sensitive_fd &&\n> +\t\tgit branch foo/bar &&\n> +\t\tgit branch Foo\n> \t)\n> '\n> \n\nNice idea to use the reftable format here.\n\n> @@ -1546,6 +1552,20 @@ test_expect_success CASE_INSENSITIVE_FS,REFFILES 'existing references in a case\n> \t)\n> '\n> \n> +test_expect_success CASE_INSENSITIVE_FS,REFFILES 'F/D conflict on case insensitive filesystem' '\n> +\ttest_when_finished rm -rf case_insensitive &&\n> +\t(\n> +\t\tgit init --bare case_insensitive &&\n> +\t\tcd case_insensitive &&\n> +\t\tgit remote add origin -- ../case_sensitive_fd &&\n> +\t\ttest_must_fail git fetch -f origin \"refs/heads/*:refs/heads/*\" 2>err &&\n> +\t\ttest_grep \"failed: refname conflict\" err &&\n> +\t\tgit rev-parse refs/heads/main >expect &&\n> +\t\tgit rev-parse refs/heads/foo/bar >actual &&\n> +\t\ttest_cmp expect actual\n> +\t)\n> +'\n\nOkay, so we again only end up with one of these references, which is the\nbest we can do.\n\natrick\n"},{"id":"525417","messageId":"CAOLa=ZTxZXWuFxaTWXmVpfoFjao_F0Mcu469aKQRu5fNNB0KEA@mail.gmail.com","threadId":"64069","inReplyTo":"aLfw8xiys53A-azC@pks.im","subject":"Re: [PATCH 1/2] refs/files: use correct error type when locking fails","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-09-03T10:38:04Z","receivedAt":"2025-09-03T10:38:06Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Patrick Steinhardt writes:\n\n> On Tue, Sep 02, 2025 at 10:34:25AM +0200, Karthik Nayak wrote:\n>> During the 'prepare' phase of reference transaction in the files\n>> backend, we create the lock files for references to be created. When\n>> using batched updates on case-insensitive filesystems, the transactions\n>> would be aborted if there are conflicting names such as:\n>>\n>> refs/heads/Foo\n>> refs/heads/foo\n>>\n>> This affects all commands which were migrated to use batched updates in\n>> Git 2.51, including 'git-fetch(1)' and 'git-receive-pack(1)'. Before\n>> that, references updates would be applied serially with one transaction\n>\n> s/references/reference/\n>\n>> used per update. When users fetched multiple references on\n>> case-insensitive systems, subsequent references would simply overwrite\n>> any earlier references. So when fetching:\n>>\n>> refs/heads/foo: 5f34ec0bfeac225b1c854340257a65b106f70ea6\n>> refs/heads/Foo: ec3053b0977e83d9b67fc32c4527a117953994f3\n>> refs/heads/sample: 2eefd1150e06d8fca1ddfa684dec016f36bf4e56\n>>\n>> The user would simply end up with:\n>>\n>> refs/heads/foo: ec3053b0977e83d9b67fc32c4527a117953994f3\n>> refs/heads/sample: 2eefd1150e06d8fca1ddfa684dec016f36bf4e56\n>>\n>> This is buggy behavior since the user is never intimated about the\n>> overrides performed and missing references. Nevertheless, the user is\n>> left with a working repository with a subset of the references. Since\n>> Git 2.51, in such situations fetches would simply fail without applying\n>> any references. Which is also buggy behavior and worse off since the\n>> user is left without any references.\n>\n> Yup, agreed.\n>\n>> The error is triggered in `lock_raw_ref()` where the files backend\n>> attempts to create a lock file. When a lock file already exists the\n>> function returns a 'REF_TRANSACTION_ERROR_GENERIC'. Change this to return\n>> 'REF_TRANSACTION_ERROR_CREATE_EXISTS' instead to aid the batched update\n>> mechanism to simply reject such errors.\n>>\n>> This bubbles the error type up to `files_transaction_prepare()` which\n>> tries to lock each reference update. So if the locking fails, we check\n>> if the rejection type can be ignored, which is done by calling\n>> `ref_transaction_maybe_set_rejected()`.\n>>\n>> As the error type is now 'REF_TRANSACTION_ERROR_CREATE_EXISTS', the\n>> specific reference update would simply be rejected, while other updates\n>> in the transaction would continue to be applied. This allows partial\n>> application of references in case-insensitive filesystems when fetching\n>> colliding references.\n>\n> Okay. Does that mean that both git-fetch(1) and git-receive-pack(1) are\n> already told to evict unsuccessful updates? If so, this bit of info\n> should probably be added to the commit message to say that it was\n> already the intent, but that it didn't work out because of the\n> unexpected error type.\n>\n>> While the earlier implementation allowed the last reference to be\n>> applied overriding the initial references, this change would allow the\n>> first reference to be applied while rejecting consequent collisions.\n>> This should be an OKAY compromise since with the files backend, there is\n>\n> I don't quite get why we're shouting :) In any case I think the\n> compromise is acceptable, but we very much should warn the user about\n> this error. Ideally, we'd even guide them towards the reftable backend.\n> But let's read on, maybe you already do that.\n>\n\nLet me remove that shout :D\n\n>> no scenario possible where we would retain all colliding references.\n>>\n>> The change only affects batched updates since batched updates will\n>> reject individual updates with non-generic errors. So specifically this\n>> would only affect:\n>>\n>> 1. git fetch\n>> 2. git receive-pack\n>> 3. git update-ref --batch-updates\n>\n> Okay, here you mention that we already use batched updates for those\n> commands. I think it would help the reader if this was explained before\n> going into the individual error codes.\n>\n\nYeah it would read better earlier on, let me move it around.\n\n>> Let's also be more pro-active and notify users on case-insensitive\n>> filesystems about such problems by providing a brief about the issue\n>> while also recommending using the reftable backend, which doesn't have\n>> the same issue.\n>\n> And yup, you already do exactly what I was proposing. Nice!\n>\n\nIt was copied from your suggestion!\n\n>> diff --git a/builtin/fetch.c b/builtin/fetch.c\n>> index 24645c4653..9563abbe12 100644\n>> --- a/builtin/fetch.c\n>> +++ b/builtin/fetch.c\n>> @@ -1643,7 +1643,8 @@ static int set_head(const struct ref *remote_refs, struct remote *remote)\n>>\n>> struct ref_rejection_data {\n>> \tint *retcode;\n>> -\tint conflict_msg_shown;\n>> +\tbool conflict_msg_shown;\n>> +\tbool case_sensitive_msg_shown;\n>> \tconst char *remote_name;\n>> };\n>>\n>> @@ -1657,11 +1658,25 @@ static void ref_transaction_rejection_handler(const char *refname,\n>> {\n>> \tstruct ref_rejection_data *data = cb_data;\n>>\n>> -\tif (err == REF_TRANSACTION_ERROR_NAME_CONFLICT && !data->conflict_msg_shown) {\n>> +\tif (err == REF_TRANSACTION_ERROR_CREATE_EXISTS && ignore_case &&\n>> +\t !data->case_sensitive_msg_shown) {\n>> +\t\terror(_(\"You're on a case-insensitive filesystem, and the remote you are\\n\"\n>> +\t\t\t\"trying to fetch from has references that only differ in casing. It\\n\"\n>> +\t\t\t\"is impossible to store such references with the 'files' backend. You\\n\"\n>> +\t\t\t\"can either accept this as-is, in which case you won't be able to\\n\"\n>> +\t\t\t\"store all remote references on disk. Or you can alternatively\\n\"\n>> +\t\t\t\"migrate your repository to use the 'reftable' backend with the\\n\"\n>> +\t\t\t\"following command:\\n\\n git refs migrate --ref-format=reftable\\n\\n\"\n>> +\t\t\t\"Please keep in mind that not all implementations of Git support this\\n\"\n>> +\t\t\t\"new format yet. So if you use tools other than Git to access this\\n\"\n>> +\t\t\t\"repository it may not be an option to migrate to reftables.\\n\"));\n>\n> This reads familiar :)\n>\n\nWhich I failed to attribute to you, sorry for missing that, will add in\na 'Helped-by'.\n\n>> +\t\tdata->case_sensitive_msg_shown = true;\n>> +\t} else if (err == REF_TRANSACTION_ERROR_NAME_CONFLICT &&\n>> +\t\t !data->conflict_msg_shown) {\n>> \t\terror(_(\"some local refs could not be updated; try running\\n\"\n>> \t\t\t\" 'git remote prune %s' to remove any old, conflicting \"\n>> \t\t\t\"branches\"), data->remote_name);\n>> -\t\tdata->conflict_msg_shown = 1;\n>> +\t\tdata->conflict_msg_shown = true;\n>> \t} else {\n>> \t\tconst char *reason = ref_transaction_error_msg(err);\n>>\n>> diff --git a/refs/files-backend.c b/refs/files-backend.c\n>> index 088b52c740..9f58ea4858 100644\n>> --- a/refs/files-backend.c\n>> +++ b/refs/files-backend.c\n>> @@ -776,6 +776,8 @@ static enum ref_transaction_error lock_raw_ref(struct files_ref_store *refs,\n>> \t\t\tgoto retry;\n>> \t\t} else {\n>> \t\t\tunable_to_lock_message(ref_file.buf, myerr, err);\n>> +\t\t\tif (myerr == EEXIST)\n>> +\t\t\t\tret = REF_TRANSACTION_ERROR_CREATE_EXISTS;\n>> \t\t\tgoto error_return;\n>> \t\t}\n>> \t}\n>\n> This here is the actual bug fix that makes us treat the error\n> gracefully.\n>\n>> diff --git a/t/t1400-update-ref.sh b/t/t1400-update-ref.sh\n>> index 96648a6e5d..e37a5d83e8 100755\n>> --- a/t/t1400-update-ref.sh\n>> +++ b/t/t1400-update-ref.sh\n>> @@ -2294,6 +2294,30 @@ do\n>> \t\t)\n>> \t'\n>>\n>> +\ttest_expect_success CASE_INSENSITIVE_FS,REFFILES \"stdin $type batch-updates existing reference\" '\n>> +\t\tgit init repo &&\n>> +\t\ttest_when_finished \"rm -fr repo\" &&\n>> +\t\t(\n>> +\t\t\tcd repo &&\n>> +\t\t\ttest_commit one &&\n>> +\t\t\told_head=$(git rev-parse HEAD) &&\n>> +\t\t\ttest_commit two &&\n>> +\t\t\thead=$(git rev-parse HEAD) &&\n>> +\n>> +\t\t\tformat_command $type \"create refs/heads/foo\" \"$head\" >stdin &&\n>> +\t\t\tformat_command $type \"create refs/heads/ref\" \"$old_head\" >>stdin &&\n>> +\t\t\tformat_command $type \"create refs/heads/Foo\" \"$old_head\" >>stdin &&\n>\n> These could be written as:\n>\n> {\n> format_command $type \"create refs/heads/foo\" \"$head\" &&\n> format_command $type \"create refs/heads/ref\" \"$old_head\" &&\n> format_command $type \"create refs/heads/Foo\" \"$old_head\"\n> } >stdin\n>\n\nThat reads much better, thanks.\n\n>> +\t\t\tgit update-ref $type --stdin --batch-updates stdout &&\n>> +\n>> +\t\t\techo $head >expect &&\n>> +\t\t\tgit rev-parse refs/heads/foo >actual &&\n>> +\t\t\techo $old_head >expect &&\n>> +\t\t\tgit rev-parse refs/heads/ref >actual &&\n>> +\t\t\ttest_cmp expect actual &&\n>> +\t\t\ttest_grep -q \"reference already exists\" stdout\n>> +\t\t)\n>> +\t'\n>> +\n>> \ttest_expect_success \"stdin $type batch-updates delete incorrect symbolic ref\" '\n>> \t\tgit init repo &&\n>> \t\ttest_when_finished \"rm -fr repo\" &&\n>\n> We could think about making these tests not depend on the REFFILES\n> prerequisite and then verify that with the reftable backend things work\n> as expected.\n>\n> Patrick\n\nThat would be a good test to add, will add it in the next version.\n"},{"id":"525420","messageId":"aLgrAs1J2k47uH6v@pks.im","threadId":"64069","inReplyTo":"CAOLa=ZTxZXWuFxaTWXmVpfoFjao_F0Mcu469aKQRu5fNNB0KEA@mail.gmail.com","subject":"Re: [PATCH 1/2] refs/files: use correct error type when locking fails","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-09-03T11:48:18Z","receivedAt":"2025-09-03T11:48:25Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Wed, Sep 03, 2025 at 03:38:04AM -0700, Karthik Nayak wrote:\n> Patrick Steinhardt writes:\n> > On Tue, Sep 02, 2025 at 10:34:25AM +0200, Karthik Nayak wrote:\n> >> diff --git a/builtin/fetch.c b/builtin/fetch.c\n> >> index 24645c4653..9563abbe12 100644\n> >> --- a/builtin/fetch.c\n> >> +++ b/builtin/fetch.c\n> >> @@ -1657,11 +1658,25 @@ static void ref_transaction_rejection_handler(const char *refname,\n> >> {\n> >> \tstruct ref_rejection_data *data = cb_data;\n> >>\n> >> -\tif (err == REF_TRANSACTION_ERROR_NAME_CONFLICT && !data->conflict_msg_shown) {\n> >> +\tif (err == REF_TRANSACTION_ERROR_CREATE_EXISTS && ignore_case &&\n> >> +\t !data->case_sensitive_msg_shown) {\n> >> +\t\terror(_(\"You're on a case-insensitive filesystem, and the remote you are\\n\"\n> >> +\t\t\t\"trying to fetch from has references that only differ in casing. It\\n\"\n> >> +\t\t\t\"is impossible to store such references with the 'files' backend. You\\n\"\n> >> +\t\t\t\"can either accept this as-is, in which case you won't be able to\\n\"\n> >> +\t\t\t\"store all remote references on disk. Or you can alternatively\\n\"\n> >> +\t\t\t\"migrate your repository to use the 'reftable' backend with the\\n\"\n> >> +\t\t\t\"following command:\\n\\n git refs migrate --ref-format=reftable\\n\\n\"\n> >> +\t\t\t\"Please keep in mind that not all implementations of Git support this\\n\"\n> >> +\t\t\t\"new format yet. So if you use tools other than Git to access this\\n\"\n> >> +\t\t\t\"repository it may not be an option to migrate to reftables.\\n\"));\n> >\n> > This reads familiar :)\n> >\n> \n> Which I failed to attribute to you, sorry for missing that, will add in\n> a 'Helped-by'.\n\nNo worries, I didn't mind it at all.\n\nPatrick\n"},{"id":"525449","messageId":"xmqqv7lzfpeu.fsf@gitster.g","threadId":"64069","inReplyTo":"20250902-587-git-fetch-1-fails-fetches-on-case-insensitive-repositories-v1-1-35e69bbb507d@gmail.com","subject":"Re: [PATCH 1/2] refs/files: use correct error type when locking fails","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-09-03T16:53:45Z","receivedAt":"2025-09-03T16:53:48Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Karthik Nayak writes:\n\n> During the 'prepare' phase of reference transaction in the files\n> backend, we create the lock files for references to be created. When\n> using batched updates on case-insensitive filesystems, the transactions\n> would be aborted if there are conflicting names such as:\n>\n> refs/heads/Foo\n> refs/heads/foo\n>\n> This affects all commands which were migrated to use batched updates in\n> Git 2.51, including 'git-fetch(1)' and 'git-receive-pack(1)'. Before\n> that, references updates would be applied serially with one transaction\n> used per update. When users fetched multiple references on\n> case-insensitive systems, subsequent references would simply overwrite\n> any earlier references. So when fetching:\n>\n> refs/heads/foo: 5f34ec0bfeac225b1c854340257a65b106f70ea6\n> refs/heads/Foo: ec3053b0977e83d9b67fc32c4527a117953994f3\n> refs/heads/sample: 2eefd1150e06d8fca1ddfa684dec016f36bf4e56\n>\n> The user would simply end up with:\n>\n> refs/heads/foo: ec3053b0977e83d9b67fc32c4527a117953994f3\n> refs/heads/sample: 2eefd1150e06d8fca1ddfa684dec016f36bf4e56\n>\n> This is buggy behavior since the user is never intimated about the\n\n\"intimated\" -> \"informed\" or simply \"told\".\n\n> overrides performed and missing references. Nevertheless, the user is\n> left with a working repository with a subset of the references. Since\n> Git 2.51, in such situations fetches would simply fail without applying\n\n\"applying\" -> \"updating\".\n\n> any references. Which is also buggy behavior and worse off since the\n> user is left without any references.\n\nVery true.\n\n> The error is triggered in `lock_raw_ref()` where the files backend\n> attempts to create a lock file. When a lock file already exists the\n> function returns a 'REF_TRANSACTION_ERROR_GENERIC'. Change this to return\n> 'REF_TRANSACTION_ERROR_CREATE_EXISTS' instead to aid the batched update\n> mechanism to simply reject such errors.\n\nIn the above description, both \"batched\" and \"transaction\" are used\nbut they mean different things and their difference is critical to\nthis description, right? IIUC, the mechanism for \"batched updates\"\nis based on the transaction mechanism where all-or-none is the norm,\nand when in batched mode, that all-or-none-ness that makes it a\ntransaction is deliberately broken and lets certain types of errors\ncause operations on refs individually rejected.\n\nAfter \"The error is triggerred...a REF_TRANSACTION_ERROR_GENERIC\"\nbut before \"Change this\", you would want to say what the code does\n(i.e. \"When this happens, the entire batched updates, not individual\noperation, is aborted as if it were in a transaction\") to highlight\nwhy you would want to \"Change this\", wouldn't you?\n\n> While the earlier implementation allowed the last reference to be\n> applied overriding the initial references, this change would allow the\n> first reference to be applied while rejecting consequent collisions.\n> This should be an OKAY compromise since with the files backend, there is\n> no scenario possible where we would retain all colliding references.\n\nOK. How do we know that a existing lockfile on a case insensitive\nfilesystem can only be due to somebody tried to lock a ref that is\nonly different in case, and not a leftover lockfile or lockfile held\nby some competing process? Don't we _know_ all the refs that are\ninvolved in _our_ batched update when we find that we failed to lock\none particular ref? We can inspect other refs we have locked so far\n(assuming that the transaction mechanism knows what refs it is\nupdating) and see if one of them is truly conflicting only in case,\nand if the code did so, I am happy if the code ignored that lock\nfailure (and the ref update). But I feel a bit uneasy to see that\nany \"ah there already is _somebody_ holding a lock on this ref\"\nwithout checking that it is _we_ that took the lock for another ref\nwhose path is only different in case and ignoring the failure.\n\n> diff --git a/refs/files-backend.c b/refs/files-backend.c\n> index 088b52c740..9f58ea4858 100644\n> --- a/refs/files-backend.c\n> +++ b/refs/files-backend.c\n> @@ -776,6 +776,8 @@ static enum ref_transaction_error lock_raw_ref(struct files_ref_store *refs,\n> \t\t\tgoto retry;\n> \t\t} else {\n> \t\t\tunable_to_lock_message(ref_file.buf, myerr, err);\n> +\t\t\tif (myerr == EEXIST)\n> +\t\t\t\tret = REF_TRANSACTION_ERROR_CREATE_EXISTS;\n> \t\t\tgoto error_return;\n\nI guess the place to check would be here in EEXIST case. Since it\nis an error codepath, we can afford to be more careful (probably\nwith an out-of-line logic implemented in a helper function call made\nfrom here).\n\nThanks.\n"},{"id":"525452","messageId":"xmqqms7bfll4.fsf@gitster.g","threadId":"64069","inReplyTo":"20250902-587-git-fetch-1-fails-fetches-on-case-insensitive-repositories-v1-2-35e69bbb507d@gmail.com","subject":"Re: [PATCH 2/2] refs/files: handle F/D conflicts in case-insensitive FS","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-09-03T18:16:23Z","receivedAt":"2025-09-03T18:16:26Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Karthik Nayak writes:\n\n> Similar to the previous commit, when using the files-backend on\n> case-insensitive filesystems, there is possibility of hitting F/D\n> conflicts when creating references within a single transaction, such as:\n>\n> - 'refs/heads/foo'\n> - 'refs/heads/Foo/bar'\n>\n> Ideally such conflicts are caught in `refs_verify_refnames_available()`\n> which is responsible for checking F/D conflicts within a given\n> transaction. This utility function is shared across the reference\n> backends. As such, it doesn't consider the issues of using a\n> case-insensitive, which only affects the files-backend.\n\nSounds like a sensible way to separate the issues and\nresponsibilities between higher and lower layers.\n\n> While one solution would be to make the function aware of such issues.\n> This feels like...\n\nThe first line alone is only half a sentence. \"such issues. This\"\n-> \"such issues, this\".\n\n> ... leaking implementation details of file-backend specific\n> issues into the utility function.\n\nVery true.\n\n> So opt for the more simpler option, of\n> lowercasing all references sent to this function when on a\n> case-insensitive filesystem and operating on the files-backend.\n\nSo when you are trying to lock \"Foo\", you lock \"foo\", for example?\nHow would that let the generic code liks verify_refname_available\nnotice that an existing ref \"Foo/bar\" would crash with the name you\nare trying to take, which is now downcased to be \"foo\"? I am not\nsure if the above explanation is sufficiently clear to convince\nreaders why it is sufficient..\n\n> Reported-by: Junio C Hamano \n\nHmph, I do not recall reporting anything, but perhaps it was a long\ntime ago...\n\n> Signed-off-by: Karthik Nayak \n> ---\n> refs/files-backend.c | 19 +++++++++++++++++--\n> t/t5510-fetch.sh | 20 ++++++++++++++++++++\n> 2 files changed, 37 insertions(+), 2 deletions(-)\n>\n> diff --git a/refs/files-backend.c b/refs/files-backend.c\n> index 9f58ea4858..466cdfe121 100644\n> --- a/refs/files-backend.c\n> +++ b/refs/files-backend.c\n> @@ -869,8 +869,23 @@ static enum ref_transaction_error lock_raw_ref(struct files_ref_store *refs,\n> \t\t * If the ref did not exist and we are creating it, we have to\n> \t\t * make sure there is no existing packed ref that conflicts\n> \t\t * with refname. This check is deferred so that we can batch it.\n> +\t\t *\n> +\t\t * For case-insensitive filesystems, we should also check for F/D\n> +\t\t * conflicts between 'foo' and 'Foo/bar'. So let's lowercase\n> +\t\t * the refname.\n> \t\t */\n> -\t\titem = string_list_append(refnames_to_check, refname);\n> +\t\tif (ignore_case) {\n> +\t\t\tstruct strbuf lower = STRBUF_INIT;\n> +\n> +\t\t\tstrbuf_addstr(&lower, refname);\n> +\t\t\tstrbuf_tolower(&lower);\n> +\n> +\t\t\titem = string_list_append(refnames_to_check, lower.buf);\n> +\t\t\tstrbuf_release(&lower);\n> +\t\t} else {\n> +\t\t\titem = string_list_append(refnames_to_check, refname);\n> +\t\t}\n> +\n> \t\titem->util = xmalloc(sizeof(update_idx));\n> \t\tmemcpy(item->util, &update_idx, sizeof(update_idx));\n> \t}\n> @@ -2796,7 +2811,7 @@ static int files_transaction_prepare(struct ref_store *ref_store,\n> \t\t\t \"ref_transaction_prepare\");\n> \tsize_t i;\n> \tint ret = 0;\n> -\tstruct string_list refnames_to_check = STRING_LIST_INIT_NODUP;\n> +\tstruct string_list refnames_to_check = STRING_LIST_INIT_DUP;\n> \tchar *head_ref = NULL;\n> \tint head_type;\n> \tstruct files_transaction_backend_data *backend_data;\n> diff --git a/t/t5510-fetch.sh b/t/t5510-fetch.sh\n> index 57f60da81b..84dc68e5f3 100755\n> --- a/t/t5510-fetch.sh\n> +++ b/t/t5510-fetch.sh\n> @@ -53,6 +53,12 @@ test_expect_success \"clone and setup child repos\" '\n> \t\tcd case_sensitive &&\n> \t\tgit branch branch1 &&\n> \t\tgit branch bRanch1\n> +\t) &&\n> +\tgit clone --ref-format=reftable . case_sensitive_fd &&\n> +\t(\n> +\t\tcd case_sensitive_fd &&\n> +\t\tgit branch foo/bar &&\n> +\t\tgit branch Foo\n> \t)\n> '\n> \n> @@ -1546,6 +1552,20 @@ test_expect_success CASE_INSENSITIVE_FS,REFFILES 'existing references in a case\n> \t)\n> '\n> \n> +test_expect_success CASE_INSENSITIVE_FS,REFFILES 'F/D conflict on case insensitive filesystem' '\n> +\ttest_when_finished rm -rf case_insensitive &&\n> +\t(\n> +\t\tgit init --bare case_insensitive &&\n> +\t\tcd case_insensitive &&\n> +\t\tgit remote add origin -- ../case_sensitive_fd &&\n> +\t\ttest_must_fail git fetch -f origin \"refs/heads/*:refs/heads/*\" 2>err &&\n> +\t\ttest_grep \"failed: refname conflict\" err &&\n> +\t\tgit rev-parse refs/heads/main >expect &&\n> +\t\tgit rev-parse refs/heads/foo/bar >actual &&\n> +\t\ttest_cmp expect actual\n> +\t)\n> +'\n> +\n> . \"$TEST_DIRECTORY\"/lib-httpd.sh\n> start_httpd\n"},{"id":"525562","messageId":"CAOLa=ZSj5iggtNuX3FDNdDsDFKE24+bmQSe1xeXmsfcWGagkWQ@mail.gmail.com","threadId":"64069","inReplyTo":"CAPx1GvdA3dhWN8b6wORdL1MktdVoJ3E2s9sgoAyUDwFK+7WdzA@mail.gmail.com","subject":"Re: [PATCH 1/2] refs/files: use correct error type when locking fails","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-09-04T20:24:43Z","receivedAt":"2025-09-04T20:24:44Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Chris Torek writes:\n\n> Minor:\n>\n> On Tue, Sep 2, 2025 at 1:40 AM Karthik Nayak wrote:\n>> During the 'prepare' phase of reference transaction in the files\n>> backend, we create the lock files for references to be created. When\n> [mass snippage for space]\n>>\n>> This is buggy behavior since the user is never intimated about the\n>> overrides performed and missing references. Nevertheless, the user is\n>\n> \"Intimated\" is the wrong word (also this is a rare form, \"intimate\"\n> as a verb that is, at least in US and British English). I'd suggest\n> \"informed\" here, or some alternate phrasing.\n>\n\nThanks, will swap it out!\n\n> Chris\n"},{"id":"525569","messageId":"xmqqfrd17s9p.fsf@gitster.g","threadId":"64069","inReplyTo":"xmqqms7bfll4.fsf@gitster.g","subject":"Re: [PATCH 2/2] refs/files: handle F/D conflicts in case-insensitive FS","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-09-04T22:43:46Z","receivedAt":"2025-09-04T22:43:48Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n>> So opt for the more simpler option, of\n>> lowercasing all references sent to this function when on a\n>> case-insensitive filesystem and operating on the files-backend.\n>\n> So when you are trying to lock \"Foo\", you lock \"foo\", for example?\n> How would that let the generic code liks verify_refname_available\n> notice that an existing ref \"Foo/bar\" would crash with the name you\n> are trying to take, which is now downcased to be \"foo\"? I am not\n> sure if the above explanation is sufficiently clear to convince\n> readers why it is sufficient..\n\nHere is a quick-and-dirty monkey-see-monkey-do patch on top of your\nseries, and with it jobs on osx boxes seem to fail.\n\nhttps://github.com/git/git/actions/runs/17477662700/job/49641377250#step:4:1912\n\n t/t5510-fetch.sh | 20 ++++++++++++++++++++\n 1 file changed, 20 insertions(+)\n\ndiff --git a/t/t5510-fetch.sh b/t/t5510-fetch.sh\nindex 84dc68e5f3..76af769f5c 100755\n--- a/t/t5510-fetch.sh\n+++ b/t/t5510-fetch.sh\n@@ -59,6 +59,12 @@ test_expect_success \"clone and setup child repos\" '\n \t\tcd case_sensitive_fd &&\n \t\tgit branch foo/bar &&\n \t\tgit branch Foo\n+\t) &&\n+\tgit clone --ref-format=reftable . case_sensitive_df &&\n+\t(\n+\t\tcd case_sensitive_df &&\n+\t\tgit branch Foo/bar &&\n+\t\tgit branch foo\n \t)\n '\n \n@@ -1566,6 +1572,20 @@ test_expect_success CASE_INSENSITIVE_FS,REFFILES 'F/D conflict on case insensiti\n \t)\n '\n \n+test_expect_success CASE_INSENSITIVE_FS,REFFILES 'D/F conflict on case insensitive filesystem' '\n+\ttest_when_finished rm -rf case_insensitive &&\n+\t(\n+\t\tgit init --bare case_insensitive &&\n+\t\tcd case_insensitive &&\n+\t\tgit remote add origin -- ../case_sensitive_df &&\n+\t\ttest_must_fail git fetch -f origin \"refs/heads/*:refs/heads/*\" 2>err &&\n+\t\ttest_grep \"failed: refname conflict\" err &&\n+\t\tgit rev-parse refs/heads/main >expect &&\n+\t\tgit rev-parse refs/heads/Foo/bar >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n . \"$TEST_DIRECTORY\"/lib-httpd.sh\n start_httpd\n \n-- \n2.51.0-314-g9a15cfd6dc\n\n\n"},{"id":"525781","messageId":"CAOLa=ZTC1=KRB6pANUcwpoiVKSbbCGJQPG=WW_TLEjHziyO6ig@mail.gmail.com","threadId":"64069","inReplyTo":"aLfw-peLY8NEKSZd@pks.im","subject":"Re: [PATCH 2/2] refs/files: handle F/D conflicts in case-insensitive FS","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-09-08T07:27:03Z","receivedAt":"2025-09-08T07:27:06Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Patrick Steinhardt writes:\n\n> On Tue, Sep 02, 2025 at 10:34:26AM +0200, Karthik Nayak wrote:\n>> Similar to the previous commit, when using the files-backend on\n>> case-insensitive filesystems, there is possibility of hitting F/D\n>> conflicts when creating references within a single transaction, such as:\n>>\n>> - 'refs/heads/foo'\n>> - 'refs/heads/Foo/bar'\n>\n> Great, I wanted to ask about this scenario.\n>\n>> Ideally such conflicts are caught in `refs_verify_refnames_available()`\n>> which is responsible for checking F/D conflicts within a given\n>> transaction. This utility function is shared across the reference\n>> backends. As such, it doesn't consider the issues of using a\n>> case-insensitive, which only affects the files-backend.\n>>\n>> While one solution would be to make the function aware of such issues.\n>> This feels like leaking implementation details of file-backend specific\n>> issues into the utility function. So opt for the more simpler option, of\n>> lowercasing all references sent to this function when on a\n>> case-insensitive filesystem and operating on the files-backend.\n>>\n>> To do this, simply use a `struct strbuf` to convert the refname to a\n>> lower case and append it to the list of refnames to be checked. Since we\n>> use a `struct strbuf` and the memory is cleared right after, make sure\n>> that the string list duplicates all provided string.\n>>\n>> Without this change, the user would simply be left with a repository\n>> with '.lock' files which were created in the 'prepare' phase of the\n>> transaction, as the 'commit' phase would simply abort and not do the\n>> necessary cleanup.\n>\n> Oh, that's a clever hack. Does this also work for the case where we have\n> preexisting refs already that differ only in casing? I guess it should\n> given that the lookups we perform should yield those refs regardless of\n> their casing.\n>\n> In any case, if we don't already have such a test it would be great to\n> also verify that this works as expected.\n>\n\nThere is a case which I missed and Junio also highlighted, I will fix\nthat in the consequent version.\n\n>> diff --git a/refs/files-backend.c b/refs/files-backend.c\n>> index 9f58ea4858..466cdfe121 100644\n>> --- a/refs/files-backend.c\n>> +++ b/refs/files-backend.c\n>> @@ -869,8 +869,23 @@ static enum ref_transaction_error lock_raw_ref(struct files_ref_store *refs,\n>> \t\t * If the ref did not exist and we are creating it, we have to\n>> \t\t * make sure there is no existing packed ref that conflicts\n>> \t\t * with refname. This check is deferred so that we can batch it.\n>> +\t\t *\n>> +\t\t * For case-insensitive filesystems, we should also check for F/D\n>> +\t\t * conflicts between 'foo' and 'Foo/bar'. So let's lowercase\n>> +\t\t * the refname.\n>> \t\t */\n>> -\t\titem = string_list_append(refnames_to_check, refname);\n>> +\t\tif (ignore_case) {\n>> +\t\t\tstruct strbuf lower = STRBUF_INIT;\n>> +\n>> +\t\t\tstrbuf_addstr(&lower, refname);\n>> +\t\t\tstrbuf_tolower(&lower);\n>> +\n>> +\t\t\titem = string_list_append(refnames_to_check, lower.buf);\n>> +\t\t\tstrbuf_release(&lower);\n>\n> Can we use `string_list_append_nodup()` together with `strbuf_detach()`\n> here to avoid one memory allocation?\n>\n\nThat's clever, I'll add that in, thanks!\n\n>> +\t\t} else {\n>> +\t\t\titem = string_list_append(refnames_to_check, refname);\n>> +\t\t}\n>> +\n>> \t\titem->util = xmalloc(sizeof(update_idx));\n>> \t\tmemcpy(item->util, &update_idx, sizeof(update_idx));\n>> \t}\n>> diff --git a/t/t5510-fetch.sh b/t/t5510-fetch.sh\n>> index 57f60da81b..84dc68e5f3 100755\n>> --- a/t/t5510-fetch.sh\n>> +++ b/t/t5510-fetch.sh\n>> @@ -53,6 +53,12 @@ test_expect_success \"clone and setup child repos\" '\n>> \t\tcd case_sensitive &&\n>> \t\tgit branch branch1 &&\n>> \t\tgit branch bRanch1\n>> +\t) &&\n>> +\tgit clone --ref-format=reftable . case_sensitive_fd &&\n>> +\t(\n>> +\t\tcd case_sensitive_fd &&\n>> +\t\tgit branch foo/bar &&\n>> +\t\tgit branch Foo\n>> \t)\n>> '\n>>\n>\n> Nice idea to use the reftable format here.\n>\n\nYeah, that was the only way.\n\n>> @@ -1546,6 +1552,20 @@ test_expect_success CASE_INSENSITIVE_FS,REFFILES 'existing references in a case\n>> \t)\n>> '\n>>\n>> +test_expect_success CASE_INSENSITIVE_FS,REFFILES 'F/D conflict on case insensitive filesystem' '\n>> +\ttest_when_finished rm -rf case_insensitive &&\n>> +\t(\n>> +\t\tgit init --bare case_insensitive &&\n>> +\t\tcd case_insensitive &&\n>> +\t\tgit remote add origin -- ../case_sensitive_fd &&\n>> +\t\ttest_must_fail git fetch -f origin \"refs/heads/*:refs/heads/*\" 2>err &&\n>> +\t\ttest_grep \"failed: refname conflict\" err &&\n>> +\t\tgit rev-parse refs/heads/main >expect &&\n>> +\t\tgit rev-parse refs/heads/foo/bar >actual &&\n>> +\t\ttest_cmp expect actual\n>> +\t)\n>> +'\n>\n> Okay, so we again only end up with one of these references, which is the\n> best we can do.\n>\n\nExactly.\n\n> atrick\n"},{"id":"525783","messageId":"CAOLa=ZT7vUdkB_6YaK7P1P72VwvHjVxix36q+G3CEqfze8Km5w@mail.gmail.com","threadId":"64069","inReplyTo":"xmqqms7bfll4.fsf@gitster.g","subject":"Re: [PATCH 2/2] refs/files: handle F/D conflicts in case-insensitive FS","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-09-08T09:27:13Z","receivedAt":"2025-09-08T09:27:15Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Junio C Hamano writes:\n\n> Karthik Nayak writes:\n>\n>> Similar to the previous commit, when using the files-backend on\n>> case-insensitive filesystems, there is possibility of hitting F/D\n>> conflicts when creating references within a single transaction, such as:\n>>\n>> - 'refs/heads/foo'\n>> - 'refs/heads/Foo/bar'\n>>\n>> Ideally such conflicts are caught in `refs_verify_refnames_available()`\n>> which is responsible for checking F/D conflicts within a given\n>> transaction. This utility function is shared across the reference\n>> backends. As such, it doesn't consider the issues of using a\n>> case-insensitive, which only affects the files-backend.\n>\n> Sounds like a sensible way to separate the issues and\n> responsibilities between higher and lower layers.\n>\n\nI totally agree, it shouldn't know/care about specific reference backends.\n\n>> While one solution would be to make the function aware of such issues.\n>> This feels like...\n>\n> The first line alone is only half a sentence. \"such issues. This\"\n> -> \"such issues, this\".\n>\n\nTotally fair. Fixed.\n\n>> ... leaking implementation details of file-backend specific\n>> issues into the utility function.\n>\n> Very true.\n>\n>> So opt for the more simpler option, of\n>> lowercasing all references sent to this function when on a\n>> case-insensitive filesystem and operating on the files-backend.\n>\n> So when you are trying to lock \"Foo\", you lock \"foo\", for example?\n> How would that let the generic code liks verify_refname_available\n> notice that an existing ref \"Foo/bar\" would crash with the name you\n> are trying to take, which is now downcased to be \"foo\"? I am not\n> sure if the above explanation is sufficiently clear to convince\n> readers why it is sufficient..\n>\n\nThanks for pointing it out and also providing a test. That indeed is a\nproblem, but that is an issue caught much earlier, creating 'foo', when\n'Foo/bar' exists, fails at the locking step, since the locking step also\nchecks for D/F conflicts. I've added another commit around this, fixing\nthe check that you mentioned.\n\n>> Reported-by: Junio C Hamano \n>\n> Hmph, I do not recall reporting anything, but perhaps it was a long\n> time ago...\n>\n\n[snip]\n"},{"id":"525785","messageId":"CAOLa=ZRshHZXWxTeNqgi253R2b0TB7o42sq6mPmNHQWN_T733g@mail.gmail.com","threadId":"64069","inReplyTo":"CAOLa=ZT7vUdkB_6YaK7P1P72VwvHjVxix36q+G3CEqfze8Km5w@mail.gmail.com","subject":"Re: [PATCH 2/2] refs/files: handle F/D conflicts in case-insensitive FS","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-09-08T09:29:52Z","receivedAt":"2025-09-08T09:29:54Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Karthik Nayak writes:\n\n[snip]\n\n>>> Reported-by: Junio C Hamano \n>>\n>> Hmph, I do not recall reporting anything, but perhaps it was a long\n>> time ago...\n>>\n>\n\nReported-by might not be the right field here, You mentioned about the\nD/F conflict in an earlier mail [1]. I would've missed it otherwise. So\nI wanted to attribute you for that :)\n\n[1]: https://lore.kernel.org/all/xmqq8qjbxn8n.fsf@gitster.g/\n\n> [snip]\n"},{"id":"525794","messageId":"20250908-587-git-fetch-1-fails-fetches-on-case-insensitive-repositories-v2-0-b2eb2459befb@gmail.com","threadId":"64069","inReplyTo":"20250902-587-git-fetch-1-fails-fetches-on-case-insensitive-repositories-v1-0-35e69bbb507d@gmail.com","subject":"[PATCH v2 0/4] refs/files: fix issues with git-fetch on case-insensitive FS","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-09-08T12:37:34Z","receivedAt":"2025-09-08T12:37:40Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Hello!\n\nWith Git 2.51 we moved 'git-fetch(1)' and 'git-receive-pack(1)' to use\nbatched updates while doing reference updates. This provided a nice perf\nboost since both commands will now use a single transaction for\nreference updates. This removes the overhead of using individual\ntransaction per reference update and also avoids unnecessary\nauto-compaction between reference updates in the reftable backend.\n\nHowever, in the files-backend it does introduce a few bugs around\nconflicts. The reported bug was around case-insensitive filesystems [1],\nbut we also fix some adjacent issues:\n\n1. When fetching references such as:\n\n - refs/heads/foo\n - refs/heads/Foo\n\nEarlier we would simply overwrite the first reference with the second\nand continue. Since Git 2.51 we simply abort stating a conflict.\n\nThis is resolved in the first commit by explicitly categorizing the\nerror as non-GENERIC. This allows batched updates to reject the\nparticular update, while updating the rest.\n\n2. When fetching references and a lock for a particular reference\nalready exits. We treat this is a GENERIC error, which fails the entire\nupdate. By categorizing this error as non-GENERIC, we can reject this\nspecific update and update the other references.\n\n3. When fetching references such as with F/D conflict:\n\n - refs/heads/foo\n - refs/heads/Foo/bar\n\nEarlier we would apply the first, while the second would fail due to\nconflict. Since Git 2.51, the lock files for both would be created, but\nthe 'commit' phase would abruptly end leaving the lock files.\n\nThe second commit fixes this by ensuring that on case-insensitive\nfilesystems we lowercase the refnames for availability check to ensure\nF/D are caught and reported to the user.\n\n4. When fetching references with D/F conflict:\n\n - refs/heads/Foo/bar\n - refs/heads/foo\n\nThe creation of the second reference's lock in `lock_raw_ref()` catches\nthe D/F conflict, but we mark this as a GENERIC error. By categorizing\nthis as non-GENERIC, we can allow the updates to continue while\nrejecting this specific error.\n\n- Karthik\n\n[1]: https://lore.kernel.org/all/YQXPR01MB3046197EF39296549EE6DD669A33A@YQXPR01MB3046.CANPRD01.PROD.OUTLOOK.COM/\n\nSigned-off-by: Karthik Nayak \n---\nChanges in v2:\n- This version fixes two more issues:\n - Fetching while locks already exist in the repository\n - D/F conflicts while fetching\n- Add a specific error to the first case, so we can nicely show a\n relevant error. Also check explicitly that the issue is due to\n case-insensitive filesystems.\n- Cleanup the commit messages.\n- Use `string_list_append_nodup()` with `strbuf_detach`, reducing the\n number of allocations.\n\n---\n builtin/fetch.c | 21 ++++++++++--\n refs.c | 10 +++++-\n refs.h | 2 ++\n refs/files-backend.c | 51 ++++++++++++++++++++++++-----\n t/t1400-update-ref.sh | 53 +++++++++++++++++++++++++++++++\n t/t5510-fetch.sh | 88 ++++++++++++++++++++++++++++++++++++++++++++++++++-\n 6 files changed, 212 insertions(+), 13 deletions(-)\n\nKarthik Nayak (4):\n refs/files: catch conflicts on case-insensitive file-systems\n refs/files: use correct error type when lock exists\n refs/files: handle F/D conflicts in case-insensitive FS\n refs/files: handle D/F conflicts during locking\n\nRange-diff versus v1:\n\n1: fe6e2c12e7 ! 1: bab864e28a refs/files: use correct error type when locking fails\n @@ Metadata\n Author: Karthik Nayak \n \n ## Commit message ##\n - refs/files: use correct error type when locking fails\n + refs/files: catch conflicts on case-insensitive file-systems\n \n During the 'prepare' phase of reference transaction in the files\n backend, we create the lock files for references to be created. When\n - using batched updates on case-insensitive filesystems, the transactions\n - would be aborted if there are conflicting names such as:\n + using batched updates on case-insensitive filesystems, the entire\n + batched updates would be aborted if there are conflicting names such as:\n \n refs/heads/Foo\n refs/heads/foo\n \n This affects all commands which were migrated to use batched updates in\n Git 2.51, including 'git-fetch(1)' and 'git-receive-pack(1)'. Before\n - that, references updates would be applied serially with one transaction\n + that, reference updates would be applied serially with one transaction\n used per update. When users fetched multiple references on\n case-insensitive systems, subsequent references would simply overwrite\n any earlier references. So when fetching:\n @@ Commit message\n refs/heads/foo: ec3053b0977e83d9b67fc32c4527a117953994f3\n refs/heads/sample: 2eefd1150e06d8fca1ddfa684dec016f36bf4e56\n \n - This is buggy behavior since the user is never intimated about the\n + This is buggy behavior since the user is never informed about the\n overrides performed and missing references. Nevertheless, the user is\n left with a working repository with a subset of the references. Since\n - Git 2.51, in such situations fetches would simply fail without applying\n + Git 2.51, in such situations fetches would simply fail without updating\n any references. Which is also buggy behavior and worse off since the\n user is left without any references.\n \n The error is triggered in `lock_raw_ref()` where the files backend\n attempts to create a lock file. When a lock file already exists the\n - function returns a 'REF_TRANSACTION_ERROR_GENERIC'. Change this to return\n - 'REF_TRANSACTION_ERROR_CREATE_EXISTS' instead to aid the batched update\n - mechanism to simply reject such errors.\n + function returns a 'REF_TRANSACTION_ERROR_GENERIC'. When this happens,\n + the entire batched updates, not individual operation, is aborted as if\n + it were in a transaction.\n +\n + Change this to return 'REF_TRANSACTION_ERROR_CASE_CONFLICT' instead to\n + aid the batched update mechanism to simply reject such errors. The\n + change only affects batched updates since batched updates will reject\n + individual updates with non-generic errors. So specifically this would\n + only affect:\n +\n + 1. git fetch\n + 2. git receive-pack\n + 3. git update-ref --batch-updates\n \n This bubbles the error type up to `files_transaction_prepare()` which\n tries to lock each reference update. So if the locking fails, we check\n if the rejection type can be ignored, which is done by calling\n `ref_transaction_maybe_set_rejected()`.\n \n - As the error type is now 'REF_TRANSACTION_ERROR_CREATE_EXISTS', the\n - specific reference update would simply be rejected, while other updates\n - in the transaction would continue to be applied. This allows partial\n - application of references in case-insensitive filesystems when fetching\n - colliding references.\n + As the error type is now 'REF_TRANSACTION_ERROR_CASE_CONFLICT',\n + the specific reference update would simply be rejected, while other\n + updates in the transaction would continue to be applied. This allows\n + partial application of references in case-insensitive filesystems when\n + fetching colliding references.\n \n While the earlier implementation allowed the last reference to be\n applied overriding the initial references, this change would allow the\n first reference to be applied while rejecting consequent collisions.\n - This should be an OKAY compromise since with the files backend, there is\n + This should be an okay compromise since with the files backend, there is\n no scenario possible where we would retain all colliding references.\n \n - The change only affects batched updates since batched updates will\n - reject individual updates with non-generic errors. So specifically this\n - would only affect:\n -\n - 1. git fetch\n - 2. git receive-pack\n - 3. git update-ref --batch-updates\n -\n Let's also be more pro-active and notify users on case-insensitive\n filesystems about such problems by providing a brief about the issue\n while also recommending using the reftable backend, which doesn't have\n the same issue.\n \n Reported-by: Joe Drew \n + Helped-by: Patrick Steinhardt \n Signed-off-by: Karthik Nayak \n \n ## builtin/fetch.c ##\n @@ builtin/fetch.c: static void ref_transaction_rejection_handler(const char *refna\n \tstruct ref_rejection_data *data = cb_data;\n \n -\tif (err == REF_TRANSACTION_ERROR_NAME_CONFLICT && !data->conflict_msg_shown) {\n -+\tif (err == REF_TRANSACTION_ERROR_CREATE_EXISTS && ignore_case &&\n ++\tif (err == REF_TRANSACTION_ERROR_CASE_CONFLICT && ignore_case &&\n +\t !data->case_sensitive_msg_shown) {\n +\t\terror(_(\"You're on a case-insensitive filesystem, and the remote you are\\n\"\n +\t\t\t\"trying to fetch from has references that only differ in casing. It\\n\"\n @@ builtin/fetch.c: static void ref_transaction_rejection_handler(const char *refna\n \t\tconst char *reason = ref_transaction_error_msg(err);\n \n \n + ## refs.c ##\n +@@ refs.c: const char *ref_transaction_error_msg(enum ref_transaction_error err)\n + \t\treturn \"invalid new value provided\";\n + \tcase REF_TRANSACTION_ERROR_EXPECTED_SYMREF:\n + \t\treturn \"expected symref but found regular ref\";\n ++\tcase REF_TRANSACTION_ERROR_CASE_CONFLICT:\n ++\t\treturn \"reference conflict due to case-insensitive filesystem\";\n + \tdefault:\n + \t\treturn \"unknown failure\";\n + \t}\n +\n + ## refs.h ##\n +@@ refs.h: enum ref_transaction_error {\n + \tREF_TRANSACTION_ERROR_INVALID_NEW_VALUE = -6,\n + \t/* Expected ref to be symref, but is a regular ref */\n + \tREF_TRANSACTION_ERROR_EXPECTED_SYMREF = -7,\n ++\t/* Cannot create ref due to case-insensitive filesystem */\n ++\tREF_TRANSACTION_ERROR_CASE_CONFLICT = -8,\n + };\n + \n + /*\n +\n ## refs/files-backend.c ##\n +@@ refs/files-backend.c: static void unlock_ref(struct ref_lock *lock)\n + \t}\n + }\n + \n ++static bool duplicate_reference_case_cmp(struct ref_transaction *transaction,\n ++\t\t\t\t\t struct ref_update *update)\n ++{\n ++\tfor (size_t i = 0; i < transaction->nr; i++) {\n ++\t\tif (transaction->updates[i] == update)\n ++\t\t\tbreak;\n ++\n ++\t\tif (!strcasecmp(transaction->updates[i]->refname, update->refname))\n ++\t\t\treturn true;\n ++\t}\n ++\treturn false;\n ++}\n ++\n + /*\n + * Lock refname, without following symrefs, and set *lock_p to point\n + * at a newly-allocated lock object. Fill in lock->old_oid, referent,\n +@@ refs/files-backend.c: static void unlock_ref(struct ref_lock *lock)\n + * - Generate informative error messages in the case of failure\n + */\n + static enum ref_transaction_error lock_raw_ref(struct files_ref_store *refs,\n +-\t\t\t\t\t struct ref_update *update,\n ++\t\t\t\t\t struct ref_transaction *transaction,\n + \t\t\t\t\t size_t update_idx,\n + \t\t\t\t\t int mustexist,\n + \t\t\t\t\t struct string_list *refnames_to_check,\n +-\t\t\t\t\t const struct string_list *extras,\n + \t\t\t\t\t struct ref_lock **lock_p,\n + \t\t\t\t\t struct strbuf *referent,\n + \t\t\t\t\t struct strbuf *err)\n + {\n + \tenum ref_transaction_error ret = REF_TRANSACTION_ERROR_GENERIC;\n ++\tstruct ref_update *update = transaction->updates[update_idx];\n ++\tconst struct string_list *extras = &transaction->refnames;\n + \tconst char *refname = update->refname;\n + \tunsigned int *type = &update->type;\n + \tstruct ref_lock *lock;\n @@ refs/files-backend.c: static enum ref_transaction_error lock_raw_ref(struct files_ref_store *refs,\n \t\t\tgoto retry;\n \t\t} else {\n \t\t\tunable_to_lock_message(ref_file.buf, myerr, err);\n -+\t\t\tif (myerr == EEXIST)\n -+\t\t\t\tret = REF_TRANSACTION_ERROR_CREATE_EXISTS;\n ++\t\t\tif (myerr == EEXIST && ignore_case &&\n ++\t\t\t duplicate_reference_case_cmp(transaction, update))\n ++\t\t\t\tret = REF_TRANSACTION_ERROR_CASE_CONFLICT;\n \t\t\tgoto error_return;\n \t\t}\n \t}\n +@@ refs/files-backend.c: static enum ref_transaction_error lock_ref_for_update(struct files_ref_store *re\n + \tif (lock) {\n + \t\tlock->count++;\n + \t} else {\n +-\t\tret = lock_raw_ref(refs, update, update_idx, mustexist,\n +-\t\t\t\t refnames_to_check, &transaction->refnames,\n +-\t\t\t\t &lock, &referent, err);\n ++\t\tret = lock_raw_ref(refs, transaction, update_idx, mustexist,\n ++\t\t\t\t refnames_to_check, &lock, &referent, err);\n + \t\tif (ret) {\n + \t\t\tchar *reason;\n + \n \n ## t/t1400-update-ref.sh ##\n @@ t/t1400-update-ref.sh: do\n @@ t/t1400-update-ref.sh: do\n +\t\t\ttest_commit two &&\n +\t\t\thead=$(git rev-parse HEAD) &&\n +\n -+\t\t\tformat_command $type \"create refs/heads/foo\" \"$head\" >stdin &&\n -+\t\t\tformat_command $type \"create refs/heads/ref\" \"$old_head\" >>stdin &&\n -+\t\t\tformat_command $type \"create refs/heads/Foo\" \"$old_head\" >>stdin &&\n ++\t\t\t{\n ++\t\t\t\tformat_command $type \"create refs/heads/foo\" \"$head\" &&\n ++\t\t\t\tformat_command $type \"create refs/heads/ref\" \"$old_head\" &&\n ++\t\t\t\tformat_command $type \"create refs/heads/Foo\" \"$old_head\"\n ++\t\t\t} >stdin &&\n ++\t\t\tgit update-ref $type --stdin --batch-updates stdout &&\n ++\n ++\t\t\techo $head >expect &&\n ++\t\t\tgit rev-parse refs/heads/foo >actual &&\n ++\t\t\techo $old_head >expect &&\n ++\t\t\tgit rev-parse refs/heads/ref >actual &&\n ++\t\t\ttest_cmp expect actual &&\n ++\t\t\ttest_grep -q \"reference conflict due to case-insensitive filesystem\" stdout\n ++\t\t)\n ++\t'\n ++\n ++\ttest_expect_success CASE_INSENSITIVE_FS \"stdin $type batch-updates existing reference\" '\n ++\t\tgit init --ref-format=reftable repo &&\n ++\t\ttest_when_finished \"rm -fr repo\" &&\n ++\t\t(\n ++\t\t\tcd repo &&\n ++\t\t\ttest_commit one &&\n ++\t\t\told_head=$(git rev-parse HEAD) &&\n ++\t\t\ttest_commit two &&\n ++\t\t\thead=$(git rev-parse HEAD) &&\n ++\n ++\t\t\t{\n ++\t\t\t\tformat_command $type \"create refs/heads/foo\" \"$head\" &&\n ++\t\t\t\tformat_command $type \"create refs/heads/ref\" \"$old_head\" &&\n ++\t\t\t\tformat_command $type \"create refs/heads/Foo\" \"$old_head\"\n ++\t\t\t} >stdin &&\n +\t\t\tgit update-ref $type --stdin --batch-updates stdout &&\n +\n +\t\t\techo $head >expect &&\n @@ t/t1400-update-ref.sh: do\n +\t\t\techo $old_head >expect &&\n +\t\t\tgit rev-parse refs/heads/ref >actual &&\n +\t\t\ttest_cmp expect actual &&\n -+\t\t\ttest_grep -q \"reference already exists\" stdout\n ++\t\t\tgit rev-parse refs/heads/Foo >actual &&\n ++\t\t\ttest_cmp expect actual\n +\t\t)\n +\t'\n +\n-: ---------- > 2: b0ecf6f10d refs/files: use correct error type when lock exists\n2: 30a2629ebc ! 3: 1842ddee90 refs/files: handle F/D conflicts in case-insensitive FS\n @@ Metadata\n ## Commit message ##\n refs/files: handle F/D conflicts in case-insensitive FS\n \n - Similar to the previous commit, when using the files-backend on\n - case-insensitive filesystems, there is possibility of hitting F/D\n - conflicts when creating references within a single transaction, such as:\n + When using the files-backend on case-insensitive filesystems, there is\n + possibility of hitting F/D conflicts when creating references within a\n + single transaction, such as:\n \n - 'refs/heads/foo'\n - 'refs/heads/Foo/bar'\n @@ Commit message\n which is responsible for checking F/D conflicts within a given\n transaction. This utility function is shared across the reference\n backends. As such, it doesn't consider the issues of using a\n - case-insensitive, which only affects the files-backend.\n + case-insensitive file system, which only affects the files-backend.\n \n - While one solution would be to make the function aware of such issues.\n - This feels like leaking implementation details of file-backend specific\n + While one solution would be to make the function aware of such issues,\n + this feels like leaking implementation details of file-backend specific\n issues into the utility function. So opt for the more simpler option, of\n lowercasing all references sent to this function when on a\n case-insensitive filesystem and operating on the files-backend.\n @@ refs/files-backend.c: static enum ref_transaction_error lock_raw_ref(struct file\n +\t\t\tstrbuf_addstr(&lower, refname);\n +\t\t\tstrbuf_tolower(&lower);\n +\n -+\t\t\titem = string_list_append(refnames_to_check, lower.buf);\n -+\t\t\tstrbuf_release(&lower);\n ++\t\t\titem = string_list_append_nodup(refnames_to_check,\n ++\t\t\t\t\t\t\tstrbuf_detach(&lower, NULL));\n +\t\t} else {\n +\t\t\titem = string_list_append(refnames_to_check, refname);\n +\t\t}\n @@ t/t5510-fetch.sh: test_expect_success \"clone and setup child repos\" '\n \t)\n '\n \n -@@ t/t5510-fetch.sh: test_expect_success CASE_INSENSITIVE_FS,REFFILES 'existing references in a case\n +@@ t/t5510-fetch.sh: test_expect_success REFFILES 'existing reference lock in repo' '\n \t)\n '\n \n-: ---------- > 4: 81759b0e51 refs/files: handle D/F conflicts during locking\n\n\nbase-commit: c44beea485f0f2feaf460e2ac87fdd5608d63cf0\nchange-id: 20250822-587-git-fetch-1-fails-fetches-on-case-insensitive-repositories-0a187c7ac41e\n\nThanks\n- Karthik\n\n"},{"id":"525795","messageId":"20250908-587-git-fetch-1-fails-fetches-on-case-insensitive-repositories-v2-1-b2eb2459befb@gmail.com","threadId":"64069","inReplyTo":"20250908-587-git-fetch-1-fails-fetches-on-case-insensitive-repositories-v2-0-b2eb2459befb@gmail.com","subject":"[PATCH v2 1/4] refs/files: catch conflicts on case-insensitive file-systems","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-09-08T12:37:35Z","receivedAt":"2025-09-08T12:37:41Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"During the 'prepare' phase of reference transaction in the files\nbackend, we create the lock files for references to be created. When\nusing batched updates on case-insensitive filesystems, the entire\nbatched updates would be aborted if there are conflicting names such as:\n\n refs/heads/Foo\n refs/heads/foo\n\nThis affects all commands which were migrated to use batched updates in\nGit 2.51, including 'git-fetch(1)' and 'git-receive-pack(1)'. Before\nthat, reference updates would be applied serially with one transaction\nused per update. When users fetched multiple references on\ncase-insensitive systems, subsequent references would simply overwrite\nany earlier references. So when fetching:\n\n refs/heads/foo: 5f34ec0bfeac225b1c854340257a65b106f70ea6\n refs/heads/Foo: ec3053b0977e83d9b67fc32c4527a117953994f3\n refs/heads/sample: 2eefd1150e06d8fca1ddfa684dec016f36bf4e56\n\nThe user would simply end up with:\n\n refs/heads/foo: ec3053b0977e83d9b67fc32c4527a117953994f3\n refs/heads/sample: 2eefd1150e06d8fca1ddfa684dec016f36bf4e56\n\nThis is buggy behavior since the user is never informed about the\noverrides performed and missing references. Nevertheless, the user is\nleft with a working repository with a subset of the references. Since\nGit 2.51, in such situations fetches would simply fail without updating\nany references. Which is also buggy behavior and worse off since the\nuser is left without any references.\n\nThe error is triggered in `lock_raw_ref()` where the files backend\nattempts to create a lock file. When a lock file already exists the\nfunction returns a 'REF_TRANSACTION_ERROR_GENERIC'. When this happens,\nthe entire batched updates, not individual operation, is aborted as if\nit were in a transaction.\n\nChange this to return 'REF_TRANSACTION_ERROR_CASE_CONFLICT' instead to\naid the batched update mechanism to simply reject such errors. The\nchange only affects batched updates since batched updates will reject\nindividual updates with non-generic errors. So specifically this would\nonly affect:\n\n 1. git fetch\n 2. git receive-pack\n 3. git update-ref --batch-updates\n\nThis bubbles the error type up to `files_transaction_prepare()` which\ntries to lock each reference update. So if the locking fails, we check\nif the rejection type can be ignored, which is done by calling\n`ref_transaction_maybe_set_rejected()`.\n\nAs the error type is now 'REF_TRANSACTION_ERROR_CASE_CONFLICT',\nthe specific reference update would simply be rejected, while other\nupdates in the transaction would continue to be applied. This allows\npartial application of references in case-insensitive filesystems when\nfetching colliding references.\n\nWhile the earlier implementation allowed the last reference to be\napplied overriding the initial references, this change would allow the\nfirst reference to be applied while rejecting consequent collisions.\nThis should be an okay compromise since with the files backend, there is\nno scenario possible where we would retain all colliding references.\n\nLet's also be more pro-active and notify users on case-insensitive\nfilesystems about such problems by providing a brief about the issue\nwhile also recommending using the reftable backend, which doesn't have\nthe same issue.\n\nReported-by: Joe Drew \nHelped-by: Patrick Steinhardt \nSigned-off-by: Karthik Nayak \n---\n builtin/fetch.c | 21 +++++++++++++++++---\n refs.c | 2 ++\n refs.h | 2 ++\n refs/files-backend.c | 26 ++++++++++++++++++++-----\n t/t1400-update-ref.sh | 53 +++++++++++++++++++++++++++++++++++++++++++++++++++\n t/t5510-fetch.sh | 22 ++++++++++++++++++++-\n 6 files changed, 117 insertions(+), 9 deletions(-)\n\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex 24645c4653..c7ff3480fb 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -1643,7 +1643,8 @@ static int set_head(const struct ref *remote_refs, struct remote *remote)\n \n struct ref_rejection_data {\n \tint *retcode;\n-\tint conflict_msg_shown;\n+\tbool conflict_msg_shown;\n+\tbool case_sensitive_msg_shown;\n \tconst char *remote_name;\n };\n \n@@ -1657,11 +1658,25 @@ static void ref_transaction_rejection_handler(const char *refname,\n {\n \tstruct ref_rejection_data *data = cb_data;\n \n-\tif (err == REF_TRANSACTION_ERROR_NAME_CONFLICT && !data->conflict_msg_shown) {\n+\tif (err == REF_TRANSACTION_ERROR_CASE_CONFLICT && ignore_case &&\n+\t !data->case_sensitive_msg_shown) {\n+\t\terror(_(\"You're on a case-insensitive filesystem, and the remote you are\\n\"\n+\t\t\t\"trying to fetch from has references that only differ in casing. It\\n\"\n+\t\t\t\"is impossible to store such references with the 'files' backend. You\\n\"\n+\t\t\t\"can either accept this as-is, in which case you won't be able to\\n\"\n+\t\t\t\"store all remote references on disk. Or you can alternatively\\n\"\n+\t\t\t\"migrate your repository to use the 'reftable' backend with the\\n\"\n+\t\t\t\"following command:\\n\\n git refs migrate --ref-format=reftable\\n\\n\"\n+\t\t\t\"Please keep in mind that not all implementations of Git support this\\n\"\n+\t\t\t\"new format yet. So if you use tools other than Git to access this\\n\"\n+\t\t\t\"repository it may not be an option to migrate to reftables.\\n\"));\n+\t\tdata->case_sensitive_msg_shown = true;\n+\t} else if (err == REF_TRANSACTION_ERROR_NAME_CONFLICT &&\n+\t\t !data->conflict_msg_shown) {\n \t\terror(_(\"some local refs could not be updated; try running\\n\"\n \t\t\t\" 'git remote prune %s' to remove any old, conflicting \"\n \t\t\t\"branches\"), data->remote_name);\n-\t\tdata->conflict_msg_shown = 1;\n+\t\tdata->conflict_msg_shown = true;\n \t} else {\n \t\tconst char *reason = ref_transaction_error_msg(err);\n \ndiff --git a/refs.c b/refs.c\nindex bfdbe718b7..4c1c339ed9 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -3321,6 +3321,8 @@ const char *ref_transaction_error_msg(enum ref_transaction_error err)\n \t\treturn \"invalid new value provided\";\n \tcase REF_TRANSACTION_ERROR_EXPECTED_SYMREF:\n \t\treturn \"expected symref but found regular ref\";\n+\tcase REF_TRANSACTION_ERROR_CASE_CONFLICT:\n+\t\treturn \"reference conflict due to case-insensitive filesystem\";\n \tdefault:\n \t\treturn \"unknown failure\";\n \t}\ndiff --git a/refs.h b/refs.h\nindex eedbb599c5..41915086b3 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -31,6 +31,8 @@ enum ref_transaction_error {\n \tREF_TRANSACTION_ERROR_INVALID_NEW_VALUE = -6,\n \t/* Expected ref to be symref, but is a regular ref */\n \tREF_TRANSACTION_ERROR_EXPECTED_SYMREF = -7,\n+\t/* Cannot create ref due to case-insensitive filesystem */\n+\tREF_TRANSACTION_ERROR_CASE_CONFLICT = -8,\n };\n \n /*\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 088b52c740..58005d2732 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -647,6 +647,19 @@ static void unlock_ref(struct ref_lock *lock)\n \t}\n }\n \n+static bool duplicate_reference_case_cmp(struct ref_transaction *transaction,\n+\t\t\t\t\t struct ref_update *update)\n+{\n+\tfor (size_t i = 0; i < transaction->nr; i++) {\n+\t\tif (transaction->updates[i] == update)\n+\t\t\tbreak;\n+\n+\t\tif (!strcasecmp(transaction->updates[i]->refname, update->refname))\n+\t\t\treturn true;\n+\t}\n+\treturn false;\n+}\n+\n /*\n * Lock refname, without following symrefs, and set *lock_p to point\n * at a newly-allocated lock object. Fill in lock->old_oid, referent,\n@@ -677,16 +690,17 @@ static void unlock_ref(struct ref_lock *lock)\n * - Generate informative error messages in the case of failure\n */\n static enum ref_transaction_error lock_raw_ref(struct files_ref_store *refs,\n-\t\t\t\t\t struct ref_update *update,\n+\t\t\t\t\t struct ref_transaction *transaction,\n \t\t\t\t\t size_t update_idx,\n \t\t\t\t\t int mustexist,\n \t\t\t\t\t struct string_list *refnames_to_check,\n-\t\t\t\t\t const struct string_list *extras,\n \t\t\t\t\t struct ref_lock **lock_p,\n \t\t\t\t\t struct strbuf *referent,\n \t\t\t\t\t struct strbuf *err)\n {\n \tenum ref_transaction_error ret = REF_TRANSACTION_ERROR_GENERIC;\n+\tstruct ref_update *update = transaction->updates[update_idx];\n+\tconst struct string_list *extras = &transaction->refnames;\n \tconst char *refname = update->refname;\n \tunsigned int *type = &update->type;\n \tstruct ref_lock *lock;\n@@ -776,6 +790,9 @@ static enum ref_transaction_error lock_raw_ref(struct files_ref_store *refs,\n \t\t\tgoto retry;\n \t\t} else {\n \t\t\tunable_to_lock_message(ref_file.buf, myerr, err);\n+\t\t\tif (myerr == EEXIST && ignore_case &&\n+\t\t\t duplicate_reference_case_cmp(transaction, update))\n+\t\t\t\tret = REF_TRANSACTION_ERROR_CASE_CONFLICT;\n \t\t\tgoto error_return;\n \t\t}\n \t}\n@@ -2583,9 +2600,8 @@ static enum ref_transaction_error lock_ref_for_update(struct files_ref_store *re\n \tif (lock) {\n \t\tlock->count++;\n \t} else {\n-\t\tret = lock_raw_ref(refs, update, update_idx, mustexist,\n-\t\t\t\t refnames_to_check, &transaction->refnames,\n-\t\t\t\t &lock, &referent, err);\n+\t\tret = lock_raw_ref(refs, transaction, update_idx, mustexist,\n+\t\t\t\t refnames_to_check, &lock, &referent, err);\n \t\tif (ret) {\n \t\t\tchar *reason;\n \ndiff --git a/t/t1400-update-ref.sh b/t/t1400-update-ref.sh\nindex 96648a6e5d..08d5df2af7 100755\n--- a/t/t1400-update-ref.sh\n+++ b/t/t1400-update-ref.sh\n@@ -2294,6 +2294,59 @@ do\n \t\t)\n \t'\n \n+\ttest_expect_success CASE_INSENSITIVE_FS,REFFILES \"stdin $type batch-updates existing reference\" '\n+\t\tgit init repo &&\n+\t\ttest_when_finished \"rm -fr repo\" &&\n+\t\t(\n+\t\t\tcd repo &&\n+\t\t\ttest_commit one &&\n+\t\t\told_head=$(git rev-parse HEAD) &&\n+\t\t\ttest_commit two &&\n+\t\t\thead=$(git rev-parse HEAD) &&\n+\n+\t\t\t{\n+\t\t\t\tformat_command $type \"create refs/heads/foo\" \"$head\" &&\n+\t\t\t\tformat_command $type \"create refs/heads/ref\" \"$old_head\" &&\n+\t\t\t\tformat_command $type \"create refs/heads/Foo\" \"$old_head\"\n+\t\t\t} >stdin &&\n+\t\t\tgit update-ref $type --stdin --batch-updates stdout &&\n+\n+\t\t\techo $head >expect &&\n+\t\t\tgit rev-parse refs/heads/foo >actual &&\n+\t\t\techo $old_head >expect &&\n+\t\t\tgit rev-parse refs/heads/ref >actual &&\n+\t\t\ttest_cmp expect actual &&\n+\t\t\ttest_grep -q \"reference conflict due to case-insensitive filesystem\" stdout\n+\t\t)\n+\t'\n+\n+\ttest_expect_success CASE_INSENSITIVE_FS \"stdin $type batch-updates existing reference\" '\n+\t\tgit init --ref-format=reftable repo &&\n+\t\ttest_when_finished \"rm -fr repo\" &&\n+\t\t(\n+\t\t\tcd repo &&\n+\t\t\ttest_commit one &&\n+\t\t\told_head=$(git rev-parse HEAD) &&\n+\t\t\ttest_commit two &&\n+\t\t\thead=$(git rev-parse HEAD) &&\n+\n+\t\t\t{\n+\t\t\t\tformat_command $type \"create refs/heads/foo\" \"$head\" &&\n+\t\t\t\tformat_command $type \"create refs/heads/ref\" \"$old_head\" &&\n+\t\t\t\tformat_command $type \"create refs/heads/Foo\" \"$old_head\"\n+\t\t\t} >stdin &&\n+\t\t\tgit update-ref $type --stdin --batch-updates stdout &&\n+\n+\t\t\techo $head >expect &&\n+\t\t\tgit rev-parse refs/heads/foo >actual &&\n+\t\t\techo $old_head >expect &&\n+\t\t\tgit rev-parse refs/heads/ref >actual &&\n+\t\t\ttest_cmp expect actual &&\n+\t\t\tgit rev-parse refs/heads/Foo >actual &&\n+\t\t\ttest_cmp expect actual\n+\t\t)\n+\t'\n+\n \ttest_expect_success \"stdin $type batch-updates delete incorrect symbolic ref\" '\n \t\tgit init repo &&\n \t\ttest_when_finished \"rm -fr repo\" &&\ndiff --git a/t/t5510-fetch.sh b/t/t5510-fetch.sh\nindex ebc696546b..57f60da81b 100755\n--- a/t/t5510-fetch.sh\n+++ b/t/t5510-fetch.sh\n@@ -47,7 +47,13 @@ test_expect_success \"clone and setup child repos\" '\n \t\tgit config set branch.main.merge refs/heads/one\n \t) &&\n \tgit clone . bundle &&\n-\tgit clone . seven\n+\tgit clone . seven &&\n+\tgit clone --ref-format=reftable . case_sensitive &&\n+\t(\n+\t\tcd case_sensitive &&\n+\t\tgit branch branch1 &&\n+\t\tgit branch bRanch1\n+\t)\n '\n \n test_expect_success \"fetch test\" '\n@@ -1526,6 +1532,20 @@ test_expect_success SYMLINKS 'clone does not get confused by a D/F conflict' '\n \ttest_path_is_missing whoops\n '\n \n+test_expect_success CASE_INSENSITIVE_FS,REFFILES 'existing references in a case insensitive filesystem' '\n+\ttest_when_finished rm -rf case_insensitive &&\n+\t(\n+\t\tgit init --bare case_insensitive &&\n+\t\tcd case_insensitive &&\n+\t\tgit remote add origin -- ../case_sensitive &&\n+\t\ttest_must_fail git fetch -f origin \"refs/heads/*:refs/heads/*\" 2>err &&\n+\t\ttest_grep \"You${SQ}re on a case-insensitive filesystem\" err &&\n+\t\tgit rev-parse refs/heads/main >expect &&\n+\t\tgit rev-parse refs/heads/branch1 >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n . \"$TEST_DIRECTORY\"/lib-httpd.sh\n start_httpd\n \n\n-- \n2.50.1\n\n"},{"id":"525796","messageId":"20250908-587-git-fetch-1-fails-fetches-on-case-insensitive-repositories-v2-2-b2eb2459befb@gmail.com","threadId":"64069","inReplyTo":"20250908-587-git-fetch-1-fails-fetches-on-case-insensitive-repositories-v2-0-b2eb2459befb@gmail.com","subject":"[PATCH v2 2/4] refs/files: use correct error type when lock exists","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-09-08T12:37:36Z","receivedAt":"2025-09-08T12:37:42Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"When fetching references into a repository, if a lock for a particular\nreference exists, then `lock_raw_ref()` throws the generic error\n'REF_TRANSACTION_ERROR_GENERIC'. This causes the entire set of batched\nupdates to fail.\n\nInstead, return a 'REF_TRANSACTION_ERROR_CREATE_EXISTS' error. This\nallows batched updates to reject the individual update which conflicts\nwith the existing file, while updating the rest of the references.\n\nSigned-off-by: Karthik Nayak \n---\n refs/files-backend.c | 10 +++++++---\n t/t5510-fetch.sh | 26 ++++++++++++++++++++++++++\n 2 files changed, 33 insertions(+), 3 deletions(-)\n\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 58005d2732..2730713d23 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -790,9 +790,13 @@ static enum ref_transaction_error lock_raw_ref(struct files_ref_store *refs,\n \t\t\tgoto retry;\n \t\t} else {\n \t\t\tunable_to_lock_message(ref_file.buf, myerr, err);\n-\t\t\tif (myerr == EEXIST && ignore_case &&\n-\t\t\t duplicate_reference_case_cmp(transaction, update))\n-\t\t\t\tret = REF_TRANSACTION_ERROR_CASE_CONFLICT;\n+\t\t\tif (myerr == EEXIST) {\n+\t\t\t\tif (ignore_case && duplicate_reference_case_cmp(transaction, update))\n+\t\t\t\t\tret = REF_TRANSACTION_ERROR_CASE_CONFLICT;\n+\t\t\t\telse\n+\t\t\t\t\tret = REF_TRANSACTION_ERROR_CREATE_EXISTS;\n+\t\t\t}\n+\n \t\t\tgoto error_return;\n \t\t}\n \t}\ndiff --git a/t/t5510-fetch.sh b/t/t5510-fetch.sh\nindex 57f60da81b..6f8db0ace4 100755\n--- a/t/t5510-fetch.sh\n+++ b/t/t5510-fetch.sh\n@@ -1546,6 +1546,32 @@ test_expect_success CASE_INSENSITIVE_FS,REFFILES 'existing references in a case\n \t)\n '\n \n+test_expect_success REFFILES 'existing reference lock in repo' '\n+\ttest_when_finished rm -rf base repo &&\n+\t(\n+\t\tgit init --ref-format=reftable base &&\n+\t\tcd base &&\n+\t\techo >file update &&\n+\t\tgit add . &&\n+\t\tgit commit -m \"updated\" &&\n+\t\tgit branch -M main &&\n+\n+\t\tgit update-ref refs/heads/foo @ &&\n+\t\tgit update-ref refs/heads/branch @ &&\n+\t\tcd .. &&\n+\n+\t\tgit init --ref-format=files --bare repo &&\n+\t\tcd repo &&\n+\t\tgit remote add origin ../base &&\n+\t\ttouch refs/heads/foo.lock &&\n+\t\ttest_must_fail git fetch -f origin \"refs/heads/*:refs/heads/*\" 2>err &&\n+\t\ttest_grep \"error: fetching ref refs/heads/foo failed: reference already exists\" err &&\n+\t\tgit rev-parse refs/heads/main >expect &&\n+\t\tgit rev-parse refs/heads/branch >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n . \"$TEST_DIRECTORY\"/lib-httpd.sh\n start_httpd\n \n\n-- \n2.50.1\n\n"},{"id":"525797","messageId":"20250908-587-git-fetch-1-fails-fetches-on-case-insensitive-repositories-v2-3-b2eb2459befb@gmail.com","threadId":"64069","inReplyTo":"20250908-587-git-fetch-1-fails-fetches-on-case-insensitive-repositories-v2-0-b2eb2459befb@gmail.com","subject":"[PATCH v2 3/4] refs/files: handle F/D conflicts in case-insensitive FS","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-09-08T12:37:37Z","receivedAt":"2025-09-08T12:37:43Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"When using the files-backend on case-insensitive filesystems, there is\npossibility of hitting F/D conflicts when creating references within a\nsingle transaction, such as:\n\n - 'refs/heads/foo'\n - 'refs/heads/Foo/bar'\n\nIdeally such conflicts are caught in `refs_verify_refnames_available()`\nwhich is responsible for checking F/D conflicts within a given\ntransaction. This utility function is shared across the reference\nbackends. As such, it doesn't consider the issues of using a\ncase-insensitive file system, which only affects the files-backend.\n\nWhile one solution would be to make the function aware of such issues,\nthis feels like leaking implementation details of file-backend specific\nissues into the utility function. So opt for the more simpler option, of\nlowercasing all references sent to this function when on a\ncase-insensitive filesystem and operating on the files-backend.\n\nTo do this, simply use a `struct strbuf` to convert the refname to a\nlower case and append it to the list of refnames to be checked. Since we\nuse a `struct strbuf` and the memory is cleared right after, make sure\nthat the string list duplicates all provided string.\n\nWithout this change, the user would simply be left with a repository\nwith '.lock' files which were created in the 'prepare' phase of the\ntransaction, as the 'commit' phase would simply abort and not do the\nnecessary cleanup.\n\nReported-by: Junio C Hamano \nSigned-off-by: Karthik Nayak \n---\n refs/files-backend.c | 19 +++++++++++++++++--\n t/t5510-fetch.sh | 20 ++++++++++++++++++++\n 2 files changed, 37 insertions(+), 2 deletions(-)\n\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 2730713d23..85f2e14e93 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -888,8 +888,23 @@ static enum ref_transaction_error lock_raw_ref(struct files_ref_store *refs,\n \t\t * If the ref did not exist and we are creating it, we have to\n \t\t * make sure there is no existing packed ref that conflicts\n \t\t * with refname. This check is deferred so that we can batch it.\n+\t\t *\n+\t\t * For case-insensitive filesystems, we should also check for F/D\n+\t\t * conflicts between 'foo' and 'Foo/bar'. So let's lowercase\n+\t\t * the refname.\n \t\t */\n-\t\titem = string_list_append(refnames_to_check, refname);\n+\t\tif (ignore_case) {\n+\t\t\tstruct strbuf lower = STRBUF_INIT;\n+\n+\t\t\tstrbuf_addstr(&lower, refname);\n+\t\t\tstrbuf_tolower(&lower);\n+\n+\t\t\titem = string_list_append_nodup(refnames_to_check,\n+\t\t\t\t\t\t\tstrbuf_detach(&lower, NULL));\n+\t\t} else {\n+\t\t\titem = string_list_append(refnames_to_check, refname);\n+\t\t}\n+\n \t\titem->util = xmalloc(sizeof(update_idx));\n \t\tmemcpy(item->util, &update_idx, sizeof(update_idx));\n \t}\n@@ -2814,7 +2829,7 @@ static int files_transaction_prepare(struct ref_store *ref_store,\n \t\t\t \"ref_transaction_prepare\");\n \tsize_t i;\n \tint ret = 0;\n-\tstruct string_list refnames_to_check = STRING_LIST_INIT_NODUP;\n+\tstruct string_list refnames_to_check = STRING_LIST_INIT_DUP;\n \tchar *head_ref = NULL;\n \tint head_type;\n \tstruct files_transaction_backend_data *backend_data;\ndiff --git a/t/t5510-fetch.sh b/t/t5510-fetch.sh\nindex 6f8db0ace4..08dbea6503 100755\n--- a/t/t5510-fetch.sh\n+++ b/t/t5510-fetch.sh\n@@ -53,6 +53,12 @@ test_expect_success \"clone and setup child repos\" '\n \t\tcd case_sensitive &&\n \t\tgit branch branch1 &&\n \t\tgit branch bRanch1\n+\t) &&\n+\tgit clone --ref-format=reftable . case_sensitive_fd &&\n+\t(\n+\t\tcd case_sensitive_fd &&\n+\t\tgit branch foo/bar &&\n+\t\tgit branch Foo\n \t)\n '\n \n@@ -1572,6 +1578,20 @@ test_expect_success REFFILES 'existing reference lock in repo' '\n \t)\n '\n \n+test_expect_success CASE_INSENSITIVE_FS,REFFILES 'F/D conflict on case insensitive filesystem' '\n+\ttest_when_finished rm -rf case_insensitive &&\n+\t(\n+\t\tgit init --bare case_insensitive &&\n+\t\tcd case_insensitive &&\n+\t\tgit remote add origin -- ../case_sensitive_fd &&\n+\t\ttest_must_fail git fetch -f origin \"refs/heads/*:refs/heads/*\" 2>err &&\n+\t\ttest_grep \"failed: refname conflict\" err &&\n+\t\tgit rev-parse refs/heads/main >expect &&\n+\t\tgit rev-parse refs/heads/foo/bar >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n . \"$TEST_DIRECTORY\"/lib-httpd.sh\n start_httpd\n \n\n-- \n2.50.1\n\n"},{"id":"525798","messageId":"20250908-587-git-fetch-1-fails-fetches-on-case-insensitive-repositories-v2-4-b2eb2459befb@gmail.com","threadId":"64069","inReplyTo":"20250908-587-git-fetch-1-fails-fetches-on-case-insensitive-repositories-v2-0-b2eb2459befb@gmail.com","subject":"[PATCH v2 4/4] refs/files: handle D/F conflicts during locking","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-09-08T12:37:38Z","receivedAt":"2025-09-08T12:37:43Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"The previous commit, added the necessary validation and checks for F/D\nconflicts in the files backend when working on case insensitive systems.\n\nThere is still a possibility for D/F conflicts. This is a different from\nthe F/D since for F/D conflicts, there would not be a conflict during\nthe lock creation phase:\n\n refs/heads/foo.lock\n refs/heads/foo/bar.lock\n\nHowever there would be a conflict when the locks are committed, since we\ncannot have 'refs/heads/foo/bar' and 'refs/heads/foo'. These kinds of\nconflicts are checked and resolved in\n`refs_verify_refnames_available()`, so the previous commit ensured that\nfor case-insensitive filesystems, we would lowercase the inputs to that\nfunction.\n\nFor D/F conflicts, there is a conflict during the lock creation phase\nitself:\n\n refs/heads/foo/bar.lock\n refs/heads/foo.lock\n\nAs in `lock_raw_ref()` after creating the lock, we also check for D/F\nconflicts. To fix this, simply categorize the error as a name conflict.\nAlso remove this reference from the list of valid refnames for\navailability checks.\n\nBy categorizing the error and removing it from the list of valid\nreferences, batched updates now knows to reject such reference updates\nand apply the other reference updates.\n\nFix a small typo in `ref_transaction_maybe_set_rejected()` while here.\n\nHelped-by: Junio C Hamano \nSigned-off-by: Karthik Nayak \n---\n refs.c | 8 +++++++-\n refs/files-backend.c | 2 +-\n t/t5510-fetch.sh | 20 ++++++++++++++++++++\n 3 files changed, 28 insertions(+), 2 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex 4c1c339ed9..ec4f0e9502 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -1223,7 +1223,7 @@ int ref_transaction_maybe_set_rejected(struct ref_transaction *transaction,\n \t\treturn 0;\n \n \tif (!transaction->rejections)\n-\t\tBUG(\"transaction not inititalized with failure support\");\n+\t\tBUG(\"transaction not initialized with failure support\");\n \n \t/*\n \t * Don't accept generic errors, since these errors are not user\n@@ -1232,6 +1232,12 @@ int ref_transaction_maybe_set_rejected(struct ref_transaction *transaction,\n \tif (err == REF_TRANSACTION_ERROR_GENERIC)\n \t\treturn 0;\n \n+\t/*\n+\t * Remove this refname from the list of refnames used for validation\n+\t */\n+\tstring_list_remove(&transaction->refnames,\n+\t\t\t transaction->updates[update_idx]->refname, 0);\n+\n \ttransaction->updates[update_idx]->rejection_err = err;\n \tALLOC_GROW(transaction->rejections->update_indices,\n \t\t transaction->rejections->nr + 1,\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 85f2e14e93..ceeec272ff 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -852,6 +852,7 @@ static enum ref_transaction_error lock_raw_ref(struct files_ref_store *refs,\n \t\t\t\tgoto error_return;\n \t\t\t} else if (remove_dir_recursively(&ref_file,\n \t\t\t\t\t\t\t REMOVE_DIR_EMPTY_ONLY)) {\n+\t\t\t\tret = REF_TRANSACTION_ERROR_NAME_CONFLICT;\n \t\t\t\tif (refs_verify_refname_available(\n \t\t\t\t\t\t &refs->base, refname,\n \t\t\t\t\t\t extras, NULL, 0, err)) {\n@@ -859,7 +860,6 @@ static enum ref_transaction_error lock_raw_ref(struct files_ref_store *refs,\n \t\t\t\t\t * The error message set by\n \t\t\t\t\t * verify_refname_available() is OK.\n \t\t\t\t\t */\n-\t\t\t\t\tret = REF_TRANSACTION_ERROR_NAME_CONFLICT;\n \t\t\t\t\tgoto error_return;\n \t\t\t\t} else {\n \t\t\t\t\t/*\ndiff --git a/t/t5510-fetch.sh b/t/t5510-fetch.sh\nindex 08dbea6503..ffaa956cde 100755\n--- a/t/t5510-fetch.sh\n+++ b/t/t5510-fetch.sh\n@@ -59,6 +59,12 @@ test_expect_success \"clone and setup child repos\" '\n \t\tcd case_sensitive_fd &&\n \t\tgit branch foo/bar &&\n \t\tgit branch Foo\n+\t) &&\n+\tgit clone --ref-format=reftable . case_sensitive_df &&\n+\t(\n+\t\tcd case_sensitive_df &&\n+\t\tgit branch Foo/bar &&\n+\t\tgit branch foo\n \t)\n '\n \n@@ -1592,6 +1598,20 @@ test_expect_success CASE_INSENSITIVE_FS,REFFILES 'F/D conflict on case insensiti\n \t)\n '\n \n+test_expect_success CASE_INSENSITIVE_FS,REFFILES 'D/F conflict on case insensitive filesystem' '\n+\ttest_when_finished rm -rf case_insensitive &&\n+\t(\n+\t\tgit init --bare case_insensitive &&\n+\t\tcd case_insensitive &&\n+\t\tgit remote add origin -- ../case_sensitive_df &&\n+\t\ttest_must_fail git fetch -f origin \"refs/heads/*:refs/heads/*\" 2>err &&\n+\t\ttest_grep \"failed: refname conflict\" err &&\n+\t\tgit rev-parse refs/heads/main >expect &&\n+\t\tgit rev-parse refs/heads/Foo/bar >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n . \"$TEST_DIRECTORY\"/lib-httpd.sh\n start_httpd\n \n\n-- \n2.50.1\n\n"},{"id":"525909","messageId":"aL_SwghVaAXL-yeX@pks.im","threadId":"64069","inReplyTo":"20250908-587-git-fetch-1-fails-fetches-on-case-insensitive-repositories-v2-1-b2eb2459befb@gmail.com","subject":"Re: [PATCH v2 1/4] refs/files: catch conflicts on case-insensitive file-systems","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-09-09T07:09:54Z","receivedAt":"2025-09-09T07:10:02Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, Sep 08, 2025 at 02:37:35PM +0200, Karthik Nayak wrote:\n> diff --git a/refs.h b/refs.h\n> index eedbb599c5..41915086b3 100644\n> --- a/refs.h\n> +++ b/refs.h\n> @@ -31,6 +31,8 @@ enum ref_transaction_error {\n> \tREF_TRANSACTION_ERROR_INVALID_NEW_VALUE = -6,\n> \t/* Expected ref to be symref, but is a regular ref */\n> \tREF_TRANSACTION_ERROR_EXPECTED_SYMREF = -7,\n> +\t/* Cannot create ref due to case-insensitive filesystem */\n> +\tREF_TRANSACTION_ERROR_CASE_CONFLICT = -8,\n\nNice that we now have a specific error code for this error case. It\nremoves some of the guesswork we previously had to do.\n\n> diff --git a/refs/files-backend.c b/refs/files-backend.c\n> index 088b52c740..58005d2732 100644\n> --- a/refs/files-backend.c\n> +++ b/refs/files-backend.c\n> @@ -647,6 +647,19 @@ static void unlock_ref(struct ref_lock *lock)\n> \t}\n> }\n> \n> +static bool duplicate_reference_case_cmp(struct ref_transaction *transaction,\n> +\t\t\t\t\t struct ref_update *update)\n\nI think the name could use some improvement. How about\n`transaction_has_case_conflicting_update()`?\n\n> +{\n> +\tfor (size_t i = 0; i < transaction->nr; i++) {\n> +\t\tif (transaction->updates[i] == update)\n> +\t\t\tbreak;\n\nWhy do we break here? Shouldn't we continue?\n\n> @@ -776,6 +790,9 @@ static enum ref_transaction_error lock_raw_ref(struct files_ref_store *refs,\n> \t\t\tgoto retry;\n> \t\t} else {\n> \t\t\tunable_to_lock_message(ref_file.buf, myerr, err);\n> +\t\t\tif (myerr == EEXIST && ignore_case &&\n> +\t\t\t duplicate_reference_case_cmp(transaction, update))\n> +\t\t\t\tret = REF_TRANSACTION_ERROR_CASE_CONFLICT;\n> \t\t\tgoto error_return;\n> \t\t}\n> \t}\n\nOkay. If we cannot lock the reference we now try to detect whether this\nis because of a case conflict. That only catches the case though where\nwe have a case conflict in the same transaction, right? How about the\ncase where there's preexisting refs on disk that cause a conflict?\n\nPatrick\n"},{"id":"525910","messageId":"aL_SydrNjXdQrtFC@pks.im","threadId":"64069","inReplyTo":"20250908-587-git-fetch-1-fails-fetches-on-case-insensitive-repositories-v2-2-b2eb2459befb@gmail.com","subject":"Re: [PATCH v2 2/4] refs/files: use correct error type when lock exists","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-09-09T07:10:01Z","receivedAt":"2025-09-09T07:10:06Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, Sep 08, 2025 at 02:37:36PM +0200, Karthik Nayak wrote:\n> When fetching references into a repository, if a lock for a particular\n> reference exists, then `lock_raw_ref()` throws the generic error\n> 'REF_TRANSACTION_ERROR_GENERIC'. This causes the entire set of batched\n> updates to fail.\n\nThis isn't quite accurate anymore as we may also raise\n`REF_TRANSACTION_ERROR_CASE_CONFLICT` now.\n\n> diff --git a/refs/files-backend.c b/refs/files-backend.c\n> index 58005d2732..2730713d23 100644\n> --- a/refs/files-backend.c\n> +++ b/refs/files-backend.c\n> @@ -790,9 +790,13 @@ static enum ref_transaction_error lock_raw_ref(struct files_ref_store *refs,\n> \t\t\tgoto retry;\n> \t\t} else {\n> \t\t\tunable_to_lock_message(ref_file.buf, myerr, err);\n> -\t\t\tif (myerr == EEXIST && ignore_case &&\n> -\t\t\t duplicate_reference_case_cmp(transaction, update))\n> -\t\t\t\tret = REF_TRANSACTION_ERROR_CASE_CONFLICT;\n> +\t\t\tif (myerr == EEXIST) {\n> +\t\t\t\tif (ignore_case && duplicate_reference_case_cmp(transaction, update))\n> +\t\t\t\t\tret = REF_TRANSACTION_ERROR_CASE_CONFLICT;\n> +\t\t\t\telse\n> +\t\t\t\t\tret = REF_TRANSACTION_ERROR_CREATE_EXISTS;\n> +\t\t\t}\n> +\n> \t\t\tgoto error_return;\n> \t\t}\n> \t}\n\nHm. So if I understand correctly, we now return CREATE_EXISTS in case we\nhave a conflict with a preexisting case-conflicting reference, but we\nreturn CASE_CONFLICT in case we have a conflict with an update in the\nsame transaction?\n\nIt feels awkward, but I guess that's the best thing we can do. We\nhappily overwrite case-conflicting preexisting refs, so we wouldn't even\nsee EEXIST in that case. So the only case where we still see that error\nis on a D/F conflict, and in that case it makes sense to return\nCREATE_EXISTS.\n\nI feel like this should be added as a comment though, as it gives some\nimportant non-obvious context.\n\nPatrick\n"},{"id":"525911","messageId":"aL_SznrF61hbUMBo@pks.im","threadId":"64069","inReplyTo":"20250908-587-git-fetch-1-fails-fetches-on-case-insensitive-repositories-v2-4-b2eb2459befb@gmail.com","subject":"Re: [PATCH v2 4/4] refs/files: handle D/F conflicts during locking","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-09-09T07:10:06Z","receivedAt":"2025-09-09T07:10:13Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, Sep 08, 2025 at 02:37:38PM +0200, Karthik Nayak wrote:\n> The previous commit, added the necessary validation and checks for F/D\n\ns/commit,/commit/\n\n> diff --git a/refs.c b/refs.c\n> index 4c1c339ed9..ec4f0e9502 100644\n> --- a/refs.c\n> +++ b/refs.c\n> @@ -1232,6 +1232,12 @@ int ref_transaction_maybe_set_rejected(struct ref_transaction *transaction,\n> \tif (err == REF_TRANSACTION_ERROR_GENERIC)\n> \t\treturn 0;\n> \n> +\t/*\n> +\t * Remove this refname from the list of refnames used for validation\n> +\t */\n\nNit: it's obvious that we remove the refname from that list, so the\ncomment is not helping much. It's much more important to explain _why_\nwe do that though to give readers the necessary context.\n\n> +\tstring_list_remove(&transaction->refnames,\n> +\t\t\t transaction->updates[update_idx]->refname, 0);\n> +\n> \ttransaction->updates[update_idx]->rejection_err = err;\n> \tALLOC_GROW(transaction->rejections->update_indices,\n> \t\t transaction->rejections->nr + 1,\n> diff --git a/refs/files-backend.c b/refs/files-backend.c\n> index 85f2e14e93..ceeec272ff 100644\n> --- a/refs/files-backend.c\n> +++ b/refs/files-backend.c\n> @@ -852,6 +852,7 @@ static enum ref_transaction_error lock_raw_ref(struct files_ref_store *refs,\n> \t\t\t\tgoto error_return;\n> \t\t\t} else if (remove_dir_recursively(&ref_file,\n> \t\t\t\t\t\t\t REMOVE_DIR_EMPTY_ONLY)) {\n> +\t\t\t\tret = REF_TRANSACTION_ERROR_NAME_CONFLICT;\n> \t\t\t\tif (refs_verify_refname_available(\n> \t\t\t\t\t\t &refs->base, refname,\n> \t\t\t\t\t\t extras, NULL, 0, err)) {\n> @@ -859,7 +860,6 @@ static enum ref_transaction_error lock_raw_ref(struct files_ref_store *refs,\n> \t\t\t\t\t * The error message set by\n> \t\t\t\t\t * verify_refname_available() is OK.\n> \t\t\t\t\t */\n> -\t\t\t\t\tret = REF_TRANSACTION_ERROR_NAME_CONFLICT;\n> \t\t\t\t\tgoto error_return;\n> \t\t\t\t} else {\n> \t\t\t\t\t/*\n\nHm, interesting. Previously we'd have returned a generic error in the\n`else` branch, which reads like this:\n\n\t} else {\n\t\t/*\n\t\t * We can't delete the directory,\n\t\t * but we also don't know of any\n\t\t * references that it should\n\t\t * contain.\n\t\t */\n\t\tstrbuf_addf(err, \"there is a non-empty directory '%s' \"\n\t\t\t \"blocking reference '%s'\",\n\t\t\t ref_file.buf, refname);\n\t\tgoto error_return;\n\t}\n\nSo that directory contains something, even though we've previously\nverified that it shouldn't, if I understand correctly. The test case you\nadd does seem to indicate that there are valid cases though where this\ncan happen on a case insensitive filesystem?\n\nIf so, the comment definitely needs to be updated to explain this\nadditional error case.\n\nOne more question is whether it's the correct thing to unconditionally\nreturn REF_TRANSACTION_ERROR_NAME_CONFLICT in that case now. Could it be\nthat the directory exists but contains some garbage?\n\nPatrick\n"},{"id":"526117","messageId":"CAOLa=ZRDY6ORDngq0Np0Y_8SFXRWWKAmmsecQfpqeqrLCfbN0g@mail.gmail.com","threadId":"64069","inReplyTo":"aL_SwghVaAXL-yeX@pks.im","subject":"Re: [PATCH v2 1/4] refs/files: catch conflicts on case-insensitive file-systems","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-09-11T09:35:52Z","receivedAt":"2025-09-11T09:35:54Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Patrick Steinhardt writes:\n\n> On Mon, Sep 08, 2025 at 02:37:35PM +0200, Karthik Nayak wrote:\n>> diff --git a/refs.h b/refs.h\n>> index eedbb599c5..41915086b3 100644\n>> --- a/refs.h\n>> +++ b/refs.h\n>> @@ -31,6 +31,8 @@ enum ref_transaction_error {\n>> \tREF_TRANSACTION_ERROR_INVALID_NEW_VALUE = -6,\n>> \t/* Expected ref to be symref, but is a regular ref */\n>> \tREF_TRANSACTION_ERROR_EXPECTED_SYMREF = -7,\n>> +\t/* Cannot create ref due to case-insensitive filesystem */\n>> +\tREF_TRANSACTION_ERROR_CASE_CONFLICT = -8,\n>\n> Nice that we now have a specific error code for this error case. It\n> removes some of the guesswork we previously had to do.\n>\n\nYeah Agreed. I dislike adding a common error which only affects a single\nreference backend, but this is nicer.\n\n>> diff --git a/refs/files-backend.c b/refs/files-backend.c\n>> index 088b52c740..58005d2732 100644\n>> --- a/refs/files-backend.c\n>> +++ b/refs/files-backend.c\n>> @@ -647,6 +647,19 @@ static void unlock_ref(struct ref_lock *lock)\n>> \t}\n>> }\n>>\n>> +static bool duplicate_reference_case_cmp(struct ref_transaction *transaction,\n>> +\t\t\t\t\t struct ref_update *update)\n>\n> I think the name could use some improvement. How about\n> `transaction_has_case_conflicting_update()`?\n>\n\nThat does read better, will change.\n\n>> +{\n>> +\tfor (size_t i = 0; i < transaction->nr; i++) {\n>> +\t\tif (transaction->updates[i] == update)\n>> +\t\t\tbreak;\n>\n> Why do we break here? Shouldn't we continue?\n>\n\nWe break because we only care about matching updates up to the index of\nthe provided updates. Further updates should recall the function.\n\nI'll add a comment to explain this.\n\n>> @@ -776,6 +790,9 @@ static enum ref_transaction_error lock_raw_ref(struct files_ref_store *refs,\n>> \t\t\tgoto retry;\n>> \t\t} else {\n>> \t\t\tunable_to_lock_message(ref_file.buf, myerr, err);\n>> +\t\t\tif (myerr == EEXIST && ignore_case &&\n>> +\t\t\t duplicate_reference_case_cmp(transaction, update))\n>> +\t\t\t\tret = REF_TRANSACTION_ERROR_CASE_CONFLICT;\n>> \t\t\tgoto error_return;\n>> \t\t}\n>> \t}\n>\n> Okay. If we cannot lock the reference we now try to detect whether this\n> is because of a case conflict. That only catches the case though where\n> we have a case conflict in the same transaction, right? How about the\n> case where there's preexisting refs on disk that cause a conflict?\n>\n\nExisting references aren't an issue since in those situations we can\ncreate the lock file. The issue here arises because within the same\ntransaction we try to create the lock file twice which causes the\nconflict.\n\n> Patrick\n"},{"id":"526118","messageId":"CAOLa=ZTf=Es5MJvdC=zTUWoJnASF=Mqf9+RXrnHOYDZ9YuFEtg@mail.gmail.com","threadId":"64069","inReplyTo":"aL_SydrNjXdQrtFC@pks.im","subject":"Re: [PATCH v2 2/4] refs/files: use correct error type when lock exists","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-09-11T10:14:20Z","receivedAt":"2025-09-11T10:14:22Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Patrick Steinhardt writes:\n\n> On Mon, Sep 08, 2025 at 02:37:36PM +0200, Karthik Nayak wrote:\n>> When fetching references into a repository, if a lock for a particular\n>> reference exists, then `lock_raw_ref()` throws the generic error\n>> 'REF_TRANSACTION_ERROR_GENERIC'. This causes the entire set of batched\n>> updates to fail.\n>\n> This isn't quite accurate anymore as we may also raise\n> `REF_TRANSACTION_ERROR_CASE_CONFLICT` now.\n>\n\nGood catch, will fixup.\n\n>> diff --git a/refs/files-backend.c b/refs/files-backend.c\n>> index 58005d2732..2730713d23 100644\n>> --- a/refs/files-backend.c\n>> +++ b/refs/files-backend.c\n>> @@ -790,9 +790,13 @@ static enum ref_transaction_error lock_raw_ref(struct files_ref_store *refs,\n>> \t\t\tgoto retry;\n>> \t\t} else {\n>> \t\t\tunable_to_lock_message(ref_file.buf, myerr, err);\n>> -\t\t\tif (myerr == EEXIST && ignore_case &&\n>> -\t\t\t duplicate_reference_case_cmp(transaction, update))\n>> -\t\t\t\tret = REF_TRANSACTION_ERROR_CASE_CONFLICT;\n>> +\t\t\tif (myerr == EEXIST) {\n>> +\t\t\t\tif (ignore_case && duplicate_reference_case_cmp(transaction, update))\n>> +\t\t\t\t\tret = REF_TRANSACTION_ERROR_CASE_CONFLICT;\n>> +\t\t\t\telse\n>> +\t\t\t\t\tret = REF_TRANSACTION_ERROR_CREATE_EXISTS;\n>> +\t\t\t}\n>> +\n>> \t\t\tgoto error_return;\n>> \t\t}\n>> \t}\n>\n> Hm. So if I understand correctly, we now return CREATE_EXISTS in case we\n> have a conflict with a preexisting case-conflicting reference, but we\n> return CASE_CONFLICT in case we have a conflict with an update in the\n> same transaction?\n>\n\nIt's not a reference, rather the lock that causes the conflict. So a\npreexisting case-conflicting reference lock would raise the GENERIC\nerror. Which would fail all updates.\n\n> It feels awkward, but I guess that's the best thing we can do. We\n> happily overwrite case-conflicting preexisting refs, so we wouldn't even\n> see EEXIST in that case. So the only case where we still see that error\n> is on a D/F conflict, and in that case it makes sense to return\n> CREATE_EXISTS.\n>\n> I feel like this should be added as a comment though, as it gives some\n> important non-obvious context.\n>\n\nYeah, it would be beneficial, will add some comments here.\n\n> Patrick\n"},{"id":"526176","messageId":"CAOLa=ZRt=HY_MrKHxGrC1g1QjOjngSYENzMwNDgUpi1gbg5yEA@mail.gmail.com","threadId":"64069","inReplyTo":"aL_SznrF61hbUMBo@pks.im","subject":"Re: [PATCH v2 4/4] refs/files: handle D/F conflicts during locking","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-09-12T11:53:04Z","receivedAt":"2025-09-12T11:53:06Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Patrick Steinhardt writes:\n\n> On Mon, Sep 08, 2025 at 02:37:38PM +0200, Karthik Nayak wrote:\n>> The previous commit, added the necessary validation and checks for F/D\n>\n> s/commit,/commit/\n>\n\nThanks, will change.\n\n>> diff --git a/refs.c b/refs.c\n>> index 4c1c339ed9..ec4f0e9502 100644\n>> --- a/refs.c\n>> +++ b/refs.c\n>> @@ -1232,6 +1232,12 @@ int ref_transaction_maybe_set_rejected(struct ref_transaction *transaction,\n>> \tif (err == REF_TRANSACTION_ERROR_GENERIC)\n>> \t\treturn 0;\n>>\n>> +\t/*\n>> +\t * Remove this refname from the list of refnames used for validation\n>> +\t */\n>\n> Nit: it's obvious that we remove the refname from that list, so the\n> comment is not helping much. It's much more important to explain _why_\n> we do that though to give readers the necessary context.\n>\n\nIndeed, I'll add this instead\n\n Rejected refnames shouldn't be considered in the availability checks,\n so remove them from the list.\n\n>> +\tstring_list_remove(&transaction->refnames,\n>> +\t\t\t transaction->updates[update_idx]->refname, 0);\n>> +\n>> \ttransaction->updates[update_idx]->rejection_err = err;\n>> \tALLOC_GROW(transaction->rejections->update_indices,\n>> \t\t transaction->rejections->nr + 1,\n>> diff --git a/refs/files-backend.c b/refs/files-backend.c\n>> index 85f2e14e93..ceeec272ff 100644\n>> --- a/refs/files-backend.c\n>> +++ b/refs/files-backend.c\n>> @@ -852,6 +852,7 @@ static enum ref_transaction_error lock_raw_ref(struct files_ref_store *refs,\n>> \t\t\t\tgoto error_return;\n>> \t\t\t} else if (remove_dir_recursively(&ref_file,\n>> \t\t\t\t\t\t\t REMOVE_DIR_EMPTY_ONLY)) {\n>> +\t\t\t\tret = REF_TRANSACTION_ERROR_NAME_CONFLICT;\n>> \t\t\t\tif (refs_verify_refname_available(\n>> \t\t\t\t\t\t &refs->base, refname,\n>> \t\t\t\t\t\t extras, NULL, 0, err)) {\n>> @@ -859,7 +860,6 @@ static enum ref_transaction_error lock_raw_ref(struct files_ref_store *refs,\n>> \t\t\t\t\t * The error message set by\n>> \t\t\t\t\t * verify_refname_available() is OK.\n>> \t\t\t\t\t */\n>> -\t\t\t\t\tret = REF_TRANSACTION_ERROR_NAME_CONFLICT;\n>> \t\t\t\t\tgoto error_return;\n>> \t\t\t\t} else {\n>> \t\t\t\t\t/*\n>\n> Hm, interesting. Previously we'd have returned a generic error in the\n> `else` branch, which reads like this:\n>\n> \t} else {\n> \t\t/*\n> \t\t * We can't delete the directory,\n> \t\t * but we also don't know of any\n> \t\t * references that it should\n> \t\t * contain.\n> \t\t */\n> \t\tstrbuf_addf(err, \"there is a non-empty directory '%s' \"\n> \t\t\t \"blocking reference '%s'\",\n> \t\t\t ref_file.buf, refname);\n> \t\tgoto error_return;\n> \t}\n>\n> So that directory contains something, even though we've previously\n> verified that it shouldn't, if I understand correctly. The test case you\n> add does seem to indicate that there are valid cases though where this\n> can happen on a case insensitive filesystem?\n>\n> If so, the comment definitely needs to be updated to explain this\n> additional error case.\n>\n\nYeah I think that comment needs to be rewritten.\n\n> One more question is whether it's the correct thing to unconditionally\n> return REF_TRANSACTION_ERROR_NAME_CONFLICT in that case now. Could it be\n> that the directory exists but contains some garbage?\n>\n> Patrick\n\nThis does affect case-sensitive systems too. I should've added a test\nthere as well to showcase this.\n\nBasically if there is a lock file in a directory\n'refs/heads/foo/bar.lock' while fetching 'refs/heads/foo', this would\nreach this section of the code. Pre batched-updates, we would skip this\nref and update other refs. So it makes sense to retain this behavior.\nI'll add in a test and amend the commit message to reflect this\nbehavior.\n\n- Karthik\n"},{"id":"526257","messageId":"20250913-587-git-fetch-1-fails-fetches-on-case-insensitive-repositories-v3-0-195569740b57@gmail.com","threadId":"64069","inReplyTo":"20250902-587-git-fetch-1-fails-fetches-on-case-insensitive-repositories-v1-0-35e69bbb507d@gmail.com","subject":"[PATCH v3 0/4] refs/files: fix issues with git-fetch on case-insensitive FS","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-09-13T20:54:28Z","receivedAt":"2025-09-13T20:54:35Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Hello!\n\nWith Git 2.51 we moved 'git-fetch(1)' and 'git-receive-pack(1)' to use\nbatched updates while doing reference updates. This provided a nice perf\nboost since both commands will now use a single transaction for\nreference updates. This removes the overhead of using individual\ntransaction per reference update and also avoids unnecessary\nauto-compaction between reference updates in the reftable backend.\n\nHowever, in the files-backend it does introduce a few bugs around\nconflicts. The reported bug was around case-insensitive filesystems [1],\nbut we also fix some adjacent issues:\n\n1. When fetching references such as:\n\n - refs/heads/foo\n - refs/heads/Foo\n\nEarlier we would simply overwrite the first reference with the second\nand continue. Since Git 2.51 we simply abort stating a conflict.\n\nThis is resolved in the first commit by explicitly categorizing the\nerror as non-GENERIC. This allows batched updates to reject the\nparticular update, while updating the rest.\n\n2. When fetching references and a lock for a particular reference\nalready exits. We treat this is a GENERIC error, which fails the entire\nupdate. By categorizing this error as non-GENERIC, we can reject this\nspecific update and update the other references.\n\n3. When fetching references such as with F/D conflict:\n\n - refs/heads/foo\n - refs/heads/Foo/bar\n\nEarlier we would apply the first, while the second would fail due to\nconflict. Since Git 2.51, the lock files for both would be created, but\nthe 'commit' phase would abruptly end leaving the lock files.\n\nThe second commit fixes this by ensuring that on case-insensitive\nfilesystems we lowercase the refnames for availability check to ensure\nF/D are caught and reported to the user.\n\n4. When fetching references with D/F conflict:\n\n - refs/heads/Foo/bar\n - refs/heads/foo\n\nThe creation of the second reference's lock in `lock_raw_ref()` catches\nthe D/F conflict, but we mark this as a GENERIC error. By categorizing\nthis as non-GENERIC, we can allow the updates to continue while\nrejecting this specific error.\n\nThis also applies to D/F conflicts in case-sensitive systems where there\nexists a lock in the repo 'refs/heads/foo/bar.lock' causing a conflict\nwhile fetching 'refs/heads/foo'.\n\n- Karthik\n\n[1]: https://lore.kernel.org/all/YQXPR01MB3046197EF39296549EE6DD669A33A@YQXPR01MB3046.CANPRD01.PROD.OUTLOOK.COM/\n\nSigned-off-by: Karthik Nayak \n---\nChanges in v3:\n- Rename duplicate_reference_case_cmp() to\n transaction_has_case_conflicting_update() and add comments.\n- Improve commit messages.\n- Add an additional test in the 4th commit to showcase D/F conflicts in\n case-sensistive file systems.\n- Link to v2: https://lore.kernel.org/r/20250908-587-git-fetch-1-fails-fetches-on-case-insensitive-repositories-v2-0-b2eb2459befb@gmail.com\n\nChanges in v2:\n- This version fixes two more issues:\n - Fetching while locks already exist in the repository\n - D/F conflicts while fetching\n- Add a specific error to the first case, so we can nicely show a\n relevant error. Also check explicitly that the issue is due to\n case-insensitive filesystems.\n- Cleanup the commit messages.\n- Use `string_list_append_nodup()` with `strbuf_detach`, reducing the\n number of allocations.\n\n---\n builtin/fetch.c | 21 ++++++++--\n refs.c | 11 ++++-\n refs.h | 2 +\n refs/files-backend.c | 77 ++++++++++++++++++++++++++++------\n t/t1400-update-ref.sh | 53 +++++++++++++++++++++++\n t/t5510-fetch.sh | 114 +++++++++++++++++++++++++++++++++++++++++++++++++-\n 6 files changed, 261 insertions(+), 17 deletions(-)\n\nKarthik Nayak (4):\n refs/files: catch conflicts on case-insensitive file-systems\n refs/files: use correct error type when lock exists\n refs/files: handle F/D conflicts in case-insensitive FS\n refs/files: handle D/F conflicts during locking\n\nRange-diff versus v2:\n\n1: 570b24fc09 ! 1: 1d4ff64d90 refs/files: catch conflicts on case-insensitive file-systems\n @@ refs/files-backend.c: static void unlock_ref(struct ref_lock *lock)\n \t}\n }\n \n -+static bool duplicate_reference_case_cmp(struct ref_transaction *transaction,\n -+\t\t\t\t\t struct ref_update *update)\n ++/*\n ++ * Check if the transaction has another update with a case-insensitive refname\n ++ * match.\n ++ *\n ++ * If the update is part of the transaction, we only check up to that index.\n ++ * Further updates are expected to call this function to match previous indices.\n ++ */\n ++static bool transaction_has_case_conflicting_update(struct ref_transaction *transaction,\n ++\t\t\t\t\t\t struct ref_update *update)\n +{\n +\tfor (size_t i = 0; i < transaction->nr; i++) {\n +\t\tif (transaction->updates[i] == update)\n @@ refs/files-backend.c: static enum ref_transaction_error lock_raw_ref(struct file\n \t\t} else {\n \t\t\tunable_to_lock_message(ref_file.buf, myerr, err);\n +\t\t\tif (myerr == EEXIST && ignore_case &&\n -+\t\t\t duplicate_reference_case_cmp(transaction, update))\n ++\t\t\t transaction_has_case_conflicting_update(transaction, update))\n +\t\t\t\tret = REF_TRANSACTION_ERROR_CASE_CONFLICT;\n \t\t\tgoto error_return;\n \t\t}\n2: 10bfb8c1f2 ! 2: fdc69b233e refs/files: use correct error type when lock exists\n @@ Commit message\n refs/files: use correct error type when lock exists\n \n When fetching references into a repository, if a lock for a particular\n - reference exists, then `lock_raw_ref()` throws the generic error\n - 'REF_TRANSACTION_ERROR_GENERIC'. This causes the entire set of batched\n - updates to fail.\n + reference exists, then `lock_raw_ref()` throws:\n +\n + - REF_TRANSACTION_ERROR_CASE_CONFLICT: when there is a conflict\n + because transaction contains conflicting references while being on a\n + case-insensitive filesystem.\n +\n + - REF_TRANSACTION_ERROR_GENERIC: for all other errors.\n +\n + The latter causes the entire set of batched updates to fail, even in\n + case sensitive filessystems.\n \n Instead, return a 'REF_TRANSACTION_ERROR_CREATE_EXISTS' error. This\n allows batched updates to reject the individual update which conflicts\n @@ refs/files-backend.c: static enum ref_transaction_error lock_raw_ref(struct file\n \t\t} else {\n \t\t\tunable_to_lock_message(ref_file.buf, myerr, err);\n -\t\t\tif (myerr == EEXIST && ignore_case &&\n --\t\t\t duplicate_reference_case_cmp(transaction, update))\n +-\t\t\t transaction_has_case_conflicting_update(transaction, update))\n -\t\t\t\tret = REF_TRANSACTION_ERROR_CASE_CONFLICT;\n +\t\t\tif (myerr == EEXIST) {\n -+\t\t\t\tif (ignore_case && duplicate_reference_case_cmp(transaction, update))\n ++\t\t\t\tif (ignore_case &&\n ++\t\t\t\t transaction_has_case_conflicting_update(transaction, update))\n ++\t\t\t\t\t/*\n ++\t\t\t\t\t * In case-insensitive filesystems, ensure that conflicts within a\n ++\t\t\t\t\t * given transaction are handled. Pre-existing refs on a\n ++\t\t\t\t\t * case-insensitive system will be overridden without any issue.\n ++\t\t\t\t\t */\n +\t\t\t\t\tret = REF_TRANSACTION_ERROR_CASE_CONFLICT;\n +\t\t\t\telse\n ++\t\t\t\t\t/*\n ++\t\t\t\t\t * Pre-existing case-conflicting reference locks should also be\n ++\t\t\t\t\t * specially categorized to avoid failing all batched updates.\n ++\t\t\t\t\t */\n +\t\t\t\t\tret = REF_TRANSACTION_ERROR_CREATE_EXISTS;\n +\t\t\t}\n +\n3: 23868f2218 = 3: ff7fb95b66 refs/files: handle F/D conflicts in case-insensitive FS\n4: 5af3dc2a27 ! 4: a610f9478b refs/files: handle D/F conflicts during locking\n @@ Metadata\n ## Commit message ##\n refs/files: handle D/F conflicts during locking\n \n - The previous commit, added the necessary validation and checks for F/D\n + The previous commit added the necessary validation and checks for F/D\n conflicts in the files backend when working on case insensitive systems.\n \n There is still a possibility for D/F conflicts. This is a different from\n @@ Commit message\n refs/heads/foo.lock\n \n As in `lock_raw_ref()` after creating the lock, we also check for D/F\n - conflicts. To fix this, simply categorize the error as a name conflict.\n - Also remove this reference from the list of valid refnames for\n - availability checks.\n + conflicts. This can occur in case-insensitive filesystems when trying to\n + fetch case-conflicted references like:\n \n + refs/heads/Foo/new\n + refs/heads/foo\n +\n + D/F conflicts can also occur in case-sensitive filesystems, when the\n + repository already contains a directory with a lock file\n + 'refs/heads/foo/bar.lock' and trying to fetch 'refs/heads/foo'. This\n + doesn't concern directories containing garbage files as those are\n + handled on a higher level.\n +\n + To fix this, simply categorize the error as a name conflict. Also remove\n + this reference from the list of valid refnames for availability checks.\n By categorizing the error and removing it from the list of valid\n references, batched updates now knows to reject such reference updates\n and apply the other reference updates.\n @@ refs.c: int ref_transaction_maybe_set_rejected(struct ref_transaction *transacti\n \t\treturn 0;\n \n +\t/*\n -+\t * Remove this refname from the list of refnames used for validation\n ++\t * Rejected refnames shouldn't be considered in the availability\n ++\t * checks, so remove them from the list.\n +\t */\n +\tstring_list_remove(&transaction->refnames,\n +\t\t\t transaction->updates[update_idx]->refname, 0);\n @@ refs/files-backend.c: static enum ref_transaction_error lock_raw_ref(struct file\n \t\t\t\t\tgoto error_return;\n \t\t\t\t} else {\n \t\t\t\t\t/*\n +-\t\t\t\t\t * We can't delete the directory,\n +-\t\t\t\t\t * but we also don't know of any\n +-\t\t\t\t\t * references that it should\n +-\t\t\t\t\t * contain.\n ++\t\t\t\t\t * Directory conflicts can occur if there\n ++\t\t\t\t\t * is an existing lock file in the directory\n ++\t\t\t\t\t * or if the filesystem is case-insensitive\n ++\t\t\t\t\t * and the directory contains a valid reference\n ++\t\t\t\t\t * but conflicts with the update.\n + \t\t\t\t\t */\n + \t\t\t\t\tstrbuf_addf(err, \"there is a non-empty directory '%s' \"\n + \t\t\t\t\t\t \"blocking reference '%s'\",\n \n ## t/t5510-fetch.sh ##\n @@ t/t5510-fetch.sh: test_expect_success \"clone and setup child repos\" '\n @@ t/t5510-fetch.sh: test_expect_success CASE_INSENSITIVE_FS,REFFILES 'F/D conflict\n +\t\ttest_cmp expect actual\n +\t)\n +'\n ++\n ++test_expect_success REFFILES 'D/F conflict on case sensitive filesystem with lock' '\n ++\t(\n ++\t\tgit init --ref-format=reftable base &&\n ++\t\tcd base &&\n ++\t\techo >file update &&\n ++\t\tgit add . &&\n ++\t\tgit commit -m \"updated\" &&\n ++\t\tgit branch -M main &&\n ++\n ++\t\tgit update-ref refs/heads/foo @ &&\n ++\t\tgit update-ref refs/heads/branch @ &&\n ++\t\tcd .. &&\n ++\n ++\t\tgit init --ref-format=files --bare repo &&\n ++\t\tcd repo &&\n ++\t\tgit remote add origin ../base &&\n ++\t\tmkdir refs/heads/foo &&\n ++\t\ttouch refs/heads/foo/random.lock &&\n ++\t\ttest_must_fail git fetch origin \"refs/heads/*:refs/heads/*\" 2>err &&\n ++\t\ttest_grep \"some local refs could not be updated; try running\" err &&\n ++\t\tgit rev-parse refs/heads/main >expect &&\n ++\t\tgit rev-parse refs/heads/branch >actual &&\n ++\t\ttest_cmp expect actual\n ++\t)\n ++'\n +\n . \"$TEST_DIRECTORY\"/lib-httpd.sh\n start_httpd\n\n\nbase-commit: c44beea485f0f2feaf460e2ac87fdd5608d63cf0\nchange-id: 20250822-587-git-fetch-1-fails-fetches-on-case-insensitive-repositories-0a187c7ac41e\n\nThanks\n- Karthik\n\n"},{"id":"526258","messageId":"20250913-587-git-fetch-1-fails-fetches-on-case-insensitive-repositories-v3-1-195569740b57@gmail.com","threadId":"64069","inReplyTo":"20250913-587-git-fetch-1-fails-fetches-on-case-insensitive-repositories-v3-0-195569740b57@gmail.com","subject":"[PATCH v3 1/4] refs/files: catch conflicts on case-insensitive file-systems","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-09-13T20:54:29Z","receivedAt":"2025-09-13T20:54:36Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"During the 'prepare' phase of reference transaction in the files\nbackend, we create the lock files for references to be created. When\nusing batched updates on case-insensitive filesystems, the entire\nbatched updates would be aborted if there are conflicting names such as:\n\n refs/heads/Foo\n refs/heads/foo\n\nThis affects all commands which were migrated to use batched updates in\nGit 2.51, including 'git-fetch(1)' and 'git-receive-pack(1)'. Before\nthat, reference updates would be applied serially with one transaction\nused per update. When users fetched multiple references on\ncase-insensitive systems, subsequent references would simply overwrite\nany earlier references. So when fetching:\n\n refs/heads/foo: 5f34ec0bfeac225b1c854340257a65b106f70ea6\n refs/heads/Foo: ec3053b0977e83d9b67fc32c4527a117953994f3\n refs/heads/sample: 2eefd1150e06d8fca1ddfa684dec016f36bf4e56\n\nThe user would simply end up with:\n\n refs/heads/foo: ec3053b0977e83d9b67fc32c4527a117953994f3\n refs/heads/sample: 2eefd1150e06d8fca1ddfa684dec016f36bf4e56\n\nThis is buggy behavior since the user is never informed about the\noverrides performed and missing references. Nevertheless, the user is\nleft with a working repository with a subset of the references. Since\nGit 2.51, in such situations fetches would simply fail without updating\nany references. Which is also buggy behavior and worse off since the\nuser is left without any references.\n\nThe error is triggered in `lock_raw_ref()` where the files backend\nattempts to create a lock file. When a lock file already exists the\nfunction returns a 'REF_TRANSACTION_ERROR_GENERIC'. When this happens,\nthe entire batched updates, not individual operation, is aborted as if\nit were in a transaction.\n\nChange this to return 'REF_TRANSACTION_ERROR_CASE_CONFLICT' instead to\naid the batched update mechanism to simply reject such errors. The\nchange only affects batched updates since batched updates will reject\nindividual updates with non-generic errors. So specifically this would\nonly affect:\n\n 1. git fetch\n 2. git receive-pack\n 3. git update-ref --batch-updates\n\nThis bubbles the error type up to `files_transaction_prepare()` which\ntries to lock each reference update. So if the locking fails, we check\nif the rejection type can be ignored, which is done by calling\n`ref_transaction_maybe_set_rejected()`.\n\nAs the error type is now 'REF_TRANSACTION_ERROR_CASE_CONFLICT',\nthe specific reference update would simply be rejected, while other\nupdates in the transaction would continue to be applied. This allows\npartial application of references in case-insensitive filesystems when\nfetching colliding references.\n\nWhile the earlier implementation allowed the last reference to be\napplied overriding the initial references, this change would allow the\nfirst reference to be applied while rejecting consequent collisions.\nThis should be an okay compromise since with the files backend, there is\nno scenario possible where we would retain all colliding references.\n\nLet's also be more pro-active and notify users on case-insensitive\nfilesystems about such problems by providing a brief about the issue\nwhile also recommending using the reftable backend, which doesn't have\nthe same issue.\n\nReported-by: Joe Drew \nHelped-by: Patrick Steinhardt \nSigned-off-by: Karthik Nayak \n---\n builtin/fetch.c | 21 +++++++++++++++++---\n refs.c | 2 ++\n refs.h | 2 ++\n refs/files-backend.c | 33 +++++++++++++++++++++++++++-----\n t/t1400-update-ref.sh | 53 +++++++++++++++++++++++++++++++++++++++++++++++++++\n t/t5510-fetch.sh | 22 ++++++++++++++++++++-\n 6 files changed, 124 insertions(+), 9 deletions(-)\n\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex 24645c4653..c7ff3480fb 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -1643,7 +1643,8 @@ static int set_head(const struct ref *remote_refs, struct remote *remote)\n \n struct ref_rejection_data {\n \tint *retcode;\n-\tint conflict_msg_shown;\n+\tbool conflict_msg_shown;\n+\tbool case_sensitive_msg_shown;\n \tconst char *remote_name;\n };\n \n@@ -1657,11 +1658,25 @@ static void ref_transaction_rejection_handler(const char *refname,\n {\n \tstruct ref_rejection_data *data = cb_data;\n \n-\tif (err == REF_TRANSACTION_ERROR_NAME_CONFLICT && !data->conflict_msg_shown) {\n+\tif (err == REF_TRANSACTION_ERROR_CASE_CONFLICT && ignore_case &&\n+\t !data->case_sensitive_msg_shown) {\n+\t\terror(_(\"You're on a case-insensitive filesystem, and the remote you are\\n\"\n+\t\t\t\"trying to fetch from has references that only differ in casing. It\\n\"\n+\t\t\t\"is impossible to store such references with the 'files' backend. You\\n\"\n+\t\t\t\"can either accept this as-is, in which case you won't be able to\\n\"\n+\t\t\t\"store all remote references on disk. Or you can alternatively\\n\"\n+\t\t\t\"migrate your repository to use the 'reftable' backend with the\\n\"\n+\t\t\t\"following command:\\n\\n git refs migrate --ref-format=reftable\\n\\n\"\n+\t\t\t\"Please keep in mind that not all implementations of Git support this\\n\"\n+\t\t\t\"new format yet. So if you use tools other than Git to access this\\n\"\n+\t\t\t\"repository it may not be an option to migrate to reftables.\\n\"));\n+\t\tdata->case_sensitive_msg_shown = true;\n+\t} else if (err == REF_TRANSACTION_ERROR_NAME_CONFLICT &&\n+\t\t !data->conflict_msg_shown) {\n \t\terror(_(\"some local refs could not be updated; try running\\n\"\n \t\t\t\" 'git remote prune %s' to remove any old, conflicting \"\n \t\t\t\"branches\"), data->remote_name);\n-\t\tdata->conflict_msg_shown = 1;\n+\t\tdata->conflict_msg_shown = true;\n \t} else {\n \t\tconst char *reason = ref_transaction_error_msg(err);\n \ndiff --git a/refs.c b/refs.c\nindex bfdbe718b7..4c1c339ed9 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -3321,6 +3321,8 @@ const char *ref_transaction_error_msg(enum ref_transaction_error err)\n \t\treturn \"invalid new value provided\";\n \tcase REF_TRANSACTION_ERROR_EXPECTED_SYMREF:\n \t\treturn \"expected symref but found regular ref\";\n+\tcase REF_TRANSACTION_ERROR_CASE_CONFLICT:\n+\t\treturn \"reference conflict due to case-insensitive filesystem\";\n \tdefault:\n \t\treturn \"unknown failure\";\n \t}\ndiff --git a/refs.h b/refs.h\nindex eedbb599c5..41915086b3 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -31,6 +31,8 @@ enum ref_transaction_error {\n \tREF_TRANSACTION_ERROR_INVALID_NEW_VALUE = -6,\n \t/* Expected ref to be symref, but is a regular ref */\n \tREF_TRANSACTION_ERROR_EXPECTED_SYMREF = -7,\n+\t/* Cannot create ref due to case-insensitive filesystem */\n+\tREF_TRANSACTION_ERROR_CASE_CONFLICT = -8,\n };\n \n /*\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 088b52c740..01df32904b 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -647,6 +647,26 @@ static void unlock_ref(struct ref_lock *lock)\n \t}\n }\n \n+/*\n+ * Check if the transaction has another update with a case-insensitive refname\n+ * match.\n+ *\n+ * If the update is part of the transaction, we only check up to that index.\n+ * Further updates are expected to call this function to match previous indices.\n+ */\n+static bool transaction_has_case_conflicting_update(struct ref_transaction *transaction,\n+\t\t\t\t\t\t struct ref_update *update)\n+{\n+\tfor (size_t i = 0; i < transaction->nr; i++) {\n+\t\tif (transaction->updates[i] == update)\n+\t\t\tbreak;\n+\n+\t\tif (!strcasecmp(transaction->updates[i]->refname, update->refname))\n+\t\t\treturn true;\n+\t}\n+\treturn false;\n+}\n+\n /*\n * Lock refname, without following symrefs, and set *lock_p to point\n * at a newly-allocated lock object. Fill in lock->old_oid, referent,\n@@ -677,16 +697,17 @@ static void unlock_ref(struct ref_lock *lock)\n * - Generate informative error messages in the case of failure\n */\n static enum ref_transaction_error lock_raw_ref(struct files_ref_store *refs,\n-\t\t\t\t\t struct ref_update *update,\n+\t\t\t\t\t struct ref_transaction *transaction,\n \t\t\t\t\t size_t update_idx,\n \t\t\t\t\t int mustexist,\n \t\t\t\t\t struct string_list *refnames_to_check,\n-\t\t\t\t\t const struct string_list *extras,\n \t\t\t\t\t struct ref_lock **lock_p,\n \t\t\t\t\t struct strbuf *referent,\n \t\t\t\t\t struct strbuf *err)\n {\n \tenum ref_transaction_error ret = REF_TRANSACTION_ERROR_GENERIC;\n+\tstruct ref_update *update = transaction->updates[update_idx];\n+\tconst struct string_list *extras = &transaction->refnames;\n \tconst char *refname = update->refname;\n \tunsigned int *type = &update->type;\n \tstruct ref_lock *lock;\n@@ -776,6 +797,9 @@ static enum ref_transaction_error lock_raw_ref(struct files_ref_store *refs,\n \t\t\tgoto retry;\n \t\t} else {\n \t\t\tunable_to_lock_message(ref_file.buf, myerr, err);\n+\t\t\tif (myerr == EEXIST && ignore_case &&\n+\t\t\t transaction_has_case_conflicting_update(transaction, update))\n+\t\t\t\tret = REF_TRANSACTION_ERROR_CASE_CONFLICT;\n \t\t\tgoto error_return;\n \t\t}\n \t}\n@@ -2583,9 +2607,8 @@ static enum ref_transaction_error lock_ref_for_update(struct files_ref_store *re\n \tif (lock) {\n \t\tlock->count++;\n \t} else {\n-\t\tret = lock_raw_ref(refs, update, update_idx, mustexist,\n-\t\t\t\t refnames_to_check, &transaction->refnames,\n-\t\t\t\t &lock, &referent, err);\n+\t\tret = lock_raw_ref(refs, transaction, update_idx, mustexist,\n+\t\t\t\t refnames_to_check, &lock, &referent, err);\n \t\tif (ret) {\n \t\t\tchar *reason;\n \ndiff --git a/t/t1400-update-ref.sh b/t/t1400-update-ref.sh\nindex 96648a6e5d..08d5df2af7 100755\n--- a/t/t1400-update-ref.sh\n+++ b/t/t1400-update-ref.sh\n@@ -2294,6 +2294,59 @@ do\n \t\t)\n \t'\n \n+\ttest_expect_success CASE_INSENSITIVE_FS,REFFILES \"stdin $type batch-updates existing reference\" '\n+\t\tgit init repo &&\n+\t\ttest_when_finished \"rm -fr repo\" &&\n+\t\t(\n+\t\t\tcd repo &&\n+\t\t\ttest_commit one &&\n+\t\t\told_head=$(git rev-parse HEAD) &&\n+\t\t\ttest_commit two &&\n+\t\t\thead=$(git rev-parse HEAD) &&\n+\n+\t\t\t{\n+\t\t\t\tformat_command $type \"create refs/heads/foo\" \"$head\" &&\n+\t\t\t\tformat_command $type \"create refs/heads/ref\" \"$old_head\" &&\n+\t\t\t\tformat_command $type \"create refs/heads/Foo\" \"$old_head\"\n+\t\t\t} >stdin &&\n+\t\t\tgit update-ref $type --stdin --batch-updates stdout &&\n+\n+\t\t\techo $head >expect &&\n+\t\t\tgit rev-parse refs/heads/foo >actual &&\n+\t\t\techo $old_head >expect &&\n+\t\t\tgit rev-parse refs/heads/ref >actual &&\n+\t\t\ttest_cmp expect actual &&\n+\t\t\ttest_grep -q \"reference conflict due to case-insensitive filesystem\" stdout\n+\t\t)\n+\t'\n+\n+\ttest_expect_success CASE_INSENSITIVE_FS \"stdin $type batch-updates existing reference\" '\n+\t\tgit init --ref-format=reftable repo &&\n+\t\ttest_when_finished \"rm -fr repo\" &&\n+\t\t(\n+\t\t\tcd repo &&\n+\t\t\ttest_commit one &&\n+\t\t\told_head=$(git rev-parse HEAD) &&\n+\t\t\ttest_commit two &&\n+\t\t\thead=$(git rev-parse HEAD) &&\n+\n+\t\t\t{\n+\t\t\t\tformat_command $type \"create refs/heads/foo\" \"$head\" &&\n+\t\t\t\tformat_command $type \"create refs/heads/ref\" \"$old_head\" &&\n+\t\t\t\tformat_command $type \"create refs/heads/Foo\" \"$old_head\"\n+\t\t\t} >stdin &&\n+\t\t\tgit update-ref $type --stdin --batch-updates stdout &&\n+\n+\t\t\techo $head >expect &&\n+\t\t\tgit rev-parse refs/heads/foo >actual &&\n+\t\t\techo $old_head >expect &&\n+\t\t\tgit rev-parse refs/heads/ref >actual &&\n+\t\t\ttest_cmp expect actual &&\n+\t\t\tgit rev-parse refs/heads/Foo >actual &&\n+\t\t\ttest_cmp expect actual\n+\t\t)\n+\t'\n+\n \ttest_expect_success \"stdin $type batch-updates delete incorrect symbolic ref\" '\n \t\tgit init repo &&\n \t\ttest_when_finished \"rm -fr repo\" &&\ndiff --git a/t/t5510-fetch.sh b/t/t5510-fetch.sh\nindex ebc696546b..57f60da81b 100755\n--- a/t/t5510-fetch.sh\n+++ b/t/t5510-fetch.sh\n@@ -47,7 +47,13 @@ test_expect_success \"clone and setup child repos\" '\n \t\tgit config set branch.main.merge refs/heads/one\n \t) &&\n \tgit clone . bundle &&\n-\tgit clone . seven\n+\tgit clone . seven &&\n+\tgit clone --ref-format=reftable . case_sensitive &&\n+\t(\n+\t\tcd case_sensitive &&\n+\t\tgit branch branch1 &&\n+\t\tgit branch bRanch1\n+\t)\n '\n \n test_expect_success \"fetch test\" '\n@@ -1526,6 +1532,20 @@ test_expect_success SYMLINKS 'clone does not get confused by a D/F conflict' '\n \ttest_path_is_missing whoops\n '\n \n+test_expect_success CASE_INSENSITIVE_FS,REFFILES 'existing references in a case insensitive filesystem' '\n+\ttest_when_finished rm -rf case_insensitive &&\n+\t(\n+\t\tgit init --bare case_insensitive &&\n+\t\tcd case_insensitive &&\n+\t\tgit remote add origin -- ../case_sensitive &&\n+\t\ttest_must_fail git fetch -f origin \"refs/heads/*:refs/heads/*\" 2>err &&\n+\t\ttest_grep \"You${SQ}re on a case-insensitive filesystem\" err &&\n+\t\tgit rev-parse refs/heads/main >expect &&\n+\t\tgit rev-parse refs/heads/branch1 >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n . \"$TEST_DIRECTORY\"/lib-httpd.sh\n start_httpd\n \n\n-- \n2.50.1\n\n"},{"id":"526259","messageId":"20250913-587-git-fetch-1-fails-fetches-on-case-insensitive-repositories-v3-2-195569740b57@gmail.com","threadId":"64069","inReplyTo":"20250913-587-git-fetch-1-fails-fetches-on-case-insensitive-repositories-v3-0-195569740b57@gmail.com","subject":"[PATCH v3 2/4] refs/files: use correct error type when lock exists","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-09-13T20:54:30Z","receivedAt":"2025-09-13T20:54:37Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"When fetching references into a repository, if a lock for a particular\nreference exists, then `lock_raw_ref()` throws:\n\n - REF_TRANSACTION_ERROR_CASE_CONFLICT: when there is a conflict\n because transaction contains conflicting references while being on a\n case-insensitive filesystem.\n\n - REF_TRANSACTION_ERROR_GENERIC: for all other errors.\n\nThe latter causes the entire set of batched updates to fail, even in\ncase sensitive filessystems.\n\nInstead, return a 'REF_TRANSACTION_ERROR_CREATE_EXISTS' error. This\nallows batched updates to reject the individual update which conflicts\nwith the existing file, while updating the rest of the references.\n\nSigned-off-by: Karthik Nayak \n---\n refs/files-backend.c | 20 +++++++++++++++++---\n t/t5510-fetch.sh | 26 ++++++++++++++++++++++++++\n 2 files changed, 43 insertions(+), 3 deletions(-)\n\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 01df32904b..69e50a16db 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -797,9 +797,23 @@ static enum ref_transaction_error lock_raw_ref(struct files_ref_store *refs,\n \t\t\tgoto retry;\n \t\t} else {\n \t\t\tunable_to_lock_message(ref_file.buf, myerr, err);\n-\t\t\tif (myerr == EEXIST && ignore_case &&\n-\t\t\t transaction_has_case_conflicting_update(transaction, update))\n-\t\t\t\tret = REF_TRANSACTION_ERROR_CASE_CONFLICT;\n+\t\t\tif (myerr == EEXIST) {\n+\t\t\t\tif (ignore_case &&\n+\t\t\t\t transaction_has_case_conflicting_update(transaction, update))\n+\t\t\t\t\t/*\n+\t\t\t\t\t * In case-insensitive filesystems, ensure that conflicts within a\n+\t\t\t\t\t * given transaction are handled. Pre-existing refs on a\n+\t\t\t\t\t * case-insensitive system will be overridden without any issue.\n+\t\t\t\t\t */\n+\t\t\t\t\tret = REF_TRANSACTION_ERROR_CASE_CONFLICT;\n+\t\t\t\telse\n+\t\t\t\t\t/*\n+\t\t\t\t\t * Pre-existing case-conflicting reference locks should also be\n+\t\t\t\t\t * specially categorized to avoid failing all batched updates.\n+\t\t\t\t\t */\n+\t\t\t\t\tret = REF_TRANSACTION_ERROR_CREATE_EXISTS;\n+\t\t\t}\n+\n \t\t\tgoto error_return;\n \t\t}\n \t}\ndiff --git a/t/t5510-fetch.sh b/t/t5510-fetch.sh\nindex 57f60da81b..6f8db0ace4 100755\n--- a/t/t5510-fetch.sh\n+++ b/t/t5510-fetch.sh\n@@ -1546,6 +1546,32 @@ test_expect_success CASE_INSENSITIVE_FS,REFFILES 'existing references in a case\n \t)\n '\n \n+test_expect_success REFFILES 'existing reference lock in repo' '\n+\ttest_when_finished rm -rf base repo &&\n+\t(\n+\t\tgit init --ref-format=reftable base &&\n+\t\tcd base &&\n+\t\techo >file update &&\n+\t\tgit add . &&\n+\t\tgit commit -m \"updated\" &&\n+\t\tgit branch -M main &&\n+\n+\t\tgit update-ref refs/heads/foo @ &&\n+\t\tgit update-ref refs/heads/branch @ &&\n+\t\tcd .. &&\n+\n+\t\tgit init --ref-format=files --bare repo &&\n+\t\tcd repo &&\n+\t\tgit remote add origin ../base &&\n+\t\ttouch refs/heads/foo.lock &&\n+\t\ttest_must_fail git fetch -f origin \"refs/heads/*:refs/heads/*\" 2>err &&\n+\t\ttest_grep \"error: fetching ref refs/heads/foo failed: reference already exists\" err &&\n+\t\tgit rev-parse refs/heads/main >expect &&\n+\t\tgit rev-parse refs/heads/branch >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n . \"$TEST_DIRECTORY\"/lib-httpd.sh\n start_httpd\n \n\n-- \n2.50.1\n\n"},{"id":"526260","messageId":"20250913-587-git-fetch-1-fails-fetches-on-case-insensitive-repositories-v3-3-195569740b57@gmail.com","threadId":"64069","inReplyTo":"20250913-587-git-fetch-1-fails-fetches-on-case-insensitive-repositories-v3-0-195569740b57@gmail.com","subject":"[PATCH v3 3/4] refs/files: handle F/D conflicts in case-insensitive FS","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-09-13T20:54:31Z","receivedAt":"2025-09-13T20:54:38Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"When using the files-backend on case-insensitive filesystems, there is\npossibility of hitting F/D conflicts when creating references within a\nsingle transaction, such as:\n\n - 'refs/heads/foo'\n - 'refs/heads/Foo/bar'\n\nIdeally such conflicts are caught in `refs_verify_refnames_available()`\nwhich is responsible for checking F/D conflicts within a given\ntransaction. This utility function is shared across the reference\nbackends. As such, it doesn't consider the issues of using a\ncase-insensitive file system, which only affects the files-backend.\n\nWhile one solution would be to make the function aware of such issues,\nthis feels like leaking implementation details of file-backend specific\nissues into the utility function. So opt for the more simpler option, of\nlowercasing all references sent to this function when on a\ncase-insensitive filesystem and operating on the files-backend.\n\nTo do this, simply use a `struct strbuf` to convert the refname to a\nlower case and append it to the list of refnames to be checked. Since we\nuse a `struct strbuf` and the memory is cleared right after, make sure\nthat the string list duplicates all provided string.\n\nWithout this change, the user would simply be left with a repository\nwith '.lock' files which were created in the 'prepare' phase of the\ntransaction, as the 'commit' phase would simply abort and not do the\nnecessary cleanup.\n\nReported-by: Junio C Hamano \nSigned-off-by: Karthik Nayak \n---\n refs/files-backend.c | 19 +++++++++++++++++--\n t/t5510-fetch.sh | 20 ++++++++++++++++++++\n 2 files changed, 37 insertions(+), 2 deletions(-)\n\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 69e50a16db..817b56f4ce 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -905,8 +905,23 @@ static enum ref_transaction_error lock_raw_ref(struct files_ref_store *refs,\n \t\t * If the ref did not exist and we are creating it, we have to\n \t\t * make sure there is no existing packed ref that conflicts\n \t\t * with refname. This check is deferred so that we can batch it.\n+\t\t *\n+\t\t * For case-insensitive filesystems, we should also check for F/D\n+\t\t * conflicts between 'foo' and 'Foo/bar'. So let's lowercase\n+\t\t * the refname.\n \t\t */\n-\t\titem = string_list_append(refnames_to_check, refname);\n+\t\tif (ignore_case) {\n+\t\t\tstruct strbuf lower = STRBUF_INIT;\n+\n+\t\t\tstrbuf_addstr(&lower, refname);\n+\t\t\tstrbuf_tolower(&lower);\n+\n+\t\t\titem = string_list_append_nodup(refnames_to_check,\n+\t\t\t\t\t\t\tstrbuf_detach(&lower, NULL));\n+\t\t} else {\n+\t\t\titem = string_list_append(refnames_to_check, refname);\n+\t\t}\n+\n \t\titem->util = xmalloc(sizeof(update_idx));\n \t\tmemcpy(item->util, &update_idx, sizeof(update_idx));\n \t}\n@@ -2831,7 +2846,7 @@ static int files_transaction_prepare(struct ref_store *ref_store,\n \t\t\t \"ref_transaction_prepare\");\n \tsize_t i;\n \tint ret = 0;\n-\tstruct string_list refnames_to_check = STRING_LIST_INIT_NODUP;\n+\tstruct string_list refnames_to_check = STRING_LIST_INIT_DUP;\n \tchar *head_ref = NULL;\n \tint head_type;\n \tstruct files_transaction_backend_data *backend_data;\ndiff --git a/t/t5510-fetch.sh b/t/t5510-fetch.sh\nindex 6f8db0ace4..08dbea6503 100755\n--- a/t/t5510-fetch.sh\n+++ b/t/t5510-fetch.sh\n@@ -53,6 +53,12 @@ test_expect_success \"clone and setup child repos\" '\n \t\tcd case_sensitive &&\n \t\tgit branch branch1 &&\n \t\tgit branch bRanch1\n+\t) &&\n+\tgit clone --ref-format=reftable . case_sensitive_fd &&\n+\t(\n+\t\tcd case_sensitive_fd &&\n+\t\tgit branch foo/bar &&\n+\t\tgit branch Foo\n \t)\n '\n \n@@ -1572,6 +1578,20 @@ test_expect_success REFFILES 'existing reference lock in repo' '\n \t)\n '\n \n+test_expect_success CASE_INSENSITIVE_FS,REFFILES 'F/D conflict on case insensitive filesystem' '\n+\ttest_when_finished rm -rf case_insensitive &&\n+\t(\n+\t\tgit init --bare case_insensitive &&\n+\t\tcd case_insensitive &&\n+\t\tgit remote add origin -- ../case_sensitive_fd &&\n+\t\ttest_must_fail git fetch -f origin \"refs/heads/*:refs/heads/*\" 2>err &&\n+\t\ttest_grep \"failed: refname conflict\" err &&\n+\t\tgit rev-parse refs/heads/main >expect &&\n+\t\tgit rev-parse refs/heads/foo/bar >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n . \"$TEST_DIRECTORY\"/lib-httpd.sh\n start_httpd\n \n\n-- \n2.50.1\n\n"},{"id":"526261","messageId":"20250913-587-git-fetch-1-fails-fetches-on-case-insensitive-repositories-v3-4-195569740b57@gmail.com","threadId":"64069","inReplyTo":"20250913-587-git-fetch-1-fails-fetches-on-case-insensitive-repositories-v3-0-195569740b57@gmail.com","subject":"[PATCH v3 4/4] refs/files: handle D/F conflicts during locking","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-09-13T20:54:32Z","receivedAt":"2025-09-13T20:54:39Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"The previous commit added the necessary validation and checks for F/D\nconflicts in the files backend when working on case insensitive systems.\n\nThere is still a possibility for D/F conflicts. This is a different from\nthe F/D since for F/D conflicts, there would not be a conflict during\nthe lock creation phase:\n\n refs/heads/foo.lock\n refs/heads/foo/bar.lock\n\nHowever there would be a conflict when the locks are committed, since we\ncannot have 'refs/heads/foo/bar' and 'refs/heads/foo'. These kinds of\nconflicts are checked and resolved in\n`refs_verify_refnames_available()`, so the previous commit ensured that\nfor case-insensitive filesystems, we would lowercase the inputs to that\nfunction.\n\nFor D/F conflicts, there is a conflict during the lock creation phase\nitself:\n\n refs/heads/foo/bar.lock\n refs/heads/foo.lock\n\nAs in `lock_raw_ref()` after creating the lock, we also check for D/F\nconflicts. This can occur in case-insensitive filesystems when trying to\nfetch case-conflicted references like:\n\n refs/heads/Foo/new\n refs/heads/foo\n\nD/F conflicts can also occur in case-sensitive filesystems, when the\nrepository already contains a directory with a lock file\n'refs/heads/foo/bar.lock' and trying to fetch 'refs/heads/foo'. This\ndoesn't concern directories containing garbage files as those are\nhandled on a higher level.\n\nTo fix this, simply categorize the error as a name conflict. Also remove\nthis reference from the list of valid refnames for availability checks.\nBy categorizing the error and removing it from the list of valid\nreferences, batched updates now knows to reject such reference updates\nand apply the other reference updates.\n\nFix a small typo in `ref_transaction_maybe_set_rejected()` while here.\n\nHelped-by: Junio C Hamano \nSigned-off-by: Karthik Nayak \n---\n refs.c | 9 ++++++++-\n refs/files-backend.c | 11 ++++++-----\n t/t5510-fetch.sh | 46 ++++++++++++++++++++++++++++++++++++++++++++++\n 3 files changed, 60 insertions(+), 6 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex 4c1c339ed9..e7109ea5fe 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -1223,7 +1223,7 @@ int ref_transaction_maybe_set_rejected(struct ref_transaction *transaction,\n \t\treturn 0;\n \n \tif (!transaction->rejections)\n-\t\tBUG(\"transaction not inititalized with failure support\");\n+\t\tBUG(\"transaction not initialized with failure support\");\n \n \t/*\n \t * Don't accept generic errors, since these errors are not user\n@@ -1232,6 +1232,13 @@ int ref_transaction_maybe_set_rejected(struct ref_transaction *transaction,\n \tif (err == REF_TRANSACTION_ERROR_GENERIC)\n \t\treturn 0;\n \n+\t/*\n+\t * Rejected refnames shouldn't be considered in the availability\n+\t * checks, so remove them from the list.\n+\t */\n+\tstring_list_remove(&transaction->refnames,\n+\t\t\t transaction->updates[update_idx]->refname, 0);\n+\n \ttransaction->updates[update_idx]->rejection_err = err;\n \tALLOC_GROW(transaction->rejections->update_indices,\n \t\t transaction->rejections->nr + 1,\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 817b56f4ce..fec7713ea6 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -869,6 +869,7 @@ static enum ref_transaction_error lock_raw_ref(struct files_ref_store *refs,\n \t\t\t\tgoto error_return;\n \t\t\t} else if (remove_dir_recursively(&ref_file,\n \t\t\t\t\t\t\t REMOVE_DIR_EMPTY_ONLY)) {\n+\t\t\t\tret = REF_TRANSACTION_ERROR_NAME_CONFLICT;\n \t\t\t\tif (refs_verify_refname_available(\n \t\t\t\t\t\t &refs->base, refname,\n \t\t\t\t\t\t extras, NULL, 0, err)) {\n@@ -876,14 +877,14 @@ static enum ref_transaction_error lock_raw_ref(struct files_ref_store *refs,\n \t\t\t\t\t * The error message set by\n \t\t\t\t\t * verify_refname_available() is OK.\n \t\t\t\t\t */\n-\t\t\t\t\tret = REF_TRANSACTION_ERROR_NAME_CONFLICT;\n \t\t\t\t\tgoto error_return;\n \t\t\t\t} else {\n \t\t\t\t\t/*\n-\t\t\t\t\t * We can't delete the directory,\n-\t\t\t\t\t * but we also don't know of any\n-\t\t\t\t\t * references that it should\n-\t\t\t\t\t * contain.\n+\t\t\t\t\t * Directory conflicts can occur if there\n+\t\t\t\t\t * is an existing lock file in the directory\n+\t\t\t\t\t * or if the filesystem is case-insensitive\n+\t\t\t\t\t * and the directory contains a valid reference\n+\t\t\t\t\t * but conflicts with the update.\n \t\t\t\t\t */\n \t\t\t\t\tstrbuf_addf(err, \"there is a non-empty directory '%s' \"\n \t\t\t\t\t\t \"blocking reference '%s'\",\ndiff --git a/t/t5510-fetch.sh b/t/t5510-fetch.sh\nindex 08dbea6503..6b2739db26 100755\n--- a/t/t5510-fetch.sh\n+++ b/t/t5510-fetch.sh\n@@ -59,6 +59,12 @@ test_expect_success \"clone and setup child repos\" '\n \t\tcd case_sensitive_fd &&\n \t\tgit branch foo/bar &&\n \t\tgit branch Foo\n+\t) &&\n+\tgit clone --ref-format=reftable . case_sensitive_df &&\n+\t(\n+\t\tcd case_sensitive_df &&\n+\t\tgit branch Foo/bar &&\n+\t\tgit branch foo\n \t)\n '\n \n@@ -1592,6 +1598,46 @@ test_expect_success CASE_INSENSITIVE_FS,REFFILES 'F/D conflict on case insensiti\n \t)\n '\n \n+test_expect_success CASE_INSENSITIVE_FS,REFFILES 'D/F conflict on case insensitive filesystem' '\n+\ttest_when_finished rm -rf case_insensitive &&\n+\t(\n+\t\tgit init --bare case_insensitive &&\n+\t\tcd case_insensitive &&\n+\t\tgit remote add origin -- ../case_sensitive_df &&\n+\t\ttest_must_fail git fetch -f origin \"refs/heads/*:refs/heads/*\" 2>err &&\n+\t\ttest_grep \"failed: refname conflict\" err &&\n+\t\tgit rev-parse refs/heads/main >expect &&\n+\t\tgit rev-parse refs/heads/Foo/bar >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success REFFILES 'D/F conflict on case sensitive filesystem with lock' '\n+\t(\n+\t\tgit init --ref-format=reftable base &&\n+\t\tcd base &&\n+\t\techo >file update &&\n+\t\tgit add . &&\n+\t\tgit commit -m \"updated\" &&\n+\t\tgit branch -M main &&\n+\n+\t\tgit update-ref refs/heads/foo @ &&\n+\t\tgit update-ref refs/heads/branch @ &&\n+\t\tcd .. &&\n+\n+\t\tgit init --ref-format=files --bare repo &&\n+\t\tcd repo &&\n+\t\tgit remote add origin ../base &&\n+\t\tmkdir refs/heads/foo &&\n+\t\ttouch refs/heads/foo/random.lock &&\n+\t\ttest_must_fail git fetch origin \"refs/heads/*:refs/heads/*\" 2>err &&\n+\t\ttest_grep \"some local refs could not be updated; try running\" err &&\n+\t\tgit rev-parse refs/heads/main >expect &&\n+\t\tgit rev-parse refs/heads/branch >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n . \"$TEST_DIRECTORY\"/lib-httpd.sh\n start_httpd\n \n\n-- \n2.50.1\n\n"},{"id":"526427","messageId":"aMkW-HOAsU8YZ8Lp@pks.im","threadId":"64069","inReplyTo":"20250913-587-git-fetch-1-fails-fetches-on-case-insensitive-repositories-v3-2-195569740b57@gmail.com","subject":"Re: [PATCH v3 2/4] refs/files: use correct error type when lock exists","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-09-16T07:51:20Z","receivedAt":"2025-09-16T07:51:27Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Sat, Sep 13, 2025 at 10:54:30PM +0200, Karthik Nayak wrote:\n> diff --git a/refs/files-backend.c b/refs/files-backend.c\n> index 01df32904b..69e50a16db 100644\n> --- a/refs/files-backend.c\n> +++ b/refs/files-backend.c\n> @@ -797,9 +797,23 @@ static enum ref_transaction_error lock_raw_ref(struct files_ref_store *refs,\n> \t\t\tgoto retry;\n> \t\t} else {\n> \t\t\tunable_to_lock_message(ref_file.buf, myerr, err);\n> -\t\t\tif (myerr == EEXIST && ignore_case &&\n> -\t\t\t transaction_has_case_conflicting_update(transaction, update))\n> -\t\t\t\tret = REF_TRANSACTION_ERROR_CASE_CONFLICT;\n> +\t\t\tif (myerr == EEXIST) {\n> +\t\t\t\tif (ignore_case &&\n> +\t\t\t\t transaction_has_case_conflicting_update(transaction, update))\n> +\t\t\t\t\t/*\n> +\t\t\t\t\t * In case-insensitive filesystems, ensure that conflicts within a\n> +\t\t\t\t\t * given transaction are handled. Pre-existing refs on a\n> +\t\t\t\t\t * case-insensitive system will be overridden without any issue.\n> +\t\t\t\t\t */\n> +\t\t\t\t\tret = REF_TRANSACTION_ERROR_CASE_CONFLICT;\n> +\t\t\t\telse\n> +\t\t\t\t\t/*\n> +\t\t\t\t\t * Pre-existing case-conflicting reference locks should also be\n> +\t\t\t\t\t * specially categorized to avoid failing all batched updates.\n> +\t\t\t\t\t */\n> +\t\t\t\t\tret = REF_TRANSACTION_ERROR_CREATE_EXISTS;\n> +\t\t\t}\n\nTiniest nit: I think it would be preferable to use curly braces for such\nmulti-line comments. This nit isn't worth a reroll though.\n\nOther than that my feedback got addressed, so this looks good to me.\nThanks!\n\nPatrick\n"},{"id":"526502","messageId":"uv4ifkvxcujjjj3lsc4tf5b5dnc7pkaaw62t6ahagnskxn4kg6@oyxjgupdilrc","threadId":"64069","inReplyTo":"20250913-587-git-fetch-1-fails-fetches-on-case-insensitive-repositories-v3-1-195569740b57@gmail.com","subject":"Re: [PATCH v3 1/4] refs/files: catch conflicts on case-insensitive file-systems","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2025-09-16T21:13:52Z","receivedAt":"2025-09-16T21:13:54Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 25/09/13 10:54PM, Karthik Nayak wrote:\n> During the 'prepare' phase of reference transaction in the files\n\ns/reference/a reference/\n\n> backend, we create the lock files for references to be created. When\n> using batched updates on case-insensitive filesystems, the entire\n> batched updates would be aborted if there are conflicting names such as:\n> \n> refs/heads/Foo\n> refs/heads/foo\n\nOk so this is only a problem now because the reference updates are\nperformed in a single transaction and the resulting error causes the\nentire transaction to be aborted.\n\n> This affects all commands which were migrated to use batched updates in\n> Git 2.51, including 'git-fetch(1)' and 'git-receive-pack(1)'. Before\n> that, reference updates would be applied serially with one transaction\n> used per update. When users fetched multiple references on\n> case-insensitive systems, subsequent references would simply overwrite\n> any earlier references. So when fetching:\n> \n> refs/heads/foo: 5f34ec0bfeac225b1c854340257a65b106f70ea6\n> refs/heads/Foo: ec3053b0977e83d9b67fc32c4527a117953994f3\n> refs/heads/sample: 2eefd1150e06d8fca1ddfa684dec016f36bf4e56\n> \n> The user would simply end up with:\n> \n> refs/heads/foo: ec3053b0977e83d9b67fc32c4527a117953994f3\n> refs/heads/sample: 2eefd1150e06d8fca1ddfa684dec016f36bf4e56\n\nMakes sense.\n\n> This is buggy behavior since the user is never informed about the\n> overrides performed and missing references. Nevertheless, the user is\n> left with a working repository with a subset of the references. Since\n> Git 2.51, in such situations fetches would simply fail without updating\n> any references. Which is also buggy behavior and worse off since the\n> user is left without any references.\n> \n> The error is triggered in `lock_raw_ref()` where the files backend\n> attempts to create a lock file. When a lock file already exists the\n> function returns a 'REF_TRANSACTION_ERROR_GENERIC'. When this happens,\n> the entire batched updates, not individual operation, is aborted as if\n> it were in a transaction.\n> \n> Change this to return 'REF_TRANSACTION_ERROR_CASE_CONFLICT' instead to\n> aid the batched update mechanism to simply reject such errors. \n\nSo does this mean that we return `REF_TRANSACTION_ERROR_CASE_CONFLICT`\nin all cases where a a lockfile already exists for a reference? Or do we\nonly actually care about scenarios where the lockfile already exists\nspecific to case-insensitive filesystems?\n\n> The\n> change only affects batched updates since batched updates will reject\n> individual updates with non-generic errors. So specifically this would\n> only affect:\n> \n> 1. git fetch\n> 2. git receive-pack\n> 3. git update-ref --batch-updates\n\nJust to clarify, is this saying that this new error is not ignored in a\nstandard reference transaction? Only the above operations?\n\n> This bubbles the error type up to `files_transaction_prepare()` which\n> tries to lock each reference update. So if the locking fails, we check\n> if the rejection type can be ignored, which is done by calling\n> `ref_transaction_maybe_set_rejected()`.\n> \n> As the error type is now 'REF_TRANSACTION_ERROR_CASE_CONFLICT',\n> the specific reference update would simply be rejected, while other\n> updates in the transaction would continue to be applied. This allows\n> partial application of references in case-insensitive filesystems when\n> fetching colliding references.\n> \n> While the earlier implementation allowed the last reference to be\n> applied overriding the initial references, this change would allow the\n> first reference to be applied while rejecting consequent collisions.\n> This should be an okay compromise since with the files backend, there is\n> no scenario possible where we would retain all colliding references.\n> \n> Let's also be more pro-active and notify users on case-insensitive\n\ns/pro-active/proactive/\n\n> filesystems about such problems by providing a brief about the issue\n> while also recommending using the reftable backend, which doesn't have\n> the same issue.\n> \n> Reported-by: Joe Drew \n> Helped-by: Patrick Steinhardt \n> Signed-off-by: Karthik Nayak \n> ---\n> builtin/fetch.c | 21 +++++++++++++++++---\n> refs.c | 2 ++\n> refs.h | 2 ++\n> refs/files-backend.c | 33 +++++++++++++++++++++++++++-----\n> t/t1400-update-ref.sh | 53 +++++++++++++++++++++++++++++++++++++++++++++++++++\n> t/t5510-fetch.sh | 22 ++++++++++++++++++++-\n> 6 files changed, 124 insertions(+), 9 deletions(-)\n> \n> diff --git a/builtin/fetch.c b/builtin/fetch.c\n> index 24645c4653..c7ff3480fb 100644\n> --- a/builtin/fetch.c\n> +++ b/builtin/fetch.c\n> @@ -1643,7 +1643,8 @@ static int set_head(const struct ref *remote_refs, struct remote *remote)\n> \n> struct ref_rejection_data {\n> \tint *retcode;\n> -\tint conflict_msg_shown;\n> +\tbool conflict_msg_shown;\n> +\tbool case_sensitive_msg_shown;\n> \tconst char *remote_name;\n> };\n> \n> @@ -1657,11 +1658,25 @@ static void ref_transaction_rejection_handler(const char *refname,\n> {\n> \tstruct ref_rejection_data *data = cb_data;\n> \n> -\tif (err == REF_TRANSACTION_ERROR_NAME_CONFLICT && !data->conflict_msg_shown) {\n> +\tif (err == REF_TRANSACTION_ERROR_CASE_CONFLICT && ignore_case &&\n> +\t !data->case_sensitive_msg_shown) {\n> +\t\terror(_(\"You're on a case-insensitive filesystem, and the remote you are\\n\"\n> +\t\t\t\"trying to fetch from has references that only differ in casing. It\\n\"\n> +\t\t\t\"is impossible to store such references with the 'files' backend. You\\n\"\n> +\t\t\t\"can either accept this as-is, in which case you won't be able to\\n\"\n> +\t\t\t\"store all remote references on disk. Or you can alternatively\\n\"\n> +\t\t\t\"migrate your repository to use the 'reftable' backend with the\\n\"\n> +\t\t\t\"following command:\\n\\n git refs migrate --ref-format=reftable\\n\\n\"\n> +\t\t\t\"Please keep in mind that not all implementations of Git support this\\n\"\n> +\t\t\t\"new format yet. So if you use tools other than Git to access this\\n\"\n> +\t\t\t\"repository it may not be an option to migrate to reftables.\\n\"));\n\nNice error message.\n\n[snip]\n> +/*\n> + * Check if the transaction has another update with a case-insensitive refname\n> + * match.\n> + *\n> + * If the update is part of the transaction, we only check up to that index.\n> + * Further updates are expected to call this function to match previous indices.\n> + */\n> +static bool transaction_has_case_conflicting_update(struct ref_transaction *transaction,\n> +\t\t\t\t\t\t struct ref_update *update)\n> +{\n> +\tfor (size_t i = 0; i < transaction->nr; i++) {\n> +\t\tif (transaction->updates[i] == update)\n> +\t\t\tbreak;\n> +\n> +\t\tif (!strcasecmp(transaction->updates[i]->refname, update->refname))\n> +\t\t\treturn true;\n> +\t}\n> +\treturn false;\n> +}\n\nAh ok, so we do validate that the pre-existing lockfile comes from this\ntransaction. That is how we know it is related to case-insensitive\nfilesystems.\n\n> +\n> /*\n> * Lock refname, without following symrefs, and set *lock_p to point\n> * at a newly-allocated lock object. Fill in lock->old_oid, referent,\n> @@ -677,16 +697,17 @@ static void unlock_ref(struct ref_lock *lock)\n> * - Generate informative error messages in the case of failure\n> */\n> static enum ref_transaction_error lock_raw_ref(struct files_ref_store *refs,\n> -\t\t\t\t\t struct ref_update *update,\n> +\t\t\t\t\t struct ref_transaction *transaction,\n> \t\t\t\t\t size_t update_idx,\n> \t\t\t\t\t int mustexist,\n> \t\t\t\t\t struct string_list *refnames_to_check,\n> -\t\t\t\t\t const struct string_list *extras,\n> \t\t\t\t\t struct ref_lock **lock_p,\n> \t\t\t\t\t struct strbuf *referent,\n> \t\t\t\t\t struct strbuf *err)\n> {\n> \tenum ref_transaction_error ret = REF_TRANSACTION_ERROR_GENERIC;\n> +\tstruct ref_update *update = transaction->updates[update_idx];\n> +\tconst struct string_list *extras = &transaction->refnames;\n> \tconst char *refname = update->refname;\n> \tunsigned int *type = &update->type;\n> \tstruct ref_lock *lock;\n> @@ -776,6 +797,9 @@ static enum ref_transaction_error lock_raw_ref(struct files_ref_store *refs,\n> \t\t\tgoto retry;\n> \t\t} else {\n> \t\t\tunable_to_lock_message(ref_file.buf, myerr, err);\n\nhuh, so if if we have a lockfile error due to a case-insensitve\nfilesystem, does this mean we print the error message from\n`unable_to_lock_message()` and the new message?\n\nIf so, I wonder if we would be better off skipping the former since it\ncould be a bit misleading.\n\n> +\t\t\tif (myerr == EEXIST && ignore_case &&\n> +\t\t\t transaction_has_case_conflicting_update(transaction, update))\n> +\t\t\t\tret = REF_TRANSACTION_ERROR_CASE_CONFLICT;\n> \t\t\tgoto error_return;\n> \t\t}\n> \t}\n\n-Justin\n"},{"id":"526503","messageId":"672jdldtxegx4y3gdtcyrnsz5nsy4sh6pk76eftqrukwug3f3c@xwso5f4vukn5","threadId":"64069","inReplyTo":"20250913-587-git-fetch-1-fails-fetches-on-case-insensitive-repositories-v3-2-195569740b57@gmail.com","subject":"Re: [PATCH v3 2/4] refs/files: use correct error type when lock exists","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2025-09-16T21:31:04Z","receivedAt":"2025-09-16T21:31:06Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 25/09/13 10:54PM, Karthik Nayak wrote:\n> When fetching references into a repository, if a lock for a particular\n> reference exists, then `lock_raw_ref()` throws:\n> \n> - REF_TRANSACTION_ERROR_CASE_CONFLICT: when there is a conflict\n> because transaction contains conflicting references while being on a\n\ns/transaction/the transaction/\n\n> case-insensitive filesystem.\n> \n> - REF_TRANSACTION_ERROR_GENERIC: for all other errors.\n> \n> The latter causes the entire set of batched updates to fail, even in\n> case sensitive filessystems.\n\nOk so this issue isn't related to case-insensitive filesystems. The\nissue is that now we use batch updated, a single pre-existing lockfile\ncauses the entire transaction to fail. Prior to batch updates, only the\nindividual update would fail, but wouldn't stop others.\n\n> Instead, return a 'REF_TRANSACTION_ERROR_CREATE_EXISTS' error. This\n> allows batched updates to reject the individual update which conflicts\n> with the existing file, while updating the rest of the references.\n\nMake sense.\n\n> Signed-off-by: Karthik Nayak \n> ---\n> refs/files-backend.c | 20 +++++++++++++++++---\n> t/t5510-fetch.sh | 26 ++++++++++++++++++++++++++\n> 2 files changed, 43 insertions(+), 3 deletions(-)\n> \n> diff --git a/refs/files-backend.c b/refs/files-backend.c\n> index 01df32904b..69e50a16db 100644\n> --- a/refs/files-backend.c\n> +++ b/refs/files-backend.c\n> @@ -797,9 +797,23 @@ static enum ref_transaction_error lock_raw_ref(struct files_ref_store *refs,\n> \t\t\tgoto retry;\n> \t\t} else {\n> \t\t\tunable_to_lock_message(ref_file.buf, myerr, err);\n> -\t\t\tif (myerr == EEXIST && ignore_case &&\n> -\t\t\t transaction_has_case_conflicting_update(transaction, update))\n> -\t\t\t\tret = REF_TRANSACTION_ERROR_CASE_CONFLICT;\n> +\t\t\tif (myerr == EEXIST) {\n> +\t\t\t\tif (ignore_case &&\n> +\t\t\t\t transaction_has_case_conflicting_update(transaction, update))\n> +\t\t\t\t\t/*\n> +\t\t\t\t\t * In case-insensitive filesystems, ensure that conflicts within a\n> +\t\t\t\t\t * given transaction are handled. Pre-existing refs on a\n> +\t\t\t\t\t * case-insensitive system will be overridden without any issue.\n> +\t\t\t\t\t */\n> +\t\t\t\t\tret = REF_TRANSACTION_ERROR_CASE_CONFLICT;\n> +\t\t\t\telse\n> +\t\t\t\t\t/*\n> +\t\t\t\t\t * Pre-existing case-conflicting reference locks should also be\n> +\t\t\t\t\t * specially categorized to avoid failing all batched updates.\n> +\t\t\t\t\t */\n> +\t\t\t\t\tret = REF_TRANSACTION_ERROR_CREATE_EXISTS;\n\nIIUC, by returning a non-generic error here the individual reference\nwill be rejected during batch updates instead of aborting the\ntransaction.\n\n-Justin\n"},{"id":"526506","messageId":"bp4vpyje3ehlq2uezjmkuk5qdigi6fo73sh7eyoofvxxbau7go@oksetdbbtqz7","threadId":"64069","inReplyTo":"20250913-587-git-fetch-1-fails-fetches-on-case-insensitive-repositories-v3-3-195569740b57@gmail.com","subject":"Re: [PATCH v3 3/4] refs/files: handle F/D conflicts in case-insensitive FS","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2025-09-16T21:52:50Z","receivedAt":"2025-09-16T21:52:51Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 25/09/13 10:54PM, Karthik Nayak wrote:\n> When using the files-backend on case-insensitive filesystems, there is\n> possibility of hitting F/D conflicts when creating references within a\n> single transaction, such as:\n> \n> - 'refs/heads/foo'\n> - 'refs/heads/Foo/bar'\n> \n> Ideally such conflicts are caught in `refs_verify_refnames_available()`\n> which is responsible for checking F/D conflicts within a given\n> transaction. This utility function is shared across the reference\n> backends. As such, it doesn't consider the issues of using a\n> case-insensitive file system, which only affects the files-backend.\n> \n> While one solution would be to make the function aware of such issues,\n> this feels like leaking implementation details of file-backend specific\n> issues into the utility function. So opt for the more simpler option, of\n> lowercasing all references sent to this function when on a\n> case-insensitive filesystem and operating on the files-backend.\n> \n> To do this, simply use a `struct strbuf` to convert the refname to a\n> lower case and append it to the list of refnames to be checked. Since we\n\ns/a lower case/lowercase/\n\n> use a `struct strbuf` and the memory is cleared right after, make sure\n> that the string list duplicates all provided string.\n> \n> Without this change, the user would simply be left with a repository\n> with '.lock' files which were created in the 'prepare' phase of the\n> transaction, as the 'commit' phase would simply abort and not do the\n> necessary cleanup.\n\nSo IIUC, this also isn't related to the batched updates change and is\njust an existing issue caused by case-insensitive filesystems and F/D\nconflicts. With this change, we now properly detect F/D conflicts in\nthese situations and thus are able to cleanup lockfiles that would\npreviously be left behind.\n\n> Reported-by: Junio C Hamano \n> Signed-off-by: Karthik Nayak \n> ---\n> refs/files-backend.c | 19 +++++++++++++++++--\n> t/t5510-fetch.sh | 20 ++++++++++++++++++++\n> 2 files changed, 37 insertions(+), 2 deletions(-)\n> \n> diff --git a/refs/files-backend.c b/refs/files-backend.c\n> index 69e50a16db..817b56f4ce 100644\n> --- a/refs/files-backend.c\n> +++ b/refs/files-backend.c\n> @@ -905,8 +905,23 @@ static enum ref_transaction_error lock_raw_ref(struct files_ref_store *refs,\n> \t\t * If the ref did not exist and we are creating it, we have to\n> \t\t * make sure there is no existing packed ref that conflicts\n> \t\t * with refname. This check is deferred so that we can batch it.\n> +\t\t *\n> +\t\t * For case-insensitive filesystems, we should also check for F/D\n> +\t\t * conflicts between 'foo' and 'Foo/bar'. So let's lowercase\n> +\t\t * the refname.\n> \t\t */\n> -\t\titem = string_list_append(refnames_to_check, refname);\n> +\t\tif (ignore_case) {\n> +\t\t\tstruct strbuf lower = STRBUF_INIT;\n> +\n> +\t\t\tstrbuf_addstr(&lower, refname);\n> +\t\t\tstrbuf_tolower(&lower);\n> +\n> +\t\t\titem = string_list_append_nodup(refnames_to_check,\n> +\t\t\t\t\t\t\tstrbuf_detach(&lower, NULL));\n\nFor case-insensitive file-systems, we instead append a lowercased\nversion of the reference name which gets used to check for F/D\nconflicts. Makes sense.\n\n> +\t\t} else {\n> +\t\t\titem = string_list_append(refnames_to_check, refname);\n> +\t\t}\n> +\n\n-Justin\n"},{"id":"526507","messageId":"xmqqh5x2m5d3.fsf@gitster.g","threadId":"64069","inReplyTo":"20250913-587-git-fetch-1-fails-fetches-on-case-insensitive-repositories-v3-0-195569740b57@gmail.com","subject":"Re: [PATCH v3 0/4] refs/files: fix issues with git-fetch on case-insensitive FS","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-09-16T21:53:28Z","receivedAt":"2025-09-16T21:53:31Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Karthik Nayak writes:\n\n> Changes in v3:\n> - Rename duplicate_reference_case_cmp() to\n> transaction_has_case_conflicting_update() and add comments.\n> - Improve commit messages.\n> - Add an additional test in the 4th commit to showcase D/F conflicts in\n> case-sensistive file systems.\n> - Link to v2: https://lore.kernel.org/r/20250908-587-git-fetch-1-fails-fetches-on-case-insensitive-repositories-v2-0-b2eb2459befb@gmail.com\n\nI think I like this \"latest first and then historical\" order in the\ncover letter much better than the other way around.\n\nI see that this topic is pretty much done? There still are a few\nquestions from Justin's reply that may want to be answered, but I\nhave a feeling that the answer to them would not require a new\niteration.\n\nLooking good. Thanks.\n"},{"id":"526546","messageId":"CAOLa=ZR1LNs0i5TDPFG_QvUYS3=CMx3ov-68ph6QKNM273JUHQ@mail.gmail.com","threadId":"64069","inReplyTo":"uv4ifkvxcujjjj3lsc4tf5b5dnc7pkaaw62t6ahagnskxn4kg6@oyxjgupdilrc","subject":"Re: [PATCH v3 1/4] refs/files: catch conflicts on case-insensitive file-systems","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-09-17T07:33:24Z","receivedAt":"2025-09-17T07:33:26Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Justin Tobler writes:\n\n> On 25/09/13 10:54PM, Karthik Nayak wrote:\n>> During the 'prepare' phase of reference transaction in the files\n>\n> s/reference/a reference/\n>\n\nI thought 'phase of reference transaction' without an 'a' is an accepted\nform.. I'll add this locally for now.\n\n>> backend, we create the lock files for references to be created. When\n>> using batched updates on case-insensitive filesystems, the entire\n>> batched updates would be aborted if there are conflicting names such as:\n>>\n>> refs/heads/Foo\n>> refs/heads/foo\n>\n> Ok so this is only a problem now because the reference updates are\n> performed in a single transaction and the resulting error causes the\n> entire transaction to be aborted.\n>\n\nYup, exactly. So conflicting references within the same transaction\nbecome an issue since they try to lock the same file.\n\n>> This affects all commands which were migrated to use batched updates in\n>> Git 2.51, including 'git-fetch(1)' and 'git-receive-pack(1)'. Before\n>> that, reference updates would be applied serially with one transaction\n>> used per update. When users fetched multiple references on\n>> case-insensitive systems, subsequent references would simply overwrite\n>> any earlier references. So when fetching:\n>>\n>> refs/heads/foo: 5f34ec0bfeac225b1c854340257a65b106f70ea6\n>> refs/heads/Foo: ec3053b0977e83d9b67fc32c4527a117953994f3\n>> refs/heads/sample: 2eefd1150e06d8fca1ddfa684dec016f36bf4e56\n>>\n>> The user would simply end up with:\n>>\n>> refs/heads/foo: ec3053b0977e83d9b67fc32c4527a117953994f3\n>> refs/heads/sample: 2eefd1150e06d8fca1ddfa684dec016f36bf4e56\n>\n> Makes sense.\n>\n>> This is buggy behavior since the user is never informed about the\n>> overrides performed and missing references. Nevertheless, the user is\n>> left with a working repository with a subset of the references. Since\n>> Git 2.51, in such situations fetches would simply fail without updating\n>> any references. Which is also buggy behavior and worse off since the\n>> user is left without any references.\n>>\n>> The error is triggered in `lock_raw_ref()` where the files backend\n>> attempts to create a lock file. When a lock file already exists the\n>> function returns a 'REF_TRANSACTION_ERROR_GENERIC'. When this happens,\n>> the entire batched updates, not individual operation, is aborted as if\n>> it were in a transaction.\n>>\n>> Change this to return 'REF_TRANSACTION_ERROR_CASE_CONFLICT' instead to\n>> aid the batched update mechanism to simply reject such errors.\n>\n> So does this mean that we return `REF_TRANSACTION_ERROR_CASE_CONFLICT`\n> in all cases where a a lockfile already exists for a reference? Or do we\n> only actually care about scenarios where the lockfile already exists\n> specific to case-insensitive filesystems?\n>\n\nThis patch specifically only returns when there is a conflict in\ncase-insensitive systems.\n\n>> The\n>> change only affects batched updates since batched updates will reject\n>> individual updates with non-generic errors. So specifically this would\n>> only affect:\n>>\n>> 1. git fetch\n>> 2. git receive-pack\n>> 3. git update-ref --batch-updates\n>\n> Just to clarify, is this saying that this new error is not ignored in a\n> standard reference transaction? Only the above operations?\n>\n\nYes, only batched updates care about error types. Regular transactions\ntreat all errors the same.\n\n>> This bubbles the error type up to `files_transaction_prepare()` which\n>> tries to lock each reference update. So if the locking fails, we check\n>> if the rejection type can be ignored, which is done by calling\n>> `ref_transaction_maybe_set_rejected()`.\n>>\n>> As the error type is now 'REF_TRANSACTION_ERROR_CASE_CONFLICT',\n>> the specific reference update would simply be rejected, while other\n>> updates in the transaction would continue to be applied. This allows\n>> partial application of references in case-insensitive filesystems when\n>> fetching colliding references.\n>>\n>> While the earlier implementation allowed the last reference to be\n>> applied overriding the initial references, this change would allow the\n>> first reference to be applied while rejecting consequent collisions.\n>> This should be an okay compromise since with the files backend, there is\n>> no scenario possible where we would retain all colliding references.\n>>\n>> Let's also be more pro-active and notify users on case-insensitive\n>\n> s/pro-active/proactive/\n>\n\nWill change.\n\n>> @@ -1657,11 +1658,25 @@ static void ref_transaction_rejection_handler(const char *refname,\n>> {\n>> \tstruct ref_rejection_data *data = cb_data;\n>>\n>> -\tif (err == REF_TRANSACTION_ERROR_NAME_CONFLICT && !data->conflict_msg_shown) {\n>> +\tif (err == REF_TRANSACTION_ERROR_CASE_CONFLICT && ignore_case &&\n>> +\t !data->case_sensitive_msg_shown) {\n>> +\t\terror(_(\"You're on a case-insensitive filesystem, and the remote you are\\n\"\n>> +\t\t\t\"trying to fetch from has references that only differ in casing. It\\n\"\n>> +\t\t\t\"is impossible to store such references with the 'files' backend. You\\n\"\n>> +\t\t\t\"can either accept this as-is, in which case you won't be able to\\n\"\n>> +\t\t\t\"store all remote references on disk. Or you can alternatively\\n\"\n>> +\t\t\t\"migrate your repository to use the 'reftable' backend with the\\n\"\n>> +\t\t\t\"following command:\\n\\n git refs migrate --ref-format=reftable\\n\\n\"\n>> +\t\t\t\"Please keep in mind that not all implementations of Git support this\\n\"\n>> +\t\t\t\"new format yet. So if you use tools other than Git to access this\\n\"\n>> +\t\t\t\"repository it may not be an option to migrate to reftables.\\n\"));\n>\n> Nice error message.\n>\n\nAll thanks to Patrick!\n\n>> +\n>> /*\n>> * Lock refname, without following symrefs, and set *lock_p to point\n>> * at a newly-allocated lock object. Fill in lock->old_oid, referent,\n>> @@ -677,16 +697,17 @@ static void unlock_ref(struct ref_lock *lock)\n>> * - Generate informative error messages in the case of failure\n>> */\n>> static enum ref_transaction_error lock_raw_ref(struct files_ref_store *refs,\n>> -\t\t\t\t\t struct ref_update *update,\n>> +\t\t\t\t\t struct ref_transaction *transaction,\n>> \t\t\t\t\t size_t update_idx,\n>> \t\t\t\t\t int mustexist,\n>> \t\t\t\t\t struct string_list *refnames_to_check,\n>> -\t\t\t\t\t const struct string_list *extras,\n>> \t\t\t\t\t struct ref_lock **lock_p,\n>> \t\t\t\t\t struct strbuf *referent,\n>> \t\t\t\t\t struct strbuf *err)\n>> {\n>> \tenum ref_transaction_error ret = REF_TRANSACTION_ERROR_GENERIC;\n>> +\tstruct ref_update *update = transaction->updates[update_idx];\n>> +\tconst struct string_list *extras = &transaction->refnames;\n>> \tconst char *refname = update->refname;\n>> \tunsigned int *type = &update->type;\n>> \tstruct ref_lock *lock;\n>> @@ -776,6 +797,9 @@ static enum ref_transaction_error lock_raw_ref(struct files_ref_store *refs,\n>> \t\t\tgoto retry;\n>> \t\t} else {\n>> \t\t\tunable_to_lock_message(ref_file.buf, myerr, err);\n>\n> huh, so if if we have a lockfile error due to a case-insensitve\n> filesystem, does this mean we print the error message from\n> `unable_to_lock_message()` and the new message?\n>\n> If so, I wonder if we would be better off skipping the former since it\n> could be a bit misleading.\n>\n\nI would say both are necessary. The errors added here are more technical\nand really talk about why we faced an issue. The error in\n'builtin/fetch.c' is more about guidance to how to overcome that issue.\n\nAlso this error is client agnostic, so we'd add the error here for users\nof both regular transactions and batched updates. The error in\n'builtin/fetch.c' is very specific to users of 'git-fetch(1)'. So I\nthink both hold value.\n\n>> +\t\t\tif (myerr == EEXIST && ignore_case &&\n>> +\t\t\t transaction_has_case_conflicting_update(transaction, update))\n>> +\t\t\t\tret = REF_TRANSACTION_ERROR_CASE_CONFLICT;\n>> \t\t\tgoto error_return;\n>> \t\t}\n>> \t}\n>\n> -Justin\n"},{"id":"526547","messageId":"CAOLa=ZTeduuWYmkbOknxGhAQ6QOazPSFUU=AbeBg7Bs_fru2BA@mail.gmail.com","threadId":"64069","inReplyTo":"aMkW-HOAsU8YZ8Lp@pks.im","subject":"Re: [PATCH v3 2/4] refs/files: use correct error type when lock exists","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-09-17T07:37:17Z","receivedAt":"2025-09-17T07:37:18Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Patrick Steinhardt writes:\n\n> On Sat, Sep 13, 2025 at 10:54:30PM +0200, Karthik Nayak wrote:\n>> diff --git a/refs/files-backend.c b/refs/files-backend.c\n>> index 01df32904b..69e50a16db 100644\n>> --- a/refs/files-backend.c\n>> +++ b/refs/files-backend.c\n>> @@ -797,9 +797,23 @@ static enum ref_transaction_error lock_raw_ref(struct files_ref_store *refs,\n>> \t\t\tgoto retry;\n>> \t\t} else {\n>> \t\t\tunable_to_lock_message(ref_file.buf, myerr, err);\n>> -\t\t\tif (myerr == EEXIST && ignore_case &&\n>> -\t\t\t transaction_has_case_conflicting_update(transaction, update))\n>> -\t\t\t\tret = REF_TRANSACTION_ERROR_CASE_CONFLICT;\n>> +\t\t\tif (myerr == EEXIST) {\n>> +\t\t\t\tif (ignore_case &&\n>> +\t\t\t\t transaction_has_case_conflicting_update(transaction, update))\n>> +\t\t\t\t\t/*\n>> +\t\t\t\t\t * In case-insensitive filesystems, ensure that conflicts within a\n>> +\t\t\t\t\t * given transaction are handled. Pre-existing refs on a\n>> +\t\t\t\t\t * case-insensitive system will be overridden without any issue.\n>> +\t\t\t\t\t */\n>> +\t\t\t\t\tret = REF_TRANSACTION_ERROR_CASE_CONFLICT;\n>> +\t\t\t\telse\n>> +\t\t\t\t\t/*\n>> +\t\t\t\t\t * Pre-existing case-conflicting reference locks should also be\n>> +\t\t\t\t\t * specially categorized to avoid failing all batched updates.\n>> +\t\t\t\t\t */\n>> +\t\t\t\t\tret = REF_TRANSACTION_ERROR_CREATE_EXISTS;\n>> +\t\t\t}\n>\n> Tiniest nit: I think it would be preferable to use curly braces for such\n> multi-line comments. This nit isn't worth a reroll though.\n>\n\nI did contemplate about this in my mind, thanks for raising this. I will\nchange it locally for now and push a new version if needed.\n\n> Other than that my feedback got addressed, so this looks good to me.\n> Thanks!\n>\n> Patrick\n\nThanks for your reviews!\n"},{"id":"526548","messageId":"CAOLa=ZSy06tp7PbTOq+L77U4YqA6G1OEZbPb4F6S2dUw4RkqUQ@mail.gmail.com","threadId":"64069","inReplyTo":"672jdldtxegx4y3gdtcyrnsz5nsy4sh6pk76eftqrukwug3f3c@xwso5f4vukn5","subject":"Re: [PATCH v3 2/4] refs/files: use correct error type when lock exists","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-09-17T07:39:31Z","receivedAt":"2025-09-17T07:39:33Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Justin Tobler writes:\n\n> On 25/09/13 10:54PM, Karthik Nayak wrote:\n>> When fetching references into a repository, if a lock for a particular\n>> reference exists, then `lock_raw_ref()` throws:\n>>\n>> - REF_TRANSACTION_ERROR_CASE_CONFLICT: when there is a conflict\n>> because transaction contains conflicting references while being on a\n>\n> s/transaction/the transaction/\n>\n\nMakes sense.\n\n>> case-insensitive filesystem.\n>>\n>> - REF_TRANSACTION_ERROR_GENERIC: for all other errors.\n>>\n>> The latter causes the entire set of batched updates to fail, even in\n>> case sensitive filessystems.\n>\n> Ok so this issue isn't related to case-insensitive filesystems. The\n> issue is that now we use batch updated, a single pre-existing lockfile\n> causes the entire transaction to fail. Prior to batch updates, only the\n> individual update would fail, but wouldn't stop others.\n>\n>> Instead, return a 'REF_TRANSACTION_ERROR_CREATE_EXISTS' error. This\n>> allows batched updates to reject the individual update which conflicts\n>> with the existing file, while updating the rest of the references.\n>\n> Make sense.\n>\n>> Signed-off-by: Karthik Nayak \n>> ---\n>> refs/files-backend.c | 20 +++++++++++++++++---\n>> t/t5510-fetch.sh | 26 ++++++++++++++++++++++++++\n>> 2 files changed, 43 insertions(+), 3 deletions(-)\n>>\n>> diff --git a/refs/files-backend.c b/refs/files-backend.c\n>> index 01df32904b..69e50a16db 100644\n>> --- a/refs/files-backend.c\n>> +++ b/refs/files-backend.c\n>> @@ -797,9 +797,23 @@ static enum ref_transaction_error lock_raw_ref(struct files_ref_store *refs,\n>> \t\t\tgoto retry;\n>> \t\t} else {\n>> \t\t\tunable_to_lock_message(ref_file.buf, myerr, err);\n>> -\t\t\tif (myerr == EEXIST && ignore_case &&\n>> -\t\t\t transaction_has_case_conflicting_update(transaction, update))\n>> -\t\t\t\tret = REF_TRANSACTION_ERROR_CASE_CONFLICT;\n>> +\t\t\tif (myerr == EEXIST) {\n>> +\t\t\t\tif (ignore_case &&\n>> +\t\t\t\t transaction_has_case_conflicting_update(transaction, update))\n>> +\t\t\t\t\t/*\n>> +\t\t\t\t\t * In case-insensitive filesystems, ensure that conflicts within a\n>> +\t\t\t\t\t * given transaction are handled. Pre-existing refs on a\n>> +\t\t\t\t\t * case-insensitive system will be overridden without any issue.\n>> +\t\t\t\t\t */\n>> +\t\t\t\t\tret = REF_TRANSACTION_ERROR_CASE_CONFLICT;\n>> +\t\t\t\telse\n>> +\t\t\t\t\t/*\n>> +\t\t\t\t\t * Pre-existing case-conflicting reference locks should also be\n>> +\t\t\t\t\t * specially categorized to avoid failing all batched updates.\n>> +\t\t\t\t\t */\n>> +\t\t\t\t\tret = REF_TRANSACTION_ERROR_CREATE_EXISTS;\n>\n> IIUC, by returning a non-generic error here the individual reference\n> will be rejected during batch updates instead of aborting the\n> transaction.\n>\n> -Justin\n\nIndeed. Batched updates allow rejecting of individual updates. It only\naborts when faced with a GENERIC error. So by marking this as\nnon-generic, it doesn't change the flow for regular transactions but\nonly how batched updates would handle it.\n"},{"id":"526549","messageId":"CAOLa=ZQW_yYaBmMQk7uovGiK2mxDd2R-wY=w8BS-_LXU2rJUBg@mail.gmail.com","threadId":"64069","inReplyTo":"bp4vpyje3ehlq2uezjmkuk5qdigi6fo73sh7eyoofvxxbau7go@oksetdbbtqz7","subject":"Re: [PATCH v3 3/4] refs/files: handle F/D conflicts in case-insensitive FS","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-09-17T07:42:37Z","receivedAt":"2025-09-17T07:42:38Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Justin Tobler writes:\n\n> On 25/09/13 10:54PM, Karthik Nayak wrote:\n>> When using the files-backend on case-insensitive filesystems, there is\n>> possibility of hitting F/D conflicts when creating references within a\n>> single transaction, such as:\n>>\n>> - 'refs/heads/foo'\n>> - 'refs/heads/Foo/bar'\n>>\n>> Ideally such conflicts are caught in `refs_verify_refnames_available()`\n>> which is responsible for checking F/D conflicts within a given\n>> transaction. This utility function is shared across the reference\n>> backends. As such, it doesn't consider the issues of using a\n>> case-insensitive file system, which only affects the files-backend.\n>>\n>> While one solution would be to make the function aware of such issues,\n>> this feels like leaking implementation details of file-backend specific\n>> issues into the utility function. So opt for the more simpler option, of\n>> lowercasing all references sent to this function when on a\n>> case-insensitive filesystem and operating on the files-backend.\n>>\n>> To do this, simply use a `struct strbuf` to convert the refname to a\n>> lower case and append it to the list of refnames to be checked. Since we\n>\n> s/a lower case/lowercase/\n>\n\nWill change. Thanks.\n\n>> use a `struct strbuf` and the memory is cleared right after, make sure\n>> that the string list duplicates all provided string.\n>>\n>> Without this change, the user would simply be left with a repository\n>> with '.lock' files which were created in the 'prepare' phase of the\n>> transaction, as the 'commit' phase would simply abort and not do the\n>> necessary cleanup.\n>\n> So IIUC, this also isn't related to the batched updates change and is\n> just an existing issue caused by case-insensitive filesystems and F/D\n> conflicts. With this change, we now properly detect F/D conflicts in\n> these situations and thus are able to cleanup lockfiles that would\n> previously be left behind.\n>\n\nYup. That's correct!\n\n>> Reported-by: Junio C Hamano \n>> Signed-off-by: Karthik Nayak \n>> ---\n>> refs/files-backend.c | 19 +++++++++++++++++--\n>> t/t5510-fetch.sh | 20 ++++++++++++++++++++\n>> 2 files changed, 37 insertions(+), 2 deletions(-)\n>>\n>> diff --git a/refs/files-backend.c b/refs/files-backend.c\n>> index 69e50a16db..817b56f4ce 100644\n>> --- a/refs/files-backend.c\n>> +++ b/refs/files-backend.c\n>> @@ -905,8 +905,23 @@ static enum ref_transaction_error lock_raw_ref(struct files_ref_store *refs,\n>> \t\t * If the ref did not exist and we are creating it, we have to\n>> \t\t * make sure there is no existing packed ref that conflicts\n>> \t\t * with refname. This check is deferred so that we can batch it.\n>> +\t\t *\n>> +\t\t * For case-insensitive filesystems, we should also check for F/D\n>> +\t\t * conflicts between 'foo' and 'Foo/bar'. So let's lowercase\n>> +\t\t * the refname.\n>> \t\t */\n>> -\t\titem = string_list_append(refnames_to_check, refname);\n>> +\t\tif (ignore_case) {\n>> +\t\t\tstruct strbuf lower = STRBUF_INIT;\n>> +\n>> +\t\t\tstrbuf_addstr(&lower, refname);\n>> +\t\t\tstrbuf_tolower(&lower);\n>> +\n>> +\t\t\titem = string_list_append_nodup(refnames_to_check,\n>> +\t\t\t\t\t\t\tstrbuf_detach(&lower, NULL));\n>\n> For case-insensitive file-systems, we instead append a lowercased\n> version of the reference name which gets used to check for F/D\n> conflicts. Makes sense.\n>\n>> +\t\t} else {\n>> +\t\t\titem = string_list_append(refnames_to_check, refname);\n>> +\t\t}\n>> +\n>\n> -Justin\n"},{"id":"526552","messageId":"CAOLa=ZRvAMq7MNm6=dBneAXdDe6OjQ=NqYA9-Vd7E+nvDWGH0w@mail.gmail.com","threadId":"64069","inReplyTo":"xmqqh5x2m5d3.fsf@gitster.g","subject":"Re: [PATCH v3 0/4] refs/files: fix issues with git-fetch on case-insensitive FS","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-09-17T07:45:39Z","receivedAt":"2025-09-17T07:45:40Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Junio C Hamano writes:\n\n> Karthik Nayak writes:\n>\n>> Changes in v3:\n>> - Rename duplicate_reference_case_cmp() to\n>> transaction_has_case_conflicting_update() and add comments.\n>> - Improve commit messages.\n>> - Add an additional test in the 4th commit to showcase D/F conflicts in\n>> case-sensistive file systems.\n>> - Link to v2: https://lore.kernel.org/r/20250908-587-git-fetch-1-fails-fetches-on-case-insensitive-repositories-v2-0-b2eb2459befb@gmail.com\n>\n> I think I like this \"latest first and then historical\" order in the\n> cover letter much better than the other way around.\n>\n> I see that this topic is pretty much done? There still are a few\n> questions from Justin's reply that may want to be answered, but I\n> have a feeling that the answer to them would not require a new\n> iteration.\n>\n> Looking good. Thanks.\n\nI did respond to Justin, I think there were a few small nits around\ngrammar in the commit messages and a question around error reporting.\n\nI have applied the grammar fixes locally and don't think they warrant a\nre-roll. Regarding the error reporting. I think it is good the way it\nis.\n\nSo I would say this is good as is and I will refrain for sending in\na new version, unless there is some other concern.\n\nThanks,\nKarthik\n"},{"id":"526576","messageId":"874it1noh2.fsf@iotcl.com","threadId":"64069","inReplyTo":"CAOLa=ZRvAMq7MNm6=dBneAXdDe6OjQ=NqYA9-Vd7E+nvDWGH0w@mail.gmail.com","subject":"Re: [PATCH v3 0/4] refs/files: fix issues with git-fetch on case-insensitive FS","fromName":"Toon Claes","fromEmail":"toon@iotcl.com","sentAt":"2025-09-17T14:27:37Z","receivedAt":"2025-09-17T14:27:50Z","isPatch":true,"sender":{"key":"toon@iotcl.com","avatar":"https://avatars.githubusercontent.com/u/121621?v=4"},"body":"Karthik Nayak writes:\n\n> I did respond to Justin, I think there were a few small nits around\n> grammar in the commit messages and a question around error reporting.\n>\n> I have applied the grammar fixes locally and don't think they warrant a\n> re-roll. Regarding the error reporting. I think it is good the way it\n> is.\n>\n> So I would say this is good as is and I will refrain for sending in\n> a new version, unless there is some other concern.\n\nI also gave it a round of review, and I agree it's good to go.\n\nI've noticed another issue though, at the moment it's possible to create\na ref like `refs/heads/foo.Lock`. I can image this gives issue when the\nremote has `refs/heads/foo` and `refs/heads/foo.Lock` and you pull those\nin on a case-insensitive FS. Unfortunately I wasn't able to verify this.\nBut anyhow, I don't think that any reason to hold back on this current\npatch series. I approve.\n\n-- Toon\n"},{"id":"526579","messageId":"xmqq5xdhgnbk.fsf@gitster.g","threadId":"64069","inReplyTo":"CAOLa=ZRvAMq7MNm6=dBneAXdDe6OjQ=NqYA9-Vd7E+nvDWGH0w@mail.gmail.com","subject":"Re: [PATCH v3 0/4] refs/files: fix issues with git-fetch on case-insensitive FS","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-09-17T14:34:23Z","receivedAt":"2025-09-17T14:34:26Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Karthik Nayak writes:\n\n> Junio C Hamano writes:\n>\n>> Karthik Nayak writes:\n>>\n>>> Changes in v3:\n>>> - Rename duplicate_reference_case_cmp() to\n>>> transaction_has_case_conflicting_update() and add comments.\n>>> - Improve commit messages.\n>>> - Add an additional test in the 4th commit to showcase D/F conflicts in\n>>> case-sensistive file systems.\n>>> - Link to v2: https://lore.kernel.org/r/20250908-587-git-fetch-1-fails-fetches-on-case-insensitive-repositories-v2-0-b2eb2459befb@gmail.com\n>>\n>> I think I like this \"latest first and then historical\" order in the\n>> cover letter much better than the other way around.\n>>\n>> I see that this topic is pretty much done? There still are a few\n>> questions from Justin's reply that may want to be answered, but I\n>> have a feeling that the answer to them would not require a new\n>> iteration.\n>>\n>> Looking good. Thanks.\n>\n> I did respond to Justin, I think there were a few small nits around\n> grammar in the commit messages and a question around error reporting.\n\nWell, if it is already locally ready, let's have a quick \"small and\nfinal\" update before merging it to 'next'.\n\nThanks.\n"},{"id":"526585","messageId":"CAOLa=ZSBpSyU0V08OBrr-QPrZPsYnO1bTZewESY=C7Jy4NTcWQ@mail.gmail.com","threadId":"64069","inReplyTo":"874it1noh2.fsf@iotcl.com","subject":"Re: [PATCH v3 0/4] refs/files: fix issues with git-fetch on case-insensitive FS","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-09-17T15:24:40Z","receivedAt":"2025-09-17T15:24:43Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Toon Claes writes:\n\n> Karthik Nayak writes:\n>\n>> I did respond to Justin, I think there were a few small nits around\n>> grammar in the commit messages and a question around error reporting.\n>>\n>> I have applied the grammar fixes locally and don't think they warrant a\n>> re-roll. Regarding the error reporting. I think it is good the way it\n>> is.\n>>\n>> So I would say this is good as is and I will refrain for sending in\n>> a new version, unless there is some other concern.\n>\n> I also gave it a round of review, and I agree it's good to go.\n>\n\nThanks, appreciate the review!\n\n> I've noticed another issue though, at the moment it's possible to create\n> a ref like `refs/heads/foo.Lock`. I can image this gives issue when the\n> remote has `refs/heads/foo` and `refs/heads/foo.Lock` and you pull those\n> in on a case-insensitive FS. Unfortunately I wasn't able to verify this.\n> But anyhow, I don't think that any reason to hold back on this current\n> patch series. I approve.\n\nWe don't fetch locks from remote. The locking mechanism is simply a\nconstruct used to update files locally in race-free manner. Locking a\nfile ensures no other concurrent writes can happen.\n\nIn-short when you fetch references, the prepare stage of the reference\ntransaction will create the necessary lock files. This locks in the\nupdates with guarantee that no other process can update/create the refs.\nThe commit phase simply removes the lock files post updating the refs.\n\n>\n> -- Toon\n"},{"id":"526586","messageId":"20250917-587-git-fetch-1-fails-fetches-on-case-insensitive-repositories-v4-0-da3c74a08ed0@gmail.com","threadId":"64069","inReplyTo":"20250902-587-git-fetch-1-fails-fetches-on-case-insensitive-repositories-v1-0-35e69bbb507d@gmail.com","subject":"[PATCH v4 0/4] refs/files: fix issues with git-fetch on case-insensitive FS","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-09-17T15:25:10Z","receivedAt":"2025-09-17T15:25:23Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Hello!\n\nWith Git 2.51 we moved 'git-fetch(1)' and 'git-receive-pack(1)' to use\nbatched updates while doing reference updates. This provided a nice perf\nboost since both commands will now use a single transaction for\nreference updates. This removes the overhead of using individual\ntransaction per reference update and also avoids unnecessary\nauto-compaction between reference updates in the reftable backend.\n\nHowever, in the files-backend it does introduce a few bugs around\nconflicts. The reported bug was around case-insensitive filesystems [1],\nbut we also fix some adjacent issues:\n\n1. When fetching references such as:\n\n - refs/heads/foo\n - refs/heads/Foo\n\nEarlier we would simply overwrite the first reference with the second\nand continue. Since Git 2.51 we simply abort stating a conflict.\n\nThis is resolved in the first commit by explicitly categorizing the\nerror as non-GENERIC. This allows batched updates to reject the\nparticular update, while updating the rest.\n\n2. When fetching references and a lock for a particular reference\nalready exits. We treat this is a GENERIC error, which fails the entire\nupdate. By categorizing this error as non-GENERIC, we can reject this\nspecific update and update the other references.\n\n3. When fetching references such as with F/D conflict:\n\n - refs/heads/foo\n - refs/heads/Foo/bar\n\nEarlier we would apply the first, while the second would fail due to\nconflict. Since Git 2.51, the lock files for both would be created, but\nthe 'commit' phase would abruptly end leaving the lock files.\n\nThe second commit fixes this by ensuring that on case-insensitive\nfilesystems we lowercase the refnames for availability check to ensure\nF/D are caught and reported to the user.\n\n4. When fetching references with D/F conflict:\n\n - refs/heads/Foo/bar\n - refs/heads/foo\n\nThe creation of the second reference's lock in `lock_raw_ref()` catches\nthe D/F conflict, but we mark this as a GENERIC error. By categorizing\nthis as non-GENERIC, we can allow the updates to continue while\nrejecting this specific error.\n\nThis also applies to D/F conflicts in case-sensitive systems where there\nexists a lock in the repo 'refs/heads/foo/bar.lock' causing a conflict\nwhile fetching 'refs/heads/foo'.\n\n- Karthik\n\n[1]: https://lore.kernel.org/all/YQXPR01MB3046197EF39296549EE6DD669A33A@YQXPR01MB3046.CANPRD01.PROD.OUTLOOK.COM/\n\nSigned-off-by: Karthik Nayak \n---\nChanges in v4:\n- Fixes to typos in commit messages.\n- Use curly braces for if..else clause with inlined comments.\n- Link to v3: https://lore.kernel.org/r/20250913-587-git-fetch-1-fails-fetches-on-case-insensitive-repositories-v3-0-195569740b57@gmail.com\n\nChanges in v3:\n- Rename duplicate_reference_case_cmp() to\n transaction_has_case_conflicting_update() and add comments.\n- Improve commit messages.\n- Add an additional test in the 4th commit to showcase D/F conflicts in\n case-sensistive file systems.\n- Link to v2: https://lore.kernel.org/r/20250908-587-git-fetch-1-fails-fetches-on-case-insensitive-repositories-v2-0-b2eb2459befb@gmail.com\n\nChanges in v2:\n- This version fixes two more issues:\n - Fetching while locks already exist in the repository\n - D/F conflicts while fetching\n- Add a specific error to the first case, so we can nicely show a\n relevant error. Also check explicitly that the issue is due to\n case-insensitive filesystems.\n- Cleanup the commit messages.\n- Use `string_list_append_nodup()` with `strbuf_detach`, reducing the\n number of allocations.\n\n---\n builtin/fetch.c | 21 ++++++++--\n refs.c | 11 ++++-\n refs.h | 2 +\n refs/files-backend.c | 78 ++++++++++++++++++++++++++++------\n t/t1400-update-ref.sh | 53 +++++++++++++++++++++++\n t/t5510-fetch.sh | 114 +++++++++++++++++++++++++++++++++++++++++++++++++-\n 6 files changed, 262 insertions(+), 17 deletions(-)\n\nKarthik Nayak (4):\n refs/files: catch conflicts on case-insensitive file-systems\n refs/files: use correct error type when lock exists\n refs/files: handle F/D conflicts in case-insensitive FS\n refs/files: handle D/F conflicts during locking\n\nRange-diff versus v3:\n\n1: ec89ebee70 ! 1: 203a32f814 refs/files: catch conflicts on case-insensitive file-systems\n @@ Metadata\n ## Commit message ##\n refs/files: catch conflicts on case-insensitive file-systems\n \n - During the 'prepare' phase of reference transaction in the files\n + During the 'prepare' phase of a reference transaction in the files\n backend, we create the lock files for references to be created. When\n using batched updates on case-insensitive filesystems, the entire\n batched updates would be aborted if there are conflicting names such as:\n @@ Commit message\n This should be an okay compromise since with the files backend, there is\n no scenario possible where we would retain all colliding references.\n \n - Let's also be more pro-active and notify users on case-insensitive\n + Let's also be more proactive and notify users on case-insensitive\n filesystems about such problems by providing a brief about the issue\n while also recommending using the reftable backend, which doesn't have\n the same issue.\n2: e231d44125 ! 2: f06c1a89b9 refs/files: use correct error type when lock exists\n @@ Commit message\n reference exists, then `lock_raw_ref()` throws:\n \n - REF_TRANSACTION_ERROR_CASE_CONFLICT: when there is a conflict\n - because transaction contains conflicting references while being on a\n - case-insensitive filesystem.\n + because the transaction contains conflicting references while being\n + on a case-insensitive filesystem.\n \n - REF_TRANSACTION_ERROR_GENERIC: for all other errors.\n \n @@ refs/files-backend.c: static enum ref_transaction_error lock_raw_ref(struct file\n -\t\t\t\tret = REF_TRANSACTION_ERROR_CASE_CONFLICT;\n +\t\t\tif (myerr == EEXIST) {\n +\t\t\t\tif (ignore_case &&\n -+\t\t\t\t transaction_has_case_conflicting_update(transaction, update))\n ++\t\t\t\t transaction_has_case_conflicting_update(transaction, update)) {\n +\t\t\t\t\t/*\n +\t\t\t\t\t * In case-insensitive filesystems, ensure that conflicts within a\n +\t\t\t\t\t * given transaction are handled. Pre-existing refs on a\n +\t\t\t\t\t * case-insensitive system will be overridden without any issue.\n +\t\t\t\t\t */\n +\t\t\t\t\tret = REF_TRANSACTION_ERROR_CASE_CONFLICT;\n -+\t\t\t\telse\n ++\t\t\t\t} else {\n +\t\t\t\t\t/*\n +\t\t\t\t\t * Pre-existing case-conflicting reference locks should also be\n +\t\t\t\t\t * specially categorized to avoid failing all batched updates.\n +\t\t\t\t\t */\n +\t\t\t\t\tret = REF_TRANSACTION_ERROR_CREATE_EXISTS;\n ++\t\t\t\t}\n +\t\t\t}\n +\n \t\t\tgoto error_return;\n3: 0100f5a48d ! 3: e1b3f8106c refs/files: handle F/D conflicts in case-insensitive FS\n @@ Commit message\n lowercasing all references sent to this function when on a\n case-insensitive filesystem and operating on the files-backend.\n \n - To do this, simply use a `struct strbuf` to convert the refname to a\n - lower case and append it to the list of refnames to be checked. Since we\n + To do this, simply use a `struct strbuf` to convert the refname to\n + lowercase and append it to the list of refnames to be checked. Since we\n use a `struct strbuf` and the memory is cleared right after, make sure\n that the string list duplicates all provided string.\n \n4: 64def2c0fb = 4: 5c4886a1fb refs/files: handle D/F conflicts during locking\n\n\nbase-commit: c44beea485f0f2feaf460e2ac87fdd5608d63cf0\nchange-id: 20250822-587-git-fetch-1-fails-fetches-on-case-insensitive-repositories-0a187c7ac41e\n\nThanks\n- Karthik\n\n"},{"id":"526587","messageId":"20250917-587-git-fetch-1-fails-fetches-on-case-insensitive-repositories-v4-1-da3c74a08ed0@gmail.com","threadId":"64069","inReplyTo":"20250917-587-git-fetch-1-fails-fetches-on-case-insensitive-repositories-v4-0-da3c74a08ed0@gmail.com","subject":"[PATCH v4 1/4] refs/files: catch conflicts on case-insensitive file-systems","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-09-17T15:25:11Z","receivedAt":"2025-09-17T15:25:24Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"During the 'prepare' phase of a reference transaction in the files\nbackend, we create the lock files for references to be created. When\nusing batched updates on case-insensitive filesystems, the entire\nbatched updates would be aborted if there are conflicting names such as:\n\n refs/heads/Foo\n refs/heads/foo\n\nThis affects all commands which were migrated to use batched updates in\nGit 2.51, including 'git-fetch(1)' and 'git-receive-pack(1)'. Before\nthat, reference updates would be applied serially with one transaction\nused per update. When users fetched multiple references on\ncase-insensitive systems, subsequent references would simply overwrite\nany earlier references. So when fetching:\n\n refs/heads/foo: 5f34ec0bfeac225b1c854340257a65b106f70ea6\n refs/heads/Foo: ec3053b0977e83d9b67fc32c4527a117953994f3\n refs/heads/sample: 2eefd1150e06d8fca1ddfa684dec016f36bf4e56\n\nThe user would simply end up with:\n\n refs/heads/foo: ec3053b0977e83d9b67fc32c4527a117953994f3\n refs/heads/sample: 2eefd1150e06d8fca1ddfa684dec016f36bf4e56\n\nThis is buggy behavior since the user is never informed about the\noverrides performed and missing references. Nevertheless, the user is\nleft with a working repository with a subset of the references. Since\nGit 2.51, in such situations fetches would simply fail without updating\nany references. Which is also buggy behavior and worse off since the\nuser is left without any references.\n\nThe error is triggered in `lock_raw_ref()` where the files backend\nattempts to create a lock file. When a lock file already exists the\nfunction returns a 'REF_TRANSACTION_ERROR_GENERIC'. When this happens,\nthe entire batched updates, not individual operation, is aborted as if\nit were in a transaction.\n\nChange this to return 'REF_TRANSACTION_ERROR_CASE_CONFLICT' instead to\naid the batched update mechanism to simply reject such errors. The\nchange only affects batched updates since batched updates will reject\nindividual updates with non-generic errors. So specifically this would\nonly affect:\n\n 1. git fetch\n 2. git receive-pack\n 3. git update-ref --batch-updates\n\nThis bubbles the error type up to `files_transaction_prepare()` which\ntries to lock each reference update. So if the locking fails, we check\nif the rejection type can be ignored, which is done by calling\n`ref_transaction_maybe_set_rejected()`.\n\nAs the error type is now 'REF_TRANSACTION_ERROR_CASE_CONFLICT',\nthe specific reference update would simply be rejected, while other\nupdates in the transaction would continue to be applied. This allows\npartial application of references in case-insensitive filesystems when\nfetching colliding references.\n\nWhile the earlier implementation allowed the last reference to be\napplied overriding the initial references, this change would allow the\nfirst reference to be applied while rejecting consequent collisions.\nThis should be an okay compromise since with the files backend, there is\nno scenario possible where we would retain all colliding references.\n\nLet's also be more proactive and notify users on case-insensitive\nfilesystems about such problems by providing a brief about the issue\nwhile also recommending using the reftable backend, which doesn't have\nthe same issue.\n\nReported-by: Joe Drew \nHelped-by: Patrick Steinhardt \nSigned-off-by: Karthik Nayak \n---\n builtin/fetch.c | 21 +++++++++++++++++---\n refs.c | 2 ++\n refs.h | 2 ++\n refs/files-backend.c | 33 +++++++++++++++++++++++++++-----\n t/t1400-update-ref.sh | 53 +++++++++++++++++++++++++++++++++++++++++++++++++++\n t/t5510-fetch.sh | 22 ++++++++++++++++++++-\n 6 files changed, 124 insertions(+), 9 deletions(-)\n\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex 24645c4653..c7ff3480fb 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -1643,7 +1643,8 @@ static int set_head(const struct ref *remote_refs, struct remote *remote)\n \n struct ref_rejection_data {\n \tint *retcode;\n-\tint conflict_msg_shown;\n+\tbool conflict_msg_shown;\n+\tbool case_sensitive_msg_shown;\n \tconst char *remote_name;\n };\n \n@@ -1657,11 +1658,25 @@ static void ref_transaction_rejection_handler(const char *refname,\n {\n \tstruct ref_rejection_data *data = cb_data;\n \n-\tif (err == REF_TRANSACTION_ERROR_NAME_CONFLICT && !data->conflict_msg_shown) {\n+\tif (err == REF_TRANSACTION_ERROR_CASE_CONFLICT && ignore_case &&\n+\t !data->case_sensitive_msg_shown) {\n+\t\terror(_(\"You're on a case-insensitive filesystem, and the remote you are\\n\"\n+\t\t\t\"trying to fetch from has references that only differ in casing. It\\n\"\n+\t\t\t\"is impossible to store such references with the 'files' backend. You\\n\"\n+\t\t\t\"can either accept this as-is, in which case you won't be able to\\n\"\n+\t\t\t\"store all remote references on disk. Or you can alternatively\\n\"\n+\t\t\t\"migrate your repository to use the 'reftable' backend with the\\n\"\n+\t\t\t\"following command:\\n\\n git refs migrate --ref-format=reftable\\n\\n\"\n+\t\t\t\"Please keep in mind that not all implementations of Git support this\\n\"\n+\t\t\t\"new format yet. So if you use tools other than Git to access this\\n\"\n+\t\t\t\"repository it may not be an option to migrate to reftables.\\n\"));\n+\t\tdata->case_sensitive_msg_shown = true;\n+\t} else if (err == REF_TRANSACTION_ERROR_NAME_CONFLICT &&\n+\t\t !data->conflict_msg_shown) {\n \t\terror(_(\"some local refs could not be updated; try running\\n\"\n \t\t\t\" 'git remote prune %s' to remove any old, conflicting \"\n \t\t\t\"branches\"), data->remote_name);\n-\t\tdata->conflict_msg_shown = 1;\n+\t\tdata->conflict_msg_shown = true;\n \t} else {\n \t\tconst char *reason = ref_transaction_error_msg(err);\n \ndiff --git a/refs.c b/refs.c\nindex bfdbe718b7..4c1c339ed9 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -3321,6 +3321,8 @@ const char *ref_transaction_error_msg(enum ref_transaction_error err)\n \t\treturn \"invalid new value provided\";\n \tcase REF_TRANSACTION_ERROR_EXPECTED_SYMREF:\n \t\treturn \"expected symref but found regular ref\";\n+\tcase REF_TRANSACTION_ERROR_CASE_CONFLICT:\n+\t\treturn \"reference conflict due to case-insensitive filesystem\";\n \tdefault:\n \t\treturn \"unknown failure\";\n \t}\ndiff --git a/refs.h b/refs.h\nindex eedbb599c5..41915086b3 100644\n--- a/refs.h\n+++ b/refs.h\n@@ -31,6 +31,8 @@ enum ref_transaction_error {\n \tREF_TRANSACTION_ERROR_INVALID_NEW_VALUE = -6,\n \t/* Expected ref to be symref, but is a regular ref */\n \tREF_TRANSACTION_ERROR_EXPECTED_SYMREF = -7,\n+\t/* Cannot create ref due to case-insensitive filesystem */\n+\tREF_TRANSACTION_ERROR_CASE_CONFLICT = -8,\n };\n \n /*\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 088b52c740..01df32904b 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -647,6 +647,26 @@ static void unlock_ref(struct ref_lock *lock)\n \t}\n }\n \n+/*\n+ * Check if the transaction has another update with a case-insensitive refname\n+ * match.\n+ *\n+ * If the update is part of the transaction, we only check up to that index.\n+ * Further updates are expected to call this function to match previous indices.\n+ */\n+static bool transaction_has_case_conflicting_update(struct ref_transaction *transaction,\n+\t\t\t\t\t\t struct ref_update *update)\n+{\n+\tfor (size_t i = 0; i < transaction->nr; i++) {\n+\t\tif (transaction->updates[i] == update)\n+\t\t\tbreak;\n+\n+\t\tif (!strcasecmp(transaction->updates[i]->refname, update->refname))\n+\t\t\treturn true;\n+\t}\n+\treturn false;\n+}\n+\n /*\n * Lock refname, without following symrefs, and set *lock_p to point\n * at a newly-allocated lock object. Fill in lock->old_oid, referent,\n@@ -677,16 +697,17 @@ static void unlock_ref(struct ref_lock *lock)\n * - Generate informative error messages in the case of failure\n */\n static enum ref_transaction_error lock_raw_ref(struct files_ref_store *refs,\n-\t\t\t\t\t struct ref_update *update,\n+\t\t\t\t\t struct ref_transaction *transaction,\n \t\t\t\t\t size_t update_idx,\n \t\t\t\t\t int mustexist,\n \t\t\t\t\t struct string_list *refnames_to_check,\n-\t\t\t\t\t const struct string_list *extras,\n \t\t\t\t\t struct ref_lock **lock_p,\n \t\t\t\t\t struct strbuf *referent,\n \t\t\t\t\t struct strbuf *err)\n {\n \tenum ref_transaction_error ret = REF_TRANSACTION_ERROR_GENERIC;\n+\tstruct ref_update *update = transaction->updates[update_idx];\n+\tconst struct string_list *extras = &transaction->refnames;\n \tconst char *refname = update->refname;\n \tunsigned int *type = &update->type;\n \tstruct ref_lock *lock;\n@@ -776,6 +797,9 @@ static enum ref_transaction_error lock_raw_ref(struct files_ref_store *refs,\n \t\t\tgoto retry;\n \t\t} else {\n \t\t\tunable_to_lock_message(ref_file.buf, myerr, err);\n+\t\t\tif (myerr == EEXIST && ignore_case &&\n+\t\t\t transaction_has_case_conflicting_update(transaction, update))\n+\t\t\t\tret = REF_TRANSACTION_ERROR_CASE_CONFLICT;\n \t\t\tgoto error_return;\n \t\t}\n \t}\n@@ -2583,9 +2607,8 @@ static enum ref_transaction_error lock_ref_for_update(struct files_ref_store *re\n \tif (lock) {\n \t\tlock->count++;\n \t} else {\n-\t\tret = lock_raw_ref(refs, update, update_idx, mustexist,\n-\t\t\t\t refnames_to_check, &transaction->refnames,\n-\t\t\t\t &lock, &referent, err);\n+\t\tret = lock_raw_ref(refs, transaction, update_idx, mustexist,\n+\t\t\t\t refnames_to_check, &lock, &referent, err);\n \t\tif (ret) {\n \t\t\tchar *reason;\n \ndiff --git a/t/t1400-update-ref.sh b/t/t1400-update-ref.sh\nindex 96648a6e5d..08d5df2af7 100755\n--- a/t/t1400-update-ref.sh\n+++ b/t/t1400-update-ref.sh\n@@ -2294,6 +2294,59 @@ do\n \t\t)\n \t'\n \n+\ttest_expect_success CASE_INSENSITIVE_FS,REFFILES \"stdin $type batch-updates existing reference\" '\n+\t\tgit init repo &&\n+\t\ttest_when_finished \"rm -fr repo\" &&\n+\t\t(\n+\t\t\tcd repo &&\n+\t\t\ttest_commit one &&\n+\t\t\told_head=$(git rev-parse HEAD) &&\n+\t\t\ttest_commit two &&\n+\t\t\thead=$(git rev-parse HEAD) &&\n+\n+\t\t\t{\n+\t\t\t\tformat_command $type \"create refs/heads/foo\" \"$head\" &&\n+\t\t\t\tformat_command $type \"create refs/heads/ref\" \"$old_head\" &&\n+\t\t\t\tformat_command $type \"create refs/heads/Foo\" \"$old_head\"\n+\t\t\t} >stdin &&\n+\t\t\tgit update-ref $type --stdin --batch-updates stdout &&\n+\n+\t\t\techo $head >expect &&\n+\t\t\tgit rev-parse refs/heads/foo >actual &&\n+\t\t\techo $old_head >expect &&\n+\t\t\tgit rev-parse refs/heads/ref >actual &&\n+\t\t\ttest_cmp expect actual &&\n+\t\t\ttest_grep -q \"reference conflict due to case-insensitive filesystem\" stdout\n+\t\t)\n+\t'\n+\n+\ttest_expect_success CASE_INSENSITIVE_FS \"stdin $type batch-updates existing reference\" '\n+\t\tgit init --ref-format=reftable repo &&\n+\t\ttest_when_finished \"rm -fr repo\" &&\n+\t\t(\n+\t\t\tcd repo &&\n+\t\t\ttest_commit one &&\n+\t\t\told_head=$(git rev-parse HEAD) &&\n+\t\t\ttest_commit two &&\n+\t\t\thead=$(git rev-parse HEAD) &&\n+\n+\t\t\t{\n+\t\t\t\tformat_command $type \"create refs/heads/foo\" \"$head\" &&\n+\t\t\t\tformat_command $type \"create refs/heads/ref\" \"$old_head\" &&\n+\t\t\t\tformat_command $type \"create refs/heads/Foo\" \"$old_head\"\n+\t\t\t} >stdin &&\n+\t\t\tgit update-ref $type --stdin --batch-updates stdout &&\n+\n+\t\t\techo $head >expect &&\n+\t\t\tgit rev-parse refs/heads/foo >actual &&\n+\t\t\techo $old_head >expect &&\n+\t\t\tgit rev-parse refs/heads/ref >actual &&\n+\t\t\ttest_cmp expect actual &&\n+\t\t\tgit rev-parse refs/heads/Foo >actual &&\n+\t\t\ttest_cmp expect actual\n+\t\t)\n+\t'\n+\n \ttest_expect_success \"stdin $type batch-updates delete incorrect symbolic ref\" '\n \t\tgit init repo &&\n \t\ttest_when_finished \"rm -fr repo\" &&\ndiff --git a/t/t5510-fetch.sh b/t/t5510-fetch.sh\nindex ebc696546b..57f60da81b 100755\n--- a/t/t5510-fetch.sh\n+++ b/t/t5510-fetch.sh\n@@ -47,7 +47,13 @@ test_expect_success \"clone and setup child repos\" '\n \t\tgit config set branch.main.merge refs/heads/one\n \t) &&\n \tgit clone . bundle &&\n-\tgit clone . seven\n+\tgit clone . seven &&\n+\tgit clone --ref-format=reftable . case_sensitive &&\n+\t(\n+\t\tcd case_sensitive &&\n+\t\tgit branch branch1 &&\n+\t\tgit branch bRanch1\n+\t)\n '\n \n test_expect_success \"fetch test\" '\n@@ -1526,6 +1532,20 @@ test_expect_success SYMLINKS 'clone does not get confused by a D/F conflict' '\n \ttest_path_is_missing whoops\n '\n \n+test_expect_success CASE_INSENSITIVE_FS,REFFILES 'existing references in a case insensitive filesystem' '\n+\ttest_when_finished rm -rf case_insensitive &&\n+\t(\n+\t\tgit init --bare case_insensitive &&\n+\t\tcd case_insensitive &&\n+\t\tgit remote add origin -- ../case_sensitive &&\n+\t\ttest_must_fail git fetch -f origin \"refs/heads/*:refs/heads/*\" 2>err &&\n+\t\ttest_grep \"You${SQ}re on a case-insensitive filesystem\" err &&\n+\t\tgit rev-parse refs/heads/main >expect &&\n+\t\tgit rev-parse refs/heads/branch1 >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n . \"$TEST_DIRECTORY\"/lib-httpd.sh\n start_httpd\n \n\n-- \n2.51.0\n\n"},{"id":"526588","messageId":"20250917-587-git-fetch-1-fails-fetches-on-case-insensitive-repositories-v4-2-da3c74a08ed0@gmail.com","threadId":"64069","inReplyTo":"20250917-587-git-fetch-1-fails-fetches-on-case-insensitive-repositories-v4-0-da3c74a08ed0@gmail.com","subject":"[PATCH v4 2/4] refs/files: use correct error type when lock exists","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-09-17T15:25:12Z","receivedAt":"2025-09-17T15:25:26Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"When fetching references into a repository, if a lock for a particular\nreference exists, then `lock_raw_ref()` throws:\n\n - REF_TRANSACTION_ERROR_CASE_CONFLICT: when there is a conflict\n because the transaction contains conflicting references while being\n on a case-insensitive filesystem.\n\n - REF_TRANSACTION_ERROR_GENERIC: for all other errors.\n\nThe latter causes the entire set of batched updates to fail, even in\ncase sensitive filessystems.\n\nInstead, return a 'REF_TRANSACTION_ERROR_CREATE_EXISTS' error. This\nallows batched updates to reject the individual update which conflicts\nwith the existing file, while updating the rest of the references.\n\nSigned-off-by: Karthik Nayak \n---\n refs/files-backend.c | 21 ++++++++++++++++++---\n t/t5510-fetch.sh | 26 ++++++++++++++++++++++++++\n 2 files changed, 44 insertions(+), 3 deletions(-)\n\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex 01df32904b..d1af5d6bc7 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -797,9 +797,24 @@ static enum ref_transaction_error lock_raw_ref(struct files_ref_store *refs,\n \t\t\tgoto retry;\n \t\t} else {\n \t\t\tunable_to_lock_message(ref_file.buf, myerr, err);\n-\t\t\tif (myerr == EEXIST && ignore_case &&\n-\t\t\t transaction_has_case_conflicting_update(transaction, update))\n-\t\t\t\tret = REF_TRANSACTION_ERROR_CASE_CONFLICT;\n+\t\t\tif (myerr == EEXIST) {\n+\t\t\t\tif (ignore_case &&\n+\t\t\t\t transaction_has_case_conflicting_update(transaction, update)) {\n+\t\t\t\t\t/*\n+\t\t\t\t\t * In case-insensitive filesystems, ensure that conflicts within a\n+\t\t\t\t\t * given transaction are handled. Pre-existing refs on a\n+\t\t\t\t\t * case-insensitive system will be overridden without any issue.\n+\t\t\t\t\t */\n+\t\t\t\t\tret = REF_TRANSACTION_ERROR_CASE_CONFLICT;\n+\t\t\t\t} else {\n+\t\t\t\t\t/*\n+\t\t\t\t\t * Pre-existing case-conflicting reference locks should also be\n+\t\t\t\t\t * specially categorized to avoid failing all batched updates.\n+\t\t\t\t\t */\n+\t\t\t\t\tret = REF_TRANSACTION_ERROR_CREATE_EXISTS;\n+\t\t\t\t}\n+\t\t\t}\n+\n \t\t\tgoto error_return;\n \t\t}\n \t}\ndiff --git a/t/t5510-fetch.sh b/t/t5510-fetch.sh\nindex 57f60da81b..6f8db0ace4 100755\n--- a/t/t5510-fetch.sh\n+++ b/t/t5510-fetch.sh\n@@ -1546,6 +1546,32 @@ test_expect_success CASE_INSENSITIVE_FS,REFFILES 'existing references in a case\n \t)\n '\n \n+test_expect_success REFFILES 'existing reference lock in repo' '\n+\ttest_when_finished rm -rf base repo &&\n+\t(\n+\t\tgit init --ref-format=reftable base &&\n+\t\tcd base &&\n+\t\techo >file update &&\n+\t\tgit add . &&\n+\t\tgit commit -m \"updated\" &&\n+\t\tgit branch -M main &&\n+\n+\t\tgit update-ref refs/heads/foo @ &&\n+\t\tgit update-ref refs/heads/branch @ &&\n+\t\tcd .. &&\n+\n+\t\tgit init --ref-format=files --bare repo &&\n+\t\tcd repo &&\n+\t\tgit remote add origin ../base &&\n+\t\ttouch refs/heads/foo.lock &&\n+\t\ttest_must_fail git fetch -f origin \"refs/heads/*:refs/heads/*\" 2>err &&\n+\t\ttest_grep \"error: fetching ref refs/heads/foo failed: reference already exists\" err &&\n+\t\tgit rev-parse refs/heads/main >expect &&\n+\t\tgit rev-parse refs/heads/branch >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n . \"$TEST_DIRECTORY\"/lib-httpd.sh\n start_httpd\n \n\n-- \n2.51.0\n\n"},{"id":"526589","messageId":"20250917-587-git-fetch-1-fails-fetches-on-case-insensitive-repositories-v4-3-da3c74a08ed0@gmail.com","threadId":"64069","inReplyTo":"20250917-587-git-fetch-1-fails-fetches-on-case-insensitive-repositories-v4-0-da3c74a08ed0@gmail.com","subject":"[PATCH v4 3/4] refs/files: handle F/D conflicts in case-insensitive FS","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-09-17T15:25:13Z","receivedAt":"2025-09-17T15:25:27Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"When using the files-backend on case-insensitive filesystems, there is\npossibility of hitting F/D conflicts when creating references within a\nsingle transaction, such as:\n\n - 'refs/heads/foo'\n - 'refs/heads/Foo/bar'\n\nIdeally such conflicts are caught in `refs_verify_refnames_available()`\nwhich is responsible for checking F/D conflicts within a given\ntransaction. This utility function is shared across the reference\nbackends. As such, it doesn't consider the issues of using a\ncase-insensitive file system, which only affects the files-backend.\n\nWhile one solution would be to make the function aware of such issues,\nthis feels like leaking implementation details of file-backend specific\nissues into the utility function. So opt for the more simpler option, of\nlowercasing all references sent to this function when on a\ncase-insensitive filesystem and operating on the files-backend.\n\nTo do this, simply use a `struct strbuf` to convert the refname to\nlowercase and append it to the list of refnames to be checked. Since we\nuse a `struct strbuf` and the memory is cleared right after, make sure\nthat the string list duplicates all provided string.\n\nWithout this change, the user would simply be left with a repository\nwith '.lock' files which were created in the 'prepare' phase of the\ntransaction, as the 'commit' phase would simply abort and not do the\nnecessary cleanup.\n\nReported-by: Junio C Hamano \nSigned-off-by: Karthik Nayak \n---\n refs/files-backend.c | 19 +++++++++++++++++--\n t/t5510-fetch.sh | 20 ++++++++++++++++++++\n 2 files changed, 37 insertions(+), 2 deletions(-)\n\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex d1af5d6bc7..bfdf85121a 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -906,8 +906,23 @@ static enum ref_transaction_error lock_raw_ref(struct files_ref_store *refs,\n \t\t * If the ref did not exist and we are creating it, we have to\n \t\t * make sure there is no existing packed ref that conflicts\n \t\t * with refname. This check is deferred so that we can batch it.\n+\t\t *\n+\t\t * For case-insensitive filesystems, we should also check for F/D\n+\t\t * conflicts between 'foo' and 'Foo/bar'. So let's lowercase\n+\t\t * the refname.\n \t\t */\n-\t\titem = string_list_append(refnames_to_check, refname);\n+\t\tif (ignore_case) {\n+\t\t\tstruct strbuf lower = STRBUF_INIT;\n+\n+\t\t\tstrbuf_addstr(&lower, refname);\n+\t\t\tstrbuf_tolower(&lower);\n+\n+\t\t\titem = string_list_append_nodup(refnames_to_check,\n+\t\t\t\t\t\t\tstrbuf_detach(&lower, NULL));\n+\t\t} else {\n+\t\t\titem = string_list_append(refnames_to_check, refname);\n+\t\t}\n+\n \t\titem->util = xmalloc(sizeof(update_idx));\n \t\tmemcpy(item->util, &update_idx, sizeof(update_idx));\n \t}\n@@ -2832,7 +2847,7 @@ static int files_transaction_prepare(struct ref_store *ref_store,\n \t\t\t \"ref_transaction_prepare\");\n \tsize_t i;\n \tint ret = 0;\n-\tstruct string_list refnames_to_check = STRING_LIST_INIT_NODUP;\n+\tstruct string_list refnames_to_check = STRING_LIST_INIT_DUP;\n \tchar *head_ref = NULL;\n \tint head_type;\n \tstruct files_transaction_backend_data *backend_data;\ndiff --git a/t/t5510-fetch.sh b/t/t5510-fetch.sh\nindex 6f8db0ace4..08dbea6503 100755\n--- a/t/t5510-fetch.sh\n+++ b/t/t5510-fetch.sh\n@@ -53,6 +53,12 @@ test_expect_success \"clone and setup child repos\" '\n \t\tcd case_sensitive &&\n \t\tgit branch branch1 &&\n \t\tgit branch bRanch1\n+\t) &&\n+\tgit clone --ref-format=reftable . case_sensitive_fd &&\n+\t(\n+\t\tcd case_sensitive_fd &&\n+\t\tgit branch foo/bar &&\n+\t\tgit branch Foo\n \t)\n '\n \n@@ -1572,6 +1578,20 @@ test_expect_success REFFILES 'existing reference lock in repo' '\n \t)\n '\n \n+test_expect_success CASE_INSENSITIVE_FS,REFFILES 'F/D conflict on case insensitive filesystem' '\n+\ttest_when_finished rm -rf case_insensitive &&\n+\t(\n+\t\tgit init --bare case_insensitive &&\n+\t\tcd case_insensitive &&\n+\t\tgit remote add origin -- ../case_sensitive_fd &&\n+\t\ttest_must_fail git fetch -f origin \"refs/heads/*:refs/heads/*\" 2>err &&\n+\t\ttest_grep \"failed: refname conflict\" err &&\n+\t\tgit rev-parse refs/heads/main >expect &&\n+\t\tgit rev-parse refs/heads/foo/bar >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n . \"$TEST_DIRECTORY\"/lib-httpd.sh\n start_httpd\n \n\n-- \n2.51.0\n\n"},{"id":"526590","messageId":"20250917-587-git-fetch-1-fails-fetches-on-case-insensitive-repositories-v4-4-da3c74a08ed0@gmail.com","threadId":"64069","inReplyTo":"20250917-587-git-fetch-1-fails-fetches-on-case-insensitive-repositories-v4-0-da3c74a08ed0@gmail.com","subject":"[PATCH v4 4/4] refs/files: handle D/F conflicts during locking","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-09-17T15:25:14Z","receivedAt":"2025-09-17T15:25:29Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"The previous commit added the necessary validation and checks for F/D\nconflicts in the files backend when working on case insensitive systems.\n\nThere is still a possibility for D/F conflicts. This is a different from\nthe F/D since for F/D conflicts, there would not be a conflict during\nthe lock creation phase:\n\n refs/heads/foo.lock\n refs/heads/foo/bar.lock\n\nHowever there would be a conflict when the locks are committed, since we\ncannot have 'refs/heads/foo/bar' and 'refs/heads/foo'. These kinds of\nconflicts are checked and resolved in\n`refs_verify_refnames_available()`, so the previous commit ensured that\nfor case-insensitive filesystems, we would lowercase the inputs to that\nfunction.\n\nFor D/F conflicts, there is a conflict during the lock creation phase\nitself:\n\n refs/heads/foo/bar.lock\n refs/heads/foo.lock\n\nAs in `lock_raw_ref()` after creating the lock, we also check for D/F\nconflicts. This can occur in case-insensitive filesystems when trying to\nfetch case-conflicted references like:\n\n refs/heads/Foo/new\n refs/heads/foo\n\nD/F conflicts can also occur in case-sensitive filesystems, when the\nrepository already contains a directory with a lock file\n'refs/heads/foo/bar.lock' and trying to fetch 'refs/heads/foo'. This\ndoesn't concern directories containing garbage files as those are\nhandled on a higher level.\n\nTo fix this, simply categorize the error as a name conflict. Also remove\nthis reference from the list of valid refnames for availability checks.\nBy categorizing the error and removing it from the list of valid\nreferences, batched updates now knows to reject such reference updates\nand apply the other reference updates.\n\nFix a small typo in `ref_transaction_maybe_set_rejected()` while here.\n\nHelped-by: Junio C Hamano \nSigned-off-by: Karthik Nayak \n---\n refs.c | 9 ++++++++-\n refs/files-backend.c | 11 ++++++-----\n t/t5510-fetch.sh | 46 ++++++++++++++++++++++++++++++++++++++++++++++\n 3 files changed, 60 insertions(+), 6 deletions(-)\n\ndiff --git a/refs.c b/refs.c\nindex 4c1c339ed9..e7109ea5fe 100644\n--- a/refs.c\n+++ b/refs.c\n@@ -1223,7 +1223,7 @@ int ref_transaction_maybe_set_rejected(struct ref_transaction *transaction,\n \t\treturn 0;\n \n \tif (!transaction->rejections)\n-\t\tBUG(\"transaction not inititalized with failure support\");\n+\t\tBUG(\"transaction not initialized with failure support\");\n \n \t/*\n \t * Don't accept generic errors, since these errors are not user\n@@ -1232,6 +1232,13 @@ int ref_transaction_maybe_set_rejected(struct ref_transaction *transaction,\n \tif (err == REF_TRANSACTION_ERROR_GENERIC)\n \t\treturn 0;\n \n+\t/*\n+\t * Rejected refnames shouldn't be considered in the availability\n+\t * checks, so remove them from the list.\n+\t */\n+\tstring_list_remove(&transaction->refnames,\n+\t\t\t transaction->updates[update_idx]->refname, 0);\n+\n \ttransaction->updates[update_idx]->rejection_err = err;\n \tALLOC_GROW(transaction->rejections->update_indices,\n \t\t transaction->rejections->nr + 1,\ndiff --git a/refs/files-backend.c b/refs/files-backend.c\nindex bfdf85121a..4a0343d827 100644\n--- a/refs/files-backend.c\n+++ b/refs/files-backend.c\n@@ -870,6 +870,7 @@ static enum ref_transaction_error lock_raw_ref(struct files_ref_store *refs,\n \t\t\t\tgoto error_return;\n \t\t\t} else if (remove_dir_recursively(&ref_file,\n \t\t\t\t\t\t\t REMOVE_DIR_EMPTY_ONLY)) {\n+\t\t\t\tret = REF_TRANSACTION_ERROR_NAME_CONFLICT;\n \t\t\t\tif (refs_verify_refname_available(\n \t\t\t\t\t\t &refs->base, refname,\n \t\t\t\t\t\t extras, NULL, 0, err)) {\n@@ -877,14 +878,14 @@ static enum ref_transaction_error lock_raw_ref(struct files_ref_store *refs,\n \t\t\t\t\t * The error message set by\n \t\t\t\t\t * verify_refname_available() is OK.\n \t\t\t\t\t */\n-\t\t\t\t\tret = REF_TRANSACTION_ERROR_NAME_CONFLICT;\n \t\t\t\t\tgoto error_return;\n \t\t\t\t} else {\n \t\t\t\t\t/*\n-\t\t\t\t\t * We can't delete the directory,\n-\t\t\t\t\t * but we also don't know of any\n-\t\t\t\t\t * references that it should\n-\t\t\t\t\t * contain.\n+\t\t\t\t\t * Directory conflicts can occur if there\n+\t\t\t\t\t * is an existing lock file in the directory\n+\t\t\t\t\t * or if the filesystem is case-insensitive\n+\t\t\t\t\t * and the directory contains a valid reference\n+\t\t\t\t\t * but conflicts with the update.\n \t\t\t\t\t */\n \t\t\t\t\tstrbuf_addf(err, \"there is a non-empty directory '%s' \"\n \t\t\t\t\t\t \"blocking reference '%s'\",\ndiff --git a/t/t5510-fetch.sh b/t/t5510-fetch.sh\nindex 08dbea6503..6b2739db26 100755\n--- a/t/t5510-fetch.sh\n+++ b/t/t5510-fetch.sh\n@@ -59,6 +59,12 @@ test_expect_success \"clone and setup child repos\" '\n \t\tcd case_sensitive_fd &&\n \t\tgit branch foo/bar &&\n \t\tgit branch Foo\n+\t) &&\n+\tgit clone --ref-format=reftable . case_sensitive_df &&\n+\t(\n+\t\tcd case_sensitive_df &&\n+\t\tgit branch Foo/bar &&\n+\t\tgit branch foo\n \t)\n '\n \n@@ -1592,6 +1598,46 @@ test_expect_success CASE_INSENSITIVE_FS,REFFILES 'F/D conflict on case insensiti\n \t)\n '\n \n+test_expect_success CASE_INSENSITIVE_FS,REFFILES 'D/F conflict on case insensitive filesystem' '\n+\ttest_when_finished rm -rf case_insensitive &&\n+\t(\n+\t\tgit init --bare case_insensitive &&\n+\t\tcd case_insensitive &&\n+\t\tgit remote add origin -- ../case_sensitive_df &&\n+\t\ttest_must_fail git fetch -f origin \"refs/heads/*:refs/heads/*\" 2>err &&\n+\t\ttest_grep \"failed: refname conflict\" err &&\n+\t\tgit rev-parse refs/heads/main >expect &&\n+\t\tgit rev-parse refs/heads/Foo/bar >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success REFFILES 'D/F conflict on case sensitive filesystem with lock' '\n+\t(\n+\t\tgit init --ref-format=reftable base &&\n+\t\tcd base &&\n+\t\techo >file update &&\n+\t\tgit add . &&\n+\t\tgit commit -m \"updated\" &&\n+\t\tgit branch -M main &&\n+\n+\t\tgit update-ref refs/heads/foo @ &&\n+\t\tgit update-ref refs/heads/branch @ &&\n+\t\tcd .. &&\n+\n+\t\tgit init --ref-format=files --bare repo &&\n+\t\tcd repo &&\n+\t\tgit remote add origin ../base &&\n+\t\tmkdir refs/heads/foo &&\n+\t\ttouch refs/heads/foo/random.lock &&\n+\t\ttest_must_fail git fetch origin \"refs/heads/*:refs/heads/*\" 2>err &&\n+\t\ttest_grep \"some local refs could not be updated; try running\" err &&\n+\t\tgit rev-parse refs/heads/main >expect &&\n+\t\tgit rev-parse refs/heads/branch >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n . \"$TEST_DIRECTORY\"/lib-httpd.sh\n start_httpd\n \n\n-- \n2.51.0\n\n"},{"id":"526591","messageId":"CAOLa=ZSaOhUnXqPsWP8MW3AH1EC-rQu7cGtU0W35APZQUhKiBQ@mail.gmail.com","threadId":"64069","inReplyTo":"xmqq5xdhgnbk.fsf@gitster.g","subject":"Re: [PATCH v3 0/4] refs/files: fix issues with git-fetch on case-insensitive FS","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-09-17T15:25:52Z","receivedAt":"2025-09-17T15:25:54Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Junio C Hamano writes:\n\n> Karthik Nayak writes:\n>\n>> Junio C Hamano writes:\n>>\n>>> Karthik Nayak writes:\n>>>\n>>>> Changes in v3:\n>>>> - Rename duplicate_reference_case_cmp() to\n>>>> transaction_has_case_conflicting_update() and add comments.\n>>>> - Improve commit messages.\n>>>> - Add an additional test in the 4th commit to showcase D/F conflicts in\n>>>> case-sensistive file systems.\n>>>> - Link to v2: https://lore.kernel.org/r/20250908-587-git-fetch-1-fails-fetches-on-case-insensitive-repositories-v2-0-b2eb2459befb@gmail.com\n>>>\n>>> I think I like this \"latest first and then historical\" order in the\n>>> cover letter much better than the other way around.\n>>>\n>>> I see that this topic is pretty much done? There still are a few\n>>> questions from Justin's reply that may want to be answered, but I\n>>> have a feeling that the answer to them would not require a new\n>>> iteration.\n>>>\n>>> Looking good. Thanks.\n>>\n>> I did respond to Justin, I think there were a few small nits around\n>> grammar in the commit messages and a question around error reporting.\n>\n> Well, if it is already locally ready, let's have a quick \"small and\n> final\" update before merging it to 'next'.\n>\n> Thanks.\n\nSurely, I've sent in v4.\n\nThanks,\nKarthik\n"},{"id":"526593","messageId":"CAOLa=ZQ7RBhqQCd4zgq4htcORWLxDsDX6t=diN+Z=TmR7H32bw@mail.gmail.com","threadId":"64069","inReplyTo":"CAOLa=ZR1LNs0i5TDPFG_QvUYS3=CMx3ov-68ph6QKNM273JUHQ@mail.gmail.com","subject":"Re: [PATCH v3 1/4] refs/files: catch conflicts on case-insensitive file-systems","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-09-17T15:30:18Z","receivedAt":"2025-09-17T15:30:20Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Karthik Nayak writes:\n\n>>> +\n>>> /*\n>>> * Lock refname, without following symrefs, and set *lock_p to point\n>>> * at a newly-allocated lock object. Fill in lock->old_oid, referent,\n>>> @@ -677,16 +697,17 @@ static void unlock_ref(struct ref_lock *lock)\n>>> * - Generate informative error messages in the case of failure\n>>> */\n>>> static enum ref_transaction_error lock_raw_ref(struct files_ref_store *refs,\n>>> -\t\t\t\t\t struct ref_update *update,\n>>> +\t\t\t\t\t struct ref_transaction *transaction,\n>>> \t\t\t\t\t size_t update_idx,\n>>> \t\t\t\t\t int mustexist,\n>>> \t\t\t\t\t struct string_list *refnames_to_check,\n>>> -\t\t\t\t\t const struct string_list *extras,\n>>> \t\t\t\t\t struct ref_lock **lock_p,\n>>> \t\t\t\t\t struct strbuf *referent,\n>>> \t\t\t\t\t struct strbuf *err)\n>>> {\n>>> \tenum ref_transaction_error ret = REF_TRANSACTION_ERROR_GENERIC;\n>>> +\tstruct ref_update *update = transaction->updates[update_idx];\n>>> +\tconst struct string_list *extras = &transaction->refnames;\n>>> \tconst char *refname = update->refname;\n>>> \tunsigned int *type = &update->type;\n>>> \tstruct ref_lock *lock;\n>>> @@ -776,6 +797,9 @@ static enum ref_transaction_error lock_raw_ref(struct files_ref_store *refs,\n>>> \t\t\tgoto retry;\n>>> \t\t} else {\n>>> \t\t\tunable_to_lock_message(ref_file.buf, myerr, err);\n>>\n>> huh, so if if we have a lockfile error due to a case-insensitve\n>> filesystem, does this mean we print the error message from\n>> `unable_to_lock_message()` and the new message?\n>>\n>> If so, I wonder if we would be better off skipping the former since it\n>> could be a bit misleading.\n>>\n>\n> I would say both are necessary. The errors added here are more technical\n> and really talk about why we faced an issue. The error in\n> 'builtin/fetch.c' is more about guidance to how to overcome that issue.\n>\n> Also this error is client agnostic, so we'd add the error here for users\n> of both regular transactions and batched updates. The error in\n> 'builtin/fetch.c' is very specific to users of 'git-fetch(1)'. So I\n> think both hold value.\n>\n\nMy response was a bit confusing here. To answer your question directly.\nNo we do not print both messages. The error message is appended to the\n`strbuf err` as you see.\n\nThis 'strbuf' is provided by the callee, specifically in 'git-fetch(1)',\nwe only print this error message if the ref transaction failed. Since\n'git-fetch(1)' uses batched updates, the transaction will only fail when\nwe encounter a GENERIC error type. Otherwise, the transaction succeeds\nwith/without rejected updates. After which 'git-fetch(1)' will print out\nspecific messages for rejected updates.\n\nFor example, with the test added in this patch:\n\n $ ~/code/git/build/bin-wrappers/git fetch -f origin\n\"refs/heads/*:refs/heads/*\"\n remote: Enumerating objects: 3, done.\n remote: Counting objects: 100% (3/3), done.\n remote: Total 3 (delta 0), reused 0 (delta 0), pack-reused 0 (from 0)\n Unpacking objects: 100% (3/3), 713 bytes | 713.00 KiB/s, done.\n From ../base\n * [new branch] bRanch1 -> bRanch1\n * [new branch] branch1 -> branch1\n * [new branch] master -> master\n * [new branch] bRanch1 -> origin/bRanch1\n * [new branch] branch1 -> origin/branch1\n * [new branch] master -> origin/master\n error: You're on a case-insensitive filesystem, and the remote you are\n trying to fetch from has references that only differ in casing. It\n is impossible to store such references with the 'files' backend. You\n can either accept this as-is, in which case you won't be able to\n store all remote references on disk. Or you can alternatively\n migrate your repository to use the 'reftable' backend with the\n following command:\n\n git refs migrate --ref-format=reftable\n\n Please keep in mind that not all implementations of Git support this\n new format yet. So if you use tools other than Git to access this\n repository it may not be an option to migrate to reftables.\n\n error: fetching ref refs/remotes/origin/branch1 failed: reference\nconflict due to case-insensitive filesystem\n\nHope that clarifies it!\n"},{"id":"526595","messageId":"eiu2y4r3nncfnmsg3ar3jhzya6ge4dnmwqyrkhsinhcjaxd4qn@ehu6l7xngtrg","threadId":"64069","inReplyTo":"CAOLa=ZR1LNs0i5TDPFG_QvUYS3=CMx3ov-68ph6QKNM273JUHQ@mail.gmail.com","subject":"Re: [PATCH v3 1/4] refs/files: catch conflicts on case-insensitive file-systems","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2025-09-17T15:34:41Z","receivedAt":"2025-09-17T15:34:43Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 25/09/17 12:33AM, Karthik Nayak wrote:\n> Justin Tobler writes:\n> > On 25/09/13 10:54PM, Karthik Nayak wrote:\n> >> +\n> >> /*\n> >> * Lock refname, without following symrefs, and set *lock_p to point\n> >> * at a newly-allocated lock object. Fill in lock->old_oid, referent,\n> >> @@ -677,16 +697,17 @@ static void unlock_ref(struct ref_lock *lock)\n> >> * - Generate informative error messages in the case of failure\n> >> */\n> >> static enum ref_transaction_error lock_raw_ref(struct files_ref_store *refs,\n> >> -\t\t\t\t\t struct ref_update *update,\n> >> +\t\t\t\t\t struct ref_transaction *transaction,\n> >> \t\t\t\t\t size_t update_idx,\n> >> \t\t\t\t\t int mustexist,\n> >> \t\t\t\t\t struct string_list *refnames_to_check,\n> >> -\t\t\t\t\t const struct string_list *extras,\n> >> \t\t\t\t\t struct ref_lock **lock_p,\n> >> \t\t\t\t\t struct strbuf *referent,\n> >> \t\t\t\t\t struct strbuf *err)\n> >> {\n> >> \tenum ref_transaction_error ret = REF_TRANSACTION_ERROR_GENERIC;\n> >> +\tstruct ref_update *update = transaction->updates[update_idx];\n> >> +\tconst struct string_list *extras = &transaction->refnames;\n> >> \tconst char *refname = update->refname;\n> >> \tunsigned int *type = &update->type;\n> >> \tstruct ref_lock *lock;\n> >> @@ -776,6 +797,9 @@ static enum ref_transaction_error lock_raw_ref(struct files_ref_store *refs,\n> >> \t\t\tgoto retry;\n> >> \t\t} else {\n> >> \t\t\tunable_to_lock_message(ref_file.buf, myerr, err);\n> >\n> > huh, so if if we have a lockfile error due to a case-insensitve\n> > filesystem, does this mean we print the error message from\n> > `unable_to_lock_message()` and the new message?\n> >\n> > If so, I wonder if we would be better off skipping the former since it\n> > could be a bit misleading.\n> >\n> \n> I would say both are necessary. The errors added here are more technical\n> and really talk about why we faced an issue. The error in\n> 'builtin/fetch.c' is more about guidance to how to overcome that issue.\n> \n> Also this error is client agnostic, so we'd add the error here for users\n> of both regular transactions and batched updates. The error in\n> 'builtin/fetch.c' is very specific to users of 'git-fetch(1)'. So I\n> think both hold value.\n\nAh ok, I was orginally concerned that both error messages would be\nprinted when a reference update is rejected due to the case conflict\nerror. I see now that the \"Unable to create '%s.lock': %s.\" message only\ngets printed if the transaction actually fails. With this change, a case\nconflict error on an individual references in batched updates doesn't\nresult in the entire transaction failing. Thus, only the new message is\nprinted. Makes sense.\n\nThanks,\n-Justin\n"},{"id":"526597","messageId":"pjvwiyrcdrtpkeyro2mvmfljj6a32u7bhli7ry2cndjhn4pi22@6hzanr46cyij","threadId":"64069","inReplyTo":"20250917-587-git-fetch-1-fails-fetches-on-case-insensitive-repositories-v4-0-da3c74a08ed0@gmail.com","subject":"Re: [PATCH v4 0/4] refs/files: fix issues with git-fetch on case-insensitive FS","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2025-09-17T15:43:02Z","receivedAt":"2025-09-17T15:43:04Z","isPatch":true,"sender":{"key":"jltobler@gmail.com","avatar":"https://avatars.githubusercontent.com/u/53454972?v=4"},"body":"On 25/09/17 05:25PM, Karthik Nayak wrote:\n> Changes in v4:\n> - Fixes to typos in commit messages.\n> - Use curly braces for if..else clause with inlined comments.\n> - Link to v3: https://lore.kernel.org/r/20250913-587-git-fetch-1-fails-fetches-on-case-insensitive-repositories-v3-0-195569740b57@gmail.com\n\nLooking at the range-diff, this version looks good to me.\n\nThanks,\n-Justin\n"},{"id":"526600","messageId":"xmqqecs5dpaa.fsf@gitster.g","threadId":"64069","inReplyTo":"20250917-587-git-fetch-1-fails-fetches-on-case-insensitive-repositories-v4-0-da3c74a08ed0@gmail.com","subject":"Re: [PATCH v4 0/4] refs/files: fix issues with git-fetch on case-insensitive FS","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-09-17T16:20:13Z","receivedAt":"2025-09-17T16:20:16Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Karthik Nayak writes:\n\n> Changes in v4:\n> - Fixes to typos in commit messages.\n> - Use curly braces for if..else clause with inlined comments.\n> - Link to v3: https://lore.kernel.org/r/20250913-587-git-fetch-1-fails-fetches-on-case-insensitive-repositories-v3-0-195569740b57@gmail.com\n\nThanks for a pleasant read. Will replace.\n"},{"id":"526603","messageId":"xmqqjz1xc8nm.fsf@gitster.g","threadId":"64069","inReplyTo":"CAOLa=ZSBpSyU0V08OBrr-QPrZPsYnO1bTZewESY=C7Jy4NTcWQ@mail.gmail.com","subject":"Re: [PATCH v3 0/4] refs/files: fix issues with git-fetch on case-insensitive FS","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-09-17T17:04:45Z","receivedAt":"2025-09-17T17:04:48Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Karthik Nayak writes:\n\n>> I've noticed another issue though, at the moment it's possible to create\n>> a ref like `refs/heads/foo.Lock`. I can image this gives issue when the\n>> remote has `refs/heads/foo` and `refs/heads/foo.Lock` and you pull those\n>> in on a case-insensitive FS. Unfortunately I wasn't able to verify this.\n>> But anyhow, I don't think that any reason to hold back on this current\n>> patch series. I approve.\n>\n> We don't fetch locks from remote. The locking mechanism is simply a\n> construct used to update files locally in race-free manner. Locking a\n> file ensures no other concurrent writes can happen.\n\nI am not Toon, but I think Toon meant \"foo.Lock\" to be a funnily\nnamed but a valid branch the remote has.\n\nDoesn't the remote advertise refs/heads/foo and refs/heads/foo.Lock\nin such a case? And we can ask for both of them. When updating\nfoo, we would locally create \"refs/heads/foo.lock\" and then rename\nit to refs/heads/foo\", right? I think Toon's point was what happens\nwhen the fetch of \"foo.Lock\" from there somehow has completed first.\nOr we can simply fetch foo.Lock branch and then foo branch in a\nseparate invocation of \"git fetch\"---now wouldn't the second\ninvocation have trouble creating the lock file foo.lock for foo on\ncertain filesystems?\n\n> In-short when you fetch references, the prepare stage of the reference\n> transaction will create the necessary lock files. This locks in the\n> updates with guarantee that no other process can update/create the refs.\n> The commit phase simply removes the lock files post updating the refs.\n"},{"id":"526614","messageId":"871po4or7m.fsf@iotcl.com","threadId":"64069","inReplyTo":"pjvwiyrcdrtpkeyro2mvmfljj6a32u7bhli7ry2cndjhn4pi22@6hzanr46cyij","subject":"Re: [PATCH v4 0/4] refs/files: fix issues with git-fetch on case-insensitive FS","fromName":"Toon Claes","fromEmail":"toon@iotcl.com","sentAt":"2025-09-17T18:43:09Z","receivedAt":"2025-09-17T18:43:27Z","isPatch":true,"sender":{"key":"toon@iotcl.com","avatar":"https://avatars.githubusercontent.com/u/121621?v=4"},"body":"Justin Tobler writes:\n\n> Looking at the range-diff, this version looks good to me.\n>\n> Thanks,\n> -Justin\n>\n\nI also have got nothing to add.\n\n-- \nCheers,\nToon\n"},{"id":"526616","messageId":"xmqqtt10c0rr.fsf@gitster.g","threadId":"64069","inReplyTo":"871po4or7m.fsf@iotcl.com","subject":"Re: [PATCH v4 0/4] refs/files: fix issues with git-fetch on case-insensitive FS","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-09-17T19:55:04Z","receivedAt":"2025-09-17T19:55:07Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Toon Claes writes:\n\n> Justin Tobler writes:\n>\n>> Looking at the range-diff, this version looks good to me.\n>>\n>> Thanks,\n>> -Justin\n>>\n>\n> I also have got nothing to add.\n\nThanks, all. Let's declare victory and mark it for 'next' then.\n\n"},{"id":"526629","messageId":"CAOLa=ZSfqcaP1+yxGzY=kEZJPeUqkqm3rO1ak6QihCQ5urT-0A@mail.gmail.com","threadId":"64069","inReplyTo":"xmqqjz1xc8nm.fsf@gitster.g","subject":"Re: [PATCH v3 0/4] refs/files: fix issues with git-fetch on case-insensitive FS","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-09-17T20:40:49Z","receivedAt":"2025-09-17T20:40:51Z","isPatch":true,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"Junio C Hamano writes:\n\n> Karthik Nayak writes:\n>\n>>> I've noticed another issue though, at the moment it's possible to create\n>>> a ref like `refs/heads/foo.Lock`. I can image this gives issue when the\n>>> remote has `refs/heads/foo` and `refs/heads/foo.Lock` and you pull those\n>>> in on a case-insensitive FS. Unfortunately I wasn't able to verify this.\n>>> But anyhow, I don't think that any reason to hold back on this current\n>>> patch series. I approve.\n>>\n>> We don't fetch locks from remote. The locking mechanism is simply a\n>> construct used to update files locally in race-free manner. Locking a\n>> file ensures no other concurrent writes can happen.\n>\n> I am not Toon, but I think Toon meant \"foo.Lock\" to be a funnily\n> named but a valid branch the remote has.\n>\n\nOops. Thanks for explaining.\n\n> Doesn't the remote advertise refs/heads/foo and refs/heads/foo.Lock\n> in such a case? And we can ask for both of them. When updating\n> foo, we would locally create \"refs/heads/foo.lock\" and then rename\n> it to refs/heads/foo\", right?\n\nYeah with batched updates, it would\n\n1. create 'refs/heads/foo.lock'\n2. before creating 'ref/heads/foo.Lock.lock', it would try to verify if\n 'refs/heads/foo.Lock' already exists. Which it does from #1, so not\n create it at all.\n\nTo be honest, I'm not sure what the best way to solve this would be for\ncase-insensitive filesystems using the files backend.\n\n> I think Toon's point was what happens\n> when the fetch of \"foo.Lock\" from there somehow has completed first.\n\nThat shouldn't happen AFAIK since the reference updates are ordered\nlexicographically.\n\n> Or we can simply fetch foo.Lock branch and then foo branch in a\n> separate invocation of \"git fetch\"---now wouldn't the second\n> invocation have trouble creating the lock file foo.lock for foo on\n> certain filesystems?\n\nSo if we:\n\n1. fetch 'refs/heads/foo.Lock' to create 'refs/heads/foo.Lock' locally\n2. fetch 'refs/heads/foo' after\n\n#2 would fail since it would determine that 'refs/heads/foo.lock'\nalready exists. This would be the same behavior without batched updates\ntoo though.\n\n---\n\nOverall, I think the only issue left would be when you try to fetch\n'refs/heads/foo' and 'refs/heads/foo.Lock' together on a\ncase-insensitive filesystem with files backend. For such a system, with\nbatched updates, the latter reference wouldn't be created.\n\nThinking more about it, without batched updates, both would be created,\nsince we create the locks individually in separate transactions. But\nafter the first fetch, we would never be able to update 'refs/heads/foo'\nsince it would always think that a lock file for that reference exists.\nSo it was always broken in some sense. Sigh!\n\n>> In-short when you fetch references, the prepare stage of the reference\n>> transaction will create the necessary lock files. This locks in the\n>> updates with guarantee that no other process can update/create the refs.\n>> The commit phase simply removes the lock files post updating the refs.\n"}]}