{"thread":{"id":"63690","subject":"[bug] pull --prune could not delete references due to lock file already exists error","startedAt":"2025-06-25T12:32:20Z","lastAt":"2025-07-02T15:38:00Z","messageCount":14,"participants":["ryenus","K Jayatheerth","brian m. carlson","JAYATHEERTH K","Karthik Nayak","Junio C Hamano","Patrick Steinhardt","Chris Torek"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"520682","messageId":"CAKkAvaw0sZ0sW9o_0NZdnZknS8M34UST3PetaPBQj5wwvJyjBA@mail.gmail.com","threadId":"63690","inReplyTo":null,"subject":"[bug] pull --prune could not delete references due to lock file already exists error","fromName":"ryenus","fromEmail":"ryenus@gmail.com","sentAt":"2025-06-25T12:32:07Z","receivedAt":"2025-06-25T12:32:20Z","isPatch":false,"sender":{"key":"ryenus@gmail.com","avatar":"https://avatars.githubusercontent.com/u/610161?v=4"},"body":"today I tried to prune the remote refs in a local repo, but `git pull --prune`\nfailed with an error saying it could not delete references. I retried several\ntimes and it failed consistently, without a single ref pruned.\n\nI found this interesting comment while looking into the code here:\nhttps://github.com/git/git/blob/bd99d6e8db5e2c56dd24395e9711ee7ee564bf4f/refs.c#L2863-L2895\n\n> /*\n>  * Since we don't check the references' old_oids, the\n>  * individual updates can't fail, so we can pack all of the\n>  * updates into a single transaction.\n>  */\n\nThe problem is the assume is wrong, coz things could fail, one such case\nis to have two refs like below:\n\n1. origin/TOM/b1\n2. origin/TOM/b2\n3. origin/tom/b2\n\nNotice there's `TOM` vs `tom` in the path. Another factor is that my local file\nsystem is case insensitive. As a result the 3rd ref would fail and cause the\ntransaction to be always rolled back.\n"},{"id":"520698","messageId":"20250625141849.78834-1-jayatheerthkulkarni2005@gmail.com","threadId":"63690","inReplyTo":"CAKkAvaw0sZ0sW9o_0NZdnZknS8M34UST3PetaPBQj5wwvJyjBA@mail.gmail.com","subject":"Re [bug] pull --prune could not delete references due to lock file already exists error","fromName":"K Jayatheerth","fromEmail":"jayatheerthkulkarni2005@gmail.com","sentAt":"2025-06-25T14:18:49Z","receivedAt":"2025-06-25T14:18:59Z","isPatch":false,"sender":{"key":"jayatheerthkulkarni2005@gmail.com","avatar":"https://avatars.githubusercontent.com/u/148841023?v=4"},"body":"First off thanks for reporting the bug :)\nSo I cannot test this bug as my files system is case sensitive\nbut to just read the code and give a thought in a direction (Assuming that the bug is recreatable)\n\nint refs_delete_refs(struct ref_store *refs, const char *logmsg,\n\t\t     struct string_list *refnames, unsigned int flags)\n{\n...\n\tstruct ref_transaction *transaction;\n...\n\tfor_each_string_list_item(item, refnames) {\n\n\t\ttransaction = ref_store_transaction_begin(refs, 0, &err);\n\t\tif (!transaction) {\n\t\t\twarning(_(\"could not begin transaction to delete %s: %s\"),\n\t\t\t\titem->string, err.buf);\n\t\t\tstrbuf_reset(&err);\n\t\t\tfailures = 1;\n\t\t\tcontinue;\n\t\t}\n\n\t\tret = ref_transaction_delete(transaction, item->string,\n\t\t\t\t\t     NULL, NULL, flags, msg, &err);\n\t\tif (ret) {\n\t\t\twarning(_(\"could not delete reference %s: %s\"),\n\t\t\t\titem->string, err.buf);\n\t\t\tstrbuf_reset(&err);\n\t\t\tref_transaction_free(transaction);\n\t\t\tfailures = 1;\n\t\t\tcontinue;\n\t\t}\n\n\t\tret = ref_transaction_commit(transaction, &err);\n\t\tif (ret) {\n\t\t\twarning(_(\"could not commit deletion of %s: %s\"),\n\t\t\t\titem->string, err.buf);\n\t\t\tstrbuf_reset(&err);\n\t\t\tfailures = 1;\n\t\t}\n\n\t\tref_transaction_free(transaction);\n\t}\n\n\tif (failures)\n\t\tret = -1;\n\n\tstrbuf_release(&err);\n\tfree(msg);\n\treturn ret;\n}\n\nThe original implementation:\nStarts a single transaction using ref_store_transaction_begin().\nAdds all deletions to that transaction.\nCommits the transaction.\nIf any deletion fails, the entire transaction is aborted.\nOn case-insensitive file systems, two refs like:\nmay conflict at the file system level (e.g. both mapped to the same file or directory). \nIf Git tries to delete both in one go, the transaction fails due to a lock file or unlink error.\n(Above are my assumptions till now).\n\nWhat has changed is:\nDeletes each reference in its own transaction\nstruct ref_transaction *transaction = ref_store_transaction_begin(...);\nref_transaction_delete(transaction, ...);\nref_transaction_commit(transaction, ...);\nref_transaction_free(transaction);\nIf one deletion fails due to a case conflict, the others still proceed.\nIt avoids rolling back the entire prune operation just because of a single failure.\nKeeps failure count and returns appropriately\nSignals that something went wrong, but Git can now give partial success feedback.\n\n\nThe question I have is\nIf this approach seems viable or perhaps any solution, \nwould it be possible to write a test case for this scenario?\n\nThank you \n\n- Jayatheerth\n"},{"id":"520820","messageId":"aF7qFEfPi25_e3bq@fruit.crustytoothpaste.net","threadId":"63690","inReplyTo":"20250625141849.78834-1-jayatheerthkulkarni2005@gmail.com","subject":"Re: Re [bug] pull --prune could not delete references due to lock file already exists error","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2025-06-27T18:59:32Z","receivedAt":"2025-06-27T18:59:34Z","isPatch":false,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On 2025-06-25 at 14:18:49, K Jayatheerth wrote:\n> First off thanks for reporting the bug :)\n> So I cannot test this bug as my files system is case sensitive\n> but to just read the code and give a thought in a direction (Assuming that the bug is recreatable)\n\nJust so you know, on Linux, you can create a case-insensitive JFS\npartition on a loopback device and on macOS, you can create a\ncase-insensitive APFS or HFS partition in a disk image file that can\nthen be mounted (I think using `hdiutil` or the directions at [0]).\n\nI have used the former in the rare occasion that I need to test a\ncase-insensitive file system.\n\n[0] https://support.apple.com/en-ca/guide/disk-utility/dskutl11888/mac\n-- \nbrian m. carlson (they/them)\nToronto, Ontario, CA\n"},{"id":"520900","messageId":"CA+rGoLeb_rgHNPu7JAX80eRF4+EiQCyrTR1hbgQtWw8a-D4Nfg@mail.gmail.com","threadId":"63690","inReplyTo":"aF7qFEfPi25_e3bq@fruit.crustytoothpaste.net","subject":"Re: Re [bug] pull --prune could not delete references due to lock file already exists error","fromName":"JAYATHEERTH K","fromEmail":"jayatheerthkulkarni2005@gmail.com","sentAt":"2025-06-30T07:26:12Z","receivedAt":"2025-06-30T07:26:24Z","isPatch":false,"sender":{"key":"jayatheerthkulkarni2005@gmail.com","avatar":"https://avatars.githubusercontent.com/u/148841023?v=4"},"body":"On Sat, Jun 28, 2025 at 12:29 AM brian m. carlson\n<sandals@crustytoothpaste.net> wrote:\n>\n> On 2025-06-25 at 14:18:49, K Jayatheerth wrote:\n> > First off thanks for reporting the bug :)\n> > So I cannot test this bug as my files system is case sensitive\n> > but to just read the code and give a thought in a direction (Assuming that the bug is recreatable)\n>\n> Just so you know, on Linux, you can create a case-insensitive JFS\n> partition on a loopback device and on macOS, you can create a\n> case-insensitive APFS or HFS partition in a disk image file that can\n> then be mounted (I think using `hdiutil` or the directions at [0]).\n>\n> I have used the former in the rare occasion that I need to test a\n> case-insensitive file system.\n>\n\nThanks Brian, that’s really helpful! I’ll try setting up a loopback filesystem\nand use it for testing similar bugs in the future.\nAppreciate the tip!\n\n- Jayatheerth\n"},{"id":"520917","messageId":"CAOLa=ZSA273KGPnwZ2aRBU_ybcCTYogBHvUwpa+5CfDOc2bEWg@mail.gmail.com","threadId":"63690","inReplyTo":"20250625141849.78834-1-jayatheerthkulkarni2005@gmail.com","subject":"Re: Re [bug] pull --prune could not delete references due to lock file already exists error","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-06-30T13:46:35Z","receivedAt":"2025-06-30T13:46:37Z","isPatch":false,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"K Jayatheerth <jayatheerthkulkarni2005@gmail.com> writes:\n\n[snip]\n\n> The original implementation:\n> Starts a single transaction using ref_store_transaction_begin().\n> Adds all deletions to that transaction.\n> Commits the transaction.\n> If any deletion fails, the entire transaction is aborted.\n> On case-insensitive file systems, two refs like:\n> may conflict at the file system level (e.g. both mapped to the same file or directory).\n> If Git tries to delete both in one go, the transaction fails due to a lock file or unlink error.\n> (Above are my assumptions till now).\n>\n> What has changed is:\n> Deletes each reference in its own transaction\n> struct ref_transaction *transaction = ref_store_transaction_begin(...);\n> ref_transaction_delete(transaction, ...);\n> ref_transaction_commit(transaction, ...);\n> ref_transaction_free(transaction);\n> If one deletion fails due to a case conflict, the others still proceed.\n> It avoids rolling back the entire prune operation just because of a single failure.\n> Keeps failure count and returns appropriately\n> Signals that something went wrong, but Git can now give partial success feedback.\n>\n>\n> The question I have is\n> If this approach seems viable or perhaps any solution,\n> would it be possible to write a test case for this scenario?\n>\n\nYou analysis is right. With 'kn/fetch-push-bulk-ref-update' in the works\n(possibly be merged to next soon), we will start using batched updates\nin git-fetch(1) too. Batched updates allow individual updates to fail,\nwhile allowing the transaction as a whole to succeed.\n\nUnfortunately, because our transaction mechanism doesn't handle\nconflicts, we separate out pruning as a pre-step. So this bug would\nstill be present there.\n\nThe issue with the fix you're suggesting is a huge performance drop,\nsince creating individual transaction for each deletion has a lot of\noverhead and the reftable backend would perform a lot worse in such\nsituations.\n\nI can see few solutions overall (including the one you suggested).\n\nOne solution is to drop duplicates in case insensitive systems, this is\nthe shortest and easiest fix for now.\n\nPerhaps something like (untested back of the hand code):\n\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex cc0a3deb61..bc79d74b82 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -1352,10 +1352,16 @@ static int prune_refs(struct display_state\n*display_state,\n \t\t\t\t\tgoto cleanup;\n \t\t\t}\n \t\t} else {\n+\t\t\tconst char *prev;\n \t\t\tstruct string_list refnames = STRING_LIST_INIT_NODUP;\n\n-\t\t\tfor (ref = stale_refs; ref; ref = ref->next)\n+\t\t\tfor (ref = stale_refs; ref; ref = ref->next) {\n+\t\t\t\tif (ignore_case && prev && !strcasecmp(ref->next, prev))\n+\t\t\t\t\tcontinue;\n+\n \t\t\t\tstring_list_append(&refnames, ref->name);\n+\t\t\t\tprev = ref->name;\n+\t\t\t}\n\n \t\t\tresult = refs_delete_refs(get_main_ref_store(the_repository),\n \t\t\t\t\t\t  \"fetch: prune\", &refnames,\n\n\nA bigger and eventual goal is to simply introduce conflict resolution in\nreference transactions. This would allow us to use batched transaction\ntogether for pruning and updating of refs, and using batched\ntransactions would ensure that single reference changes can fail without\nfailing the entire batch.\n\n- Karthik\n"},{"id":"520921","messageId":"aGKdICvmKlumU0ru@fruit.crustytoothpaste.net","threadId":"63690","inReplyTo":"CAOLa=ZSA273KGPnwZ2aRBU_ybcCTYogBHvUwpa+5CfDOc2bEWg@mail.gmail.com","subject":"Re: Re [bug] pull --prune could not delete references due to lock file already exists error","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2025-06-30T14:20:16Z","receivedAt":"2025-06-30T14:20:24Z","isPatch":false,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On 2025-06-30 at 13:46:35, Karthik Nayak wrote:\n> I can see few solutions overall (including the one you suggested).\n> \n> One solution is to drop duplicates in case insensitive systems, this is\n> the shortest and easiest fix for now.\n> \n> Perhaps something like (untested back of the hand code):\n> \n> diff --git a/builtin/fetch.c b/builtin/fetch.c\n> index cc0a3deb61..bc79d74b82 100644\n> --- a/builtin/fetch.c\n> +++ b/builtin/fetch.c\n> @@ -1352,10 +1352,16 @@ static int prune_refs(struct display_state\n> *display_state,\n>  \t\t\t\t\tgoto cleanup;\n>  \t\t\t}\n>  \t\t} else {\n> +\t\t\tconst char *prev;\n>  \t\t\tstruct string_list refnames = STRING_LIST_INIT_NODUP;\n> \n> -\t\t\tfor (ref = stale_refs; ref; ref = ref->next)\n> +\t\t\tfor (ref = stale_refs; ref; ref = ref->next) {\n> +\t\t\t\tif (ignore_case && prev && !strcasecmp(ref->next, prev))\n> +\t\t\t\t\tcontinue;\n> +\n>  \t\t\t\tstring_list_append(&refnames, ref->name);\n> +\t\t\t\tprev = ref->name;\n> +\t\t\t}\n\nThis won't work in the general case, since the two refs that match case\ninsensitively aren't guaranteed to be adjacent.  For instance:\n\n    refs/heads/AAAA\n    refs/heads/AAAB\n    refs/heads/aaaa\n    refs/heads/aaab\n\nThey'll be in the above order for a bytewise comparison, but the\nmatching entries won't be adjacent in the list.\n\nAnother option is for users on case-insensitive systems to use reftable,\nwhich won't have the same problems as the file-based backend and will\npreserve case properly.\n-- \nbrian m. carlson (they/them)\nToronto, Ontario, CA\n"},{"id":"520973","messageId":"xmqq1pr1lyur.fsf@gitster.g","threadId":"63690","inReplyTo":"aGKdICvmKlumU0ru@fruit.crustytoothpaste.net","subject":"Re: Re [bug] pull --prune could not delete references due to lock file already exists error","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-06-30T21:10:36Z","receivedAt":"2025-06-30T21:10:38Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"brian m. carlson\" <sandals@crustytoothpaste.net> writes:\n\n> Another option is for users on case-insensitive systems to use reftable,\n> which won't have the same problems as the file-based backend and will\n> preserve case properly.\n\nThe more guinea-pigs^Wadopters we have, the merrier we are ;-).\n\n\n"},{"id":"520979","messageId":"CA+rGoLd0dhxFx0TuCdrGhRgU0=1imLw+dM8eM6Ca3t9r_Q5XyQ@mail.gmail.com","threadId":"63690","inReplyTo":"CAOLa=ZSA273KGPnwZ2aRBU_ybcCTYogBHvUwpa+5CfDOc2bEWg@mail.gmail.com","subject":"Re: Re [bug] pull --prune could not delete references due to lock file already exists error","fromName":"JAYATHEERTH K","fromEmail":"jayatheerthkulkarni2005@gmail.com","sentAt":"2025-07-01T00:52:47Z","receivedAt":"2025-07-01T00:52:59Z","isPatch":false,"sender":{"key":"jayatheerthkulkarni2005@gmail.com","avatar":"https://avatars.githubusercontent.com/u/148841023?v=4"},"body":"On Mon, Jun 30, 2025 at 7:16 PM Karthik Nayak <karthik.188@gmail.com> wrote:\n>\n> K Jayatheerth <jayatheerthkulkarni2005@gmail.com> writes:\n>\n> [snip]\n>\n> > The original implementation:\n> > Starts a single transaction using ref_store_transaction_begin().\n> > Adds all deletions to that transaction.\n> > Commits the transaction.\n> > If any deletion fails, the entire transaction is aborted.\n> > On case-insensitive file systems, two refs like:\n> > may conflict at the file system level (e.g. both mapped to the same file or directory).\n> > If Git tries to delete both in one go, the transaction fails due to a lock file or unlink error.\n> > (Above are my assumptions till now).\n> >\n> > What has changed is:\n> > Deletes each reference in its own transaction\n> > struct ref_transaction *transaction = ref_store_transaction_begin(...);\n> > ref_transaction_delete(transaction, ...);\n> > ref_transaction_commit(transaction, ...);\n> > ref_transaction_free(transaction);\n> > If one deletion fails due to a case conflict, the others still proceed.\n> > It avoids rolling back the entire prune operation just because of a single failure.\n> > Keeps failure count and returns appropriately\n> > Signals that something went wrong, but Git can now give partial success feedback.\n> >\n> >\n> > The question I have is\n> > If this approach seems viable or perhaps any solution,\n> > would it be possible to write a test case for this scenario?\n> >\n>\n> You analysis is right. With 'kn/fetch-push-bulk-ref-update' in the works\n> (possibly be merged to next soon), we will start using batched updates\n> in git-fetch(1) too. Batched updates allow individual updates to fail,\n> while allowing the transaction as a whole to succeed.\n>\n> Unfortunately, because our transaction mechanism doesn't handle\n> conflicts, we separate out pruning as a pre-step. So this bug would\n> still be present there.\n>\n> The issue with the fix you're suggesting is a huge performance drop,\n> since creating individual transaction for each deletion has a lot of\n> overhead and the reftable backend would perform a lot worse in such\n> situations.\n>\n> I can see few solutions overall (including the one you suggested).\n>\n> One solution is to drop duplicates in case insensitive systems, this is\n> the shortest and easiest fix for now.\n>\n> Perhaps something like (untested back of the hand code):\n>\n> diff --git a/builtin/fetch.c b/builtin/fetch.c\n> index cc0a3deb61..bc79d74b82 100644\n> --- a/builtin/fetch.c\n> +++ b/builtin/fetch.c\n> @@ -1352,10 +1352,16 @@ static int prune_refs(struct display_state\n> *display_state,\n>                                         goto cleanup;\n>                         }\n>                 } else {\n> +                       const char *prev;\n>                         struct string_list refnames = STRING_LIST_INIT_NODUP;\n>\n> -                       for (ref = stale_refs; ref; ref = ref->next)\n> +                       for (ref = stale_refs; ref; ref = ref->next) {\n> +                               if (ignore_case && prev && !strcasecmp(ref->next, prev))\n> +                                       continue;\n> +\n>                                 string_list_append(&refnames, ref->name);\n> +                               prev = ref->name;\n> +                       }\n>\n>                         result = refs_delete_refs(get_main_ref_store(the_repository),\n>                                                   \"fetch: prune\", &refnames,\n>\n>\n> A bigger and eventual goal is to simply introduce conflict resolution in\n> reference transactions. This would allow us to use batched transaction\n> together for pruning and updating of refs, and using batched\n> transactions would ensure that single reference changes can fail without\n> failing the entire batch.\n>\n> - Karthik\n\n\n\n\nAgreed that was a lazy fix.\nI had thought it(multiple transactions) was not a viable solution.\n\nSomething I find interesting is\n\nLoop through the refs marked for deletion (stale_refs)\nIf the filesystem is case-insensitive\nConvert the ref name to lowercase\nCheck if this lowercase version was already seen\nIf it's not a duplicate, store the lowercase name\nAdd the actual (original-cased) ref to the refnames list\nFree up the memory used in seen_refs map\n\n---\n\nstruct strmap seen_refs = STRMAP_INIT;\n...\n\nfor (ref = stale_refs; ref; ref = ref->next) {\n        if (ignore_case) {\n                char *lower = xstrdup(ref->name);\n                str_tolower(lower);\n\n                if (strmap_contains(&seen_refs, lower)) {\n                        /* Skip this ref it would collide on\ncase-insensitive FS */\n                        warning(_(\"Skipping deletion of '%s' due to\ncase-insensitive conflict\"), ref->name);\n                        free(lower);\n                        continue;\n                }\n\n                /* Keep the lowercased key in the map to catch future\nconflicts */\n                strmap_put(&seen_refs, lower, (void *)1);\n        }\n\n        string_list_append(&refnames, ref->name);\n}\n\nstrmap_clear(&seen_refs, 1);\n\n---\n\nwarning(_(\"Skipping deletion of '%s' due to case-insensitive\nconflict\"), ref->name);\nWhen we find duplicates it should work fine I guess.\n\nLet me know if this direction makes sense or if you'd prefer a\ndifferent handling strategy,\nhappy to start sending patches.\n"},{"id":"520996","messageId":"CAOLa=ZSEtd2aHt-j9vOoXGR72g7qbndX3VYkaeD3hKF=pvKfkw@mail.gmail.com","threadId":"63690","inReplyTo":"aGKdICvmKlumU0ru@fruit.crustytoothpaste.net","subject":"Re: Re [bug] pull --prune could not delete references due to lock file already exists error","fromName":"Karthik Nayak","fromEmail":"karthik.188@gmail.com","sentAt":"2025-07-01T08:20:17Z","receivedAt":"2025-07-01T08:20:19Z","isPatch":false,"sender":{"key":"karthik.188@gmail.com","avatar":"https://avatars.githubusercontent.com/u/1786334?v=4"},"body":"\"brian m. carlson\" <sandals@crustytoothpaste.net> writes:\n\n> On 2025-06-30 at 13:46:35, Karthik Nayak wrote:\n>> I can see few solutions overall (including the one you suggested).\n>>\n>> One solution is to drop duplicates in case insensitive systems, this is\n>> the shortest and easiest fix for now.\n>>\n>> Perhaps something like (untested back of the hand code):\n>>\n>> diff --git a/builtin/fetch.c b/builtin/fetch.c\n>> index cc0a3deb61..bc79d74b82 100644\n>> --- a/builtin/fetch.c\n>> +++ b/builtin/fetch.c\n>> @@ -1352,10 +1352,16 @@ static int prune_refs(struct display_state\n>> *display_state,\n>>  \t\t\t\t\tgoto cleanup;\n>>  \t\t\t}\n>>  \t\t} else {\n>> +\t\t\tconst char *prev;\n>>  \t\t\tstruct string_list refnames = STRING_LIST_INIT_NODUP;\n>>\n>> -\t\t\tfor (ref = stale_refs; ref; ref = ref->next)\n>> +\t\t\tfor (ref = stale_refs; ref; ref = ref->next) {\n>> +\t\t\t\tif (ignore_case && prev && !strcasecmp(ref->next, prev))\n>> +\t\t\t\t\tcontinue;\n>> +\n>>  \t\t\t\tstring_list_append(&refnames, ref->name);\n>> +\t\t\t\tprev = ref->name;\n>> +\t\t\t}\n>\n> This won't work in the general case, since the two refs that match case\n> insensitively aren't guaranteed to be adjacent.  For instance:\n>\n>     refs/heads/AAAA\n>     refs/heads/AAAB\n>     refs/heads/aaaa\n>     refs/heads/aaab\n>\n> They'll be in the above order for a bytewise comparison, but the\n> matching entries won't be adjacent in the list.\n>\n\nIndeed. Good catch, this wouldn't work! I guess you'd have to use a\nhashmap in that case, I think that is what Jayatheerth also mentioned in\nhis reply.\n\n> Another option is for users on case-insensitive systems to use reftable,\n> which won't have the same problems as the file-based backend and will\n> preserve case properly.\n\nYes, that's always recommended! But until we start marking the files\nbackend as deprecated, we do have to plug this bug.\n\n> --\n> brian m. carlson (they/them)\n> Toronto, Ontario, CA\n"},{"id":"521006","messageId":"aGO46urHzZTZvDve@pks.im","threadId":"63690","inReplyTo":"xmqq1pr1lyur.fsf@gitster.g","subject":"Re: Re [bug] pull --prune could not delete references due to lock file already exists error","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-07-01T10:31:06Z","receivedAt":"2025-07-01T10:31:14Z","isPatch":false,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, Jun 30, 2025 at 02:10:36PM -0700, Junio C Hamano wrote:\n> \"brian m. carlson\" <sandals@crustytoothpaste.net> writes:\n> \n> > Another option is for users on case-insensitive systems to use reftable,\n> > which won't have the same problems as the file-based backend and will\n> > preserve case properly.\n> \n> The more guinea-pigs^Wadopters we have, the merrier we are ;-).\n\nI'd really like to start thinking about reftables as the default\nbackend. They fix filesystem-specific issues, compress better, are more\nefficient in most (not all) use cases, sometimes significantly so, have\nbetter properties when it comes to repository maintenance. We could for\nexample make \"features.experimental\" enable the reftable backend by\ndefault and add a note to the Git 3.0 breaking changes in that spirit.\n\nI'm quite certain that reftables are stable enough to not cause any\nproblems as used via Git. We have it rolled out to a couple hundred\nthousand of repositories on our staging systems at GitLab, and will soon\nroll them out to production systems. I myself have been running with\nreftables for 1.5 years now and haven't seen a bug after the first\ncouple months.\n\nBut the bigger issue unfortunately is the ecosystem. JGit supports\nreftables, but other implementations like libgit2 and Gitoxide don't. I\npartially got myself to blame here, because I haven't gotten around to\nupdating the libgit2 pull request and am lacking the time to finish that\none. So it's probably a non-starter if all tools that use those libraries\ncannot access such reftable-enabled Git repositories at all anymore.\n\nI bet there's also tons of scripts out there that just reach into the\nfilesystem to do stuff, but that's something that we cannot really help\nwith.\n\nToo bad :/\n\nPatrick\n"},{"id":"521078","messageId":"xmqqtt3vkhwk.fsf@gitster.g","threadId":"63690","inReplyTo":"aGO46urHzZTZvDve@pks.im","subject":"Re: Re [bug] pull --prune could not delete references due to lock file already exists error","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-07-01T16:14:19Z","receivedAt":"2025-07-01T16:14:21Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt <ps@pks.im> writes:\n\n> I'd really like to start thinking about reftables as the default\n> backend. They fix filesystem-specific issues, compress better, are more\n> efficient in most (not all) use cases, sometimes significantly so, have\n> better properties when it comes to repository maintenance. We could for\n> example make \"features.experimental\" enable the reftable backend by\n> default and add a note to the Git 3.0 breaking changes in that spirit.\n\nI am a bit surprised that there is no reference to reftable in the\nbreaking changes document ;-)\n\nI am all for allowing users to opt into it dynamically, and\nfeature.experimental is the ideal mechanism to do so.  Currently it\nsets four configuration variables, and making it five would not make\nit too scary for adventurous users.\n\n> I bet there's also tons of scripts out there that just reach into the\n> filesystem to do stuff, but that's something that we cannot really help\n> with.\n\nI thought you've done enough to make sure that common things people\nwould want to do by direct access to the .git/refs/ hierarchy can be\neasily done with plumbing commands instead, so it would probably be\na matter of writing and publicizing the \"how to migrate to the world\nwhere you cannot write into files under .git/refs/ directory\"\ndocument?\n\nThanks.\n"},{"id":"521147","messageId":"CAPx1Gveenh075k5W-KAKnyJKfdzMNL0iry-043mLB8rKZ_2wuQ@mail.gmail.com","threadId":"63690","inReplyTo":"aGKdICvmKlumU0ru@fruit.crustytoothpaste.net","subject":"Re: Re [bug] pull --prune could not delete references due to lock file already exists error","fromName":"Chris Torek","fromEmail":"chris.torek@gmail.com","sentAt":"2025-07-02T04:50:37Z","receivedAt":"2025-07-02T04:50:52Z","isPatch":false,"sender":{"key":"chris.torek@gmail.com","avatar":"https://avatars.githubusercontent.com/u/16826774?v=4"},"body":"On Mon, Jun 30, 2025 at 7:21 AM brian m. carlson\n<sandals@crustytoothpaste.net> wrote:\n[regarding]\n> > +                             if (ignore_case && prev && !strcasecmp(ref->next, prev))\n> This won't work in the general case, since the two refs that match case\n> insensitively aren't guaranteed to be adjacent.\n\nAlso worth mention: it's not just case-folding that matters.\n\nOn OS X (Macs), path names get \"normalized\" so that the names\n\n   s c h combining-umlaut o n\n\nand\n\n   s c h umlaut-o n\n\nrefer to the *same* file or directory. On a typical Linux/Unix FS, they differ.\n\n(I don't know what Windows does!)\n\nSo, if you have a \"folder-full\" of \"pretty\" German refnames, some\nspelled one way and some another, well...\n\n(It's not clear to me what, if anything, Git should attempt to do here.)\n\nChris\n"},{"id":"521150","messageId":"aGTy8qX9uvwWv_ap@pks.im","threadId":"63690","inReplyTo":"xmqqtt3vkhwk.fsf@gitster.g","subject":"Re: Re [bug] pull --prune could not delete references due to lock file already exists error","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-07-02T08:50:58Z","receivedAt":"2025-07-02T08:51:05Z","isPatch":false,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Jul 01, 2025 at 09:14:19AM -0700, Junio C Hamano wrote:\n> Patrick Steinhardt <ps@pks.im> writes:\n> > I bet there's also tons of scripts out there that just reach into the\n> > filesystem to do stuff, but that's something that we cannot really help\n> > with.\n> \n> I thought you've done enough to make sure that common things people\n> would want to do by direct access to the .git/refs/ hierarchy can be\n> easily done with plumbing commands instead, so it would probably be\n> a matter of writing and publicizing the \"how to migrate to the world\n> where you cannot write into files under .git/refs/ directory\"\n> document?\n\nYeah, all the tools are there. I'm not even sure whether such a guide\nwould be required in the first place -- tools are just what you are used\nto already anyway. So I don't know whether I really need to explain how\nto use git-update-ref(1) and friends.\n\nI am of course biased here, so if disagree and think that this would be\nrequired I might just do it.\n\nAnyway, I'll hack something up and send it to the mailing list soonish\nto get the discussion going.\n\nPatrick\n"},{"id":"521178","messageId":"xmqq5xgafvs9.fsf@gitster.g","threadId":"63690","inReplyTo":"CAPx1Gveenh075k5W-KAKnyJKfdzMNL0iry-043mLB8rKZ_2wuQ@mail.gmail.com","subject":"Re: Re [bug] pull --prune could not delete references due to lock file already exists error","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-07-02T15:37:58Z","receivedAt":"2025-07-02T15:38:00Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Chris Torek <chris.torek@gmail.com> writes:\n\n> On Mon, Jun 30, 2025 at 7:21 AM brian m. carlson\n> <sandals@crustytoothpaste.net> wrote:\n> [regarding]\n>> > +                             if (ignore_case && prev && !strcasecmp(ref->next, prev))\n>> This won't work in the general case, since the two refs that match case\n>> insensitively aren't guaranteed to be adjacent.\n>\n> Also worth mention: it's not just case-folding that matters.\n>\n> On OS X (Macs), path names get \"normalized\" so that the names\n>\n>    s c h combining-umlaut o n\n>\n> and\n>\n>    s c h umlaut-o n\n>\n> refer to the *same* file or directory. On a typical Linux/Unix FS, they differ.\n>\n> (I don't know what Windows does!)\n>\n> So, if you have a \"folder-full\" of \"pretty\" German refnames, some\n> spelled one way and some another, well...\n>\n> (It's not clear to me what, if anything, Git should attempt to do here.)\n\nThe system supplied argv[] is fed to the precompose_argv_prefix()\nhelper in compat/precompose_utf8.c; opendir/readdir/closedir are\nalso wrapped with similar NFD/NFC normalization (really, UTF-8-MAC\nvs UTF-8) helpers defined in the same file.  So the path you read\n(via the opendir/readdir like dir.c does) from the system, or the\npath you are fed from the command line (via argv[]), are normalized\nbefore code in Git above the compat layer even sees them.  The path\nrecoreded in various mechanisms in Git like the index and the tree\nobjects are all normalized.\n\n"}]}