{"thread":{"id":"55612","subject":"Bug: Changing folder case with `git mv` crashes on case-insensitive file system","startedAt":"2021-05-03T17:25:57Z","lastAt":"2021-05-06T21:03:31Z","messageCount":14,"participants":["Mark Amery","brian m. carlson","Junio C Hamano","Torsten Bögershausen","Johannes Schindelin","Bagas Sanjaya"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"423511","messageId":"CAD8jeghpOQoibk0xM0QgLsOwLNw9GdM=4rhYuzV-NSkw8LinMQ@mail.gmail.com","threadId":"55612","inReplyTo":null,"subject":"Bug: Changing folder case with `git mv` crashes on case-insensitive file system","fromName":"Mark Amery","fromEmail":"markrobertamery@gmail.com","sentAt":"2021-05-03T17:25:43Z","receivedAt":"2021-05-03T17:25:57Z","isPatch":false,"sender":{"key":"markrobertamery@gmail.com","avatar":null},"body":"Attempting to change the case of a folder's name using a command like\n`git mv foo FOO` crashes on case-insensitive file systems, like the\ndefault APFS used on Apple Macs.\n\nHere are simple steps to repro this:\n\n    $ mkdir testrepo && cd testrepo && git init\n    Initialized empty Git repository in /Users/markamery/testrepo/.git/\n    $ mkdir foo && touch foo/bar && git add foo && git commit -m bla\n    [master (root-commit) a7e9f5f] bla\n    1 file changed, 0 insertions(+), 0 deletions(-)\n    create mode 100644 foo/bar\n    $ git mv foo FOO\n    fatal: renaming 'foo' failed: Invalid argument\n    $ echo $?\n    128\n    $ git status\n    On branch master\n    nothing to commit, working tree clean\n\nIf I create a case-sensitive APFS volume using Disk Utility and try\nthe commands above on that volume, `git mv foo FOO` works correctly:\nit emits no output, exits with a 0 status code, and stages a change\nrenaming `foo/bar` to `FOO/bar`. However, on my main case-insensitive\nvolume, `git mv` behaves as shown above: it exits with code 128,\nprints an \"Invalid argument\" error message, and does not stage any\nchanges.\n\nThe command still fails in the same way if you use `git mv --force`\ninstead of just `git mv`.\n\nNote that previously, `git mv` could not change the case of *file*\nnames on case-insensitive file systems, until that was fixed in commit\nhttps://github.com/git/git/commit/baa37bff9a845471754d3f47957d58a6ccc30058.\nI'm guessing there's a different code path that needs fixing for\nchanging the case of *folders*.\n\nAs far as I can tell, this error has never been reported to the Git\nmailing list, but it seems to be encountered frequently;\nhttps://stackoverflow.com/questions/3011625/git-mv-and-only-change-case-of-directory\nmentions this bug and has 86000 views.\n\nIn case it's relevant, here's my system info as output by `git bugreport`:\n\n    [System Info]\n    git version:\n    git version 2.31.1\n    cpu: x86_64\n    no commit associated with this build\n    sizeof-long: 8\n    sizeof-size_t: 8\n    shell-path: /bin/sh\n    uname: Darwin 18.7.0 Darwin Kernel Version 18.7.0: Mon Apr 27\n20:09:39 PDT 2020; root:xnu-4903.278.35~1/RELEASE_X86_64 x86_64\n    compiler info: clang: 11.0.0 (clang-1100.0.33.17)\n    libc info: no libc information available\n    $SHELL (typically, interactive shell): /bin/bash\n"},{"id":"423543","messageId":"YJCABtAGT7s70jl1@camp.crustytoothpaste.net","threadId":"55612","inReplyTo":"CAD8jeghpOQoibk0xM0QgLsOwLNw9GdM=4rhYuzV-NSkw8LinMQ@mail.gmail.com","subject":"Re: Bug: Changing folder case with `git mv` crashes on case-insensitive file system","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2021-05-03T22:58:14Z","receivedAt":"2021-05-03T22:58:22Z","isPatch":false,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On 2021-05-03 at 17:25:43, Mark Amery wrote:\n> Attempting to change the case of a folder's name using a command like\n> `git mv foo FOO` crashes on case-insensitive file systems, like the\n> default APFS used on Apple Macs.\n> \n> Here are simple steps to repro this:\n> \n>     $ mkdir testrepo && cd testrepo && git init\n>     Initialized empty Git repository in /Users/markamery/testrepo/.git/\n>     $ mkdir foo && touch foo/bar && git add foo && git commit -m bla\n>     [master (root-commit) a7e9f5f] bla\n>     1 file changed, 0 insertions(+), 0 deletions(-)\n>     create mode 100644 foo/bar\n>     $ git mv foo FOO\n>     fatal: renaming 'foo' failed: Invalid argument\n>     $ echo $?\n>     128\n>     $ git status\n>     On branch master\n>     nothing to commit, working tree clean\n> \n> If I create a case-sensitive APFS volume using Disk Utility and try\n> the commands above on that volume, `git mv foo FOO` works correctly:\n> it emits no output, exits with a 0 status code, and stages a change\n> renaming `foo/bar` to `FOO/bar`. However, on my main case-insensitive\n> volume, `git mv` behaves as shown above: it exits with code 128,\n> prints an \"Invalid argument\" error message, and does not stage any\n> changes.\n\nYeah, this is because your operating system returns EINVAL in this case.\nPOSIX specifies EINVAL when you're trying to make a directory a\nsubdirectory of itself.  Which, I mean, I guess is a valid\ninterpretation here, but it of course makes renaming the path needlessly\ndifficult.\n\n> The command still fails in the same way if you use `git mv --force`\n> instead of just `git mv`.\n> \n> Note that previously, `git mv` could not change the case of *file*\n> names on case-insensitive file systems, until that was fixed in commit\n> https://github.com/git/git/commit/baa37bff9a845471754d3f47957d58a6ccc30058.\n> I'm guessing there's a different code path that needs fixing for\n> changing the case of *folders*.\n\nMy guess is this is because when the argument is a directory, the mode\nis set to WORKING_DIRECTORY, and then we check that the mode isn't INDEX\nand so rename(2) gets called.  However, I believe there is a -k option\nwhich should make this work, since we ignore the errors.\n\nI suspect part of the problem here is two fold: on macOS we can't\ndistinguish an attempt to rename the path due to it folding or\ncanonicalizing to the same thing from a real attempt to move an actual\ndirectory into itself.  The latter would be a problem we'd want to\nreport, and the former is not.  Unfortunately, detecting this is\ndifficult because that means we'd have to implement the macOS\ncanonicalization algorithm in Git and we don't want to do that.\n\nMaybe someone who frequently uses a macOS system that uses a\ncase-sensitive file system can chime in here, but for now, I think -k\nshould do what you want.\n-- \nbrian m. carlson (he/him or they/them)\nHouston, Texas, US\n"},{"id":"423561","messageId":"xmqqfsz36u0r.fsf@gitster.g","threadId":"55612","inReplyTo":"YJCABtAGT7s70jl1@camp.crustytoothpaste.net","subject":"Re: Bug: Changing folder case with `git mv` crashes on case-insensitive file system","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-05-04T03:46:12Z","receivedAt":"2021-05-04T03:46:16Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"brian m. carlson\" <sandals@crustytoothpaste.net> writes:\n\n> Yeah, this is because your operating system returns EINVAL in this case.\n> POSIX specifies EINVAL when you're trying to make a directory a\n> subdirectory of itself.  Which, I mean, I guess is a valid\n> interpretation here, but it of course makes renaming the path needlessly\n> difficult.\n> ...\n> I suspect part of the problem here is two fold: on macOS we can't\n> distinguish an attempt to rename the path due to it folding or\n> canonicalizing to the same thing from a real attempt to move an actual\n> directory into itself.  The latter would be a problem we'd want to\n> report, and the former is not.  Unfortunately, detecting this is\n> difficult because that means we'd have to implement the macOS\n> canonicalization algorithm in Git and we don't want to do that.\n\nI agree we'd probably need to resort to macOS specific hack (like we\nhave NFS or Coda specific hacks), but it may not be too bad.\n\nAfter seeing EINVAL, we can lstat src 'foo' and dst 'FOO', and\nrealize that both are directories and have the same st_dev/st_ino,\nwhich should be fairly straightforward, no?\n\nFor that, we do not exactly have to depend on any part of macOS-ism;\nwe do depend on the traditional \"within the same device, inum is a\ngood way to tell if two filesystem entities are the same\".\n\nThe (mis)design of \"git mv a b c d ... DST\" that turns the request\ninto \"mv a DST/a && mv b DST/b && ...\" too early may make the\nfallback behaviour a bit cumbersome to implement (mainly, we need to\nstrip out the '/foo' part out of the failing dst FOO/foo to get the\nreal destination directory 'FOO' the user gave us, before checking\nwith that lstat dance), but since it is an error codepath, we can be\nas inefficient as we like, as long as we are correct ;-)\n\n"},{"id":"423570","messageId":"YJEuBqVVa/7x+jrZ@camp.crustytoothpaste.net","threadId":"55612","inReplyTo":"xmqqfsz36u0r.fsf@gitster.g","subject":"Re: Bug: Changing folder case with `git mv` crashes on case-insensitive file system","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2021-05-04T11:20:38Z","receivedAt":"2021-05-04T11:21:38Z","isPatch":false,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On 2021-05-04 at 03:46:12, Junio C Hamano wrote:\n> \"brian m. carlson\" <sandals@crustytoothpaste.net> writes:\n> \n> > Yeah, this is because your operating system returns EINVAL in this case.\n> > POSIX specifies EINVAL when you're trying to make a directory a\n> > subdirectory of itself.  Which, I mean, I guess is a valid\n> > interpretation here, but it of course makes renaming the path needlessly\n> > difficult.\n> > ...\n> > I suspect part of the problem here is two fold: on macOS we can't\n> > distinguish an attempt to rename the path due to it folding or\n> > canonicalizing to the same thing from a real attempt to move an actual\n> > directory into itself.  The latter would be a problem we'd want to\n> > report, and the former is not.  Unfortunately, detecting this is\n> > difficult because that means we'd have to implement the macOS\n> > canonicalization algorithm in Git and we don't want to do that.\n> \n> I agree we'd probably need to resort to macOS specific hack (like we\n> have NFS or Coda specific hacks), but it may not be too bad.\n> \n> After seeing EINVAL, we can lstat src 'foo' and dst 'FOO', and\n> realize that both are directories and have the same st_dev/st_ino,\n> which should be fairly straightforward, no?\n> \n> For that, we do not exactly have to depend on any part of macOS-ism;\n> we do depend on the traditional \"within the same device, inum is a\n> good way to tell if two filesystem entities are the same\".\n\nYes, although that won't work on Windows, which I don't believe has the\nconcept of inodes and almost certainly has the same problem.  CCing\nDscho in case he has some ideas on how we can make this more resilient\nthere.\n\nIn any event, I'm not planning on writing a patch for this since I have\nno way to test it, but I'm sure someone who uses macOS could probably\nwrite one reasonably easily.\n-- \nbrian m. carlson (he/him or they/them)\nHouston, Texas, US\n"},{"id":"423580","messageId":"20210504151954.aczbvml4rh7t2svc@tb-raspi4","threadId":"55612","inReplyTo":"CAD8jeghpOQoibk0xM0QgLsOwLNw9GdM=4rhYuzV-NSkw8LinMQ@mail.gmail.com","subject":"Re: Bug: Changing folder case with `git mv` crashes on case-insensitive file system","fromName":"Torsten Bögershausen","fromEmail":"tboegi@web.de","sentAt":"2021-05-04T15:19:54Z","receivedAt":"2021-05-04T15:20:00Z","isPatch":false,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"On Mon, May 03, 2021 at 06:25:43PM +0100, Mark Amery wrote:\n> Attempting to change the case of a folder's name using a command like\n> `git mv foo FOO` crashes on case-insensitive file systems, like the\n> default APFS used on Apple Macs.\n>\n> Here are simple steps to repro this:\n>\n>     $ mkdir testrepo && cd testrepo && git init\n>     Initialized empty Git repository in /Users/markamery/testrepo/.git/\n>     $ mkdir foo && touch foo/bar && git add foo && git commit -m bla\n>     [master (root-commit) a7e9f5f] bla\n>     1 file changed, 0 insertions(+), 0 deletions(-)\n>     create mode 100644 foo/bar\n>     $ git mv foo FOO\n>     fatal: renaming 'foo' failed: Invalid argument\n>     $ echo $?\n>     128\n>     $ git status\n>     On branch master\n>     nothing to commit, working tree clean\n>\n> If I create a case-sensitive APFS volume using Disk Utility and try\n> the commands above on that volume, `git mv foo FOO` works correctly:\n> it emits no output, exits with a 0 status code, and stages a change\n> renaming `foo/bar` to `FOO/bar`. However, on my main case-insensitive\n> volume, `git mv` behaves as shown above: it exits with code 128,\n> prints an \"Invalid argument\" error message, and does not stage any\n> changes.\n>\n> The command still fails in the same way if you use `git mv --force`\n> instead of just `git mv`.\n>\n> Note that previously, `git mv` could not change the case of *file*\n> names on case-insensitive file systems, until that was fixed in commit\n> https://github.com/git/git/commit/baa37bff9a845471754d3f47957d58a6ccc30058.\n> I'm guessing there's a different code path that needs fixing for\n> changing the case of *folders*.\n>\n> As far as I can tell, this error has never been reported to the Git\n> mailing list, but it seems to be encountered frequently;\n> https://stackoverflow.com/questions/3011625/git-mv-and-only-change-case-of-directory\n> mentions this bug and has 86000 views.\n>\n> In case it's relevant, here's my system info as output by `git bugreport`:\n>\n>     [System Info]\n>     git version:\n>     git version 2.31.1\n>     cpu: x86_64\n>     no commit associated with this build\n>     sizeof-long: 8\n>     sizeof-size_t: 8\n>     shell-path: /bin/sh\n>     uname: Darwin 18.7.0 Darwin Kernel Version 18.7.0: Mon Apr 27\n> 20:09:39 PDT 2020; root:xnu-4903.278.35~1/RELEASE_X86_64 x86_64\n>     compiler info: clang: 11.0.0 (clang-1100.0.33.17)\n>     libc info: no libc information available\n>     $SHELL (typically, interactive shell): /bin/bash\n\n\nThanks for reporting - that's always good.\n\nTo my undestanding we try to rename\nfoo/ into FOO/.\nBut because FOO/ already \"exists\" as directory,\nGit tries to move foo/ into FOO/foo, which fails.\n\nAnd no, the problem is probably not restricted to MacOs,\nWindows and all case-insenstive file systems should show\nthe same, but I haven't tested yet, so it's more a suspicion.\n\nThe following diff allows to move foo/ into FOO/\nIf someone wants to make a patch out if, that would be good.\n\n\n\ndiff --git a/builtin/mv.c b/builtin/mv.c\nindex 3fccdcb6452..fbf184bcfa9 100644\n--- a/builtin/mv.c\n+++ b/builtin/mv.c\n@@ -163,8 +163,10 @@ int cmd_mv(int argc, const char **argv, const char *prefix)\n \t\tdestination = internal_prefix_pathspec(dest_path[0], argv, argc, DUP_BASENAME);\n \telse if (!lstat(dest_path[0], &st) &&\n \t\t\tS_ISDIR(st.st_mode)) {\n-\t\tdest_path[0] = add_slash(dest_path[0]);\n-\t\tdestination = internal_prefix_pathspec(dest_path[0], argv, argc, DUP_BASENAME);\n+\t\tif (!ignore_case || strcasecmp(source[0], dest_path[0])) {\n+\t\t\tdest_path[0] = add_slash(dest_path[0]);\n+\t\t\tdestination = internal_prefix_pathspec(dest_path[0], argv, argc, DUP_BASENAME);\n+\t\t}\n \t} else {\n \t\tif (argc != 1)\n \t\t\tdie(_(\"destination '%s' is not a directory\"), dest_path[0]);\n@@ -187,9 +189,12 @@ int cmd_mv(int argc, const char **argv, const char *prefix)\n \t\t\t\t(dst[length] == 0 || dst[length] == '/')) {\n \t\t\tbad = _(\"can not move directory into itself\");\n \t\t} else if ((src_is_dir = S_ISDIR(st.st_mode))\n-\t\t\t\t&& lstat(dst, &st) == 0)\n-\t\t\tbad = _(\"cannot move directory over file\");\n-\t\telse if (src_is_dir) {\n+\t\t\t   && lstat(dst, &st) == 0) {\n+\t\t\tif (!ignore_case || strcasecmp(src, dst)){\n+\t\t\t\tbad = _(\"cannot move directory over file\");\n+\t\t\t}\n+\t\t}\n+\t\tif (!bad && src_is_dir) {\n \t\t\tint first = cache_name_pos(src, length), last;\n\n \t\t\tif (first >= 0)\n@@ -277,7 +282,7 @@ int cmd_mv(int argc, const char **argv, const char *prefix)\n \t\tif (mode != INDEX && rename(src, dst) < 0) {\n \t\t\tif (ignore_errors)\n \t\t\t\tcontinue;\n-\t\t\tdie_errno(_(\"renaming '%s' failed\"), src);\n+\t\t\tdie_errno(_(\"renaming '%s' into '%s' failed\"), src, dst);\n \t\t}\n \t\tif (submodule_gitfile[i]) {\n \t\t\tif (!update_path_in_gitmodules(src, dst))\n"},{"id":"423626","messageId":"xmqqtuni58ra.fsf@gitster.g","threadId":"55612","inReplyTo":"20210504151954.aczbvml4rh7t2svc@tb-raspi4","subject":"Re: Bug: Changing folder case with `git mv` crashes on case-insensitive file system","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-05-05T00:23:05Z","receivedAt":"2021-05-05T00:23:09Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Torsten Bögershausen <tboegi@web.de> writes:\n\n> To my undestanding we try to rename\n> foo/ into FOO/.\n> But because FOO/ already \"exists\" as directory,\n> Git tries to move foo/ into FOO/foo, which fails.\n>\n> And no, the problem is probably not restricted to MacOs,\n> Windows and all case-insenstive file systems should show\n> the same, but I haven't tested yet, so it's more a suspicion.\n>\n> The following diff allows to move foo/ into FOO/\n> If someone wants to make a patch out if, that would be good.\n\nIs strcasecmp() sufficient for macOS whose filesystem has not just\ncase insensitivity but UTF-8 normalization issues?\n\n\n"},{"id":"423636","messageId":"YJH/J+ungqbqT3Zb@camp.crustytoothpaste.net","threadId":"55612","inReplyTo":"xmqqtuni58ra.fsf@gitster.g","subject":"Re: Bug: Changing folder case with `git mv` crashes on case-insensitive file system","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2021-05-05T02:12:55Z","receivedAt":"2021-05-05T02:13:36Z","isPatch":false,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On 2021-05-05 at 00:23:05, Junio C Hamano wrote:\n> Torsten Bögershausen <tboegi@web.de> writes:\n> \n> > To my undestanding we try to rename\n> > foo/ into FOO/.\n> > But because FOO/ already \"exists\" as directory,\n> > Git tries to move foo/ into FOO/foo, which fails.\n> >\n> > And no, the problem is probably not restricted to MacOs,\n> > Windows and all case-insenstive file systems should show\n> > the same, but I haven't tested yet, so it's more a suspicion.\n> >\n> > The following diff allows to move foo/ into FOO/\n> > If someone wants to make a patch out if, that would be good.\n> \n> Is strcasecmp() sufficient for macOS whose filesystem has not just\n> case insensitivity but UTF-8 normalization issues?\n\nstrcasecmp is locale sensitive, so I would say it is not a good choice\nhere.  Notably, that means we have differing behavior with \"i\" and \"I\"\nif the locale is Turkish versus English.  Only one of those behaviors\nwill be a correct match for what the kernel does, since it is not locale\nsensitive.  Moreover, on Windows, the version of Unicode used for case\nmapping is written into an NTFS file system when it's created, so a\nfresh install may have different behavior from an upgrade, and\nstrcasecmp won't honor that.\n\nAnd, as you pointed out, it doesn't handle Unicode normalization at all.\n-- \nbrian m. carlson (he/him or they/them)\nHouston, Texas, US\n"},{"id":"423678","messageId":"nycvar.QRO.7.76.6.2105051528030.50@tvgsbejvaqbjf.bet","threadId":"55612","inReplyTo":"YJEuBqVVa/7x+jrZ@camp.crustytoothpaste.net","subject":"Re: Bug: Changing folder case with `git mv` crashes on case-insensitive file system","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2021-05-05T13:51:14Z","receivedAt":"2021-05-05T13:51:30Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Tue, 4 May 2021, brian m. carlson wrote:\n\n> On 2021-05-04 at 03:46:12, Junio C Hamano wrote:\n> > \"brian m. carlson\" <sandals@crustytoothpaste.net> writes:\n> >\n> > > Yeah, this is because your operating system returns EINVAL in this case.\n> > > POSIX specifies EINVAL when you're trying to make a directory a\n> > > subdirectory of itself.  Which, I mean, I guess is a valid\n> > > interpretation here, but it of course makes renaming the path needlessly\n> > > difficult.\n> > > ...\n> > > I suspect part of the problem here is two fold: on macOS we can't\n> > > distinguish an attempt to rename the path due to it folding or\n> > > canonicalizing to the same thing from a real attempt to move an actual\n> > > directory into itself.  The latter would be a problem we'd want to\n> > > report, and the former is not.  Unfortunately, detecting this is\n> > > difficult because that means we'd have to implement the macOS\n> > > canonicalization algorithm in Git and we don't want to do that.\n> >\n> > I agree we'd probably need to resort to macOS specific hack (like we\n> > have NFS or Coda specific hacks), but it may not be too bad.\n> >\n> > After seeing EINVAL, we can lstat src 'foo' and dst 'FOO', and\n> > realize that both are directories and have the same st_dev/st_ino,\n> > which should be fairly straightforward, no?\n> >\n> > For that, we do not exactly have to depend on any part of macOS-ism;\n> > we do depend on the traditional \"within the same device, inum is a\n> > good way to tell if two filesystem entities are the same\".\n>\n> Yes, although that won't work on Windows, which I don't believe has the\n> concept of inodes and almost certainly has the same problem.  CCing\n> Dscho in case he has some ideas on how we can make this more resilient\n> there.\n\nWindows does not really have inodes. But it has what it calls \"file IDs\".\nThis concept is pretty much what you expect on inodes, except on that\nstill-used file system called FAT (for full details, see\nhttps://docs.microsoft.com/en-us/windows/win32/api/fileapi/ns-fileapi-by_handle_file_information):\n\n\tIn the FAT file system, the file ID is generated from the first\n\tcluster of the containing directory and the byte offset within the\n\tdirectory of the entry for the file. Some defragmentation products\n\tchange this byte offset. (Windows in-box defragmentation does\n\tnot.) Thus, a FAT file ID can change over time. Renaming a file in\n\tthe FAT file system can also change the file ID, but only if the\n\tnew file name is longer than the old one.\n\nIn this instance, because we're not actually renaming (yet), the file ID\nwould probably be okay. But in general, we should assume that we do not\nhave inodes on Windows.\n\nThere is another complication: it is not actually cheap to get to that\nfile ID. For performance reasons, we introduced that (optional) FSCache\nfeature in Git for Windows where cache the `lstat()` results because the\nassumption that `lstat()` is fast really only holds true on Linux.\n\nIn fact, what we do is to use `FindFirstFile()`/`FindNextFile()` to\nenumerate an entire directory's worth of `lstat()` data, because funnily\nenough, it is a lot faster when we need an entire directory's worth of\n`lstat()` data anyway (calling `GetFileAttributes()` for individual files\nis of course faster, but not for a dozen files or so).\n\nBut even `GetFileAttributes()` won't get you that \"file ID\". You have to\ncreate a file handle (via `CreateFile()`, which is *expensive*) and then\ncall `GetFileInformationByHandle()`.\n\nNow, way too much of Git's source code still pretends that `lstat()` is\njust this fast operation and we can do it left and right and not say what\nwe _actually_ want to know. That function is called when we need only\nparts of the `lstat()` data. Sometimes it is even used to determine\nwhether a file or directory is present. But since Git does not have proper\nabstraction, `mingw_lstat()` _still_ has to fill in all the information.\n\nSo _if_ we need that file ID information, I would be very much in favor of\nintroducing a proper abstraction, where differentiate between the\nintention (think `get_inode(const char *path)`) from the\nplatform-dependent implementation detail (think `lstat()`, `CreateFile()`\nand `GetFileInformationByHandle()`).\n\nCiao,\nDscho\n\n> In any event, I'm not planning on writing a patch for this since I have\n> no way to test it, but I'm sure someone who uses macOS could probably\n> write one reasonably easily.\n> --\n> brian m. carlson (he/him or they/them)\n> Houston, Texas, US\n>\n"},{"id":"423723","messageId":"xmqqeeek3de1.fsf@gitster.g","threadId":"55612","inReplyTo":"nycvar.QRO.7.76.6.2105051528030.50@tvgsbejvaqbjf.bet","subject":"Re: Bug: Changing folder case with `git mv` crashes on case-insensitive file system","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-05-06T00:38:14Z","receivedAt":"2021-05-06T00:38:19Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:\n\n> So _if_ we need that file ID information, I would be very much in favor of\n> introducing a proper abstraction, where differentiate between the\n> intention (think `get_inode(const char *path)`) from the\n> platform-dependent implementation detail (think `lstat()`, `CreateFile()`\n> and `GetFileInformationByHandle()`).\n\nI agree in principle.  Essentially, we need to\n\n (1) examine all calls to lstat(2) we make in our codebase, and find\n     out what members of \"stat\" are really used out of the result\n     for each callsite.  This will be different from caller to\n     caller (some callers may want only ino, other callers may want\n     ino, size, and mtime, etc.), and we would learn that there are\n     N patterns.\n\n (2) write N abstracted helper functions (or a single helper that\n     takes const char *path, struct stat *, and an enum to tell\n     which one of N patterns this call is about).\n\n (3) replace each lstat(2) call with one of these N abstract helper\n     functions.\n\nget_inode() might be one of these N functions, but what is important\nis that the current callers that want ino and something else should\nnot be penalized by making two separte calls get_inode() and\nget_other_things(), which, when done naively, would result in two\nlstat(2) calls.\n\n\n\n"},{"id":"423738","messageId":"20210506043429.zqgzxjrj643avrns@tb-raspi4","threadId":"55612","inReplyTo":"xmqqtuni58ra.fsf@gitster.g","subject":"Re: Bug: Changing folder case with `git mv` crashes on case-insensitive file system","fromName":"Torsten Bögershausen","fromEmail":"tboegi@web.de","sentAt":"2021-05-06T04:34:29Z","receivedAt":"2021-05-06T04:34:37Z","isPatch":false,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"On Wed, May 05, 2021 at 09:23:05AM +0900, Junio C Hamano wrote:\n> Torsten Bögershausen <tboegi@web.de> writes:\n>\n> > To my undestanding we try to rename\n> > foo/ into FOO/.\n> > But because FOO/ already \"exists\" as directory,\n> > Git tries to move foo/ into FOO/foo, which fails.\n> >\n> > And no, the problem is probably not restricted to MacOs,\n> > Windows and all case-insenstive file systems should show\n> > the same, but I haven't tested yet, so it's more a suspicion.\n> >\n> > The following diff allows to move foo/ into FOO/\n> > If someone wants to make a patch out if, that would be good.\n>\n> Is strcasecmp() sufficient for macOS whose filesystem has not just\n> case insensitivity but UTF-8 normalization issues?\n>\n\nStrictly speaking: no.\n\nThe Git code doesn't handle UTF-8 uppper/lower case at all:\ngit mv bar.txt BAR.TXT works because strcasecmp() is catching it.\n\ngit mv bär.txt BÄR.TXT needs the long way:\ngit mv bär.txt baer.txt && git mv baer.txt BÄR.TXT\n\nWe have been restricting the case-change-is-allowed to ASCII filenames\nall the time.\nThere is no information, which code points map onto each other in Git,\nsince this is all file system dependent.\nNTFS has one way, HFS+, APFS another, VFAT a third one, and if I expose\next4 via SAMBA we probably have another one.\nNot mentioniong that ext4 can be use case-insensitve on later Linux kernels,\nwhich sticks to unicode.\nOr Git repos running on machines using ISO-8859-1, those should be rare these\ndays.\n\nThat said, people are renaming files in ASCII only and are happy,\nand in that sense renaming directories in ASCII can be supported\nwithout major hassle.\n\nAnd the inode approach mentioned as well:\nThis could go on top of strcasecmp() to cover non-ASCII filenames\nor other oddities, if someone implements it.\n\n\n"},{"id":"423761","messageId":"CAD8jeghZKDcp=weHtcMZ4z8KaO1jQJqfPqaRtYgtiwrX-1+NNg@mail.gmail.com","threadId":"55612","inReplyTo":"20210506043429.zqgzxjrj643avrns@tb-raspi4","subject":"Re: Bug: Changing folder case with `git mv` crashes on case-insensitive file system","fromName":"Mark Amery","fromEmail":"markrobertamery@gmail.com","sentAt":"2021-05-06T09:12:40Z","receivedAt":"2021-05-06T09:12:55Z","isPatch":false,"sender":{"key":"markrobertamery@gmail.com","avatar":null},"body":"So, I'm just a dumb Git user who doesn't even write C, so much of this\ndiscussion is over my head, but I have a few thoughts that may be\nhelpful:\n\n• The mv utility on Mac is capable of doing `mv bär.txt bÄr.txt` just\nfine. Maybe `git mv` can learn something from whatever `mv` does?\n\n• On a case-insensitive file system, `git mv somedir sOMEdir` is a\nrename. But on a case-sensitive file system, it might NOT be a rename;\nit might be the case that `somedir` and `sOMEdir` both exist and that\nthe command should put `somedir` inside `sOMEdir`. I mention this\nbecause I can imagine some naive attempts at fixing the original bug\nby doing a case-insensitive comparison of the two names ending up\nbreaking this behaviour on case-sensitive file systems by wrongly\ntreating such a command as a rename. It's probably worth having a test\nthat this scenario gets handled cleanly on case-sensitive file\nsystems? (I haven't checked whether Torsten's proposed diff falls into\nthis trap or not.)\n\n• Above, Torsten mentions that there are filesystem-specific rules\nabout what names are equal to each other that Git can't easily handle,\nbecause they go beyond just ASCII case changes. In that case, maybe\nthe right solution is to always defer the question to the filesystem\nrather than Git trying to figure out the answer \"in its head\"?\n\n  That is: first check the inode or file ID of the src and dst passed\nto `git mv`. If they are different and the second one is a folder,\nmove src inside the existing folder. If either they are the same or\nthe second one is not a folder, then do a rename.\n\n  It seems to me that this approach automatically handles stuff like\n`git mv bär.txt bÄr.txt` plus any other rules about names being equal\n(like two different sequences of code points that both express \"à\"),\nall without Git ever needing to explicitly check whether two names are\ncase-insensitively equal. Am I missing something?\n\nSorry if any of the above is dumb or if I'm reiterating things others\nhave already said without realising it.\n\nOn Thu, May 6, 2021 at 5:34 AM Torsten Bögershausen <tboegi@web.de> wrote:\n>\n> On Wed, May 05, 2021 at 09:23:05AM +0900, Junio C Hamano wrote:\n> > Torsten Bögershausen <tboegi@web.de> writes:\n> >\n> > > To my undestanding we try to rename\n> > > foo/ into FOO/.\n> > > But because FOO/ already \"exists\" as directory,\n> > > Git tries to move foo/ into FOO/foo, which fails.\n> > >\n> > > And no, the problem is probably not restricted to MacOs,\n> > > Windows and all case-insenstive file systems should show\n> > > the same, but I haven't tested yet, so it's more a suspicion.\n> > >\n> > > The following diff allows to move foo/ into FOO/\n> > > If someone wants to make a patch out if, that would be good.\n> >\n> > Is strcasecmp() sufficient for macOS whose filesystem has not just\n> > case insensitivity but UTF-8 normalization issues?\n> >\n>\n> Strictly speaking: no.\n>\n> The Git code doesn't handle UTF-8 uppper/lower case at all:\n> git mv bar.txt BAR.TXT works because strcasecmp() is catching it.\n>\n> git mv bär.txt BÄR.TXT needs the long way:\n> git mv bär.txt baer.txt && git mv baer.txt BÄR.TXT\n>\n> We have been restricting the case-change-is-allowed to ASCII filenames\n> all the time.\n> There is no information, which code points map onto each other in Git,\n> since this is all file system dependent.\n> NTFS has one way, HFS+, APFS another, VFAT a third one, and if I expose\n> ext4 via SAMBA we probably have another one.\n> Not mentioniong that ext4 can be use case-insensitve on later Linux kernels,\n> which sticks to unicode.\n> Or Git repos running on machines using ISO-8859-1, those should be rare these\n> days.\n>\n> That said, people are renaming files in ASCII only and are happy,\n> and in that sense renaming directories in ASCII can be supported\n> without major hassle.\n>\n> And the inode approach mentioned as well:\n> This could go on top of strcasecmp() to cover non-ASCII filenames\n> or other oddities, if someone implements it.\n>\n>\n"},{"id":"423772","messageId":"15ba96cb-c682-f6d2-0bc2-eee4094c243c@gmail.com","threadId":"55612","inReplyTo":"CAD8jeghZKDcp=weHtcMZ4z8KaO1jQJqfPqaRtYgtiwrX-1+NNg@mail.gmail.com","subject":"Re: Bug: Changing folder case with `git mv` crashes on case-insensitive file system","fromName":"Bagas Sanjaya","fromEmail":"bagasdotme@gmail.com","sentAt":"2021-05-06T13:11:10Z","receivedAt":"2021-05-06T13:11:16Z","isPatch":false,"sender":{"key":"bagasdotme@gmail.com","avatar":"https://avatars.githubusercontent.com/u/40219486?v=4"},"body":"On 06/05/21 16.12, Mark Amery wrote:\n\n> • Above, Torsten mentions that there are filesystem-specific rules\n> about what names are equal to each other that Git can't easily handle,\n> because they go beyond just ASCII case changes. In that case, maybe\n> the right solution is to always defer the question to the filesystem\n> rather than Git trying to figure out the answer \"in its head\"?\n> \n>    That is: first check the inode or file ID of the src and dst passed\n> to `git mv`. If they are different and the second one is a folder,\n> move src inside the existing folder. If either they are the same or\n> the second one is not a folder, then do a rename.\n> \n>    It seems to me that this approach automatically handles stuff like\n> `git mv bär.txt bÄr.txt` plus any other rules about names being equal\n> (like two different sequences of code points that both express \"à\"),\n> all without Git ever needing to explicitly check whether two names are\n> case-insensitively equal. Am I missing something?\n\nOn case when src and dst have different inodes and dst is a file, rather\nthan a directory, mv will overwrite dst with contents of src (on Linux).\n\n-- \nAn old man doll... just what I always wanted! - Clara\n"},{"id":"423778","messageId":"20210506145315.rrqcndm7esequ4ok@tb-raspi4","threadId":"55612","inReplyTo":"CAD8jeghZKDcp=weHtcMZ4z8KaO1jQJqfPqaRtYgtiwrX-1+NNg@mail.gmail.com","subject":"Re: Bug: Changing folder case with `git mv` crashes on case-insensitive file system","fromName":"Torsten Bögershausen","fromEmail":"tboegi@web.de","sentAt":"2021-05-06T14:53:15Z","receivedAt":"2021-05-06T14:53:24Z","isPatch":false,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"On Thu, May 06, 2021 at 10:12:40AM +0100, Mark Amery wrote:\n> So, I'm just a dumb Git user who doesn't even write C, so much of this\n> discussion is over my head, but I have a few thoughts that may be\n> helpful:\n>\n> • The mv utility on Mac is capable of doing `mv bär.txt bÄr.txt` just\n> fine. Maybe `git mv` can learn something from whatever `mv` does?\n\nYes, Git can do the same.\nThe thing is, that Git avoids to overwrite of existing files.\nJust to be nice.\nIn that sense, if the destination exists, Git refuses the mv,\nunless you use git mv -f\nHaving said that,\ngit mv bär.txt bÄr.txt will not work (without the -f)\n\ngit mv bear.txt BEAR.txt does work,\n\n>\n> • On a case-insensitive file system, `git mv somedir sOMEdir` is a\n> rename. But on a case-sensitive file system, it might NOT be a rename;\n> it might be the case that `somedir` and `sOMEdir` both exist and that\n> the command should put `somedir` inside `sOMEdir`. I mention this\n> because I can imagine some naive attempts at fixing the original bug\n> by doing a case-insensitive comparison of the two names ending up\n> breaking this behaviour on case-sensitive file systems by wrongly\n> treating such a command as a rename. It's probably worth having a test\n> that this scenario gets handled cleanly on case-sensitive file\n> systems? (I haven't checked whether Torsten's proposed diff falls into\n> this trap or not.)\n\nTests are needed - I should have started with those.\nBut I didn't intend to send a patch (yet), just sharing\nideas and knowledge. Which may enable someone to write a patch.\n\n>\n> • Above, Torsten mentions that there are filesystem-specific rules\n> about what names are equal to each other that Git can't easily handle,\n> because they go beyond just ASCII case changes. In that case, maybe\n> the right solution is to always defer the question to the filesystem\n> rather than Git trying to figure out the answer \"in its head\"?\n\nThere are different trade-offs:\nSo far I am only aware of people asking for the\ngit mv bear.txt BEAR.txt rename.\nJust because they are all SW developers ? I don't know.\nAnd just because SW developers are developping Git,\nthe case-insensitive string compare is good enough,\nit is working for them/us.\nSo things are as they are.\n\n>\n>   That is: first check the inode or file ID of the src and dst passed\n> to `git mv`. If they are different and the second one is a folder,\n> move src inside the existing folder. If either they are the same or\n> the second one is not a folder, then do a rename.\n\nYes. In short: patches are welcome.\nIn long: inodes don't work on Windows (without a major effort)\n\n>\n>   It seems to me that this approach automatically handles stuff like\n> `git mv bär.txt bÄr.txt` plus any other rules about names being equal\n> (like two different sequences of code points that both express \"à\"),\n> all without Git ever needing to explicitly check whether two names are\n> case-insensitively equal. Am I missing something?\n\nThat could be a solution. There may are situations/configurations,\nwhere inodes don't work:\nWhat happens if a Windows server exports a file system to MacOs ?\nTo Linux ?\nDo we have working inodes ?\nWhat about other networking combinations ?\nOur code should handle them well as well.\n\n>\n> Sorry if any of the above is dumb or if I'm reiterating things others\n> have already said without realising it.\n\nNo problem. Actually I realized that we used top-posting here,\nSo I remove the reset to make it more readable.\n\n[]\n"},{"id":"423818","messageId":"xmqqlf8ry3qe.fsf@gitster.g","threadId":"55612","inReplyTo":"CAD8jeghZKDcp=weHtcMZ4z8KaO1jQJqfPqaRtYgtiwrX-1+NNg@mail.gmail.com","subject":"Re: Bug: Changing folder case with `git mv` crashes on case-insensitive file system","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-05-06T21:03:21Z","receivedAt":"2021-05-06T21:03:31Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Mark Amery <markrobertamery@gmail.com> writes:\n\n>   That is: first check the inode or file ID of the src and dst passed\n> to `git mv`. If they are different and the second one is a folder,\n> move src inside the existing folder. If either they are the same or\n> the second one is not a folder, then do a rename.\n>\n>   It seems to me that this approach automatically handles stuff like\n> `git mv bär.txt bÄr.txt` plus any other rules about names being equal\n> (like two different sequences of code points that both express \"à\"),\n> all without Git ever needing to explicitly check whether two names are\n> case-insensitively equal. Am I missing something?\n>\n> Sorry if any of the above is dumb or if I'm reiterating things others\n> have already said without realising it.\n\nThere is nothing dumb about the idea.  I think that is essentially\nwhat brian and I outlined in the earlier exchange, except that we\nneed to use something other than inum on Windows to decide if two\npathnames refer to the same filesystem entity, and the approach we\nsuggested was to use the inum-based check _after_ seeing the current\ncode fail with EINVAL in the error codepath, instead of doing so\nupfront.\n\nThanks.\n\n"}]}