{"thread":{"id":"26582","subject":"RFD: Handling case-colliding filenames on case-insensitive filesystems","startedAt":"2011-02-23T17:11:45Z","lastAt":"2011-02-24T08:50:27Z","messageCount":12,"participants":["Johan Herland","Junio C Hamano","Shawn Pearce","Jay Soffian","Matthieu Moy","Marc Branchaud","Greg Troxel"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"162035","messageId":"201102231811.45948.johan@herland.net","threadId":"26582","inReplyTo":null,"subject":"RFD: Handling case-colliding filenames on case-insensitive filesystems","fromName":"Johan Herland","fromEmail":"johan@herland.net","sentAt":"2011-02-23T17:11:45Z","receivedAt":"2011-02-23T17:11:45Z","isPatch":false,"sender":{"key":"johan@herland.net","avatar":"https://avatars.githubusercontent.com/u/547031?v=4"},"body":"Hi,\n\nAt $dayjob we recently had a problem where a developer pushed a commit \nthat added new files, two of which were named \"foobar.TXT\" \nand \"FOOBAR.txt\". When this commit (or anything based on it) is checked \nout by one of our Windows developers, Git maps two files in its index \nto a single file on the filesystem, and ends up reporting a diff on one \nof those files. The diff won't go away unless one (or both) of the \ncase-colliding files is removed from the repo. Obivously, the \npersisting diff prevents the developer from easily rebasing, switching \nbranches, merging, bisecting and a number of other useful tasks.\n\nThe root of the problem is that the case-colliding files were added in \nthe first place, and this should obviously be prevented in projects \nthat aim to be compatible with case-insensitive filesystems. To that \nend, I'm currently writing an update hook which will prevent \ncase-colliding files from being pushed to our central repo.\n\nHowever, given that this has already happened, how can we design Git to \nhandle this situation more gracefully. In other words, how can we \nbetter handle checking out filenames that collide on case-insensitive \nfilesystems?\n\nMy first idea was to simply refuse checking out trees with \ncase-colliding filenames. I.e. when core.ignoreCase is enabled, we \ncheck whether any of the files we're about to checkout map to the same \nfilesystem representation, and if they do, we abort the checkout and \ncomplain loudly to the user. However, that doesn't really help the user \nat all. Failure to checkout would only make it much harder to fix the \nissue.\n\nA colleague suggested instead that Git should notice that the collision \nwill occur, and work around the failure to represent the repository \nobjects in the file system with a one-to-one match. Either by checking \nout only _one_ of the colliding files, or by using a non-colliding name \nfor the second file. After all, Git already has functionality for \nmanipulating the file contents on checkout (CRLF conversion). Doesn't \nit make sense to add functionality for manipulating the _directory_ \ncontents on checkout as well? Even if that makes sense, I'm not sure \nthat implementing it will be straightforward.\n\nAre there better suggestions on how to deal with this?\n\n\nThanks,\n\n...Johan\n\n-- \nJohan Herland, \nwww.herland.net\n"},{"id":"162041","messageId":"7vmxlm8trt.fsf@alter.siamese.dyndns.org","threadId":"26582","inReplyTo":"201102231811.45948.johan@herland.net","subject":"Re: RFD: Handling case-colliding filenames on case-insensitive filesystems","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-02-23T18:56:06Z","receivedAt":"2011-02-23T18:56:06Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johan Herland writes:\n\n> Are there better suggestions on how to deal with this?\n\nJust from the top off my head, perhaps you can go to the same route as\nsymbolic link support on filesystems that are not symlink-capable?\n"},{"id":"162042","messageId":"AANLkTi=gAM3LGwU47_EkENerZeKmjwuhWhpHZJGSiW=n@mail.gmail.com","threadId":"26582","inReplyTo":"7vmxlm8trt.fsf@alter.siamese.dyndns.org","subject":"Re: RFD: Handling case-colliding filenames on case-insensitive filesystems","fromName":"Shawn Pearce","fromEmail":"spearce@spearce.org","sentAt":"2011-02-23T19:01:03Z","receivedAt":"2011-02-23T19:01:03Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"On Wed, Feb 23, 2011 at 10:56, Junio C Hamano wrote:\n> Johan Herland writes:\n>\n>> Are there better suggestions on how to deal with this?\n>\n> Just from the top off my head, perhaps you can go to the same route as\n> symbolic link support on filesystems that are not symlink-capable?\n\nI don't know how that helps here Junio. On those systems we write a\ntext file holding the symlink contents. That text file name is at\nleast still unique in the working directory.\n\n\nPerhaps instead the \"colliding file\" becomes a directory that stores\nall of the files below it, each with a unique name and a table of\ncontents, e.g.:\n\n foo.txt/\n\n git-contents:\n file-A foo.TXT\n file-B FOO.txt\n\n file-A:\n ... the contents of foo.TXT ..\n\n file-B:\n ... the contents of FOO.txt ..\n\nIt would be hard to work with in the index, and the project's build\nsystem might fail, but at least the user can edit both files using\nnormal tools in the working tree, and can see which one is which using\nthe magic git-contents file.\n\nThis is such an odd corner case though, we need really good tests for\nit, because it won't come up in daily usage very often. :-(\n\n-- \nShawn.\n"},{"id":"162043","messageId":"AANLkTi=MyqXYi=zNZ+MFcUW2p-_icwg0m_aMQpVioT8J@mail.gmail.com","threadId":"26582","inReplyTo":"201102231811.45948.johan@herland.net","subject":"Re: RFD: Handling case-colliding filenames on case-insensitive filesystems","fromName":"Jay Soffian","fromEmail":"jaysoffian@gmail.com","sentAt":"2011-02-23T19:07:35Z","receivedAt":"2011-02-23T19:07:35Z","isPatch":false,"sender":{"key":"jaysoffian@gmail.com","avatar":"https://avatars.githubusercontent.com/u/155970?v=4"},"body":"On Wed, Feb 23, 2011 at 12:11 PM, Johan Herland wrote:\n> A colleague suggested instead that Git should notice that the collision\n> will occur, and work around the failure to represent the repository\n> objects in the file system with a one-to-one match. Either by checking\n> out only _one_ of the colliding files, or by using a non-colliding name\n> for the second file. After all, Git already has functionality for\n> manipulating the file contents on checkout (CRLF conversion). Doesn't\n> it make sense to add functionality for manipulating the _directory_\n> contents on checkout as well? Even if that makes sense, I'm not sure\n> that implementing it will be straightforward.\n>\n> Are there better suggestions on how to deal with this?\n\nThe general problem is aliasing in the working-tree, of which\ncase-insenitivity is the most common form, but it also happens due to\nHFS's use of NFD. A search on gmane for \"insensitive\" or \"nfd\" will\nreturn many hits.\n\nI think the argument against remapping filenames is that it doesn't\nreally help the user.\n\nLet's say (for the sake of argument) that git supported remapping\nbetween the index and the working-tree. Further, my repo has:\n\n$ cat Foo.c\n#include \"Foo.h\"\n\n$ cat foo.c\n#include \"foo.h\"\n\nAnd on a case-insensitive file-system, git has remapped foo.[ch] to\nfoo~2.[ch] for the purposes of avoiding collisions on checkout.\n\nThe checkout can't be compiled correctly, so what's the point of even\nallowing it?\n\n(I'm not saying this is right/wrong, just that's been one of the\narguments against remapping.)\n\nj.\n"},{"id":"162045","messageId":"vpq62sa36ik.fsf@bauges.imag.fr","threadId":"26582","inReplyTo":"AANLkTi=MyqXYi=zNZ+MFcUW2p-_icwg0m_aMQpVioT8J@mail.gmail.com","subject":"Re: RFD: Handling case-colliding filenames on case-insensitive filesystems","fromName":"Matthieu Moy","fromEmail":"matthieu.moy@grenoble-inp.fr","sentAt":"2011-02-23T19:17:23Z","receivedAt":"2011-02-23T19:17:23Z","isPatch":false,"sender":{"key":"matthieu.moy@grenoble-inp.fr","avatar":"https://gravatar.com/avatar/72c8a2705971a25dfaff23cece15130d405685845d911aedd5667ace277f3fc5?d=mp&s=160"},"body":"Jay Soffian writes:\n\n> The checkout can't be compiled correctly, so what's the point of even\n> allowing it?\n\nThere's at least one: allow the user to fix it.\n\nI'm not a user of case-insensitive filesystem, but I guess it must be\nterribly frustrating for a user to have the tool say \"your repo is so\nbroken that I'm not even going to show you what's it in\".\n\nNow, it doesn't solve the problem of people having case-colliding\nfilenames on purpose ...\n\n-- \nMatthieu Moy\nhttp://www-verimag.imag.fr/~moy/\n"},{"id":"162047","messageId":"7vfwre8sax.fsf@alter.siamese.dyndns.org","threadId":"26582","inReplyTo":"AANLkTi=gAM3LGwU47_EkENerZeKmjwuhWhpHZJGSiW=n@mail.gmail.com","subject":"Re: RFD: Handling case-colliding filenames on case-insensitive filesystems","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-02-23T19:27:50Z","receivedAt":"2011-02-23T19:27:50Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Shawn Pearce writes:\n\n> On Wed, Feb 23, 2011 at 10:56, Junio C Hamano wrote:\n>> Johan Herland writes:\n>>\n>>> Are there better suggestions on how to deal with this?\n>>\n>> Just from the top off my head, perhaps you can go to the same route as\n>> symbolic link support on filesystems that are not symlink-capable?\n>\n> I don't know how that helps here Junio. On those systems we write a\n> text file holding the symlink contents. That text file name is at\n> least still unique in the working directory.\n\nHeh, I probably should have more explicitly hinted that I was suggesting\nto rename, e.g. foo-1.txt, when checking out conflicting paths. I chose\nnot to be precise because I knew readers were intelligent enough to read\nwhat's between my lines themselves ;-).\n\nJust like a text file that records link target is useless as a symlink,\nsuch a file would be useless for its original purpose (e.g. renaming\nxt_TCPMSS.c to xt_tcpmss-1.c to avoid a collision with xt_tcpmss.c would\nnot help when its associated Makefile wants to build xt_TCPMSS.o and\nxt_tcpmss.o next to each other), just like your \"treat everybody the same\nway and make that a directory\" approach.\n\nI think two things are sensible to do, are relatively low hanging fruits,\nand are of low risk:\n\n - break checkout on such a tree on incapable filesystems; and\n\n - per project configuration (or attribute given to paths underneath a\n particular directory) that forbids or warns addition of case colliding\n paths to the index; enforce it at write_index() codepath; and\n\n - if we choose to just warn in the second item above instead of downright\n forbidding, barf in cache_tree_update() codepath when the per project\n configuration (or attribute) triggers upon case colliding paths, to\n prevent a commit from being made.\n\nI think \"warn at add time, fail at write-tree time\" is more preferrable,\nas it might be more convenient if you can add hello.c while you still have\nHELLO.c in the index as long as you do not forget to remove HELLO.c from\nthe index before making your next commit.\n"},{"id":"162062","messageId":"4D658FAD.2010505@xiplink.com","threadId":"26582","inReplyTo":"AANLkTi=MyqXYi=zNZ+MFcUW2p-_icwg0m_aMQpVioT8J@mail.gmail.com","subject":"Re: RFD: Handling case-colliding filenames on case-insensitive filesystems","fromName":"Marc Branchaud","fromEmail":"marcnarc@xiplink.com","sentAt":"2011-02-23T22:52:29Z","receivedAt":"2011-02-23T22:52:29Z","isPatch":false,"sender":{"key":"marcnarc@xiplink.com","avatar":"https://avatars.githubusercontent.com/u/14980203?v=4"},"body":"On 11-02-23 02:07 PM, Jay Soffian wrote:\n> On Wed, Feb 23, 2011 at 12:11 PM, Johan Herland wrote:\n>> A colleague suggested instead that Git should notice that the collision\n>> will occur, and work around the failure to represent the repository\n>> objects in the file system with a one-to-one match. Either by checking\n>> out only _one_ of the colliding files, or by using a non-colliding name\n>> for the second file. After all, Git already has functionality for\n>> manipulating the file contents on checkout (CRLF conversion). Doesn't\n>> it make sense to add functionality for manipulating the _directory_\n>> contents on checkout as well? Even if that makes sense, I'm not sure\n>> that implementing it will be straightforward.\n>>\n>> Are there better suggestions on how to deal with this?\n> \n> The general problem is aliasing in the working-tree, of which\n> case-insenitivity is the most common form, but it also happens due to\n> HFS's use of NFD. A search on gmane for \"insensitive\" or \"nfd\" will\n> return many hits.\n> \n> I think the argument against remapping filenames is that it doesn't\n> really help the user.\n> \n> Let's say (for the sake of argument) that git supported remapping\n> between the index and the working-tree. Further, my repo has:\n> \n> $ cat Foo.c\n> #include \"Foo.h\"\n> \n> $ cat foo.c\n> #include \"foo.h\"\n> \n> And on a case-insensitive file-system, git has remapped foo.[ch] to\n> foo~2.[ch] for the purposes of avoiding collisions on checkout.\n> \n> The checkout can't be compiled correctly, so what's the point of even\n> allowing it?\n\nIn our case it would be useful to still have that checkout because the people\nworking on the case-insensitive systems are dealing with a different part of\nthe tree and don't care about the part with the collision.\n\nA build designed to exploit case-sensitivity obviously won't work on a\ncase-insensitive system, but there's no reason to expect a git repo to have a\nsingle, monolithic build. There are a couple of parts of our code tree --\nparts that are out of our control -- that use case sensitive file names, but\nmost of it doesn't. It would be good if git would allow people on\ncase-insensitive systems to work with the repository, if not the complete build.\n\nI suggest:\n\n1. Git should emit a warning when checking out a case-colliding file (or\ndirectory) on a case-insensitive system. I don't really care _what_ gets\nchecked out for that file -- whatever it is ain't gonna work anyway. Let's\nsay it checks out the associated blob the first time it runs across\nthing.foo, but then emits the warning when it tries to check out Thing.Foo.\n\n2. Git should forbid (yes, *forbid*) a user on a case-insensitive system from\nadding any change to any files stored in the repository under\ncase-conflicting names. The error message should basically be \"You need to\nuse a case-sensitive system to work on this file.\"\n\n3. I'm OK with git allowing case-insensitive users to forcibly delete\ncase-conflicting files. \"git rm thing.foo\" should, on case-insensitive\nsystems, fail and display all case-colliding names for\n[tT][hH][iI][nN][gG].[fF][oO][oO], and tell the user to use -f if they really\nwant to delete *all* those files.\n\n\t\tM.\n"},{"id":"162069","messageId":"rmi39newdpp.fsf@fnord.ir.bbn.com","threadId":"26582","inReplyTo":"4D658FAD.2010505@xiplink.com","subject":"Re: RFD: Handling case-colliding filenames on case-insensitive filesystems","fromName":"Greg Troxel","fromEmail":"gdt@ir.bbn.com","sentAt":"2011-02-23T23:09:06Z","receivedAt":"2011-02-23T23:09:06Z","isPatch":false,"sender":{"key":"gdt@ir.bbn.com","avatar":null},"body":"\nMarc Branchaud writes:\n\n> On 11-02-23 02:07 PM, Jay Soffian wrote:\n>\n>> And on a case-insensitive file-system, git has remapped foo.[ch] to\n>> foo~2.[ch] for the purposes of avoiding collisions on checkout.\n>> \n>> The checkout can't be compiled correctly, so what's the point of even\n>> allowing it?\n>\n> In our case it would be useful to still have that checkout because the\n> people working on the case-insensitive systems are dealing with a\n> different part of the tree and don't care about the part with the\n> collision.\n\nAgreed; I've had this problem too. In particular, a repository with\nmultiple packages imported, one of which was a Linux flavor that has\nconflicting names in case-preserving filesystems. The result was an\napparently modified checkout, but the offending files were not\ninteresting to the project. So some sort of remapping, and the\nsubsequent prohibition on modifications (perhaps to either) seems like a\ngood plan.\n\nPerhaps by default the checkout should just error out, but then it would\nbe good to have a variable to instead translate the duplicated names.\n"},{"id":"162086","messageId":"201102240130.05800.johan@herland.net","threadId":"26582","inReplyTo":"201102231811.45948.johan@herland.net","subject":"Re: RFD: Handling case-colliding filenames on case-insensitive filesystems","fromName":"Johan Herland","fromEmail":"johan@herland.net","sentAt":"2011-02-24T00:30:05Z","receivedAt":"2011-02-24T00:30:05Z","isPatch":false,"sender":{"key":"johan@herland.net","avatar":"https://avatars.githubusercontent.com/u/547031?v=4"},"body":"On Wednesday 23 February 2011, Johan Herland wrote:\n> Are there better suggestions on how to deal with this?\n\nJust a small note that I forgot in the first email:\n\nFor the record, this issue has been discussed on stackoverflow and the last comment there suggests an \nalternative way to work around the case-colliding problem without having to \nremove either file from the repo:\n\nUse sparse-checkout to exclude the case-colliding files from the working \ntree.\n\nObviously, this isn't a permanent solution, but I thought I'd just throw it \nout there, as something to consider until a more permanent solution is in \nplace.\n\n\n...Johan\n\n-- \nJohan Herland, \nwww.herland.net\n"},{"id":"162091","messageId":"201102240158.24363.johan@herland.net","threadId":"26582","inReplyTo":"7vfwre8sax.fsf@alter.siamese.dyndns.org","subject":"Re: RFD: Handling case-colliding filenames on case-insensitive filesystems","fromName":"Johan Herland","fromEmail":"johan@herland.net","sentAt":"2011-02-24T00:58:24Z","receivedAt":"2011-02-24T00:58:24Z","isPatch":false,"sender":{"key":"johan@herland.net","avatar":"https://avatars.githubusercontent.com/u/547031?v=4"},"body":"On Wednesday 23 February 2011, Junio C Hamano wrote:\n> I think two things are sensible to do, are relatively low hanging fruits,\n> and are of low risk:\n> \n> - break checkout on such a tree on incapable filesystems; and\n\nWouldn't that be a regression from the current state (where the poor user in \na case-insensitive worktree can at least \"git rm\" the offending files, and \nkeep working without assistance from a case-sensitive worktree)?\n\nWhat about giving a warning on checkout, instead, explaining the problem, \nand advising that - for now - the user can remove the offending files with \n\"git rm\"?\n\n> - per project configuration (or attribute given to paths underneath a\n> particular directory) that forbids or warns addition of case colliding\n> paths to the index; enforce it at write_index() codepath; and\n> \n> - if we choose to just warn in the second item above instead of\n> downright forbidding, barf in cache_tree_update() codepath when the per\n> project configuration (or attribute) triggers upon case colliding paths,\n> to prevent a commit from being made.\n\nI support making this a per-project configuration that will trigger at tree-\ncreation (i.e. commit) time. I would even argue that the default should be \nto warn about (though maybe not refuse) case-colliding filenames, since they \nare either (a) directly harmful for cross-platform projects, or (b) probably \nunwanted in most projects anyway.\n\nHaving a per-project configuration sure beats trying to solve the problem in \na hook script (using \"pre-commit\" introduces the logistical problem making \nsure everybody installs/enables the hook, whereas using \"update\" requires \n(precious) server runtime, triggers too late in the developer's workflow \n(forcing developer to amend/rebase), and probably confuses newbie developers \nas well).\n\n> I think \"warn at add time, fail at write-tree time\" is more preferrable,\n> as it might be more convenient if you can add hello.c while you still\n> have HELLO.c in the index as long as you do not forget to remove HELLO.c\n> from the index before making your next commit.\n\nAgreed.\n\n\n...Johan\n\n-- \nJohan Herland, \nwww.herland.net\n"},{"id":"162095","messageId":"7v8vx643z4.fsf@alter.siamese.dyndns.org","threadId":"26582","inReplyTo":"201102240158.24363.johan@herland.net","subject":"Re: RFD: Handling case-colliding filenames on case-insensitive filesystems","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-02-24T01:26:55Z","receivedAt":"2011-02-24T01:26:55Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johan Herland writes:\n\n> On Wednesday 23 February 2011, Junio C Hamano wrote:\n>> I think two things are sensible to do, are relatively low hanging fruits,\n>> and are of low risk:\n>> \n>> - break checkout on such a tree on incapable filesystems; and\n>\n> Wouldn't that be a regression from the current state (where the poor user in \n> a case-insensitive worktree can at least \"git rm\" the offending files, and \n> keep working without assistance from a case-sensitive worktree)?\n\nDepends on the definition of \"break\". I meant \"exit with non-zero\nstatus\", not necessarily changing what is left in the working tree from\nwhat the current code gives us.\n"},{"id":"162115","messageId":"201102240950.27139.johan@herland.net","threadId":"26582","inReplyTo":"7v8vx643z4.fsf@alter.siamese.dyndns.org","subject":"Re: RFD: Handling case-colliding filenames on case-insensitive filesystems","fromName":"Johan Herland","fromEmail":"johan@herland.net","sentAt":"2011-02-24T08:50:27Z","receivedAt":"2011-02-24T08:50:27Z","isPatch":false,"sender":{"key":"johan@herland.net","avatar":"https://avatars.githubusercontent.com/u/547031?v=4"},"body":"On Thursday 24 February 2011, Junio C Hamano wrote:\n> Johan Herland writes:\n> > On Wednesday 23 February 2011, Junio C Hamano wrote:\n> >> I think two things are sensible to do, are relatively low hanging\n> >> fruits,\n> >> \n> >> and are of low risk:\n> >> - break checkout on such a tree on incapable filesystems; and\n> > \n> > Wouldn't that be a regression from the current state (where the poor\n> > user in a case-insensitive worktree can at least \"git rm\" the\n> > offending files, and keep working without assistance from a\n> > case-sensitive worktree)?\n> \n> Depends on the definition of \"break\". I meant \"exit with non-zero\n> status\", not necessarily changing what is left in the working tree from\n> what the current code gives us.\n\nAh, ok. I have no problem with that.\n\n\n...Johan\n\n-- \nJohan Herland, \nwww.herland.net\n"}]}