{"thread":{"id":"16131","subject":".gitattributes glob matching broken","startedAt":"2008-11-02T16:33:51Z","lastAt":"2008-11-05T03:07:03Z","messageCount":9,"participants":["Hannu Koivisto","Jeff King","Dmitry Potapov","Kelly F. Hickel"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"94669","messageId":"83od0yaxzk.fsf@kalahari.s2.org","threadId":"16131","inReplyTo":null,"subject":".gitattributes glob matching broken","fromName":"Hannu Koivisto","fromEmail":"azure@iki.fi","sentAt":"2008-11-02T16:33:51Z","receivedAt":"2008-11-02T16:33:51Z","isPatch":false,"sender":{"key":"azure@iki.fi","avatar":null},"body":"Greetings,\n\nIt seems that, for example, glob pattern *.s matches files with .sh\nextension at least with checkout and reset --hard but git status\nthinks otherwise:\n\nmkdir test\ncd test\ngit init\necho -e \"*.sh -crlf\\n*.s crlf\" > .gitattributes\necho -e \"foobar\\nfoobar\\nfoobar\" > kala.s\necho -e \"foobar\\nfoobar\\nfoobar\" > kala.sh\ngit add .gitattributes kala.s kala.sh\ngit commit -m \"Foo.\"\ncd ..\ngit clone -n test test2\ncd test2\ngit config core.autocrlf true\ngit checkout\ngit status\n\n# On branch master\n# Changed but not updated:\n#   (use \"git add <file>...\" to update what will be committed)\n#   (use \"git checkout -- <file>...\" to discard changes in working\n# directory)\n#\n#       modified:   kala.sh\n#\nno changes added to commit (use \"git add\" and/or \"git commit -a\")\n\nfile kala.s kala.sh\n\nkala.s:  ASCII text, with CRLF line terminators\nkala.sh: ASCII text, with CRLF line terminators\n\nTested in Linux with git 1.6.0.3.535.g933bb (master as of this\nwriting) but also witnessed in Windows and with slightly older\ngit versions.\n\nThis makes git use in a Windows environment pretty much impossible\nif you don't want to / can't rely on git guessing \"text\"\nvs. \"binary\" files correctly so I hope a solution is found soon.\n\nIt would also be good to document what kind of glob patterns git\nactually supports.  I made the assumption that at least on Linux it\nsupports whatever glob(7) says but even if that assumption is\ncorrect (which it may not be, of course) for example Windows users\nmay not realize to look for such a manual page.\n\n-- \nHannu\n"},{"id":"94731","messageId":"20081103090932.GA18424@coredump.intra.peff.net","threadId":"16131","inReplyTo":"83od0yaxzk.fsf@kalahari.s2.org","subject":"Re: .gitattributes glob matching broken","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2008-11-03T09:09:33Z","receivedAt":"2008-11-03T09:09:33Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sun, Nov 02, 2008 at 06:33:51PM +0200, Hannu Koivisto wrote:\n\n> It seems that, for example, glob pattern *.s matches files with .sh\n> extension at least with checkout and reset --hard but git status\n> thinks otherwise:\n\nI think your analysis is incorrect. I will try to explain what is\nhappening.\n\n> mkdir test\n> cd test\n> git init\n> echo -e \"*.sh -crlf\\n*.s crlf\" > .gitattributes\n> echo -e \"foobar\\nfoobar\\nfoobar\" > kala.s\n> echo -e \"foobar\\nfoobar\\nfoobar\" > kala.sh\n> git add .gitattributes kala.s kala.sh\n> git commit -m \"Foo.\"\n\nOK, so here we have two files, one of which we are telling git is text\nand one of which we are telling git is not text. Since we don't have\nautocrlf set at all, of course nothing happens here.\n\n> git clone -n test test2\n\nAnd here we clone without checking out, so there are no files yet.\n\n> cd test2\n> git config core.autocrlf true\n> git checkout\n\nAnd now we do check out the files, with autocrlf applied. But what are\nwe left with? When I run this, _both_ files were detected as text and\nhave CRLF line endings. So here I think is where git didn't do what you\nexpected: kala.sh should not have had CRLF conversion applied.\n\nThis is a known limitation of the attributes mechanism: it only reads\nfrom .gitattributes in the filesystem (or from .git/info/attributes),\nand not from the tree that is being checked out. This is something that\nshould be addressed, but nobody has stepped up with a patch yet (though\nthere has been some preliminary discussion).\n\n> git status\n> \n> # On branch master\n> # Changed but not updated:\n> #   (use \"git add <file>...\" to update what will be committed)\n> #   (use \"git checkout -- <file>...\" to discard changes in working\n> # directory)\n> #\n> #       modified:   kala.sh\n> #\n> no changes added to commit (use \"git add\" and/or \"git commit -a\")\n\nSo yes, this status makes perfect sense, then. The file \"kala.sh\" has\nCRLFs in the filesystem, but we have told git that it is not a file\nwhich gets converted. So it looks like those CRs have been added.\n\nThe problem, again, is that we have inconsistently applied the\ngitattributes. They were _not_ applied during checkout (because\n.gitattributes did not exist yet), but they _are_ being applied here.\n\nTo \"fix\" this, you can then do a \"git reset --hard\" which will respect\nyour .gitattributes (since it is now checked out). And further file\ncreation and checkout should work OK.\n\n-Peff\n"},{"id":"94767","messageId":"83y700alzf.fsf_-_@kalahari.s2.org","threadId":"16131","inReplyTo":"20081103090932.GA18424@coredump.intra.peff.net","subject":"CRLF support bugs (was: Re: .gitattributes glob matching broken)","fromName":"Hannu Koivisto","fromEmail":"azure@iki.fi","sentAt":"2008-11-03T15:05:24Z","receivedAt":"2008-11-03T15:05:24Z","isPatch":false,"sender":{"key":"azure@iki.fi","avatar":null},"body":"Jeff King <peff@peff.net> writes:\n\n> I think your analysis is incorrect. I will try to explain what is\n> happening.\n\nYes, you are right.  The behaviour I saw in my actual use case was\nso odd that I got completely confused.\n\nI suspect one part of that \"oddness\" was caused by git applying its\nheuristics in checkout as it doesn't use .gitattributes at that\ntime.  For example, it seems that it recognized some of my .sh\nfiles as text files and the rest as binary files.  I suppose I was\ncorrect to assume that it would be stupid to rely on git guessing\nfile type and the only sensible way is to use .gitattributes.  If\nit was supported in checkout too, that is.\n\nI don't know what purpose the autodetection aims to serve but I'd\nadd a big warning in the core.autocrlf documentation about it and\ninstructions on how to configure things so that it is never applied\nbut instead the types must always be specified explicitly.\n\n> The problem, again, is that we have inconsistently applied the\n> gitattributes. They were _not_ applied during checkout (because\n> .gitattributes did not exist yet), but they _are_ being applied here.\n>\n> To \"fix\" this, you can then do a \"git reset --hard\" which will respect\n> your .gitattributes (since it is now checked out). And further file\n> creation and checkout should work OK.\n\nSince I'm trying to launch git in a company environment, I think I\ncan't rely on people remembering to do that.\n\nActually, even if .gitattributes were applied in checkout, I think\nthe whole CRLF support is broken by design because people will have\nto remember to use -n in clone, then enable core.autocrlf support\nand then checkout.  This makes it unneccessarily complicated to\ncreate \"quick local clones\" as well.  You might suggest that\nWindows users should enable core.autocrlf globally but it may not\nbe the right thing to do for all projects/repositories either.\n\nI think CRLF conversion support should have some attribute (be it\n.gitattributes attribute or something else) that is somehow\ninherited from the parent repository.  It would basically say that\n\"you should use platform's native line end type for text files with\nthis repository and its children\".  To go with that, one would\nmaybe have a configuration option to tell what that platform\ndefault line end type is (just in case someone wants to pretend\nCygwin is Unix or something like that).\n\nI also observed this problem:\n\n# Pretend someone does this on Unix\nmkdir test1\ncd test1\ngit init\necho \"*.c crlf\" > .gitattributes\necho -en \"foo\\r\\nfoo\\r\\nfoo\\r\\n\" > kala.c\ngit add .gitattributes kala.c\ngit commit -m \"* Initial checkin.\"\ncd ..\n# Pretend someone else does this on Windows\ngit clone -n test1 test2\ncd test2\ngit config core.autocrlf true\ngit checkout\ngit status\n\n...\n#       modified:   kala.c\n...\n\ngit reset --hard\ngit status\n...\n#       modified:   kala.c\n...\n\nNow, even if .gitattributes were obeyed by checkout, I suspect the end\nresult would be the same(?)  I'm sure someone argues that this makes\nsense.  But try to put yourself in the position of a random Window\nuser.  I think it's far from obvious what is going on and what\nshould be done in this situation.\n\n-- \nHannu\n"},{"id":"94769","messageId":"83skq8al29.fsf@kalahari.s2.org","threadId":"16131","inReplyTo":"83y700alzf.fsf_-_@kalahari.s2.org","subject":"Re: CRLF support bugs","fromName":"Hannu Koivisto","fromEmail":"azure@iki.fi","sentAt":"2008-11-03T15:25:18Z","receivedAt":"2008-11-03T15:25:18Z","isPatch":false,"sender":{"key":"azure@iki.fi","avatar":null},"body":"Hannu Koivisto <azure@iki.fi> writes:\n\n> Actually, even if .gitattributes were applied in checkout, I think\n> the whole CRLF support is broken by design because people will have\n> to remember to use -n in clone, then enable core.autocrlf support\n> and then checkout.  This makes it unneccessarily complicated to\n\nI forgot one thing: so what if someone forgets to use -n or just\nimagines that you can set core.autocrlf afterwards?\n\n# Pretend someone does this on Unix\nmkdir test1\ncd test1\ngit init\necho \"*.c crlf\" > .gitattributes\necho -e \"foo\\nfoo\\nfoo\" > kala.c\ngit add .gitattributes kala.c\ngit commit -m \"Initial checkin.\"\ncd ..\n# Pretend test1 is not a local repository and someone else does this on Windows\ngit clone test1 test2\ncd test2\ngit config core.autocrlf true\ngit status\n\n# On branch master\nnothing to commit (working directory clean)\n\nNow the user would have to know that even though git status claims\neverything is ok, that is not the case.  The user would have to\nknow to say (according to #git):\n\nrm .git/index\ngit reset --hard\n\nJust for the record, when I started to learn git, one of the first\nquestions I had was \"how do I undo checkout?\"  It wasn't until now\nthat I learned I need to remove .git/index (in addition to all\nfiles).\n\n-- \nHannu\n"},{"id":"94779","messageId":"20081103164626.GG21650@dpotapov.dyndns.org","threadId":"16131","inReplyTo":"83y700alzf.fsf_-_@kalahari.s2.org","subject":"Re: CRLF support bugs (was: Re: .gitattributes glob matching broken)","fromName":"Dmitry Potapov","fromEmail":"dpotapov@gmail.com","sentAt":"2008-11-03T16:46:26Z","receivedAt":"2008-11-03T16:46:26Z","isPatch":false,"sender":{"key":"dpotapov@gmail.com","avatar":"https://avatars.githubusercontent.com/u/6568595?v=4"},"body":"On Mon, Nov 03, 2008 at 05:05:24PM +0200, Hannu Koivisto wrote:\n> \n> Actually, even if .gitattributes were applied in checkout, I think\n> the whole CRLF support is broken by design because people will have\n> to remember to use -n in clone, then enable core.autocrlf support\n> and then checkout.  This makes it unneccessarily complicated to\n> create \"quick local clones\" as well.  You might suggest that\n> Windows users should enable core.autocrlf globally but it may not\n> be the right thing to do for all projects/repositories either.\n\ncore.autocrlf was exactly meant to be set globally. Basically,\nit says what end-of-line should be on your system. It is strange\nto have it different for different repositories.\n\n> \n> I think CRLF conversion support should have some attribute (be it\n> .gitattributes attribute or something else) that is somehow\n> inherited from the parent repository.  It would basically say that\n> \"you should use platform's native line end type for text files with\n> this repository and its children\".\n\nIt is already so. All text files are treated accordingly to users\npreferences, and for those files where automatic heuristic produces\nundesirable result, it can be disabled using .gitattributes. (Alas,\nthere is a bug with checkout that you encountered earlier).\n\n> To go with that, one would\n> maybe have a configuration option to tell what that platform\n> default line end type is (just in case someone wants to pretend\n> Cygwin is Unix or something like that).\n\nThat is exactly what core.autocrlf is about. One user may want to\nhave LF on Windows and another wants CRLF. So, core.autocrlf defines\nhow text files should be treated. It is what each user may have in\nhis/her own ~/.gitconfig.\n\n> \n> I also observed this problem:\n> \n> # Pretend someone does this on Unix\n> mkdir test1\n> cd test1\n> git init\n> echo \"*.c crlf\" > .gitattributes\n> echo -en \"foo\\r\\nfoo\\r\\nfoo\\r\\n\" > kala.c\n\nThe 'crlf' attribute means that the file should be treated as 'text'\nwithout applying heuristic. The correct ending for text files on Unix\nis '\\n', not '\\r\\n'.  So, you put a text file with incorrect ending,\nnot surprisingly it causes problems for Windows users later.\n\n\nDmitry\n"},{"id":"94820","messageId":"83mygga1o6.fsf@kalahari.s2.org","threadId":"16131","inReplyTo":"20081103164626.GG21650@dpotapov.dyndns.org","subject":"Re: CRLF support bugs","fromName":"Hannu Koivisto","fromEmail":"azure@iki.fi","sentAt":"2008-11-03T22:24:09Z","receivedAt":"2008-11-03T22:24:09Z","isPatch":false,"sender":{"key":"azure@iki.fi","avatar":null},"body":"Dmitry Potapov <dpotapov@gmail.com> writes:\n\n> On Mon, Nov 03, 2008 at 05:05:24PM +0200, Hannu Koivisto wrote:\n>\n> core.autocrlf was exactly meant to be set globally. Basically,\n> it says what end-of-line should be on your system. It is strange\n> to have it different for different repositories.\n\nMaybe so from the point of view of what it was intended for, but\nsince there is nothing else that could be used to control end of\nline conversion on a repository basis, it certainly doesn't feel\nstrange to me to use it like that.\n\nWhen you clone & checkout, say, the official git repository on\nWindows, are you comfortable doing it with core.autocrlf globally\nset to true?  Maybe you know it's fine to do that so you actually\nare but I'm not.  How about some other random free software\nmainly-Unix project you would like to develop / build under\nWindows?  Even if text vs. binary autodetection worked perfectly\n(and it doesn't), CRLF line ends may still be the wrong choice for\nsome project.  I recall one such project and while admittedly the\nsituation with it may have changed since I last used it, that\ndoesn't change the point.\n\nI certainly wouldn't want to have core.autocrlf globally set to\ntrue on Windows.  No automatic conversion is a much safer default.\nI only want CRLF conversion to happen with projects that have\nactually considered such checkouts and if necessary have been\ncarefully set up to support it by using .gitattributes.\n\n>> I also observed this problem:\n>> \n>> # Pretend someone does this on Unix\n>> mkdir test1\n>> cd test1\n>> git init\n>> echo \"*.c crlf\" > .gitattributes\n>> echo -en \"foo\\r\\nfoo\\r\\nfoo\\r\\n\" > kala.c\n>\n> The 'crlf' attribute means that the file should be treated as 'text'\n> without applying heuristic. The correct ending for text files on Unix\n> is '\\n', not '\\r\\n'.  So, you put a text file with incorrect ending,\n> not surprisingly it causes problems for Windows users later.\n\nIt seems to me you are looking at this, too, from the technical\npoint of view.  Yes, given the way CRLF support is implemented, the\nend result was expected.  But that doesn't mean it was ok from the\nuser's point of view.  Consider usability instead.  A user makes a\nmistake and adds a file from a colleague who uses Windows without\nfirst converting it.  Are you really saying \"so he made a mistake,\nwho cares if repository users face problems\"?  I think it's just\nvery bad usability that by making such a small mistake you cause\nthe system to end up in a state that doesn't make any sense,\ni.e. git claims you have modifications right after clone & checkout\neven though you haven't modified anything.\n\n-- \nHannu\n"},{"id":"94852","messageId":"20081104051432.GD31276@coredump.intra.peff.net","threadId":"16131","inReplyTo":"83y700alzf.fsf_-_@kalahari.s2.org","subject":"Re: CRLF support bugs (was: Re: .gitattributes glob matching broken)","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2008-11-04T05:14:32Z","receivedAt":"2008-11-04T05:14:32Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Nov 03, 2008 at 05:05:24PM +0200, Hannu Koivisto wrote:\n\n> I suspect one part of that \"oddness\" was caused by git applying its\n> heuristics in checkout as it doesn't use .gitattributes at that\n> time.\n\nIt _does_ apply them in checkout, you just didn't have a .gitattributes\nfile yet. So it is part of the same problem.\n\n> For example, it seems that it recognized some of my .sh\n> files as text files and the rest as binary files.  I suppose I was\n> correct to assume that it would be stupid to rely on git guessing\n> file type and the only sensible way is to use .gitattributes.  If\n\nI think it depends on what's in your scripts, since many people have not\nhad trouble with the auto-detection. Perhaps some are UTF-16 which\ncontain NULs?\n\nIf the auto-detection is not working, I am sure people would love to see\nsamples of what fooled it (since it is, after all, just a guess, and we\nwould like to make the guess more accurate).\n\n> > To \"fix\" this, you can then do a \"git reset --hard\" which will respect\n> > your .gitattributes (since it is now checked out). And further file\n> > creation and checkout should work OK.\n> \n> Since I'm trying to launch git in a company environment, I think I\n> can't rely on people remembering to do that.\n\nOh, absolutely. I think this is a shortcoming in git. The reset is\nsimply a workaround until it is actually fixed.\n\n> Actually, even if .gitattributes were applied in checkout, I think\n> the whole CRLF support is broken by design because people will have\n> to remember to use -n in clone, then enable core.autocrlf support\n> and then checkout.  This makes it unneccessarily complicated to\n\nYes, that is a little bit annoying. I think there are four options:\n\n  - people set core.autocrlf in their global ~/.gitconfig. The downside,\n    as you mentioned, is that you might not want it for all projects\n\n  - clone should take an extra \"options\" parameter which can set this up\n    after doing the 'init'. Like:\n\n      git clone -O core.autocrlf=true /path/to/repo\n\n  - after setting autocrlf, people need to tell git to re-checkout with\n    the updated settings. I don't know of a straightforward way to tell\n    git everything needs to be updated. So I would do:\n\n      git ls-files | xargs touch\n      git reset --hard\n\n    which is not ideal. Probably some sort of \"re-checkout\" option to\n    git-checkout would be better.\n\n  - you could do this \"re-checkout\" automagically when core.autocrlf is\n    set via \"git config\". There are two obvious problems with this\n    magic, though:\n\n      - that may not be what the user wants, if they have work in\n        progress in the directory. And normally calling \"git config\"\n        has no such side effects, so it is certainly unexpected.\n\n      - we don't even know when the config is updated, since the user\n        may simply edit the file behind git's back\n\n    So that is a little too magic for my taste.\n\n> I think CRLF conversion support should have some attribute (be it\n> .gitattributes attribute or something else) that is somehow\n> inherited from the parent repository.  It would basically say that\n> \"you should use platform's native line end type for text files with\n> this repository and its children\".  To go with that, one would\n> maybe have a configuration option to tell what that platform\n> default line end type is (just in case someone wants to pretend\n> Cygwin is Unix or something like that).\n\nI think others have complained before about something like this, in that\nit really is a _local_ decision and not a _project_ decision to make. I\nam fortunate enough to work exclusively on platforms with sane line\nendings, so I don't know what is normal.\n\nBut if you really wanted to do such a thing for some set of corporate\nusers, maybe it would make sense to have a \"clone\" hook that runs after\ninit and can set up any relevant config (e.g., by copying certain config\nvalues from the parent repo).\n\n-Peff\n"},{"id":"94889","messageId":"63BEA5E623E09F4D92233FB12A9F79430296676E@emailmn.mqsoftware.com","threadId":"16131","inReplyTo":"20081104051432.GD31276@coredump.intra.peff.net","subject":"RE: CRLF support bugs (was: Re: .gitattributes glob matchingbroken)","fromName":"Kelly F. Hickel","fromEmail":"kfh@mqsoftware.com","sentAt":"2008-11-04T12:37:27Z","receivedAt":"2008-11-04T12:37:27Z","isPatch":false,"sender":{"key":"kfh@mqsoftware.com","avatar":null},"body":"> -----Original Message-----\n> From: git-owner@vger.kernel.org [mailto:git-owner@vger.kernel.org] On\n> Behalf Of Jeff King\n> Sent: Monday, November 03, 2008 11:15 PM\n> To: Hannu Koivisto\n> Cc: git@vger.kernel.org\n> Subject: Re: CRLF support bugs (was: Re: .gitattributes glob\n> matchingbroken)\n> \n> On Mon, Nov 03, 2008 at 05:05:24PM +0200, Hannu Koivisto wrote:\n> \n<snip>\n> > I think CRLF conversion support should have some attribute (be it\n> > .gitattributes attribute or something else) that is somehow\n> > inherited from the parent repository.  It would basically say that\n> > \"you should use platform's native line end type for text files with\n> > this repository and its children\".  To go with that, one would\n> > maybe have a configuration option to tell what that platform\n> > default line end type is (just in case someone wants to pretend\n> > Cygwin is Unix or something like that).\n> \n> I think others have complained before about something like this, in\n> that\n> it really is a _local_ decision and not a _project_ decision to make. I\n> am fortunate enough to work exclusively on platforms with sane line\n> endings, so I don't know what is normal.\n\nFrom my point of view, the factoid that a particular file should be subjected to having its line endings munged is a _project_ decision.  Whether or not to munge them on any given platform is a _local_ decision.\n\nI work on various UNIXes, Linux, Windows, z/OS, etc, etc, and I want the tool to just do the right thing so that I don't have to think about it on a daily basis.\n\nMy $0.02....\n\n-Kelly \n\n> \n> But if you really wanted to do such a thing for some set of corporate\n> users, maybe it would make sense to have a \"clone\" hook that runs after\n> init and can set up any relevant config (e.g., by copying certain\n> config\n> values from the parent repo).\n> \n> -Peff\n"},{"id":"94952","messageId":"20081105030702.GD20907@coredump.intra.peff.net","threadId":"16131","inReplyTo":"63BEA5E623E09F4D92233FB12A9F79430296676E@emailmn.mqsoftware.com","subject":"Re: CRLF support bugs (was: Re: .gitattributes glob matchingbroken)","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2008-11-05T03:07:03Z","receivedAt":"2008-11-05T03:07:03Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Nov 04, 2008 at 06:37:27AM -0600, Kelly F. Hickel wrote:\n\n> From my point of view, the factoid that a particular file should be\n> subjected to having its line endings munged is a _project_ decision.\n> Whether or not to munge them on any given platform is a _local_\n> decision.\n\nNow that you spell it out, I am reminded that that is the argument I\nremember having seen in past discussions. And of course, that argues for\n.gitattributes being carried by the project, but the core.autocrlf\n_config_ being a local decision.\n\nWhich, perhaps not coincidentally, is how it is currently implemented. :)\n\n-Peff\n"}]}