{"thread":{"id":"19315","subject":"Re: Cross-Platform Version Control","startedAt":"2009-05-12T15:06:05Z","lastAt":"2009-06-20T12:14:39Z","messageCount":59,"participants":["Esko Luontola","Shawn O. Pearce","Johannes Schindelin","Jeff King","Dmitry Potapov","Martin Langhoff","Jakub Narebski","Heiko Voigt","Alex Riesen","John Tapsell","Jay Soffian","Nicolas Pitre","Linus Torvalds","Andreas Ericsson","Matthias Andree","Theodore Tso","Daniel Barkalow","Miles Bader","Peter Krefting","Junio C Hamano","Johannes Sixt","Julian Phillips"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"113640","messageId":"419AD153-53B4-4DAB-AF72-4127C17B1CA0@gmail.com","threadId":"19315","inReplyTo":null,"subject":"Re: Cross-Platform Version Control","fromName":"Esko Luontola","fromEmail":"esko.luontola@gmail.com","sentAt":"2009-05-12T15:06:05Z","receivedAt":"2009-05-12T15:06:05Z","isPatch":false,"sender":{"key":"esko.luontola@gmail.com","avatar":"https://gravatar.com/avatar/86ea2beca289ab5e1e70b2d13b48e6c20e0e44478f3afe74b923090517f66a30?d=mp&s=160"},"body":"A good start for making Git cross-platform, would be storing the text \nencoding of every file name and commit message together with the \ncommit. Currently, because Git is oblivious to the encodings and just \nconsiders them as a series of bytes, there is no way to make them \ncross-platform. It's as http://www.joelonsoftware.com/articles/Unicode.html \n says, \"It does not make sense to have a string without knowing what \nencoding it uses.\" Without explicit encoding information, making a \nsystem that works even on the three main platforms, let alone in all \ncountries and languages, is simply not possible.\n\nOn the other hand, if the encoding is explicitly stated in the \nrepository, then it is possible for platform and locale aware Git \nclients to handle the file names and commit messages in whatever way \nmakes most sense for the platform (for example convert the file names \nto the platform's encoding, if it differs from the committer's \nplatform encoding). Then it would also be possible to create a Mac \nversion of Git, which compensates for Mac OS X's file system's file \nname encoding peculiarities. Also the system could then warn (on \"git \nadd\") if the data does not look like it has been encoded with the said \nencoding.\n\nIf the platform's and the repository's encoding happen to be the same \n(which in reality might be possible only inside a small company where \neverybody is forced to use the same OS and is configured by a single \nsysadmin), then no conversions need to be done. Also Git purists, who \nthink that the byte sequence representing a file name are more \nimportant than the human readable version of the file name, may use \nsome configuration switch that disables all conversions - but even \nthen the current encoding should be stored together with the commit.\n\nAre there any plans on storing the encoding information of file names \nand commit messages in the Git repository? How much time would \nimplementing it take? Any ideas on how to maintain backwards \ncompatibility (for old commits that do not have the encoding \ninformation)?\n\n- Esko\n"},{"id":"113641","messageId":"20090512151403.GS30527@spearce.org","threadId":"19315","inReplyTo":"419AD153-53B4-4DAB-AF72-4127C17B1CA0@gmail.com","subject":"Re: Cross-Platform Version Control","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-05-12T15:14:03Z","receivedAt":"2009-05-12T15:14:03Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Esko Luontola wrote:\n> Are there any plans on storing the encoding information of file names \n> and commit messages in the Git repository?\n\nCommit messages already store their encoding in an optional\n\"encoding\" header if the message isn't stored in UTF-8, or\nUS-ASCII, which is a strict subset of UTF-8.\n\nAs for file names, no plans, its a sequence of bytes, but I think a\nlot of people wind up using some subset of US-ASCII for their file\nnames, especially if their project is going to be cross platform.\n\n-- \nShawn.\n"},{"id":"113653","messageId":"alpine.DEB.1.00.0905121808480.4447@intel-tinevez-2-302","threadId":"19315","inReplyTo":"20090512151403.GS30527@spearce.org","subject":"Re: Cross-Platform Version Control","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2009-05-12T16:13:51Z","receivedAt":"2009-05-12T16:13:51Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Tue, 12 May 2009, Shawn O. Pearce wrote:\n\n> Esko Luontola wrote:\n> > Are there any plans on storing the encoding information of file names \n> > and commit messages in the Git repository?\n> \n> Commit messages already store their encoding in an optional \"encoding\" \n> header if the message isn't stored in UTF-8, or US-ASCII, which is a \n> strict subset of UTF-8.\n> \n> As for file names, no plans, its a sequence of bytes, but I think a\n> lot of people wind up using some subset of US-ASCII for their file\n> names, especially if their project is going to be cross platform.\n\nSome context: this issue cropped up in msysGit, of course.\n\nAs to storing all file names in UTF-8, my point about Unicode being not \nnecessarily appropriate for everyone still stands.\n\nUTF-8 _might_ be the de-facto standard for Linux filesystems, but \nIMHO we should not take away the freedom for everybody to decide what they \nwant their file names to be encoded as.\n\nHowever, I see that there might be a need to be able to encode the file \nnames differently, such as on Windows. IMHO the best solution would be \na config variable controlling the reencoding of file names.\n\nFor some time, it looked as if two people were interested in implementing \nsomething like that (Peter and Robin IIRC), but efforts have stalled.\n\nCiao,\nDscho\n"},{"id":"113656","messageId":"20090512161638.GB29566@coredump.intra.peff.net","threadId":"19315","inReplyTo":"20090512151403.GS30527@spearce.org","subject":"Re: Cross-Platform Version Control","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2009-05-12T16:16:38Z","receivedAt":"2009-05-12T16:16:38Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, May 12, 2009 at 08:14:03AM -0700, Shawn O. Pearce wrote:\n\n> As for file names, no plans, its a sequence of bytes, but I think a\n> lot of people wind up using some subset of US-ASCII for their file\n> names, especially if their project is going to be cross platform.\n\nOr they use a single encoding like utf8 so that there are no surprises.\nYou can still run into normalization problems with filenames on some\nfilesystems, though. Linus's name_hash code sets up the framework to\nhandle \"these two names are actually equivalent\", but right now I think\nthere is just code for handling case-sensitivity, not utf8 normalization\n(but I just skimmed the code, so I might be wrong).\n\n-Peff\n"},{"id":"113667","messageId":"alpine.DEB.1.00.0905121856250.4447@intel-tinevez-2-302","threadId":"19315","inReplyTo":"20090512161638.GB29566@coredump.intra.peff.net","subject":"Re: Cross-Platform Version Control","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2009-05-12T16:57:47Z","receivedAt":"2009-05-12T16:57:47Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Tue, 12 May 2009, Jeff King wrote:\n\n> On Tue, May 12, 2009 at 08:14:03AM -0700, Shawn O. Pearce wrote:\n> \n> > As for file names, no plans, its a sequence of bytes, but I think a \n> > lot of people wind up using some subset of US-ASCII for their file \n> > names, especially if their project is going to be cross platform.\n> \n> Or they use a single encoding like utf8 so that there are no surprises. \n> You can still run into normalization problems with filenames on some \n> filesystems, though. Linus's name_hash code sets up the framework to \n> handle \"these two names are actually equivalent\", but right now I think \n> there is just code for handling case-sensitivity, not utf8 normalization \n> (but I just skimmed the code, so I might be wrong).\n\nBack then I actually started on a patch to make Git capable of determining \nUTF-8 equivalence, but at the same time somebody started such an annoying \nmail thread that I stopped working on the issue completely.\n\nCiao,\nDscho\n"},{"id":"113675","messageId":"D1E08DA2-8D8E-4D45-A50A-F32E3448D6E9@gmail.com","threadId":"19315","inReplyTo":"alpine.DEB.1.00.0905121808480.4447@intel-tinevez-2-302","subject":"Re: Cross-Platform Version Control","fromName":"Esko Luontola","fromEmail":"esko.luontola@gmail.com","sentAt":"2009-05-12T17:56:55Z","receivedAt":"2009-05-12T17:56:55Z","isPatch":false,"sender":{"key":"esko.luontola@gmail.com","avatar":"https://gravatar.com/avatar/86ea2beca289ab5e1e70b2d13b48e6c20e0e44478f3afe74b923090517f66a30?d=mp&s=160"},"body":"On 12.5.2009, at 19:13, Johannes Schindelin wrote:\n> As to storing all file names in UTF-8, my point about Unicode being \n> not\n> necessarily appropriate for everyone still stands.\n>\n> UTF-8 _might_ be the de-facto standard for Linux filesystems, but\n> IMHO we should not take away the freedom for everybody to decide \n> what they\n> want their file names to be encoded as.\n>\n> However, I see that there might be a need to be able to encode the \n> file\n> names differently, such as on Windows. IMHO the best solution would \n> be\n> a config variable controlling the reencoding of file names.\n\nExactly. The system should not force the use of a specific encoding. \nIt should only offer a recommendation, but be also fully compatible if \nthe user uses some other encoding.\n\nThat's why it's best to always store the information about what \nencoding was used. It shouldn't matter, whether the data is encoded \nwith ISO-8859-1, UTF-8, Shift_JIS, Big5 or some other encoding, as \nlong as it is explicitly said that what the encoding is. Then the \nreader of the data can best decide, how to show that data on the \ncurrent platform.\n\nA config variable for defining, that what encoding should be used when \ncommitting the file names, would make sense. Git should also try to \nautodetect, that what encoding is used in its current environment. In \nthe case of UTF-8, you should also be able to specify which \nnormalization form is used (http://www.unicode.org/unicode/reports/ \ntr15/), or whether it is normalized at all.\n\nFor example, it should be possible to configure Git so, that when a \nfile is checked out on Mac, its file name is converted to the current \nfile system's encoding (UTF-8 NFD, I think), and when the file is \ncommitted on Mac, the file name is normalized back to the same UTF-8 \nform as is used on Linux (UTF-8 NFC).\n\nIt would be nice to have config variables for saying, that all file \nnames in this repository must use UTF-8 NFC, and all commit messages \nmust use UTF-8 NFC (with Unix newlines). Then the Git client would \nautodetect the current environment's encoding, and convert the text, \nif necessary, to match the repository's encoding.\n\n- Esko\n"},{"id":"113681","messageId":"20090512182828.GA26812@dpotapov.dyndns.org","threadId":"19315","inReplyTo":"419AD153-53B4-4DAB-AF72-4127C17B1CA0@gmail.com","subject":"Re: Cross-Platform Version Control","fromName":"Dmitry Potapov","fromEmail":"dpotapov@gmail.com","sentAt":"2009-05-12T18:28:28Z","receivedAt":"2009-05-12T18:28:28Z","isPatch":false,"sender":{"key":"dpotapov@gmail.com","avatar":"https://avatars.githubusercontent.com/u/6568595?v=4"},"body":"On Tue, May 12, 2009 at 06:06:05PM +0300, Esko Luontola wrote:\n> A good start for making Git cross-platform, would be storing the text \n> encoding of every file name and commit message together with the commit. \n> Currently, because Git is oblivious to the encodings and just considers \n> them as a series of bytes, there is no way to make them cross-platform. \n\n1. Git already stores the endcoding for all commit messages that are not\n in UTF-8.\n\n2. If you really want to be cross-platform portable, you should not use\n any characters in filenames outside of [A-Za-z0-9._-] (i.e. Portable\n Filename Character Set)\n http://www.opengroup.org/onlinepubs/000095399/basedefs/xbd_chap03.html#tag_03_276\n\n\nDmitry\n"},{"id":"113682","messageId":"46a038f90905121140n61902c61qc3cd41100efee4ae@mail.gmail.com","threadId":"19315","inReplyTo":"20090512182828.GA26812@dpotapov.dyndns.org","subject":"Re: Cross-Platform Version Control","fromName":"Martin Langhoff","fromEmail":"martin.langhoff@gmail.com","sentAt":"2009-05-12T18:40:20Z","receivedAt":"2009-05-12T18:40:20Z","isPatch":false,"sender":{"key":"martin.langhoff@gmail.com","avatar":"https://gravatar.com/avatar/1e3f311b6c4c15836501901ca58f8c0b0667246488084ba524d8bc9867e22fd9?d=mp&s=160"},"body":"On Tue, May 12, 2009 at 8:28 PM, Dmitry Potapov wrote:\n> 2. If you really want to be cross-platform portable, you should not use\n>   any characters in filenames outside of [A-Za-z0-9._-] (i.e. Portable\n>   Filename Character Set)\n>   http://www.opengroup.org/onlinepubs/000095399/basedefs/xbd_chap03.html#tag_03_276\n\nWould it make sense to have warnings at 'git add' time about\n\n - filenames outside of that charset (as the strictest mode, perhaps\neven default)\n - filenames that have a potential conflict wrt case-sensitivity\n - filenames that have potential conflict in the same tree due to\nutf-8 encoding vagaries\n\nMHO is that a strict \"start your project portable from day one\" mode\nis best as a default. But I'd be happy with any default, actually ;-)\n\n\n\nm\n-- \n martin.langhoff@gmail.com\n martin@laptop.org -- School Server Architect\n - ask interesting questions\n - don't get distracted with shiny stuff - working code first\n - http://wiki.laptop.org/go/User:Martinlanghoff\n"},{"id":"113683","messageId":"m3r5yukv6g.fsf@localhost.localdomain","threadId":"19315","inReplyTo":"46a038f90905121140n61902c61qc3cd41100efee4ae@mail.gmail.com","subject":"Re: Cross-Platform Version Control","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2009-05-12T18:55:39Z","receivedAt":"2009-05-12T18:55:39Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Martin Langhoff writes:\n> On Tue, May 12, 2009 at 8:28 PM, Dmitry Potapov wrote:\n\n> > 2. If you really want to be cross-platform portable, you should not use\n> >   any characters in filenames outside of [A-Za-z0-9._-] (i.e. Portable\n> >   Filename Character Set)\n> >   http://www.opengroup.org/onlinepubs/000095399/basedefs/xbd_chap03.html#tag_03_276\n> \n> Would it make sense to have warnings at 'git add' time about\n> \n> - filenames outside of that charset (as the strictest mode, perhaps\n> even default)\n> - filenames that have a potential conflict wrt case-sensitivity\n> - filenames that have potential conflict in the same tree due to\n> utf-8 encoding vagaries\n> \n> MHO is that a strict \"start your project portable from day one\" mode\n> is best as a default. But I'd be happy with any default, actually ;-)\n\nSomebody asked for a pre-add hook in the past; it would be good place\nto put such check. But in meantime you can do it using pre-commit\nhook instead, isn't it?\n\n-- \nJakub Narebski\nPoland\nShadeHawk on #git\n"},{"id":"113690","messageId":"alpine.DEB.1.00.0905122237230.27348@pacific.mpi-cbg.de","threadId":"19315","inReplyTo":"D1E08DA2-8D8E-4D45-A50A-F32E3448D6E9@gmail.com","subject":"Re: Cross-Platform Version Control","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2009-05-12T20:38:10Z","receivedAt":"2009-05-12T20:38:10Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Tue, 12 May 2009, Esko Luontola wrote:\n\n> On 12.5.2009, at 19:13, Johannes Schindelin wrote:\n> >As to storing all file names in UTF-8, my point about Unicode being not \n> >necessarily appropriate for everyone still stands.\n> >\n> >UTF-8 _might_ be the de-facto standard for Linux filesystems, but IMHO \n> >we should not take away the freedom for everybody to decide what they \n> >want their file names to be encoded as.\n> >\n> >However, I see that there might be a need to be able to encode the file \n> >names differently, such as on Windows. IMHO the best solution would be \n> >a config variable controlling the reencoding of file names.\n> \n> Exactly. The system should not force the use of a specific encoding. It \n> should only offer a recommendation, but be also fully compatible if the \n> user uses some other encoding.\n> \n> That's why it's best to always store the information about what encoding \n> was used. It shouldn't matter, whether the data is encoded with \n> ISO-8859-1, UTF-8, Shift_JIS, Big5 or some other encoding, as long as it \n> is explicitly said that what the encoding is. Then the reader of the \n> data can best decide, how to show that data on the current platform.\n> \n> A config variable for defining, that what encoding should be used when \n> committing the file names, would make sense. Git should also try to \n> autodetect, that what encoding is used in its current environment. In \n> the case of UTF-8, you should also be able to specify which \n> normalization form is used \n> (http://www.unicode.org/unicode/reports/tr15/), or whether it is \n> normalized at all.\n> \n> For example, it should be possible to configure Git so, that when a file \n> is checked out on Mac, its file name is converted to the current file \n> system's encoding (UTF-8 NFD, I think), and when the file is committed \n> on Mac, the file name is normalized back to the same UTF-8 form as is \n> used on Linux (UTF-8 NFC).\n> \n> It would be nice to have config variables for saying, that all file \n> names in this repository must use UTF-8 NFC, and all commit messages \n> must use UTF-8 NFC (with Unix newlines). Then the Git client would \n> autodetect the current environment's encoding, and convert the text, if \n> necessary, to match the repository's encoding.\n\nThat is a nice analysis. How about implementing it?\n\nCiao,\nDscho\n"},{"id":"113693","messageId":"4A09E719.4090205@gmail.com","threadId":"19315","inReplyTo":"alpine.DEB.1.00.0905122237230.27348@pacific.mpi-cbg.de","subject":"Re: Cross-Platform Version Control","fromName":"Esko Luontola","fromEmail":"esko.luontola@gmail.com","sentAt":"2009-05-12T21:16:09Z","receivedAt":"2009-05-12T21:16:09Z","isPatch":false,"sender":{"key":"esko.luontola@gmail.com","avatar":"https://gravatar.com/avatar/86ea2beca289ab5e1e70b2d13b48e6c20e0e44478f3afe74b923090517f66a30?d=mp&s=160"},"body":"Johannes Schindelin wrote on 12.5.2009 23:38:\n> That is a nice analysis. How about implementing it?\n> \n\nDo we have here somebody, who knows Git's code well and is motivated to \nimplement this?\n\nI don't think that I would be capable, because of not having used C \nmuch, being new to Git's codebase and having too little time. But I can \nhelp with the requirements specification, interaction design and system \ntesting.\n\n-- \nEsko Luontola\nwww.orfjackal.net\n"},{"id":"113696","messageId":"20090512214250.GA22974@macbook.lan","threadId":"19315","inReplyTo":"m3r5yukv6g.fsf@localhost.localdomain","subject":"[PATCH] Extend sample pre-commit hook to check for non ascii file/usernames","fromName":"Heiko Voigt","fromEmail":"hvoigt@hvoigt.net","sentAt":"2009-05-12T21:43:00Z","receivedAt":"2009-05-12T21:43:00Z","isPatch":true,"sender":{"key":"hvoigt@hvoigt.net","avatar":"https://avatars.githubusercontent.com/u/184958?v=4"},"body":"At the moment non-ascii encodings of file/usernames are not very well\nsupported by git. This will most likely change in the future but to\nallow repositories to be portable among different file/operating systems\nthis check is enabled by default.\n\nSigned-off-by: Heiko Voigt \n---\nOn Tue, May 12, 2009 at 11:55:39AM -0700, Jakub Narebski wrote:\n> Somebody asked for a pre-add hook in the past; it would be good place\n> to put such check. But in meantime you can do it using pre-commit\n> hook instead, isn't it?\n\nI actually had this in my queue to be submitted...\n\n templates/hooks--pre-commit.sample | 33 +++++++++++++++++++++++++++++++++\n 1 files changed, 33 insertions(+), 0 deletions(-)\n\ndiff --git a/templates/hooks--pre-commit.sample b/templates/hooks--pre-commit.sample\nindex 0e49279..83ff873 100755\n--- a/templates/hooks--pre-commit.sample\n+++ b/templates/hooks--pre-commit.sample\n@@ -7,6 +7,39 @@\n #\n # To enable this hook, rename this file to \"pre-commit\".\n \n+# If you want to allow non-ascii filenames or usernames set\n+# this variable to true.\n+allownonascii=$(git config hooks.allownonascii)\n+\n+function is_ascii () {\n+ test -z \"$(cat | sed -e \"s/[\\ -~]*//g\")\"\n+ return $?\n+}\n+\n+if [ \"$allownonascii\" != \"true\" ]\n+then\n+\t# until git can handle non-ascii filenames gracefully\n+\t# prevent them to be added into the repository\n+\tif ! git diff --cached --name-only --diff-filter=A -z \\\n+\t\t\t| tr \"\\0\" \"\\n\" | is_ascii; then\n+\t\techo \"Non-ascii filenames are not allowed !\"\n+\t\techo \"Please rename the file ...\"\n+\t\texit 1\n+\tfi\n+\n+\t# non-ascii username issue a warning in git gui so tell the\n+\t# user to change it\n+\tif ! git config user.name | is_ascii; then\n+\t\techo \"Please only use ascii characters in your username!\"\n+\t\texit 1\n+\tfi\n+\n+\tif ! git config user.email | is_ascii; then\n+\t\techo \"Please only use ascii characters in your email!\"\n+\t\texit 1\n+\tfi\n+fi\n+\n if git-rev-parse --verify HEAD 2>/dev/null\n then\n \tagainst=HEAD\n-- \n1.6.3\n"},{"id":"113697","messageId":"200905122356.00659.jnareb@gmail.com","threadId":"19315","inReplyTo":"20090512214250.GA22974@macbook.lan","subject":"Re: [PATCH] Extend sample pre-commit hook to check for non ascii file/usernames","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2009-05-12T21:55:59Z","receivedAt":"2009-05-12T21:55:59Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"On Tue, 12 May 2009, Heiko Voigt wrote:\n\n> At the moment non-ascii encodings of file/usernames are not very well\n> supported by git. This will most likely change in the future but to\n> allow repositories to be portable among different file/operating systems\n> this check is enabled by default.\n\n> +\t# non-ascii username issue a warning in git gui so tell the\n> +\t# user to change it\n> +\tif ! git config user.name | is_ascii; then\n> +\t\techo \"Please only use ascii characters in your username!\"\n> +\t\texit 1\n> +\tfi\n> +\n> +\tif ! git config user.email | is_ascii; then\n> +\t\techo \"Please only use ascii characters in your email!\"\n> +\t\texit 1\n> +\tfi\n\nActually 1.) there is no easy way to avoid non-ASCII names at least\nin user.name (I think they are not allowed in email), but 2.) there\nis no trouble with non-ASCII encoding of commits, as they have \n'encoding' header if it is not uft-8 (see *encoding* config variables).\n\n-- \nJakub Narebski\nPoland\n"},{"id":"113714","messageId":"alpine.DEB.1.00.0905130222460.27348@pacific.mpi-cbg.de","threadId":"19315","inReplyTo":"4A09E719.4090205@gmail.com","subject":"Re: Cross-Platform Version Control","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2009-05-13T00:23:11Z","receivedAt":"2009-05-13T00:23:11Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Wed, 13 May 2009, Esko Luontola wrote:\n\n> Johannes Schindelin wrote on 12.5.2009 23:38:\n> > That is a nice analysis. How about implementing it?\n> > \n> \n> Do we have here somebody, who knows Git's code well and is motivated to\n> implement this?\n> \n> I don't think that I would be capable, because of not having used C \n> much, being new to Git's codebase and having too little time.\n\nWell, that rather settles things, no?\n\nCiao,\nDscho\n"},{"id":"113721","messageId":"4A0A5BDB.9030602@gmail.com","threadId":"19315","inReplyTo":"alpine.DEB.1.00.0905130222460.27348@pacific.mpi-cbg.de","subject":"Re: Cross-Platform Version Control","fromName":"Esko Luontola","fromEmail":"esko.luontola@gmail.com","sentAt":"2009-05-13T05:34:19Z","receivedAt":"2009-05-13T05:34:19Z","isPatch":false,"sender":{"key":"esko.luontola@gmail.com","avatar":"https://gravatar.com/avatar/86ea2beca289ab5e1e70b2d13b48e6c20e0e44478f3afe74b923090517f66a30?d=mp&s=160"},"body":"Johannes Schindelin wrote on 13.5.2009 3:23:\n> Well, that rather settles things, no?\n> \n\nThere is need for the feature, but it's unfortunate that the Git \ndevelopers do not see its value. There are many users for whom using \nnon-ASCII names is necessary (for example all of Asia and most of \nEurope), but now it seems that Bazaar is the only DVCS that handles \nencodings correctly: \nhttp://stackoverflow.com/questions/829682/what-dvcs-support-unicode-filenames\n\nLet's see if I have time later this or next year to work on it. At least \nit would be good practise in getting acquainted with a new codebase and \nlearning C. But it would be better for someone else do it, to get it \ndone within a reasonable amount of time.\n\nI see that there are some tests in the /t directory. Which command will \nrun all of them, how good coverage do the tests have, how reproducable \nand isolated they are, how many seconds does it take to run all the \ntests? Is there some high-level documentation for new developers?\n\n-- \nEsko Luontola\nwww.orfjackal.net\n"},{"id":"113727","messageId":"81b0412b0905122349n729f0081j2f9b5b546da80306@mail.gmail.com","threadId":"19315","inReplyTo":"4A0A5BDB.9030602@gmail.com","subject":"Re: Cross-Platform Version Control","fromName":"Alex Riesen","fromEmail":"raa.lkml@gmail.com","sentAt":"2009-05-13T06:49:58Z","receivedAt":"2009-05-13T06:49:58Z","isPatch":false,"sender":{"key":"raa.lkml@gmail.com","avatar":"https://avatars.githubusercontent.com/u/324101?v=4"},"body":"2009/5/13 Esko Luontola :\n> Johannes Schindelin wrote on 13.5.2009 3:23:\n>>\n>> Well, that rather settles things, no?\n>>\n>\n> There is need for the feature, but it's unfortunate that the Git developers\n> do not see its value. There are many users for whom using non-ASCII names is\n> necessary (for example all of Asia and most of Europe), but now it seems\n> that Bazaar is the only DVCS that handles encodings correctly:\n> http://stackoverflow.com/questions/829682/what-dvcs-support-unicode-filenames\n\nMany Git developers just use systems which don't care about the file names\nencoding at all and just keep the names as they were. So interoperability\nproblem does not exist for them. So, they either don't need the feature,\nor can trivially avoid or workaround any problems.\n\n> I see that there are some tests in the /t directory. Which command will run\n> all of them, how good coverage do the tests have, how reproducable and\n> isolated they are, how many seconds does it take to run all the tests? Is\n> there some high-level documentation for new developers?\n\nmake test. See also t/README. We like them. I always run test suite before\ndeployment and sometimes run it just for fun (unless I have to run it\non Windows).\n"},{"id":"113762","messageId":"alpine.DEB.1.00.0905131215010.5046@intel-tinevez-2-302","threadId":"19315","inReplyTo":"4A0A5BDB.9030602@gmail.com","subject":"Re: Cross-Platform Version Control","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2009-05-13T10:15:51Z","receivedAt":"2009-05-13T10:15:51Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Wed, 13 May 2009, Esko Luontola wrote:\n\n> Johannes Schindelin wrote on 13.5.2009 3:23:\n> > Well, that rather settles things, no?\n> \n> There is need for the feature, but it's unfortunate that the Git \n> developers do not see its value.\n\nI see a value. But it is not my itch. And since it is your itch and you \nsaid that you will not do anything about it (I don't count writing emails \nhere ;-), I concluded that it settles the issue.\n\nCiao,\nDscho\n"},{"id":"113769","messageId":"43d8ce650905130341g516c899fsdc6b50a45381d4c6@mail.gmail.com","threadId":"19315","inReplyTo":"43d8ce650905130340q596043d5g45b342b62fe20e8d@mail.gmail.com","subject":"Cross-Platform Version Control","fromName":"John Tapsell","fromEmail":"johnflux@gmail.com","sentAt":"2009-05-13T10:41:54Z","receivedAt":"2009-05-13T10:41:54Z","isPatch":false,"sender":{"key":"johnflux@gmail.com","avatar":"https://gravatar.com/avatar/25f70d4c0f96396b84a2e34bcd9bdc233462c7b4be29b5fdca8266fc53f30b0c?d=mp&s=160"},"body":"2009/5/13 Johannes Schindelin :\n> Hi,\n>\n> On Wed, 13 May 2009, Esko Luontola wrote:\n>\n>> Johannes Schindelin wrote on 13.5.2009 3:23:\n>> > Well, that rather settles things, no?\n>>\n>> There is need for the feature, but it's unfortunate that the Git\n>> developers do not see its value.\n>\n> I see a value.  But it is not my itch.  And since it is your itch and you\n> said that you will not do anything about it (I don't count writing emails\n> here ;-), I concluded that it settles the issue.\n\nI don't know why the git developers are being so hostile/dismisisve,\nbut I also hope that somebody volunteers to fix this.\nEsko, you have my moral support :-)\n\nJohn\n"},{"id":"113785","messageId":"76718490905130642h792e9701sf2ec7fbf2e7de96a@mail.gmail.com","threadId":"19315","inReplyTo":"43d8ce650905130341g516c899fsdc6b50a45381d4c6@mail.gmail.com","subject":"Re: Cross-Platform Version Control","fromName":"Jay Soffian","fromEmail":"jaysoffian@gmail.com","sentAt":"2009-05-13T13:42:37Z","receivedAt":"2009-05-13T13:42:37Z","isPatch":false,"sender":{"key":"jaysoffian@gmail.com","avatar":"https://avatars.githubusercontent.com/u/155970?v=4"},"body":"On Wed, May 13, 2009 at 6:41 AM, John Tapsell wrote:\n> I don't know why the git developers are being so hostile/dismisisve,\n\nAre you serious?\n\nj.\n"},{"id":"113786","messageId":"81b0412b0905130644u7e22bde4ta97616a4ed8be1e0@mail.gmail.com","threadId":"19315","inReplyTo":"76718490905130642h792e9701sf2ec7fbf2e7de96a@mail.gmail.com","subject":"Re: Cross-Platform Version Control","fromName":"Alex Riesen","fromEmail":"raa.lkml@gmail.com","sentAt":"2009-05-13T13:44:38Z","receivedAt":"2009-05-13T13:44:38Z","isPatch":false,"sender":{"key":"raa.lkml@gmail.com","avatar":"https://avatars.githubusercontent.com/u/324101?v=4"},"body":"2009/5/13 Jay Soffian :\n> On Wed, May 13, 2009 at 6:41 AM, John Tapsell wrote:\n>> I don't know why the git developers are being so hostile/dismisisve,\n>\n> Are you serious?\n>\n\n...because we'll kill you if aren't >:-E\n"},{"id":"113787","messageId":"76718490905130650u4d047e13pa73e0a96289c452a@mail.gmail.com","threadId":"19315","inReplyTo":"81b0412b0905130644u7e22bde4ta97616a4ed8be1e0@mail.gmail.com","subject":"Re: Cross-Platform Version Control","fromName":"Jay Soffian","fromEmail":"jaysoffian@gmail.com","sentAt":"2009-05-13T13:50:21Z","receivedAt":"2009-05-13T13:50:21Z","isPatch":false,"sender":{"key":"jaysoffian@gmail.com","avatar":"https://avatars.githubusercontent.com/u/155970?v=4"},"body":"On Wed, May 13, 2009 at 9:44 AM, Alex Riesen wrote:\n> 2009/5/13 Jay Soffian :\n>> On Wed, May 13, 2009 at 6:41 AM, John Tapsell wrote:\n>>> I don't know why the git developers are being so hostile/dismisisve,\n>>\n>> Are you serious?\n>>\n>\n> ...because we'll kill you if aren't >:-E\n\nI'm just flabbergasted by some people's expectations. Perhaps John\ndoesn't realize the git developers are all volunteers, and that it is\nnever appropriate to criticize a volunteer. A \"thank you for all your\nhard work on git\" would have done nicely.\n\nj.\n"},{"id":"113788","messageId":"43d8ce650905130657q2de9686q683fea84345a6651@mail.gmail.com","threadId":"19315","inReplyTo":"76718490905130650u4d047e13pa73e0a96289c452a@mail.gmail.com","subject":"Re: Cross-Platform Version Control","fromName":"John Tapsell","fromEmail":"johnflux@gmail.com","sentAt":"2009-05-13T13:57:00Z","receivedAt":"2009-05-13T13:57:00Z","isPatch":false,"sender":{"key":"johnflux@gmail.com","avatar":"https://gravatar.com/avatar/25f70d4c0f96396b84a2e34bcd9bdc233462c7b4be29b5fdca8266fc53f30b0c?d=mp&s=160"},"body":"2009/5/13 Jay Soffian :\n> On Wed, May 13, 2009 at 9:44 AM, Alex Riesen wrote:\n>> 2009/5/13 Jay Soffian :\n>>> On Wed, May 13, 2009 at 6:41 AM, John Tapsell wrote:\n>>>> I don't know why the git developers are being so hostile/dismisisve,\n>>>\n>>> Are you serious?\n>>>\n>>\n>> ...because we'll kill you if aren't >:-E\n>\n> I'm just flabbergasted by some people's expectations. Perhaps John\n> doesn't realize the git developers are all volunteers, and that it is\n> never appropriate to criticize a volunteer. A \"thank you for all your\n> hard work on git\" would have done nicely.\n\nI'm as much of an open source developer as anyone else here. I spend\na huge amount of my time programming for KDE. But I've never told a\nuser \"well that settles it\" because they won't code it themselves :-/\nI certaintly get a huge number of bug/wishes that I can't/won't code\nmyself, but I try to be a bit more diplomatic about it.\nBut then the kernel mailing lists tend to be a lot more.. direct..\nthan the kde mailing lists, so I guess it comes from that. Requiring\npeople to have a thick skin and all that.\n\n\nJohn\n"},{"id":"113796","messageId":"alpine.LFD.2.00.0905131117120.6741@xanadu.home","threadId":"19315","inReplyTo":"43d8ce650905130657q2de9686q683fea84345a6651@mail.gmail.com","subject":"Re: Cross-Platform Version Control","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2009-05-13T15:27:57Z","receivedAt":"2009-05-13T15:27:57Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Wed, 13 May 2009, John Tapsell wrote:\n\n> I'm as much of an open source developer as anyone else here. I spend\n> a huge amount of my time programming for KDE. But I've never told a\n> user \"well that settles it\" because they won't code it themselves :-/\n> I certaintly get a huge number of bug/wishes that I can't/won't code\n> myself, but I try to be a bit more diplomatic about it.\n> But then the kernel mailing lists tend to be a lot more.. direct..\n> than the kde mailing lists, so I guess it comes from that. Requiring\n> people to have a thick skin and all that.\n\nThis is not the kernel mailing list. In fact this list is quite \nfriendlier and accommodating that the kernel list.\n\nThe remark alluded above comes from _one_ of the git developers. And \nDscho is apparently in a rather sad mood these days. While the substance \nof Dscho's remark is entirely pertinent, it would be wrong to use its \nform and style as a characterization of git developers in general.\n\n\nNicolas\n"},{"id":"113802","messageId":"alpine.DEB.1.00.0905131815230.5046@intel-tinevez-2-302","threadId":"19315","inReplyTo":"alpine.LFD.2.00.0905131117120.6741@xanadu.home","subject":"Re: Cross-Platform Version Control","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2009-05-13T16:22:58Z","receivedAt":"2009-05-13T16:22:58Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Wed, 13 May 2009, Nicolas Pitre wrote:\n\n> On Wed, 13 May 2009, John Tapsell wrote:\n> \n> > I'm as much of an open source developer as anyone else here. I spend \n> > a huge amount of my time programming for KDE. But I've never told a \n> > user \"well that settles it\" because they won't code it themselves :-/ \n> > I certaintly get a huge number of bug/wishes that I can't/won't code \n> > myself, but I try to be a bit more diplomatic about it.\n> >\n> > But then the kernel mailing lists tend to be a lot more.. direct.. \n> > than the kde mailing lists, so I guess it comes from that. Requiring \n> > people to have a thick skin and all that.\n> \n> This is not the kernel mailing list. In fact this list is quite \n> friendlier and accommodating that the kernel list.\n> \n> The remark alluded above comes from _one_ of the git developers. And \n> Dscho is apparently in a rather sad mood these days. While the substance \n> of Dscho's remark is entirely pertinent, it would be wrong to use its \n> form and style as a characterization of git developers in general.\n\nEven if I were in a better mood, the whole thread has a back story on an \nmsysGit issue, and this led me to try to stop what I feared would become a \nrather long mail thread without much of an outcome, such as that infamous \nthread about MacOSX UTF-8 filename handling.\n\nAlas, it seems that Robin is willing to work on the issues, so my fears \nhave been totally and completely unfounded.\n\nCiao,\nDscho\n"},{"id":"113803","messageId":"alpine.LFD.2.01.0905130915540.3343@localhost.localdomain","threadId":"19315","inReplyTo":"20090512161638.GB29566@coredump.intra.peff.net","subject":"Re: Cross-Platform Version Control","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2009-05-13T16:26:19Z","receivedAt":"2009-05-13T16:26:19Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Tue, 12 May 2009, Jeff King wrote:\n>\n> Or they use a single encoding like utf8 so that there are no surprises.\n> You can still run into normalization problems with filenames on some\n> filesystems, though. Linus's name_hash code sets up the framework to\n> handle \"these two names are actually equivalent\", but right now I think\n> there is just code for handling case-sensitivity, not utf8 normalization\n> (but I just skimmed the code, so I might be wrong).\n\nutf-8 normalization was one goal, and shouldn't be _that_ hard to do. But \nquite frankly, the index is only part of it, and probably not the worst \npart.\n\nThe real pain of filename handling is all the \"read tree recursively with \nreaddir()\" issues. Along with just an absolute sh*t-load of issues about \nwhat to do when people ended up using different versions of the \"same\" \nname in different branches.\n\nThere's also the issue that \"cross-platform\" really can be a pretty damn \nbig pain. What do you do for platforms that simply are pure shit? I \nrealize that OS X people have a hard time accepting it, but OS X \nfilesystems are generally total and utter crap - even more so than \nWindows.\n\nYes, yes, you can tell OS X that case matters, but that's not the normal \ncase - and what do you do with projects that simply _do_ care about case. \nThe kernel is one such project.\n\nSure, you can \"encode\" the filenames on such broken filesystems in a way \nthat they'd be different - but that won't really help the project, since \nmakefiles etc won't work anyway.\n\nSo one reason I didn't bother with utf-8 is that the much more fundamental \nissues are simply in plain old 7-bit US-ASCII. \n\nThat said, if the only issue is that you want to encode regular utf-8 in a \ncoherent way (and ignore the case issues), then we could probably do that \npart fairly easily with a \"convert_to_internal()\" and \n\"convert_to_filename()\" thing that acts very much like the CRLF conversion \n(except on filenames, not data).\n\nAnd yes, it's probably worth doing, since we'd need that for fuller case \nsupport anyway.\n\nIt's just a fair amount of churn - not fundamentally _hard_, but not \ntrivial either. And it needs a _lot_ of care, and a fair amount of \ntesting that is probably hard to do on sane filesystems (ie the case where \nthe filesystem actually _changes_ the name is going to be hard to test on \nanything sane).\n\n\t\t\tLinus\n"},{"id":"113808","messageId":"alpine.LFD.2.01.0905130951100.3343@localhost.localdomain","threadId":"19315","inReplyTo":"alpine.LFD.2.01.0905130915540.3343@localhost.localdomain","subject":"Re: Cross-Platform Version Control","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2009-05-13T17:12:45Z","receivedAt":"2009-05-13T17:12:45Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Wed, 13 May 2009, Linus Torvalds wrote:\n> \n> utf-8 normalization was one goal, and shouldn't be _that_ hard to do. But \n> quite frankly, the index is only part of it, and probably not the worst \n> part.\n> \n> The real pain of filename handling is all the \"read tree recursively with \n> readdir()\" issues. Along with just an absolute sh*t-load of issues about \n> what to do when people ended up using different versions of the \"same\" \n> name in different branches.\n\nBtw, if people care mainly just about OS X, and don't worry so much about \ncase, but about the idiotic and insane OS X behavior of turning UTF-8 \nfilenames into that crazy NFD format, here's a simple patch that may be \nuseful for that.\n\nThere _will_ certainly be other places, but this handles the one big case \nof \"read_directory_recursive()\", and can turn NFD into the sane NFC \nformat.\n\nSince OS X will then accept NFC (and internally turn it back to NFD) when \nyou pass them as filenames, that means that converting the other way is \nnot necessary.\n\nNOTE NOTE NOTE! This really just handles one case, and is not enough for \nany kind of general case. For example, it does NOT handle the case where \nyou do\n\n\tgit add filename_with_åäö\n\nexplicitly, because if the \"filename_with_åäö\" is done using NFD \n(tab-completion etc), now git won't _match_ it with the filename it reads \nusing readdir() any more (which got converted to NFC), so at a minimum \nwe'd need to do that crazy NFD->NFC conversion in all the pathspecs too. \n\nSee \"get_pathspec()\" in setup.c for that latter case.\n\nBut with that, and this crazy thing, OS X users might be already a lot \nbetter off. Totally untested, of course. \n\nOh, and somebody needs to fill in that \n\n\tconvert_name_from_nfd_to_nfc()\n\nimplementation. It's designed so that if it notices that the string is \njust plain US-ASCII, it can return 0 and no extra work is done. That, in \nturn, can easily be done by some simple and efficient pre-processign that \nchecks that there are no high bits set (on a 64-bit platform, do it 8 \ncharacters at a time with a \"& 0x8080808080808080\"), so that the common \ncase doesn't need to have barely any overhead at all.\n\nUse and stringprep_utf8_nfkc_normalize() or something to do \nthe actual normalization if you find characters with the high bit set. And \nsince I know that the OS X filesystems are so buggy as to not even do that \nwhole NFD thing right, there is probably some OS-X specific \"use this for \nfilesystem names\" conversion function.\n\nHmm. Anybody want to take this on? It really shouldn't be too complex to \nget it working for the common case on just OS X. It's really the case \nsensitivity that is the biggest problem, if you ignore that for now, the \nproblem space is _much_ smaller.\n\nIn other words, I think we can reasonably easily support a subset of \n_common_ issues with some trivial patches like this. But getting it right \nin _all_ the cases is going to be much more work (there are lots of other \nuses of \"readdir()\" too, this one just happens to be one of the more \ncentral ones).\n\nOf course, it probably makes sense to have a whole \"git_readdir()\" that \ndoes this thing in general. That \"create_full_path()\" thing makes sense \nregardless, though, in that it also simplifies a lot of \"baselen+len\" \nusage in just \"len\".\n\n\t\tLinus\n\n---\n dir.c | 40 ++++++++++++++++++++++++++++++++--------\n 1 files changed, 32 insertions(+), 8 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex 6aae09a..4cbfc24 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -566,6 +566,30 @@ static int get_dtype(struct dirent *de, const char *path)\n }\n \n /*\n+ * Take the readdir output, in (d_name,len), and append it to\n+ * our base name in (fullname,baselen) with any required\n+ * readdir fs->internal translation.\n+ *\n+ * Put the result in 'fullname', and return the final length.\n+ *\n+ * Right now we have no translation, and just do a memcpy()\n+ * (the +1 is to copy the final NUL character too).\n+ */\n+static int create_full_path(char *fullname, int baselen, const char *d_name, int len)\n+{\n+#ifdef OS_X_IS_SOME_CRAZY_SHxAT\n+\tchar temp[256], nlen;\n+\tnlen = convert_name_from_nfd_to_nfc(d_name, len, temp, sizeof(temp));\n+\tif (nlen) {\n+\t\tlen = nlen;\n+\t\td_name = temp;\n+\t}\n+#endif\n+\tmemcpy(fullname + baselen, d_name, len + 1);\n+\treturn baselen + len;\n+}\n+\n+/*\n * Read a directory tree. We currently ignore anything but\n * directories, regular files and symlinks. That's because git\n * doesn't handle them at all yet. Maybe that will change some\n@@ -595,15 +619,15 @@ static int read_directory_recursive(struct dir_struct *dir, const char *path, co\n \t\t\t/* Ignore overly long pathnames! */\n \t\t\tif (len + baselen + 8 > sizeof(fullname))\n \t\t\t\tcontinue;\n-\t\t\tmemcpy(fullname + baselen, de->d_name, len+1);\n-\t\t\tif (simplify_away(fullname, baselen + len, simplify))\n+\t\t\tlen = create_full_path(fullname, baselen, de->d_name, len);\n+\t\t\tif (simplify_away(fullname, len, simplify))\n \t\t\t\tcontinue;\n \n \t\t\tdtype = DTYPE(de);\n \t\t\texclude = excluded(dir, fullname, &dtype);\n \t\t\tif (exclude && (dir->flags & DIR_COLLECT_IGNORED)\n-\t\t\t && in_pathspec(fullname, baselen + len, simplify))\n-\t\t\t\tdir_add_ignored(dir, fullname, baselen + len);\n+\t\t\t && in_pathspec(fullname, len, simplify))\n+\t\t\t\tdir_add_ignored(dir, fullname, len);\n \n \t\t\t/*\n \t\t\t * Excluded? If we don't explicitly want to show\n@@ -630,9 +654,9 @@ static int read_directory_recursive(struct dir_struct *dir, const char *path, co\n \t\t\tdefault:\n \t\t\t\tcontinue;\n \t\t\tcase DT_DIR:\n-\t\t\t\tmemcpy(fullname + baselen + len, \"/\", 2);\n+\t\t\t\tmemcpy(fullname + len, \"/\", 2);\n \t\t\t\tlen++;\n-\t\t\t\tswitch (treat_directory(dir, fullname, baselen + len, simplify)) {\n+\t\t\t\tswitch (treat_directory(dir, fullname, len, simplify)) {\n \t\t\t\tcase show_directory:\n \t\t\t\t\tif (exclude != !!(dir->flags\n \t\t\t\t\t\t\t& DIR_SHOW_IGNORED))\n@@ -640,7 +664,7 @@ static int read_directory_recursive(struct dir_struct *dir, const char *path, co\n \t\t\t\t\tbreak;\n \t\t\t\tcase recurse_into_directory:\n \t\t\t\t\tcontents += read_directory_recursive(dir,\n-\t\t\t\t\t\tfullname, fullname, baselen + len, 0, simplify);\n+\t\t\t\t\t\tfullname, fullname, len, 0, simplify);\n \t\t\t\t\tcontinue;\n \t\t\t\tcase ignore_directory:\n \t\t\t\t\tcontinue;\n@@ -654,7 +678,7 @@ static int read_directory_recursive(struct dir_struct *dir, const char *path, co\n \t\t\tif (check_only)\n \t\t\t\tgoto exit_early;\n \t\t\telse\n-\t\t\t\tdir_add_name(dir, fullname, baselen + len);\n+\t\t\t\tdir_add_name(dir, fullname, len);\n \t\t}\n exit_early:\n \t\tclosedir(fdir);\n"},{"id":"113810","messageId":"4A0B0265.8000100@op5.se","threadId":"19315","inReplyTo":"43d8ce650905130657q2de9686q683fea84345a6651@mail.gmail.com","subject":"Re: Cross-Platform Version Control","fromName":"Andreas Ericsson","fromEmail":"ae@op5.se","sentAt":"2009-05-13T17:24:53Z","receivedAt":"2009-05-13T17:24:53Z","isPatch":false,"sender":{"key":"ae@op5.se","avatar":"https://gravatar.com/avatar/426e89595c75a8f5252dd0c989e5fabe5bcac616e68557427ad9aef6b0ca342a?d=mp&s=160"},"body":"John Tapsell wrote:\n> 2009/5/13 Jay Soffian :\n>> On Wed, May 13, 2009 at 9:44 AM, Alex Riesen wrote:\n>>> 2009/5/13 Jay Soffian :\n>>>> On Wed, May 13, 2009 at 6:41 AM, John Tapsell wrote:\n>>>>> I don't know why the git developers are being so hostile/dismisisve,\n>>>> Are you serious?\n>>>>\n>>> ...because we'll kill you if aren't >:-E\n>> I'm just flabbergasted by some people's expectations. Perhaps John\n>> doesn't realize the git developers are all volunteers, and that it is\n>> never appropriate to criticize a volunteer. A \"thank you for all your\n>> hard work on git\" would have done nicely.\n> \n> I'm as much of an open source developer as anyone else here. I spend\n> a huge amount of my time programming for KDE. But I've never told a\n> user \"well that settles it\" because they won't code it themselves :-/\n> I certaintly get a huge number of bug/wishes that I can't/won't code\n> myself, but I try to be a bit more diplomatic about it.\n> But then the kernel mailing lists tend to be a lot more.. direct..\n> than the kde mailing lists, so I guess it comes from that. Requiring\n> people to have a thick skin and all that.\n> \n\nI think much of the perceived malignancy stems from the fact that the\ngit list has a high ratio of developer-to-luser mailings on it, being\nby nature a developer tool most of the time. When the unaware user\nappears on the list with demands rather than polite requests, they're\ntreated that much harder. Especially by the developer who happens to\nbe, as it were, the butt of the request.\n\nPersonally, I've only ever found Dscho being anything but friendly on\nthis list, and even then, I really didn't find it offensive. If viewed\nin a happy mood, it matches quite nicely with a swedish sketch whose\ntheme is \"men ja ente bitter\". It's often quite funny, really :-)\n\n-- \nAndreas Ericsson andreas.ericsson@op5.se\nOP5 AB www.op5.se\nTel: +46 8-230225 Fax: +46 8-230231\n\nRegister now for Nordic Meet on Nagios, June 3-4 in Stockholm\n http://nordicmeetonnagios.op5.org/\n\nConsidering the successes of the wars on alcohol, poverty, drugs and\nterror, I think we should give some serious thought to declaring war\non peace.\n"},{"id":"113812","messageId":"4A0B03FC.9020101@op5.se","threadId":"19315","inReplyTo":"alpine.LFD.2.01.0905130951100.3343@localhost.localdomain","subject":"Re: Cross-Platform Version Control","fromName":"Andreas Ericsson","fromEmail":"ae@op5.se","sentAt":"2009-05-13T17:31:40Z","receivedAt":"2009-05-13T17:31:40Z","isPatch":false,"sender":{"key":"ae@op5.se","avatar":"https://gravatar.com/avatar/426e89595c75a8f5252dd0c989e5fabe5bcac616e68557427ad9aef6b0ca342a?d=mp&s=160"},"body":"Linus Torvalds wrote:\n> \n> Of course, it probably makes sense to have a whole \"git_readdir()\" that \n> does this thing in general. That \"create_full_path()\" thing makes sense \n> regardless, though, in that it also simplifies a lot of \"baselen+len\" \n> usage in just \"len\".\n> \n\nIn a flash of premonitory insight, libgit2 has \n\n\tgitfo_foreach_dirent(path, callback)\n\nwhich would probably be well suited for this kind of thing.\n\n-- \nAndreas Ericsson andreas.ericsson@op5.se\nOP5 AB www.op5.se\nTel: +46 8-230225 Fax: +46 8-230231\n\nRegister now for Nordic Meet on Nagios, June 3-4 in Stockholm\n http://nordicmeetonnagios.op5.org/\n\nConsidering the successes of the wars on alcohol, poverty, drugs and\nterror, I think we should give some serious thought to declaring war\non peace.\n"},{"id":"113816","messageId":"alpine.LFD.2.01.0905131036040.3343@localhost.localdomain","threadId":"19315","inReplyTo":"alpine.LFD.2.01.0905130951100.3343@localhost.localdomain","subject":"Re: Cross-Platform Version Control","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2009-05-13T17:46:53Z","receivedAt":"2009-05-13T17:46:53Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Wed, 13 May 2009, Linus Torvalds wrote:\n>\n> Of course, it probably makes sense to have a whole \"git_readdir()\" that \n> does this thing in general.\n\nActually, the more I think about that, the less true I think it is.\n\nIt _sounds_ like a nice simplification (\"just do it once in readdir, and \nforget about it everywhere else\"), but it's in fact a stupid thing to do.\n\nWhy?\n\nIf we _ever_ want to fix this in the general case, then the code that does \nthe readdir() will actually have to remember both the \"raw filesystem\" \nform _and_ the \"cleaned-up utf-8 form\".\n\nWhy? Because when we do readdir(), we'll also do 'lstat()' on the end \nresult to check the types, and opendir() in case it's a directory and we \nthen want to do things recursively etc. And that happens to work on OS X \n(because we can use our \"fixed\" filename for lstat too), but it does not \nwork in the general case.\n\nAnd you can say \"well, just do the stat inside the wrapped readdir()\", but \nthat doesn't work _either_, since\n\n - we don't want to do the lstat() if it's unnecessary. Even if we don't \n have \"de->d_type\" information, we can often avoid the need for it, if \n we can tell that the name isn't interestign (due to being ignored).\n\n Avoiding the lstat is a huge performance issue for cold-cache cases. \n It's basically a seek.\n\n So we really want to do the lstat() later, which implies that the \n caller needs to know _both_ the original \"real\" filesystem name _and_ \n the converted one.\n\n - it doesn't handle the opendir() case anyway - so the end result is that \n a real implementation will _always_ need to carry around both the \n \"filesystem view\" filename _and_ the \"what we've converted it into\".\n\nNow, the point of the patch I sent out was that for the specific case of \nOS X, which does UTF-8 conversions (wrong) but also is happy to get our \nproperly normalized name, we don't care. So my patch is \"correct\" for that \nspecial case - and so would a plain readdir() wrapper be.\n\nBut my patch is _also_ correct for the case where a readdir() wrapper \nwould do the wrong thing. My patch doesn't _handle_ it (since it doesn't \nchange the code to pass both \"filesystem view\" and \"cleaned-up view\" \npathnames), but the patch I sent out also doesn't make it any harder to do \nright.\n\nIn contrast, doing a readdir() wrapper makes it much harder to do right \nlater, because it's just doing the conversion at the wrong level (you \ncould make that \"wrapper\" return both the original and the fixed \nfilename, but at that point the wrapper doesn't really help - you might \nas well just have the \"convert\" function, and it would be a hell of a lot \nmore obvious what is really going on).\n\nSo I take it back. A readdir() wrapper is not a good idea. It gets us a \ntiny bit of the way, but it would actually take us a step back from the \n\"real\" solution.\n\n\t\t\tLinus\n"},{"id":"113820","messageId":"46a038f90905131126s79e5b8e1qd0c6c100ec836127@mail.gmail.com","threadId":"19315","inReplyTo":"alpine.LFD.2.01.0905131036040.3343@localhost.localdomain","subject":"Re: Cross-Platform Version Control","fromName":"Martin Langhoff","fromEmail":"martin.langhoff@gmail.com","sentAt":"2009-05-13T18:26:26Z","receivedAt":"2009-05-13T18:26:26Z","isPatch":false,"sender":{"key":"martin.langhoff@gmail.com","avatar":"https://gravatar.com/avatar/1e3f311b6c4c15836501901ca58f8c0b0667246488084ba524d8bc9867e22fd9?d=mp&s=160"},"body":"On Wed, May 13, 2009 at 7:46 PM, Linus Torvalds\n wrote:\n> So I take it back. A readdir() wrapper is not a good idea. It gets us a\n> tiny bit of the way, but it would actually take us a step back from the\n> \"real\" solution.\n\nDo we need to take the real solution to the core of git?\n\nWhat I am wondering is whether we can keep this simple in git\ninternals and catch problem filenames at git-add time. This would\nallow git to keep treating filenames as a bag of bytes, and it does a\nbetter thing for users.\n\nIn cross platform projects, most users don't even know that there are\nproblems, and even if they do, they don't know what the problems are.\n\nIf git add can be told to warn & refuse to add a path with portability\nproblems, then we educate our users, prevent them from committing\nfilenames that will later cause trouble to others in their projects,\netc.\n\nfrom-the-keep-it-simple-and-informative-dept,\n\n\nm\n-- \n martin.langhoff@gmail.com\n martin@laptop.org -- School Server Architect\n - ask interesting questions\n - don't get distracted with shiny stuff - working code first\n - http://wiki.laptop.org/go/User:Martinlanghoff\n"},{"id":"113824","messageId":"alpine.LFD.2.01.0905131131240.3343@localhost.localdomain","threadId":"19315","inReplyTo":"46a038f90905131126s79e5b8e1qd0c6c100ec836127@mail.gmail.com","subject":"Re: Cross-Platform Version Control","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2009-05-13T18:37:28Z","receivedAt":"2009-05-13T18:37:28Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Wed, 13 May 2009, Martin Langhoff wrote:\n> \n> Do we need to take the real solution to the core of git?\n\nWell, I suspect that if we really want to support it, then we'd better.\n\n> What I am wondering is whether we can keep this simple in git\n> internals and catch problem filenames at git-add time.\n\nI can almost guarantee that it will just cause more problems than it \nsolves, and generate some nasty cases that just aren't solvable.\n\nBecause it really isn't just \"git add\". It's every single thing that does \na lstat() on a filename inside of git.\n\nNow, the simple OS X case is not a huge problem, since the lstat will \nsucceed with the fixed-up filename too. But as mentioned, the OS X case is \nthe thing that doesn't need a lot of infrastructure _anyway_ - I can \nalmost guarantee that my posted patch (with the added setup.c stuff for \nget_pathspec()) is going to be _fewer_ lines than some wrapper logic.\n\nNote: in all of the above, I assume that people care more about just plain \nUTF characters (and the insane NFD form OS X uses) than about worrying \nabout the _really_ subtle issues of case-independence. Those are a major \npain, but they will need even more \"internal\" support, because there \nsimply isn't any sane wrapping method.\n\n(You could wrap everything to force lower-casing of all filesystem ops or \nsomething, but that would not be acceptable to any sane environment. So in \nreality you need to accept mixed-case things, and then there is no way to \nknow from the \"outside\" whether one external mixed-case thing matches some \ninternal index mixed-case thing).\n\n\t\t\tLinus\n"},{"id":"113841","messageId":"op.utvy5mie1e62zd@merlin.emma.line.org","threadId":"19315","inReplyTo":"alpine.LFD.2.01.0905130951100.3343@localhost.localdomain","subject":"Re: Cross-Platform Version Control","fromName":"Matthias Andree","fromEmail":"matthias.andree@gmx.de","sentAt":"2009-05-13T20:57:12Z","receivedAt":"2009-05-13T20:57:12Z","isPatch":false,"sender":{"key":"matthias.andree@gmx.de","avatar":null},"body":"Am 13.05.2009, 19:12 Uhr, schrieb Linus Torvalds \n:\n\n> Use and stringprep_utf8_nfkc_normalize() or something to \n> do the actual normalization if you find characters with the high bit \n> set. And since I know that the OS X filesystems are so buggy as to not \n> even do that whole NFD thing right, there is probably some OS-X specific \n> \"use this for\n> filesystem names\" conversion function.\n\nSorry for interrupting, but NF_K_C? You don't want that (K for \ncompatibility, rather than canonical, normalization) for anything except \nnormalizing temporary variables inside strcasecmp(3) or similar. Probably \nnot even that. The normalizations done are often irreversible and also \nsurprising. You don't want to turn 2³.c into 23.c, do you?\n\n-- \nMatthias Andree\n"},{"id":"113842","messageId":"20090513210425.GD6579@mit.edu","threadId":"19315","inReplyTo":"alpine.LFD.2.01.0905131131240.3343@localhost.localdomain","subject":"Re: Cross-Platform Version Control","fromName":"Theodore Tso","fromEmail":"tytso@mit.edu","sentAt":"2009-05-13T21:04:25Z","receivedAt":"2009-05-13T21:04:25Z","isPatch":false,"sender":{"key":"tytso@mit.edu","avatar":"https://avatars.githubusercontent.com/u/51416?v=4"},"body":"On Wed, May 13, 2009 at 11:37:28AM -0700, Linus Torvalds wrote:\n> Note: in all of the above, I assume that people care more about just plain \n> UTF characters (and the insane NFD form OS X uses) than about worrying \n> about the _really_ subtle issues of case-independence. Those are a major \n> pain, but they will need even more \"internal\" support, because there \n> simply isn't any sane wrapping method.\n\nStupid question --- if we get something that works for Windows and\nMacOS X, is there any reason why we need to solve the general problem\nof case-insentive filesystems? It's really backwards compatibility\nwith Legacy OS's that most important, right? Are there any other\nsystems other than Windows and Mac OS X which (a) perpetrate case\ninsensitivity on application programmers, and (b) which current or\nfuture git users are likely to care about?\n\n\t\t\t\t\t\t- Ted\n"},{"id":"113843","messageId":"alpine.LNX.2.00.0905131639580.2147@iabervon.org","threadId":"19315","inReplyTo":"alpine.LFD.2.01.0905131131240.3343@localhost.localdomain","subject":"Re: Cross-Platform Version Control","fromName":"Daniel Barkalow","fromEmail":"barkalow@iabervon.org","sentAt":"2009-05-13T21:08:31Z","receivedAt":"2009-05-13T21:08:31Z","isPatch":false,"sender":{"key":"barkalow@iabervon.org","avatar":"https://avatars.githubusercontent.com/u/55364219?v=4"},"body":"On Wed, 13 May 2009, Linus Torvalds wrote:\n\n> On Wed, 13 May 2009, Martin Langhoff wrote:\n> > \n> > Do we need to take the real solution to the core of git?\n> \n> Well, I suspect that if we really want to support it, then we'd better.\n> \n> > What I am wondering is whether we can keep this simple in git\n> > internals and catch problem filenames at git-add time.\n> \n> I can almost guarantee that it will just cause more problems than it \n> solves, and generate some nasty cases that just aren't solvable.\n> \n> Because it really isn't just \"git add\". It's every single thing that does \n> a lstat() on a filename inside of git.\n> \n> Now, the simple OS X case is not a huge problem, since the lstat will \n> succeed with the fixed-up filename too.\n\nI'm not seeing what the general case is, and how it could possibly behave.\n\nThere's the \"insensitive\" behavior: if you create \"foo\" and look for \n\"FOO\", it's there, but readdir() reports \"foo\".\n\nThere's the \"converting\" behavior: if you create \"foo\", readdir() reports \n\"FOO\", but lstat(\"foo\") returns it.\n\nThe obvious general case is: if you create \"foo\", readdir() reports \"FOO\", \nand lstat(\"foo\") doesn't find a match. But if you create \"foo\" again... it \ndoesn't find \"foo\", so it creates a new file, which it also calls \"FOO\", \nand the filesystem now has two files with identical names?\n\nIt seems to me that the limits of minimally functional, non-inode-losing \nfilesystems are: lstat() might take a filename and return the data for a \nnon-byte-identical filename; open(name, O_CREAT|O_EXCL) might replace the \ngiven name with a non-byte-identical filename. But surely open(name) and \nlstat(name) (with the same name) must find the same file, even if \nreaddir() would report it with a different name.\n\nAnd I assume that a filesystem that rejected any non-NFD filenames or any \nnon-NFC filenames would be totally unusable, in that users will manage to \nget unnormalized filenames into programs and find that the filesystem just \ndoesn't work.\n\n\t-Daniel\n*This .sig left intentionally blank*\n"},{"id":"113844","messageId":"alpine.LFD.2.01.0905131401170.3343@localhost.localdomain","threadId":"19315","inReplyTo":"op.utvy5mie1e62zd@merlin.emma.line.org","subject":"Re: Cross-Platform Version Control","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2009-05-13T21:10:17Z","receivedAt":"2009-05-13T21:10:17Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Wed, 13 May 2009, Matthias Andree wrote:\n\n> Am 13.05.2009, 19:12 Uhr, schrieb Linus Torvalds\n> :\n> \n> > Use and stringprep_utf8_nfkc_normalize() or something to do\n> > the actual normalization if you find characters with the high bit set. And\n> > since I know that the OS X filesystems are so buggy as to not even do that\n> > whole NFD thing right, there is probably some OS-X specific \"use this for\n> > filesystem names\" conversion function.\n> \n> Sorry for interrupting, but NF_K_C? You don't want that (K for compatibility,\n> rather than canonical, normalization) for anything except normalizing\n> temporary variables inside strcasecmp(3) or similar. Probably not even that.\n> The normalizations done are often irreversible and also surprising. You don't\n> want to turn 2³.c into 23.c, do you?\n\nNo, you're right. We want just plain NFC. I just googled for how some \nother projects handled this, and found the stringprep thing in a post \nabout rsync, and didn't look any closer.\n\nBut yes, you're absolutely right, stringprep is total crap, and nfkc is \nhorrible.\n\nI have no idea of what library to use, though. For perl, there's \nUnicode::Normalize, but that's likely still subtly incorrect for the OS-X \ncase due to the filesystem not using _strict_ NFD.\n\nI have this dim memory of somebody actually pointing to the documentation \nof exactly which characters OS X ends up decomposing. Maybe we could just \ndo a git-specific inverse of that, knowing that NOBODY ELSE IN THE WHOLE \nUNIVERSE IS SO TERMINALLY STUPID AS TO DO THAT DECOMPOSITION, and thus the \nOS X case is the only one we need to care about?\n\n\t\t\tLinus\n"},{"id":"113845","messageId":"alpine.LFD.2.01.0905131410260.3343@localhost.localdomain","threadId":"19315","inReplyTo":"20090513210425.GD6579@mit.edu","subject":"Re: Cross-Platform Version Control","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2009-05-13T21:20:27Z","receivedAt":"2009-05-13T21:20:27Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Wed, 13 May 2009, Theodore Tso wrote:\n> \n> Stupid question --- if we get something that works for Windows and\n> MacOS X, is there any reason why we need to solve the general problem\n> of case-insentive filesystems?\n\nQutie frankly, I don't think we're even very close to getting anything \nthat works for Windows of OS X.\n\nCase-insensitivity is _hard_.\n\nThe \"easy\" case is to just handle the OS X craxy pseudo-NFD format, and at \nleast turn that into NFC (and perhaps add a config option to do latin1 and \nEUC-JP to utf-8 too) and. At that point, we at least handle regular utf-8 \nthe same way.\n\nDoing the latin1/EUC-JP thing would actually to some degree be more \ninteresting than the OS X NFD case, because that really does require \ntwo-way conversion, and we can \"test\" that even on sane filesystems (ie \nplay at having a Latin1 filesystem).\n\nThat said, I suspect there aren't that many people who care about latin1 \nfilesystems. I dunno about EUC-JP (and variants - for all I know, \nshift-JIS and other cases may be the more common ones).\n\nOf course, if we do everything right, maybe the windows people would \nactually like us to keep the filesystem-native representation in UTF-16LE \nor whatever the crazy format is that Windows really uses deep down.\n\nMy point being that all of these things happen even without the added \nworry about case. And in many ways, not worrying about case should \nprobably be the first step. We do have some support for worrying about \ncase, but trying to solve both things at the same time isn't going to be \nworkable, I suspect.\n\nCase insensitivity should never ever involve a _conversion_ (if it does, \nyou get all kinds of crazy behavior), it's just purely a _comparison_ \nissue, so the two really are fundamentally different.\n\nOf course, the reason OS-X seems to be so messed up is exactly that the \nmorons at Apple didn't understand the difference between conversion and \ncomparison, and mixed them up.\n\n\t\tLinus\n"},{"id":"113851","messageId":"alpine.LFD.2.01.0905131420500.3343@localhost.localdomain","threadId":"19315","inReplyTo":"alpine.LNX.2.00.0905131639580.2147@iabervon.org","subject":"Re: Cross-Platform Version Control","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2009-05-13T21:29:24Z","receivedAt":"2009-05-13T21:29:24Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Wed, 13 May 2009, Daniel Barkalow wrote:\n> > \n> > Now, the simple OS X case is not a huge problem, since the lstat will \n> > succeed with the fixed-up filename too.\n> \n> I'm not seeing what the general case is, and how it could possibly behave.\n\nHere's a simple example.\n\nLet's say that your company uses Latin1 internally for your filesystems, \nbecause your tools really aren't utf-8 ready. \n\nThis is NOT AT ALL unnatural - it's how lots of people used to work with \nLinux over the years, and it's largely how people still use FAT, I suspect \n(except it's not latin1, it's some windows-specific 8-bits-per-character \nmapping).\n\n\nIOW, if you have a file called 'åäö', it literally is encoded as \n'\\xe5\\xe4\\xf6' (if you wonder why I picked those three letters, it's \nbecause they are the regular extra letters in Swedish - Swedish has 29 \nletters in its alphabet, and those three letters really are letters in \ntheir own right, they are NOT 'a' and 'o' with some dots/rings on top).\n\nIOW, if you open such a file, you need to use those three bytes.\n\nNow, even if you happen to have an OS and use Latin1 on disk, you may \nrealize that you'd like to interact with others that use UTF-8, and would \nwant to have your git archive that you export use nice portable UTF-8.\n\nBut you absolutely MUST NOT just do a conversion at \"readdir()\" time. If \nyou do that, then your three-byte filename turns into a six-byte utf-8 \nsequence of '\\xc3\\xa5\\xc3\\xa4\\xc3\\xb6' and the thing is, now \"lstat()\" \nwon't work on that sequence.\n\nSo obviously you could always turn things _back_ for lstat(), but quite \nfrankly, that's (a) insane (b) incompetent and (c) not even always \nwell-defined.\n\n> There's the \"insensitive\" behavior: if you create \"foo\" and look for \n> \"FOO\", it's there, but readdir() reports \"foo\".\n> \n> There's the \"converting\" behavior: if you create \"foo\", readdir() reports \n> \"FOO\", but lstat(\"foo\") returns it.\n\nThen there's the behaviour above: you want your git repository to have \nutf-8, but your filesystem doesn't convert anything at all, and all your \nregular tools (think editors etc) are all Latin1.\n\nLatin1 is going away, I hope, but I bet EUC-JP etc still exist. \n\n\t\tLinus\n"},{"id":"113852","messageId":"76718490905131430x6f3dcc7fv71b5c880f841be47@mail.gmail.com","threadId":"19315","inReplyTo":"alpine.LFD.2.01.0905131401170.3343@localhost.localdomain","subject":"Re: Cross-Platform Version Control","fromName":"Jay Soffian","fromEmail":"jaysoffian@gmail.com","sentAt":"2009-05-13T21:30:15Z","receivedAt":"2009-05-13T21:30:15Z","isPatch":false,"sender":{"key":"jaysoffian@gmail.com","avatar":"https://avatars.githubusercontent.com/u/155970?v=4"},"body":"On Wed, May 13, 2009 at 5:10 PM, Linus Torvalds\n wrote:\n> I have this dim memory of somebody actually pointing to the documentation\n> of exactly which characters OS X ends up decomposing.\n\nhttp://developer.apple.com/technotes/tn/tn1150.html#UnicodeSubtleties\nhttp://developer.apple.com/technotes/tn/tn1150table.html\n\nj.\n"},{"id":"113855","messageId":"op.utv1gwsi1e62zd@merlin.emma.line.org","threadId":"19315","inReplyTo":"alpine.LFD.2.01.0905131401170.3343@localhost.localdomain","subject":"Re: Cross-Platform Version Control","fromName":"Matthias Andree","fromEmail":"matthias.andree@gmx.de","sentAt":"2009-05-13T21:47:10Z","receivedAt":"2009-05-13T21:47:10Z","isPatch":false,"sender":{"key":"matthias.andree@gmx.de","avatar":null},"body":"Am 13.05.2009, 23:10 Uhr, schrieb Linus Torvalds \n:\n\n>\n>\n> On Wed, 13 May 2009, Matthias Andree wrote:\n>\n>> Am 13.05.2009, 19:12 Uhr, schrieb Linus Torvalds\n>> :\n>>\n>> > Use and stringprep_utf8_nfkc_normalize() or something \n>> to do\n>> > the actual normalization if you find characters with the high bit \n>> set. And\n>> > since I know that the OS X filesystems are so buggy as to not even do \n>> that\n>> > whole NFD thing right, there is probably some OS-X specific \"use this \n>> for\n>> > filesystem names\" conversion function.\n>>\n>> Sorry for interrupting, but NF_K_C? You don't want that (K for \n>> compatibility,\n>> rather than canonical, normalization) for anything except normalizing\n>> temporary variables inside strcasecmp(3) or similar. Probably not even \n>> that.\n>> The normalizations done are often irreversible and also surprising. You \n>> don't\n>> want to turn 2³.c into 23.c, do you?\n>\n> No, you're right. We want just plain NFC. I just googled for how some\n> other projects handled this, and found the stringprep thing in a post\n> about rsync, and didn't look any closer.\n>\n> But yes, you're absolutely right, stringprep is total crap, and nfkc is\n> horrible.\n\nCrap? It's just besides the purpose and some limited form of fuzzy match. \nAnyways...\n\n> I have no idea of what library to use, though. For perl, there's\n> Unicode::Normalize, but that's likely still subtly incorrect for the OS-X\n> case due to the filesystem not using _strict_ NFD.\n\nPerhaps ICU (ICU4C), from http://site.icu-project.org/\n\n-- \nMatthias Andree\n"},{"id":"113861","messageId":"buooctwqwr8.fsf@dhlpc061.dev.necel.com","threadId":"19315","inReplyTo":"43d8ce650905130657q2de9686q683fea84345a6651@mail.gmail.com","subject":"Re: Cross-Platform Version Control","fromName":"Miles Bader","fromEmail":"miles@gnu.org","sentAt":"2009-05-14T01:49:31Z","receivedAt":"2009-05-14T01:49:31Z","isPatch":false,"sender":{"key":"miles@gnu.org","avatar":"https://gravatar.com/avatar/01069b69593af7bff28e2f97afeb3644ae6fe2f5f56cb3a8cf34c5fb8c36efe5?d=mp&s=160"},"body":"John Tapsell writes:\n> I'm as much of an open source developer as anyone else here. I spend\n> a huge amount of my time programming for KDE. But I've never told a\n> user \"well that settles it\" because they won't code it themselves :-/\n\nFWIW, Johannes' use of \"Well, that rather settles things, no?\" in this\nthread this didn't strike me as being rude or truly dismissive (even\nthough it's literally so).\n\nIt seemed more just a timely and to the point reminder that however fun\nit is to talk about random feature X, someone's gotta do the work if\nit's going to actually be implemented, and that the direction of git\ndevelopment very much follows the whims of those doing the actual\nhacking (perhaps more so than other projects).\n\n[and I don't even have particularly thick skin, I think -- I'm often\nvery annoyed by brusqueness one sees on many developer mailing lists...]\n\n-Miles\n\n-- \nAcquaintance, n. A person whom we know well enough to borrow from, but not\nwell enough to lend to.\n"},{"id":"113937","messageId":"alpine.DEB.2.00.0905141441200.20117@perkele.intern.softwolves.pp.se","threadId":"19315","inReplyTo":"419AD153-53B4-4DAB-AF72-4127C17B1CA0@gmail.com","subject":"Re: Cross-Platform Version Control","fromName":"Peter Krefting","fromEmail":"peter@softwolves.pp.se","sentAt":"2009-05-14T13:48:40Z","receivedAt":"2009-05-14T13:48:40Z","isPatch":false,"sender":{"key":"peter@softwolves.pp.se","avatar":"https://avatars.githubusercontent.com/u/990764?v=4"},"body":"Esko Luontola:\n\n> A good start for making Git cross-platform, would be storing the text \n> encoding of every file name and commit message together with the commit.\n\nIs it really necessary to store the encoding for every single file name, \nshould it not be enough to just store encoding information for all file \nnames at once (i.e., for the object that contains the list of file names and \ntheir associated blobs)?\n\nI did publish, as a request for comments, the beginnings of a patch that \nwould change the Windows version of Git to expect file names to be UTF-8 \nencoded. There were some comments about it, especially that I could not just \nassume that UTF-8 was the right thing to assume.\n\nPerhaps if we added some meta-data, maybe using the same fall-back mechanism \nas for commit messages (i.e., assume UTF-8 unless otherwise specified), it \nwould be easier to do.\n\nOn Windows, the file APIs allow you to use Unicode (UTF-16) to specify file \nnames, and the file systems will handle any necessary conversion to whatever \nbyte sequences are used to store the file names. UTF-16 and UTF-8 are \ntrivial to convert between, and Windows does contain APIs to convert between \nother character encodings and UTF-16.\n\nOn Mac OS X, I believe the file system APIs assume you use some kind of \nnormalized UTF-8. That should also be possible to create, possibly \nconverting back and forth between different normalization forms, if necessary.\n\nOn Linux and other Unixes we could just use iconv() to convert from the \nrepository file name encoding to whatever the current locale has set up. The \ntrick here is to handle file names outside the current encoding. Some kind \nof escaping mechanism will probably need to be introduced.\n\nThe best way would be to define this in the Git core once and for all, and \nadd support to it for all the platforms in the same go, instead of trying to \nhack around the issue whenever it pops up on the various platforms.\n\nMy main use-case for Git on Windows has disappeared as my $dayjob went \nbankrupt, but I am happy to assist with whatever insight I may be able to \nbring.\n\n-- \n\\\\// Peter - http://www.softwolves.pp.se/\n"},{"id":"113943","messageId":"20090514175850.GA26267@macbook.lan","threadId":"19315","inReplyTo":"200905122356.00659.jnareb@gmail.com","subject":"[PATCH v2] Extend sample pre-commit hook to check for non ascii filenames","fromName":"Heiko Voigt","fromEmail":"hvoigt@hvoigt.net","sentAt":"2009-05-14T17:59:01Z","receivedAt":"2009-05-14T17:59:01Z","isPatch":true,"sender":{"key":"hvoigt@hvoigt.net","avatar":"https://avatars.githubusercontent.com/u/184958?v=4"},"body":"At the moment non-ascii encodings of filenames are not portably converted\nbetween different filesystems by git. This will most likely change in the\nfuture but to allow repositories to be portable among different file/operating\nsystems this check is enabled by default.\n\nSigned-off-by: Heiko Voigt \n---\nOn Tue, May 12, 2009 at 11:55:59PM +0200, Jakub Narebski wrote:\n> On Tue, 12 May 2009, Heiko Voigt wrote:\n> \n> > At the moment non-ascii encodings of file/usernames are not very well\n> > supported by git. This will most likely change in the future but to\n> > allow repositories to be portable among different file/operating systems\n> > this check is enabled by default.\n> \n> > +\t# non-ascii username issue a warning in git gui so tell the\n> > +\t# user to change it\n> > +\tif ! git config user.name | is_ascii; then\n> > +\t\techo \"Please only use ascii characters in your username!\"\n> > +\t\texit 1\n> > +\tfi\n> > +\n> > +\tif ! git config user.email | is_ascii; then\n> > +\t\techo \"Please only use ascii characters in your email!\"\n> > +\t\texit 1\n> > +\tfi\n> \n> Actually 1.) there is no easy way to avoid non-ASCII names at least\n> in user.name (I think they are not allowed in email), but 2.) there\n> is no trouble with non-ASCII encoding of commits, as they have \n> 'encoding' header if it is not uft-8 (see *encoding* config variables).\n\nI tried it and indeed it seems to work now. This hook originated from a\nwindows installation were having non-ascii characters resulted in a\nstrange warning from git gui each time you commit. So here is the\ncorrected patch.\n\n templates/hooks--pre-commit.sample | 20 ++++++++++++++++++++\n 1 files changed, 20 insertions(+), 0 deletions(-)\n\ndiff --git a/templates/hooks--pre-commit.sample b/templates/hooks--pre-commit.sample\nindex 0e49279..3083735 100755\n--- a/templates/hooks--pre-commit.sample\n+++ b/templates/hooks--pre-commit.sample\n@@ -7,6 +7,26 @@\n #\n # To enable this hook, rename this file to \"pre-commit\".\n \n+# If you want to allow non-ascii filenames set this variable to true.\n+allownonascii=$(git config hooks.allownonascii)\n+\n+function is_ascii () {\n+ test -z \"$(cat | sed -e \"s/[\\ -~]*//g\")\"\n+ return $?\n+}\n+\n+if [ \"$allownonascii\" != \"true\" ]\n+then\n+\t# until git can handle non-ascii filenames gracefully\n+\t# prevent them to be added into the repository\n+\tif ! git diff --cached --name-only --diff-filter=A -z \\\n+\t\t\t| tr \"\\0\" \"\\n\" | is_ascii; then\n+\t\techo \"Non-ascii filenames are not allowed !\"\n+\t\techo \"Please rename the file ...\"\n+\t\texit 1\n+\tfi\n+fi\n+\n if git-rev-parse --verify HEAD 2>/dev/null\n then\n \tagainst=HEAD\n-- \n1.6.3\n"},{"id":"113957","messageId":"4A0C77D9.5050402@gmail.com","threadId":"19315","inReplyTo":"alpine.DEB.2.00.0905141441200.20117@perkele.intern.softwolves.pp.se","subject":"Re: Cross-Platform Version Control","fromName":"Esko Luontola","fromEmail":"esko.luontola@gmail.com","sentAt":"2009-05-14T19:58:17Z","receivedAt":"2009-05-14T19:58:17Z","isPatch":false,"sender":{"key":"esko.luontola@gmail.com","avatar":"https://gravatar.com/avatar/86ea2beca289ab5e1e70b2d13b48e6c20e0e44478f3afe74b923090517f66a30?d=mp&s=160"},"body":"Peter Krefting wrote on 14.5.2009 16:48:\n> Is it really necessary to store the encoding for every single file name, \n> should it not be enough to just store encoding information for all file \n> names at once (i.e., for the object that contains the list of file names \n> and their associated blobs)?\n\nWhat about if some disorganized project has people committing with many \ndifferent encodings? Should we allow it, that a directory has the names \nof some files using one encoding, and the names of other files using \nanother encoding? Or should we force the whole repository to use the \nsame encoding?\n\n> The best way would be to define this in the Git core once and for all, \n> and add support to it for all the platforms in the same go, instead of \n> trying to hack around the issue whenever it pops up on the various \n> platforms.\n\n+1\n\n-- \nEsko Luontola\nwww.orfjackal.net\n"},{"id":"113958","messageId":"4A0C7D35.7090404@op5.se","threadId":"19315","inReplyTo":"4A0C77D9.5050402@gmail.com","subject":"Re: Cross-Platform Version Control","fromName":"Andreas Ericsson","fromEmail":"ae@op5.se","sentAt":"2009-05-14T20:21:09Z","receivedAt":"2009-05-14T20:21:09Z","isPatch":false,"sender":{"key":"ae@op5.se","avatar":"https://gravatar.com/avatar/426e89595c75a8f5252dd0c989e5fabe5bcac616e68557427ad9aef6b0ca342a?d=mp&s=160"},"body":"Esko Luontola wrote:\n> Peter Krefting wrote on 14.5.2009 16:48:\n>> Is it really necessary to store the encoding for every single file \n>> name, should it not be enough to just store encoding information for \n>> all file names at once (i.e., for the object that contains the list of \n>> file names and their associated blobs)?\n> \n> What about if some disorganized project has people committing with many \n> different encodings? Should we allow it, that a directory has the names \n> of some files using one encoding, and the names of other files using \n> another encoding? Or should we force the whole repository to use the \n> same encoding?\n> \n\nIf encodings are on a per-tree basis, we could add a special mode-flag for\nit without breaking backwards incompatibility (I think, anyways). Older\ngits just won't know how to handle it and will treat it as a byte-stream.\n\n>> The best way would be to define this in the Git core once and for all, \n>> and add support to it for all the platforms in the same go, instead of \n>> trying to hack around the issue whenever it pops up on the various \n>> platforms.\n> \n> +1\n> \n\nThere's still the problem that noone's stepped forward to do all that\nwork yet, so apparently this isn't important enough for people to put\ntheir patches where their mouths are. Often when issues generate long\ndiscussions and no code, it's of high academic interest and of little\nreal-world value.\n\nI believe the \"little real-world value\" here comes from the fact that\ncross-platform projects often enforce 7-bit ascii compatible filenames\nfrom the start, because they *know* they may run into problems on other\nfilesystems otherwise. Remember it's not only git that has to get\nthings right. It's also build-systems and compilers that have to locate\nthe correct files (the Makefile and the filesystem may use different\nencodings), so in the real world, people really do stay away from\nfilenames with åäö or other non-ascii chars in them.\n\nIt's fun to discuss, but I won't spend any time on it. Good luck to\nthose who do though. I'd quite like to see if someone could pull it\noff without breaking backwards compatibility or impacting performance\ntoo much.\n\n-- \nAndreas Ericsson andreas.ericsson@op5.se\nOP5 AB www.op5.se\nTel: +46 8-230225 Fax: +46 8-230231\n\nRegister now for Nordic Meet on Nagios, June 3-4 in Stockholm\n http://nordicmeetonnagios.op5.org/\n\nConsidering the successes of the wars on alcohol, poverty, drugs and\nterror, I think we should give some serious thought to declaring war\non peace.\n"},{"id":"113969","messageId":"alpine.DEB.1.00.0905150022400.26154@pacific.mpi-cbg.de","threadId":"19315","inReplyTo":"4A0C77D9.5050402@gmail.com","subject":"Re: Cross-Platform Version Control","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2009-05-14T22:25:34Z","receivedAt":"2009-05-14T22:25:34Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Thu, 14 May 2009, Esko Luontola wrote:\n\n> Peter Krefting wrote on 14.5.2009 16:48:\n> \n> > The best way would be to define this in the Git core once and for all, \n> > and add support to it for all the platforms in the same go, instead of \n> > trying to hack around the issue whenever it pops up on the various \n> > platforms.\n> \n> +1\n\nYou might be enthusiastic about this cunning idea. However, if it costs \nme performance on Linux, and all the benefits go to Windows users, then I \nwill remove this \"solution\" from my personal Git tree _right away_, and \nI'd expect a lot of other people, too.\n\nI repeat this just once more: if you add complexity, you'll have to have a \ncompelling reason to do so. If there is no benefit for Linux users, why \nshould they bear the cost?\n\nBut as Andreas remarked, I sincerely think that there has been enough talk \nabout the issue. It's time to see some patches, or to stop the \ndiscussion.\n\nCiao,\nDscho\n"},{"id":"113991","messageId":"46a038f90905150352q76157227q24c6b8c0befa4108@mail.gmail.com","threadId":"19315","inReplyTo":"20090514175850.GA26267@macbook.lan","subject":"Re: [PATCH v2] Extend sample pre-commit hook to check for non ascii filenames","fromName":"Martin Langhoff","fromEmail":"martin.langhoff@gmail.com","sentAt":"2009-05-15T10:52:41Z","receivedAt":"2009-05-15T10:52:41Z","isPatch":true,"sender":{"key":"martin.langhoff@gmail.com","avatar":"https://gravatar.com/avatar/1e3f311b6c4c15836501901ca58f8c0b0667246488084ba524d8bc9867e22fd9?d=mp&s=160"},"body":"On Thu, May 14, 2009 at 7:59 PM, Heiko Voigt wrote:\n> At the moment non-ascii encodings of filenames are not portably converted\n> between different filesystems by git. This will most likely change in the\n> future but to allow repositories to be portable among different file/operating\n> systems this check is enabled by default.\n\nNice!\n\n - It'd be a good idea to add to the mix a check for filenames that\nare equivalent in case-insensitive FSs.\n\n - Should all of this be a general \"portablefilenames\" setting?\n\ncheers,\n\n\nm\n-- \n martin.langhoff@gmail.com\n martin@laptop.org -- School Server Architect\n - ask interesting questions\n - don't get distracted with shiny stuff - working code first\n - http://wiki.laptop.org/go/User:Martinlanghoff\n"},{"id":"113992","messageId":"20090515111859.GB26812@dpotapov.dyndns.org","threadId":"19315","inReplyTo":"4A0C77D9.5050402@gmail.com","subject":"Re: Cross-Platform Version Control","fromName":"Dmitry Potapov","fromEmail":"dpotapov@gmail.com","sentAt":"2009-05-15T11:18:59Z","receivedAt":"2009-05-15T11:18:59Z","isPatch":false,"sender":{"key":"dpotapov@gmail.com","avatar":"https://avatars.githubusercontent.com/u/6568595?v=4"},"body":"On Thu, May 14, 2009 at 10:58:17PM +0300, Esko Luontola wrote:\n>\n> What about if some disorganized project has people committing with many \n> different encodings? Should we allow it, that a directory has the names \n> of some files using one encoding, and the names of other files using \n> another encoding? Or should we force the whole repository to use the \n> same encoding?\n\nThe whole repository should have the same encoding internally. Anything\nelse will be too complex and too slow... Have you seen any file system\nwhere file names would be stored in different encodings? And Git does\nfar more operation on file names than a file system does. So, it is\nclearly to me that the whole repository should have a single encoding.\n\nNow, I don't think that you will find many open source projects that use\nnon-ASCII in file names. Moreover, most Linux users are either use UTF-8\nalready or switch to it in the near future. Mac OS X uses UTF-8 (though\nthere is a problem with decomposed characters, but Linus posted a\npossible solution). So, the only platform were non-ASCII characters may\nbe interesting to Git users and that does not support UTF-8 is Windows.\nAFAIK, Cygwin 1.7 has UTF-8 support. So, it is mostly a problem for\nmsysGit... Though adding support for legacy encodings can help to some\ndegree, it means that every system call involving a file name will go\nthrough UTF-8 <-> LEGACY_ENC <-> UTF-16LE conversion. IMHO, having a\nlegacy encoding involved is far from the best possible solution; but\nto avoid that, you need to change MSYS to be able to work with UTF-8.\n(I have never looked at MSYS myself, but I suspect it may be not easy).\n\n\nDmitry\n"},{"id":"113996","messageId":"200905151657.47225.jnareb@gmail.com","threadId":"19315","inReplyTo":"20090514175850.GA26267@macbook.lan","subject":"Re: [PATCH v2] Extend sample pre-commit hook to check for non ascii filenames","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2009-05-15T14:57:45Z","receivedAt":"2009-05-15T14:57:45Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":" ;-)\n\nIn short: good idea, don't be discouraged by criticism...\n\nOn Thu, 14 May 2009, Heiko Voigt wrote:\n\n> At the moment non-ascii encodings of filenames are not portably converted\n> between different filesystems by git. This will most likely change in the\n> future but to allow repositories to be portable among different file/operating\n> systems this check is enabled by default.\n\nBy the way, you might consider choosing shorter line length for commits,\nsomething around 70-76 characters per line; otherwise it is harder to\nreply to without linewrapping. 80 characters that you used is, IMHO,\nabsolute maximum, and it is good that you kept to it.\n\n> \n> Signed-off-by: Heiko Voigt \n> ---\n\n> +# If you want to allow non-ascii filenames set this variable to true.\n> +allownonascii=$(git config hooks.allownonascii)\n> +\n> +function is_ascii () {\n> + test -z \"$(cat | sed -e \"s/[\\ -~]*//g\")\"\n> + return $?\n> +}\n\n>From CodingGuidelines for shell scripts:\n - We do not write the noiseword \"function\" in front of shell\n functions.\n\n(in short: do not use bash-specific features... unless, of course,\nyou are modifying bash-completion script).\n\nSecond, it would be nice to have comment about how to use this\nfunction (as it does not check file given by its argument, but\nrather its standard input). And perhaps also a comment that it\nworks because ASCII printable characters begin with ' ' space\n(does it have to be escaped?) and end with '~' tilde[2].\n\nThird, isn't it useless use of 'cat'[3]? And wouldn't it be better\nto use 'tr' to either delete printable characters and check for\nanything left (as you do; BTW. wouldn't \"return test ...\" be simpler?),\nor use 'tr' to count non portable characters?\n\n[1] http://www.dwheeler.com/essays/fixing-unix-linux-filenames.html\n[2] http://en.wikipedia.org/wiki/ASCII#ASCII_printable_characters\n[3] http://partmaps.org/era/unix/award.html#cat\n\n> +\n> +if [ \"$allownonascii\" != \"true\" ]\n> +then\n> +\t# until git can handle non-ascii filenames gracefully\n> +\t# prevent them to be added into the repository\n> +\tif ! git diff --cached --name-only --diff-filter=A -z \\\n> +\t\t\t| tr \"\\0\" \"\\n\" | is_ascii; then\n> +\t\techo \"Non-ascii filenames are not allowed !\"\n> +\t\techo \"Please rename the file ...\"\n> +\t\texit 1\n> +\tfi\n> +fi\n> +\n> if git-rev-parse --verify HEAD 2>/dev/null\n> then\n> \tagainst=HEAD\n> -- \n> 1.6.3\n> \n> \n> \n> \n\n-- \nJakub Narebski\nPoland\n"},{"id":"114011","messageId":"7vbpputewv.fsf@alter.siamese.dyndns.org","threadId":"19315","inReplyTo":"20090514175850.GA26267@macbook.lan","subject":"Re: [PATCH v2] Extend sample pre-commit hook to check for non ascii filenames","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2009-05-15T18:11:12Z","receivedAt":"2009-05-15T18:11:12Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Heiko Voigt writes:\n\n> diff --git a/templates/hooks--pre-commit.sample b/templates/hooks--pre-commit.sample\n> index 0e49279..3083735 100755\n> --- a/templates/hooks--pre-commit.sample\n> +++ b/templates/hooks--pre-commit.sample\n> @@ -7,6 +7,26 @@\n> #\n> # To enable this hook, rename this file to \"pre-commit\".\n> \n> +# If you want to allow non-ascii filenames set this variable to true.\n> +allownonascii=$(git config hooks.allownonascii)\n> +\n> +function is_ascii () {\n\nWe do not say \"#!/bin/bash\" at the beginning (hopefully), so let's not say\n\"function \" here.\n\n> + test -z \"$(cat | sed -e \"s/[\\ -~]*//g\")\"\n\nDo you need \"cat | \"?\n\nDoes this script run under LC_ALL=C? Can an i18n'ized sed interfere with\nwhat you are trying to do?\n\n> + return $?\n\nDo you need this, or does the function return the result of the last\nstatment anyway?\n\n> +\t\techo \"Non-ascii filenames are not allowed !\"\n> +\t\techo \"Please rename the file ...\"\n\nCan we make this sound more like a _sample_ project policy? It's not like\nwe enforce that policy to other people's projects.\n\n> +\t\texit 1\n> +\tfi\n> +fi\n> +\n> if git-rev-parse --verify HEAD 2>/dev/null\n> then\n> \tagainst=HEAD\n> -- \n> 1.6.3\n"},{"id":"114170","messageId":"20090518093655.GA910@heiko-voigts-macbook.local","threadId":"19315","inReplyTo":"46a038f90905150352q76157227q24c6b8c0befa4108@mail.gmail.com","subject":"Re: [PATCH v2] Extend sample pre-commit hook to check for non ascii filenames","fromName":"Heiko Voigt","fromEmail":"hvoigt@hvoigt.net","sentAt":"2009-05-18T09:37:10Z","receivedAt":"2009-05-18T09:37:10Z","isPatch":true,"sender":{"key":"hvoigt@hvoigt.net","avatar":"https://avatars.githubusercontent.com/u/184958?v=4"},"body":"On Fri, May 15, 2009 at 12:52:41PM +0200, Martin Langhoff wrote:\n> On Thu, May 14, 2009 at 7:59 PM, Heiko Voigt wrote:\n> > At the moment non-ascii encodings of filenames are not portably converted\n> > between different filesystems by git. This will most likely change in the\n> > future but to allow repositories to be portable among different file/operating\n> > systems this check is enabled by default.\n> \n> Nice!\n> \n> - It'd be a good idea to add to the mix a check for filenames that\n> are equivalent in case-insensitive FSs.\n\nI agree, but that will be an extension in another patch. BTW, if anyone\nhas a good idea how to efficiently do that kind of check in a hook I'd\ncook up a patch on top of this.\n\n> - Should all of this be a general \"portablefilenames\" setting?\n\nWell, if you can specify what general portable filenames would have as\nproperties.\n\nQuestions like:\n\n * What is the portable maximum path length?\n * How long may a filename be (DOS 8.3 ?)\n * Are windows keywords (PRN, ...) allowed?\n * ...\n\nSo I think this should be on a per property basis providing sensible\ndefaults to support the most standard case.\n\ncheers Heiko\n"},{"id":"114172","messageId":"20090518095044.GB910@heiko-voigts-macbook.local","threadId":"19315","inReplyTo":"200905151657.47225.jnareb@gmail.com","subject":"[PATCH] Extend sample pre-commit hook to check for non ascii filenames","fromName":"Heiko Voigt","fromEmail":"hvoigt@hvoigt.net","sentAt":"2009-05-18T09:50:44Z","receivedAt":"2009-05-18T09:50:44Z","isPatch":true,"sender":{"key":"hvoigt@hvoigt.net","avatar":"https://avatars.githubusercontent.com/u/184958?v=4"},"body":"At the moment non-ascii encodings of filenames are not portably converted\nbetween different filesystems by git. This will most likely change in the\nfuture but to allow repositories to be portable among different file/operating\nsystems this check is enabled by default.\n\nSigned-off-by: Heiko \n---\nso here is a third version ...\n\nOn Fri, May 15, 2009 at 04:57:45PM +0200, Jakub Narebski wrote:\n> On Thu, 14 May 2009, Heiko Voigt wrote:\n> \n> > At the moment non-ascii encodings of filenames are not portably converted\n> > between different filesystems by git. This will most likely change in the\n> > future but to allow repositories to be portable among different file/operating\n> > systems this check is enabled by default.\n> \n> By the way, you might consider choosing shorter line length for commits,\n> something around 70-76 characters per line; otherwise it is harder to\n> reply to without linewrapping. 80 characters that you used is, IMHO,\n> absolute maximum, and it is good that you kept to it.\n\nYeah, I admit they were a little bit long.\n\n> > +function is_ascii () {\n> > + test -z \"$(cat | sed -e \"s/[\\ -~]*//g\")\"\n> > + return $?\n> > +}\n> \n> From CodingGuidelines for shell scripts:\n> - We do not write the noiseword \"function\" in front of shell\n> functions.\n> \n> (in short: do not use bash-specific features... unless, of course,\n> you are modifying bash-completion script).\n\nAddressed.\n\n> Second, it would be nice to have comment about how to use this\n> function (as it does not check file given by its argument, but\n> rather its standard input). And perhaps also a comment that it\n> works because ASCII printable characters begin with ' ' space\n> (does it have to be escaped?) and end with '~' tilde[2].\n\nDone\n\n> \n> Third, isn't it useless use of 'cat'[3]? And wouldn't it be better\n> to use 'tr' to either delete printable characters and check for\n> anything left (as you do; BTW. wouldn't \"return test ...\" be simpler?),\n> or use 'tr' to count non portable characters?\n\nYes indeed it was useless. I also switched from sed to tr.\n\n> \n> [1] http://www.dwheeler.com/essays/fixing-unix-linux-filenames.html\n> [2] http://en.wikipedia.org/wiki/ASCII#ASCII_printable_characters\n> [3] http://partmaps.org/era/unix/award.html#cat\n\nOn Fri, May 15, 2009 at 11:11:12AM -0700, Junio C Hamano wrote:\n> Heiko Voigt writes:\n> > +function is_ascii () {\n> \n> We do not say \"#!/bin/bash\" at the beginning (hopefully), so let's not say\n> \"function \" here.\n\nSee above.\n\n> > + test -z \"$(cat | sed -e \"s/[\\ -~]*//g\")\"\n> \n> Do you need \"cat | \"?\n\nAlso above.\n\n> Does this script run under LC_ALL=C? Can an i18n'ized sed interfere with\n> what you are trying to do?\n\nI now explicitely set LC_ALL=C for the tr call which should now be robust\nagainst such cases.\n\n> \n> > + return $?\n> \n> Do you need this, or does the function return the result of the last\n> statment anyway?\n\nI wasn't aware of that. Removed the return.\n\n> > +\t\techo \"Non-ascii filenames are not allowed !\"\n> > +\t\techo \"Please rename the file ...\"\n> \n> Can we make this sound more like a _sample_ project policy? It's not like\n> we enforce that policy to other people's projects.\n\nI've polished this so we are now more user friendly as well.\n\n templates/hooks--pre-commit.sample | 32 ++++++++++++++++++++++++++++++++\n 1 files changed, 32 insertions(+), 0 deletions(-)\n\ndiff --git a/templates/hooks--pre-commit.sample b/templates/hooks--pre-commit.sample\nindex 0e49279..91ab563 100755\n--- a/templates/hooks--pre-commit.sample\n+++ b/templates/hooks--pre-commit.sample\n@@ -7,6 +7,38 @@\n #\n # To enable this hook, rename this file to \"pre-commit\".\n \n+# If you want to allow non-ascii filenames set this variable to true.\n+allownonascii=$(git config hooks.allownonascii)\n+\n+# is_ascii() Tests the string given given on standard input for\n+# printable ascii conformance. We exploit the fact that the printable\n+# range starts at the space character and ends with tilde.\n+is_ascii() {\n+ test -z \"$(LC_ALL=C tr -d \\ -~)\"\n+}\n+\n+if [ \"$allownonascii\" != \"true\" ]\n+then\n+\t# until git can handle non-ascii filenames gracefully\n+\t# prevent them to be added into the repository\n+\tif ! git diff --cached --name-only --diff-filter=A -z \\\n+\t | tr \"\\0\" \"\\n\" | is_ascii; then\n+\t\techo \"Error: Preventing to add a non-ascii filename.\"\n+\t\techo\n+\t\techo \"This can cause problems if you want to work together\"\n+\t\techo \"with people on other platforms than you.\"\n+\t\techo\n+\t\techo \"To be portable it is adviseable to rename the file ...\"\n+\t\techo\n+\t\techo \"If you know what you are doing you can disable this\"\n+\t\techo \"check using:\"\n+\t\techo\n+\t\techo \" git config hooks.allownonascii true\"\n+\t\techo\n+\t\texit 1\n+\tfi\n+fi\n+\n if git-rev-parse --verify HEAD 2>/dev/null\n then\n \tagainst=HEAD\n-- \n1.6.3\n"},{"id":"114173","messageId":"4A113B09.6040301@viscovery.net","threadId":"19315","inReplyTo":"20090518095044.GB910@heiko-voigts-macbook.local","subject":"Re: [PATCH] Extend sample pre-commit hook to check for non ascii filenames","fromName":"Johannes Sixt","fromEmail":"j.sixt@viscovery.net","sentAt":"2009-05-18T10:40:09Z","receivedAt":"2009-05-18T10:40:09Z","isPatch":true,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"Heiko Voigt schrieb:\n> +# is_ascii() Tests the string given given on standard input for\n> +# printable ascii conformance. We exploit the fact that the printable\n> +# range starts at the space character and ends with tilde.\n> +is_ascii() {\n> + test -z \"$(LC_ALL=C tr -d \\ -~)\"\n> +}\n> +\n> +if [ \"$allownonascii\" != \"true\" ]\n> +then\n> +\t# until git can handle non-ascii filenames gracefully\n> +\t# prevent them to be added into the repository\n> +\tif ! git diff --cached --name-only --diff-filter=A -z \\\n> +\t | tr \"\\0\" \"\\n\" | is_ascii; then\n\nWill this not fail to add more than one file with allowed names? The \\n is\nnot removed in is_ascii(), and so the resulting string will not be empty.\n\nBTW, not all tr work well with NULs. See the commit message of e85fe4d8,\nfor example. Otherwise, I would have suggested to convert the NUL to some\nallowed ASCII character, e.g. 'A'. BTW, you should really use '\\0' and\n'\\n' (single-quotes) to guarantee that the shell does not ignore the\nbackslash.\n\n-- Hannes\n"},{"id":"114175","messageId":"20090518115002.GA3374@heiko-voigts-macbook.local","threadId":"19315","inReplyTo":"4A113B09.6040301@viscovery.net","subject":"Re: [PATCH] Extend sample pre-commit hook to check for non ascii filenames","fromName":"Heiko Voigt","fromEmail":"hvoigt@hvoigt.net","sentAt":"2009-05-18T11:50:11Z","receivedAt":"2009-05-18T11:50:11Z","isPatch":true,"sender":{"key":"hvoigt@hvoigt.net","avatar":"https://avatars.githubusercontent.com/u/184958?v=4"},"body":"On Mon, May 18, 2009 at 12:40:09PM +0200, Johannes Sixt wrote:\n> Heiko Voigt schrieb:\n> > +# is_ascii() Tests the string given given on standard input for\n> > +# printable ascii conformance. We exploit the fact that the printable\n> > +# range starts at the space character and ends with tilde.\n> > +is_ascii() {\n> > + test -z \"$(LC_ALL=C tr -d \\ -~)\"\n> > +}\n> > +\n> > +if [ \"$allownonascii\" != \"true\" ]\n> > +then\n> > +\t# until git can handle non-ascii filenames gracefully\n> > +\t# prevent them to be added into the repository\n> > +\tif ! git diff --cached --name-only --diff-filter=A -z \\\n> > +\t | tr \"\\0\" \"\\n\" | is_ascii; then\n> \n> Will this not fail to add more than one file with allowed names? The \\n is\n> not removed in is_ascii(), and so the resulting string will not be empty.\n\nNo currently it does not. At least on my system, but good point.\n\n> BTW, not all tr work well with NULs. See the commit message of e85fe4d8,\n> for example. Otherwise, I would have suggested to convert the NUL to some\n> allowed ASCII character, e.g. 'A'. BTW, you should really use '\\0' and\n> '\\n' (single-quotes) to guarantee that the shell does not ignore the\n> backslash.\n\nAre there any problems with '\\0' and tr other than swallowing of it. In\ncase not I would just change\n\n\ttr \"\\0\" \"\\n\"\nto\n \ttr -d '\\0'\n\nThat way there are no '\\n's left over and it doesn't matter if tr\nswallows the '\\0'.\n\nWaiting for further comments before sending the cleanup.\n\ncheers Heiko\n"},{"id":"114176","messageId":"4A114EB8.60609@viscovery.net","threadId":"19315","inReplyTo":"20090518115002.GA3374@heiko-voigts-macbook.local","subject":"Re: [PATCH] Extend sample pre-commit hook to check for non ascii filenames","fromName":"Johannes Sixt","fromEmail":"j.sixt@viscovery.net","sentAt":"2009-05-18T12:04:08Z","receivedAt":"2009-05-18T12:04:08Z","isPatch":true,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"Heiko Voigt schrieb:\n> Are there any problems with '\\0' and tr other than swallowing of it.\n\nI can't tell. But the commits ae90e16..aab0abf are interesting to study in\nw.r.t. portability.\n\n> In\n> case not I would just change\n> \n> \ttr \"\\0\" \"\\n\"\n> to\n> \ttr -d '\\0'\n\nIn which case I'd suggest that you call tr only once, in isascii():\n\n tr -d '[ -~]\\0'\n\n-- Hannes\n"},{"id":"114181","messageId":"7v3ab2ealk.fsf@alter.siamese.dyndns.org","threadId":"19315","inReplyTo":"20090518095044.GB910@heiko-voigts-macbook.local","subject":"Re: [PATCH] Extend sample pre-commit hook to check for non ascii filenames","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2009-05-18T14:42:31Z","receivedAt":"2009-05-18T14:42:31Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Heiko Voigt writes:\n\n> +if [ \"$allownonascii\" != \"true\" ]\n> +then\n> +\t# until git can handle non-ascii filenames gracefully\n> +\t# prevent them to be added into the repository\n\nI think you can inline your is_ascii shell function in the pipeline below.\nYou made it a separate function and I agree that it has a very good\ndocumentation value, but the mention of \"non-ascii filenames\" in this\ncomment here is enough clue to let anybody know what is going on.\n\n\tSide note: I am not sure \"Until ... can ... gracefully\" is a good\n\tdescription of the general problem. It probably is more neutral\n\tto say \"Cross platform projects tend to avoid non-ascii filenames;\n prevent them from being added to the repository.\"\n\n> +\tif ! git diff --cached --name-only --diff-filter=A -z \\\n> +\t | tr \"\\0\" \"\\n\" | is_ascii; then\n\nA standard trick while writing a long pipeline in shell is to change line\nafter a pipe, like:\n\n\tcmd1 |\n cmd2 |\n cmd3\n\nwhich allows you to lose the BS-before-LF sequence.\n\nI think comments from J6t and others are valuable but clear enough that I\nwouldn't have to repeat them.\n"},{"id":"114199","messageId":"alpine.LNX.2.00.0905182132170.27823@reaper.quantumfyre.co.uk","threadId":"19315","inReplyTo":"20090518095044.GB910@heiko-voigts-macbook.local","subject":"Re: [PATCH] Extend sample pre-commit hook to check for non ascii filenames","fromName":"Julian Phillips","fromEmail":"julian@quantumfyre.co.uk","sentAt":"2009-05-18T20:35:19Z","receivedAt":"2009-05-18T20:35:19Z","isPatch":true,"sender":{"key":"julian@quantumfyre.co.uk","avatar":"https://avatars.githubusercontent.com/u/948888?v=4"},"body":"On Mon, 18 May 2009, Heiko Voigt wrote:\n\n> +if [ \"$allownonascii\" != \"true\" ]\n> +then\n> +\t# until git can handle non-ascii filenames gracefully\n> +\t# prevent them to be added into the repository\n> +\tif ! git diff --cached --name-only --diff-filter=A -z \\\n> +\t | tr \"\\0\" \"\\n\" | is_ascii; then\n> +\t\techo \"Error: Preventing to add a non-ascii filename.\"\n\nThis would read better as:\n\n+\t\techo \"Error: Attempt to add a non-ascii filename.\"\n\n(after all the prevention itself is a result of the error, not the cause \nof it)\n\nIf you want to keep the preventing, then you need to at least correct the \nenglish:\n\n> +\t\techo \"Error: Preventing addition of a non-ascii filename.\"\n\n-- \nJulian\n\n ---\nQOTD:\n \tMoney isn't everything, but at least it keeps the kids in touch.\n"},{"id":"114204","messageId":"200905190026.37291.jnareb@gmail.com","threadId":"19315","inReplyTo":"20090518093655.GA910@heiko-voigts-macbook.local","subject":"Re: [PATCH v2] Extend sample pre-commit hook to check for non ascii filenames","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2009-05-18T22:26:35Z","receivedAt":"2009-05-18T22:26:35Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"On Mon, 18 May 2009, Heiko Voigt wrote:\n> On Fri, May 15, 2009 at 12:52:41PM +0200, Martin Langhoff wrote:\n\n> >  - Should all of this be a general \"portablefilenames\" setting?\n> \n> Well, if you can specify what general portable filenames would have as\n> properties.\n\n\"Fixing Unix/Linux/POSIX Filenames: Control Characters (such as \nNewline), Leading Dashes, and Other Problems\" by David A. Wheeler\nhttp://www.dwheeler.com/essays/fixing-unix-linux-filenames.html\n\n-- \nJakub Narebski\nPoland\n"},{"id":"114300","messageId":"20090519200153.GA1926@heiko-voigts-macbook.local","threadId":"19315","inReplyTo":"4A113B09.6040301@viscovery.net","subject":"[PATCH v4] Extend sample pre-commit hook to check for non ascii filenames","fromName":"Heiko Voigt","fromEmail":"hvoigt@hvoigt.net","sentAt":"2009-05-19T20:01:54Z","receivedAt":"2009-05-19T20:01:54Z","isPatch":true,"sender":{"key":"hvoigt@hvoigt.net","avatar":"https://avatars.githubusercontent.com/u/184958?v=4"},"body":"At the moment non-ascii encodings of filenames are not portably\nconverted between different filesystems by git. This will most likely\nchange in the future but to allow repositories to be portable among\ndifferent file/operating systems this check is enabled by default.\n\nSigned-off-by: Heiko Voigt \n---\n\nThanks for all comments. I now hopefully have a satisfying patch.\n\n\nOn Mon, May 18, 2009 at 12:40:09PM +0200, Johannes Sixt wrote:\n> Heiko Voigt schrieb:\n> > +\tif ! git diff --cached --name-only --diff-filter=A -z \\\n> > +\t | tr \"\\0\" \"\\n\" | is_ascii; then\n> \n> Will this not fail to add more than one file with allowed names? The \\n is\n> not removed in is_ascii(), and so the resulting string will not be empty.\n> \n> BTW, not all tr work well with NULs. See the commit message of e85fe4d8,\n> for example. Otherwise, I would have suggested to convert the NUL to some\n> allowed ASCII character, e.g. 'A'. BTW, you should really use '\\0' and\n> '\\n' (single-quotes) to guarantee that the shell does not ignore the\n> backslash.\n\nI removed all \\0 characters and hopefully use the correct platform\nindependent syntax as described in the commits you send.\n\n\nOn Mon, May 18, 2009 at 02:04:08PM +0200, Johannes Sixt wrote:\n> Heiko Voigt schrieb:\n> > Are there any problems with '\\0' and tr other than swallowing of it.\n> \n> I can't tell. But the commits ae90e16..aab0abf are interesting to study in\n> w.r.t. portability.\n> \n> > In\n> > case not I would just change\n> > \n> > \ttr \"\\0\" \"\\n\"\n> > to\n> > \ttr -d '\\0'\n> \n> In which case I'd suggest that you call tr only once, in isascii():\n> \n> tr -d '[ -~]\\0'\n\nAfter reading a little about the portability things. This seems to be\nthe right way and is now included.\n\n\nOn Mon, May 18, 2009 at 07:42:31AM -0700, Junio C Hamano wrote:\n> Heiko Voigt writes:\n> \n> > +if [ \"$allownonascii\" != \"true\" ]\n> > +then\n> > +\t# until git can handle non-ascii filenames gracefully\n> > +\t# prevent them to be added into the repository\n> \n> I think you can inline your is_ascii shell function in the pipeline below.\n> You made it a separate function and I agree that it has a very good\n> documentation value, but the mention of \"non-ascii filenames\" in this\n> comment here is enough clue to let anybody know what is going on.\n\nI agree. I thought it would probably be useful in other places but we\njust need it once so its inlined now.\n\n> \n> \tSide note: I am not sure \"Until ... can ... gracefully\" is a good\n> \tdescription of the general problem. It probably is more neutral\n> \tto say \"Cross platform projects tend to avoid non-ascii filenames;\n> prevent them from being added to the repository.\"\n\nChanged that.\n\n> \n> > +\tif ! git diff --cached --name-only --diff-filter=A -z \\\n> > +\t | tr \"\\0\" \"\\n\" | is_ascii; then\n> \n> A standard trick while writing a long pipeline in shell is to change line\n> after a pipe, like:\n> \n> \tcmd1 |\n> cmd2 |\n> cmd3\n> \n> which allows you to lose the BS-before-LF sequence.\n\nWasn't aware of that. Changed it accordingly.\n\n\nOn Mon, May 18, 2009 at 09:35:19PM +0100, Julian Phillips wrote:\n> On Mon, 18 May 2009, Heiko Voigt wrote:\n>> +\t\techo \"Error: Preventing to add a non-ascii filename.\"\n>\n> This would read better as:\n>\n> +\t\techo \"Error: Attempt to add a non-ascii filename.\"\n>\n> (after all the prevention itself is a result of the error, not the cause \n> of it)\n\nThat really sounds better. Thanks.\n\n templates/hooks--pre-commit.sample | 25 +++++++++++++++++++++++++\n 1 files changed, 25 insertions(+), 0 deletions(-)\n\ndiff --git a/templates/hooks--pre-commit.sample b/templates/hooks--pre-commit.sample\nindex 0e49279..ad892a2 100755\n--- a/templates/hooks--pre-commit.sample\n+++ b/templates/hooks--pre-commit.sample\n@@ -7,6 +7,31 @@\n #\n # To enable this hook, rename this file to \"pre-commit\".\n \n+# If you want to allow non-ascii filenames set this variable to true.\n+allownonascii=$(git config hooks.allownonascii)\n+\n+# Cross platform projects tend to avoid non-ascii filenames; prevent\n+# them from being added to the repository. We exploit the fact that the\n+# printable range starts at the space character and ends with tilde.\n+if [ \"$allownonascii\" != \"true\" ] &&\n+\ttest \"$(git diff --cached --name-only --diff-filter=A -z |\n+\t LC_ALL=C tr -d '[ -~]\\0')\"\n+then\n+\techo \"Error: Attempt to add a non-ascii filename.\"\n+\techo\n+\techo \"This can cause problems if you want to work together\"\n+\techo \"with people on other platforms than you.\"\n+\techo\n+\techo \"To be portable it is adviseable to rename the file ...\"\n+\techo\n+\techo \"If you know what you are doing you can disable this\"\n+\techo \"check using:\"\n+\techo\n+\techo \" git config hooks.allownonascii true\"\n+\techo\n+\texit 1\n+fi\n+\n if git-rev-parse --verify HEAD 2>/dev/null\n then\n \tagainst=HEAD\n-- \n1.6.3\n"},{"id":"116670","messageId":"20090620121427.GC4876@heiko-voigts-macbook.local","threadId":"19315","inReplyTo":"46a038f90905150352q76157227q24c6b8c0befa4108@mail.gmail.com","subject":"[RFC PATCH] check for filenames that only differ in case to sample pre-commit hook","fromName":"Heiko Voigt","fromEmail":"hvoigt@hvoigt.net","sentAt":"2009-06-20T12:14:39Z","receivedAt":"2009-06-20T12:14:39Z","isPatch":true,"sender":{"key":"hvoigt@hvoigt.net","avatar":"https://avatars.githubusercontent.com/u/184958?v=4"},"body":"This helps cross-platform projects on the case-sensitive filename side\nof operating systems to use filenames that are nice for the\ncase-insensitive side\n\n---\nOn Fri, May 15, 2009 at 12:52:41PM +0200, Martin Langhoff wrote:\n> On Thu, May 14, 2009 at 7:59 PM, Heiko Voigt wrote:\n> > At the moment non-ascii encodings of filenames are not portably converted\n> > between different filesystems by git. This will most likely change in the\n> > future but to allow repositories to be portable among different file/operating\n> > systems this check is enabled by default.\n> - It'd be a good idea to add to the mix a check for filenames that\n> are equivalent in case-insensitive FSs.\n\nTotally untested. Just to get feedback if someone has ideas how this can\nbe solved more efficiently. I suspect that processing all files will\nyield an unbearable performance degradation on large projects.\n\nLet me know what you think. The wording of the error message is not yet\nfinal.\n\n templates/hooks--pre-commit.sample | 21 +++++++++++++++++++++\n 1 files changed, 21 insertions(+), 0 deletions(-)\n\ndiff --git a/templates/hooks--pre-commit.sample b/templates/hooks--pre-commit.sample\nindex b11ad6a..32d1809 100755\n--- a/templates/hooks--pre-commit.sample\n+++ b/templates/hooks--pre-commit.sample\n@@ -9,6 +9,10 @@\n \n # If you want to allow non-ascii filenames set this variable to true.\n allownonascii=$(git config hooks.allownonascii)\n+# If you want to allow filenames that only differ in case set this\n+# variable to true. NOTE: This can degrade performance on project with\n+# lots of files\n+allowcaseonly=$(git config hooks.allowcaseonly)\n \n # Cross platform projects tend to avoid non-ascii filenames; prevent\n # them from being added to the repository. We exploit the fact that the\n@@ -32,6 +36,23 @@ then\n \texit 1\n fi\n \n+# check for names that already exist but only differ in case\n+# which can be problematic on non-casesensitive filesystems\n+if [ \"$allowcaseonly\" != \"true\" ] &&\n+\ttest -z \"$(git ls-files | LC_ALL=C tr -s [A-Z] [a-z] | uniq -d)\"\n+then\n+\techo \"Error: Attempt to add file which already exists in different case\"\n+\techo\n+\techo \"If you know what you are doing you can disable this\"\n+\techo \"check using:\"\n+\techo\n+\techo \" git config hooks.allowcaseonly true\"\n+\techo\n+\texit 1\n+fi\n+\n if git-rev-parse --verify HEAD >/dev/null 2>&1\n then\n \tagainst=HEAD\n-- \n1.6.3.2.203.g9a122\n"}]}