{"thread":{"id":"4994","subject":"Moving a directory into another fails","startedAt":"2006-07-26T15:00:48Z","lastAt":"2006-12-05T17:11:35Z","messageCount":24,"participants":["Jon Smirl","Nicolas Vilz","Petr Baudis","Linus Torvalds","Johannes Schindelin","Stefan Pfetzing","Shawn Pearce","Jakub Narebski"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"24136","messageId":"9e4733910607260800v618edf0em7b0f5c3332bf8fc5@mail.gmail.com","threadId":"4994","inReplyTo":null,"subject":"Moving a directory into another fails","fromName":"Jon Smirl","fromEmail":"jonsmirl@gmail.com","sentAt":"2006-07-26T15:00:48Z","receivedAt":"2006-07-26T15:00:48Z","isPatch":false,"sender":{"key":"jonsmirl@gmail.com","avatar":"https://gravatar.com/avatar/cff3bf5bfdfa6708b905712ff91f0f9b8aaca161659f38c02b787920d5d28b7e?d=mp&s=160"},"body":"I cloned a git project. Then in the original I did mkdir for a new\ndirectory and use git mv to move an existing directory into it. I then\nused cg diff to generate a patch for the move.\n\nWhen I use cg patch to apply this patch to the cloned tree it fails.\nThis seems to be a problem in the git code, not cg.  It is not picking\nup the creation of the new intervening subdirectory correctly.\n\nI just synced and this does not work in the current code.\n\n-- \nJon Smirl\njonsmirl@gmail.com\n"},{"id":"24174","messageId":"20060726223459.GA30601@vsectoor.geht-ab-wie-schnitzel.de","threadId":"4994","inReplyTo":"9e4733910607260800v618edf0em7b0f5c3332bf8fc5@mail.gmail.com","subject":"Re: Moving a directory into another fails","fromName":"Nicolas Vilz","fromEmail":"niv@iaglans.de","sentAt":"2006-07-26T22:34:59Z","receivedAt":"2006-07-26T22:34:59Z","isPatch":false,"sender":{"key":"niv@iaglans.de","avatar":"https://gravatar.com/avatar/e4d43a32d721241212d4edb1d2210327e28423c913071b4bfeeaa0ce15296110?d=mp&s=160"},"body":"On Wed, Jul 26, 2006 at 11:00:48AM -0400, Jon Smirl wrote:\n> I cloned a git project. Then in the original I did mkdir for a new\n> directory and use git mv to move an existing directory into it. I then\n> used cg diff to generate a patch for the move.\n> \n> When I use cg patch to apply this patch to the cloned tree it fails.\n> This seems to be a problem in the git code, not cg.  It is not picking\n> up the creation of the new intervening subdirectory correctly.\n> \n> I just synced and this does not work in the current code.\n\nI tried to reproduce your scenario and before that I setup a test\nrepository. \n\n(1) mkdir git_test\n(2) cd git_test\n(3) git init-db\n(4) vim test.txt \n# fill in some bogus text\n(5) mkdir testing\n(6) cd testing\n(7) vim test1.txt\n# again, fill in some bogus text\n(8) cd ..\n(9) cg add test.txt testing/test1.txt\n(10) cg commit -C \n# just give a fancy commit message...\n(11) cd ..\n(12) mkdir bare_git\n(13) cd bare_git\n(14) mkdir git_test.git\n(15) GIT_DIR=git_test.git git init-db \n(16) cd ../git_test\n(17) git push ../bare_git/git_test.git --all\n(18) cd ../\n(19) git clone bare_git/git_test.git git_test2\n(20) cd git_test\n(21) mkdir blah_test\n(22) git mv testing/ blah_test/\n(23) cg diff > ../mkdir_patch.diff\n(24) cd ..\n(25) cd git_test2/\n(26) cg patch < ../mkdir_patch.diff\n\nfrom the last one (26) i get \n\npatching file blah_test/testing/test1.txt\npatching file testing/test1.txt\ntouch: cannot touch `testing/test1.txt': No such file or directory\nAdding file blah_test/testing/test1.txt\nRemoving file testing/test1.txt\n\nbut the result is correct. There is no testing-directory in here\nanymore, and inside blah_test, there is my testing dir with the file\ntest1.txt in it...\n\ndid I miss something?\n\nI use cogito-0.17.3 with git version 1.4.1\n(obviously without that recently rewritten git-mv...)\n\nNicolas\n"},{"id":"24175","messageId":"9e4733910607261603m6772602cr333d8c58f555edaa@mail.gmail.com","threadId":"4994","inReplyTo":"20060726223459.GA30601@vsectoor.geht-ab-wie-schnitzel.de","subject":"Re: Moving a directory into another fails","fromName":"Jon Smirl","fromEmail":"jonsmirl@gmail.com","sentAt":"2006-07-26T23:03:30Z","receivedAt":"2006-07-26T23:03:30Z","isPatch":false,"sender":{"key":"jonsmirl@gmail.com","avatar":"https://gravatar.com/avatar/cff3bf5bfdfa6708b905712ff91f0f9b8aaca161659f38c02b787920d5d28b7e?d=mp&s=160"},"body":"This is a simpler sequence\n\ncg clone git foo\ncg clone git foo1\ncd foo\nmkdir zzz\ngit mv gitweb zzz\ncg diff >patch\ncg ../foo1\ncg patch <../foo/patch\n\nFails with these errors. We have determined that git apply patch is ok\nand this is a bug in cg patch.\n\n[jonsmirl@jonsmirl foo1]$ cg patch <../foo/patch\nmv: cannot move `gitweb/README' to `zzz/gitweb/README': No such file\nor directory\nmv: cannot move `gitweb/gitweb.cgi' to `zzz/gitweb/gitweb.cgi': No\nsuch file or directory\nmv: cannot move `gitweb/gitweb.css' to `zzz/gitweb/gitweb.css': No\nsuch file or directory\nmv: cannot stat `\"gitweb/test/M\\\\303\\\\244rchen\"': No such file or directory\nmv: cannot move `gitweb/test/file with spaces' to\n`zzz/gitweb/test/file with spaces': No such file or directory\nmv: cannot move `gitweb/test/file+plus+sign' to\n`zzz/gitweb/test/file+plus+sign': No such file or directory\npatch: **** Only garbage was found in the patch input.\nRemoving file gitweb/README\nAdding file zzz/gitweb/README\nerror: zzz/gitweb/README: does not exist and --remove not passed\nfatal: Unable to process file zzz/gitweb/README\ncg-add: warning: not all items could have been added\nRemoving file gitweb/gitweb.cgi\nAdding file zzz/gitweb/gitweb.cgi\nerror: zzz/gitweb/gitweb.cgi: does not exist and --remove not passed\nfatal: Unable to process file zzz/gitweb/gitweb.cgi\ncg-add: warning: not all items could have been added\nRemoving file gitweb/gitweb.css\nAdding file zzz/gitweb/gitweb.css\nerror: zzz/gitweb/gitweb.css: does not exist and --remove not passed\nfatal: Unable to process file zzz/gitweb/gitweb.css\ncg-add: warning: not all items could have been added\nRemoving file \"gitweb/test/Märchen\"\nAdding file \"zzz/gitweb/test/Märchen\"\nerror: \"zzz/gitweb/test/Märchen\": does not exist and --remove not passed\nfatal: Unable to process file \"zzz/gitweb/test/Märchen\"\ncg-add: warning: not all items could have been added\nRemoving file gitweb/test/file with spaces\nAdding file zzz/gitweb/test/file with spaces\nerror: zzz/gitweb/test/file with spaces: does not exist and --remove not passed\nfatal: Unable to process file zzz/gitweb/test/file with spaces\ncg-add: warning: not all items could have been added\nRemoving file gitweb/test/file+plus+sign\nAdding file zzz/gitweb/test/file+plus+sign\nerror: zzz/gitweb/test/file+plus+sign: does not exist and --remove not passed\nfatal: Unable to process file zzz/gitweb/test/file+plus+sign\ncg-add: warning: not all items could have been added\n[jonsmirl@jonsmirl foo1]$\n\n\n\n-- \nJon Smirl\njonsmirl@gmail.com\n"},{"id":"24176","messageId":"20060726232530.GC30601@vsectoor.geht-ab-wie-schnitzel.de","threadId":"4994","inReplyTo":"9e4733910607261603m6772602cr333d8c58f555edaa@mail.gmail.com","subject":"Re: Moving a directory into another fails","fromName":"Nicolas Vilz","fromEmail":"niv@iaglans.de","sentAt":"2006-07-26T23:25:30Z","receivedAt":"2006-07-26T23:25:30Z","isPatch":false,"sender":{"key":"niv@iaglans.de","avatar":"https://gravatar.com/avatar/e4d43a32d721241212d4edb1d2210327e28423c913071b4bfeeaa0ce15296110?d=mp&s=160"},"body":"On Wed, Jul 26, 2006 at 07:03:30PM -0400, Jon Smirl wrote:\n> This is a simpler sequence\n> \n> cg clone git foo\n> cg clone git foo1\n> cd foo\n> mkdir zzz\n> git mv gitweb zzz\n> cg diff >patch\n> cg ../foo1\n> cg patch <../foo/patch\n> \n> Fails with these errors. We have determined that git apply patch is ok\n> and this is a bug in cg patch.\n\nWell, perhaps i should react faster and I shouldn't pause my fetchmail\nfor 2 or 3 hours... this is bad for this list :) You are kind of fast :) \n\nNicolas\n"},{"id":"24275","messageId":"20060728014350.GI13776@pasky.or.cz","threadId":"4994","inReplyTo":"9e4733910607261603m6772602cr333d8c58f555edaa@mail.gmail.com","subject":"Re: Moving a directory into another fails","fromName":"Petr Baudis","fromEmail":"pasky@suse.cz","sentAt":"2006-07-28T01:43:50Z","receivedAt":"2006-07-28T01:43:50Z","isPatch":false,"sender":{"key":"pasky@ucw.cz","avatar":"https://avatars.githubusercontent.com/u/18439?v=4"},"body":"Dear diary, on Thu, Jul 27, 2006 at 01:03:30AM CEST, I got a letter\nwhere Jon Smirl <jonsmirl@gmail.com> said that...\n> This is a simpler sequence\n> \n> cg clone git foo\n> cg clone git foo1\n> cd foo\n> mkdir zzz\n> git mv gitweb zzz\n> cg diff >patch\n> cg ../foo1\n> cg patch <../foo/patch\n\nEven simpler one:\n\n\tmkdir zzz\n\tcg-mv gitweb zzz\n\tcg-diff | cg-patch -R\n\n(which would even undo the mess supposing that it worked properly)\n\n> [jonsmirl@jonsmirl foo1]$ cg patch <../foo/patch\n> mv: cannot move `gitweb/README' to `zzz/gitweb/README': No such file\n> or directory\n\nOops. Thanks, fixed with this:\n\ndiff --git a/cg-patch b/cg-patch\nindex cc82f1f..923df0e 100755\n--- a/cg-patch\n+++ b/cg-patch\n@@ -145,6 +145,8 @@ redzone_border()\n \t\t\techo \"$file1: rename destination $file2 already exists, NOT RENAMING\" >&2\n \t\t\treturn\n \t\tfi\n+\t\t# FIXME: Remove stale empty directories related to $mvfrom\n+\t\tcase $mvto in */*) mkdir -p \"${mvto%/*}\";; esac\n \t\tmv \"$mvfrom\" \"$mvto\"\n \tfi\n \tif [ \"$op\" = \"delete\" -o \"$op\" = \"rename\" ]; then\n\n> mv: cannot stat `\"gitweb/test/M\\\\303\\\\244rchen\"': No such file or directory\n\nJunio, how am I supposed to unmangle this *censored* stuff?\n\n-- \n\t\t\t\tPetr \"Pasky\" Baudis\nStuff: http://pasky.or.cz/\nSnow falling on Perl. White noise covering line noise.\nHides all the bugs too. -- J. Putnam\n"},{"id":"294428","messageId":"f3d7535d0612041019q4bda01a1k9938b056d51f8a78@mail.gmail.com","threadId":"4994","inReplyTo":"20060728014350.GI13776@pasky.or.cz","subject":"Re: Re: Moving a directory into another fails","fromName":"Stefan Pfetzing","fromEmail":"stefan.pfetzing@gmail.com","sentAt":"2006-12-04T18:19:58Z","receivedAt":"2006-12-04T18:19:58Z","isPatch":false,"sender":{"key":"stefan.pfetzing@gmail.com","avatar":null},"body":"Hi Folks,\n\n2006/7/28, Petr Baudis <pasky@suse.cz>:\n> > mv: cannot stat `\"gitweb/test/M\\\\303\\\\244rchen\"': No such file or directory\nsince when is this file in the official git.git tree?\n\nIts quite problematic when used on HFS+ because it uses UTF-16 internally IMHO.\n\nGit always thinks there is a new file in my git.git clone.\n\n--- snip ---\ndreamind@paris:~/src/git% git status\n# Untracked files:\n#   (use \"git add\" to add to commit)\n#\n#       gitweb/test/MaÌrchen\nnothing to commit\n--- snap ---\n\nbye\n\ndreamind\n\n-- \n       http://www.dreamind.de/\n"},{"id":"296876","messageId":"el1qtr$bca$1@sea.gmane.org","threadId":"4994","inReplyTo":"f3d7535d0612041019q4bda01a1k9938b056d51f8a78@mail.gmail.com","subject":"Re: Re: Moving a directory into another fails","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2006-12-04T18:56:57Z","receivedAt":"2006-12-04T18:56:57Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Stefan Pfetzing wrote:\n\n> 2006/7/28, Petr Baudis <pasky@suse.cz>:\n>>>\n>>> mv: cannot stat `\"gitweb/test/M\\\\303\\\\244rchen\"': No such file or directory\n>>>\n> since when is this file in the official git.git tree?\n\nSince merging in gitweb, Sat Jun 10, 2006.\n \n> Its quite problematic when used on HFS+ because it uses UTF-16 internally IMHO.\n> \n> Git always thinks there is a new file in my git.git clone.\n\nThat is the problem that git tries to be content agnostict, and it\nincludes being coding agnostic.\n\nI personally think that because the same repository might be deployed\non different systems with different file name encoding (and this is not\nsomething you have control over, contrary to commit/tag message encoding,\nand encoding in files), git should acquire core.filesystemEncoding\nconfiguration variable which would encode from filesystem encoding used\nin working directory and perhaps index to UTF-8 encoding used in repository\n(in tree objects) and perhaps index.\n-- \nJakub Narebski\nWarsaw, Poland\nShadeHawk on #git\n\n"},{"id":"294220","messageId":"Pine.LNX.4.63.0612042001320.28348@wbgn013.biozentrum.uni-wuerzburg.de","threadId":"4994","inReplyTo":"el1qtr$bca$1@sea.gmane.org","subject":"Re: Re: Moving a directory into another fails","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2006-12-04T19:03:46Z","receivedAt":"2006-12-04T19:03:46Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Mon, 4 Dec 2006, Jakub Narebski wrote:\n\n> [...] git should acquire core.filesystemEncoding configuration variable \n> which would encode from filesystem encoding used in working directory \n> and perhaps index to UTF-8 encoding used in repository (in tree objects) \n> and perhaps index.\n\nSo, you want to pull in all thinkable encodings? Of course, you could rely \non libiconv, adding yet another dependency to git. (Yes, I know, mailinfo \nuses it already. But I never use mailinfo, so I do not need libiconv.)\n\nCiao,\nDscho\n"},{"id":"295607","messageId":"el1rmm$bca$2@sea.gmane.org","threadId":"4994","inReplyTo":"Pine.LNX.4.63.0612042001320.28348@wbgn013.biozentrum.uni-wuerzburg.de","subject":"Re: Re: Moving a directory into another fails","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2006-12-04T19:10:12Z","receivedAt":"2006-12-04T19:10:12Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Johannes Schindelin wrote:\n\n> On Mon, 4 Dec 2006, Jakub Narebski wrote:\n> \n>> [...] git should acquire core.filesystemEncoding configuration variable \n>> which would encode from filesystem encoding used in working directory \n>> and perhaps index to UTF-8 encoding used in repository (in tree objects) \n>> and perhaps index.\n> \n> So, you want to pull in all thinkable encodings? Of course, you could rely \n> on libiconv, adding yet another dependency to git. (Yes, I know, mailinfo \n> uses it already. But I never use mailinfo, so I do not need libiconv.)\n\nA conditional dependency. If you don't have libiconv, this feature wouldn't\nbe available.\n\n-- \nJakub Narebski\nWarsaw, Poland\nShadeHawk on #git\n\n"},{"id":"294615","messageId":"Pine.LNX.4.63.0612042009590.28348@wbgn013.biozentrum.uni-wuerzburg.de","threadId":"4994","inReplyTo":"el1rmm$bca$2@sea.gmane.org","subject":"Re: Re: Moving a directory into another fails","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2006-12-04T19:10:30Z","receivedAt":"2006-12-04T19:10:30Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Mon, 4 Dec 2006, Jakub Narebski wrote:\n\n> Johannes Schindelin wrote:\n> \n> > On Mon, 4 Dec 2006, Jakub Narebski wrote:\n> > \n> >> [...] git should acquire core.filesystemEncoding configuration variable \n> >> which would encode from filesystem encoding used in working directory \n> >> and perhaps index to UTF-8 encoding used in repository (in tree objects) \n> >> and perhaps index.\n> > \n> > So, you want to pull in all thinkable encodings? Of course, you could rely \n> > on libiconv, adding yet another dependency to git. (Yes, I know, mailinfo \n> > uses it already. But I never use mailinfo, so I do not need libiconv.)\n> \n> A conditional dependency. If you don't have libiconv, this feature wouldn't\n> be available.\n\nYou are speaking as somebody compiling git from source. We are a minority.\n\nCiao,\nDscho\n"},{"id":"297387","messageId":"el1t9a$lmo$1@sea.gmane.org","threadId":"4994","inReplyTo":"Pine.LNX.4.63.0612042009590.28348@wbgn013.biozentrum.uni-wuerzburg.de","subject":"Re: Re: Moving a directory into another fails","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2006-12-04T19:37:12Z","receivedAt":"2006-12-04T19:37:12Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Johannes Schindelin wrote:\n\n> On Mon, 4 Dec 2006, Jakub Narebski wrote:\n> \n>> Johannes Schindelin wrote:\n>> \n>>> On Mon, 4 Dec 2006, Jakub Narebski wrote:\n>>> \n>>>> [...] git should acquire core.filesystemEncoding configuration variable \n>>>> which would encode from filesystem encoding used in working directory \n>>>> and perhaps index to UTF-8 encoding used in repository (in tree objects) \n>>>> and perhaps index.\n>>> \n>>> So, you want to pull in all thinkable encodings? Of course, you could rely \n>>> on libiconv, adding yet another dependency to git. (Yes, I know, mailinfo \n>>> uses it already. But I never use mailinfo, so I do not need libiconv.)\n>> \n>> A conditional dependency. If you don't have libiconv, this feature wouldn't\n>> be available.\n> \n> You are speaking as somebody compiling git from source. We are a minority.\n\nUsually iconv is in libc.\n\n# Define NEEDS_LIBICONV if linking with libc is not enough (Darwin).\n\nHmm... perhaps not that usually. The uname based configuration in Makefile\n(not the test based configuration provided by autoconf generated\n./configure script) sets NEEDS_LIBICONV for: Darwin, SunOS 5.8, Cygwin,\nFreeBSD and OpenBSD, some versions of NetBSD, AIX.\n\nAnd HFS+ is on MacOS X / Darwin, without iconv in libc...\n-- \nJakub Narebski\nWarsaw, Poland\nShadeHawk on #git\n\n"},{"id":"296635","messageId":"Pine.LNX.4.64.0612041114240.3476@woody.osdl.org","threadId":"4994","inReplyTo":"Pine.LNX.4.63.0612042009590.28348@wbgn013.biozentrum.uni-wuerzburg.de","subject":"Re: Re: Moving a directory into another fails","fromName":"Linus Torvalds","fromEmail":"torvalds@osdl.org","sentAt":"2006-12-04T20:26:42Z","receivedAt":"2006-12-04T20:26:42Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Mon, 4 Dec 2006, Johannes Schindelin wrote:\n> \n> On Mon, 4 Dec 2006, Jakub Narebski wrote:\n> \n> > Johannes Schindelin wrote:\n> > \n> > > On Mon, 4 Dec 2006, Jakub Narebski wrote:\n> > > \n> > >> [...] git should acquire core.filesystemEncoding configuration variable \n> > >> which would encode from filesystem encoding used in working directory \n> > >> and perhaps index to UTF-8 encoding used in repository (in tree objects) \n> > >> and perhaps index.\n> > > \n> > > So, you want to pull in all thinkable encodings? Of course, you could rely \n> > > on libiconv, adding yet another dependency to git. (Yes, I know, mailinfo \n> > > uses it already. But I never use mailinfo, so I do not need libiconv.)\n> > \n> > A conditional dependency. If you don't have libiconv, this feature wouldn't\n> > be available.\n> \n> You are speaking as somebody compiling git from source. We are a minority.\n\nYou guys are ignoring the _real_ problem. \n\nIt has nothing at all to do with dependencies on external packages. The \nREAL problem is that if you do locale-dependent trees and other git \nobjects, git will STOP WORKING.\n\nA filename in a tree object _has_ to be see as a pure 8-bit character \nstream. They _have_ to be compared with \"memcmp()\", and they have to sort \nthe same way and mean EXACTLY the same thing for everybody.\n\nIf a filesystem cannot represent that name AS THAT BYTE SEQUENCE then the \nfilesystem is broken. No ifs, buts, maybes about it. I'm sorry, but that's \nhow it is.\n\nThis is _exactly_ the same issue as case independence. Git does not ignore \ncase, and it really CANNOT ignore case. Ignoring case would cause horrible \nand deep problems, and it has nothing to do with dependencies on libraries \n(although it _would_ get much much worse from locale settings, and again \nhaving different locales compare the same name differently because case \nrules are different).\n\nSo it really boils down to one one: git saves a byte stream. Not text. \n\nThis is true for all levels of the git archive. It's true for blob \ncontent, it's true for filenames in trees, and it is true for commits. The \ncommit message is actually somewhat easier (because we have nothing to \n\"compare\" it to afterwards in the checked-out tree), so the commit message \nis the _one_ thing we can kind of play games with, but even there, once \nit's done, it's done, and it's just a stream of bytes.\n\n"},{"id":"297182","messageId":"Pine.LNX.4.64.0612041246150.3476@woody.osdl.org","threadId":"4994","inReplyTo":"Pine.LNX.4.64.0612041114240.3476@woody.osdl.org","subject":"Re: Re: Moving a directory into another fails","fromName":"Linus Torvalds","fromEmail":"torvalds@osdl.org","sentAt":"2006-12-04T20:51:08Z","receivedAt":"2006-12-04T20:51:08Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Mon, 4 Dec 2006, Linus Torvalds wrote:\n> \n> If a filesystem cannot represent that name AS THAT BYTE SEQUENCE then the \n> filesystem is broken. No ifs, buts, maybes about it. I'm sorry, but that's \n> how it is.\n\nBtw, what this means in practice is that when git creates a file with a \ncertain sequence of bytes, then \n (a) readdir had better return _that_ sequence of bytes, or git will see \n     it as somethign else.\n (b) opening it with that same sequence of bytes had better work.\n\nThis does not mean that a filesystem may not internally use some other \nencoding. It just means that if the filesystem - when converting back and \nforth between the internal encoding and the one it shows to user space - \nhad better convert back to the exact same thing.\n\nAlso, note that for most projects, even a broken filesystem doesn't \nactually matter - it's enough that the filesystem gets the conversions \nright for the particular set of names in a particular project. So any \nproject that just has 7-bit filenames will obviously never even see any \nissues at all, even if the filesystem it runs on then does something \nstrange with 8-bit filenames.\n\nThis is one reason why UNIX's \"everything is a stream of bytes\" is so \nimportant, and whyprograms should generally work with byte streams, not \n\"wide strings\" or similar. It's the only way that you can reliably work \nacross different locales. Use wide strings and locale-specific stuff \n_only_ for actually showing users something on the tty, for example.\n\n"},{"id":"294944","messageId":"20061204205407.GB6764@spearce.org","threadId":"4994","inReplyTo":"Pine.LNX.4.64.0612041114240.3476@woody.osdl.org","subject":"Re: Re: Moving a directory into another fails","fromName":"Shawn Pearce","fromEmail":"spearce@spearce.org","sentAt":"2006-12-04T20:54:07Z","receivedAt":"2006-12-04T20:54:07Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Linus Torvalds <torvalds@osdl.org> wrote:\n> You guys are ignoring the _real_ problem. \n> \n> It has nothing at all to do with dependencies on external packages. The \n> REAL problem is that if you do locale-dependent trees and other git \n> objects, git will STOP WORKING.\n\nYes!\n\nIn jgit I assumed all tree entry names were encoded in UTF8.\nThen I later learned they aren't.  Foolish me.\n\nAs Linus points out its a HUGE problem that the caller of\ngit-write-tree gets to decide what encoding should be used for\nthat tree.  Especially if someone else wants to use a different\nencoding for the same filename (think ISO-8859-1 vs. UTF-8)!\n\nI'd rather just force the tree entry names to be encoded in UTF-8\nalways, as its compact for most western texts (which many filenames\nare), and at least degrades to supporting the non western texts.\n\nA per-project setting is essentially impossible as we have\nno such concept today, and a per-repository setting (like\ni18n.commitEncoding) lets two different users encode the same\nfilename differently, which means two different tree SHA1s with\nthe exact same content... not correct!\n \n> This is true for all levels of the git archive. It's true for blob \n> content, it's true for filenames in trees, and it is true for commits. The \n> commit message is actually somewhat easier (because we have nothing to \n> \"compare\" it to afterwards in the checked-out tree), so the commit message \n> is the _one_ thing we can kind of play games with, but even there, once \n> it's done, it's done, and it's just a stream of bytes.\n\nCommit encoding is a problem.  Clearly the \"header parts\"\n(tree, parent) are US-ASCII but the author and committer lines\ncan be anything.  So can the body.  And we have no way of knowing\nwhat encoding was used years later, we can only guess and display\nit wrong.\n\nWe really should either normalize all commit messages to a single\nencoding (again, UTF-8) or embed the encoding as part of the headers\nsomehow (e.g. look at how XML embeds the document encoding in the\nstart of the document).\n\n-- \n"},{"id":"297638","messageId":"200612042156.44963.jnareb@gmail.com","threadId":"4994","inReplyTo":"Pine.LNX.4.64.0612041114240.3476@woody.osdl.org","subject":"Re: Moving a directory into another fails","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2006-12-04T20:56:44Z","receivedAt":"2006-12-04T20:56:44Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Dnia poniedziałek 4. grudnia 2006 21:26, Linus Torvalds napisał:\n \n>>>> On Mon, 4 Dec 2006, Jakub Narebski wrote:\n>>>> \n>>>>> [...] git should acquire core.filesystemEncoding configuration variable \n>>>>> which would encode from filesystem encoding used in working directory \n>>>>> and perhaps index to UTF-8 encoding used in repository (in tree objects) \n>>>>> and perhaps index.\n\n> You guys are ignoring the _real_ problem. \n> \n> It has nothing at all to do with dependencies on external packages. The \n> REAL problem is that if you do locale-dependent trees and other git \n> objects, git will STOP WORKING.\n> \n> A filename in a tree object _has_ to be see as a pure 8-bit character \n> stream. They _have_ to be compared with \"memcmp()\", and they have to sort \n> the same way and mean EXACTLY the same thing for everybody.\n\nWhat I propose is having filename in tree object UTF-8 encoded. I don't\nknow if git relies heavily that filename encoding on filesystem (in working\narea) is the same as in the index, is the same as in a tree object.\n \nAlthough I'm not sure what is the problem. You checkout non US-ASCII filename\nout of git; the file can have strange characters in a name, but should\nencode to the filename as is in git. The problem migh be some forbidden by\nfilesystem characters in a filename perhaps. \n\nAlthough Wolfgang Fischer wrote (to me and Johannes Schindelin) that HFS+\nuses UTF8-NFC (Normalization-Form-Composed) when creating a file, while\nreaddir returns encoding used by HFS+, which is UTF8-NFD (Normalization-Form-\nDecomposed). [Explitive censored]\n\n> If a filesystem cannot represent that name AS THAT BYTE SEQUENCE then the \n> filesystem is broken. No ifs, buts, maybes about it. I'm sorry, but that's \n> how it is.\n\nWe have some configuration variables to work around broken filesystems,\nlike core.ignoreStat, so why not core.filesystemEncoding.\n\n-- \nJakub Narebski\n"},{"id":"295393","messageId":"Pine.LNX.4.63.0612042155480.28348@wbgn013.biozentrum.uni-wuerzburg.de","threadId":"4994","inReplyTo":"7617FA7E-D49A-4A4C-B033-C2CB20623F5F@wf227.com","subject":"Re: Moving a directory into another fails","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2006-12-04T21:01:34Z","receivedAt":"2006-12-04T21:01:34Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nthank you, Wolfgang, for back Cc'ing me (Jakub never does that...), but I \nam Cc'ing the git list here, also. Hope both of you don't mind.\n\nOn Mon, 4 Dec 2006, Wolfgang Fischer wrote:\n\n> On 04.12.2006, at 20:37, Jakub Narebski wrote:\n> \n> > And HFS+ is on MacOS X / Darwin, without iconv in libc...\n> \n> And, what is even worse, is the fact that HFS+ uses an encoding, which is not\n> represented in libiconv.\n> \n> If you CREATE a file, you can use UTF8-NFC \n> (Normalization-Form-Composed), but if you later READDIR a directory, you \n> will get the very same name back in the encoding used by HFS+, which is \n> UTF8-NFD Normalization-Form-Decomposed. The difference is noticeable for \n> some non-ASCII characters like e.g.\n> \n> LATIN SMALL LETTER A WITH DIAERESIS  U+00E4 or U+0061 U+0308 in Unicode.\n> \n> If you need a sane backward mapping, one has to use some CoreFoundation \n> interface, for which I removed the details out of my brain, in order to \n> reclaim that memory area (garbage collection!). But I can help you with \n> some details and probably code, if you really need that conversion \n> direction.\n\nYes. When my iBook was still alive, I saw that problem, too: writing and \nreading filenames were completely different issues.\n\nWorse, you can experience the same on USB-Sticks when accessing them with \ndifferent OSes. For example, when checking out a git repo on a stick with \nLinux, and then calling git-status on the same stick with Windows XP, you \nsee an issue with the file \"Märchen\", like you did on MacOSX.\n\nSo, please, please, please do not try to be smart about filename encodings \nin git, but just DO NOT USE ANYTHING BUT ASCII IN FILENAMES IF THE \nREPOSITORY IS GOING TO BE PUT ON DIFFERENT OPERATING SYSTEMS/FILE SYSTEMS. \n(Wow, the Caps Lock key is _not_ dead after all. I must have been infected \nby Linus...)\n\nCiao,\nDscho\n"},{"id":"296579","messageId":"Pine.LNX.4.63.0612042201420.28348@wbgn013.biozentrum.uni-wuerzburg.de","threadId":"4994","inReplyTo":"Pine.LNX.4.64.0612041114240.3476@woody.osdl.org","subject":"Re: Re: Moving a directory into another fails","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2006-12-04T21:05:12Z","receivedAt":"2006-12-04T21:05:12Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Mon, 4 Dec 2006, Linus Torvalds wrote:\n\n> On Mon, 4 Dec 2006, Johannes Schindelin wrote:\n> > \n> > On Mon, 4 Dec 2006, Jakub Narebski wrote:\n> > \n> > > Johannes Schindelin wrote:\n> > > \n> > > > On Mon, 4 Dec 2006, Jakub Narebski wrote:\n> > > > \n> > > >> [...] git should acquire core.filesystemEncoding configuration variable \n> > > >> which would encode from filesystem encoding used in working directory \n> > > >> and perhaps index to UTF-8 encoding used in repository (in tree objects) \n> > > >> and perhaps index.\n> > > > \n> > > > So, you want to pull in all thinkable encodings? Of course, you could rely \n> > > > on libiconv, adding yet another dependency to git. (Yes, I know, mailinfo \n> > > > uses it already. But I never use mailinfo, so I do not need libiconv.)\n> > > \n> > > A conditional dependency. If you don't have libiconv, this feature wouldn't\n> > > be available.\n> > \n> > You are speaking as somebody compiling git from source. We are a minority.\n> \n> You guys are ignoring the _real_ problem. \n> \n> It has nothing at all to do with dependencies on external packages. The \n> REAL problem is that if you do locale-dependent trees and other git \n> objects, git will STOP WORKING.\n\nThe issue was _not_ locale-dependent trees, but file systems which \n_change_ the encoding. And even then, Jakub's proposition reencoding could \nwork, because it is an _encoding_ after all, i.e. bijective (reversable \nmapping for you non-Math guys). Not at all comparable to cases \ninsensitivity, which _loses_ information.\n\nBut for reasons described in another mail, there are more fundamental \nproblems with encodings, especially with MacOSX which (braindeadly) \nencodes _differently_ when writing and reading.\n\nSo, we reach the same conclusion, but for different reasons.\n\nCiao,\n"},{"id":"298008","messageId":"Pine.LNX.4.64.0612041317480.3476@woody.osdl.org","threadId":"4994","inReplyTo":"Pine.LNX.4.63.0612042201420.28348@wbgn013.biozentrum.uni-wuerzburg.de","subject":"Re: Re: Moving a directory into another fails","fromName":"Linus Torvalds","fromEmail":"torvalds@osdl.org","sentAt":"2006-12-04T21:23:25Z","receivedAt":"2006-12-04T21:23:25Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Mon, 4 Dec 2006, Johannes Schindelin wrote:\n> \n> The issue was _not_ locale-dependent trees, but file systems which \n> _change_ the encoding.\n\nCorrect. However, it doesn't really change the issue: some byte streams \nmay simply not work in certain encodings.\n\nYou could, of course, basically do some kind of \"escape high characters\" \non the filename if it has characters in it that you suspect might cause \nproblems, but you'd better make 100% sure that it really is 100% \nreversible (and you need to do all the real operations on the _native_git_ \nversion of the filename).\n\nSo we _could_ use a flag that says \"escape all filenames\", but it would \nnot be a _locale_ setting, it would really be a per-repository setting, \nand it wouldn't be \"iconv\", it would be something similar to what we do \nfor \"git diff\" when we escape filenames with strange characters in them.\n\nWe could do it by changing ever \"open()/creat()\" and \"[l]stat()\" on the \nworking tree with somethign that first escapes the filename.\n\nThen, people with broken filesystems could set\n\n\t[core]\n\t\tescapefilenames = true\n\nand instead of seeing 8-bit filenames, they'd see filenames with 7 bits \nand escapes. They could work with such a repo, for sure. It would be ugly \nas hell, though.\n\n"},{"id":"298391","messageId":"Pine.LNX.4.63.0612050831550.28348@wbgn013.biozentrum.uni-wuerzburg.de","threadId":"4994","inReplyTo":"Pine.LNX.4.64.0612041317480.3476@woody.osdl.org","subject":"Re: Re: Moving a directory into another fails","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2006-12-05T07:34:33Z","receivedAt":"2006-12-05T07:34:33Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Mon, 4 Dec 2006, Linus Torvalds wrote:\n\n> \t[core]\n> \t\tescapefilenames = true\n\nI think this goes too far. The problem _only_ showed up with a made-up \ntest case for gitweb. Let's bite the apple when we _have_ to (which I \ndoubt will happen, because for the most part, developers understand that \nspaces and umlauts have _no_ place in filenames, basically since UNIX was \ninvented by stupid US Americans who did not know anything about nice \nfilenames, let alone other languages than English and C).\n\nCiao,\nDscho\n"},{"id":"298324","messageId":"200612051036.13645.jnareb@gmail.com","threadId":"4994","inReplyTo":"Pine.LNX.4.63.0612050831550.28348@wbgn013.biozentrum.uni-wuerzburg.de","subject":"Re: Moving a directory into another fails","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2006-12-05T09:36:13Z","receivedAt":"2006-12-05T09:36:13Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Johannes Schindelin wrote:\n\n> On Mon, 4 Dec 2006, Linus Torvalds wrote:\n> \n>> \t[core]\n>> \t\tescapefilenames = true\n> \n> I think this goes too far. The problem _only_ showed up with a made-up \n> test case for gitweb. Let's bite the apple when we _have_ to (which I \n> doubt will happen, because for the most part, developers understand that \n> spaces and umlauts have _no_ place in filenames, basically since UNIX was \n> invented by stupid US Americans who did not know anything about nice \n> filenames, let alone other languages than English and C).\n\nNo, the problem showed with stupid HFS+ which uses different encoding\nfor creating file, and different for readdir.\n\nPerhaps we should remove gitweb/test directory, and move testing gitweb\nto proper place, t/ directory. \n\nBy the way, would it be correct to use external tools (if they exist),\nnamely HTMLtidy in gitweb output test to-be-written?\n-- \nJakub Narebski\n"},{"id":"297701","messageId":"Pine.LNX.4.63.0612051507170.28348@wbgn013.biozentrum.uni-wuerzburg.de","threadId":"4994","inReplyTo":"200612051036.13645.jnareb@gmail.com","subject":"filesystem encodings and gitweb tests, was Re: Moving a directory into another fails","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2006-12-05T14:11:32Z","receivedAt":"2006-12-05T14:11:32Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Tue, 5 Dec 2006, Jakub Narebski wrote:\n\n> Johannes Schindelin wrote:\n> \n> > On Mon, 4 Dec 2006, Linus Torvalds wrote:\n> > \n> >> \t[core]\n> >> \t\tescapefilenames = true\n> > \n> > I think this goes too far. The problem _only_ showed up with a made-up \n> > test case for gitweb. Let's bite the apple when we _have_ to (which I \n> > doubt will happen, because for the most part, developers understand that \n> > spaces and umlauts have _no_ place in filenames, basically since UNIX was \n> > invented by stupid US Americans who did not know anything about nice \n> > filenames, let alone other languages than English and C).\n> \n> No, the problem showed with stupid HFS+ which uses different encoding\n> for creating file, and different for readdir.\n\nThis is just one of the problems. I described another problem in this \nthread, namely a repo on a usb stick being accessed from different hosts.\n\n> Perhaps we should remove gitweb/test directory, and move testing gitweb \n> to proper place, t/ directory.\n\nIf you do that, please make sure that these tests can be disabled (a la \nsvn tests), so that people not being interested in gitweb, or lacking the \nprograms to test it, do not have to suffer.\n\n> By the way, would it be correct to use external tools (if they exist),\n> namely HTMLtidy in gitweb output test to-be-written?\n\n(Yes, they exist. HTMLtidy for example ;-)\n\nIMHO if such a tool is common enough, you should use it. If anybody steps \nforward providing automated HTTP tests, I will not complain, and certainly \nnot about testing with things like HTMLtidy.\n\nCiao,\nDscho\n"},{"id":"296873","messageId":"200612051529.57174.jnareb@gmail.com","threadId":"4994","inReplyTo":"Pine.LNX.4.63.0612051507170.28348@wbgn013.biozentrum.uni-wuerzburg.de","subject":"Re: filesystem encodings and gitweb tests, was Re: Moving a directory into another fails","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2006-12-05T14:29:56Z","receivedAt":"2006-12-05T14:29:56Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Johannes Schindelin wrote:\n\n> On Tue, 5 Dec 2006, Jakub Narebski wrote:\n> \n>> No, the problem showed with stupid HFS+ which uses different encoding\n>> for creating file, and different for readdir.\n> \n> This is just one of the problems. I described another problem in this \n> thread, namely a repo on a usb stick being accessed from different hosts.\n\nThat is not much a problem. Yes, the filenames on different hosts would\n_look_ different, but shouldn't be detected as new file.\n-- \nJakub Narebski\n"},{"id":"295273","messageId":"Pine.LNX.4.63.0612051539030.28348@wbgn013.biozentrum.uni-wuerzburg.de","threadId":"4994","inReplyTo":"200612051529.57174.jnareb@gmail.com","subject":"Re: filesystem encodings and gitweb tests, was Re: Moving a directory into another fails","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2006-12-05T14:40:04Z","receivedAt":"2006-12-05T14:40:04Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Tue, 5 Dec 2006, Jakub Narebski wrote:\n\n> Johannes Schindelin wrote:\n> \n> > On Tue, 5 Dec 2006, Jakub Narebski wrote:\n> > \n> >> No, the problem showed with stupid HFS+ which uses different encoding\n> >> for creating file, and different for readdir.\n> > \n> > This is just one of the problems. I described another problem in this \n> > thread, namely a repo on a usb stick being accessed from different hosts.\n> \n> That is not much a problem. Yes, the filenames on different hosts would\n> _look_ different, but shouldn't be detected as new file.\n\nSorry, I should have been clearer. I meant a repo _and_ a working \ndirectory going along with it, on a USB stick. They do look different on \ndifferent hosts, and git-status looks different as a consequence ;-)\n\nCiao,\nDscho\n"},{"id":"294183","messageId":"Pine.LNX.4.64.0612050908520.3542@woody.osdl.org","threadId":"4994","inReplyTo":"Pine.LNX.4.63.0612050831550.28348@wbgn013.biozentrum.uni-wuerzburg.de","subject":"Re: Re: Moving a directory into another fails","fromName":"Linus Torvalds","fromEmail":"torvalds@osdl.org","sentAt":"2006-12-05T17:11:35Z","receivedAt":"2006-12-05T17:11:35Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Tue, 5 Dec 2006, Johannes Schindelin wrote:\n> \n> On Mon, 4 Dec 2006, Linus Torvalds wrote:\n> \n> > \t[core]\n> > \t\tescapefilenames = true\n> \n> I think this goes too far.\n\nSure., I agree that in _practice_ this isn't actually a problem, because \npeople have long since learnt to avoid strange filenames in SCM's, simply \nbecause you can't get it right with insane filesystems.\n\nThat said, it might be a good idea to abstract out the create/read phase \nfor filenames in the working tree regardless, since that also tends to be \nan area where other issues can come up (whoops - '/' vs '\\' as the \ndirectory separator etc).\n\n"}]}