{"thread":{"id":"11703","subject":"[RFC] File system difference handling in git","startedAt":"2008-01-22T09:21:02Z","lastAt":"2008-01-22T21:32:14Z","messageCount":8,"participants":["Reece Dunn","Junio C Hamano","Linus Torvalds","Steffen Prohaska","David Kastrup","Jonathan del Strother"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"66295","messageId":"3f4fd2640801220121w60cb9a69u8519a7ceb81d3414@mail.gmail.com","threadId":"11703","inReplyTo":null,"subject":"[RFC] File system difference handling in git","fromName":"Reece Dunn","fromEmail":"msclrhd@googlemail.com","sentAt":"2008-01-22T09:21:02Z","receivedAt":"2008-01-22T09:21:02Z","isPatch":false,"sender":{"key":"msclrhd@googlemail.com","avatar":null},"body":"Hi,\n\nObserving the various comments w.r.t. the different (potentially\nbraindead) filesystems that are available, there are two general\ncategories for behavioural differences:\n\n 1. File name representation\n\nFor Linux file systems (correct me if I am wrong here), they all store\nthe file name as-is. The question here is what happens on\nWindows-based file systems (e.g. NTFS) that are being read on Linux?\n\nFor Mac filesystems, you have the Unicode character decomposition\nissues to deal with.\n\nFor Windows, you have UTF-16 filename support.\n\nThere are two basic usages for file/directory names: passing the name\nto the Operating System; getting the name from the Operating System.\nTherefore, you have:\n\n os_to_git_path( const NATIVECHAR * ospath, strbuf * gitpath );\n git_to_os_path( const char * gitpath, const NATIVECHAR * ospath, int oslen );\n\nThese can then be used to handle Operating System differences (e.g.\nuse WideCharToMultiByte/MultiByteToWideChar conversion on Windows to\nmap between UTF-8 and UCS-2/UTF-16).\n\nIf Mac has an API to handle its strange behaviour, that can be used\nhere as well.\n\n 2. Case (in)sensitivity\n\nHere, you have the following cases:\n\n 1. git and the filesystem say that the files are different.\n\nUpdate the git directory tree and move the file on the filesystem.\n\n 2. git and the filesystem say that the files are the same.\n\nGenerate an error as is currently done in git.\n\n 3. git says that the files are different, but the filesystem says\nthat the files are the same.\n\nAllow the move, updating the git directory tree only.\n\n- Reece\n"},{"id":"66298","messageId":"7vtzl6qhtk.fsf@gitster.siamese.dyndns.org","threadId":"11703","inReplyTo":"3f4fd2640801220121w60cb9a69u8519a7ceb81d3414@mail.gmail.com","subject":"Re: [RFC] File system difference handling in git","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2008-01-22T10:24:07Z","receivedAt":"2008-01-22T10:24:07Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Reece Dunn\" writes:\n\n> 1. File name representation\n>\n> For Linux file systems ...\n> Therefore, you have:\n>\n> os_to_git_path( const NATIVECHAR * ospath, strbuf * gitpath );\n> git_to_os_path( const char * gitpath, const NATIVECHAR * ospath, int oslen );\n\nIt is not that simple, I am afraid. Legacy encodings can be\nused in pathnames. With bog-standard traditional UNIX pathname\nsemantics, all pathnames are sequences of non-NUL, non-slash\nbytes, separated with slashes, so if you do not allow choices\n(which is a very sensible ideal world scenario), you can declare\nthat the \"git\" encoding is UTF-8 and always check things out\nas-is.\n\nBut if you want a project (\"git\" in your above parlance) to be\nchecked out in two repositories, one with legacy and the other\nwith UTF-8, you cannot just say os_to_git/git_to_os. You would\nneed a bit more information from the repository owners what\nencodings are suitable. So your os_to_git()/git_to_os() will\nnot be an identity function even on Linux to support such.\n\nI used to have a data directory on my Linux box with EUC-JP\npathname and exported as an SMB share to my wife's Windows box,\ntelling samba to transliterate to whatever encoding the other\nend liked. I did not want to have the pathname on the Linux end\nin UTF-8 because I did not have enough energey to update my\nEmacs configuration to grok Japanese in UTF-8 (even though I\nfinally bit the bullet and switched to UTF-8 on the Linux side\nrecently).\n\nI know, this is painful. Real life hurts. Even on Linux,\nnot everybody can live in UTF-8-only world.\n\n> 2. Case (in)sensitivity\n>\n> Here, you have the following cases:\n> ...\n> 3. git says that the files are different, but the filesystem says\n> that the files are the same.\n>\n> Allow the move, updating the git directory tree only.\n\nSorry, I cannot really tell what you are talking about. You\nseem to imply, with \"Allow the move\", that you are describing a\nscenario that involves a move of one existing file to another,\nbut it is not clear. E.g. did you mean, by 3, \"When the user\nsays 'move a b', and if git says a and b are different but if\nthe filesystem says a and b are the same, then...\"?\n"},{"id":"66302","messageId":"3f4fd2640801220252q6a6557cdma4ca0a2c41223593@mail.gmail.com","threadId":"11703","inReplyTo":"7vtzl6qhtk.fsf@gitster.siamese.dyndns.org","subject":"Re: [RFC] File system difference handling in git","fromName":"Reece Dunn","fromEmail":"msclrhd@googlemail.com","sentAt":"2008-01-22T10:52:30Z","receivedAt":"2008-01-22T10:52:30Z","isPatch":false,"sender":{"key":"msclrhd@googlemail.com","avatar":null},"body":"On 22/01/2008, Junio C Hamano wrote:\n> \"Reece Dunn\" writes:\n>\n> > 1. File name representation\n> >\n> > For Linux file systems ...\n> > Therefore, you have:\n> >\n> > os_to_git_path( const NATIVECHAR * ospath, strbuf * gitpath );\n> > git_to_os_path( const char * gitpath, const NATIVECHAR * ospath, int oslen );\n>\n> It is not that simple, I am afraid. Legacy encodings can be\n> used in pathnames. With bog-standard traditional UNIX pathname\n> semantics, all pathnames are sequences of non-NUL, non-slash\n> bytes, separated with slashes, so if you do not allow choices\n> (which is a very sensible ideal world scenario), you can declare\n> that the \"git\" encoding is UTF-8 and always check things out\n> as-is.\n\nSo the upshot of this is that you need to use a platform (Operating\nSystem, filesystem, locale, etc.) that match what the git repository\nwas created in, otherwise there are going to be issues when\ninterpreting paths correctly.\n\nThe locale issue asside, can the above proposal help users working on\nMac, Linux and Windows interoperate with each other?\n\nI understand that there is not going to be a universal magic fix; what\nI'm interested in is minimising the differences between Operating\nSystems. This may be a futile effort, as it is likely you will need\nsome knowledge of the properties of the filesystem being used (as\nfilesystems with different properties can be used on the same\nOperating System).\n\n> > 2. Case (in)sensitivity\n> >\n> > Here, you have the following cases:\n> > ...\n> > 3. git says that the files are different, but the filesystem says\n> > that the files are the same.\n> >\n> > Allow the move, updating the git directory tree only.\n>\n> Sorry, I cannot really tell what you are talking about. You\n> seem to imply, with \"Allow the move\", that you are describing a\n> scenario that involves a move of one existing file to another,\n> but it is not clear. E.g. did you mean, by 3, \"When the user\n> says 'move a b', and if git says a and b are different but if\n> the filesystem says a and b are the same, then...\"?\n\nThis is what I am saying. For example, if you say:\n\n git mv myfile.H myfile.h\n\non a case sensitive filesystem (e.g. ext3), this will work, however on\na case insensitive filesystem (e.g. ntfs) git would complain that the\nfiles are the same.\n\nThe workaround is to say:\n\n git mv myfile.H myfile.h.tmp\n git mv myfile.h.tmp myfile.h\n\nbut this is not ideal, especially if you are automating some move operations.\n\nThis also applies to the VCS importers (e.g. git-p4) that can delete a\nfile that is a case-only move on case insensitive filesystems.\n\nThe question then becomes what happens on Mac (with the Unicode\ndecomposing behaviour) if they differ in the way they are stored (e.g.\nin Linus' 'ä' vs 'a¨' example)?\n\n- Reece\n"},{"id":"66319","messageId":"alpine.LFD.1.00.0801220829080.2957@woody.linux-foundation.org","threadId":"11703","inReplyTo":"3f4fd2640801220121w60cb9a69u8519a7ceb81d3414@mail.gmail.com","subject":"Re: [RFC] File system difference handling in git","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2008-01-22T16:56:28Z","receivedAt":"2008-01-22T16:56:28Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Tue, 22 Jan 2008, Reece Dunn wrote:\n> \n> 1. File name representation\n> \n> For Linux file systems (correct me if I am wrong here), they all store\n> the file name as-is. The question here is what happens on\n> Windows-based file systems (e.g. NTFS) that are being read on Linux?\n\nGenerally, Linux tries to follow the conventions of the filesystem, so \nit's generally case-preserving and case-sensitive (but not normalizing in \nany way - the case sensitivity is literally a upcase lookup table, so you \ndo \"upcase(c1) == upcase(c2)\" for each UCS-2 character, no combining or \ndecomposition).\n\nBut the fs volume can specify if it's a case-sensitive volume or not. And \nthe volume will also actually contain the \"upcase[]\" array that defines \nthe case-sensitivity, so exactly *which* characters are equivalent isn't \nactually defined by any external entity, it's defined by the particular \nfilesystem instance itself!\n\nThere's a default upcase table which is probably the one almost everybody \nuses.\n\nCaveat: I've never used NTFS myself, so I don't have any personal \nknowledge. I can see the sources, and what it thinks it is doing, but \nwhether it works that way or not I'll leave to others.\n\nAlso, note: at least as far as Linux is concerned, NTFS is pure UCS-2. Not \nUTF-16. \n\n> For Mac filesystems, you have the Unicode character decomposition\n> issues to deal with.\n> \n> For Windows, you have UTF-16 filename support.\n\n.. and for pretty much all unixes, you also have potentially Latin1 or any \nother local convention (eg EUC-JP or EUC-KR). \n\nSometimes you'd have to guess from the name itself what it is (ie there \nmight be a mixture). In those cases, it's probably best to *not* even try \nto convert to unicode.\n\n> There are two basic usages for file/directory names: passing the name\n> to the Operating System; getting the name from the Operating System.\n> Therefore, you have:\n> \n> os_to_git_path( const NATIVECHAR * ospath, strbuf * gitpath );\n> git_to_os_path( const char * gitpath, const NATIVECHAR * ospath, int oslen );\n\nIt's not going to be that simple. And if you want type safety, it's the \n\"ospath\" that needs to be \"char *\", since that's what you get from the OS \nand it's really the \"index form\" that you want to protect from giving \nunconverted by mistake to \"lstat()\" and friends.\n\nAnd it's going to be really quite painful even with the compiler pointing \nout each point where you use an \"index name\" with an operation that wants \nan \"OS name\".\n\nIt would be interesting to see how painful it is (make the actual \n\"conversion\" be a no-op at first, just casting the pointer), but I suspect \nthe answer is \"very\".\n\n\t\t\tLinus\n"},{"id":"66322","messageId":"0AEE94F5-C0B2-4B92-AD99-8BF2298428AF@zib.de","threadId":"11703","inReplyTo":"3f4fd2640801220252q6a6557cdma4ca0a2c41223593@mail.gmail.com","subject":"Re: [RFC] File system difference handling in git","fromName":"Steffen Prohaska","fromEmail":"prohaska@zib.de","sentAt":"2008-01-22T17:44:00Z","receivedAt":"2008-01-22T17:44:00Z","isPatch":false,"sender":{"key":"prohaska@zib.de","avatar":"https://avatars.githubusercontent.com/u/217580?v=4"},"body":"\nOn Jan 22, 2008, at 11:52 AM, Reece Dunn wrote:\n\n>>> 2. Case (in)sensitivity\n>>>\n>>> Here, you have the following cases:\n>>> ...\n>>> 3. git says that the files are different, but the filesystem says\n>>> that the files are the same.\n>>>\n>>> Allow the move, updating the git directory tree only.\n>>\n>> Sorry, I cannot really tell what you are talking about. You\n>> seem to imply, with \"Allow the move\", that you are describing a\n>> scenario that involves a move of one existing file to another,\n>> but it is not clear. E.g. did you mean, by 3, \"When the user\n>> says 'move a b', and if git says a and b are different but if\n>> the filesystem says a and b are the same, then...\"?\n>\n> This is what I am saying. For example, if you say:\n>\n> git mv myfile.H myfile.h\n>\n> on a case sensitive filesystem (e.g. ext3), this will work, however on\n> a case insensitive filesystem (e.g. ntfs) git would complain that the\n> files are the same.\n>\n> The workaround is to say:\n>\n> git mv myfile.H myfile.h.tmp\n> git mv myfile.h.tmp myfile.h\n>\n> but this is not ideal, especially if you are automating some move \n> operations.\n>\n> This also applies to the VCS importers (e.g. git-p4) that can delete a\n> file that is a case-only move on case insensitive filesystems.\n>\n> The question then becomes what happens on Mac (with the Unicode\n> decomposing behaviour) if they differ in the way they are stored (e.g.\n> in Linus' 'ä' vs 'a¨' example)?\n\n\nYou can work around the problem as you described; but later git\nwill hit you again and fails unexpectedly when you try to merge\nyour change.\n\nSo better avoid renames that only change case until git at least\npasses the two test below.\n\n\tSteffen\n\n---- snip ---\n\nGit behaves strangely (from a user's point of view) on filesystems\nthat preserve case but do not distinguish it. The two major examples\nare Windows and Mac OS X. Simple operations such as \"git mv\" or \"git\nmerge\" can fail unexpectedly.\n\nThis commit adds two simple tests. Both tests currently fail on\nWindows and Mac, although they pass on Linux.\n\nSigned-off-by: Steffen Prohaska \n---\n t/t0050-filesystems.sh | 36 ++++++++++++++++++++++++++++++++++++\n 1 files changed, 36 insertions(+), 0 deletions(-)\n create mode 100755 t/t0050-filesystems.sh\n\ndiff --git a/t/t0050-filesystems.sh b/t/t0050-filesystems.sh\nnew file mode 100755\nindex 0000000..953b02b\n--- /dev/null\n+++ b/t/t0050-filesystems.sh\n@@ -0,0 +1,36 @@\n+#!/bin/sh\n+\n+test_description='Various filesystems issues'\n+\n+. ./test-lib.sh\n+\n+test_expect_success setup '\n+\n+\ttouch camelcase &&\n+\tgit add camelcase &&\n+\tgit commit -m \"initial\" &&\n+\tgit tag initial &&\n+\tgit checkout -b topic &&\n+\tgit mv camelcase tmp &&\n+\tgit mv tmp CamelCase &&\n+\tgit commit -m \"rename\" &&\n+\tgit checkout -f master\n+\n+'\n+\n+test_expect_success 'rename (case change)' '\n+\n+\tgit mv camelcase CamelCase &&\n+\tgit commit -m \"rename\"\n+\n+'\n+\n+test_expect_success 'merge (case change)' '\n+\n+\tgit reset --hard initial &&\n+\tgit merge topic\n+\n+'\n+\n+test_done\n-- \n1.5.4.rc4\n"},{"id":"66325","messageId":"85ve5l393k.fsf@lola.goethe.zz","threadId":"11703","inReplyTo":"alpine.LFD.1.00.0801220829080.2957@woody.linux-foundation.org","subject":"Re: [RFC] File system difference handling in git","fromName":"David Kastrup","fromEmail":"dak@gnu.org","sentAt":"2008-01-22T20:21:03Z","receivedAt":"2008-01-22T20:21:03Z","isPatch":false,"sender":{"key":"dak@gnu.org","avatar":"https://avatars.githubusercontent.com/u/52141349?v=4"},"body":"Linus Torvalds writes:\n\n> On Tue, 22 Jan 2008, Reece Dunn wrote:\n>> \n>> 1. File name representation\n>> \n>> For Linux file systems (correct me if I am wrong here), they all store\n>> the file name as-is. The question here is what happens on\n>> Windows-based file systems (e.g. NTFS) that are being read on Linux?\n>\n> Generally, Linux tries to follow the conventions of the filesystem, so \n> it's generally case-preserving and case-sensitive (but not normalizing in \n> any way - the case sensitivity is literally a upcase lookup table, so you \n> do \"upcase(c1) == upcase(c2)\" for each UCS-2 character, no combining or \n> decomposition).\n\ns/sensitiv/insensitiv/g\n\n-- \nDavid Kastrup, Kriemhildstr. 15, 44793 Bochum\n"},{"id":"66326","messageId":"57518fd10801221254u19888475s6192fedbbdd0b329@mail.gmail.com","threadId":"11703","inReplyTo":"3f4fd2640801220252q6a6557cdma4ca0a2c41223593@mail.gmail.com","subject":"Re: [RFC] File system difference handling in git","fromName":"Jonathan del Strother","fromEmail":"maillist@steelskies.com","sentAt":"2008-01-22T20:54:51Z","receivedAt":"2008-01-22T20:54:51Z","isPatch":false,"sender":{"key":"jon.delstrother@bestbefore.tv","avatar":"https://gravatar.com/avatar/754e21ab701c00e2d21fc261187254c34b2a1c0b959d9ee5be1a295990be3081?d=mp&s=160"},"body":"On Jan 22, 2008 10:52 AM, Reece Dunn wrote:\n> This is what I am saying. For example, if you say:\n>\n> git mv myfile.H myfile.h\n>\n> on a case sensitive filesystem (e.g. ext3), this will work, however on\n> a case insensitive filesystem (e.g. ntfs) git would complain that the\n> files are the same.\n>\n> The workaround is to say:\n>\n> git mv myfile.H myfile.h.tmp\n> git mv myfile.h.tmp myfile.h\n>\n> but this is not ideal, especially if you are automating some move operations.\n>\n\nIf I remember correctly, this fails when it comes to applying the\ncommit containing that move, at least on HFS+. You could create 2\ncommits (one with the first mv, one with the second), and apply them\none at a time, but it's a pretty unpleasant workaround.\n"},{"id":"66329","messageId":"alpine.LFD.1.00.0801221331520.1741@woody.linux-foundation.org","threadId":"11703","inReplyTo":"85ve5l393k.fsf@lola.goethe.zz","subject":"Re: [RFC] File system difference handling in git","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2008-01-22T21:32:14Z","receivedAt":"2008-01-22T21:32:14Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Tue, 22 Jan 2008, David Kastrup wrote:\n> \n> s/sensitiv/insensitiv/g\n\nDuh. Yes.\n\n\t\tLinus\n"}]}