Re-casing directories on case-insensitive systems

Somehow I managed to change the case of a directory without git realizing it. I thought I issued `git mv CS4536 cs4536` but since that won't work in my efforts to reproduce the problem, I must have simply issued the `mv` outside of git and then re-added it.

Anyway, here's the state of my directory:

kevin@KBALLARD:~/Documents/School/C07> git ls-tree HEAD 040000 tree b47c8103e2e01fcf145bdc237c0e56ffc61f1c47 CS4536 040000 tree dbf7fc51ef3effebdf9b4e9172e4c86cae52b163 cs4536 040000 tree 15834a7b6534a285bf6930be4e5404b37e1dc718 ece3601 040000 tree 62d229b8c4a389b550df20a3752d666c48c767a4 ma2071

Note that I have both versions of the directory present. Unfortunately, only one of them can be present on the filesystem. If I run `mv cs4536 CS4536; git reset --hard` I end up with a different working tree.

Git should be able to detect this sort of conflict on a case- insensitive system. I didn't even realize what I'd done until I pushed back to the master repo and ran `git reset --hard` there, then wondered why the new file I added to cs4536/ was missing and why my directory was still named CS4536.

-Kevin Ballard

Wow, it's even worse. I made a tmp branch and used git-filter-branch to remove the commit that introduced CS4536, leaving only the cs4536 directory. But now if I try and run `git co master` it refuses, as it thinks it's going to overwrite the untracked file CS4536/ introduction.txt. I believe it's actually seeing the tracked file cs4536/introduction.txt.

-Kevin Ballard

On Jan 11, 2008, at 3:19 PM, Kevin Ballard wrote:

Err, make that git-rebase (for removing the commit).

-Kevin Ballard

On Jan 11, 2008, at 4:09 PM, Kevin Ballard wrote:

On Fri, 11 Jan 2008, Kevin Ballard wrote:

If you don't have any dirty state, I'd suggest removing your working tree before doing a "git checkout". That's needed anyway to make sure that your working tree has the same case as your index and git trees, because otherwise since the crazy filesystem thinks that CS4536/cs4536 ar ethe same, you might end up having all the wrong names.

Case differences are hard anyway, but you probably made them even harder by them using a rename that actually meant that the old name still *existed* in the filesystem (since the new name would always map to the old name thanks to your crazy filesystem).

I'm sure you can get into even more trouble with case-independent filesystems, but I think you did a pretty good job of hitting on one of the craziest cases ;)

		Linus

fredagen den 11 januari 2008 skrev Kevin Ballard:

I think you should try an index filter. That should help you avoid file system problems.

-- robin

On Fri, 11 Jan 2008, Kevin Ballard wrote:

Hmm. You can do something like

	git ls-files CS4536 | xargs git update-index --force-remove

which will remove gits knowledge of that directory even though "lstat()" will still claim that all the files still exists.

Case-insensitive filesystems are a pain.

I wish we had some way to handle it sanely, but I don't think a sane solution to case-insensitivity exists. If you limit it to strictly just 7bit ascii names (like in your example), then some of the problems do go away, but it would still be probably fairly major surgery to try to teach git about the whole insanity of a case-independent working tree.

		Linus

Hi,

On Fri, 11 Jan 2008, Kevin Ballard wrote:

Do not blame git for the shortcomings of your setup!

However, as luck has it, I looked into this issue again, as somebody raised it with msysgit (for obvious reasons; file systems on Windows are case challenged). If you are serious about this problem, I can give you tons of pointers how to go solve it. (Although I might be disconnected this weekend, because of the lack of competence of the IT department here.)

Ciao, Dscho

On Jan 11, 2008, at 4:29 PM, Johannes Schindelin wrote:

Oh, I'm not surprised git doesn't handle this case, nor do I think git's required to. I merely think that, given the increasing relevance of OS X and the fact that it uses a case-insensitive system by default, this sort of problem is going to occur more and more frequently and it's quite a learning experience trying to fix it by hand. It would be very helpful if git could detect these problems itself.

I think I've got a handle on it. I've already expunged the mis-cased file using git-rebase to remove the offending commit, now I just need to rewrite the second commit's message so it looks like the original commit (luckily I didn't do any work in the directory before I re- cased it). Thanks anyway.

-Kevin Ballard

Hi,

On Fri, 11 Jan 2008, Kevin Ballard wrote:

I was not talking about fixing it up in your repository. If you really think that git should help you, you gotta teach it to. Because people who do not experience the same as you will be less likely to feel the urge to teach git to help in that situation (because they did not experience that situation yet). For example, I am one of these people. And I guess a lot of these people hang out on this list.

Ciao, Dscho

On Fri, 11 Jan 2008, Kevin Ballard wrote:

I do agree that we could/should do something to help with case-insensitive filesystems.

I absolutely *detest* those things, and I think that people who design them are total morons - with MS-DOS, you could understand it (people didn't know better), but with OS X?

But considering that they exist, we should probably offer at least *some* help for people who didn't realize that you could make OS X behave better.

However, it's not like there is even a simple solution. The right place to do that check would probably be in "add_index_entry()", but doing a check whether the same file already exists (in a different case) is simply *extremely* expensive for a very critical piece of code, unless we were to change that index data structure a lot (ie add a separate hash for the filenames).

Inside the Linux kernel, we have support for insane case-insensitive filesystems, and it really does need a lot of effort to do an even half-way decent thing while not penalizing the sane case. So it's hard.

(And that's totally ignoring the fact that case-insensitivity then also has tons of i18n issues and can get *really* messy - in the above I'm talking purely about the issues that would hit us even with 7-bit straight ASCII).

So handling case-sensitivity (even when you restrict it to ASCII-only) is actually rather messy. The obvious thing to do is to sort everything using a case-insensitive sort, but that in turn would break all the rules git has for the sorting of trees.

So you can't just change the sort order: you'd literally have to have two *different* lookup keys for the index (the "strict sort" and the "case- insensitive sort"), and keep them both around.

Almost all of the code that actually touches the index is in read-cache.c, and it's not like that is a very complex data structure (or a very big file), so adding another key to the sorting probably wouldn't be too horrid. But it's definitely a lot more than just a few lines of code!

		Linus

Linus Torvalds <torvalds@linux-foundation.org> writes:

Ah, those young whippersnappers who think they are so smart... there is a history to that, you know. Early character sets (like those on punch cards) had just capital letters. Even when lowercase letters were introduced, those tended to use more space (12 instead of 6 bit) and be harder to print (and the line printers who churned out 40 lines per second did not bother with such finesse, anyway). But capital letters are not designed for readability of long lines. So when printing or even screen terminals came into use, one tended to prefer writing in lowercase letters. Which actually had the uppercase code points usually. Some early microcomputers (for which CP/M was designed) actually were hooked up with a "standard" 50 or 110 Baud teletype as I/O device, and those tended to have only lowercase letters, too, in their basic incantations. So CP/M, not knowing which kind of input device would be used and whether it would prefer (or offer exclusively) upper or lower case, had case-insensitive commands, and consequently also case-insensitive file names. And QDOS (whence MSDOS) was basically intended to be a CP/M ripoff.

OS X has an Apple inheritance, Apple has the same inheritance as other microcomputers which includes a case-insensitive BASIC interpreter (BASIC again coming from old teletype times). It is, again, a decision to drag along old history.

But actually, there is more to it nowadays: two file names containing ü, but one with a single letter and one with combining accent, look exactly the same. If they don't act exactly the same, one opens up quite a hole for spoofing attacks. Well, probably hard to avoid (since things like uppercase Alpha and uppercase A look the same and need to be different code points, too). But one also opens a can of worms for confusion. So the problem of canonical file names does not go away just with case sensitivity.

It is not like Linux does not support some case-insensitive file system types, too. So the same problems can be had there as well.

On Jan 11, 2008, at 6:10 PM, David Kastrup wrote:

In addition, while there is an option for HFS+ Case-Sensitive, using that can cause bad things to happen as Mac OS X programs are written under a case-insensitive assumption and may behave badly when presented with a case-sensitive filesystem.

-Kevin Ballard

fredagen den 11 januari 2008 skrev Linus Torvalds:

Could it be some comfort that the other SCM's I know of make a mess of these cases, regardless of the number of digits in the price tag.

[...]

Could we just have a lookup table index extension for identifying the duplicates (when checking is enabled using core configuration option #3324)? That table would keep a mapping from a normalized form (maybe include canonical encoding while we're at it) to the actual octet sequence(s) used.

Many operations would translate any supplied form throug the table before doing the lookup so if we have Foo.h and give FOO.h to git add, it would notice and perform add (update index) on Foo.h instead as that is the form we alreay know (or refuse yielding an error message; pick your poison). And, well you get the picture.

-- robin

Speaking of normalizing composed sequences, could that be the cause for the following?

kevin@KBALLARD:~/Dev/git> ls kevin@KBALLARD:~/Dev/git> ls -a ./ ../ .git/ .gitignore kevin@KBALLARD:~/Dev/git> git reset --hard HEAD is now at 58beb2c... Trim leading / off of paths in git-svn prop_walk kevin@KBALLARD:~/Dev/git> git st # On branch master # Untracked files: # (use "git add <file>..." to include in what will be committed) # # gitweb/test/Märchen nothing added to commit but untracked files present (use "git add" to track)

Some further exploration seems to support my cause:

kevin@KBALLARD:~/Dev/git> git ls-files gitweb/test "gitweb/test/M\303\244rchen" gitweb/test/file with spaces gitweb/test/file+plus+sign kevin@KBALLARD:~/Dev/git/gitweb/test> ls Märchen | xxd 0000000: 4d61 cc88 7263 6865 6e0a Ma..rchen.

As you can see, git has the file tracked using M\303\244rchen, where \303\244 (or 0xC3A4, or U+00E4) is Latin Small Letter A With Diaeresis, but the filesystem reports it as "Ma\xCC\x88rchen" where 0xCC88 (or U+0308) is Combining Diaeresis.

In other words, the git repository itself exhibits a problem under OS X. I'm not sure if I didn't notice this untracked file before, or if the filesystem (or the index) actually used the other form previously, but regardless there's a problem that I believe would be present even if I was using Case-Sensitive HFS+.

-Kevin Ballard

On Jan 11, 2008, at 6:26 PM, Robin Rosenberg wrote:

lördagen den 12 januari 2008 skrev Kevin Ballard:

[...]

Yes that is due to normalization. When adding a file by name git uses the user supplied name, but when adding files indirectly it gets the names from the file system without denormalizing them. Likewize status gets the names from the file system without denormalizing and thus you get a mismatch.

-- robin

On Jan 11, 2008, at 7:15 PM, Robin Rosenberg wrote:

Is there a reason for this? It seems like it would be trivial to end up with misdiagnosed "untracked" files when using any language other than English given this behaviuor.

-Kevin Ballard

Kevin Ballard <kevin@sb.org> writes:

No. The assumption of the code has always been that sane filesystems would return from readdir() the names you gave from creat().

Hi,

On Fri, 11 Jan 2008, Junio C Hamano wrote:

We do not really have to rehash that whole discussion for the Nth time, do we?

Ciao, Dscho

On Jan 11, 2008, at 7:40 PM, Johannes Schindelin wrote:

Apparently so. By Junio's definition, HFS+ is not a sane filesystem, and as git grows more popular with OS X users, this issue is going to crop up more frequently.

According to the HFS+ Volume Format technote[1], filenames in HFS+ are stored in normalized, canonical order. To be more specific, they're stored in a special apple variant of Unicode Normal Form D (the special variant is for preserving round-trip with older encodings with certain codepoint ranges[2]).

In other words, if you hand an HFS+ filesystem a filename that contains unicode characters, what you get back later may be in a different format. And that's going to be a problem if git doesn't deal with this.

[1]: http://developer.apple.com/technotes/tn/tn1150.html#UnicodeSubtleties [2]: http://developer.apple.com/qa/qa2001/qa1173.html

Note: CC list stripped because this is a re-sent email, as the list  
bounced the last one that contained a text/html alternate.

Kevin Ballard <kevin@sb.org> writes:

It's not "my" definition, but you asked the reason and I gave the answer. We can close this issue of "is HFS+ sane" now. HFS+ is insane, period. And as Linus said, you cannot forgive its insanity using the historical baggage argument, like MS-DOS.

HOWEVER.

It is a totally different issue if we want to refuse supporting insane filesystems. And the answer is no. It was not my intention to say that we do not intend to support them, when I explained the reason why the things are as they are, which was the original question by you.

See Robin's proposal to let us translate random names we get back from readdir() from the filesystem using an additional look-up table in the index extension section that stores mapping from canonicalized form to the form that the user registered to the index. I think that is a sane approach to tackle this issue on insane filesystems like HFS+.

On Jan 11, 2008, at 8:30 PM, Junio C Hamano wrote:

Fair enough, though I believe OS X has a good reason, namely it's an OS designed for regular users rather than servers or programmers. Case- sensitivity would confuse my mother.

Ok. I wasn't implying anything with that phrase there, I was just trying to reiterate that HFS+ is case-insensitive and emphasize that this issue will become more relevant as time goes by.

If I knew what the index extension section was, perhaps I would think that's a good idea ;) I have yet to dive into the gory details of how this stuff works.

Kevin Ballard <kevin@sb.org> writes:

If case-sensitivity would be the primary cause of confusion in mother-computer interoperation, you have a remarkable mother.

"Type things the same way and they work the same" is a simple enough rule.

On Fri, Jan 11, 2008 at 08:43:35PM -0500, Kevin Ballard wrote:

Many of *nix servers are running web-services and samba servers, yet most users are even not aware of whether they dealing with case-sensitive file system or not, let alone being confused by that. This is because most regular users will type the name only once when they create a new file and then just click on this name. So case-sensitive file systems can really confuse only some badly written applications...

Dmitry

Robin Rosenberg <robin.rosenberg@dewire.com> writes:

I would agree that the index extension, if we ever are going to do this, would be the right place to store this information, at the single repository level.

However, this opens up a can of worms. What's the canonical key should be? If you want to protect yourself from a unicode normalizing filesystem, you would use one canonicalization, while if you want to protect from a case losing filesystem you would use another? Or do we at the same time downcase and NFD normalize at the same time and be done with it?

And where should the configuration be stored? If a project wants to be interoperable across Linux and vfat, for example, that canonicalization needs to be enabled in repositories of all participants, be they on Linux or vfat, so that people on Linux can be prevented from creating and register two files xt_mark.c and xt_MARK.c in the same directory, so that people who extract the source on vfat won't have troubles.

Which means the information needs to be in-tree. But that should not be in .gitattributes (which by definition is for per-path things).

lördagen den 12 januari 2008 skrev Junio C Hamano:

The worms are out already. So the question is whether there is a way of keeping them in the can instead of having them crawl all around. I think we could to both unicode (UTF-8 or NFD) and downcase at the same time.

In the brand new ".gitconfig". It could in principle contain any config option, but that would not be safe so I guess one should only allow "safe" options there.

-- robin

Hi,

On Sat, 12 Jan 2008, Robin Rosenberg wrote:

Funny: I had the same idea (.gitconfig) for the crlf issues...

Ciao, Dscho

On Fri, Jan 11, 2008 at 02:08:35PM -0800, Linus Torvalds wrote:

After cursory look at the source code, I wonder if converting name1 and name2 to upper case before memcmp in cache_name_compare() can help case-insensitive systems. This change will change the order of file names in the index, but I suppose that it should not be a problem, because the index is host specific. Though, this fix is too simple, so I guess, I missed something.

The proper support of i18n is not simple even without case-insensitivity. For instance, there are four different encodings widely used for Russian letters. On Windows alone, you have two simulteniously in the default settings -- Windows-1251 for Windows applications and CP866 for Console applications... Actually, some console applications can change its default encoding, and it seems Cygwin programs do that. So, based on whether you use gcc from Cygwin or Visual C to compile your console program, you can get a different encoding. On *nix in Russia, koi8-r and utf-8 are most popular... So, if you have a repository shared between different systems, you cannot think about a file name just as a sequence of bytes anymore. OTOH, I doubt that many people are really interested in using non-ASCII file names with Git right now.

Dmitry

On Sat, 12 Jan 2008, Dmitry Potapov wrote:

No, the index isn't host-specific, and we also have a deep knowledge of the fact that the index order is the same as the unpacked tree order.

So no, we absolutely cannot just sort the index differently. We literally need to have a separate key for a "upper case lookup".

(That separate key can be just a hash table - it doesn't need to be something you can iterate over, so it can be pretty simple).

.. and git is very clear about this: filenames are *not* "characters" in the i18n sense, they are series of bytes. There is absolutely no room for ambiguity, and there is no locale for those things.

And that isn't going to change. It's the only sane way to do locale-independent names: people can *choose* to see the filenames as some UTF-8 sequence, or a series of Latin1, or anything, but that's not something git itself will care about.

Trying to involve locale in name comparison simply isn't possible. Two different repositories on two different filesystems would get two different answers. And that is simply unacceptable in a distributed system.

What we can do is to make the simple cases (ie the locale-*independent* ones) warn about problems with case insensitivity.

			Linus

On Sat, Jan 12, 2008 at 10:47:10AM -0800, Linus Torvalds wrote:

Unfortunately, to agree on a single encoding for different systems is even more difficult than agreeing on a single end-of-line encoding. OTOH, it is not a real issue as long as anyone use ASCII names only.

Agreed. However, the proper solution would be that all filenames are stored in UTF-8, so conversation is done when a file is added to the index. But that requires a lot of work, and as I said before, I doubt that many people really want to store files with non-ASCII names, after all, Git is a developer tool. So, as far as I am concern, it does not worth efforts.

Dmitry