Re: Cross-Platform Version Control

Esko Luontola:

Is it really necessary to store the encoding for every single file name, should it not be enough to just store encoding information for all file names at once (i.e., for the object that contains the list of file names and their associated blobs)?

I did publish, as a request for comments, the beginnings of a patch that would change the Windows version of Git to expect file names to be UTF-8 encoded. There were some comments about it, especially that I could not just assume that UTF-8 was the right thing to assume.

Perhaps if we added some meta-data, maybe using the same fall-back mechanism as for commit messages (i.e., assume UTF-8 unless otherwise specified), it would be easier to do.

On Windows, the file APIs allow you to use Unicode (UTF-16) to specify file names, and the file systems will handle any necessary conversion to whatever byte sequences are used to store the file names. UTF-16 and UTF-8 are trivial to convert between, and Windows does contain APIs to convert between other character encodings and UTF-16.

On Mac OS X, I believe the file system APIs assume you use some kind of normalized UTF-8. That should also be possible to create, possibly converting back and forth between different normalization forms, if necessary.

On Linux and other Unixes we could just use iconv() to convert from the repository file name encoding to whatever the current locale has set up. The trick here is to handle file names outside the current encoding. Some kind of escaping mechanism will probably need to be introduced.

The best way would be to define this in the Git core once and for all, and add support to it for all the platforms in the same go, instead of trying to hack around the issue whenever it pops up on the various platforms.

My main use-case for Git on Windows has disappeared as my $dayjob went bankrupt, but I am happy to assist with whatever insight I may be able to bring.