Path character encodings

Hi guys,

As part of trying to fix the problems in MSysGit around tree encodings, I would like to start a discussion on mitigating the backwards compatibility problems associated with tree path encodings being unspecified.

## History

For those folks unfamiliar with the issue, I'll provide a quick refresh - Git has traditionally not specified the string encoding of paths inside the tree object - whatever strings the OS provided from the readdir syscall was used verbatim to write out tree objects. For most operating systems, this was UTF-8 (though even on certain POSIX OS's there are some caveats with normalized sequence points, such as OS X).

However, on Windows until *very* recently (and on non-Unicode Linux locales), the strings returned by the OS are from a locale-specific OEM Code Page (i.e. Shift-JIS, Windows-1252, etc) and *not* Unicode. These repositories are currently incorrectly interpreted on other OSs (or even the same OS with a different locale configured).

Note that *blob* (i.e. content) encoding is a separate issue and is out-of-scope at the moment.

This will become a bigger problem in the near future , because MSysGit is seeking to fix this mistake on Windows by explicitly writing all tree objects in UTF-8. While this is great for new repositories, this will create a compatibility problem: people who upgrade their Git installation on their local machine will now have issues with their existing repos.

## Proposed Mitigation

For an initial mitigation plan, I'd like to propose adding a warning to either git clone or git checkout, that if invalid UTF-8 strings are detected, a warning is printed to the user.

However, without an actionable solution, it's not much of a help other than to suggest that they downgrade to a lower version of Git. Possible solutions that we've discussed are:

  * Add a git-config setting to explicitly set the code-page, defaulted to
    UTF-8. With this, the error message could instruct them to set this
    config locally. This has the additional benefit of enabling Linux users
    to use these existing Windows repositories.

  * Creating a conversion utility to rewrite all trees to use UTF-8. This is
    problematic for obvious reasons, even disregarding the fact that the
    result will be incompatible with the original repo - mainly that it may be
    non-trivial to detect which encoding the strings were originally written
    in. libicu (http://site.icu-project.org/) has code to do this.

Paul Betts <paul@paulbetts.org> writes:

Hmph, I do not see a reason to make a huge molehill in this. The pathnames are of unspecified encoding, and if a project declares that they always use UTF-8, that would be great. Older history may need to be rewritten but that is a given.

Wouldn't a flag day event per project that runs filter-branch and have participants restart their repositories be sufficient? Why does git itself have to do anything about it, and how would it help users without hurting other git users who are not involved in such a project?

Hi Junio,

On Mon, Mar 05, 2012 at 01:40:32PM -0800, Junio C Hamano wrote:

I would like to propose, that Git codifies-as-required the majority case today, that trees should *only* be encoded in UTF-8 going forward.

My concern is that users will not know what's happening or how to fix it, they'll just see messed up paths and assume Git is broken. Even a warning message would give users something explicit to put into Google to find a solution for. Not many people will possess the Git-Fu required to manage filter-branch.

Paul Betts <paul@paulbetts.org> writes:

I am afraid that that would be a hard sell.

As the pathnames are uninterpreted strings, an older project that has been using 8859-1 (or EUC for various locales, or anything that is superset of ASCII) has no good incentive or reason to follow such a unilateral decision made outside their project, only to get their participants inconvenienced. Even giving a "warning" will trigger "Yeah we know our paths are in latin-1 and everybody in our project has happily been using that, thankyouverymuch!" and annoy them.

It is not an argument that their history can be re-coded to UTF-8 without losing information, if they still have to go through the conversion process that their project do not benefit from.

Can we make this per-project opt-*in* somehow?