Path character encodings

Hi guys,

As part of trying to fix the problems in MSysGit around tree encodings, I would like to start a discussion on mitigating the backwards compatibility problems associated with tree path encodings being unspecified.

## History

For those folks unfamiliar with the issue, I'll provide a quick refresh - Git has traditionally not specified the string encoding of paths inside the tree object - whatever strings the OS provided from the readdir syscall was used verbatim to write out tree objects. For most operating systems, this was UTF-8 (though even on certain POSIX OS's there are some caveats with normalized sequence points, such as OS X).

However, on Windows until *very* recently (and on non-Unicode Linux locales), the strings returned by the OS are from a locale-specific OEM Code Page (i.e. Shift-JIS, Windows-1252, etc) and *not* Unicode. These repositories are currently incorrectly interpreted on other OSs (or even the same OS with a different locale configured).

Note that *blob* (i.e. content) encoding is a separate issue and is out-of-scope at the moment.

This will become a bigger problem in the near future , because MSysGit is seeking to fix this mistake on Windows by explicitly writing all tree objects in UTF-8. While this is great for new repositories, this will create a compatibility problem: people who upgrade their Git installation on their local machine will now have issues with their existing repos.

## Proposed Mitigation

For an initial mitigation plan, I'd like to propose adding a warning to either git clone or git checkout, that if invalid UTF-8 strings are detected, a warning is printed to the user.

However, without an actionable solution, it's not much of a help other than to suggest that they downgrade to a lower version of Git. Possible solutions that we've discussed are:

  * Add a git-config setting to explicitly set the code-page, defaulted to
    UTF-8. With this, the error message could instruct them to set this
    config locally. This has the additional benefit of enabling Linux users
    to use these existing Windows repositories.

  * Creating a conversion utility to rewrite all trees to use UTF-8. This is
    problematic for obvious reasons, even disregarding the fact that the
    result will be incompatible with the original repo - mainly that it may be
    non-trivial to detect which encoding the strings were originally written
    in. libicu (http://site.icu-project.org/) has code to do this.