{"thread":{"id":"29845","subject":"Path character encodings","startedAt":"2012-03-05T21:26:57Z","lastAt":"2012-03-05T22:18:27Z","messageCount":4,"participants":["Paul Betts","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"186142","messageId":"20120305212657.GA17903@jupiter.local","threadId":"29845","inReplyTo":null,"subject":"Path character encodings","fromName":"Paul Betts","fromEmail":"paul@paulbetts.org","sentAt":"2012-03-05T21:26:57Z","receivedAt":"2012-03-05T21:26:57Z","isPatch":false,"sender":{"key":"paul@paulbetts.org","avatar":"https://gravatar.com/avatar/0a6b8d841f415926d3822428107890edb67a784a62ddd8da6f3ec19074bd5024?d=mp&s=160"},"body":"Hi guys,\n\nAs part of trying to fix the problems in MSysGit around tree encodings, I\nwould like to start a discussion on mitigating the backwards compatibility\nproblems associated with tree path encodings being unspecified.\n\n## History\n\nFor those folks unfamiliar with the issue, I'll provide a quick refresh - Git\nhas traditionally not specified the string encoding of paths inside the tree\nobject - whatever strings the OS provided from the readdir syscall was used\nverbatim to write out tree objects. For most operating systems, this was UTF-8\n(though even on certain POSIX OS's there are some caveats with normalized\nsequence points, such as OS X).\n\nHowever, on Windows until *very* recently (and on non-Unicode Linux locales),\nthe strings returned by the OS are from a locale-specific OEM Code Page (i.e.\nShift-JIS, Windows-1252, etc) and *not* Unicode. These repositories are\ncurrently incorrectly interpreted on other OSs (or even the same OS with a\ndifferent locale configured).\n\nNote that *blob* (i.e. content) encoding is a separate issue and is\nout-of-scope at the moment.\n\nThis will become a bigger problem in the near future , because MSysGit is\nseeking to fix this mistake on Windows by explicitly writing all tree objects\nin UTF-8. While this is great for new repositories, this will create a\ncompatibility problem: people who upgrade their Git installation on their\nlocal machine will now have issues with their existing repos.\n\n## Proposed Mitigation\n\nFor an initial mitigation plan, I'd like to propose adding a warning to either\ngit clone or git checkout, that if invalid UTF-8 strings are detected, a\nwarning is printed to the user.\n\nHowever, without an actionable solution, it's not much of a help other than to\nsuggest that they downgrade to a lower version of Git. Possible solutions that\nwe've discussed are:\n\n  * Add a git-config setting to explicitly set the code-page, defaulted to\n    UTF-8. With this, the error message could instruct them to set this\n    config locally. This has the additional benefit of enabling Linux users\n    to use these existing Windows repositories.\n\n  * Creating a conversion utility to rewrite all trees to use UTF-8. This is\n    problematic for obvious reasons, even disregarding the fact that the\n    result will be incompatible with the original repo - mainly that it may be\n    non-trivial to detect which encoding the strings were originally written\n    in. libicu (http://site.icu-project.org/) has code to do this.\n\n-- \nPaul Betts <paul@paulbetts.org>\n"},{"id":"186144","messageId":"7vipiizpyn.fsf@alter.siamese.dyndns.org","threadId":"29845","inReplyTo":"20120305212657.GA17903@jupiter.local","subject":"Re: Path character encodings","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2012-03-05T21:40:32Z","receivedAt":"2012-03-05T21:40:32Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Paul Betts <paul@paulbetts.org> writes:\n\n> ## Proposed Mitigation\n>\n> For an initial mitigation plan, I'd like to propose adding a warning to either\n> git clone or git checkout, that if invalid UTF-8 strings are detected, a\n> warning is printed to the user.\n>\n> However, without an actionable solution, it's not much of a help other than to\n> suggest that they downgrade to a lower version of Git.\n\nHmph, I do not see a reason to make a huge molehill in this. The\npathnames are of unspecified encoding, and if a project declares\nthat they always use UTF-8, that would be great. Older history may\nneed to be rewritten but that is a given.\n\nWouldn't a flag day event per project that runs filter-branch and\nhave participants restart their repositories be sufficient?  Why\ndoes git itself have to do anything about it, and how would it help\nusers without hurting other git users who are not involved in such a\nproject?\n"},{"id":"186148","messageId":"20120305220232.GA18247@jupiter.local","threadId":"29845","inReplyTo":"7vipiizpyn.fsf@alter.siamese.dyndns.org","subject":"Re: Path character encodings","fromName":"Paul Betts","fromEmail":"paul@paulbetts.org","sentAt":"2012-03-05T22:02:32Z","receivedAt":"2012-03-05T22:02:32Z","isPatch":false,"sender":{"key":"paul@paulbetts.org","avatar":"https://gravatar.com/avatar/0a6b8d841f415926d3822428107890edb67a784a62ddd8da6f3ec19074bd5024?d=mp&s=160"},"body":"Hi Junio,\n\nOn Mon, Mar 05, 2012 at 01:40:32PM -0800, Junio C Hamano wrote:\n> Hmph, I do not see a reason to make a huge molehill in this. The\n> pathnames are of unspecified encoding, and if a project declares\n> that they always use UTF-8, that would be great. \n\nI would like to propose, that Git codifies-as-required the majority case\ntoday, that trees should *only* be encoded in UTF-8 going forward. \n\n> Why does git itself have to do anything about it, and how would it help\n> users without hurting other git users who are not involved in such a\n> project?\n\nMy concern is that users will not know what's happening or how to fix it,\nthey'll just see messed up paths and assume Git is broken. Even a warning\nmessage would give users something explicit to put into Google to find a\nsolution for. Not many people will possess the Git-Fu required to manage\nfilter-branch. \n\n-- \nPaul Betts <paul@paulbetts.org>\n"},{"id":"186149","messageId":"7v62eizo7g.fsf@alter.siamese.dyndns.org","threadId":"29845","inReplyTo":"20120305220232.GA18247@jupiter.local","subject":"Re: Path character encodings","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2012-03-05T22:18:27Z","receivedAt":"2012-03-05T22:18:27Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Paul Betts <paul@paulbetts.org> writes:\n\n> On Mon, Mar 05, 2012 at 01:40:32PM -0800, Junio C Hamano wrote:\n>> Hmph, I do not see a reason to make a huge molehill in this. The\n>> pathnames are of unspecified encoding, and if a project declares\n>> that they always use UTF-8, that would be great. \n>\n> I would like to propose, that Git codifies-as-required the majority case\n> today, that trees should *only* be encoded in UTF-8 going forward. \n\nI am afraid that that would be a hard sell.\n\nAs the pathnames are uninterpreted strings, an older project that\nhas been using 8859-1 (or EUC for various locales, or anything that\nis superset of ASCII) has no good incentive or reason to follow such\na unilateral decision made outside their project, only to get their\nparticipants inconvenienced.  Even giving a \"warning\" will trigger\n\"Yeah we know our paths are in latin-1 and everybody in our project\nhas happily been using that, thankyouverymuch!\" and annoy them.\n\nIt is not an argument that their history can be re-coded to UTF-8\nwithout losing information, if they still have to go through the\nconversion process that their project do not benefit from.\n\nCan we make this per-project opt-*in* somehow?\n"}]}