{"thread":{"id":"15874","subject":"[JGIT PATCH 0/4] Decodings","startedAt":"2008-10-12T22:50:56Z","lastAt":"2008-10-13T17:10:50Z","messageCount":8,"participants":["Robin Rosenberg","Shawn O. Pearce"],"isPatch":true,"patchVersion":1,"patchTotal":4},"messages":[{"id":"92880","messageId":"1223851860-13068-1-git-send-email-robin.rosenberg@dewire.com","threadId":"15874","inReplyTo":null,"subject":"[JGIT PATCH 0/4] Decodings","fromName":"Robin Rosenberg","fromEmail":"robin.rosenberg@dewire.com","sentAt":"2008-10-12T22:50:56Z","receivedAt":"2008-10-12T22:50:56Z","isPatch":true,"sender":{"key":"robin.rosenberg@dewire.com","avatar":"https://avatars.githubusercontent.com/u/46357?v=4"},"body":"These patches might be slightly controversial. Since there is no perfect\nsolution, we may want to try something that works to some extent and gives\nwhat a user might expect, i.e. presenting to a viewer the same glyphs \nthat the user who entered them saw, to the extent it is possible.\n\nWe already handle commit messages like this for the old style objects (sort of).\nThis patch set also affects other data like refs. Currenly no sane solution\nexists in git so nothing really works well outside the non-ascii range for\nrefs anyway so we can discuss what should happen with refs that contain\nnon-ascii characters. The best thing is to avoid them, but some of us live\nin countries with funny dots in what we do and other have even stranger ways\nof expressing what they do, and hence things like branch names etc.\n\nLegacy SCM to GIT conversion programs seem to do every variation of transcoding/\nnot transcoding commit messages and file names to UTF-8 so there is an issue here.\nThe nice thing about transcoding filenames to UTF-8 is that they work on all\nplatforms. A non-UTF-8 filename in a UTF-8 environement doesn't. In particular \nsuch filenames are more or less inaccessible to a Java programs. For the reverse\ncase it looks really bad. C Git currently does not transform file names. Missing from\nthis patch set is test cases. As it is quite undefined in git what happens that\nis sort of ok so far, but I'd like to define it too in the same way.\n\n-- robin\n"},{"id":"92881","messageId":"1223851860-13068-2-git-send-email-robin.rosenberg@dewire.com","threadId":"15874","inReplyTo":"1223851860-13068-1-git-send-email-robin.rosenberg@dewire.com","subject":"[JGIT PATCH 1/4] log command: Use explicit US locale for dates","fromName":"Robin Rosenberg","fromEmail":"robin.rosenberg@dewire.com","sentAt":"2008-10-12T22:50:57Z","receivedAt":"2008-10-12T22:50:57Z","isPatch":true,"sender":{"key":"robin.rosenberg@dewire.com","avatar":"https://avatars.githubusercontent.com/u/46357?v=4"},"body":"The format used is a variation of the US locale and looks odd when localized\nanyway so do not try to localize it.\n\nSigned-off-by: Robin Rosenberg <robin.rosenberg@dewire.com>\n---\n .../src/org/spearce/jgit/pgm/Log.java              |    3 ++-\n 1 files changed, 2 insertions(+), 1 deletions(-)\n\ndiff --git a/org.spearce.jgit.pgm/src/org/spearce/jgit/pgm/Log.java b/org.spearce.jgit.pgm/src/org/spearce/jgit/pgm/Log.java\nindex e16387b..e3a32c8 100644\n--- a/org.spearce.jgit.pgm/src/org/spearce/jgit/pgm/Log.java\n+++ b/org.spearce.jgit.pgm/src/org/spearce/jgit/pgm/Log.java\n@@ -40,6 +40,7 @@\n \n import java.text.DateFormat;\n import java.text.SimpleDateFormat;\n+import java.util.Locale;\n import java.util.TimeZone;\n \n import org.spearce.jgit.lib.PersonIdent;\n@@ -52,7 +53,7 @@\n \tprivate final DateFormat fmt;\n \n \tLog() {\n-\t\tfmt = new SimpleDateFormat(\"EEE MMM dd HH:mm:ss yyyy ZZZZZ\");\n+\t\tfmt = new SimpleDateFormat(\"EEE MMM dd HH:mm:ss yyyy ZZZZZ\", Locale.US);\n \t}\n \n \t@Override\n-- \n1.6.0.2.308.gef4a\n"},{"id":"92883","messageId":"1223851860-13068-3-git-send-email-robin.rosenberg@dewire.com","threadId":"15874","inReplyTo":"1223851860-13068-2-git-send-email-robin.rosenberg@dewire.com","subject":"[JGIT PATCH 2/4] jgit programs: Use i18n.logOutputEncoding or user's locale for output","fromName":"Robin Rosenberg","fromEmail":"robin.rosenberg@dewire.com","sentAt":"2008-10-12T22:50:58Z","receivedAt":"2008-10-12T22:50:58Z","isPatch":true,"sender":{"key":"robin.rosenberg@dewire.com","avatar":"https://avatars.githubusercontent.com/u/46357?v=4"},"body":"We should present the data in the user's locale to make it readable.\n\nThis prevents garbage from being displayed for user's whose locale is not\nUTF-8, when non-ascii appears in most cases. If the characters cannot be\nconverted garbage will appear in any case.\n\nWhen jgit gains the capability to present blob data this code must be\nreplaced to handle data and metadata differently.\n\nSigned-off-by: Robin Rosenberg <robin.rosenberg@dewire.com>\n---\n .../src/org/spearce/jgit/pgm/TextBuiltin.java      |   11 +++++++++--\n 1 files changed, 9 insertions(+), 2 deletions(-)\n\ndiff --git a/org.spearce.jgit.pgm/src/org/spearce/jgit/pgm/TextBuiltin.java b/org.spearce.jgit.pgm/src/org/spearce/jgit/pgm/TextBuiltin.java\nindex a68d87c..d1bf9e0 100644\n--- a/org.spearce.jgit.pgm/src/org/spearce/jgit/pgm/TextBuiltin.java\n+++ b/org.spearce.jgit.pgm/src/org/spearce/jgit/pgm/TextBuiltin.java\n@@ -86,8 +86,15 @@ final void setCommandName(final String name) {\n \n \tvoid init(final Repository repo) {\n \t\ttry {\n-\t\t\tout = new PrintWriter(new BufferedWriter(new OutputStreamWriter(\n-\t\t\t\t\tSystem.out, \"UTF-8\")));\n+\t\t\tString outputEncoding = repo.getConfig().getString(\"i18n\", null,\n+\t\t\t\t\t\"logOutputEncoding\");\n+\t\t\tSystem.out.println(\"Encoding = \"+ outputEncoding);\n+\t\t\tif (outputEncoding != null)\n+\t\t\t\tout = new PrintWriter(new BufferedWriter(\n+\t\t\t\t\t\tnew OutputStreamWriter(System.out, outputEncoding)));\n+\t\t\telse\n+\t\t\t\tout = new PrintWriter(new BufferedWriter(\n+\t\t\t\t\t\tnew OutputStreamWriter(System.out)));\n \t\t} catch (IOException e) {\n \t\t\tthrow die(\"cannot create output stream\");\n \t\t}\n-- \n1.6.0.2.308.gef4a\n"},{"id":"92882","messageId":"1223851860-13068-4-git-send-email-robin.rosenberg@dewire.com","threadId":"15874","inReplyTo":"1223851860-13068-3-git-send-email-robin.rosenberg@dewire.com","subject":"[JGIT PATCH 3/4] The git config file is case insensitive","fromName":"Robin Rosenberg","fromEmail":"robin.rosenberg@dewire.com","sentAt":"2008-10-12T22:50:59Z","receivedAt":"2008-10-12T22:50:59Z","isPatch":true,"sender":{"key":"robin.rosenberg@dewire.com","avatar":"https://avatars.githubusercontent.com/u/46357?v=4"},"body":"Signed-off-by: Robin Rosenberg <robin.rosenberg@dewire.com>\n---\n .../org/spearce/jgit/lib/RepositoryConfigTest.java |    8 ++++++++\n .../src/org/spearce/jgit/lib/RepositoryConfig.java |   18 ++++++++++++------\n 2 files changed, 20 insertions(+), 6 deletions(-)\n\ndiff --git a/org.spearce.jgit.test/tst/org/spearce/jgit/lib/RepositoryConfigTest.java b/org.spearce.jgit.test/tst/org/spearce/jgit/lib/RepositoryConfigTest.java\nindex da7e704..bd5329c 100644\n--- a/org.spearce.jgit.test/tst/org/spearce/jgit/lib/RepositoryConfigTest.java\n+++ b/org.spearce.jgit.test/tst/org/spearce/jgit/lib/RepositoryConfigTest.java\n@@ -109,4 +109,12 @@ assertTrue(Arrays.equals(values.toArray(), repositoryConfig\n \t\t\t\t.getStringList(\"my\", null, \"somename\")));\n \t\tcheckFile(cfgFile, \"[my]\\n\\tsomename = value1\\n\\tsomename = value2\\n\");\n \t}\n+\n+\tpublic void test006_readCaseInsensitive() throws IOException {\n+\t\tfinal File path = writeTrashFile(\"config_001\", \"[Foo]\\nBar\\n\");\n+\t\tRepositoryConfig repositoryConfig = new RepositoryConfig(null, path);\n+\t\tSystem.out.println(repositoryConfig.getString(\"foo\", null, \"bar\"));\n+\t\tassertEquals(true, repositoryConfig.getBoolean(\"foo\", null, \"bar\", false));\n+\t\tassertEquals(\"\", repositoryConfig.getString(\"foo\", null, \"bar\"));\n+\t}\n }\ndiff --git a/org.spearce.jgit/src/org/spearce/jgit/lib/RepositoryConfig.java b/org.spearce.jgit/src/org/spearce/jgit/lib/RepositoryConfig.java\nindex 45c2f8a..7a34cde 100644\n--- a/org.spearce.jgit/src/org/spearce/jgit/lib/RepositoryConfig.java\n+++ b/org.spearce.jgit/src/org/spearce/jgit/lib/RepositoryConfig.java\n@@ -52,12 +52,13 @@\n import java.io.PrintWriter;\n import java.util.ArrayList;\n import java.util.Collections;\n-import java.util.HashMap;\n+import java.util.Comparator;\n import java.util.HashSet;\n import java.util.Iterator;\n import java.util.List;\n import java.util.Map;\n import java.util.Set;\n+import java.util.TreeMap;\n \n import org.spearce.jgit.util.FS;\n \n@@ -236,9 +237,9 @@ protected boolean getBoolean(final String section, String subsection,\n \t\t\treturn defaultValue;\n \n \t\tn = n.toLowerCase();\n-\t\tif (MAGIC_EMPTY_VALUE.equals(n) || \"yes\".equals(n) || \"true\".equals(n) || \"1\".equals(n)) {\n+\t\tif (MAGIC_EMPTY_VALUE.equals(n) || \"yes\".equalsIgnoreCase(n) || \"true\".equalsIgnoreCase(n) || \"1\".equals(n)) {\n \t\t\treturn true;\n-\t\t} else if (\"no\".equals(n) || \"false\".equals(n) || \"0\".equals(n)) {\n+\t\t} else if (\"no\".equalsIgnoreCase(n) || \"false\".equalsIgnoreCase(n) || \"0\".equalsIgnoreCase(n)) {\n \t\t\treturn false;\n \t\t} else {\n \t\t\tthrow new IllegalArgumentException(\"Invalid boolean value: \"\n@@ -300,7 +301,7 @@ public String getString(final String section, String subsection, final String na\n \t\tfinal Set<String> result = new HashSet<String>();\n \n \t\tfor (final Entry e : entries) {\n-\t\t\tif (section.equals(e.base) && e.extendedBase != null)\n+\t\t\tif (section.equalsIgnoreCase(e.base) && e.extendedBase != null)\n \t\t\t\tresult.add(e.extendedBase);\n \t\t}\n \t\tif (baseConfig != null)\n@@ -682,7 +683,12 @@ public void load() throws IOException {\n \n \tprivate void clear() {\n \t\tentries = new ArrayList<Entry>();\n-\t\tbyName = new HashMap<String, Object>();\n+\t\tbyName = new TreeMap<String, Object>(new Comparator<String>() {\n+\n+\t\t\tpublic int compare(String o1, String o2) {\n+\t\t\t\treturn o1.compareToIgnoreCase(o2);\n+\t\t\t}\n+\t\t});\n \t}\n \n \t@SuppressWarnings(\"unchecked\")\n@@ -954,7 +960,7 @@ private static boolean eq(final String a, final String b) {\n \t\t\t\treturn true;\n \t\t\tif (a == null || b == null)\n \t\t\t\treturn false;\n-\t\t\treturn a.equals(b);\n+\t\t\treturn a.equalsIgnoreCase(b);\n \t\t}\n \t}\n }\n-- \n1.6.0.2.308.gef4a\n"},{"id":"92884","messageId":"1223851860-13068-5-git-send-email-robin.rosenberg@dewire.com","threadId":"15874","inReplyTo":"1223851860-13068-4-git-send-email-robin.rosenberg@dewire.com","subject":"[JGIT PATCH 4/4] Intelligent parsing of ambiguously encoded meta data.","fromName":"Robin Rosenberg","fromEmail":"robin.rosenberg@dewire.com","sentAt":"2008-10-12T22:51:00Z","receivedAt":"2008-10-12T22:51:00Z","isPatch":true,"sender":{"key":"robin.rosenberg@dewire.com","avatar":"https://avatars.githubusercontent.com/u/46357?v=4"},"body":"We cannot trust meta data to be encoded in any particular way, so we try\ndifferent encodings. First we try UTF-8, which is the only sane encoding\nfor non-local data, even when used in regions where eight bit legacy\nencodings are common. The chance of mistakenly parsing non-UTF-8 data\nas valid UTF-8 is varies from extremely low (western encodings) to low\nfor most other encodings. If the data does not look like UTF-8, we try the\nsuggested encoding. If that fails we try the user locale and finally, if\nthat fails we try ISO-8859-1, which cannot fail.\n\nSigned-off-by: Robin Rosenberg <robin.rosenberg@dewire.com>\n---\n .../spearce/jgit/revwalk/RevCommitParseTest.java   |  119 ++++++++++++++++++++\n .../src/org/spearce/jgit/util/RawParseUtils.java   |   63 ++++++++++-\n 2 files changed, 179 insertions(+), 3 deletions(-)\n\ndiff --git a/org.spearce.jgit.test/tst/org/spearce/jgit/revwalk/RevCommitParseTest.java b/org.spearce.jgit.test/tst/org/spearce/jgit/revwalk/RevCommitParseTest.java\nindex 3d9d42d..805e29e 100644\n--- a/org.spearce.jgit.test/tst/org/spearce/jgit/revwalk/RevCommitParseTest.java\n+++ b/org.spearce.jgit.test/tst/org/spearce/jgit/revwalk/RevCommitParseTest.java\n@@ -37,6 +37,8 @@\n \n package org.spearce.jgit.revwalk;\n \n+import java.io.ByteArrayOutputStream;\n+\n import org.spearce.jgit.lib.ObjectId;\n import org.spearce.jgit.lib.PersonIdent;\n import org.spearce.jgit.lib.RepositoryTestCase;\n@@ -130,6 +132,123 @@ public void testParse_WeirdHeaderOnlyCommit() throws Exception {\n \t\tassertEquals(\"\", c.getShortMessage());\n \t}\n \n+\tpublic void testParse_implicit_UTF8_encoded() throws Exception {\n+\t\tfinal ByteArrayOutputStream b = new ByteArrayOutputStream();\n+\t\tb.write(\"tree 9788669ad918b6fcce64af8882fc9a81cb6aba67\\n\".getBytes(\"UTF-8\"));\n+\t\tb.write(\"author F\\u00f6r fattare <a_u_thor@example.com> 1218123387 +0700\\n\".getBytes(\"UTF-8\"));\n+\t\tb.write(\"committer C O. Miter <c@example.com> 1218123390 -0500\\n\".getBytes(\"UTF-8\"));\n+\t\tb.write(\"\\n\".getBytes(\"UTF-8\"));\n+\t\tb.write(\"Sm\\u00f6rg\\u00e5sbord\\n\".getBytes(\"UTF-8\"));\n+\t\tb.write(\"\\n\".getBytes(\"UTF-8\"));\n+\t\tb.write(\"\\u304d\\u308c\\u3044\\n\".getBytes(\"UTF-8\"));\n+\t\tfinal RevCommit c;\n+\t\tc = new RevCommit(id(\"9473095c4cb2f12aefe1db8a355fe3fafba42f67\")); // bogus id\n+\t\tc.parseCanonical(new RevWalk(db), b.toByteArray());\n+\n+\t\tassertEquals(\"F\\u00f6r fattare\", c.getAuthorIdent().getName());\n+\t\tassertEquals(\"Sm\\u00f6rg\\u00e5sbord\", c.getShortMessage());\n+\t\tassertEquals(\"Sm\\u00f6rg\\u00e5sbord\\n\\n\\u304d\\u308c\\u3044\\n\", c.getFullMessage());\n+\t}\n+\n+\tpublic void testParse_implicit_mixed_encoded() throws Exception {\n+\t\tfinal ByteArrayOutputStream b = new ByteArrayOutputStream();\n+\t\tb.write(\"tree 9788669ad918b6fcce64af8882fc9a81cb6aba67\\n\".getBytes(\"UTF-8\"));\n+\t\tb.write(\"author F\\u00f6r fattare <a_u_thor@example.com> 1218123387 +0700\\n\".getBytes(\"ISO-8859-1\"));\n+\t\tb.write(\"committer C O. Miter <c@example.com> 1218123390 -0500\\n\".getBytes(\"UTF-8\"));\n+\t\tb.write(\"\\n\".getBytes(\"UTF-8\"));\n+\t\tb.write(\"Sm\\u00f6rg\\u00e5sbord\\n\".getBytes(\"UTF-8\"));\n+\t\tb.write(\"\\n\".getBytes(\"UTF-8\"));\n+\t\tb.write(\"\\u304d\\u308c\\u3044\\n\".getBytes(\"UTF-8\"));\n+\t\tfinal RevCommit c;\n+\t\tc = new RevCommit(id(\"9473095c4cb2f12aefe1db8a355fe3fafba42f67\")); // bogus id\n+\t\tc.parseCanonical(new RevWalk(db), b.toByteArray());\n+\n+\t\tassertEquals(\"F\\u00f6r fattare\", c.getAuthorIdent().getName());\n+\t\tassertEquals(\"Sm\\u00f6rg\\u00e5sbord\", c.getShortMessage());\n+\t\tassertEquals(\"Sm\\u00f6rg\\u00e5sbord\\n\\n\\u304d\\u308c\\u3044\\n\", c.getFullMessage());\n+\t}\n+\n+\t/**\n+\t * Test parsing of a commit whose encoding is given and works.\n+\t *\n+\t * @throws Exception\n+\t */\n+\tpublic void testParse_explicit_encoded() throws Exception {\n+\t\tfinal ByteArrayOutputStream b = new ByteArrayOutputStream();\n+\t\tb.write(\"tree 9788669ad918b6fcce64af8882fc9a81cb6aba67\\n\".getBytes(\"EUC-JP\"));\n+\t\tb.write(\"author F\\u00f6r fattare <a_u_thor@example.com> 1218123387 +0700\\n\".getBytes(\"EUC-JP\"));\n+\t\tb.write(\"committer C O. Miter <c@example.com> 1218123390 -0500\\n\".getBytes(\"EUC-JP\"));\n+\t\tb.write(\"encoding euc_JP\\n\".getBytes(\"EUC-JP\"));\n+\t\tb.write(\"\\n\".getBytes(\"EUC-JP\"));\n+\t\tb.write(\"\\u304d\\u308c\\u3044\\n\".getBytes(\"EUC-JP\"));\n+\t\tb.write(\"\\n\".getBytes(\"EUC-JP\"));\n+\t\tb.write(\"Hi\\n\".getBytes(\"EUC-JP\"));\n+\t\tfinal RevCommit c;\n+\t\tc = new RevCommit(id(\"9473095c4cb2f12aefe1db8a355fe3fafba42f67\")); // bogus id\n+\t\tc.parseCanonical(new RevWalk(db), b.toByteArray());\n+\n+\t\tassertEquals(\"F\\u00f6r fattare\", c.getAuthorIdent().getName());\n+\t\tassertEquals(\"\\u304d\\u308c\\u3044\", c.getShortMessage());\n+\t\tassertEquals(\"\\u304d\\u308c\\u3044\\n\\nHi\\n\", c.getFullMessage());\n+\t}\n+\n+\t/**\n+\t * This is a twisted case, but show what we expect here. We can revise the\n+\t * expectations provided this case is updated.\n+\t *\n+\t * What happens here is that an encoding us given, but data is not encoded\n+\t * that way (and we can detect it), so we try other encodings.\n+\t *\n+\t * @throws Exception\n+\t */\n+\tpublic void testParse_explicit_bad_encoded() throws Exception {\n+\t\tfinal ByteArrayOutputStream b = new ByteArrayOutputStream();\n+\t\tb.write(\"tree 9788669ad918b6fcce64af8882fc9a81cb6aba67\\n\".getBytes(\"UTF-8\"));\n+\t\tb.write(\"author F\\u00f6r fattare <a_u_thor@example.com> 1218123387 +0700\\n\".getBytes(\"ISO-8859-1\"));\n+\t\tb.write(\"committer C O. Miter <c@example.com> 1218123390 -0500\\n\".getBytes(\"UTF-8\"));\n+\t\tb.write(\"encoding EUC-JP\\n\".getBytes(\"UTF-8\"));\n+\t\tb.write(\"\\n\".getBytes(\"UTF-8\"));\n+\t\tb.write(\"\\u304d\\u308c\\u3044\\n\".getBytes(\"UTF-8\"));\n+\t\tb.write(\"\\n\".getBytes(\"UTF-8\"));\n+\t\tb.write(\"Hi\\n\".getBytes(\"UTF-8\"));\n+\t\tfinal RevCommit c;\n+\t\tc = new RevCommit(id(\"9473095c4cb2f12aefe1db8a355fe3fafba42f67\")); // bogus id\n+\t\tc.parseCanonical(new RevWalk(db), b.toByteArray());\n+\n+\t\tassertEquals(\"F\\u00f6r fattare\", c.getAuthorIdent().getName());\n+\t\tassertEquals(\"\\u304d\\u308c\\u3044\", c.getShortMessage());\n+\t\tassertEquals(\"\\u304d\\u308c\\u3044\\n\\nHi\\n\", c.getFullMessage());\n+\t}\n+\n+\t/**\n+\t * This is a twisted case too, but show what we expect here. We can revise the\n+\t * expectations provided this case is updated.\n+\t *\n+\t * What happens here is that an encoding us given, but data is not encoded\n+\t * that way (and we can detect it), so we try other encodings. Here data could\n+\t * actually be decoded in the stated encoding, but we overide using UTF-8.\n+\t *\n+\t * @throws Exception\n+\t */\n+\tpublic void testParse_explicit_bad_encoded2() throws Exception {\n+\t\tfinal ByteArrayOutputStream b = new ByteArrayOutputStream();\n+\t\tb.write(\"tree 9788669ad918b6fcce64af8882fc9a81cb6aba67\\n\".getBytes(\"UTF-8\"));\n+\t\tb.write(\"author F\\u00f6r fattare <a_u_thor@example.com> 1218123387 +0700\\n\".getBytes(\"UTF-8\"));\n+\t\tb.write(\"committer C O. Miter <c@example.com> 1218123390 -0500\\n\".getBytes(\"UTF-8\"));\n+\t\tb.write(\"encoding ISO-8859-1\\n\".getBytes(\"UTF-8\"));\n+\t\tb.write(\"\\n\".getBytes(\"UTF-8\"));\n+\t\tb.write(\"\\u304d\\u308c\\u3044\\n\".getBytes(\"UTF-8\"));\n+\t\tb.write(\"\\n\".getBytes(\"UTF-8\"));\n+\t\tb.write(\"Hi\\n\".getBytes(\"UTF-8\"));\n+\t\tfinal RevCommit c;\n+\t\tc = new RevCommit(id(\"9473095c4cb2f12aefe1db8a355fe3fafba42f67\")); // bogus id\n+\t\tc.parseCanonical(new RevWalk(db), b.toByteArray());\n+\n+\t\tassertEquals(\"F\\u00f6r fattare\", c.getAuthorIdent().getName());\n+\t\tassertEquals(\"\\u304d\\u308c\\u3044\", c.getShortMessage());\n+\t\tassertEquals(\"\\u304d\\u308c\\u3044\\n\\nHi\\n\", c.getFullMessage());\n+\t}\n+\n \tpublic void testParse_NoMessage() throws Exception {\n \t\tfinal String msg = \"\";\n \t\tfinal RevCommit c = create(msg);\ndiff --git a/org.spearce.jgit/src/org/spearce/jgit/util/RawParseUtils.java b/org.spearce.jgit/src/org/spearce/jgit/util/RawParseUtils.java\nindex a31734b..7c16394 100644\n--- a/org.spearce.jgit/src/org/spearce/jgit/util/RawParseUtils.java\n+++ b/org.spearce.jgit/src/org/spearce/jgit/util/RawParseUtils.java\n@@ -42,7 +42,10 @@\n import static org.spearce.jgit.lib.ObjectChecker.encoding;\n \n import java.nio.ByteBuffer;\n+import java.nio.charset.CharacterCodingException;\n import java.nio.charset.Charset;\n+import java.nio.charset.CharsetDecoder;\n+import java.nio.charset.CodingErrorAction;\n import java.util.Arrays;\n \n import org.spearce.jgit.lib.Constants;\n@@ -376,7 +379,10 @@ public static PersonIdent parsePersonIdent(final byte[] raw, final int nameB) {\n \t}\n \n \t/**\n-\t * Decode a region of the buffer under the specified character set.\n+\t * Decode a region of the buffer under the specified character set if possible.\n+\t *\n+\t * If the byte stream cannot be decoded that way, the platform default is tried\n+\t * and if that too fails, the fail-safe ISO-8859-1 encoding is tried.\n \t * \n \t * @param cs\n \t *            character set to use when decoding the buffer.\n@@ -393,12 +399,63 @@ public static PersonIdent parsePersonIdent(final byte[] raw, final int nameB) {\n \tpublic static String decode(final Charset cs, final byte[] buffer,\n \t\t\tfinal int start, final int end) {\n \t\tfinal ByteBuffer b = ByteBuffer.wrap(buffer, start, end - start);\n-\t\treturn cs.decode(b).toString();\n+\t\tb.mark();\n+\t\tfor (int i = 0;; ++i) {\n+\t\t\ttry {\n+\t\t\t\tCharset charset;\n+\t\t\t\tswitch (i) {\n+\t\t\t\tcase 0:\n+\t\t\t\t\t/*\n+\t\t\t\t\t * Try our built-in favorite. The assumption here is that\n+\t\t\t\t\t * decoding will fail if the data is not actually encoded\n+\t\t\t\t\t * using that encoder.\n+\t\t\t\t\t */\n+\t\t\t\t\tcharset = Constants.CHARSET;\n+\t\t\t\t\tbreak;\n+\t\t\t\tcase 1:\n+\t\t\t\t\t/*\n+\t\t\t\t\t * Try the suggested encoding, it might be right since it\n+\t\t\t\t\t * was provided\n+\t\t\t\t\t */\n+\t\t\t\t\tif (cs.equals(Constants.CHARSET))\n+\t\t\t\t\t\tcontinue;\n+\t\t\t\t\tb.reset();\n+\t\t\t\t\tcharset = cs;\n+\t\t\t\t\tbreak;\n+\t\t\t\tcase 2:\n+\t\t\t\t\t/*\n+\t\t\t\t\t * Try the default character set. A small group of people\n+\t\t\t\t\t * might actually use the same (or very similar) locale.\n+\t\t\t\t\t */\n+\t\t\t\t\tcharset = Charset.defaultCharset();\n+\t\t\t\t\tif (charset.equals(Constants.CHARSET))\n+\t\t\t\t\t\tcontinue;\n+\t\t\t\t\tif (charset.equals(cs))\n+\t\t\t\t\t\tcontinue;\n+\t\t\t\t\tb.reset();\n+\t\t\t\t\tbreak;\n+\t\t\t\tdefault:\n+\t\t\t\t\t/*\n+\t\t\t\t\t * This one is to make sure we do no fail. Data may look\n+\t\t\t\t\t * funny but there is nothing we can do here withou much\n+\t\t\t\t\t * more advanced guessing.\n+\t\t\t\t\t */\n+\t\t\t\t\tb.reset();\n+\t\t\t\t\tcharset = Charset.forName(\"ISO-8859-1\");\n+\t\t\t\t}\n+\t\t\t\tCharsetDecoder d = charset.newDecoder();\n+\t\t\t\td.onMalformedInput(CodingErrorAction.REPORT);\n+\t\t\t\td.onUnmappableCharacter(CodingErrorAction.REPORT);\n+\t\t\t\treturn d.decode(b).toString();\n+\t\t\t} catch (CharacterCodingException e1) {\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\t\t}\n \t}\n \n \t/**\n \t * Locate the position of the commit message body.\n-\t * \n+\t *\n \t * @param b\n \t *            buffer to scan.\n \t * @param ptr\n-- \n1.6.0.2.308.gef4a\n"},{"id":"92893","messageId":"20081013022708.GJ4856@spearce.org","threadId":"15874","inReplyTo":"1223851860-13068-5-git-send-email-robin.rosenberg@dewire.com","subject":"Re: [JGIT PATCH 4/4] Intelligent parsing of ambiguously encoded meta data.","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2008-10-13T02:27:08Z","receivedAt":"2008-10-13T02:27:08Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Robin Rosenberg <robin.rosenberg@dewire.com> wrote:\n> We cannot trust meta data to be encoded in any particular way, so we try\n> different encodings. First we try UTF-8, which is the only sane encoding\n> for non-local data, even when used in regions where eight bit legacy\n> encodings are common. The chance of mistakenly parsing non-UTF-8 data\n> as valid UTF-8 is varies from extremely low (western encodings) to low\n> for most other encodings. If the data does not look like UTF-8, we try the\n> suggested encoding. If that fails we try the user locale and finally, if\n> that fails we try ISO-8859-1, which cannot fail.\n\nHmm.  I'm concerned about the infinite loop you have here.\nIf ISO-8859-1 fails we'd be stuck here until the end of time.\nPlus its a bit ugly to read.\n\nI wonder if this is any better.  It passes your tests and is 2\nlines shorter.\n\ndiff --git a/org.spearce.jgit/src/org/spearce/jgit/util/RawParseUtils.java b/org.spearce.jgit/src/org/spearce/jgit/util/RawParseUtils.java\nindex a31734b..6c0e339 100644\n--- a/org.spearce.jgit/src/org/spearce/jgit/util/RawParseUtils.java\n+++ b/org.spearce.jgit/src/org/spearce/jgit/util/RawParseUtils.java\n@@ -42,7 +42,10 @@\n import static org.spearce.jgit.lib.ObjectChecker.encoding;\n \n import java.nio.ByteBuffer;\n+import java.nio.charset.CharacterCodingException;\n import java.nio.charset.Charset;\n+import java.nio.charset.CharsetDecoder;\n+import java.nio.charset.CodingErrorAction;\n import java.util.Arrays;\n \n import org.spearce.jgit.lib.Constants;\n@@ -376,7 +379,10 @@ public static PersonIdent parsePersonIdent(final byte[] raw, final int nameB) {\n \t}\n \n \t/**\n-\t * Decode a region of the buffer under the specified character set.\n+\t * Decode a region of the buffer under the specified character set if possible.\n+\t *\n+\t * If the byte stream cannot be decoded that way, the platform default is tried\n+\t * and if that too fails, the fail-safe ISO-8859-1 encoding is tried.\n \t * \n \t * @param cs\n \t *            character set to use when decoding the buffer.\n@@ -393,7 +399,56 @@ public static PersonIdent parsePersonIdent(final byte[] raw, final int nameB) {\n \tpublic static String decode(final Charset cs, final byte[] buffer,\n \t\t\tfinal int start, final int end) {\n \t\tfinal ByteBuffer b = ByteBuffer.wrap(buffer, start, end - start);\n-\t\treturn cs.decode(b).toString();\n+\t\tb.mark();\n+\n+\t\t// Try our built-in favorite. The assumption here is that\n+\t\t// decoding will fail if the data is not actually encoded\n+\t\t// using that encoder.\n+\t\t//\n+\t\ttry {\n+\t\t\treturn decode(b, Constants.CHARSET);\n+\t\t} catch (CharacterCodingException e) {\n+\t\t\tb.reset();\n+\t\t}\n+\n+\t\tif (!cs.equals(Constants.CHARSET)) {\n+\t\t\t// Try the suggested encoding, it might be right since it was\n+\t\t\t// provided by the caller.\n+\t\t\t//\n+\t\t\ttry {\n+\t\t\t\treturn decode(b, cs);\n+\t\t\t} catch (CharacterCodingException e) {\n+\t\t\t\tb.reset();\n+\t\t\t}\n+\t\t}\n+\n+\t\t// Try the default character set. A small group of people\n+\t\t// might actually use the same (or very similar) locale.\n+\t\t//\n+\t\tfinal Charset defcs = Charset.defaultCharset();\n+\t\tif (!defcs.equals(cs) && !defcs.equals(Constants.CHARSET)) {\n+\t\t\ttry {\n+\t\t\t\treturn decode(b, defcs);\n+\t\t\t} catch (CharacterCodingException e) {\n+\t\t\t\tb.reset();\n+\t\t\t}\n+\t\t}\n+\n+\t\t// Fall back to an ISO-8859-1 style encoding. At least all of\n+\t\t// the bytes will be present in the output.\n+\t\t//\n+\t\tfinal StringBuilder r = new StringBuilder(end - start);\n+\t\tfor (int i = start; i < end; i++)\n+\t\t\tr.append((char) (buffer[i] & 0xff));\n+\t\treturn r.toString();\n+\t}\n+\n+\tprivate static String decode(final ByteBuffer b, final Charset charset)\n+\t\t\tthrows CharacterCodingException {\n+\t\tfinal CharsetDecoder d = charset.newDecoder();\n+\t\td.onMalformedInput(CodingErrorAction.REPORT);\n+\t\td.onUnmappableCharacter(CodingErrorAction.REPORT);\n+\t\treturn d.decode(b).toString();\n \t}\n \n \t/**\n\n\n-- \nShawn.\n"},{"id":"92894","messageId":"20081013023636.GK4856@spearce.org","threadId":"15874","inReplyTo":"1223851860-13068-4-git-send-email-robin.rosenberg@dewire.com","subject":"Re: [JGIT PATCH 3/4] The git config file is case insensitive","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2008-10-13T02:36:36Z","receivedAt":"2008-10-13T02:36:36Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Robin Rosenberg <robin.rosenberg@dewire.com> wrote:\n> diff --git a/org.spearce.jgit/src/org/spearce/jgit/lib/RepositoryConfig.java b/org.spearce.jgit/src/org/spearce/jgit/lib/RepositoryConfig.java\n> index 45c2f8a..7a34cde 100644\n> --- a/org.spearce.jgit/src/org/spearce/jgit/lib/RepositoryConfig.java\n> +++ b/org.spearce.jgit/src/org/spearce/jgit/lib/RepositoryConfig.java\n> @@ -682,7 +683,12 @@ public void load() throws IOException {\n>  \n>  \tprivate void clear() {\n>  \t\tentries = new ArrayList<Entry>();\n> -\t\tbyName = new HashMap<String, Object>();\n> +\t\tbyName = new TreeMap<String, Object>(new Comparator<String>() {\n> +\n> +\t\t\tpublic int compare(String o1, String o2) {\n> +\t\t\t\treturn o1.compareToIgnoreCase(o2);\n> +\t\t\t}\n> +\t\t});\n>  \t}\n\nThis isn't necessary.  Everyone who does a get or a put against the\nbyName map already is forming a lower case key string.  I'd rather\nkeep the lookup O(1) than O(log N), especially if the code has a\nton of .toLowerCase() calls in it to normalize the keys.\n\nIf you are going to change it to a TreeMap with a custom Comparator\nthen maybe we should cleanup the code that operates on byName so it\ncan use the original input strings, instead of the .toLowerCase()\nforms.\n\nFor now I'm going to apply your patch without this one hunk.  If you\nwant to switch to a TreeMap lets also cleanup the get/put calls.\n  \n-- \nShawn.\n"},{"id":"92945","messageId":"200810131910.53891.robin.rosenberg.lists@dewire.com","threadId":"15874","inReplyTo":"20081013022708.GJ4856@spearce.org","subject":"Re: [JGIT PATCH 4/4] Intelligent parsing of ambiguously encoded meta data.","fromName":"Robin Rosenberg","fromEmail":"robin.rosenberg.lists@dewire.com","sentAt":"2008-10-13T17:10:50Z","receivedAt":"2008-10-13T17:10:50Z","isPatch":true,"sender":{"key":"robin.rosenberg@dewire.com","avatar":"https://avatars.githubusercontent.com/u/46357?v=4"},"body":"måndagen den 13 oktober 2008 04.27.08 skrev Shawn O. Pearce:\n> Robin Rosenberg <robin.rosenberg@dewire.com> wrote:\n> > We cannot trust meta data to be encoded in any particular way, so we try\n> > different encodings. First we try UTF-8, which is the only sane encoding\n> > for non-local data, even when used in regions where eight bit legacy\n> > encodings are common. The chance of mistakenly parsing non-UTF-8 data\n> > as valid UTF-8 is varies from extremely low (western encodings) to low\n> > for most other encodings. If the data does not look like UTF-8, we try the\n> > suggested encoding. If that fails we try the user locale and finally, if\n> > that fails we try ISO-8859-1, which cannot fail.\n> \n> Hmm.  I'm concerned about the infinite loop you have here.\n> If ISO-8859-1 fails we'd be stuck here until the end of time.\n> Plus its a bit ugly to read.\n> \n> I wonder if this is any better.  It passes your tests and is 2\n> lines shorter.\n\nYes. Not sure what I was thinking with the loop there... :)\n\n-- robin\n"}]}