{"thread":{"id":"17612","subject":"[JGIT] maven build fails on OS X","startedAt":"2009-02-06T15:08:15Z","lastAt":"2009-02-06T21:12:29Z","messageCount":6,"participants":["Nigel Magnay","Shawn O. Pearce","Robin Rosenberg"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"103495","messageId":"320075ff0902060708m5ec566b9g755829c25c7727d8@mail.gmail.com","threadId":"17612","inReplyTo":null,"subject":"[JGIT] maven build fails on OS X","fromName":"Nigel Magnay","fromEmail":"nigel.magnay@gmail.com","sentAt":"2009-02-06T15:08:15Z","receivedAt":"2009-02-06T15:08:15Z","isPatch":false,"sender":{"key":"nigel.magnay@gmail.com","avatar":"https://gravatar.com/avatar/d85cf38287bef3a8e4fa02358d2756d7589f8676c5eeb881ce2f6d731e4526c3?d=mp&s=160"},"body":"I don't know if this is known about - I get failures running the m2\nbuild for jgit:\n\nFailed tests:\n  testParse_implicit_mixed_encoded(org.spearce.jgit.revwalk.RevCommitParseTest)\n  testParse_explicit_bad_encoded(org.spearce.jgit.revwalk.RevCommitParseTest)\n  testParse_implicit_mixed_encoded(org.spearce.jgit.revwalk.RevTagParseTest)\n  testParse_explicit_bad_encoded(org.spearce.jgit.revwalk.RevTagParseTest)\n  testDequote_OctalAll(org.spearce.jgit.util.QuotedStringGitPathStyleTest)\n  testDequote_Latin1(org.spearce.jgit.util.QuotedStringGitPathStyleTest)\n  testDequote_RawLatin1(org.spearce.jgit.util.QuotedStringGitPathStyleTest)\n  testGetText_DiffCc(org.spearce.jgit.patch.GetTextTest)\n\nCuriously, the tests work in eclipse and from ./make_jgit.sh - I\nsuspect some environment setting with the mac's charset.\n\nI can try and fix it if it's new, but I wanted to check first..\n"},{"id":"103504","messageId":"20090206153155.GL26880@spearce.org","threadId":"17612","inReplyTo":"320075ff0902060708m5ec566b9g755829c25c7727d8@mail.gmail.com","subject":"Re: [JGIT] maven build fails on OS X","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2009-02-06T15:31:55Z","receivedAt":"2009-02-06T15:31:55Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Nigel Magnay <nigel.magnay@gmail.com> wrote:\n> I don't know if this is known about - I get failures running the m2\n> build for jgit:\n\nYup.\n \n> Failed tests:\n>   testParse_implicit_mixed_encoded(org.spearce.jgit.revwalk.RevCommitParseTest)\n>   testParse_explicit_bad_encoded(org.spearce.jgit.revwalk.RevCommitParseTest)\n>   testParse_implicit_mixed_encoded(org.spearce.jgit.revwalk.RevTagParseTest)\n>   testParse_explicit_bad_encoded(org.spearce.jgit.revwalk.RevTagParseTest)\n>   testDequote_OctalAll(org.spearce.jgit.util.QuotedStringGitPathStyleTest)\n>   testDequote_Latin1(org.spearce.jgit.util.QuotedStringGitPathStyleTest)\n>   testDequote_RawLatin1(org.spearce.jgit.util.QuotedStringGitPathStyleTest)\n>   testGetText_DiffCc(org.spearce.jgit.patch.GetTextTest)\n> \n> Curiously, the tests work in eclipse and from ./make_jgit.sh - I\n> suspect some environment setting with the mac's charset.\n\nmake_jgit.sh doesn't run tests.  But Eclipse does, and as you state,\nI have also seen these pass in Eclipse but fail from m2.\n\nWhat's even more curious is look at the surefire report files for\nthese tests.  The actual result looks correct, the expected result\nlooks wrong.  I think m2 miscompiled the Java sources for the tests\nsomehow and produced incorrect literal strings which are used for\nthe expected values in the tests.\n\nI don't see these failures on Linux, from either Eclipse or Maven.\n\n> I can try and fix it if it's new, but I wanted to check first..\n\nI've looked at it and gave up.  I don't know what Maven is doing here\non the Mac.  Maybe fresh eyes will have a better chance at fixing it.\n\nMy experience with Maven is it works about 5% of the time, and the\nother 95% of the time you have to work around it by skipping tests,\nor by writing massive blocks of XML in your pom.xml file, or by\nredesigning your entire project directory structure and revision\ncontrol system to use SVN instead of Git.\n\n-- \nShawn.\n"},{"id":"103550","messageId":"320075ff0902061208l6671c2a0m3d4b07b97025b53a@mail.gmail.com","threadId":"17612","inReplyTo":"20090206153155.GL26880@spearce.org","subject":"Re: [JGIT] maven build fails on OS X","fromName":"Nigel Magnay","fromEmail":"nigel.magnay@gmail.com","sentAt":"2009-02-06T20:08:18Z","receivedAt":"2009-02-06T20:08:18Z","isPatch":false,"sender":{"key":"nigel.magnay@gmail.com","avatar":"https://gravatar.com/avatar/d85cf38287bef3a8e4fa02358d2756d7589f8676c5eeb881ce2f6d731e4526c3?d=mp&s=160"},"body":"> I've looked at it and gave up.  I don't know what Maven is doing here\n> on the Mac.  Maybe fresh eyes will have a better chance at fixing it.\n>\n> My experience with Maven is it works about 5% of the time, and the\n> other 95% of the time you have to work around it by skipping tests,\n> or by writing massive blocks of XML in your pom.xml file, or by\n> redesigning your entire project directory structure and revision\n> control system to use SVN instead of Git.\n\nOk. It's not (just) a Maven problem as I can get the same tests to\nfail in Eclipse by setting the Console Encoding to MacRoman (which is\nthe default Charset.defaultCharset() when running a java app on OS X).\n\nDoing some digging, just on testGetText_DiffCc :- where it fails, I\nlook at the output of each doing getBytes() - so will be in MacRoman -\n(cols are byte no, exp before the replace, after the replace and the\nresult of fh.getScriptText. I get :\n\n94 32 32,32\n95 43 43,43 ++\n96 116 116,116 tt\n97 101 101,101 ee\n98 115 115,115 ss\n99 116 116,116 tt\n100 32 32,32\n101 -127 -127,-59 !!!!!!!!!!!!!!!!!!!!!!!!!!!! ??\n102 110 110,110 nn\n103 103 103,103 gg\n104 115 115,115 ss\n105 116 116,116 tt\n106 114 114,114 rr\n107 -102 -102,-10 !!!!!!!!!!!!!!!!!!!!!!!!!!!! ??\n108 109 109,109 mm\n109 10 10,10\n\n-10 (F6) is ^, and -59 (C5) is some wavy lines..\n\nHowever, I'm slightly confused - possibly because I don't know what\n/should/ be the case. In\n\n exp.replace(\"\\303\\205ngstr\\303\\266m\", \"\\u00c5ngstr\\u00f6m\")\n\nis \\303\\205 really meant to represent U+00C3 and U+0085 ? Shouldn't\nthe replace be being done on bytes rather than strings?\n"},{"id":"103553","messageId":"200902062154.34997.robin.rosenberg.lists@dewire.com","threadId":"17612","inReplyTo":"20090206153155.GL26880@spearce.org","subject":"Re: [JGIT] maven build fails on OS X","fromName":"Robin Rosenberg","fromEmail":"robin.rosenberg.lists@dewire.com","sentAt":"2009-02-06T20:54:34Z","receivedAt":"2009-02-06T20:54:34Z","isPatch":false,"sender":{"key":"robin.rosenberg@dewire.com","avatar":"https://avatars.githubusercontent.com/u/46357?v=4"},"body":"fredag 06 februari 2009 16:31:55 skrev Shawn O. Pearce:\n> Nigel Magnay <nigel.magnay@gmail.com> wrote:\n> > I don't know if this is known about - I get failures running the m2\n> > build for jgit:\n> \n> Yup.\n>  \n> > Failed tests:\n> >   testParse_implicit_mixed_encoded(org.spearce.jgit.revwalk.RevCommitParseTest)\n> >   testParse_explicit_bad_encoded(org.spearce.jgit.revwalk.RevCommitParseTest)\n> >   testParse_implicit_mixed_encoded(org.spearce.jgit.revwalk.RevTagParseTest)\n> >   testParse_explicit_bad_encoded(org.spearce.jgit.revwalk.RevTagParseTest)\n> >   testDequote_OctalAll(org.spearce.jgit.util.QuotedStringGitPathStyleTest)\n> >   testDequote_Latin1(org.spearce.jgit.util.QuotedStringGitPathStyleTest)\n> >   testDequote_RawLatin1(org.spearce.jgit.util.QuotedStringGitPathStyleTest)\n> >   testGetText_DiffCc(org.spearce.jgit.patch.GetTextTest)\n> > \n> > Curiously, the tests work in eclipse and from ./make_jgit.sh - I\n> > suspect some environment setting with the mac's charset.\n> \n> make_jgit.sh doesn't run tests.  But Eclipse does, and as you state,\n> I have also seen these pass in Eclipse but fail from m2.\n> \n> What's even more curious is look at the surefire report files for\n> these tests.  The actual result looks correct, the expected result\n> looks wrong.  I think m2 miscompiled the Java sources for the tests\n> somehow and produced incorrect literal strings which are used for\n> the expected values in the tests.\n> \n> I don't see these failures on Linux, from either Eclipse or Maven.\n\nOne thing to suspect could be the better-than-everybody-else decomposed\nunicode causing our tests to decompose too. I don't have a Mac so Ic\ncannot test my hypothesis myself, but consider\n\n\ttestDequote_RawLatin1(org.spearce.jgit.util.QuotedStringGitPathStyleTest)\n\nHere we intend to feed the dequote an ISO-Latin encoded string. Our decoder\nwill first try to decode it as UTF-8, which should fail, then we try to decode it\nas the platform default (Linear A, Hieroglyphs, MacRoman or something not\nknown to living mankind). A problem is that Git *forces* us to guess the encoding,\nand in some situations you'd need human intelligence to figure it out. Software\nwith a little more hindsight recognize this and declare UTF-8 to be the one\nand only encoding. JGit always encodes things as UTF-8 for this reason, but\nwe cannot trust the input. These test cases are supposed to prove that we\ncan guess things correctly at least some of the time.. Using \"platform\ndefault\" may be the wrong thing if it does not match what a C Git user on\nthis platform would encounter. \n\nSorry about the long explanation, but I didn't have the time to write a short\none.\n\n-- robin\n\n\n\n> \n> > I can try and fix it if it's new, but I wanted to check first..\n> \n> I've looked at it and gave up.  I don't know what Maven is doing here\n> on the Mac.  Maybe fresh eyes will have a better chance at fixing it.\n> \n> My experience with Maven is it works about 5% of the time, and the\n> other 95% of the time you have to work around it by skipping tests,\n> or by writing massive blocks of XML in your pom.xml file, or by\n> redesigning your entire project directory structure and revision\n> control system to use SVN instead of Git.\n> \n"},{"id":"103554","messageId":"200902062207.00941.robin.rosenberg@dewire.com","threadId":"17612","inReplyTo":"320075ff0902061208l6671c2a0m3d4b07b97025b53a@mail.gmail.com","subject":"Re: [JGIT] maven build fails on OS X","fromName":"Robin Rosenberg","fromEmail":"robin.rosenberg@dewire.com","sentAt":"2009-02-06T21:07:00Z","receivedAt":"2009-02-06T21:07:00Z","isPatch":false,"sender":{"key":"robin.rosenberg@dewire.com","avatar":"https://avatars.githubusercontent.com/u/46357?v=4"},"body":"fredag 06 februari 2009 21:08:18 skrev Nigel Magnay:\n> > I've looked at it and gave up.  I don't know what Maven is doing here\n> > on the Mac.  Maybe fresh eyes will have a better chance at fixing it.\n> >\n> > My experience with Maven is it works about 5% of the time, and the\n> > other 95% of the time you have to work around it by skipping tests,\n> > or by writing massive blocks of XML in your pom.xml file, or by\n> > redesigning your entire project directory structure and revision\n> > control system to use SVN instead of Git.\n> \n> Ok. It's not (just) a Maven problem as I can get the same tests to\n> fail in Eclipse by setting the Console Encoding to MacRoman (which is\n> the default Charset.defaultCharset() when running a java app on OS X).\n\nOk you found it. I just sent a reply...\n\n> However, I'm slightly confused - possibly because I don't know what\n> /should/ be the case. In\n> \n>  exp.replace(\"\\303\\205ngstr\\303\\266m\", \"\\u00c5ngstr\\u00f6m\")\n\nThe text is \"Ångström\".\n\n> is \\303\\205 really meant to represent U+00C3 and U+0085 ? Shouldn't\n> the replace be being done on bytes rather than strings?\n\nYes, but we fake the RAW strings by only really using the lower byte of\neach code point. arrays of bytes are really hard to read in source code. \nThe MacRoman is the problem. What encoding does C Git on Mac use?\nIsn't that UTF-8, in which case our use of platform default as fallback is\nwrong. Gah...\n\n-- robin\n"},{"id":"103556","messageId":"320075ff0902061312i1b2d960ax34bb992d8ba85fc5@mail.gmail.com","threadId":"17612","inReplyTo":"200902062154.34997.robin.rosenberg.lists@dewire.com","subject":"Re: [JGIT] maven build fails on OS X","fromName":"Nigel Magnay","fromEmail":"nigel.magnay@gmail.com","sentAt":"2009-02-06T21:12:29Z","receivedAt":"2009-02-06T21:12:29Z","isPatch":false,"sender":{"key":"nigel.magnay@gmail.com","avatar":"https://gravatar.com/avatar/d85cf38287bef3a8e4fa02358d2756d7589f8676c5eeb881ce2f6d731e4526c3?d=mp&s=160"},"body":"> Here we intend to feed the dequote an ISO-Latin encoded string. Our decoder\n> will first try to decode it as UTF-8, which should fail, then we try to decode it\n> as the platform default (Linear A, Hieroglyphs, MacRoman or something not\n> known to living mankind). A problem is that Git *forces* us to guess the encoding,\n> and in some situations you'd need human intelligence to figure it out. Software\n> with a little more hindsight recognize this and declare UTF-8 to be the one\n> and only encoding. JGit always encodes things as UTF-8 for this reason, but\n> we cannot trust the input. These test cases are supposed to prove that we\n> can guess things correctly at least some of the time.. Using \"platform\n> default\" may be the wrong thing if it does not match what a C Git user on\n> this platform would encounter.\n>\n\nI got your reply exaclty as I found the same thing :-). I'm not sure\nthat this is the common cause of all test failures, but it is for\ntestGetText_DiffCc :-\n\nWhen it parses the first string in extractFileLines, it passes to\nRawParseUtils.decode which, as you outlined, does a \"Try UTF-8, Try\ncaller suggestion, try Charset.defaultCharset(), then finally back out\nto ISO-8859-1.\n\nOn most platforms, this is Try UTF-8 (fail), caller suggestion (also\nUTF-8, skip), try default (UTF-8, skip) - finally parse ISO-8859-1.\n\nOn the mac, this is Try UTF-8 (fail), caller suggestion (UTF-8, skip),\ntry default (MacRoman) --> succeed.\n\nI'm less clear on what the right way to fix it is (or if it's just the\ntests that need to somehow force the system file.encoding?\n"}]}