{"thread":{"id":"24679","subject":"Encoding problem on OSX?","startedAt":"2010-08-09T13:58:57Z","lastAt":"2010-09-27T05:18:33Z","messageCount":14,"participants":["İsmail Dönmez","Jonathan Nieder","Kevin Ballard"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"147496","messageId":"AANLkTiky2uUHfOsh8rNXZPEqTLcNSZowy=Qcm+4Hjn_n@mail.gmail.com","threadId":"24679","inReplyTo":"AANLkTikh12guRxCK2Vf=WvshzX8P-fYTyu3qxYWNJ2px@mail.gmail.com","subject":"Encoding problem on OSX?","fromName":"İsmail Dönmez","fromEmail":"ismail@namtrac.org","sentAt":"2010-08-09T13:58:57Z","receivedAt":"2010-08-09T13:58:57Z","isPatch":false,"sender":{"key":"ismail@namtrac.org","avatar":null},"body":"Hi all;\n\nOn master & maint branch, t4201-shortlog.sh test 2 fails with:\n\nexpecting success:\n\ngit shortlog HEAD >log &&\nfuzz log >log.predictable &&\ntest_cmp expect.template log.predictable\n\n--- expect.template 2010-08-09 13:45:46.000000000 +0000\n+++ log.predictable 2010-08-09 13:45:46.000000000 +0000\n@@ -1,8 +1,8 @@\n A U Thor (5):\n       SUBJECT\n       SUBJECT\n-      SUBJECT\n-      SUBJECT\n+      SUBJECT𝄞s 𝄞s a very, very long f𝄞rst l𝄞ne for the comm𝄞t\nmessage to see 𝄞f 𝄞t 𝄞s wrapped correctly\n+      SUBJECT????s ????s a very, very long f????rst l????ne for the\ncomm????t message to see ????f ????t ????s wrapped correctly\n       SUBJECT\n\nI am not sure if this is a known problem so I am reporting it.\n\nRegards,\nismail\n"},{"id":"147587","messageId":"20100809234620.GA6418@burratino","threadId":"24679","inReplyTo":"AANLkTiky2uUHfOsh8rNXZPEqTLcNSZowy=Qcm+4Hjn_n@mail.gmail.com","subject":"Re: Encoding problem on OSX?","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2010-08-09T23:46:21Z","receivedAt":"2010-08-09T23:46:21Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"İsmail Dönmez wrote:\n\n> git shortlog HEAD >log &&\n> fuzz log >log.predictable &&\n> test_cmp expect.template log.predictable\n> \n> --- expect.template 2010-08-09 13:45:46.000000000 +0000\n> +++ log.predictable 2010-08-09 13:45:46.000000000 +0000\n> @@ -1,8 +1,8 @@\n>  A U Thor (5):\n>        SUBJECT\n>        SUBJECT\n> -      SUBJECT\n> -      SUBJECT\n> +      SUBJECT𝄞s 𝄞s a very, very long f𝄞rst l𝄞ne for the comm𝄞t\n> message to see 𝄞f 𝄞t 𝄞s wrapped correctly\n> +      SUBJECT????s ????s a very, very long f????rst l????ne for the\n> comm????t message to see ????f ????t ????s wrapped correctly\n>        SUBJECT\n\nVery interesting; thanks for a report.\n\nFrom the definition of fuzz(), it looks like\n\n\tsed \"\n\t\t\ts/$_x40/OBJECT_NAME/g\n\t\t\ts/$_x05/OBJID/g\n\t\t\ts/^ \\{6\\}[CTa].*/ SUBJECT/g\n\t\t\ts/^ \\{8\\}[^ ].*/ CONTINUATION/g\n\t\t\" log.fuzzy\n\nfailed to completely match the fourth and five lines of the shortlog:\n\n\tA U Thor (5):\n\t Test\n\t This is a very, very long first[etc]\n\t Th𝄞s 𝄞s a very, very long f𝄞rst[etc]\n\t Ths s a...\n\nCould you confirm this? What does\n\n\tlocale\n\tprintf 'Th\\360\\235\\204\\236s\\n' | sed 's/.*//g'\n\nprint?\n\nJonathan\n"},{"id":"147608","messageId":"AANLkTikqpYO9GG9PZD_==PyUSGRV8=L5LrZ0RVsESuXN@mail.gmail.com","threadId":"24679","inReplyTo":"20100809234620.GA6418@burratino","subject":"Re: Encoding problem on OSX?","fromName":"İsmail Dönmez","fromEmail":"ismail@namtrac.org","sentAt":"2010-08-10T05:52:12Z","receivedAt":"2010-08-10T05:52:12Z","isPatch":false,"sender":{"key":"ismail@namtrac.org","avatar":null},"body":"Hi;\n\nOn Tue, Aug 10, 2010 at 2:46 AM, Jonathan Nieder wrote:\n>\n>  locale\n>        printf 'Th\\360\\235\\204\\236s\\n' | sed 's/.*//g\n\n[ismail@havana][08:50:45]\n[~]> locale\nLANG=\nLC_COLLATE=\"en_US.UTF-8\"\nLC_CTYPE=\"en_US.UTF-8\"\nLC_MESSAGES=\"en_US.UTF-8\"\nLC_MONETARY=\"en_US.UTF-8\"\nLC_NUMERIC=\"en_US.UTF-8\"\nLC_TIME=\"en_US.UTF-8\"\nLC_ALL=\"en_US.UTF-8\"\n\n[ismail@havana][08:51:00]\n[~]> printf 'Th\\360\\235\\204\\236s\\n' | sed 's/.*//g'\n\n[ismail@havana][08:51:06]\n[~]>\n"},{"id":"147728","messageId":"20100811075503.GD5450@burratino","threadId":"24679","inReplyTo":"AANLkTikqpYO9GG9PZD_==PyUSGRV8=L5LrZ0RVsESuXN@mail.gmail.com","subject":"Re: Encoding problem on OSX?","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2010-08-11T07:55:03Z","receivedAt":"2010-08-11T07:55:03Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"İsmail Dönmez wrote:\n\n> [~]> printf 'Th\\360\\235\\204\\236s\\n' | sed 's/.*//g'\n> \n> [ismail@havana][08:51:06]\n> [~]>\n\nThanks for checking. So sed is not completely broken. Could you try\n\n sh t4201-shortlog.sh\n cd \"trash directory.t4201-shortlog\"\n git log\n cat \"trash directory.t4201-shortlog/log\"\n\n?\n"},{"id":"147730","messageId":"AANLkTin1bspb+bzt6P1Fz7Q11=F2ZBftC19XQuQqHL+B@mail.gmail.com","threadId":"24679","inReplyTo":"20100811075503.GD5450@burratino","subject":"Re: Encoding problem on OSX?","fromName":"İsmail Dönmez","fromEmail":"ismail@namtrac.org","sentAt":"2010-08-11T08:20:39Z","receivedAt":"2010-08-11T08:20:39Z","isPatch":false,"sender":{"key":"ismail@namtrac.org","avatar":null},"body":"Hi;\n\nOn Wed, Aug 11, 2010 at 10:55 AM, Jonathan Nieder wrote:\n> İsmail Dönmez wrote:\n>\n>> [~]> printf 'Th\\360\\235\\204\\236s\\n' | sed 's/.*//g'\n>>\n>> [ismail@havana][08:51:06]\n>> [~]>\n>\n> Thanks for checking.  So sed is not completely broken.  Could you try\n>\n>  sh t4201-shortlog.sh\n>  cd \"trash directory.t4201-shortlog\"\n>  git log\n>  cat \"trash directory.t4201-shortlog/log\"\n\nFirst of all note that this is not Mac's default sed but instead GNU sed:\n\nGNU sed version 4.2.1\nCopyright (C) 2009 Free Software Foundation, Inc.\n\nNow the output of what you requested;\n\n[~/Sources/git/t]> sh t4201-shortlog.sh\nok 1 - setup\nnot ok - 2 default output format\n#\t\n#\t\tgit shortlog HEAD >log &&\n#\t\tfuzz log >log.predictable &&\n#\t\ttest_cmp expect.template log.predictable\n#\t\nok 3 - pretty format\nok 4 - --abbrev\nok 5 - output from user-defined format is re-wrapped\nok 6 - shortlog wrapping\nok 7 - shortlog from non-git directory\nok 8 - shortlog encoding\n# failed 1 among 8 test(s)\n1..8\n[ismail@havana][11:18:24]\n[~/Sources/git/t]> cd \"trash directory.t4201-shortlog\"\n[ismail@havana][11:18:33]\n[~/Sources/git/t/trash directory.t4201-shortlog]> git log\ncommit ef6c19b4846d6a3e41f9a3ce746a3bffae653c17\nAuthor: Jöhännës \"Dschö\" Schindëlin \nDate: Wed Aug 11 08:18:24 2010 +0000\n\n set a1 to 3 and some non-ASCII chars: áæï\n\ncommit d7c0787d081716755e2863f612d171846f503d4f\nAuthor: Jöhännës \"Dschö\" Schindëlin \nDate: Wed Aug 11 08:18:24 2010 +0000\n\n set a1 to 2 and some non-ASCII chars: Äßø\n\ncommit 7e9687adfe33f5d2050f0fc4ab5004f324d3559f\nAuthor: A U Thor \nDate: Wed Aug 11 08:18:24 2010 +0000\n\n Test\n\n[~/Sources/git/t/trash directory.t4201-shortlog]> cat log\ncommit 5fc75f5794d1cd8575fc3e2e86f9c0e1aa31723e\nAuthor: Someone else \nDate: Wed Aug 11 08:18:24 2010 +0000\n\n Commit by someone else\n\ncommit 0f5955f471a9d882b0e869752614b5123af19da3\nAuthor: A U Thor \nDate: Wed Aug 11 08:18:24 2010 +0000\n\n a\t\t\t\t\t\t\t\t12\t34\t56\t78\n\ncommit 0bb7d083233c266d9051b283913bd83000c9001f\nAuthor: A U Thor \nDate: Wed Aug 11 08:18:24 2010 +0000\n\n Th????s ????s a very, very long f????rst l????ne for the comm????t\nmessage to see ????f ????t ????s wrapped correctly\n\ncommit 03a5a848c658751c51925127820491bf2a94a752\nAuthor: A U Thor \nDate: Wed Aug 11 08:18:24 2010 +0000\n\n Th𝄞s 𝄞s a very, very long f𝄞rst l𝄞ne for the comm𝄞t message\nto see 𝄞f 𝄞t 𝄞s wrapped correctly\n\ncommit fdfc106190118f705dee70b56930764007353922\nAuthor: A U Thor \nDate: Wed Aug 11 08:18:24 2010 +0000\n\n This is a very, very long first line for the commit message to see\nif it is wrapped correctly\n\ncommit 7e9687adfe33f5d2050f0fc4ab5004f324d3559f\nAuthor: A U Thor \nDate: Wed Aug 11 08:18:24 2010 +0000\n\n Test\n"},{"id":"147731","messageId":"20100811082953.GA16456@burratino","threadId":"24679","inReplyTo":"AANLkTin1bspb+bzt6P1Fz7Q11=F2ZBftC19XQuQqHL+B@mail.gmail.com","subject":"Re: Encoding problem on OSX?","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2010-08-11T08:29:53Z","receivedAt":"2010-08-11T08:29:53Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"İsmail Dönmez wrote:\n\n> [~/Sources/git/t]> sh t4201-shortlog.sh\n> ok 1 - setup\n> not ok - 2 default output format\n> #\t\n> #\t\tgit shortlog HEAD >log &&\n> #\t\tfuzz log >log.predictable &&\n> #\t\ttest_cmp expect.template log.predictable\n> #\t\n> ok 3 - pretty format\n\nOops, my bad.\n\n\tsh t4201-shortlog.sh --immediate\n\tcat \"trash directory.t4201-shortlog/log\"\n\nis what I meant. The idea is to get the log that that log.predictable\nis based on, by fetching the log from immediately after the failing test.\n\nSorry for the trouble,\nJonathan\n"},{"id":"147733","messageId":"AANLkTim5Z7M0CypudaGtb2UuON5ajf3Xn=hz6oAJwv8w@mail.gmail.com","threadId":"24679","inReplyTo":"20100811082953.GA16456@burratino","subject":"Re: Encoding problem on OSX?","fromName":"İsmail Dönmez","fromEmail":"ismail@namtrac.org","sentAt":"2010-08-11T08:33:35Z","receivedAt":"2010-08-11T08:33:35Z","isPatch":false,"sender":{"key":"ismail@namtrac.org","avatar":null},"body":"On Wed, Aug 11, 2010 at 11:29 AM, Jonathan Nieder wrote:\n> İsmail Dönmez wrote:\n>\n>> [~/Sources/git/t]>  sh t4201-shortlog.sh\n>> ok 1 - setup\n>> not ok - 2 default output format\n>> #\n>> #             git shortlog HEAD >log &&\n>> #             fuzz log >log.predictable &&\n>> #             test_cmp expect.template log.predictable\n>> #\n>> ok 3 - pretty format\n>\n> Oops, my bad.\n>\n>        sh t4201-shortlog.sh --immediate\n>        cat \"trash directory.t4201-shortlog/log\"\n>\n> is what I meant.  The idea is to get the log that that log.predictable\n> is based on, by fetching the log from immediately after the failing test.\n\nOk here we go;\n\n\n[~/Sources/git/t]> sh t4201-shortlog.sh --immediate\nok 1 - setup\nnot ok - 2 default output format\n#\t\n#\t\tgit shortlog HEAD >log &&\n#\t\tfuzz log >log.predictable &&\n#\t\ttest_cmp expect.template log.predictable\n#\t\n[ismail@havana][11:32:29]\n[~/Sources/git/t]> cat \"trash directory.t4201-shortlog/log\"\nA U Thor (5):\n Test\n This is a very, very long first line for the commit message to\nsee if it is wrapped correctly\n Th𝄞s 𝄞s a very, very long f𝄞rst l𝄞ne for the comm𝄞t message\nto see 𝄞f 𝄞t 𝄞s wrapped correctly\n Th????s ????s a very, very long f????rst l????ne for the\ncomm????t message to see ????f ????t ????s wrapped correctly\n a\t\t\t\t\t\t\t\t12\t34\t56\t78\n\nSomeone else (1):\n Commit by someone else\n"},{"id":"147743","messageId":"20100811084443.GF16495@burratino","threadId":"24679","inReplyTo":"AANLkTim5Z7M0CypudaGtb2UuON5ajf3Xn=hz6oAJwv8w@mail.gmail.com","subject":"Re: Encoding problem on OSX?","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2010-08-11T08:44:43Z","receivedAt":"2010-08-11T08:44:43Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"İsmail Dönmez wrote:\n> On Wed, Aug 11, 2010 at 11:29 AM, Jonathan Nieder wrote:\n\n>>        sh t4201-shortlog.sh --immediate\n>>        cat \"trash directory.t4201-shortlog/log\"\n>>\n>> is what I meant.  The idea is to get the log that that log.predictable\n>> is based on, by fetching the log from immediately after the failing test.\n>\n> Ok here we go;\n\nOkay, I’m stymied. It *looks* like a sed bug even if a quick\ntest did not catch it in the act.\n\nI guess the last thing to try is\n\n\tsed \"s/^ \\{6\\}[CTa].*/ SUBJECT/g\" <\"trash directory.t4201-shortlog/log\"\n\nbecause then you would have a test case to report to your sed\nsupplier.\n\nHopefully someone else with Mac OS X can reproduce this. \n\nThanks again.\n"},{"id":"147744","messageId":"AANLkTinS0BeHWNJ0mCtTbVRz=7FBwE0_OTKDfS2gPXdN@mail.gmail.com","threadId":"24679","inReplyTo":"20100811084443.GF16495@burratino","subject":"Re: Encoding problem on OSX?","fromName":"İsmail Dönmez","fromEmail":"ismail@namtrac.org","sentAt":"2010-08-11T08:47:03Z","receivedAt":"2010-08-11T08:47:03Z","isPatch":false,"sender":{"key":"ismail@namtrac.org","avatar":null},"body":"On Wed, Aug 11, 2010 at 11:44 AM, Jonathan Nieder wrote:\n>        sed \"s/^ \\{6\\}[CTa].*/      SUBJECT/g\" <\"trash directory.t4201-shortlog/log\"\n>\n\nA U Thor (5):\n SUBJECT\n SUBJECT\n SUBJECT\n SUBJECT????s ????s a very, very long f????rst l????ne for the\ncomm????t message to see ????f ????t ????s wrapped correctly\n SUBJECT\n\nSomeone else (1):\n SUBJECT\n\nI will try updating my sed, thanks!\n\nRegards,\nismail\n"},{"id":"147745","messageId":"AANLkTi=sBNmMoZmqO_-AFNO6bVqtJmXbdXYUf7-bKars@mail.gmail.com","threadId":"24679","inReplyTo":"AANLkTinS0BeHWNJ0mCtTbVRz=7FBwE0_OTKDfS2gPXdN@mail.gmail.com","subject":"Re: Encoding problem on OSX?","fromName":"İsmail Dönmez","fromEmail":"ismail@namtrac.org","sentAt":"2010-08-11T09:01:46Z","receivedAt":"2010-08-11T09:01:46Z","isPatch":false,"sender":{"key":"ismail@namtrac.org","avatar":null},"body":"Hi again;\n\nOn Wed, Aug 11, 2010 at 11:47 AM, İsmail Dönmez wrote:\n> On Wed, Aug 11, 2010 at 11:44 AM, Jonathan Nieder wrote:\n>>        sed \"s/^ \\{6\\}[CTa].*/      SUBJECT/g\" <\"trash directory.t4201-shortlog/log\"\n>>\n>\n> A U Thor (5):\n>      SUBJECT\n>      SUBJECT\n>      SUBJECT\n>      SUBJECT????s ????s a very, very long f????rst l????ne for the\n> comm????t message to see ????f ????t ????s wrapped correctly\n>      SUBJECT\n>\n> Someone else (1):\n>      SUBJECT\n>\n> I will try updating my sed, thanks!\n\nDowngrading my sed to v 4.1.5 fixed the issue. Thanks for your help!\n\nRegards,\nismail\n"},{"id":"147748","messageId":"20100811092311.GA16901@burratino","threadId":"24679","inReplyTo":"AANLkTi=sBNmMoZmqO_-AFNO6bVqtJmXbdXYUf7-bKars@mail.gmail.com","subject":"Re: Encoding problem on OSX?","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2010-08-11T09:23:11Z","receivedAt":"2010-08-11T09:23:11Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"İsmail Dönmez wrote:\n\n> Downgrading my sed to v 4.1.5 fixed the issue. Thanks for your help!\n\nI just read BUGS in the sed distribution. Strangely enough the above seems to\nbe correct behavior:\n\n Another common localization-related problem happens if your input stream\n includes invalid multibyte sequences. POSIX mandates that such\n sequences are _not_ matched by `.', so that `s/.*//' will not clear\n pattern space as you would expect. In fact, there is no way to clear\n sed's buffers in the middle of the script in most multibyte locales\n (including UTF-8 locales). For this reason, GNU sed provides a `z'\n command (for `zap') as an extension.\n\nHowever there is still a sed bug as far as I can tell, since in the\ntest suite, LC_ALL is set to C, and using the C locale is the\nsuggested workaround in the GNU sed docs. This explains where my\nfirst suggested diagnostic messed up: presumably\n\n printf 'Th\\360\\235\\204\\236s\\n' | LC_ALL=C sed \"s/.*//\"\n\nwould print\n\n s\n\nand\n\n printf 'Th\\370\\235\\204\\236s\\n' | sed \"s/.*//\"\n\nwould print\n\n ????s\n\nwith your copy of sed 4.2.1.\n\nWell, I learned something new today. Still thinking over how to fix\nthis in the test suite. Thanks again.\n"},{"id":"151777","messageId":"20100927023122.GA8802@burratino","threadId":"24679","inReplyTo":"20100811092311.GA16901@burratino","subject":"Re: Encoding problem on OSX?","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2010-09-27T02:31:22Z","receivedAt":"2010-09-27T02:31:22Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi again,\n\nİsmail Dönmez wrote:\n\n> Downgrading my sed to v 4.1.5 fixed the issue.\n\nThis is nicely explained here:\n\n https://www.opengroup.org/sophocles/show_mail.tpl?source=L&listname=austin-group-l&id=14595\n\nIt looks to be a Mac OS libc misfeature. Could you two lobby Apple to\nget this fixed? :)\n\nThanks again for the reports.\n"},{"id":"151789","messageId":"9A6F91A4-796B-45F1-B2BD-1367AB5A1679@sb.org","threadId":"24679","inReplyTo":"20100927023122.GA8802@burratino","subject":"Re: Encoding problem on OSX?","fromName":"Kevin Ballard","fromEmail":"kevin@sb.org","sentAt":"2010-09-27T05:15:58Z","receivedAt":"2010-09-27T05:15:58Z","isPatch":false,"sender":{"key":"kevin@sb.org","avatar":"https://avatars.githubusercontent.com/u/714?v=4"},"body":"On Sep 26, 2010, at 7:31 PM, Jonathan Nieder wrote:\n\n> Hi again,\n> \n> İsmail Dönmez wrote:\n> \n>> Downgrading my sed to v 4.1.5 fixed the issue.\n> \n> This is nicely explained here:\n> \n> https://www.opengroup.org/sophocles/show_mail.tpl?source=L&listname=austin-group-l&id=14595\n> \n> It looks to be a Mac OS libc misfeature. Could you two lobby Apple to\n> get this fixed? :)\n\nFWIW, /usr/bin/sed on Mac OS X 10.6 doesn't seem to be having a problem. t4201-shortlog.sh passes all tests on my machine.\n\n-Kevin Ballard"},{"id":"151790","messageId":"AANLkTim=XbrSourufbNGo39_CTnXO8QPVOjt99fJedwE@mail.gmail.com","threadId":"24679","inReplyTo":"9A6F91A4-796B-45F1-B2BD-1367AB5A1679@sb.org","subject":"Re: Encoding problem on OSX?","fromName":"İsmail Dönmez","fromEmail":"ismail@namtrac.org","sentAt":"2010-09-27T05:18:33Z","receivedAt":"2010-09-27T05:18:33Z","isPatch":false,"sender":{"key":"ismail@namtrac.org","avatar":null},"body":"On Mon, Sep 27, 2010 at 8:15 AM, Kevin Ballard wrote:\n> On Sep 26, 2010, at 7:31 PM, Jonathan Nieder wrote:\n>\n>> Hi again,\n>>\n>> İsmail Dönmez wrote:\n>>\n>>> Downgrading my sed to v 4.1.5 fixed the issue.\n>>\n>> This is nicely explained here:\n>>\n>> https://www.opengroup.org/sophocles/show_mail.tpl?source=L&listname=austin-group-l&id=14595\n>>\n>> It looks to be a Mac OS libc misfeature.  Could you two lobby Apple to\n>> get this fixed? :)\n>\n> FWIW, /usr/bin/sed on Mac OS X 10.6 doesn't seem to be having a problem. t4201-shortlog.sh passes all tests on my machine.\n\nYes the problem was with GNU sed on OSX 10.6\n\nRegards,\nismail\n"}]}