{"thread":{"id":"44678","subject":"[PATCH 2/3] update_unicode.sh: remove the plane filters","startedAt":"2016-12-11T23:34:43Z","lastAt":"2016-12-13T06:42:29Z","messageCount":11,"participants":["Beat Bolli","Torsten Bögershausen","Junio C Hamano"],"isPatch":true,"patchVersion":1,"patchTotal":3},"messages":[{"id":"307474","messageId":"1481499265-18361-2-git-send-email-dev+git@drbeat.li","threadId":"44678","inReplyTo":"1481499265-18361-1-git-send-email-dev+git@drbeat.li","subject":"[PATCH 2/3] update_unicode.sh: remove the plane filters","fromName":"Beat Bolli","fromEmail":"dev+git@drbeat.li","sentAt":"2016-12-11T23:34:24Z","receivedAt":"2016-12-11T23:34:43Z","isPatch":true,"sender":{"key":"dev+git@drbeat.li","avatar":"https://avatars.githubusercontent.com/u/21444?v=4"},"body":"The uniset upstream has accepted my patches that eliminate the Unicode\nplane offsets from the output in '--32' mode.\n\nRemove the corresponding filter in update_unicode.sh.\n\nSigned-off-by: Beat Bolli <dev+git@drbeat.li>\n---\n update_unicode.sh | 5 ++---\n 1 file changed, 2 insertions(+), 3 deletions(-)\n\ndiff --git a/update_unicode.sh b/update_unicode.sh\nindex 9ca7d8b..e595bf8 100755\n--- a/update_unicode.sh\n+++ b/update_unicode.sh\n@@ -31,11 +31,10 @@ fi &&\n \tUNICODE_DIR=. && export UNICODE_DIR &&\n \tcat >$UNICODEWIDTH_H <<-EOF\n \tstatic const struct interval zero_width[] = {\n-\t\t$(uniset/uniset --32 cat:Me,Mn,Cf + U+1160..U+11FF - U+00AD |\n-\t\t  grep -v plane)\n+\t\t$(uniset/uniset --32 cat:Me,Mn,Cf + U+1160..U+11FF - U+00AD)\n \t};\n \tstatic const struct interval double_width[] = {\n-\t\t$(uniset/uniset --32 eaw:F,W | grep -v plane)\n+\t\t$(uniset/uniset --32 eaw:F,W)\n \t};\n \tEOF\n )\n-- \n2.7.2\n"},{"id":"307475","messageId":"1481499265-18361-3-git-send-email-dev+git@drbeat.li","threadId":"44678","inReplyTo":"1481499265-18361-1-git-send-email-dev+git@drbeat.li","subject":"[PATCH 3/3] update_unicode.sh: restore hexadecimal output","fromName":"Beat Bolli","fromEmail":"dev+git@drbeat.li","sentAt":"2016-12-11T23:34:25Z","receivedAt":"2016-12-11T23:34:47Z","isPatch":true,"sender":{"key":"dev+git@drbeat.li","avatar":"https://avatars.githubusercontent.com/u/21444?v=4"},"body":"The uniset upstream has decided that decimal numbers are The True Way, so\nlet's convert them back to the usual format that's closer to the U+nnnn\nstandard.\n\nThe generated unicode_widths.h file again looks exactly the same as two\ncommits ago.\n\nSigned-off-by: Beat Bolli <dev+git@drbeat.li>\n---\n update_unicode.sh | 8 +++++++-\n 1 file changed, 7 insertions(+), 1 deletion(-)\n\ndiff --git a/update_unicode.sh b/update_unicode.sh\nindex e595bf8..d7720d5 100755\n--- a/update_unicode.sh\n+++ b/update_unicode.sh\n@@ -5,6 +5,12 @@\n #Mn Nonspacing_Mark a nonspacing combining mark (zero advance width)\n #Cf Format          a format control character\n #\n+\n+dec_to_hex() {\n+\t# convert any decimal numbers to 4-digit hex\n+\tperl -pe 's/(\\d+)/sprintf(\"0x%04X\", $1)/ge'\n+}\n+\n UNICODEWIDTH_H=../unicode_width.h\n if ! test -d unicode; then\n \tmkdir unicode\n@@ -29,7 +35,7 @@ fi &&\n \t\tmake\n \t) &&\n \tUNICODE_DIR=. && export UNICODE_DIR &&\n-\tcat >$UNICODEWIDTH_H <<-EOF\n+\tdec_to_hex >$UNICODEWIDTH_H <<-EOF\n \tstatic const struct interval zero_width[] = {\n \t\t$(uniset/uniset --32 cat:Me,Mn,Cf + U+1160..U+11FF - U+00AD)\n \t};\n-- \n2.7.2\n"},{"id":"307476","messageId":"1481499265-18361-1-git-send-email-dev+git@drbeat.li","threadId":"44678","inReplyTo":null,"subject":"[PATCH 1/3] update_unicode.sh: update the uniset repo if it exists","fromName":"Beat Bolli","fromEmail":"dev+git@drbeat.li","sentAt":"2016-12-11T23:34:23Z","receivedAt":"2016-12-11T23:34:51Z","isPatch":true,"sender":{"key":"dev+git@drbeat.li","avatar":"https://avatars.githubusercontent.com/u/21444?v=4"},"body":"We need to track the new commits in uniset, otherwise their and our code\nget out of sync.\n\nSigned-off-by: Beat Bolli <dev+git@drbeat.li>\n---\n\nJunio, these go on top of my bb/unicode-9.0 branch, please.\n\nThanks!\n\n update_unicode.sh | 5 +++++\n 1 file changed, 5 insertions(+)\n\ndiff --git a/update_unicode.sh b/update_unicode.sh\nindex 4c1ec8d..9ca7d8b 100755\n--- a/update_unicode.sh\n+++ b/update_unicode.sh\n@@ -14,6 +14,11 @@ fi &&\n \t\thttp://www.unicode.org/Public/UCD/latest/ucd/EastAsianWidth.txt &&\n \tif ! test -d uniset; then\n \t\tgit clone https://github.com/depp/uniset.git\n+\telse\n+\t(\n+\t\tcd uniset &&\n+\t\tgit pull\n+\t)\n \tfi &&\n \t(\n \t\tcd uniset &&\n-- \n2.7.2\n"},{"id":"307487","messageId":"64bc846c-0304-dd7b-73bf-a6c3a4135381@web.de","threadId":"44678","inReplyTo":"1481499265-18361-1-git-send-email-dev+git@drbeat.li","subject":"Re: [PATCH 1/3] update_unicode.sh: update the uniset repo if it exists","fromName":"Torsten Bögershausen","fromEmail":"tboegi@web.de","sentAt":"2016-12-12T05:53:16Z","receivedAt":"2016-12-12T05:53:31Z","isPatch":true,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"On 2016-12-12 00:34, Beat Bolli wrote:\n> We need to track the new commits in uniset, otherwise their and our code\n> get out of sync.\n> \n> Signed-off-by: Beat Bolli <dev+git@drbeat.li>\n> ---\n> \n> Junio, these go on top of my bb/unicode-9.0 branch, please.\n> \n> Thanks!\n> \n>  update_unicode.sh | 5 +++++\n>  1 file changed, 5 insertions(+)\n> \n> diff --git a/update_unicode.sh b/update_unicode.sh\n> index 4c1ec8d..9ca7d8b 100755\n> --- a/update_unicode.sh\n> +++ b/update_unicode.sh\n> @@ -14,6 +14,11 @@ fi &&\n>  \t\thttp://www.unicode.org/Public/UCD/latest/ucd/EastAsianWidth.txt &&\n>  \tif ! test -d uniset; then\n>  \t\tgit clone https://github.com/depp/uniset.git\n> +\telse\n> +\t(\n> +\t\tcd uniset &&\n> +\t\tgit pull\nIf upstream has accepted your patches, that's nice.\n\nMinor question, especially to the next commit:\nShould we make sure to checkout the exact version, which has been tested?\nIn this case  cb97792880625e24a9f581412d03659091a0e54f\n\nAnd this is for both a fresh clone and the git pull\nneeds to be replaced by\ngit fetch && git checkout cb97792880625e24a9f581412d03659091a0e54f\n\n\n(Which of course is a shell variable\n\n"},{"id":"307491","messageId":"c96d013c38df7737cfd551a0fce87314@drbeat.li","threadId":"44678","inReplyTo":"64bc846c-0304-dd7b-73bf-a6c3a4135381@web.de","subject":"Re: [PATCH 1/3] update_unicode.sh: update the uniset repo if it exists","fromName":"Beat Bolli","fromEmail":"dev+git@drbeat.li","sentAt":"2016-12-12T08:54:21Z","receivedAt":"2016-12-12T08:54:29Z","isPatch":true,"sender":{"key":"dev+git@drbeat.li","avatar":"https://avatars.githubusercontent.com/u/21444?v=4"},"body":"On 2016-12-12 06:53, Torsten Bögershausen wrote:\n> On 2016-12-12 00:34, Beat Bolli wrote:\n>> We need to track the new commits in uniset, otherwise their and our \n>> code\n>> get out of sync.\n>> \n>> Signed-off-by: Beat Bolli <dev+git@drbeat.li>\n>> ---\n>> \n>> Junio, these go on top of my bb/unicode-9.0 branch, please.\n>> \n>> Thanks!\n>> \n>>  update_unicode.sh | 5 +++++\n>>  1 file changed, 5 insertions(+)\n>> \n>> diff --git a/update_unicode.sh b/update_unicode.sh\n>> index 4c1ec8d..9ca7d8b 100755\n>> --- a/update_unicode.sh\n>> +++ b/update_unicode.sh\n>> @@ -14,6 +14,11 @@ fi &&\n>>  \t\thttp://www.unicode.org/Public/UCD/latest/ucd/EastAsianWidth.txt &&\n>>  \tif ! test -d uniset; then\n>>  \t\tgit clone https://github.com/depp/uniset.git\n>> +\telse\n>> +\t(\n>> +\t\tcd uniset &&\n>> +\t\tgit pull\n> If upstream has accepted your patches, that's nice.\n> \n> Minor question, especially to the next commit:\n> Should we make sure to checkout the exact version, which has been \n> tested?\n> In this case  cb97792880625e24a9f581412d03659091a0e54f\n> \n> And this is for both a fresh clone and the git pull\n> needs to be replaced by\n> git fetch && git checkout cb97792880625e24a9f581412d03659091a0e54f\n> \n> \n> (Which of course is a shell variable)\n\nI was actually wondering what the policy was for adding submodules to \nthe Git repo,\nbut then decided against it. Another option would be to fork uniset on \nGitHub and\njust let it stay on a working commit.\n\nJunio, what's your stance on this?\n\nBeat\n"},{"id":"307514","messageId":"ca10a51a-0fab-e4a4-8d7d-035673af4c06@web.de","threadId":"44678","inReplyTo":"c96d013c38df7737cfd551a0fce87314@drbeat.li","subject":"Re: [PATCH 1/3] update_unicode.sh: update the uniset repo if it exists","fromName":"Torsten Bögershausen","fromEmail":"tboegi@web.de","sentAt":"2016-12-12T18:12:38Z","receivedAt":"2016-12-12T18:12:55Z","isPatch":true,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"\n>> Minor question, especially to the next commit:\n>> Should we make sure to checkout the exact version, which has been tested?\n>> In this case  cb97792880625e24a9f581412d03659091a0e54f\n>>\n>> And this is for both a fresh clone and the git pull\n>> needs to be replaced by\n>> git fetch && git checkout cb97792880625e24a9f581412d03659091a0e54f\n>>\n>>\n>> (Which of course is a shell variable)\n> \n> I was actually wondering what the policy was for adding submodules to the Git repo,\n> but then decided against it. Another option would be to fork uniset on GitHub and\n> just let it stay on a working commit.\n> \n> Junio, what's your stance on this?\n> \n> Beat\n\nIf I run  ./update_unicode.sh on the latest master of   https://github.com/depp/uniset.git ,\ncommit  a5fac4a091857dd5429cc2d, I get a diff in  unicode_width.h like this:\n\n-{ 0x0300, 0x036F },\n\n+{ 768, 879 },\n\nIOW, all hex values are printed as decimal values.\nNot a problem for the compiler, but for the human\nto check the unicode tables.\n\nSo I think we should \"pin\" the version of uniset.\n\n"},{"id":"307520","messageId":"xmqqr35dm203.fsf@gitster.mtv.corp.google.com","threadId":"44678","inReplyTo":"ca10a51a-0fab-e4a4-8d7d-035673af4c06@web.de","subject":"Re: [PATCH 1/3] update_unicode.sh: update the uniset repo if it exists","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-12-12T18:33:00Z","receivedAt":"2016-12-12T18:33:08Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Torsten Bögershausen <tboegi@web.de> writes:\n\n> If I run ./update_unicode.sh on the latest master of\n> https://github.com/depp/uniset.git , commit\n> a5fac4a091857dd5429cc2d, I get a diff in unicode_width.h like\n> this:\n>\n> -{ 0x0300, 0x036F },\n>\n> +{ 768, 879 },\n>\n> IOW, all hex values are printed as decimal values.\n> Not a problem for the compiler, but for the human\n> to check the unicode tables.\n>\n> So I think we should \"pin\" the version of uniset.\n\nSure, and I'd rather see the update-unicode.sh script moved\nsomewhere in contrib/ while at it.  Those who are interested in\nkeeping up with the unicode standard are tiny minority of the\ndeveloper population, and most of us would treat the built width\ntable as the source (after all, that is what we ship).\n\nTo be bluntly honest, I'd rather not to see \"update-unicode.sh\"\ndownload and build uniset at all.  It's as if po/ hierarchy shipping\nwith its own script to download and build msgmerge--that's madness.\nNeedless to say, shipping the sources for uniset embedded in our\nproject tree (either as a snapshot-fork or as a submodule) is even\nworse.  Those who want to muck with po/ are expected to have\nmsgmerge and friends.  Why not expect the same for those who want to\nupdate the unicode width table?\n\nI'd rather see a written instruction telling which snapshot to get\nand from where to build and place on their $PATH in the README file,\nsitting next to the update-unicode.sh script in contrib/uniwidth/\ndirectory, for those who are interested in building the width table\n\"from the source\", and the update-unicode.sh script to assume that\nuniset is available.\n"},{"id":"307533","messageId":"f068b579-fb1a-447b-283b-006686b90861@drbeat.li","threadId":"44678","inReplyTo":"ca10a51a-0fab-e4a4-8d7d-035673af4c06@web.de","subject":"Re: [PATCH 1/3] update_unicode.sh: update the uniset repo if it exists","fromName":"Beat Bolli","fromEmail":"dev+git@drbeat.li","sentAt":"2016-12-12T19:24:20Z","receivedAt":"2016-12-12T19:24:31Z","isPatch":true,"sender":{"key":"dev+git@drbeat.li","avatar":"https://avatars.githubusercontent.com/u/21444?v=4"},"body":"On 12.12.16 19:12, Torsten Bögershausen wrote:\n> \n>>> Minor question, especially to the next commit:\n>>> Should we make sure to checkout the exact version, which has been tested?\n>>> In this case  cb97792880625e24a9f581412d03659091a0e54f\n>>>\n>>> And this is for both a fresh clone and the git pull\n>>> needs to be replaced by\n>>> git fetch && git checkout cb97792880625e24a9f581412d03659091a0e54f\n>>>\n>>>\n>>> (Which of course is a shell variable)\n>>\n>> I was actually wondering what the policy was for adding submodules to the Git repo,\n>> but then decided against it. Another option would be to fork uniset on GitHub and\n>> just let it stay on a working commit.\n>>\n>> Junio, what's your stance on this?\n>>\n>> Beat\n> \n> If I run  ./update_unicode.sh on the latest master of   https://github.com/depp/uniset.git ,\n> commit  a5fac4a091857dd5429cc2d, I get a diff in  unicode_width.h like this:\n> \n> -{ 0x0300, 0x036F },\n> \n> +{ 768, 879 },\n> \n> IOW, all hex values are printed as decimal values.\n> Not a problem for the compiler, but for the human\n> to check the unicode tables.\n> \n> So I think we should \"pin\" the version of uniset.\n\nThat's what patch 3/3 fixes.\n"},{"id":"307559","messageId":"954eed6b-c899-4f4c-eb3d-2b6d2ff4385d@drbeat.li","threadId":"44678","inReplyTo":"xmqqr35dm203.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH 1/3] update_unicode.sh: update the uniset repo if it exists","fromName":"Beat Bolli","fromEmail":"dev+git@drbeat.li","sentAt":"2016-12-12T23:50:47Z","receivedAt":"2016-12-12T23:51:00Z","isPatch":true,"sender":{"key":"dev+git@drbeat.li","avatar":"https://avatars.githubusercontent.com/u/21444?v=4"},"body":"On 12.12.16 19:33, Junio C Hamano wrote:\n> Torsten Bögershausen <tboegi@web.de> writes:\n> \n>> If I run ./update_unicode.sh on the latest master of\n>> https://github.com/depp/uniset.git , commit\n>> a5fac4a091857dd5429cc2d, I get a diff in unicode_width.h like\n>> this:\n>>\n>> -{ 0x0300, 0x036F },\n>>\n>> +{ 768, 879 },\n>>\n>> IOW, all hex values are printed as decimal values.\n>> Not a problem for the compiler, but for the human\n>> to check the unicode tables.\n>>\n>> So I think we should \"pin\" the version of uniset.\n> \n> Sure, and I'd rather see the update-unicode.sh script moved\n> somewhere in contrib/ while at it.  Those who are interested in\n> keeping up with the unicode standard are tiny minority of the\n> developer population, and most of us would treat the built width\n> table as the source (after all, that is what we ship).\n> \n> To be bluntly honest, I'd rather not to see \"update-unicode.sh\"\n> download and build uniset at all.  It's as if po/ hierarchy shipping\n> with its own script to download and build msgmerge--that's madness.\n> Needless to say, shipping the sources for uniset embedded in our\n> project tree (either as a snapshot-fork or as a submodule) is even\n> worse.  Those who want to muck with po/ are expected to have\n> msgmerge and friends.  Why not expect the same for those who want to\n> update the unicode width table?\n> \n> I'd rather see a written instruction telling which snapshot to get\n> and from where to build and place on their $PATH in the README file,\n> sitting next to the update-unicode.sh script in contrib/uniwidth/\n> directory, for those who are interested in building the width table\n> \"from the source\", and the update-unicode.sh script to assume that\n> uniset is available.\n> \n\nOK. So please don't merge bb/unicode-9.0 to next yet; I'll prepare a\nreroll following your description.\n\nTorsten, is this alright with you?\n\nCheers, Beat\n"},{"id":"307572","messageId":"d91b5e69-9b90-e21e-0dcb-8eace00ddd74@web.de","threadId":"44678","inReplyTo":"954eed6b-c899-4f4c-eb3d-2b6d2ff4385d@drbeat.li","subject":"Re: [PATCH 1/3] update_unicode.sh: update the uniset repo if it exists","fromName":"Torsten Bögershausen","fromEmail":"tboegi@web.de","sentAt":"2016-12-13T06:16:26Z","receivedAt":"2016-12-13T06:16:45Z","isPatch":true,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"\n>> Sure, and I'd rather see the update-unicode.sh script moved\n>> somewhere in contrib/ while at it.  Those who are interested in\n>> keeping up with the unicode standard are tiny minority of the\n>> developer population, and most of us would treat the built width\n>> table as the source (after all, that is what we ship).\n>>\n>> To be bluntly honest, I'd rather not to see \"update-unicode.sh\"\n>> download and build uniset at all.  It's as if po/ hierarchy shipping\n>> with its own script to download and build msgmerge--that's madness.\n>> Needless to say, shipping the sources for uniset embedded in our\n>> project tree (either as a snapshot-fork or as a submodule) is even\n>> worse.  Those who want to muck with po/ are expected to have\n>> msgmerge and friends.  Why not expect the same for those who want to\n>> update the unicode width table?\n>>\n>> I'd rather see a written instruction telling which snapshot to get\n>> and from where to build and place on their $PATH in the README file,\n>> sitting next to the update-unicode.sh script in contrib/uniwidth/\n>> directory, for those who are interested in building the width table\n>> \"from the source\", and the update-unicode.sh script to assume that\n>> uniset is available.\nOK with the contrib - that's an improvement.\nAbout the instructions how to download and compile:\n(we don't need to change the  $PATH, do we ?)\nI don't know.\nThe typical instructions I have seen are a sequence of shell commands\nto be executed, which hopefully simply work by doing \"copy-and-paste\".\nI find this error-prone, as you you may loose the last character while\nmoving the mouse, or don't check the error message or return codes.\nHaving a pre-baked shell script, which does use \"&&\" is in that way more \nattractive,\nand the README can be as simple as run \"update-unicode.sh\" and that's it.\n\nuniset is a small project and where should we put it ?\na) inside the Git tree?\nb) /tmp ?\nc) into the $HOME directory ?\nd) /usr/local\n\na) is quick and dirty\nb) probably OK\nc) Not sure about tha\nd) Needs super user rights\nCan we try to find a good place ?\n\n\"contrib/uniwidth/\" may be different to find, how about contrib/update-unicode ?\n  \n\n> OK. So please don't merge bb/unicode-9.0 to next yet; I'll prepare a\n> reroll following your description.\n>\n> Torsten, is this alright with you?\n\nsure\n\n> Cheers, Beat\n\n\n"},{"id":"307575","messageId":"xmqq37hs72k1.fsf@gitster.mtv.corp.google.com","threadId":"44678","inReplyTo":"d91b5e69-9b90-e21e-0dcb-8eace00ddd74@web.de","subject":"Re: [PATCH 1/3] update_unicode.sh: update the uniset repo if it exists","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-12-13T06:42:22Z","receivedAt":"2016-12-13T06:42:29Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Torsten Bögershausen <tboegi@web.de> writes:\n\n> The typical instructions I have seen are a sequence of shell commands\n> to be executed, which hopefully simply work by doing \"copy-and-paste\".\n> I find this error-prone, as you you may loose the last character while\n> moving the mouse, or don't check the error message or return codes.\n> Having a pre-baked shell script, which does use \"&&\" is in that way\n> more attractive,\n> and the README can be as simple as run \"update-unicode.sh\" and that's it.\n\nThat's OK as well.\n\n> \"contrib/uniwidth/\" may be different to find, how about contrib/update-unicode ?\n\nThis, too.  And as long as .gitignore pattern is set up correctly\nthere, I do not think we terribly mind \"git clone ..from..there..\"\ninto it, either.\n"}]}