{"thread":{"id":"62101","subject":"[PATCH] unicode: update the width tables to Unicode 16","startedAt":"2024-09-12T20:49:01Z","lastAt":"2024-09-30T18:12:35Z","messageCount":5,"participants":["Beat Bolli","Johannes Schindelin","Junio C Hamano"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"502718","messageId":"20240912204047.1020213-1-dev+git@drbeat.li","threadId":"62101","inReplyTo":null,"subject":"[PATCH] unicode: update the width tables to Unicode 16","fromName":"Beat Bolli","fromEmail":"dev+git@drbeat.li","sentAt":"2024-09-12T20:40:47Z","receivedAt":"2024-09-12T20:49:01Z","isPatch":true,"sender":{"key":"dev+git@drbeat.li","avatar":"https://avatars.githubusercontent.com/u/21444?v=4"},"body":"Unicode 16 has been announced on 2024-09-10 [0], so update the character\nwidth tables to the new version.\n\n[0] https://blog.unicode.org/2024/09/announcing-unicode-standard-version-160.html\n\nSigned-off-by: Beat Bolli <dev+git@drbeat.li>\n---\n unicode-width.h | 37 +++++++++++++++++++++++++------------\n 1 file changed, 25 insertions(+), 12 deletions(-)\n\ndiff --git a/unicode-width.h b/unicode-width.h\nindex be5bf8c4f2..3ffee123a0 100644\n--- a/unicode-width.h\n+++ b/unicode-width.h\n@@ -27,7 +27,7 @@ static const struct interval zero_width[] = {\n { 0x0829, 0x082D },\n { 0x0859, 0x085B },\n { 0x0890, 0x0891 },\n-{ 0x0898, 0x089F },\n+{ 0x0897, 0x089F },\n { 0x08CA, 0x0902 },\n { 0x093A, 0x093A },\n { 0x093C, 0x093C },\n@@ -227,8 +227,9 @@ static const struct interval zero_width[] = {\n { 0x10A3F, 0x10A3F },\n { 0x10AE5, 0x10AE6 },\n { 0x10D24, 0x10D27 },\n+{ 0x10D69, 0x10D6D },\n { 0x10EAB, 0x10EAC },\n-{ 0x10EFD, 0x10EFF },\n+{ 0x10EFC, 0x10EFF },\n { 0x10F46, 0x10F50 },\n { 0x10F82, 0x10F85 },\n { 0x11001, 0x11001 },\n@@ -261,6 +262,11 @@ static const struct interval zero_width[] = {\n { 0x11340, 0x11340 },\n { 0x11366, 0x1136C },\n { 0x11370, 0x11374 },\n+{ 0x113BB, 0x113C0 },\n+{ 0x113CE, 0x113CE },\n+{ 0x113D0, 0x113D0 },\n+{ 0x113D2, 0x113D2 },\n+{ 0x113E1, 0x113E2 },\n { 0x11438, 0x1143F },\n { 0x11442, 0x11444 },\n { 0x11446, 0x11446 },\n@@ -280,7 +286,8 @@ static const struct interval zero_width[] = {\n { 0x116AD, 0x116AD },\n { 0x116B0, 0x116B5 },\n { 0x116B7, 0x116B7 },\n-{ 0x1171D, 0x1171F },\n+{ 0x1171D, 0x1171D },\n+{ 0x1171F, 0x1171F },\n { 0x11722, 0x11725 },\n { 0x11727, 0x1172B },\n { 0x1182F, 0x11837 },\n@@ -319,8 +326,11 @@ static const struct interval zero_width[] = {\n { 0x11F36, 0x11F3A },\n { 0x11F40, 0x11F40 },\n { 0x11F42, 0x11F42 },\n+{ 0x11F5A, 0x11F5A },\n { 0x13430, 0x13440 },\n { 0x13447, 0x13455 },\n+{ 0x1611E, 0x16129 },\n+{ 0x1612D, 0x1612F },\n { 0x16AF0, 0x16AF4 },\n { 0x16B30, 0x16B36 },\n { 0x16F4F, 0x16F4F },\n@@ -351,6 +361,7 @@ static const struct interval zero_width[] = {\n { 0x1E2AE, 0x1E2AE },\n { 0x1E2EC, 0x1E2EF },\n { 0x1E4EC, 0x1E4EF },\n+{ 0x1E5EE, 0x1E5EF },\n { 0x1E8D0, 0x1E8D6 },\n { 0x1E944, 0x1E94A },\n { 0xE0001, 0xE0001 },\n@@ -366,8 +377,10 @@ static const struct interval double_width[] = {\n { 0x23F3, 0x23F3 },\n { 0x25FD, 0x25FE },\n { 0x2614, 0x2615 },\n+{ 0x2630, 0x2637 },\n { 0x2648, 0x2653 },\n { 0x267F, 0x267F },\n+{ 0x268A, 0x268F },\n { 0x2693, 0x2693 },\n { 0x26A1, 0x26A1 },\n { 0x26AA, 0x26AB },\n@@ -401,11 +414,10 @@ static const struct interval double_width[] = {\n { 0x3099, 0x30FF },\n { 0x3105, 0x312F },\n { 0x3131, 0x318E },\n-{ 0x3190, 0x31E3 },\n+{ 0x3190, 0x31E5 },\n { 0x31EF, 0x321E },\n { 0x3220, 0x3247 },\n-{ 0x3250, 0x4DBF },\n-{ 0x4E00, 0xA48C },\n+{ 0x3250, 0xA48C },\n { 0xA490, 0xA4C6 },\n { 0xA960, 0xA97C },\n { 0xAC00, 0xD7A3 },\n@@ -420,7 +432,7 @@ static const struct interval double_width[] = {\n { 0x16FF0, 0x16FF1 },\n { 0x17000, 0x187F7 },\n { 0x18800, 0x18CD5 },\n-{ 0x18D00, 0x18D08 },\n+{ 0x18CFF, 0x18D08 },\n { 0x1AFF0, 0x1AFF3 },\n { 0x1AFF5, 0x1AFFB },\n { 0x1AFFD, 0x1AFFE },\n@@ -430,6 +442,8 @@ static const struct interval double_width[] = {\n { 0x1B155, 0x1B155 },\n { 0x1B164, 0x1B167 },\n { 0x1B170, 0x1B2FB },\n+{ 0x1D300, 0x1D356 },\n+{ 0x1D360, 0x1D376 },\n { 0x1F004, 0x1F004 },\n { 0x1F0CF, 0x1F0CF },\n { 0x1F18E, 0x1F18E },\n@@ -470,11 +484,10 @@ static const struct interval double_width[] = {\n { 0x1F93C, 0x1F945 },\n { 0x1F947, 0x1F9FF },\n { 0x1FA70, 0x1FA7C },\n-{ 0x1FA80, 0x1FA88 },\n-{ 0x1FA90, 0x1FABD },\n-{ 0x1FABF, 0x1FAC5 },\n-{ 0x1FACE, 0x1FADB },\n-{ 0x1FAE0, 0x1FAE8 },\n+{ 0x1FA80, 0x1FA89 },\n+{ 0x1FA8F, 0x1FAC6 },\n+{ 0x1FACE, 0x1FADC },\n+{ 0x1FADF, 0x1FAE9 },\n { 0x1FAF0, 0x1FAF8 },\n { 0x20000, 0x2FFFD },\n { 0x30000, 0x3FFFD }\n-- \n2.45.2\n\n"},{"id":"502951","messageId":"7449ce5b-5308-ce6e-394d-43dec511ef16@gmx.de","threadId":"62101","inReplyTo":"20240912204047.1020213-1-dev+git@drbeat.li","subject":"Re: [PATCH] unicode: update the width tables to Unicode 16","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2024-09-17T12:37:17Z","receivedAt":"2024-09-17T12:37:24Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Beat,\n\nOn Thu, 12 Sep 2024, Beat Bolli wrote:\n\n> Unicode 16 has been announced on 2024-09-10 [0], so update the character\n> width tables to the new version.\n>\n> [0] https://blog.unicode.org/2024/09/announcing-unicode-standard-version-160.html\n\nI can confirm that the output is identical to the result of running\n./contrib/update-unicode/update_unicode.sh.\n\nMaybe we should add an automated, scheduled workflow for these updates?\n\nCiao,\nJohannes\n\n>\n> Signed-off-by: Beat Bolli <dev+git@drbeat.li>\n> ---\n>  unicode-width.h | 37 +++++++++++++++++++++++++------------\n>  1 file changed, 25 insertions(+), 12 deletions(-)\n>\n> diff --git a/unicode-width.h b/unicode-width.h\n> index be5bf8c4f2..3ffee123a0 100644\n> --- a/unicode-width.h\n> +++ b/unicode-width.h\n> @@ -27,7 +27,7 @@ static const struct interval zero_width[] = {\n>  { 0x0829, 0x082D },\n>  { 0x0859, 0x085B },\n>  { 0x0890, 0x0891 },\n> -{ 0x0898, 0x089F },\n> +{ 0x0897, 0x089F },\n>  { 0x08CA, 0x0902 },\n>  { 0x093A, 0x093A },\n>  { 0x093C, 0x093C },\n> @@ -227,8 +227,9 @@ static const struct interval zero_width[] = {\n>  { 0x10A3F, 0x10A3F },\n>  { 0x10AE5, 0x10AE6 },\n>  { 0x10D24, 0x10D27 },\n> +{ 0x10D69, 0x10D6D },\n>  { 0x10EAB, 0x10EAC },\n> -{ 0x10EFD, 0x10EFF },\n> +{ 0x10EFC, 0x10EFF },\n>  { 0x10F46, 0x10F50 },\n>  { 0x10F82, 0x10F85 },\n>  { 0x11001, 0x11001 },\n> @@ -261,6 +262,11 @@ static const struct interval zero_width[] = {\n>  { 0x11340, 0x11340 },\n>  { 0x11366, 0x1136C },\n>  { 0x11370, 0x11374 },\n> +{ 0x113BB, 0x113C0 },\n> +{ 0x113CE, 0x113CE },\n> +{ 0x113D0, 0x113D0 },\n> +{ 0x113D2, 0x113D2 },\n> +{ 0x113E1, 0x113E2 },\n>  { 0x11438, 0x1143F },\n>  { 0x11442, 0x11444 },\n>  { 0x11446, 0x11446 },\n> @@ -280,7 +286,8 @@ static const struct interval zero_width[] = {\n>  { 0x116AD, 0x116AD },\n>  { 0x116B0, 0x116B5 },\n>  { 0x116B7, 0x116B7 },\n> -{ 0x1171D, 0x1171F },\n> +{ 0x1171D, 0x1171D },\n> +{ 0x1171F, 0x1171F },\n>  { 0x11722, 0x11725 },\n>  { 0x11727, 0x1172B },\n>  { 0x1182F, 0x11837 },\n> @@ -319,8 +326,11 @@ static const struct interval zero_width[] = {\n>  { 0x11F36, 0x11F3A },\n>  { 0x11F40, 0x11F40 },\n>  { 0x11F42, 0x11F42 },\n> +{ 0x11F5A, 0x11F5A },\n>  { 0x13430, 0x13440 },\n>  { 0x13447, 0x13455 },\n> +{ 0x1611E, 0x16129 },\n> +{ 0x1612D, 0x1612F },\n>  { 0x16AF0, 0x16AF4 },\n>  { 0x16B30, 0x16B36 },\n>  { 0x16F4F, 0x16F4F },\n> @@ -351,6 +361,7 @@ static const struct interval zero_width[] = {\n>  { 0x1E2AE, 0x1E2AE },\n>  { 0x1E2EC, 0x1E2EF },\n>  { 0x1E4EC, 0x1E4EF },\n> +{ 0x1E5EE, 0x1E5EF },\n>  { 0x1E8D0, 0x1E8D6 },\n>  { 0x1E944, 0x1E94A },\n>  { 0xE0001, 0xE0001 },\n> @@ -366,8 +377,10 @@ static const struct interval double_width[] = {\n>  { 0x23F3, 0x23F3 },\n>  { 0x25FD, 0x25FE },\n>  { 0x2614, 0x2615 },\n> +{ 0x2630, 0x2637 },\n>  { 0x2648, 0x2653 },\n>  { 0x267F, 0x267F },\n> +{ 0x268A, 0x268F },\n>  { 0x2693, 0x2693 },\n>  { 0x26A1, 0x26A1 },\n>  { 0x26AA, 0x26AB },\n> @@ -401,11 +414,10 @@ static const struct interval double_width[] = {\n>  { 0x3099, 0x30FF },\n>  { 0x3105, 0x312F },\n>  { 0x3131, 0x318E },\n> -{ 0x3190, 0x31E3 },\n> +{ 0x3190, 0x31E5 },\n>  { 0x31EF, 0x321E },\n>  { 0x3220, 0x3247 },\n> -{ 0x3250, 0x4DBF },\n> -{ 0x4E00, 0xA48C },\n> +{ 0x3250, 0xA48C },\n>  { 0xA490, 0xA4C6 },\n>  { 0xA960, 0xA97C },\n>  { 0xAC00, 0xD7A3 },\n> @@ -420,7 +432,7 @@ static const struct interval double_width[] = {\n>  { 0x16FF0, 0x16FF1 },\n>  { 0x17000, 0x187F7 },\n>  { 0x18800, 0x18CD5 },\n> -{ 0x18D00, 0x18D08 },\n> +{ 0x18CFF, 0x18D08 },\n>  { 0x1AFF0, 0x1AFF3 },\n>  { 0x1AFF5, 0x1AFFB },\n>  { 0x1AFFD, 0x1AFFE },\n> @@ -430,6 +442,8 @@ static const struct interval double_width[] = {\n>  { 0x1B155, 0x1B155 },\n>  { 0x1B164, 0x1B167 },\n>  { 0x1B170, 0x1B2FB },\n> +{ 0x1D300, 0x1D356 },\n> +{ 0x1D360, 0x1D376 },\n>  { 0x1F004, 0x1F004 },\n>  { 0x1F0CF, 0x1F0CF },\n>  { 0x1F18E, 0x1F18E },\n> @@ -470,11 +484,10 @@ static const struct interval double_width[] = {\n>  { 0x1F93C, 0x1F945 },\n>  { 0x1F947, 0x1F9FF },\n>  { 0x1FA70, 0x1FA7C },\n> -{ 0x1FA80, 0x1FA88 },\n> -{ 0x1FA90, 0x1FABD },\n> -{ 0x1FABF, 0x1FAC5 },\n> -{ 0x1FACE, 0x1FADB },\n> -{ 0x1FAE0, 0x1FAE8 },\n> +{ 0x1FA80, 0x1FA89 },\n> +{ 0x1FA8F, 0x1FAC6 },\n> +{ 0x1FACE, 0x1FADC },\n> +{ 0x1FADF, 0x1FAE9 },\n>  { 0x1FAF0, 0x1FAF8 },\n>  { 0x20000, 0x2FFFD },\n>  { 0x30000, 0x3FFFD }\n> --\n> 2.45.2\n>\n>\n>\n"},{"id":"502972","messageId":"xmqqy13qhrot.fsf@gitster.g","threadId":"62101","inReplyTo":"7449ce5b-5308-ce6e-394d-43dec511ef16@gmx.de","subject":"Re: [PATCH] unicode: update the width tables to Unicode 16","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-09-17T21:54:26Z","receivedAt":"2024-09-17T21:54:29Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:\n\n> Hi Beat,\n>\n> On Thu, 12 Sep 2024, Beat Bolli wrote:\n>\n>> Unicode 16 has been announced on 2024-09-10 [0], so update the character\n>> width tables to the new version.\n>>\n>> [0] https://blog.unicode.org/2024/09/announcing-unicode-standard-version-160.html\n>\n> I can confirm that the output is identical to the result of running\n> ./contrib/update-unicode/update_unicode.sh.\n\nThanks for double checking.  I did the same when I queued the patch\nand it indeed looked good.\n\n> Maybe we should add an automated, scheduled workflow for these updates?\n\nWe could, but the consortium aims to issue major updates once a year\nin September, with minor versions and updates \"will be avoided\", so\nwe may need to devise automation that makes better use of resources\nthan to scrape http://www.unicode.org/Public/UCD/latest/ucd/ daily.\n\n44dc651132 2024-09-12T22:40:47+02:00 unicode: update the width tables to Unicode 16\n872976c37e 2023-09-25T21:07:04+02:00 unicode: update the width tables to Unicode 15.1\nb10cbdac4c 2023-03-30T21:15:17+02:00 unicode: update the width tables to Unicode 15\n187fc8b8b6 2021-09-17T12:19:20-07:00 unicode: update the width tables to Unicode 14\n65588b0b2e 2020-03-17T16:36:05+01:00 unicode: update the width tables to Unicode 13.0\n5817f9caa3 2019-05-29T22:50:45+02:00 unicode: update the width tables to Unicode 12.1\n584b62c37b 2019-03-21T22:06:17+01:00 unicode: update the width tables to Unicode 12\n570951eea2 2018-07-09T21:44:52+02:00 unicode: update the width tables to Unicode 11\ne233bef43e 2018-04-10T14:26:17-07:00 unicode_width.h: rename to use dash in file name\n"},{"id":"503664","messageId":"43570a02-3c99-1d07-4403-3e95c6e6720a@gmx.de","threadId":"62101","inReplyTo":"xmqqy13qhrot.fsf@gitster.g","subject":"Re: [PATCH] unicode: update the width tables to Unicode 16","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2024-09-29T18:58:45Z","receivedAt":"2024-09-29T18:58:54Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Junio,\n\nOn Tue, 17 Sep 2024, Junio C Hamano wrote:\n\n> Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:\n>\n> > On Thu, 12 Sep 2024, Beat Bolli wrote:\n> >\n> >> Unicode 16 has been announced on 2024-09-10 [0], so update the character\n> >> width tables to the new version.\n> >>\n> >> [0] https://blog.unicode.org/2024/09/announcing-unicode-standard-version-160.html\n> >\n> > I can confirm that the output is identical to the result of running\n> > ./contrib/update-unicode/update_unicode.sh.\n>\n> Thanks for double checking.  I did the same when I queued the patch\n> and it indeed looked good.\n>\n> > Maybe we should add an automated, scheduled workflow for these updates?\n>\n> We could, but the consortium aims to issue major updates once a year\n> in September, with minor versions and updates \"will be avoided\", so\n> we may need to devise automation that makes better use of resources\n> than to scrape http://www.unicode.org/Public/UCD/latest/ucd/ daily.\n\nOh, but I obviously was not suggesting as crude a thing as to scrape it\nunconditionally, and certainly not daily. No, I was thinking about\nsomething checking the `Last-Modified:` header and only acting upon\nupdated Unicode definitions, and checking for updates only on a weekly\nbasis. Something along these lines:\n\n\t```yml\n\tname: update Unicode definitions\n\n\ton:\n\t  schedule:\n\t    - cron: '1 15 * * 4' # 3:01pm on Wednesdays\n\t  workflow_dispatch:\n\n\tjobs:\n\t  update-repo-variable:\n\t    if: vars.UNICODE_LAST_MODIFIED != ''\n\t    runs-on: ubuntu-latest\n\t    steps:\n\t      - id: check\n\t\trun: |\n\t\t  set -x\n\t\t  latest_update=\"$(curl -I https://www.unicode.org/Public/UCD/latest/ucd/UCD.zip |\n\t\t    sed -n 's/^Last-Modified: //p')\" &&\n\t\t  if test '${{ vars.UNICODE_LAST_MODIFIED }}' = \"$latest_update\"\n\t\t  then\n\t\t    echo \"result=skip\" >>$GITHUB_OUTPUT\n\t\t    exit 0\n\t\t  fi\n\t\t  echo \"result=$latest_update\" >>$GITHUB_OUTPUT\n\t      - if: steps.check.outputs.result != 'skip'\n\t\trun: echo ::notice::_Now_ we scrape and do stuff\n\t      - if: steps.check.outputs.result != 'skip'\n\t\tenv:\n\t\t  GH_TOKEN: ${{ secrets.UNICODE_LAST_MODIFIED_PAT }}\n\t\trun: |\n\t\t  gh api -X PATCH \\\n\t\t    repos/$GITHUB_REPOSITORY/actions/variables/UNICODE_LAST_MODIFIED \\\n\t\t    -f value='${{ steps.check.outputs.result }}'\n\nThis would use the repository variable `UNICODE_LAST_MODIFIED` to store\nthe `Last-Modified:` value that was last seen (and implicitly act as the\nknob to prevent running in forks: if the variable is not yet set, the job\nwill be skipped).\n\nSadly, to update the repository variable, we cannot use `permissions:`\nbecause the workflow syntax does not offer the `variables` scope.\nTherefore a Personal Access Token would need to be stored as a repository\nsecret. I used a fine-grained token in my tests whose sope was\nRepository > Variables: read-write.\n\nCiao,\nJohannes\n\n>\n> 44dc651132 2024-09-12T22:40:47+02:00 unicode: update the width tables to Unicode 16\n> 872976c37e 2023-09-25T21:07:04+02:00 unicode: update the width tables to Unicode 15.1\n> b10cbdac4c 2023-03-30T21:15:17+02:00 unicode: update the width tables to Unicode 15\n> 187fc8b8b6 2021-09-17T12:19:20-07:00 unicode: update the width tables to Unicode 14\n> 65588b0b2e 2020-03-17T16:36:05+01:00 unicode: update the width tables to Unicode 13.0\n> 5817f9caa3 2019-05-29T22:50:45+02:00 unicode: update the width tables to Unicode 12.1\n> 584b62c37b 2019-03-21T22:06:17+01:00 unicode: update the width tables to Unicode 12\n> 570951eea2 2018-07-09T21:44:52+02:00 unicode: update the width tables to Unicode 11\n> e233bef43e 2018-04-10T14:26:17-07:00 unicode_width.h: rename to use dash in file name\n>\n>\n"},{"id":"503754","messageId":"xmqq34lh80z3.fsf@gitster.g","threadId":"62101","inReplyTo":"43570a02-3c99-1d07-4403-3e95c6e6720a@gmx.de","subject":"Re: [PATCH] unicode: update the width tables to Unicode 16","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-09-30T18:12:32Z","receivedAt":"2024-09-30T18:12:35Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:\n\n> Oh, but I obviously was not suggesting as crude a thing as to scrape it\n> unconditionally, and certainly not daily. No, I was thinking about\n> something checking the `Last-Modified:` header and only acting upon\n> updated Unicode definitions, and checking for updates only on a weekly\n> basis. Something along these lines:\n> ...\n> Sadly, to update the repository variable, we cannot use `permissions:`\n> because the workflow syntax does not offer the `variables` scope.\n> Therefore a Personal Access Token would need to be stored as a repository\n> secret. I used a fine-grained token in my tests whose sope was\n> Repository > Variables: read-write.\n\nAnd it can make a patch and send it to the list and get reviewed the\nusual way.  It is a bit curious whose Sob should be on such a patch,\nbut we can work out the details, if we were seriously to automate\nit.  It all seems workable.\n\nYou may have already noticed, but I am lazy and tolerate manual\ntasks if they do not come more often than once per quarter ;-)\n\nThanks.\n"}]}