{"thread":{"id":"64632","subject":"[PATCH v0 1/3] utf8.c: Prepare workaround for iconv under macOS 14/15","startedAt":"2025-12-15T20:45:30Z","lastAt":"2025-12-16T01:45:56Z","messageCount":2,"participants":["tboegi@web.de","Junio C Hamano"],"isPatch":true,"patchVersion":0,"patchTotal":3},"messages":[{"id":"532202","messageId":"20251215204521.1946490-1-tboegi@web.de","threadId":"64632","inReplyTo":null,"subject":"[PATCH v0 1/3] utf8.c: Prepare workaround for iconv under macOS 14/15","fromName":"","fromEmail":"tboegi@web.de","sentAt":"2025-12-15T20:45:21Z","receivedAt":"2025-12-15T20:45:30Z","isPatch":true,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"From: Torsten Bögershausen \n\nMacOS14 (Sonoma) has started to ship an iconv library with bugs.\nThe same bugs exists even in MacOS 15 (Sequoia)\n\nA bug report running the Git test suite says:\n\nthree tests of t3900 fail on macOS 26.1 for me:\n\n not ok 17 - ISO-2022-JP should be shown in UTF-8 now\n not ok 25 - ISO-2022-JP should be shown in UTF-8 now\n not ok 38 - commit --fixup into ISO-2022-JP from UTF-8\n\nHere's the verbose output of the first one:\n\n----- snip! -----\nexpecting success of 3900.17 'ISO-2022-JP should be shown in UTF-8 now':\n compare_with ISO-2022-JP \"$TEST_DIRECTORY\"/t3900/2-UTF-8.txt\n\n--- /Users/x/src/git/t/t3900/2-UTF-8.txt 2024-10-01 19:43:24.605230684 +0000\n+++ current 2025-12-08 21:52:45.786161909 +0000\n@@ -1,4 +1,4 @@\n はれひほふ\n\n しているのが、いるので。\n-濱浜ほれぷりぽれまびぐりろへ。\n+濱浜ほれぷりぽれまび$0$j$m$X!#\nnot ok 17 - ISO-2022-JP should be shown in UTF-8 now\n1..17\n----- snap! -----\n\ncompare_with runs git show to display a commit message, which in this\ncase here was encoded using ISO-2022-JP and is supposed to be reencoded\nto UTF-8, but git show only does that half-way -- the \"$0$j$m$X!#\" part\nis from the original ISO-2022-JP representation.\n\nThat botched conversion is done by utf8.c::reencode_string_iconv(). It\ncalls iconv(3) to do the actual work, initially with an output buffer of\nthe same size as the input. If the output needs more space the function\nenlarges the buffer and calls iconv(3) again.\n\niconv(3) won't tell us how much space it needs, but it will report what\npart it already managed to convert, so we can increase the buffer and\ncontinue from there. ISO-2022-JP has escape codes for switching between\ncharacter sets, so it's a stateful encoding. I guess the iconv(3) on my\nmachine forgets the state at the end of part one and then messes up part\ntwo.\n\n[end of citation]\n\nWorking around the buggy iconv shipped with the OS can be done in\ntwo ways:\na) Link Git against a different version of iconv.\nb) Improve the handling when iconv needs a larger output buffer.\n\na) is already done by default when either Fink [1]\n or MacPorts [2] is installed.\n (And a patch to do the same for homebrew [3] is on its way)\nb) is implemented here:\nWhen the output buffer is too short, increase it (as before)\nand start from scratch (this is new).\n\nThis workound needs to be enabled with\n'#define ICONV_RESTART_RESET'\nand a makefile knob will be added in the next commit\n\nSuggested-by: René Scharfe \nSigned-off-by: Torsten Bögershausen \n\n[1] https://www.finkproject.org/\n[2] https://www.macports.org/\n[3] https://brew.sh/\n\nFurther readings:\nhttps://blog.r-project.org/2024/12/11/problems-with-iconv-on-macos/\nhttps://lists.gnu.org/archive/html/bug-gnulib/2024-05/msg00375.html\n\nSigned-off-by: Torsten Bögershausen \n---\n utf8.c | 13 +++++++++++++\n 1 file changed, 13 insertions(+)\n\ndiff --git a/utf8.c b/utf8.c\nindex 35a0251939..96460cc414 100644\n--- a/utf8.c\n+++ b/utf8.c\n@@ -515,6 +515,19 @@ char *reencode_string_iconv(const char *in, size_t insz, iconv_t conv,\n \t\t\tout = xrealloc(out, outalloc);\n \t\t\toutpos = out + sofar;\n \t\t\toutsz = outalloc - sofar - 1;\n+#ifdef ICONV_RESTART_RESET\n+\t\t\t/*\n+\t\t\t * If iconv(3) messes up piecemeal conversions\n+\t\t\t * then restore the original pointers, sizes,\n+\t\t\t * and converter state, then retry converting\n+\t\t\t * the full string using the reallocated buffer.\n+\t\t\t */\n+\t\t\tinsz += cp - (iconv_ibp)in; /* Restore insz */\n+\t\t\tcp = (iconv_ibp)in; /* original start value */\n+\t\t\toutpos = out + bom_len; /* original start value */\n+\t\t\toutsz = outalloc - bom_len - 1; /* new len */\n+\t\t\ticonv(conv, NULL, NULL, NULL, NULL); /* reset iconv machinery */\n+#endif\n \t\t}\n \t\telse {\n \t\t\t*outpos = '\\0';\n-- \n2.50.0.rc0.46.g7014b55638.dirty\n\n"},{"id":"532232","messageId":"xmqqa4zj5hda.fsf@gitster.g","threadId":"64632","inReplyTo":"20251215204521.1946490-1-tboegi@web.de","subject":"Re: [PATCH v0 1/3] utf8.c: Prepare workaround for iconv under macOS 14/15","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2025-12-16T01:45:53Z","receivedAt":"2025-12-16T01:45:56Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"tboegi@web.de writes:\n\n> From: Torsten Bögershausen \n>\n> MacOS14 (Sonoma) has started to ship an iconv library with bugs.\n> The same bugs exists even in MacOS 15 (Sequoia)\n>\n> A bug report running the Git test suite says:\n>\n> three tests of t3900 fail on macOS 26.1 for me:\n>\n> not ok 17 - ISO-2022-JP should be shown in UTF-8 now\n> not ok 25 - ISO-2022-JP should be shown in UTF-8 now\n> not ok 38 - commit --fixup into ISO-2022-JP from UTF-8\n>\n> Here's the verbose output of the first one:\n>\n> ----- snip! -----\n> expecting success of 3900.17 'ISO-2022-JP should be shown in UTF-8 now':\n> compare_with ISO-2022-JP \"$TEST_DIRECTORY\"/t3900/2-UTF-8.txt\n>\n> --- /Users/x/src/git/t/t3900/2-UTF-8.txt 2024-10-01 19:43:24.605230684 +0000\n> +++ current 2025-12-08 21:52:45.786161909 +0000\n> @@ -1,4 +1,4 @@\n> はれひほふ\n>\n> しているのが、いるので。\n> -濱浜ほれぷりぽれまびぐりろへ。\n> +濱浜ほれぷりぽれまび$0$j$m$X!#\n> not ok 17 - ISO-2022-JP should be shown in UTF-8 now\n> 1..17\n> ----- snap! -----\n>\n> compare_with runs git show to display a commit message, which in this\n> case here was encoded using ISO-2022-JP and is supposed to be reencoded\n> to UTF-8, but git show only does that half-way -- the \"$0$j$m$X!#\" part\n> is from the original ISO-2022-JP representation.\n\nNext time, please try applying your patch to your own tree before\nsending.\n"}]}