{"thread":{"id":"50233","subject":"Broken interactive rebase text after some UTF-8 characters","startedAt":"2019-01-15T19:36:18Z","lastAt":"2019-02-01T19:00:43Z","messageCount":16,"participants":["Michal Nowak","Phillip Wood","Alban Gruin","Junio C Hamano","Johannes Schindelin"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"366745","messageId":"a50734d9-4d30-5847-b5df-67a8458a36cb@startmail.com","threadId":"50233","inReplyTo":null,"subject":"Broken interactive rebase text after some UTF-8 characters","fromName":"Michal Nowak","fromEmail":"mnowak@startmail.com","sentAt":"2019-01-15T19:29:01Z","receivedAt":"2019-01-15T19:36:18Z","isPatch":false,"sender":{"key":"mnowak@startmail.com","avatar":null},"body":"Hello,\n\non OpenIndiana 2018.10 (illumos kernel) line of the interactive rebase \ntext after a particular name (Gergő Mihály Doma) is broken:\n\npick 1ea94c756c 10202 loader: use screen-#rows to find bottom left \nco-ordinates Reviewed by: Toomas Soome <tsoome@me.com> Reviewed by: \nGergő Mihály Doma <domag02@gmail.com> Approved by: Robert Mustacchi \n<rm@joyent.com>\np\npick cadd68ea00 10078 smatch fixes for UCB Reviewed by: Andy Fiddaman \n<omnios@citrus-it.net> Reviewed by: Toomas Soome <tsoome@me.com> \nReviewed by: Peter Tribble <peter.tribble@gmail.com> Approved by: Robert \nMustacchi <rm@joyent.com>\n\n\nSource: https://github.com/illumos/illumos-gate.git\n\nSee the second item which only has \"p\".\n\nThis is with LC_ALL=en_US.UTF-8 (same with cs_CZ.UTF-8), with LC_ALL=C \nthe text is fine.\n\nI checked various editors (vim, nano, cat), they are all the same.\n\n`git log` (PAGER is set to `/usr/bin/less -ins`) shows the particular \ncommit correctly.\n\nI tested following git version: 2.16.5, 2.19.2, 2.20.1, git.next branch.\n\nHere's the configure output: \nhttps://paste.ec/paste/oDdydNxZ#r3avan8BL+8lldFMo928cw3eXSServTsSkGPW4jRBOd\n\nMichal\n"},{"id":"366812","messageId":"339d4dbd-b1bd-cf88-12b0-2af42f35ded7@talktalk.net","threadId":"50233","inReplyTo":"a50734d9-4d30-5847-b5df-67a8458a36cb@startmail.com","subject":"Re: Broken interactive rebase text after some UTF-8 characters","fromName":"Phillip Wood","fromEmail":"phillip.wood@talktalk.net","sentAt":"2019-01-16T10:33:55Z","receivedAt":"2019-01-16T10:34:01Z","isPatch":false,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"Dear Michal\n\nThanks for the bug report, unfortunately I'm unable to reproduce it here \nusing git 2.20.1 (see below). Knowing a little about how the todo list \nis created I cannot think how multibyte characters would break it. What \ncommand line were you using to start the rebase?\n\nBest Wishes\n\nPhillip\n\n$ git clone --shallow-since='11-1-2019' \nhttps://github.com/illumos/illumos-gate.git\nCloning into 'illumos-gate'...\nremote: Enumerating objects: 54348, done.\nremote: Counting objects: 100% (54348/54348), done.\nremote: Compressing objects: 100% (39008/39008), done.\nremote: Total 54348 (delta 19671), reused 31187 (delta 11884), pack-reused 0\nReceiving objects: 100% (54348/54348), 148.91 MiB | 1.18 MiB/s, done.\nResolving deltas: 100% (19671/19671), done.\nChecking out files: 100% (47352/47352), done.\n\n$ cd ~/src/illumos-gate\n$ git log --oneline -1\nf482e26c (HEAD -> master, origin/master, origin/HEAD) 10233 dboot: \nprocess_module() is missing newline at the end of the string Reviewed \nby: Sebastian Wiedenroth <wiedi@frubar.net> Reviewed by: Jason King \n<jbk@joyent.com> Reviewed by: Andy Stormont \n<astormont@racktopsystems.com> Approved by: Robert Mustacchi <rm@joyent.com>\n\n$ GIT_SEQUENCE_EDITOR=cat git rebase -i \n08487eea68a2fa501b5042131c6db068089f82e1\n\nhint: Waiting for your editor to close the file...\npick 1ea94c75 10202 loader: use screen-#rows to find bottom left \nco-ordinates Reviewed by: Toomas Soome <tsoome@me.com> Reviewed by: \nGergő Mihály Doma <domag02@gmail.com> Approved by: Robert Mustacchi \n<rm@joyent.com>\npick cadd68ea 10078 smatch fixes for UCB Reviewed by: Andy Fiddaman \n<omnios@citrus-it.net> Reviewed by: Toomas Soome <tsoome@me.com> \nReviewed by: Peter Tribble <peter.tribble@gmail.com> Approved by: Robert \nMustacchi <rm@joyent.com>\npick 05ede3db 10079 smatch Makefile changes for usr/src/lib Reviewed by: \nAndy Fiddaman <andy@omniosce.org> Approved by: Robert Mustacchi \n<rm@joyent.com>\npick 5661bb76 10080 smatch Makefile changes for usr/src/cmd Reviewed by: \nAndy Fiddaman <andy@omniosce.org> Approved by: Robert Mustacchi \n<rm@joyent.com>\npick 15c07adc 10081 smatch indenting fixes for usr/src/uts Reviewed by: \nToomas Soome <tsoome@me.com> Reviewed by: Peter Tribble \n<peter.tribble@gmail.com> Reviewed by: Andy Fiddaman <andy@omniosce.org> \nApproved by: Robert Mustacchi <rm@joyent.com>\npick c653bb47 10197 build smatch in parallel Reviewed by: Andy Fiddaman \n<andy@omniosce.org> Approved by: Robert Mustacchi <rm@joyent.com>\npick 161294fe 10217 mdb: r9w isn't r8w 10218 CONV_CAP_VAL_HW2_BUFSIZE \nwasn't updated 10219 AV_386_2_AVX512VPOPCDQ elfcap name should be \nconsistent Reviewed by: Dan McDonald <danmcd@joyent.com> Reviewed by: \nRob Johnston <rob.johnston@joyent.com> Reviewed by: Patrick Mooney \n<patrick.mooney@joyent.com> Reviewed by: Andy Fiddaman \n<andy@omniosce.org> Approved by: Dan McDonald <danmcd@joyent.com>\npick f482e26c 10233 dboot: process_module() is missing newline at the \nend of the string Reviewed by: Sebastian Wiedenroth <wiedi@frubar.net> \nReviewed by: Jason King <jbk@joyent.com> Reviewed by: Andy Stormont \n<astormont@racktopsystems.com> Approved by: Robert Mustacchi <rm@joyent.com>\n\n# Rebase 08487eea..f482e26c onto 08487eea (8 commands)\n#\n# Commands:\n# p, pick <commit> = use commit\n# r, reword <commit> = use commit, but edit the commit message\n# e, edit <commit> = use commit, but stop for amending\n# s, squash <commit> = use commit, but meld into previous commit\n# f, fixup <commit> = like \"squash\", but discard this commit's log message\n# x, exec <command> = run command (the rest of the line) using shell\n# b, break = stop here (continue rebase later with 'git rebase --continue')\n# d, drop <commit> = remove commit\n# l, label <label> = label current HEAD with a name\n# t, reset <label> = reset HEAD to a label\n# m, merge [-C <commit> | -c <commit>] <label> [# <oneline>]\n# .       create a merge commit using the original merge commit's\n# .       message (or the oneline, if no original merge commit was\n# .       specified). Use -c <commit> to reword the commit message.\n#\n# These lines can be re-ordered; they are executed from top to bottom.\n#\n# If you remove a line here THAT COMMIT WILL BE LOST.\n#\n# However, if you remove everything, the rebase will be aborted.\n#\n# Note that empty commits are commented out\nSuccessfully rebased and updated refs/heads/master.\n\n\nOn 15/01/2019 19:29, Michal Nowak wrote:\n> Hello,\n> \n> on OpenIndiana 2018.10 (illumos kernel) line of the interactive rebase \n> text after a particular name (Gergő Mihály Doma) is broken:\n> \n> pick 1ea94c756c 10202 loader: use screen-#rows to find bottom left \n> co-ordinates Reviewed by: Toomas Soome <tsoome@me.com> Reviewed by: \n> Gergő Mihály Doma <domag02@gmail.com> Approved by: Robert Mustacchi \n> <rm@joyent.com>\n> p\n> pick cadd68ea00 10078 smatch fixes for UCB Reviewed by: Andy Fiddaman \n> <omnios@citrus-it.net> Reviewed by: Toomas Soome <tsoome@me.com> \n> Reviewed by: Peter Tribble <peter.tribble@gmail.com> Approved by: Robert \n> Mustacchi <rm@joyent.com>\n> \n> \n> Source: https://github.com/illumos/illumos-gate.git\n> \n> See the second item which only has \"p\".\n> \n> This is with LC_ALL=en_US.UTF-8 (same with cs_CZ.UTF-8), with LC_ALL=C \n> the text is fine.\n> \n> I checked various editors (vim, nano, cat), they are all the same.\n> \n> `git log` (PAGER is set to `/usr/bin/less -ins`) shows the particular \n> commit correctly.\n> \n> I tested following git version: 2.16.5, 2.19.2, 2.20.1, git.next branch.\n> \n> Here's the configure output: \n> https://paste.ec/paste/oDdydNxZ#r3avan8BL+8lldFMo928cw3eXSServTsSkGPW4jRBOd\n> \n> Michal\n> \n\n"},{"id":"366877","messageId":"23c60f2f-43ff-94ec-6100-861c655ec80b@startmail.com","threadId":"50233","inReplyTo":"339d4dbd-b1bd-cf88-12b0-2af42f35ded7@talktalk.net","subject":"Re: Broken interactive rebase text after some UTF-8 characters","fromName":"Michal Nowak","fromEmail":"mnowak@startmail.com","sentAt":"2019-01-16T21:36:04Z","receivedAt":"2019-01-16T21:36:11Z","isPatch":false,"sender":{"key":"mnowak@startmail.com","avatar":null},"body":"Hello Phillip, thanks for reaching out.\n\nThis is what I see with your suggested command:\n\n\n$ GIT_SEQUENCE_EDITOR=cat git rebase -i \n08487eea68a2fa501b5042131c6db068089f82e1\nhint: Waiting for your editor to close the file... pick 1ea94c756c 10202 \nloader: use screen-#rows to find bottom left co-ordinates Reviewed by: \nToomas Soome <tsoome@me.com> Reviewed by: Gergő Mihály Doma \n<domag02@gmail.com> Approved by: Robert Mustacchi <rm@joyent.com>\np\npick cadd68ea00 10078 smatch fixes for UCB Reviewed by: Andy Fiddaman \n<omnios@citrus-it.net> Reviewed by: Toomas Soome <tsoome@me.com> \nReviewed by: Peter Tribble <peter.tribble@gmail.com> Approved by: Robert \nMustacchi <rm@joyent.com>\npick 05ede3db5e 10079 smatch Makefile changes for usr/src/lib Reviewed \nby: Andy Fiddaman <andy@omniosce.org> Approved by: Robert Mustacchi \n<rm@joyent.com>\npick 5661bb7641 10080 smatch Makefile changes for usr/src/cmd Reviewed \nby: Andy Fiddaman <andy@omniosce.org> Approved by: Robert Mustacchi \n<rm@joyent.com>\npick 15c07adc1c 10081 smatch indenting fixes for usr/src/uts Reviewed \nby: Toomas Soome <tsoome@me.com> Reviewed by: Peter Tribble \n<peter.tribble@gmail.com> Reviewed by: Andy Fiddaman <andy@omniosce.org> \nApproved by: Robert Mustacchi <rm@joyent.com>\npick c653bb4713 10197 build smatch in parallel Reviewed by: Andy \nFiddaman <andy@omniosce.org> Approved by: Robert Mustacchi <rm@joyent.com>\n\n# Rebase 08487eea68..c653bb4713 onto 08487eea68 (6 commands)\n#\n# Commands:\n# p, pick <commit> = use commit\n# r, reword <commit> = use commit, but edit the commit message\n# e, edit <commit> = use commit, but stop for amending\n# s, squash <commit> = use commit, but meld into previous commit\n# f, fixup <commit> = like \"squash\", but discard this commit's log message\n# x, exec <command> = run command (the rest of the line) using shell\n# b, break = stop here (continue rebase later with 'git rebase --continue')\n# d, drop <commit> = remove commit\n# l, label <label> = label current HEAD with a name\n# t, reset <label> = reset HEAD to a label\n# m, merge [-C <commit> | -c <commit>] <label> [# <oneline>]\n# .       create a merge commit using the original merge commit's\n# .       message (or the oneline, if no original merge commit was\n# .       specified). Use -c <commit> to reword the commit message.\n#\n# These lines can be re-ordered; they are executed from top to bottom.\n#\n# If you remove a line here THAT COMMIT WILL BE LOST.\n#\n# However, if you remove everything, the rebase will be aborted.\n#\n# Note that empty commits are commented out\nerror: missing arguments for pick\nerror: invalid line 2: p\nYou can fix this with 'git rebase --edit-todo' and then run 'git rebase \n--continue'.\nOr you can abort the rebase with 'git rebase --abort'.\n\n\nLet me know, if there's anything I can try myself to gather more \ninformation. Should you want to reproduce this on OpenIndiana on your \nown, Vagrant is the more straightforward way to deploy OpenIndiana.\n\nThanks,\nMichal\n\nOn 01/16/19 11:33 AM, Phillip Wood wrote:\n> Dear Michal\n> \n> Thanks for the bug report, unfortunately I'm unable to reproduce it here \n> using git 2.20.1 (see below). Knowing a little about how the todo list \n> is created I cannot think how multibyte characters would break it. What \n> command line were you using to start the rebase?\n> \n> Best Wishes\n> \n> Phillip\n> \n> $ git clone --shallow-since='11-1-2019' \n> https://github.com/illumos/illumos-gate.git\n> Cloning into 'illumos-gate'...\n> remote: Enumerating objects: 54348, done.\n> remote: Counting objects: 100% (54348/54348), done.\n> remote: Compressing objects: 100% (39008/39008), done.\n> remote: Total 54348 (delta 19671), reused 31187 (delta 11884), \n> pack-reused 0\n> Receiving objects: 100% (54348/54348), 148.91 MiB | 1.18 MiB/s, done.\n> Resolving deltas: 100% (19671/19671), done.\n> Checking out files: 100% (47352/47352), done.\n> \n> $ cd ~/src/illumos-gate\n> $ git log --oneline -1\n> f482e26c (HEAD -> master, origin/master, origin/HEAD) 10233 dboot: \n> process_module() is missing newline at the end of the string Reviewed \n> by: Sebastian Wiedenroth <wiedi@frubar.net> Reviewed by: Jason King \n> <jbk@joyent.com> Reviewed by: Andy Stormont \n> <astormont@racktopsystems.com> Approved by: Robert Mustacchi \n> <rm@joyent.com>\n> \n> $ GIT_SEQUENCE_EDITOR=cat git rebase -i \n> 08487eea68a2fa501b5042131c6db068089f82e1\n> \n> hint: Waiting for your editor to close the file...\n> pick 1ea94c75 10202 loader: use screen-#rows to find bottom left \n> co-ordinates Reviewed by: Toomas Soome <tsoome@me.com> Reviewed by: \n> Gergő Mihály Doma <domag02@gmail.com> Approved by: Robert Mustacchi \n> <rm@joyent.com>\n> pick cadd68ea 10078 smatch fixes for UCB Reviewed by: Andy Fiddaman \n> <omnios@citrus-it.net> Reviewed by: Toomas Soome <tsoome@me.com> \n> Reviewed by: Peter Tribble <peter.tribble@gmail.com> Approved by: Robert \n> Mustacchi <rm@joyent.com>\n> pick 05ede3db 10079 smatch Makefile changes for usr/src/lib Reviewed by: \n> Andy Fiddaman <andy@omniosce.org> Approved by: Robert Mustacchi \n> <rm@joyent.com>\n> pick 5661bb76 10080 smatch Makefile changes for usr/src/cmd Reviewed by: \n> Andy Fiddaman <andy@omniosce.org> Approved by: Robert Mustacchi \n> <rm@joyent.com>\n> pick 15c07adc 10081 smatch indenting fixes for usr/src/uts Reviewed by: \n> Toomas Soome <tsoome@me.com> Reviewed by: Peter Tribble \n> <peter.tribble@gmail.com> Reviewed by: Andy Fiddaman <andy@omniosce.org> \n> Approved by: Robert Mustacchi <rm@joyent.com>\n> pick c653bb47 10197 build smatch in parallel Reviewed by: Andy Fiddaman \n> <andy@omniosce.org> Approved by: Robert Mustacchi <rm@joyent.com>\n> pick 161294fe 10217 mdb: r9w isn't r8w 10218 CONV_CAP_VAL_HW2_BUFSIZE \n> wasn't updated 10219 AV_386_2_AVX512VPOPCDQ elfcap name should be \n> consistent Reviewed by: Dan McDonald <danmcd@joyent.com> Reviewed by: \n> Rob Johnston <rob.johnston@joyent.com> Reviewed by: Patrick Mooney \n> <patrick.mooney@joyent.com> Reviewed by: Andy Fiddaman \n> <andy@omniosce.org> Approved by: Dan McDonald <danmcd@joyent.com>\n> pick f482e26c 10233 dboot: process_module() is missing newline at the \n> end of the string Reviewed by: Sebastian Wiedenroth <wiedi@frubar.net> \n> Reviewed by: Jason King <jbk@joyent.com> Reviewed by: Andy Stormont \n> <astormont@racktopsystems.com> Approved by: Robert Mustacchi \n> <rm@joyent.com>\n> \n> # Rebase 08487eea..f482e26c onto 08487eea (8 commands)\n> #\n> # Commands:\n> # p, pick <commit> = use commit\n> # r, reword <commit> = use commit, but edit the commit message\n> # e, edit <commit> = use commit, but stop for amending\n> # s, squash <commit> = use commit, but meld into previous commit\n> # f, fixup <commit> = like \"squash\", but discard this commit's log message\n> # x, exec <command> = run command (the rest of the line) using shell\n> # b, break = stop here (continue rebase later with 'git rebase --continue')\n> # d, drop <commit> = remove commit\n> # l, label <label> = label current HEAD with a name\n> # t, reset <label> = reset HEAD to a label\n> # m, merge [-C <commit> | -c <commit>] <label> [# <oneline>]\n> # .       create a merge commit using the original merge commit's\n> # .       message (or the oneline, if no original merge commit was\n> # .       specified). Use -c <commit> to reword the commit message.\n> #\n> # These lines can be re-ordered; they are executed from top to bottom.\n> #\n> # If you remove a line here THAT COMMIT WILL BE LOST.\n> #\n> # However, if you remove everything, the rebase will be aborted.\n> #\n> # Note that empty commits are commented out\n> Successfully rebased and updated refs/heads/master.\n> \n> \n> On 15/01/2019 19:29, Michal Nowak wrote:\n>> Hello,\n>>\n>> on OpenIndiana 2018.10 (illumos kernel) line of the interactive rebase \n>> text after a particular name (Gergő Mihály Doma) is broken:\n>>\n>> pick 1ea94c756c 10202 loader: use screen-#rows to find bottom left \n>> co-ordinates Reviewed by: Toomas Soome <tsoome@me.com> Reviewed by: \n>> Gergő Mihály Doma <domag02@gmail.com> Approved by: Robert Mustacchi \n>> <rm@joyent.com>\n>> p\n>> pick cadd68ea00 10078 smatch fixes for UCB Reviewed by: Andy Fiddaman \n>> <omnios@citrus-it.net> Reviewed by: Toomas Soome <tsoome@me.com> \n>> Reviewed by: Peter Tribble <peter.tribble@gmail.com> Approved by: \n>> Robert Mustacchi <rm@joyent.com>\n>>\n>>\n>> Source: https://github.com/illumos/illumos-gate.git\n>>\n>> See the second item which only has \"p\".\n>>\n>> This is with LC_ALL=en_US.UTF-8 (same with cs_CZ.UTF-8), with LC_ALL=C \n>> the text is fine.\n>>\n>> I checked various editors (vim, nano, cat), they are all the same.\n>>\n>> `git log` (PAGER is set to `/usr/bin/less -ins`) shows the particular \n>> commit correctly.\n>>\n>> I tested following git version: 2.16.5, 2.19.2, 2.20.1, git.next branch.\n>>\n>> Here's the configure output: \n>> https://paste.ec/paste/oDdydNxZ#r3avan8BL+8lldFMo928cw3eXSServTsSkGPW4jRBOd \n>>\n>>\n>> Michal\n>>\n> \n"},{"id":"366909","messageId":"8c43e31b-01d8-a1c5-d19c-8efd0e5c1714@talktalk.net","threadId":"50233","inReplyTo":"23c60f2f-43ff-94ec-6100-861c655ec80b@startmail.com","subject":"Re: Broken interactive rebase text after some UTF-8 characters","fromName":"Phillip Wood","fromEmail":"phillip.wood@talktalk.net","sentAt":"2019-01-17T11:04:28Z","receivedAt":"2019-01-17T11:04:35Z","isPatch":false,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"Hi Michal\n\nOn 16/01/2019 21:36, Michal Nowak wrote:\n> Hello Phillip, thanks for reaching out.\n> \n> This is what I see with your suggested command:\n> \n> \n> $ GIT_SEQUENCE_EDITOR=cat git rebase -i \n> 08487eea68a2fa501b5042131c6db068089f82e1\n> hint: Waiting for your editor to close the file... pick 1ea94c756c 10202 \n> loader: use screen-#rows to find bottom left co-ordinates Reviewed by: \n> Toomas Soome <tsoome@me.com> Reviewed by: Gergő Mihály Doma \n> <domag02@gmail.com> Approved by: Robert Mustacchi <rm@joyent.com>\n> p\n\nThat's really weird, I'm not sure what's going on. One thing that would \nbe useful would be to check whether the output of\n\ngit log --pretty='%s' --reverse 08487eea68a2fa5..\n\nincludes the stray 'p' or not. I had a look through the configure output \nyou attached to your previous mail and it seems you're building without \niconv and gettext support, I've built git on linux with\nmake NO_ICONV=1 NO_GETTEXT=1\nto try and test that and the rebase todo list was fine. Building with \nconfigure rather than the Makefile that comes with git can be a bit \nflaky sometimes as occasionally when a new compile option gets added to \nthe Makefile updating the configure script gets overlooked but I don't \nknow if that is the problem here or not.\n\nBest Wishes\n\nPhillip\n\n> pick cadd68ea00 10078 smatch fixes for UCB Reviewed by: Andy Fiddaman \n> <omnios@citrus-it.net> Reviewed by: Toomas Soome <tsoome@me.com> \n> Reviewed by: Peter Tribble <peter.tribble@gmail.com> Approved by: Robert \n> Mustacchi <rm@joyent.com>\n> pick 05ede3db5e 10079 smatch Makefile changes for usr/src/lib Reviewed \n> by: Andy Fiddaman <andy@omniosce.org> Approved by: Robert Mustacchi \n> <rm@joyent.com>\n> pick 5661bb7641 10080 smatch Makefile changes for usr/src/cmd Reviewed \n> by: Andy Fiddaman <andy@omniosce.org> Approved by: Robert Mustacchi \n> <rm@joyent.com>\n> pick 15c07adc1c 10081 smatch indenting fixes for usr/src/uts Reviewed \n> by: Toomas Soome <tsoome@me.com> Reviewed by: Peter Tribble \n> <peter.tribble@gmail.com> Reviewed by: Andy Fiddaman <andy@omniosce.org> \n> Approved by: Robert Mustacchi <rm@joyent.com>\n> pick c653bb4713 10197 build smatch in parallel Reviewed by: Andy \n> Fiddaman <andy@omniosce.org> Approved by: Robert Mustacchi <rm@joyent.com>\n> \n> # Rebase 08487eea68..c653bb4713 onto 08487eea68 (6 commands)\n> #\n> # Commands:\n> # p, pick <commit> = use commit\n> # r, reword <commit> = use commit, but edit the commit message\n> # e, edit <commit> = use commit, but stop for amending\n> # s, squash <commit> = use commit, but meld into previous commit\n> # f, fixup <commit> = like \"squash\", but discard this commit's log message\n> # x, exec <command> = run command (the rest of the line) using shell\n> # b, break = stop here (continue rebase later with 'git rebase --continue')\n> # d, drop <commit> = remove commit\n> # l, label <label> = label current HEAD with a name\n> # t, reset <label> = reset HEAD to a label\n> # m, merge [-C <commit> | -c <commit>] <label> [# <oneline>]\n> # .       create a merge commit using the original merge commit's\n> # .       message (or the oneline, if no original merge commit was\n> # .       specified). Use -c <commit> to reword the commit message.\n> #\n> # These lines can be re-ordered; they are executed from top to bottom.\n> #\n> # If you remove a line here THAT COMMIT WILL BE LOST.\n> #\n> # However, if you remove everything, the rebase will be aborted.\n> #\n> # Note that empty commits are commented out\n> error: missing arguments for pick\n> error: invalid line 2: p\n> You can fix this with 'git rebase --edit-todo' and then run 'git rebase \n> --continue'.\n> Or you can abort the rebase with 'git rebase --abort'.\n> \n> \n> Let me know, if there's anything I can try myself to gather more \n> information. Should you want to reproduce this on OpenIndiana on your \n> own, Vagrant is the more straightforward way to deploy OpenIndiana.\n> \n> Thanks,\n> Michal\n> \n> On 01/16/19 11:33 AM, Phillip Wood wrote:\n>> Dear Michal\n>>\n>> Thanks for the bug report, unfortunately I'm unable to reproduce it \n>> here using git 2.20.1 (see below). Knowing a little about how the todo \n>> list is created I cannot think how multibyte characters would break \n>> it. What command line were you using to start the rebase?\n>>\n>> Best Wishes\n>>\n>> Phillip\n>>\n>> $ git clone --shallow-since='11-1-2019' \n>> https://github.com/illumos/illumos-gate.git\n>> Cloning into 'illumos-gate'...\n>> remote: Enumerating objects: 54348, done.\n>> remote: Counting objects: 100% (54348/54348), done.\n>> remote: Compressing objects: 100% (39008/39008), done.\n>> remote: Total 54348 (delta 19671), reused 31187 (delta 11884), \n>> pack-reused 0\n>> Receiving objects: 100% (54348/54348), 148.91 MiB | 1.18 MiB/s, done.\n>> Resolving deltas: 100% (19671/19671), done.\n>> Checking out files: 100% (47352/47352), done.\n>>\n>> $ cd ~/src/illumos-gate\n>> $ git log --oneline -1\n>> f482e26c (HEAD -> master, origin/master, origin/HEAD) 10233 dboot: \n>> process_module() is missing newline at the end of the string Reviewed \n>> by: Sebastian Wiedenroth <wiedi@frubar.net> Reviewed by: Jason King \n>> <jbk@joyent.com> Reviewed by: Andy Stormont \n>> <astormont@racktopsystems.com> Approved by: Robert Mustacchi \n>> <rm@joyent.com>\n>>\n>> $ GIT_SEQUENCE_EDITOR=cat git rebase -i \n>> 08487eea68a2fa501b5042131c6db068089f82e1\n>>\n>> hint: Waiting for your editor to close the file...\n>> pick 1ea94c75 10202 loader: use screen-#rows to find bottom left \n>> co-ordinates Reviewed by: Toomas Soome <tsoome@me.com> Reviewed by: \n>> Gergő Mihály Doma <domag02@gmail.com> Approved by: Robert Mustacchi \n>> <rm@joyent.com>\n>> pick cadd68ea 10078 smatch fixes for UCB Reviewed by: Andy Fiddaman \n>> <omnios@citrus-it.net> Reviewed by: Toomas Soome <tsoome@me.com> \n>> Reviewed by: Peter Tribble <peter.tribble@gmail.com> Approved by: \n>> Robert Mustacchi <rm@joyent.com>\n>> pick 05ede3db 10079 smatch Makefile changes for usr/src/lib Reviewed \n>> by: Andy Fiddaman <andy@omniosce.org> Approved by: Robert Mustacchi \n>> <rm@joyent.com>\n>> pick 5661bb76 10080 smatch Makefile changes for usr/src/cmd Reviewed \n>> by: Andy Fiddaman <andy@omniosce.org> Approved by: Robert Mustacchi \n>> <rm@joyent.com>\n>> pick 15c07adc 10081 smatch indenting fixes for usr/src/uts Reviewed \n>> by: Toomas Soome <tsoome@me.com> Reviewed by: Peter Tribble \n>> <peter.tribble@gmail.com> Reviewed by: Andy Fiddaman \n>> <andy@omniosce.org> Approved by: Robert Mustacchi <rm@joyent.com>\n>> pick c653bb47 10197 build smatch in parallel Reviewed by: Andy \n>> Fiddaman <andy@omniosce.org> Approved by: Robert Mustacchi \n>> <rm@joyent.com>\n>> pick 161294fe 10217 mdb: r9w isn't r8w 10218 CONV_CAP_VAL_HW2_BUFSIZE \n>> wasn't updated 10219 AV_386_2_AVX512VPOPCDQ elfcap name should be \n>> consistent Reviewed by: Dan McDonald <danmcd@joyent.com> Reviewed by: \n>> Rob Johnston <rob.johnston@joyent.com> Reviewed by: Patrick Mooney \n>> <patrick.mooney@joyent.com> Reviewed by: Andy Fiddaman \n>> <andy@omniosce.org> Approved by: Dan McDonald <danmcd@joyent.com>\n>> pick f482e26c 10233 dboot: process_module() is missing newline at the \n>> end of the string Reviewed by: Sebastian Wiedenroth <wiedi@frubar.net> \n>> Reviewed by: Jason King <jbk@joyent.com> Reviewed by: Andy Stormont \n>> <astormont@racktopsystems.com> Approved by: Robert Mustacchi \n>> <rm@joyent.com>\n>>\n>> # Rebase 08487eea..f482e26c onto 08487eea (8 commands)\n>> #\n>> # Commands:\n>> # p, pick <commit> = use commit\n>> # r, reword <commit> = use commit, but edit the commit message\n>> # e, edit <commit> = use commit, but stop for amending\n>> # s, squash <commit> = use commit, but meld into previous commit\n>> # f, fixup <commit> = like \"squash\", but discard this commit's log \n>> message\n>> # x, exec <command> = run command (the rest of the line) using shell\n>> # b, break = stop here (continue rebase later with 'git rebase \n>> --continue')\n>> # d, drop <commit> = remove commit\n>> # l, label <label> = label current HEAD with a name\n>> # t, reset <label> = reset HEAD to a label\n>> # m, merge [-C <commit> | -c <commit>] <label> [# <oneline>]\n>> # .       create a merge commit using the original merge commit's\n>> # .       message (or the oneline, if no original merge commit was\n>> # .       specified). Use -c <commit> to reword the commit message.\n>> #\n>> # These lines can be re-ordered; they are executed from top to bottom.\n>> #\n>> # If you remove a line here THAT COMMIT WILL BE LOST.\n>> #\n>> # However, if you remove everything, the rebase will be aborted.\n>> #\n>> # Note that empty commits are commented out\n>> Successfully rebased and updated refs/heads/master.\n>>\n>>\n>> On 15/01/2019 19:29, Michal Nowak wrote:\n>>> Hello,\n>>>\n>>> on OpenIndiana 2018.10 (illumos kernel) line of the interactive \n>>> rebase text after a particular name (Gergő Mihály Doma) is broken:\n>>>\n>>> pick 1ea94c756c 10202 loader: use screen-#rows to find bottom left \n>>> co-ordinates Reviewed by: Toomas Soome <tsoome@me.com> Reviewed by: \n>>> Gergő Mihály Doma <domag02@gmail.com> Approved by: Robert Mustacchi \n>>> <rm@joyent.com>\n>>> p\n>>> pick cadd68ea00 10078 smatch fixes for UCB Reviewed by: Andy Fiddaman \n>>> <omnios@citrus-it.net> Reviewed by: Toomas Soome <tsoome@me.com> \n>>> Reviewed by: Peter Tribble <peter.tribble@gmail.com> Approved by: \n>>> Robert Mustacchi <rm@joyent.com>\n>>>\n>>>\n>>> Source: https://github.com/illumos/illumos-gate.git\n>>>\n>>> See the second item which only has \"p\".\n>>>\n>>> This is with LC_ALL=en_US.UTF-8 (same with cs_CZ.UTF-8), with \n>>> LC_ALL=C the text is fine.\n>>>\n>>> I checked various editors (vim, nano, cat), they are all the same.\n>>>\n>>> `git log` (PAGER is set to `/usr/bin/less -ins`) shows the particular \n>>> commit correctly.\n>>>\n>>> I tested following git version: 2.16.5, 2.19.2, 2.20.1, git.next branch.\n>>>\n>>> Here's the configure output: \n>>> https://paste.ec/paste/oDdydNxZ#r3avan8BL+8lldFMo928cw3eXSServTsSkGPW4jRBOd \n>>>\n>>>\n>>> Michal\n>>>\n>>\n\n"},{"id":"368244","messageId":"505c2e2e-c9bc-aa57-c498-2acced0b8afa@gmail.com","threadId":"50233","inReplyTo":"8c43e31b-01d8-a1c5-d19c-8efd0e5c1714@talktalk.net","subject":"Re: Broken interactive rebase text after some UTF-8 characters","fromName":"Alban Gruin","fromEmail":"alban.gruin@gmail.com","sentAt":"2019-01-31T17:43:29Z","receivedAt":"2019-01-31T17:43:40Z","isPatch":false,"sender":{"key":"alban.gruin@gmail.com","avatar":"https://avatars.githubusercontent.com/u/6310153?v=4"},"body":"Hi Phillip and Michal,\n\nI think I found the bug.\n\nIf you look at .git/rebase-merge/git-rebase-todo.backup, which is\ncreated before the editor is opened, you can see that it does not have\nthis problem.  In between, transform_todos() is called and causes the\nproblem reported by Michal.\n\nThis seems to be caused by a single line, sequencer:4661 (on b5101f9297,\n\"Fourth batch after 2.20\", 2019-01-29)[1].  If you add just before a\nsomething like this:\n\n    fwrite(item->arg, item->arg_len, sizeof(char), stdout);\n\nYou will see that the argument is properly written to stdout.  But if\nyou write this:\n\n    printf(\"%.*s\\n\", item->arg_len, item->arg);\n\nYou will have the same broken output as in the todo file.\n\nAre we misusing C formats?\n\n[1] https://github.com/git/git/blob/master/sequencer.c#L4661\n\nCheers,\nAlban\n\n\n"},{"id":"368266","messageId":"2cbb5818-643d-bafd-6721-91e0d291a5fd@talktalk.net","threadId":"50233","inReplyTo":"505c2e2e-c9bc-aa57-c498-2acced0b8afa@gmail.com","subject":"Re: Broken interactive rebase text after some UTF-8 characters","fromName":"Phillip Wood","fromEmail":"phillip.wood@talktalk.net","sentAt":"2019-01-31T20:40:43Z","receivedAt":"2019-01-31T20:40:47Z","isPatch":false,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"Hi Alban\n\nOn 31/01/2019 17:43, Alban Gruin wrote:\n> Hi Phillip and Michal,\n> \n> I think I found the bug.\n\nGood find! Which os are you on?\n> \n> If you look at .git/rebase-merge/git-rebase-todo.backup, which is\n> created before the editor is opened, you can see that it does not have\n> this problem.  In between, transform_todos() is called and causes the\n> problem reported by Michal.\n> \n> This seems to be caused by a single line, sequencer:4661 (on b5101f9297,\n> \"Fourth batch after 2.20\", 2019-01-29)[1].  If you add just before a\n> something like this:\n> \n>      fwrite(item->arg, item->arg_len, sizeof(char), stdout);\n> \n> You will see that the argument is properly written to stdout.  But if\n> you write this:\n> \n>      printf(\"%.*s\\n\", item->arg_len, item->arg);\n> \n> You will have the same broken output as in the todo file.\n> \n> Are we misusing C formats?\n\nThe C standard and POSIX both say that the * refers to the maximum \nnumber of bytes to print but it looks like it is being treated as the \nmaximum number of characters on OpenIndiana.\n\nJohannes - Perhaps we should change it to use fwrite() unless printf() \ngets fixed and we're sure no other operating systems are affected?\n\n[1] http://pubs.opengroup.org/onlinepubs/9699919799/functions/fprintf.html\n[2] http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1548.pdf page 309\n\nBest Wishes\n\nPhillip\n\n> [1] https://github.com/git/git/blob/master/sequencer.c#L4661\n> \n> Cheers,\n> Alban\n> \n> \n\n"},{"id":"368271","messageId":"b3360111-df25-bb11-b895-15418e58d901@gmail.com","threadId":"50233","inReplyTo":"2cbb5818-643d-bafd-6721-91e0d291a5fd@talktalk.net","subject":"Re: Broken interactive rebase text after some UTF-8 characters","fromName":"Alban Gruin","fromEmail":"alban.gruin@gmail.com","sentAt":"2019-01-31T21:00:15Z","receivedAt":"2019-01-31T21:00:22Z","isPatch":false,"sender":{"key":"alban.gruin@gmail.com","avatar":"https://avatars.githubusercontent.com/u/6310153?v=4"},"body":"Hi Phillip,\n\nLe 31/01/2019 à 21:40, Phillip Wood a écrit :\n> Hi Alban\n> \n> On 31/01/2019 17:43, Alban Gruin wrote:\n>> Hi Phillip and Michal,\n>>\n>> I think I found the bug.\n> \n> Good find! Which os are you on?\n\nI’m on Linux, but I found this on OpenIndiana running under QEMU.\n\n-- Alban\n\n"},{"id":"368272","messageId":"xmqqimy41pdu.fsf@gitster-ct.c.googlers.com","threadId":"50233","inReplyTo":"2cbb5818-643d-bafd-6721-91e0d291a5fd@talktalk.net","subject":"Re: Broken interactive rebase text after some UTF-8 characters","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-01-31T21:35:57Z","receivedAt":"2019-01-31T21:36:03Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Phillip Wood <phillip.wood@talktalk.net> writes:\n\n>> Are we misusing C formats?\n>\n> The C standard and POSIX both say that the * refers to the maximum\n> number of bytes to print but it looks like it is being treated as the\n> maximum number of characters on OpenIndiana.\n>\n> Johannes - Perhaps we should change it to use fwrite() unless printf()\n> gets fixed and we're sure no other operating systems are affected?\n\nAvoid such a rewrite, as \"%*.s\" that takes (int, char *) are used in\nmany other places in our codebase, if you can.\n\n"},{"id":"368307","messageId":"nycvar.QRO.7.76.6.1902010835210.41@tvgsbejvaqbjf.bet","threadId":"50233","inReplyTo":"xmqqimy41pdu.fsf@gitster-ct.c.googlers.com","subject":"Re: Broken interactive rebase text after some UTF-8 characters","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2019-02-01T07:38:56Z","receivedAt":"2019-02-01T07:39:10Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Thu, 31 Jan 2019, Junio C Hamano wrote:\n\n> Phillip Wood <phillip.wood@talktalk.net> writes:\n> \n> >> Are we misusing C formats?\n> >\n> > The C standard and POSIX both say that the * refers to the maximum\n> > number of bytes to print but it looks like it is being treated as the\n> > maximum number of characters on OpenIndiana.\n> >\n> > Johannes - Perhaps we should change it to use fwrite() unless printf()\n> > gets fixed and we're sure no other operating systems are affected?\n> \n> Avoid such a rewrite, as \"%*.s\" that takes (int, char *) are used in\n> many other places in our codebase, if you can.\n\nYes, this would be painful in particular in cases like\n\n\tmaster:advice.c:101:           fprintf(stderr, _(\"%shint: %.*s%s\\n\"),\n\nwhere we want to write more than just a variable-length buffer.\n\nI am curious: is libintl (gettext) used on OpenIndiana? I ask because\nAFAIR fprintf() is overridden in that case, and the bug might be a lot\neasier to fix if it is in libintl rather than in libc.\n\nOf course, it might *still* be a bug in libc by virtue of handing '%.*s'\nthrough to libc's implementation.\n\nAlban, can you test this with NO_GETTEXT?\n\nThanks,\nJohannes\n"},{"id":"368316","messageId":"747726ae27ff52509f831c9615f2b102.startmail@startmail.com","threadId":"50233","inReplyTo":"nycvar.QRO.7.76.6.1902010835210.41@tvgsbejvaqbjf.bet","subject":"Re: Broken interactive rebase text after some UTF-8 characters","fromName":"Michal Nowak","fromEmail":"mnowak@startmail.com","sentAt":"2019-02-01T09:06:29Z","receivedAt":"2019-02-01T09:06:33Z","isPatch":false,"sender":{"key":"mnowak@startmail.com","avatar":null},"body":"Johannes,\n\nOn Friday, February 1, 2019 at 8:38 AM, Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:\n> Hi,\n> \n> On Thu, 31 Jan 2019, Junio C Hamano wrote:\n> \n>> Phillip Wood <phillip.wood@talktalk.net> writes:\n>>\n>> >> Are we misusing C formats?\n>> >\n>> > The C standard and POSIX both say that the * refers to the maximum\n>> > number of bytes to print but it looks like it is being treated as the\n>> > maximum number of characters on OpenIndiana.\n>> >\n>> > Johannes - Perhaps we should change it to use fwrite() unless\n>> printf()\n>> > gets fixed and we're sure no other operating systems are affected?\n>>\n>> Avoid such a rewrite, as \"%*.s\" that takes (int, char *) are used in\n>> many other places in our codebase, if you can.\n> \n> Yes, this would be painful in particular in cases like\n> \n> \tmaster:advice.c:101:           fprintf(stderr, _(\"%shint: %.*s%s\\n\"),\n> \n> where we want to write more than just a variable-length buffer.\n> \n> I am curious: is libintl (gettext) used on OpenIndiana? I ask because\n> AFAIR fprintf() is overridden in that case, and the bug might be a lot\n> easier to fix if it is in libintl rather than in libc.\n\nhere you can see the full output of the OpenIndiana git build: https://hipster.openindiana.org/logs/oi-userland/latest/git.publish.log.\n\nFrom what I see there, libintl was found.\n\nIf you believe this is illumos libc bug, it would be cool if someone created an simple testcase, which I can forward to the illumos developers.\n\nThanks,\nMichal\n\n> \n> Of course, it might *still* be a bug in libc by virtue of handing '%.*s'\n> through to libc's implementation.\n> \n> Alban, can you test this with NO_GETTEXT?\n> \n> Thanks,\n> Johannes\n"},{"id":"368329","messageId":"nycvar.QRO.7.76.6.1902011530440.41@tvgsbejvaqbjf.bet","threadId":"50233","inReplyTo":"747726ae27ff52509f831c9615f2b102.startmail@startmail.com","subject":"Re: Broken interactive rebase text after some UTF-8 characters","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2019-02-01T14:33:55Z","receivedAt":"2019-02-01T14:34:09Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Michal,\n\nOn Fri, 1 Feb 2019, Michal Nowak wrote:\n\n> On Friday, February 1, 2019 at 8:38 AM, Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:\n> > \n> > On Thu, 31 Jan 2019, Junio C Hamano wrote:\n> > \n> >> Phillip Wood <phillip.wood@talktalk.net> writes:\n> >>\n> >> >> Are we misusing C formats?\n> >> >\n> >> > The C standard and POSIX both say that the * refers to the maximum\n> >> > number of bytes to print but it looks like it is being treated as the\n> >> > maximum number of characters on OpenIndiana.\n> >> >\n> >> > Johannes - Perhaps we should change it to use fwrite() unless\n> >> printf()\n> >> > gets fixed and we're sure no other operating systems are affected?\n> >>\n> >> Avoid such a rewrite, as \"%*.s\" that takes (int, char *) are used in\n> >> many other places in our codebase, if you can.\n> > \n> > Yes, this would be painful in particular in cases like\n> > \n> > \tmaster:advice.c:101:           fprintf(stderr, _(\"%shint: %.*s%s\\n\"),\n> > \n> > where we want to write more than just a variable-length buffer.\n> > \n> > I am curious: is libintl (gettext) used on OpenIndiana? I ask because\n> > AFAIR fprintf() is overridden in that case, and the bug might be a lot\n> > easier to fix if it is in libintl rather than in libc.\n> \n> here you can see the full output of the OpenIndiana git build: https://hipster.openindiana.org/logs/oi-userland/latest/git.publish.log.\n> \n> From what I see there, libintl was found.\n> \n> If you believe this is illumos libc bug, it would be cool if someone created an simple testcase, which I can forward to the illumos developers.\n\nYou already have that example. Just take the UTF-8 text in your original\nbug report, put it into something like\n\n\tint main(int argc, char **argv)\n\t{\n\t\tchar utf8[] = \"... your text here...\";\n\n\t\tprintf(\"%.*s\", (int)(sizeof(utf8) - 1), utf8);\n\n\t\treturn 0;\n\t}\n\nYou should first verify, though, that this replicate the problem, and if\nit does not, use libintl (I think you have to `#include <gettext.h>` and\n`-lintl` or some such) and see whether that reproduces your problem.\n\nCiao,\nJohannes\n\n> \n> Thanks,\n> Michal\n> \n> > \n> > Of course, it might *still* be a bug in libc by virtue of handing '%.*s'\n> > through to libc's implementation.\n> > \n> > Alban, can you test this with NO_GETTEXT?\n> > \n> > Thanks,\n> > Johannes\n> \n"},{"id":"368334","messageId":"a3f4a6d2-7461-709a-6d1a-097785bb8664@gmail.com","threadId":"50233","inReplyTo":"nycvar.QRO.7.76.6.1902010835210.41@tvgsbejvaqbjf.bet","subject":"Re: Broken interactive rebase text after some UTF-8 characters","fromName":"Alban Gruin","fromEmail":"alban.gruin@gmail.com","sentAt":"2019-02-01T16:13:13Z","receivedAt":"2019-02-01T16:13:33Z","isPatch":false,"sender":{"key":"alban.gruin@gmail.com","avatar":"https://avatars.githubusercontent.com/u/6310153?v=4"},"body":"Hi Johannes,\n\nLe 01/02/2019 à 08:38, Johannes Schindelin a écrit :\n> Hi,\n> \n> On Thu, 31 Jan 2019, Junio C Hamano wrote:\n> \n>> Phillip Wood <phillip.wood@talktalk.net> writes:\n>>\n>>>> Are we misusing C formats?\n>>>\n>>> The C standard and POSIX both say that the * refers to the maximum\n>>> number of bytes to print but it looks like it is being treated as the\n>>> maximum number of characters on OpenIndiana.\n>>>\n>>> Johannes - Perhaps we should change it to use fwrite() unless printf()\n>>> gets fixed and we're sure no other operating systems are affected?\n>>\n>> Avoid such a rewrite, as \"%*.s\" that takes (int, char *) are used in\n>> many other places in our codebase, if you can.\n> \n> Yes, this would be painful in particular in cases like\n> \n> \tmaster:advice.c:101:           fprintf(stderr, _(\"%shint: %.*s%s\\n\"),\n> \n> where we want to write more than just a variable-length buffer.\n> \n> I am curious: is libintl (gettext) used on OpenIndiana? I ask because\n> AFAIR fprintf() is overridden in that case, and the bug might be a lot\n> easier to fix if it is in libintl rather than in libc.\n> \n> Of course, it might *still* be a bug in libc by virtue of handing '%.*s'\n> through to libc's implementation.\n> \n> Alban, can you test this with NO_GETTEXT?\n\nSure. :)\n\nThe bug no longer happens when git is built with NO_GETTEXT.  All is\nworking as expected.\n\n> \n> Thanks,\n> Johannes\n> \n\nCheers,\nAlban\n\n"},{"id":"368335","messageId":"c71f63cb-d956-96f5-4770-581a3b6b5543@gmail.com","threadId":"50233","inReplyTo":"747726ae27ff52509f831c9615f2b102.startmail@startmail.com","subject":"Re: Broken interactive rebase text after some UTF-8 characters","fromName":"Alban Gruin","fromEmail":"alban.gruin@gmail.com","sentAt":"2019-02-01T16:15:48Z","receivedAt":"2019-02-01T16:15:53Z","isPatch":false,"sender":{"key":"alban.gruin@gmail.com","avatar":"https://avatars.githubusercontent.com/u/6310153?v=4"},"body":"Hi Michal,\n\nLe 01/02/2019 à 10:06, Michal Nowak a écrit :\n> Johannes,\n> \n> On Friday, February 1, 2019 at 8:38 AM, Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:\n>> Hi,\n>>\n>> On Thu, 31 Jan 2019, Junio C Hamano wrote:\n>>\n>>> Phillip Wood <phillip.wood@talktalk.net> writes:\n>>>\n>>>>> Are we misusing C formats?\n>>>>\n>>>> The C standard and POSIX both say that the * refers to the maximum\n>>>> number of bytes to print but it looks like it is being treated as the\n>>>> maximum number of characters on OpenIndiana.\n>>>>\n>>>> Johannes - Perhaps we should change it to use fwrite() unless\n>>> printf()\n>>>> gets fixed and we're sure no other operating systems are affected?\n>>>\n>>> Avoid such a rewrite, as \"%*.s\" that takes (int, char *) are used in\n>>> many other places in our codebase, if you can.\n>>\n>> Yes, this would be painful in particular in cases like\n>>\n>> \tmaster:advice.c:101:           fprintf(stderr, _(\"%shint: %.*s%s\\n\"),\n>>\n>> where we want to write more than just a variable-length buffer.\n>>\n>> I am curious: is libintl (gettext) used on OpenIndiana? I ask because\n>> AFAIR fprintf() is overridden in that case, and the bug might be a lot\n>> easier to fix if it is in libintl rather than in libc.\n> \n> here you can see the full output of the OpenIndiana git build: https://hipster.openindiana.org/logs/oi-userland/latest/git.publish.log.\n> \n> From what I see there, libintl was found.\n> \n> If you believe this is illumos libc bug, it would be cool if someone created an simple testcase, which I can forward to the illumos developers.\n> \n\nI attached a test case to this email.  You can build it with `gcc\ntest-case.c', and run it with `./a.out'.\n\nOutput on my Linux system:\n\n    Before setting locale:\n    Expected output:\n    áaaa\n    Actual output:\n    áaaa\n\n    After setting locale:\n    Expected output:\n    áaaa\n    Actual output:\n    áaaa\n\nOutput on an OpenIndiana system:\n\n    Before setting locale:\n    Expected output:\n    áaaa\n    Actual output:\n    áaaa\n\n    After setting locale:\n    Expected output:\n    áaaa\n    Actual output:\n    áaaaa\n\n> Thanks,\n> Michal\n> \n>>\n>> Of course, it might *still* be a bug in libc by virtue of handing '%.*s'\n>> through to libc's implementation.\n>>\n>> Alban, can you test this with NO_GETTEXT?\n>>\n>> Thanks,\n>> Johannes\n\nCheers,\nAlban\n\n\n\n/*\n * Test case for OpenIndiana '%.*s' bug\n * Build with `gcc test-case.c'\n * Run with `./a.out'\n */\n\n#include <stdlib.h>\n#include <stdio.h>\n#include <string.h>\n\n#include <locale.h>\n#include <libintl.h>\n\nstatic void compare_output(const char *str, int len) {\n\tputs(\"Expected output:\");\n\tfwrite(str, len, sizeof(char), stdout);\n\n\tputs(\"\\nActual output:\");\n\tprintf(\"%.*s\\n\", len, str);\n}\n\nint main(int argc, char **argv) {\n\tchar buf[] = \"áaaaaaa\";\n\n\tputs(\"Before setting locale:\");\n\tcompare_output(buf, 5);\n\n\tsetlocale(LC_ALL, \"\");\n\n\tputs(\"\\nAfter setting locale:\");\n\tcompare_output(buf, 5);\n\n\treturn EXIT_SUCCESS;\n}\n"},{"id":"368343","messageId":"eda7b1c4-5fd6-5440-8998-75eab9f73e47@startmail.com","threadId":"50233","inReplyTo":"nycvar.QRO.7.76.6.1902011530440.41@tvgsbejvaqbjf.bet","subject":"Re: Broken interactive rebase text after some UTF-8 characters","fromName":"Michal Nowak","fromEmail":"mnowak@startmail.com","sentAt":"2019-02-01T16:24:26Z","receivedAt":"2019-02-01T16:24:39Z","isPatch":false,"sender":{"key":"mnowak@startmail.com","avatar":null},"body":"On 02/01/19 03:33 PM, Johannes Schindelin wrote:\n> Hi Michal,\n> \n> On Fri, 1 Feb 2019, Michal Nowak wrote:\n> \n>> On Friday, February 1, 2019 at 8:38 AM, Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:\n>>>\n>>> On Thu, 31 Jan 2019, Junio C Hamano wrote:\n>>>\n>>>> Phillip Wood <phillip.wood@talktalk.net> writes:\n>>>>\n>>>>>> Are we misusing C formats?\n>>>>>\n>>>>> The C standard and POSIX both say that the * refers to the maximum\n>>>>> number of bytes to print but it looks like it is being treated as the\n>>>>> maximum number of characters on OpenIndiana.\n>>>>>\n>>>>> Johannes - Perhaps we should change it to use fwrite() unless\n>>>> printf()\n>>>>> gets fixed and we're sure no other operating systems are affected?\n>>>>\n>>>> Avoid such a rewrite, as \"%*.s\" that takes (int, char *) are used in\n>>>> many other places in our codebase, if you can.\n>>>\n>>> Yes, this would be painful in particular in cases like\n>>>\n>>> \tmaster:advice.c:101:           fprintf(stderr, _(\"%shint: %.*s%s\\n\"),\n>>>\n>>> where we want to write more than just a variable-length buffer.\n>>>\n>>> I am curious: is libintl (gettext) used on OpenIndiana? I ask because\n>>> AFAIR fprintf() is overridden in that case, and the bug might be a lot\n>>> easier to fix if it is in libintl rather than in libc.\n>>\n>> here you can see the full output of the OpenIndiana git build: https://hipster.openindiana.org/logs/oi-userland/latest/git.publish.log.\n>>\n>>  From what I see there, libintl was found.\n>>\n>> If you believe this is illumos libc bug, it would be cool if someone created an simple testcase, which I can forward to the illumos developers.\n> \n> You already have that example. Just take the UTF-8 text in your original\n> bug report, put it into something like\n> \n> \tint main(int argc, char **argv)\n> \t{\n> \t\tchar utf8[] = \"... your text here...\";\n> \n> \t\tprintf(\"%.*s\", (int)(sizeof(utf8) - 1), utf8);\n> \n> \t\treturn 0;\n> \t}\n> \n> You should first verify, though, that this replicate the problem, and if\n> it does not, use libintl (I think you have to `#include <gettext.h>` and\n> `-lintl` or some such) and see whether that reproduces your problem.\n\nThank you, Johannes for the test case.\n\nHowever, I don't see any problem with the output on OpenIndiana:\n\n{global} newman@lenovo:~ $ cat printf.c\n#include <stdio.h>\n//#include <gettext.h>\nint main(int argc, char **argv) {\n   char utf8[] = \"Gergő Mihály Doma\\n\";\n   printf(\"%.*s\", (int)(sizeof(utf8) - 1), utf8);\n   return 0;\n}\n\n{global} newman@lenovo:~ $ gcc printf.c -o printf && ./printf\nGergő Mihály Doma\n\nEnabled the gettext header in the source file.\n\n{global} newman@lenovo:~ $ gcc printf.c -o printf_intl -lintl \n-I/usr/share/gettext/ && ./printf_intl\nGergő Mihály Doma\n\n{global} newman@lenovo:~ $ ldd printf printf_intl\nprintf:\n         libc.so.1 =>     /lib/libc.so.1\n         libm.so.2 =>     /lib/libm.so.2\nprintf_intl:\n         libintl.so.1 =>  /lib/libintl.so.1\n         libc.so.1 =>     /lib/libc.so.1\n         libm.so.2 =>     /lib/libm.so.2\n\n{global} newman@lenovo:~ $ locale\nLANG=en_US.UTF-8\nLC_CTYPE=\"en_US.UTF-8\"\nLC_NUMERIC=\"en_US.UTF-8\"\nLC_TIME=\"en_US.UTF-8\"\nLC_COLLATE=\"en_US.UTF-8\"\nLC_MONETARY=\"en_US.UTF-8\"\nLC_MESSAGES=\"en_US.UTF-8\"\nLC_ALL=\n\n\nI even tried more arcane characters from \nhttps://www.w3.org/2001/06/utf-8-test/UTF-8-demo.html but they are \ndisplayed correctly as well.\n\nMichal\n\n> \n> Ciao,\n> Johannes\n> \n>>\n>> Thanks,\n>> Michal\n>>\n>>>\n>>> Of course, it might *still* be a bug in libc by virtue of handing '%.*s'\n>>> through to libc's implementation.\n>>>\n>>> Alban, can you test this with NO_GETTEXT?\n>>>\n>>> Thanks,\n>>> Johannes\n>>\n"},{"id":"368344","messageId":"xmqqpnsbzaa6.fsf@gitster-ct.c.googlers.com","threadId":"50233","inReplyTo":"eda7b1c4-5fd6-5440-8998-75eab9f73e47@startmail.com","subject":"Re: Broken interactive rebase text after some UTF-8 characters","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-02-01T17:30:25Z","receivedAt":"2019-02-01T17:30:32Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Michal Nowak <mnowak@startmail.com> writes:\n\n>> You already have that example. Just take the UTF-8 text in your original\n>> bug report, put it into something like\n>>\n>> \tint main(int argc, char **argv)\n>> \t{\n>> \t\tchar utf8[] = \"... your text here...\";\n>>\n>> \t\tprintf(\"%.*s\", (int)(sizeof(utf8) - 1), utf8);\n>>\n>> \t\treturn 0;\n>> \t}\n\nWhen replayed literally, this is not a very good test.\n\n> {global} newman@lenovo:~ $ cat printf.c\n> #include <stdio.h>\n> //#include <gettext.h>\n> int main(int argc, char **argv) {\n>   char utf8[] = \"Gergő Mihály Doma\\n\";\n>   printf(\"%.*s\", (int)(sizeof(utf8) - 1), utf8);\n>   return 0;\n> }\n\nAnd this is replaying it literally.\n\nThe current working suspicion in this thread is that the platform\nprintf(\"%.*s\", num, str) emits up to num \"characters\" starting at\nstr, which is an incorrect implementation, as it should emit up to\nnum \"bytes\".  \n\nNotice that the num in this case is the byte count of that utf8[]\nstring.  That number is always larger than the number of \"characters\"\nfor a string with multi-byte character(s) in it.  Let's say that the\nsample string has N \"characters\", and it is N+X \"bytes\" long, where\nX > 1.\n\nIf the suspicion is correct, i.e. the way the printf implementation\nis broken on this platform is that it shows up to num \"characters\",\nthen the call is asking to show up to N+X \"characters\".  The buggy\nprintf shows all the available N \"characters\", notices the string\nstops there, and finishes.  So you won't _see_ the bug with that\ntest program.\n\nInstead, use something like this.\n\n        #include <stdio.h>\n\n        int main(int ac, char **av)\n        {\n                char utf8[] = \"ふabc\";\n                printf(\"%.*s\\n\", 4, utf8);\n                return 0;\n        }\n\nWith or without gettext or i18n, the output must end with 'a' followed\nby a newline, and you must not see 'b' nor 'c'.  Otherwise your printf\nis broken.\n"},{"id":"368364","messageId":"7fe4edcd-f4ec-3ad5-a9ea-e4015e4afc37@startmail.com","threadId":"50233","inReplyTo":"xmqqpnsbzaa6.fsf@gitster-ct.c.googlers.com","subject":"Re: Broken interactive rebase text after some UTF-8 characters","fromName":"Michal Nowak","fromEmail":"mnowak@startmail.com","sentAt":"2019-02-01T19:00:37Z","receivedAt":"2019-02-01T19:00:43Z","isPatch":false,"sender":{"key":"mnowak@startmail.com","avatar":null},"body":"On 02/01/19 06:30 PM, Junio C Hamano wrote:\n> Michal Nowak <mnowak@startmail.com> writes:\n> \n>>> You already have that example. Just take the UTF-8 text in your original\n>>> bug report, put it into something like\n>>>\n>>> \tint main(int argc, char **argv)\n>>> \t{\n>>> \t\tchar utf8[] = \"... your text here...\";\n>>>\n>>> \t\tprintf(\"%.*s\", (int)(sizeof(utf8) - 1), utf8);\n>>>\n>>> \t\treturn 0;\n>>> \t}\n> \n> When replayed literally, this is not a very good test.\n> \n>> {global} newman@lenovo:~ $ cat printf.c\n>> #include <stdio.h>\n>> //#include <gettext.h>\n>> int main(int argc, char **argv) {\n>>    char utf8[] = \"Gergő Mihály Doma\\n\";\n>>    printf(\"%.*s\", (int)(sizeof(utf8) - 1), utf8);\n>>    return 0;\n>> }\n> \n> And this is replaying it literally.\n> \n> The current working suspicion in this thread is that the platform\n> printf(\"%.*s\", num, str) emits up to num \"characters\" starting at\n> str, which is an incorrect implementation, as it should emit up to\n> num \"bytes\".\n> \n> Notice that the num in this case is the byte count of that utf8[]\n> string.  That number is always larger than the number of \"characters\"\n> for a string with multi-byte character(s) in it.  Let's say that the\n> sample string has N \"characters\", and it is N+X \"bytes\" long, where\n> X > 1.\n> \n> If the suspicion is correct, i.e. the way the printf implementation\n> is broken on this platform is that it shows up to num \"characters\",\n> then the call is asking to show up to N+X \"characters\".  The buggy\n> printf shows all the available N \"characters\", notices the string\n> stops there, and finishes.  So you won't _see_ the bug with that\n> test program.\n> \n> Instead, use something like this.\n> \n>          #include <stdio.h>\n> \n>          int main(int ac, char **av)\n>          {\n>                  char utf8[] = \"ふabc\";\n>                  printf(\"%.*s\\n\", 4, utf8);\n>                  return 0;\n>          }\n> \n> With or without gettext or i18n, the output must end with 'a' followed\n> by a newline, and you must not see 'b' nor 'c'.  Otherwise your printf\n> is broken.\n> \n\nThe output ends with 'a' for me on OpenIndiana:\n\n{global} newman@lenovo:~ $ gcc tp.c -o tp && ./tp\nふa\n\nThough, I can reproduce the problem with Alban's test case the exact \nsame way he wrote earlier today.\n\nMichal\n"}]}