{"thread":{"id":"50962","subject":"gettext, multiple Preferred languages, and English","startedAt":"2019-04-21T11:08:13Z","lastAt":"2019-04-23T11:45:51Z","messageCount":9,"participants":["Andrew Janke","Philip Oakley","Junio C Hamano","Duy Nguyen","Ævar Arnfjörð Bjarmason"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"374193","messageId":"d001a2b5-57c3-1eb3-70fd-679919bb2eb6@apjanke.net","threadId":"50962","inReplyTo":null,"subject":"gettext, multiple Preferred languages, and English","fromName":"Andrew Janke","fromEmail":"floss@apjanke.net","sentAt":"2019-04-21T11:08:09Z","receivedAt":"2019-04-21T11:08:13Z","isPatch":false,"sender":{"key":"floss@apjanke.net","avatar":"https://gravatar.com/avatar/cce96357f6822eb4d08552395bda5a987bf4aa3e98eb42ad062ea8cf352c514b?d=mp&s=160"},"body":"Hi, Git folks,\n\nThis is a follow-up to https://marc.info/?l=git&m=154757938429747&w=2.\n\nWith the current git 2.21.0, some users, including myself, are still\nhaving problems with git selecting the \"wrong\" language for localization.\n\nThis happens on macOS in the situation where:\n* The user has multiple Preferred languages defined in Language & Region\nsystem preferences\n* English is set as the Primary language\n* Another language, for which git has a .po translation file defined, is\nset as another Preferred language, for example, Spanish\n* Environment variable $LANG is unset\n* git was built with gettext support enabled\n\nIn this situation, when git is run, it will use the translations from\nthe secondary Preferred language instead of displaying messages in\nEnglish, the Primary language.\n\nI've seen this situation with other gettext-enabled applications before.\nI believe what's happening is that when selecting the language to use,\ngettext goes through the Preferred languages in order, looking for a .po\ntranslation file for each. It does not find one for English, but it does\nfind one for Spanish, so it uses that, instead of falling back to the\nnon-translated message strings.\n\nSome examples of this happening in the wild:\n-\nhttps://stackoverflow.com/questions/55145901/force-git-to-use-the-default-system-language/55160216\n- https://github.com/Homebrew/homebrew-core/issues/37331\n- https://github.com/Homebrew/homebrew-core/issues/31980\n\nI think an easy fix for this would be to add an \"en.po\" translation\nfile, so that when gettext does its translation selection, it finds that\nfirst for when English is the Primary language (or a Preferred language\nearlier in the order than other languages), and uses it. This .po file\nwould be an \"identity\" translation where the translated strings are all\njust the same as the original strings. I don't think it would even have\nto be actively maintained, because for new message strings that aren't\nincluded in the .po file, it would fall back to the non-translated input\nstrings, which are in English anyway, which is the desired behavior.\n\nThis would be a convenience for git users, because it would \"just work\"\nwithout any modifications to the configure/build process, or requiring\nusers to force a $LANG setting.\n\nWould you consider adding this?\n\nI've put together a patch that does this:\nhttps://github.com/apjanke/git/tree/english-dummy-translation\nhttps://github.com/apjanke/git/commit/7e6704167018e1d47399af04230521927991811b\nNot attaching a patch because it's kind of a large file. I have tested\nit locally and it fixes the language selection problem for me. I'm not\nsure if the appropriate thing to do is make a PR for this to the\ngit-l10n/git-po GitHub repo or not.\n\nCheers,\nAndrew Janke\n"},{"id":"374195","messageId":"bd5ee770-a213-b663-208c-c9980a738fe9@talktalk.net","threadId":"50962","inReplyTo":"d001a2b5-57c3-1eb3-70fd-679919bb2eb6@apjanke.net","subject":"Re: gettext, multiple Preferred languages, and English","fromName":"Philip Oakley","fromEmail":"philipoakley@talktalk.net","sentAt":"2019-04-21T12:59:48Z","receivedAt":"2019-04-21T12:59:55Z","isPatch":false,"sender":{"key":"philipoakley@talktalk.net","avatar":null},"body":"Hi Andrew,\n\nOn 21/04/2019 12:08, Andrew Janke wrote:\nhttps://public-inbox.org/git/d001a2b5-57c3-1eb3-70fd-679919bb2eb6@apjanke.net/\n> I don't think it would even have\n> to be actively maintained, because for new message strings that aren't\n> included in the .po file, it would fall back to the non-translated input\n> strings, which are in English anyway, which is the desired behavior.\nGiven the above comment, could the en.po file \n(https://github.com/apjanke/git/blob/english-dummy-translation/po/en.po) \nbe some very very short version with only one 'translated' string? This \nmay be a way-off comment, but if it could be such a simple maintenance \nfree file then that sounds sensible.\n\nalso adding in Jiang Xin <worldhello.net@gmail.com>, the coordinator for \nextra comment.\n\nPhilip\n"},{"id":"374199","messageId":"ffac5d4b-190c-16d6-497c-4b0f1f12115c@apjanke.net","threadId":"50962","inReplyTo":"bd5ee770-a213-b663-208c-c9980a738fe9@talktalk.net","subject":"Re: gettext, multiple Preferred languages, and English","fromName":"Andrew Janke","fromEmail":"floss@apjanke.net","sentAt":"2019-04-21T13:27:25Z","receivedAt":"2019-04-21T13:27:28Z","isPatch":false,"sender":{"key":"floss@apjanke.net","avatar":"https://gravatar.com/avatar/cce96357f6822eb4d08552395bda5a987bf4aa3e98eb42ad062ea8cf352c514b?d=mp&s=160"},"body":"\nOn 4/21/19 8:59 AM, Philip Oakley wrote:\n> Hi Andrew,\n> \n> On 21/04/2019 12:08, Andrew Janke wrote:\n> https://public-inbox.org/git/d001a2b5-57c3-1eb3-70fd-679919bb2eb6@apjanke.net/\n> \n>> I don't think it would even have\n>> to be actively maintained, because for new message strings that aren't\n>> included in the .po file, it would fall back to the non-translated input\n>> strings, which are in English anyway, which is the desired behavior.\n> Given the above comment, could the en.po file\n> (https://github.com/apjanke/git/blob/english-dummy-translation/po/en.po)\n> be some very very short version with only one 'translated' string?\n\nYes, I believe so. I only provided a full translation file because it\nwas trivial for me to create using the \"msginit\" instructions I found in\npo/README. Since all the translations are just identity relationships, I\nbelieve that is effectively the same as their not being there in the\nfirst place.\n\nI tested your approach locally, and it seems to work for me.\n\n> This\n> may be a way-off comment, but if it could be such a simple maintenance\n> free file then that sounds sensible.\n> \n> also adding in Jiang Xin <worldhello.net@gmail.com>, the coordinator for\n> extra comment.\n> \n> Philip\n"},{"id":"374207","messageId":"xmqqbm0yappo.fsf@gitster-ct.c.googlers.com","threadId":"50962","inReplyTo":"bd5ee770-a213-b663-208c-c9980a738fe9@talktalk.net","subject":"Re: gettext, multiple Preferred languages, and English","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-04-21T23:44:19Z","receivedAt":"2019-04-21T23:44:24Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Philip Oakley <philipoakley@talktalk.net> writes:\n\n> On 21/04/2019 12:08, Andrew Janke wrote:\n> https://public-inbox.org/git/d001a2b5-57c3-1eb3-70fd-679919bb2eb6@apjanke.net/\n>> I don't think it would even have\n>> to be actively maintained, because for new message strings that aren't\n>> included in the .po file, it would fall back to the non-translated input\n>> strings, which are in English anyway, which is the desired behavior.\n> Given the above comment, could the en.po file\n> (https://github.com/apjanke/git/blob/english-dummy-translation/po/en.po)\n> be some very very short version with only one 'translated' string?\n\nOr use LC_ALL=C and be done with it?\n\n> This may be a way-off comment, but if it could be such a simple\n> maintenance free file then that sounds sensible.\n>\n> also adding in Jiang Xin <worldhello.net@gmail.com>, the coordinator\n> for extra comment.\n>\n> Philip\n"},{"id":"374211","messageId":"CACsJy8C1w0zJm71KFb21MC+c2NAGnRLDtE3KNK21hO7U4Ax7Hg@mail.gmail.com","threadId":"50962","inReplyTo":"d001a2b5-57c3-1eb3-70fd-679919bb2eb6@apjanke.net","subject":"Re: gettext, multiple Preferred languages, and English","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2019-04-22T00:35:56Z","receivedAt":"2019-04-22T00:38:35Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Sun, Apr 21, 2019 at 6:40 PM Andrew Janke <floss@apjanke.net> wrote:\n>\n> Hi, Git folks,\n>\n> This is a follow-up to https://marc.info/?l=git&m=154757938429747&w=2.\n\nThis says the problem with \"en\" detection has been fixed. Would\nupgrading gettext fix it?\n\nYou would need to upgrade something (git or gettext) and if it's\nalready fixed in gettext I don't see why we need a workaround in git.\n\n> With the current git 2.21.0, some users, including myself, are still\n> having problems with git selecting the \"wrong\" language for localization.\n>\n> This happens on macOS in the situation where:\n> * The user has multiple Preferred languages defined in Language & Region\n> system preferences\n> * English is set as the Primary language\n> * Another language, for which git has a .po translation file defined, is\n> set as another Preferred language, for example, Spanish\n> * Environment variable $LANG is unset\n> * git was built with gettext support enabled\n>\n> In this situation, when git is run, it will use the translations from\n> the secondary Preferred language instead of displaying messages in\n> English, the Primary language.\n>\n> I've seen this situation with other gettext-enabled applications before.\n> I believe what's happening is that when selecting the language to use,\n> gettext goes through the Preferred languages in order, looking for a .po\n> translation file for each. It does not find one for English, but it does\n> find one for Spanish, so it uses that, instead of falling back to the\n> non-translated message strings.\n>\n> Some examples of this happening in the wild:\n> -\n> https://stackoverflow.com/questions/55145901/force-git-to-use-the-default-system-language/55160216\n> - https://github.com/Homebrew/homebrew-core/issues/37331\n> - https://github.com/Homebrew/homebrew-core/issues/31980\n>\n> I think an easy fix for this would be to add an \"en.po\" translation\n> file, so that when gettext does its translation selection, it finds that\n> first for when English is the Primary language (or a Preferred language\n> earlier in the order than other languages), and uses it. This .po file\n> would be an \"identity\" translation where the translated strings are all\n> just the same as the original strings. I don't think it would even have\n> to be actively maintained, because for new message strings that aren't\n> included in the .po file, it would fall back to the non-translated input\n> strings, which are in English anyway, which is the desired behavior.\n>\n> This would be a convenience for git users, because it would \"just work\"\n> without any modifications to the configure/build process, or requiring\n> users to force a $LANG setting.\n>\n> Would you consider adding this?\n>\n> I've put together a patch that does this:\n> https://github.com/apjanke/git/tree/english-dummy-translation\n> https://github.com/apjanke/git/commit/7e6704167018e1d47399af04230521927991811b\n> Not attaching a patch because it's kind of a large file. I have tested\n> it locally and it fixes the language selection problem for me. I'm not\n> sure if the appropriate thing to do is make a PR for this to the\n> git-l10n/git-po GitHub repo or not.\n>\n> Cheers,\n> Andrew Janke\n\n\n\n-- \nDuy\n"},{"id":"374212","messageId":"9d7a2fb4-8bab-2d28-1066-fbace688a5cc@apjanke.net","threadId":"50962","inReplyTo":"CACsJy8C1w0zJm71KFb21MC+c2NAGnRLDtE3KNK21hO7U4Ax7Hg@mail.gmail.com","subject":"Re: gettext, multiple Preferred languages, and English","fromName":"Andrew Janke","fromEmail":"floss@apjanke.net","sentAt":"2019-04-22T00:57:27Z","receivedAt":"2019-04-22T00:57:31Z","isPatch":false,"sender":{"key":"floss@apjanke.net","avatar":"https://gravatar.com/avatar/cce96357f6822eb4d08552395bda5a987bf4aa3e98eb42ad062ea8cf352c514b?d=mp&s=160"},"body":"\n\nOn 4/21/19 8:35 PM, Duy Nguyen wrote:\n> On Sun, Apr 21, 2019 at 6:40 PM Andrew Janke <floss@apjanke.net> wrote:\n>>\n>> Hi, Git folks,\n>>\n>> This is a follow-up to https://marc.info/?l=git&m=154757938429747&w=2.\n> \n> This says the problem with \"en\" detection has been fixed. Would\n> upgrading gettext fix it?\n> \n> You would need to upgrade something (git or gettext) and if it's\n> already fixed in gettext I don't see why we need a workaround in git.\n\nFrom reading the bug report, that does sound like it would fix it. But\nfrom what I can see, that fix hasn't made it out into a released version\nof gettext yet. I haven't downloaded the development gettext to confirm\nthe fix.\n\nLooking at the gettext ftp site at https://ftp.gnu.org/pub/gnu/gettext/,\nit looks like gettext does not make frequent releases, and the last\nrelease was two and a half years ago. Who knows when the next release\nwill be. And then it'll take longer to trickle down into Linux\ndistributions and such.\n\nFrom your release history at https://github.com/git/git/releases, it\nseems like Git is a lot more active in making releases than gettext. So\nincluding this fix in Git would get it into the hands of affected users\nsooner. And it seems like a pretty low-risk change to me.\n\nThen once the new gettext release is out, their fix is confirmed, and it\nmakes it out into common distros, the workaround could be removed from Git.\n\nCheers,\nAndrew\n"},{"id":"374214","messageId":"56e1db44-cc87-ec0c-6466-826c633ff103@apjanke.net","threadId":"50962","inReplyTo":"ffac5d4b-190c-16d6-497c-4b0f1f12115c@apjanke.net","subject":"Re: gettext, multiple Preferred languages, and English","fromName":"Andrew Janke","fromEmail":"floss@apjanke.net","sentAt":"2019-04-22T01:33:01Z","receivedAt":"2019-04-22T01:33:31Z","isPatch":false,"sender":{"key":"floss@apjanke.net","avatar":"https://gravatar.com/avatar/cce96357f6822eb4d08552395bda5a987bf4aa3e98eb42ad062ea8cf352c514b?d=mp&s=160"},"body":"\n\nOn 4/21/19 9:27 AM, Andrew Janke wrote:\n> \n> On 4/21/19 8:59 AM, Philip Oakley wrote:\n>> Hi Andrew,\n>>\n>> On 21/04/2019 12:08, Andrew Janke wrote:\n>> https://public-inbox.org/git/d001a2b5-57c3-1eb3-70fd-679919bb2eb6@apjanke.net/\n>>\n>>> I don't think it would even have\n>>> to be actively maintained, because for new message strings that aren't\n>>> included in the .po file, it would fall back to the non-translated input\n>>> strings, which are in English anyway, which is the desired behavior.\n>> Given the above comment, could the en.po file\n>> (https://github.com/apjanke/git/blob/english-dummy-translation/po/en.po)\n>> be some very very short version with only one 'translated' string?\n> \n> Yes, I believe so. I only provided a full translation file because it\n> was trivial for me to create using the \"msginit\" instructions I found in\n> po/README. Since all the translations are just identity relationships, I\n> believe that is effectively the same as their not being there in the\n> first place.\n> \n> I tested your approach locally, and it seems to work for me.\n\nBTW, here's a branch with just the \"stub\" translation file:\n\nhttps://github.com/apjanke/git/tree/english-dummy-translation-stub\n\nCheers,\nAndrew\n"},{"id":"374255","messageId":"875zr6szik.fsf@evledraar.gmail.com","threadId":"50962","inReplyTo":"9d7a2fb4-8bab-2d28-1066-fbace688a5cc@apjanke.net","subject":"Re: gettext, multiple Preferred languages, and English","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2019-04-22T17:47:31Z","receivedAt":"2019-04-22T17:47:36Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Mon, Apr 22 2019, Andrew Janke wrote:\n\n> On 4/21/19 8:35 PM, Duy Nguyen wrote:\n>> On Sun, Apr 21, 2019 at 6:40 PM Andrew Janke <floss@apjanke.net> wrote:\n>>>\n>>> Hi, Git folks,\n>>>\n>>> This is a follow-up to https://marc.info/?l=git&m=154757938429747&w=2.\n>>\n>> This says the problem with \"en\" detection has been fixed. Would\n>> upgrading gettext fix it?\n>>\n>> You would need to upgrade something (git or gettext) and if it's\n>> already fixed in gettext I don't see why we need a workaround in git.\n>\n> From reading the bug report, that does sound like it would fix it. But\n> from what I can see, that fix hasn't made it out into a released version\n> of gettext yet. I haven't downloaded the development gettext to confirm\n> the fix.\n>\n> Looking at the gettext ftp site at https://ftp.gnu.org/pub/gnu/gettext/,\n> it looks like gettext does not make frequent releases, and the last\n> release was two and a half years ago. Who knows when the next release\n> will be. And then it'll take longer to trickle down into Linux\n> distributions and such.\n>\n> From your release history at https://github.com/git/git/releases, it\n> seems like Git is a lot more active in making releases than gettext. So\n> including this fix in Git would get it into the hands of affected users\n> sooner. And it seems like a pretty low-risk change to me.\n>\n> Then once the new gettext release is out, their fix is confirmed, and it\n> makes it out into common distros, the workaround could be removed from Git.\n\nWhat does Linux distro release schedule have to do with this? Your\ninitial report and the linked-to bug on GNU savannah only talk about\nthis being an issue on OSX. Is there some more general issue I'm\nmissing?\n\nPeople have reported issues with OSX's weird language selection in the\npast. I think it makes sense to do whatever we need to hack around it as\nlong as it's some well-understood and OSX-only hack.\n\nI'm paranoid that the suggestion of adding an en.po *in general* would\nbreak stuff elsewhere. I'd be surprised if the project linked-to\nupthread that used that hack is as widely ported as we are, and that\nincludes a lot of i18n implementations, not just GNU's.\n\nUltimately setlocale() is *supposed* to be a well-understood thing. You\nset your preferred locale, programs have translations, the OS takes care\nof it. I'm concerned that us trying to be specifically smart in git will\nbackfire (e.g. it's been suggested in the past to have core.language or\nwhatever..).\n\nBut it looks like we don't need to go there, this seems like a\nworkaround needed for some specific OSX version.\n\nThat can just live behind a flag and be detected in config.mak.uname,\nno? And then we'd do whatever hack digs us out of that specific hole on\nOSX, e.g. maybe generating an en.po *just* there, and just for that list\nof known broken version(s) of OSX.\n"},{"id":"374316","messageId":"0ef2b667-fedd-da66-c87a-452f70ffc570@apjanke.net","threadId":"50962","inReplyTo":"875zr6szik.fsf@evledraar.gmail.com","subject":"Re: gettext, multiple Preferred languages, and English","fromName":"Andrew Janke","fromEmail":"floss@apjanke.net","sentAt":"2019-04-23T11:45:47Z","receivedAt":"2019-04-23T11:45:51Z","isPatch":false,"sender":{"key":"floss@apjanke.net","avatar":"https://gravatar.com/avatar/cce96357f6822eb4d08552395bda5a987bf4aa3e98eb42ad062ea8cf352c514b?d=mp&s=160"},"body":"\n\nOn 4/22/19 1:47 PM, Ævar Arnfjörð Bjarmason wrote:\n> \n> On Mon, Apr 22 2019, Andrew Janke wrote:\n> \n>> On 4/21/19 8:35 PM, Duy Nguyen wrote:\n>>> On Sun, Apr 21, 2019 at 6:40 PM Andrew Janke <floss@apjanke.net> wrote:\n>>>>\n>>>> Hi, Git folks,\n>>>>\n>>>> This is a follow-up to https://marc.info/?l=git&m=154757938429747&w=2.\n>>>\n>>> This says the problem with \"en\" detection has been fixed. Would\n>>> upgrading gettext fix it?\n>>>\n>>> You would need to upgrade something (git or gettext) and if it's\n>>> already fixed in gettext I don't see why we need a workaround in git.\n>>\n>> From reading the bug report, that does sound like it would fix it. But\n>> from what I can see, that fix hasn't made it out into a released version\n>> of gettext yet. I haven't downloaded the development gettext to confirm\n>> the fix.\n>>\n>> Looking at the gettext ftp site at https://ftp.gnu.org/pub/gnu/gettext/,\n>> it looks like gettext does not make frequent releases, and the last\n>> release was two and a half years ago. Who knows when the next release\n>> will be. And then it'll take longer to trickle down into Linux\n>> distributions and such.\n>>\n>> From your release history at https://github.com/git/git/releases, it\n>> seems like Git is a lot more active in making releases than gettext. So\n>> including this fix in Git would get it into the hands of affected users\n>> sooner. And it seems like a pretty low-risk change to me.\n>>\n>> Then once the new gettext release is out, their fix is confirmed, and it\n>> makes it out into common distros, the workaround could be removed from Git.\n> \n> What does Linux distro release schedule have to do with this? Your\n> initial report and the linked-to bug on GNU savannah only talk about\n> this being an issue on OSX. Is there some more general issue I'm\n> missing?\n> \n> People have reported issues with OSX's weird language selection in the\n> past. I think it makes sense to do whatever we need to hack around it as\n> long as it's some well-understood and OSX-only hack.\n> \n> I'm paranoid that the suggestion of adding an en.po *in general* would\n> break stuff elsewhere. I'd be surprised if the project linked-to\n> upthread that used that hack is as widely ported as we are, and that\n> includes a lot of i18n implementations, not just GNU's.\n> \n> Ultimately setlocale() is *supposed* to be a well-understood thing. You\n> set your preferred locale, programs have translations, the OS takes care\n> of it. I'm concerned that us trying to be specifically smart in git will\n> backfire (e.g. it's been suggested in the past to have core.language or\n> whatever..).\n> \n> But it looks like we don't need to go there, this seems like a\n> workaround needed for some specific OSX version.\n> \n> That can just live behind a flag and be detected in config.mak.uname,\n> no? And then we'd do whatever hack digs us out of that specific hole on\n> OSX, e.g. maybe generating an en.po *just* there, and just for that list\n> of known broken version(s) of OSX.\n> \n\nGood point. I had forgotten this was OS X-specific; Linux release\nschedules are not relevant. (I don't know enough about language\nselection on Linux to know if would ever be relevant there.) There's no\nmore general issue you're missing; as far as I know this only happens\nunder OS X's multiple-Preferred-languages setup.\n\nYeah, adding the workaround only on OS X sounds like it would work, and\nwould be the more conservative thing to do. Generating a stub en.po just\nfor OS X, and maybe just for affected versions of OS X and gettext,\nsounds sensible to me.\n\nThe bad behavior is due to gettext's interaction with OS X's language\nselection, so whether it happens is probably going to depend on both the\nversion of OS X and the version of gettext you're building against. So\nif you want to generate it selectively, I think you'll need to check the\nversion of gettext (in the expectation that a future version of gettext\nwill fix this), as well as maybe the version of OS X. This\nmulti-language-selection behavior seems to be by design in OS X; I\nwouldn't expect it to change in the future. But it seems like some older\nversions of OS X are not affected by it.\n\nI can reproduce the bad-language-selection behavior on OS X 10.12,\n10.13, and 10.14.\nI cannot reproduce it on OS X 10.11, which is the oldest version of OS X\nI can get running these days. (I'm not sure if that's due to different\nOS behavior, or if I'm just doing something wrong on that box.) All my\ntesting was done with git 2.21.0 and GNU gettext 0.19.8.1.\n\nCheers,\nAndrew\n"}]}