{"thread":{"id":"16282","subject":"Bug: UTF-16, UCS-4 and non-existing encodings for git log result in incorrect behavior","startedAt":"2008-11-12T13:32:23Z","lastAt":"2008-11-12T16:58:11Z","messageCount":6,"participants":["Constantine Plotnikov","Johannes Schindelin","Alexander Gavrilov"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"95560","messageId":"85647ef50811120532h778769ddx69f0b111dbad359a@mail.gmail.com","threadId":"16282","inReplyTo":null,"subject":"Bug: UTF-16, UCS-4 and non-existing encodings for git log result in incorrect behavior","fromName":"Constantine Plotnikov","fromEmail":"constantine.plotnikov@gmail.com","sentAt":"2008-11-12T13:32:23Z","receivedAt":"2008-11-12T13:32:23Z","isPatch":false,"sender":{"key":"constantine.plotnikov@gmail.com","avatar":null},"body":"If UTF-16[BE|LE] or UCS-4[BE|LE] encodings are used with git log, the\ngit completes successfully but commit messages and author information\nare not shown. I suggest that git should fail with fatal error if such\nzero producing encoding is used.\n\nIf the incorrect encoding name is used, the git log does not perform\nany re-encoding, but just display commits in their native encoding. I\nsuggest that git should fail with fatal error in this case as well.\n\nRegards,\nConstantine\n"},{"id":"95563","messageId":"alpine.DEB.1.00.0811121521390.30769@pacific.mpi-cbg.de","threadId":"16282","inReplyTo":"85647ef50811120532h778769ddx69f0b111dbad359a@mail.gmail.com","subject":"Re: Bug: UTF-16, UCS-4 and non-existing encodings for git log result in incorrect behavior","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2008-11-12T14:22:46Z","receivedAt":"2008-11-12T14:22:46Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Wed, 12 Nov 2008, Constantine Plotnikov wrote:\n\n> If UTF-16[BE|LE] or UCS-4[BE|LE] encodings are used with git log, the \n> git completes successfully but commit messages and author information \n> are not shown. I suggest that git should fail with fatal error if such \n> zero producing encoding is used.\n> \n> If the incorrect encoding name is used, the git log does not perform any \n> re-encoding, but just display commits in their native encoding. I \n> suggest that git should fail with fatal error in this case as well.\n\nHave you set the correct encoding with i18n.commitEncoding? If not, you \nshould not be surprised: Git's default encoding is UTF-8, and that fact is \nwell documented, AFAICT.\n\nCiao,\nDscho\n"},{"id":"95573","messageId":"alpine.DEB.1.00.0811121714280.30769@pacific.mpi-cbg.de","threadId":"16282","inReplyTo":"85647ef50811120727j730cb6e3lf4103c200d042fb9@mail.gmail.com","subject":"Re: Bug: UTF-16, UCS-4 and non-existing encodings for git log result in incorrect behavior","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2008-11-12T16:15:15Z","receivedAt":"2008-11-12T16:15:15Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\n[re Cc:ing the list]\n\nOn Wed, 12 Nov 2008, Constantine Plotnikov wrote:\n\n> On Wed, Nov 12, 2008 at 5:22 PM, Johannes Schindelin\n> wrote:\n> >\n> > On Wed, 12 Nov 2008, Constantine Plotnikov wrote:\n> >\n> >> If UTF-16[BE|LE] or UCS-4[BE|LE] encodings are used with git log, the\n> >> git completes successfully but commit messages and author information\n> >> are not shown. I suggest that git should fail with fatal error if such\n> >> zero producing encoding is used.\n> >>\n> >> If the incorrect encoding name is used, the git log does not perform any\n> >> re-encoding, but just display commits in their native encoding. I\n> >> suggest that git should fail with fatal error in this case as well.\n> >\n> > Have you set the correct encoding with i18n.commitEncoding? If not, you\n> > should not be surprised: Git's default encoding is UTF-8, and that fact is\n> > well documented, AFAICT.\n> >\n> Commit encoding is set correctly. The problem is that git log and git\n> show do not support the *output* encodings UTF-16 and UCS-4 and\n> silently fail in that case instead of reporting the error.\n\nThat looks more like an iconv bug to me. I assume you are using Windows?\n\nCiao,\nDscho\n"},{"id":"95574","messageId":"bb6f213e0811120817i20d5c0ajf0c9e289c11387a6@mail.gmail.com","threadId":"16282","inReplyTo":"alpine.DEB.1.00.0811121714280.30769@pacific.mpi-cbg.de","subject":"Re: Bug: UTF-16, UCS-4 and non-existing encodings for git log result in incorrect behavior","fromName":"Alexander Gavrilov","fromEmail":"angavrilov@gmail.com","sentAt":"2008-11-12T16:17:53Z","receivedAt":"2008-11-12T16:17:53Z","isPatch":false,"sender":{"key":"angavrilov@gmail.com","avatar":"https://avatars.githubusercontent.com/u/42666?v=4"},"body":"On Wed, Nov 12, 2008 at 7:15 PM, Johannes Schindelin\n wrote:\n> Hi,\n>\n> [re Cc:ing the list]\n>\n> On Wed, 12 Nov 2008, Constantine Plotnikov wrote:\n>\n>> On Wed, Nov 12, 2008 at 5:22 PM, Johannes Schindelin\n>> wrote:\n>> >\n>> > On Wed, 12 Nov 2008, Constantine Plotnikov wrote:\n>> >\n>> >> If UTF-16[BE|LE] or UCS-4[BE|LE] encodings are used with git log, the\n>> >> git completes successfully but commit messages and author information\n>> >> are not shown. I suggest that git should fail with fatal error if such\n>> >> zero producing encoding is used.\n>> >>\n>> >> If the incorrect encoding name is used, the git log does not perform any\n>> >> re-encoding, but just display commits in their native encoding. I\n>> >> suggest that git should fail with fatal error in this case as well.\n>> >\n>> > Have you set the correct encoding with i18n.commitEncoding? If not, you\n>> > should not be surprised: Git's default encoding is UTF-8, and that fact is\n>> > well documented, AFAICT.\n>> >\n>> Commit encoding is set correctly. The problem is that git log and git\n>> show do not support the *output* encodings UTF-16 and UCS-4 and\n>> silently fail in that case instead of reporting the error.\n>\n> That looks more like an iconv bug to me. I assume you are using Windows?\n>\n\nIconv has no way to know that git cannot work with ASCII-incompatible\nencodings, and UTF-16 is incompatible, because it fills the output\nwith loads of zero bytes. Git both truncates messages on these bytes,\nand forgets inserting them in strings that it produces itself.\n\nA separate problem is that it allows creating commits with invalid\nencoding names, which may be unnoticed for a long time in an\nenvironment with uniform commitencoding settings.\n\nAlexander\n"},{"id":"95576","messageId":"alpine.DEB.1.00.0811121740120.30769@pacific.mpi-cbg.de","threadId":"16282","inReplyTo":"bb6f213e0811120817i20d5c0ajf0c9e289c11387a6@mail.gmail.com","subject":"Re: Bug: UTF-16, UCS-4 and non-existing encodings for git log result in incorrect behavior","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2008-11-12T16:42:23Z","receivedAt":"2008-11-12T16:42:23Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Wed, 12 Nov 2008, Alexander Gavrilov wrote:\n\n> On Wed, Nov 12, 2008 at 7:15 PM, Johannes Schindelin\n> wrote:\n>\n> > [re Cc:ing the list]\n> >\n> > On Wed, 12 Nov 2008, Constantine Plotnikov wrote:\n> >\n> >> On Wed, Nov 12, 2008 at 5:22 PM, Johannes Schindelin\n> >> wrote:\n> >> >\n> >> > On Wed, 12 Nov 2008, Constantine Plotnikov wrote:\n> >> >\n> >> >> If UTF-16[BE|LE] or UCS-4[BE|LE] encodings are used with git log, \n> >> >> the git completes successfully but commit messages and author \n> >> >> information are not shown. I suggest that git should fail with \n> >> >> fatal error if such zero producing encoding is used.\n> >> >>\n> >> >> If the incorrect encoding name is used, the git log does not \n> >> >> perform any re-encoding, but just display commits in their native \n> >> >> encoding. I suggest that git should fail with fatal error in this \n> >> >> case as well.\n> >> >\n> >> > Have you set the correct encoding with i18n.commitEncoding? If \n> >> > not, you should not be surprised: Git's default encoding is UTF-8, \n> >> > and that fact is well documented, AFAICT.\n> >> >\n> >> Commit encoding is set correctly. The problem is that git log and git \n> >> show do not support the *output* encodings UTF-16 and UCS-4 and \n> >> silently fail in that case instead of reporting the error.\n> >\n> > That looks more like an iconv bug to me. I assume you are using Windows?\n> \n> Iconv has no way to know that git cannot work with ASCII-incompatible \n> encodings, and UTF-16 is incompatible, because it fills the output with \n> loads of zero bytes. Git both truncates messages on these bytes, and \n> forgets inserting them in strings that it produces itself.\n\nAh, I thought that the issue was that Git would not handle commits in that \nencoding correctly. Instead, it appears that Git cannot work with UTF-16 \n_displays_.\n\nYep, I would have expected that.\n\nCiao,\nDscho\n"},{"id":"95581","messageId":"bb6f213e0811120858lbcefa25v47a4fb14d8537839@mail.gmail.com","threadId":"16282","inReplyTo":"alpine.DEB.1.00.0811121740120.30769@pacific.mpi-cbg.de","subject":"Re: Bug: UTF-16, UCS-4 and non-existing encodings for git log result in incorrect behavior","fromName":"Alexander Gavrilov","fromEmail":"angavrilov@gmail.com","sentAt":"2008-11-12T16:58:11Z","receivedAt":"2008-11-12T16:58:11Z","isPatch":false,"sender":{"key":"angavrilov@gmail.com","avatar":"https://avatars.githubusercontent.com/u/42666?v=4"},"body":"On Wed, Nov 12, 2008 at 7:42 PM, Johannes Schindelin\n wrote:\n> On Wed, 12 Nov 2008, Alexander Gavrilov wrote:\n>> On Wed, Nov 12, 2008 at 7:15 PM, Johannes Schindelin\n>> wrote:\n>> > On Wed, 12 Nov 2008, Constantine Plotnikov wrote:\n>> >> Commit encoding is set correctly. The problem is that git log and git\n>> >> show do not support the *output* encodings UTF-16 and UCS-4 and\n>> >> silently fail in that case instead of reporting the error.\n>> >\n>> > That looks more like an iconv bug to me. I assume you are using Windows?\n>>\n>> Iconv has no way to know that git cannot work with ASCII-incompatible\n>> encodings, and UTF-16 is incompatible, because it fills the output with\n>> loads of zero bytes. Git both truncates messages on these bytes, and\n>> forgets inserting them in strings that it produces itself.\n>\n> Ah, I thought that the issue was that Git would not handle commits in that\n> encoding correctly. Instead, it appears that Git cannot work with UTF-16\n> _displays_.\n\nActually, I think that using those encodings in commits is asking for\ntrouble too, because the encoding conversion is, as far as I remember,\napplied to the entire contents of the commit object, and Git,\nnaturally, doesn't insert any null bytes in the commit headers to\nplease the decoder. The result is a completely trashed object on\noutput.\n\nAlso, I think that they are generally a poor choice of an encoding for\ndata transmission, because they are ASCII-incompatible,\nstdlib-incompatible, unreliable to loss and addition of single bytes,\nand have no way to detect encoding mismatch except by metadata or\nheuristics: almost any string of shorts is \"valid\".\n\nAlexander\n"}]}