threads / bug / 16282

Bug: UTF-16, UCS-4 and non-existing encodings for git log result in incorrect behavior

Subject: Bug: UTF-16, UCS-4 and non-existing encodings for git log result in incorrect behavior

## tl;dr

6 messages between Nov 12, 2008 and Nov 12, 2008.

replies: 5people: 3as markdown or json

Constantine Plotnikov· Nov 12, 2008, 13:32 UTC · lore

If UTF-16[BE|LE] or UCS-4[BE|LE] encodings are used with git log, the git completes successfully but commit messages and author information are not shown. I suggest that git should fail with fatal error if such zero producing encoding is used.

If the incorrect encoding name is used, the git log does not perform any re-encoding, but just display commits in their native encoding. I suggest that git should fail with fatal error in this case as well.

Regards, Constantine

Johannes Schindelin· Nov 12, 2008, 14:22 UTC · re: Constantine Plotnikov · lore

Re: Bug: UTF-16, UCS-4 and non-existing encodings for git log result in incorrect behavior

Hi,

On Wed, 12 Nov 2008, Constantine Plotnikov wrote:

Show 8 quoted lines

> If UTF-16[BE|LE] or UCS-4[BE|LE] encodings are used with git log, the 
> git completes successfully but commit messages and author information 
> are not shown. I suggest that git should fail with fatal error if such 
> zero producing encoding is used.
> 
> If the incorrect encoding name is used, the git log does not perform any 
> re-encoding, but just display commits in their native encoding. I 
> suggest that git should fail with fatal error in this case as well.

Have you set the correct encoding with i18n.commitEncoding? If not, you should not be surprised: Git's default encoding is UTF-8, and that fact is well documented, AFAICT.

Ciao, Dscho

Johannes Schindelin· Nov 12, 2008, 16:15 UTC · lore

Re: Bug: UTF-16, UCS-4 and non-existing encodings for git log result in incorrect behavior

Hi,

[re Cc:ing the list]

On Wed, 12 Nov 2008, Constantine Plotnikov wrote:

Show 21 quoted lines

> On Wed, Nov 12, 2008 at 5:22 PM, Johannes Schindelin
> <Johannes.Schindelin@gmx.de> wrote:
> >
> > On Wed, 12 Nov 2008, Constantine Plotnikov wrote:
> >
> >> If UTF-16[BE|LE] or UCS-4[BE|LE] encodings are used with git log, the
> >> git completes successfully but commit messages and author information
> >> are not shown. I suggest that git should fail with fatal error if such
> >> zero producing encoding is used.
> >>
> >> If the incorrect encoding name is used, the git log does not perform any
> >> re-encoding, but just display commits in their native encoding. I
> >> suggest that git should fail with fatal error in this case as well.
> >
> > Have you set the correct encoding with i18n.commitEncoding?  If not, you
> > should not be surprised: Git's default encoding is UTF-8, and that fact is
> > well documented, AFAICT.
> >
> Commit encoding is set correctly. The problem is that git log and git
> show do not support the *output* encodings UTF-16 and UCS-4 and
> silently fail in that case instead of reporting the error.

That looks more like an iconv bug to me.  I assume you are using Windows?

Ciao, Dscho

Alexander Gavrilov· Nov 12, 2008, 16:17 UTC · re: Johannes Schindelin · lore

Re: Bug: UTF-16, UCS-4 and non-existing encodings for git log result in incorrect behavior

On Wed, Nov 12, 2008 at 7:15 PM, Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:

Show 30 quoted lines

> Hi,
>
> [re Cc:ing the list]
>
> On Wed, 12 Nov 2008, Constantine Plotnikov wrote:
>
>> On Wed, Nov 12, 2008 at 5:22 PM, Johannes Schindelin
>> <Johannes.Schindelin@gmx.de> wrote:
>> >
>> > On Wed, 12 Nov 2008, Constantine Plotnikov wrote:
>> >
>> >> If UTF-16[BE|LE] or UCS-4[BE|LE] encodings are used with git log, the
>> >> git completes successfully but commit messages and author information
>> >> are not shown. I suggest that git should fail with fatal error if such
>> >> zero producing encoding is used.
>> >>
>> >> If the incorrect encoding name is used, the git log does not perform any
>> >> re-encoding, but just display commits in their native encoding. I
>> >> suggest that git should fail with fatal error in this case as well.
>> >
>> > Have you set the correct encoding with i18n.commitEncoding?  If not, you
>> > should not be surprised: Git's default encoding is UTF-8, and that fact is
>> > well documented, AFAICT.
>> >
>> Commit encoding is set correctly. The problem is that git log and git
>> show do not support the *output* encodings UTF-16 and UCS-4 and
>> silently fail in that case instead of reporting the error.
>
> That looks more like an iconv bug to me.  I assume you are using Windows?
>

Iconv has no way to know that git cannot work with ASCII-incompatible encodings, and UTF-16 is incompatible, because it fills the output with loads of zero bytes. Git both truncates messages on these bytes, and forgets inserting them in strings that it produces itself.

A separate problem is that it allows creating commits with invalid encoding names, which may be unnoticed for a long time in an environment with uniform commitencoding settings.

Alexander

Johannes Schindelin· Nov 12, 2008, 16:42 UTC · re: Alexander Gavrilov · lore

Re: Bug: UTF-16, UCS-4 and non-existing encodings for git log result in incorrect behavior

Hi,

On Wed, 12 Nov 2008, Alexander Gavrilov wrote:

Show 36 quoted lines

> On Wed, Nov 12, 2008 at 7:15 PM, Johannes Schindelin
> <Johannes.Schindelin@gmx.de> wrote:
>
> > [re Cc:ing the list]
> >
> > On Wed, 12 Nov 2008, Constantine Plotnikov wrote:
> >
> >> On Wed, Nov 12, 2008 at 5:22 PM, Johannes Schindelin
> >> <Johannes.Schindelin@gmx.de> wrote:
> >> >
> >> > On Wed, 12 Nov 2008, Constantine Plotnikov wrote:
> >> >
> >> >> If UTF-16[BE|LE] or UCS-4[BE|LE] encodings are used with git log, 
> >> >> the git completes successfully but commit messages and author 
> >> >> information are not shown. I suggest that git should fail with 
> >> >> fatal error if such zero producing encoding is used.
> >> >>
> >> >> If the incorrect encoding name is used, the git log does not 
> >> >> perform any re-encoding, but just display commits in their native 
> >> >> encoding. I suggest that git should fail with fatal error in this 
> >> >> case as well.
> >> >
> >> > Have you set the correct encoding with i18n.commitEncoding?  If 
> >> > not, you should not be surprised: Git's default encoding is UTF-8, 
> >> > and that fact is well documented, AFAICT.
> >> >
> >> Commit encoding is set correctly. The problem is that git log and git 
> >> show do not support the *output* encodings UTF-16 and UCS-4 and 
> >> silently fail in that case instead of reporting the error.
> >
> > That looks more like an iconv bug to me.  I assume you are using Windows?
> 
> Iconv has no way to know that git cannot work with ASCII-incompatible 
> encodings, and UTF-16 is incompatible, because it fills the output with 
> loads of zero bytes. Git both truncates messages on these bytes, and 
> forgets inserting them in strings that it produces itself.

Ah, I thought that the issue was that Git would not handle commits in that encoding correctly. Instead, it appears that Git cannot work with UTF-16 _displays_.

Yep, I would have expected that.

Ciao, Dscho

Alexander Gavrilov· Nov 12, 2008, 16:58 UTC · re: Johannes Schindelin · lore

Re: Bug: UTF-16, UCS-4 and non-existing encodings for git log result in incorrect behavior

On Wed, Nov 12, 2008 at 7:42 PM, Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:

Show 18 quoted lines

> On Wed, 12 Nov 2008, Alexander Gavrilov wrote:
>> On Wed, Nov 12, 2008 at 7:15 PM, Johannes Schindelin
>> <Johannes.Schindelin@gmx.de> wrote:
>> > On Wed, 12 Nov 2008, Constantine Plotnikov wrote:
>> >> Commit encoding is set correctly. The problem is that git log and git
>> >> show do not support the *output* encodings UTF-16 and UCS-4 and
>> >> silently fail in that case instead of reporting the error.
>> >
>> > That looks more like an iconv bug to me.  I assume you are using Windows?
>>
>> Iconv has no way to know that git cannot work with ASCII-incompatible
>> encodings, and UTF-16 is incompatible, because it fills the output with
>> loads of zero bytes. Git both truncates messages on these bytes, and
>> forgets inserting them in strings that it produces itself.
>
> Ah, I thought that the issue was that Git would not handle commits in that
> encoding correctly.  Instead, it appears that Git cannot work with UTF-16
> _displays_.

Actually, I think that using those encodings in commits is asking for trouble too, because the encoding conversion is, as far as I remember, applied to the entire contents of the commit object, and Git, naturally, doesn't insert any null bytes in the commit headers to please the decoder. The result is a completely trashed object on output.

Also, I think that they are generally a poor choice of an encoding for data transmission, because they are ASCII-incompatible, stdlib-incompatible, unreliable to loss and addition of single bytes, and have no way to detect encoding mismatch except by metadata or heuristics: almost any string of shorts is "valid".

Alexander