Re: Handling text files encoded in little-endian UTF-16 with BOM

From: Mateusz Loskot <mateusz@loskot.net>
Date: Jul 5, 2019, 18:52 UTC
Message-ID: <CABUeae-kmc7WV1iv5e1wY+BPMrpXpAoJY0H3B-iQ9D2yyg_hMg@mail.gmail.com>
In-Reply-To: <20190705162544.gckigv4cbr4k6zpz@tb-raspi4>

On Fri, 5 Jul 2019 at 18:25, Torsten Bögershausen <tboegi@web.de> wrote:

Show 22 quoted lines

> On Fri, Jul 05, 2019 at 01:35:13PM +0200, Mateusz Loskot wrote:
> >
> > Using git version 2.22.0.windows.1
> >
> > I have a repository with number of .txt files encoded in
> > little-endian UTF-16 with BOM.
> >
> > What are the best practice and recommended configuration to
> > manage such files with Git to avoid unexpected re-encoding to
> > UTF-8 or others?
> >
> > Currently, there is .gitattriuts with entries like
> >
> >    resource/*.txt   working-tree-encoding=UTF-16LE-BOM -text
> >
> > Despite that some of team members have noticed that the files
> > occacionally get re-encoded to UTF-8. It is unknow what are
> > actual steps leading to that. BTW, there a few Git clients
> > in use: git in Git Bash, VSCode, Fork.
>
> If possible, I don't want to comment on this kind of
> "sometimes something happens something on someones computer" thing.

Perfectly understood.

> A little bit more information could be helpful.

If there was more, I'd have provided.

Show 8 quoted lines

> > What bothers me in the .gitattributes is this `-text` attribute.
> >
> > Is the use of `working-tree-encoding` and `-text` together a
> > valid combination at all?
>
> Yes, it means that the content re-encoded between the repo and the working tree,
> (that is what you want)
> And the "-text" means "leave the line endings" (LF or CRLF) as is, don't convert them.

That's quite a useful insight. I understood "-text" means content is not a text, but binary.

> In that sense you can call that a legal combination, but may be not a recommended one.

Right.

Show 10 quoted lines

> > The documentation at https://git-scm.com/docs/gitattributes
> > does not seem to touch on that.
> >
> > I'll appreciate any suggestions on those UTF-16LE-BOM files.
> >
>
> My suggestion would be to use the "text" attribute:
>   resource/*.txt   working-tree-encoding=UTF-16LE-BOM text
>
> And depending on your application: Do the resource files need a special line ending ?

I need CRLF.

> The use either
>   resource/*.txt   working-tree-encoding=UTF-16LE-BOM text eol=LF
> or
>   resource/*.txt   working-tree-encoding=UTF-16LE-BOM text eol=CRLF

This is very helpful. Thanks!

-- 
Mateusz Loskot, http://mateusz.loskot.net