threads / bug / 63292

Potential bug: Git instantly converts utf-8-bom to utf-8

Subject: Potential bug: Git instantly converts utf-8-bom to utf-8

## tl;dr

4 messages between Apr 14, 2025 and Apr 14, 2025.

replies: 3people: 4as markdown or json

messester· Apr 14, 2025, 15:59 UTC · lore

How this occurs: When I clone the git repository https://github.com/NinekoTheCat/infinilore.cs.git into an empty directory and run git status it shows changes where all the files are deleted and then added again. The difference seems to be in the first line as opening it in a text editor aware of encodings shows the change from utf-8-bom to utf-8. (Attached to this email is an example patch file).

My git configuration in the directory is: --- diff.astextplain.textconv=astextplain filter.lfs.clean=git-lfs clean -- %f filter.lfs.smudge=git-lfs smudge -- %f filter.lfs.process=git-lfs filter-process filter.lfs.required=true http.sslbackend=openssl http.sslcainfo=REDACT.crt core.autocrlf=true core.fscache=true core.symlinks=false pull.rebase=false credential.helper=manager credential.https://dev.azure.com.usehttppath=true init.defaultbranch=main filter.lfs.clean=git-lfs clean -- %f filter.lfs.smudge=git-lfs smudge -- %f filter.lfs.process=git-lfs filter-process filter.lfs.required=true user.name=REDACT user.email=REDACT user.signingkey=4D973217092A925B core.editor="C:\Users\cnots\AppData\Local\Programs\Microsoft VS Code\bin\code" --wait safe.directory=REDACT commit.gpgsign=true gpg.program=C:\Program Files (x86)\GnuPG\bin\gpg.exe credential.https://github.com.helper= credential.https://github.com.helper=!REDACT credential.https://gist.github.com.helper= credential.https://gist.github.com.helper=!REDACT --- and the version of git I use is 2.49.0.windows.1.

I'd be very very grateful if anyone could explain what's going on and have a solution.
Lucas Seiki Oshiro· Apr 14, 2025, 17:19 UTC · re: messester · lore

Re: Potential bug: Git instantly converts utf-8-bom to utf-8

> When I clone the git repository https://github.com/NinekoTheCat/infinilore.cs.git into an empty directory and run git status it shows changes where all the files are deleted and then added again.
I tried here. I'm in a Mac, it shows that several file changes.
> The difference seems to be in the first line as opening it in a text editor aware of encodings shows the change from utf-8-bom to utf-8.

Given the warning messages it looks like it is more related to CRLF/LF than to UTF-8.

At least here, for example, the listed modified '*.cs' files are the ones that ends their lines in CRLF. This is, running:

``` file $(git ls-files '*.cs') | grep CRLF | cut -d ':' -f1 | sort ```

shows me the same files as

``` git status '*.cs' ```

You can also see that are some '*.cs' files that are not CRLF-terminated by running:

``` file $(git ls-files '*.cs') | grep -v CRLF | sort ```

> I'd be very very grateful if anyone could explain what's going on and have a solution.<patch.cat>
Perhaps a solution is:
- Define a CRLF or LF as a standard
- Convert all the files that don't follow the standard that you
  have chosen to the one that you have chosen
- Declare it in the .gitattributes file
Hope that it helps you!
Torsten Bögershausen· Apr 14, 2025, 18:03 UTC · re: Lucas Seiki Oshiro · lore

Re: Potential bug: Git instantly converts utf-8-bom to utf-8

On Mon, Apr 14, 2025 at 02:19:05PM -0300, Lucas Seiki Oshiro wrote:
Show 40 quoted lines
>
> > When I clone the git repository https://github.com/NinekoTheCat/infinilore.cs.git into an empty directory and run git status it shows changes where all the files are deleted and then added again.
>
> I tried here. I'm in a Mac, it shows that several file changes.
>
> > The difference seems to be in the first line as opening it in a text editor aware of encodings shows the change from utf-8-bom to utf-8.
>
> Given the warning messages it looks like it is more related
> to CRLF/LF than to UTF-8.
>
> At least here, for example, the listed modified '*.cs' files
> are the ones that ends their lines in CRLF. This is, running:
>
> ```
> file $(git ls-files '*.cs') | grep CRLF | cut -d ':' -f1 | sort
> ```
>
> shows me the same files as
>
> ```
> git status '*.cs'
> ```
>
> You can also see that are some '*.cs' files that are not
> CRLF-terminated by running:
>
> ```
> file $(git ls-files '*.cs') | grep -v CRLF | sort
> ```
>
> > I'd be very very grateful if anyone could explain what's going on and have a solution.<patch.cat>
>
> Perhaps a solution is:
>
> - Define a CRLF or LF as a standard
> - Convert all the files that don't follow the standard that you
>   have chosen to the one that you have chosen
> - Declare it in the .gitattributes file
>
> Hope that it helps you!

I think that this is already done :-) However, the repo must be normalized:

git add --renormalize .
will do that.
Junio C Hamano· Apr 14, 2025, 19:28 UTC · re: Torsten Bögershausen · lore

Re: Potential bug: Git instantly converts utf-8-bom to utf-8

Torsten Bögershausen <tboegi@web.de> writes:
Show 6 quoted lines
> I think that this is already done :-)
> However, the repo must be normalized:
>
> git add --renormalize .
>
> will do that.
Thanks ;-).

← back to recent threads