git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH v2 0/4] UTF8 BOM follow-up

From
Karsten Blees <karsten.blees@gmail.com>
Date
Apr 17, 2015, 22:44 UTC
Message-ID
<55318CE2.1000706@gmail.com>
In-Reply-To
<1429209548-32297-1-git-send-email-gitster@pobox.com>
Am 16.04.2015 um 20:39 schrieb Junio C Hamano:
Show 13 quoted lines
> This is on top of the ".gitignore can start with UTF8 BOM" patch
> from Carlos.
> 
> Second try; the first patch is new to clarify the logic in the
> codeflow after Carlos's patch, and the second one has been adjusted
> accordingly.
> 
> Junio C Hamano (4):
>   add_excludes_from_file: clarify the bom skipping logic
>   utf8-bom: introduce skip_utf8_bom() helper
>   config: use utf8_bom[] from utf.[ch] in git_parse_source()
>   attr: skip UTF8 BOM at the beginning of the input file
> 
Wouldn't it be better to just strip the BOM on commit, e.g. via a clean filter or pre-commit hook (as suggested in [1])? Or is this patch series only meant to supplement such a solution (i.e. only strip the BOM when reading files from the working-copy rather than the committed tree)?
According to rfc3629 chapter 6 [2], the use of a BOM as encoding signature should be forbidden if the encoding is *known* to be always UTF-8. And .gitignore, .gitattributes and .gitmodules contain path names, which are always UTF-8 as of Git for Windows v1.7.10.
IOW, allowing a BOM would mean that files *without* BOM are *not* UTF-8 and need to be decoded from e.g. system encoding (which unfortunately cannot be set to UTF-8 on Windows). But this makes no sense as the repository would not be portable. E.g. a .gitattributes file created on a Greek Windows, containing greek path names in Cp1253, would not work on platforms with different encoding.
On the other hand, just ignoring the BOM (as this patch series does) leaves us with two alternative binary representations of the same content file...i.e. we'll eventually end up with spurious 1st line changes as users add / remove BOMs from committed .git[ignore|attributes|modules] files, depending on their editor preference...
For local files (.gitconfig, .git/info/exclude, .git/COMMIT_EDITMSG...), auto-detecting encoding based on the presence of a BOM makes somewhat more sense. However, this will most likely break editors that follow the recommendation of the Unicode specification ("Use of a BOM is neither required nor recommended for UTF-8" [3]). So we'd probably need a core.editorEncoding or core.editorUseBom setting to tell git whether "no BOM" means UTF-8 or system encoding...
Just as a reminder: we should update the Git for Windows Unicode document [4] if we improve support for BOM-adamant editors.

Cheers, Karsten

[1] http://stackoverflow.com/questions/27223985/git-ignore-bom-prevent-git-diff-from-showing-byte-order-mark-changes [2] https://tools.ietf.org/html/rfc3629 [3] http://www.unicode.org/versions/Unicode7.0.0/ch02.pdf p.40 [4] https://github.com/msysgit/msysgit/wiki/Git-for-Windows-Unicode-Support#editor

Previous: Jeff KingNext: Junio C Hamano
Message 21 of 24 in “dir: allow a BOM at the beginning of exclude files”
  1. dir: allow a BOM at the beginning of exclude filesCarlos Martín Nieto, Apr 16, 2015
  2. Johannes SchindelinApr 16, 2015
  3. Carlos Martín NietoApr 16, 2015
  4. Carlos Martín NietoApr 16, 2015
  5. Junio C HamanoApr 16, 2015
  6. Jeff KingApr 16, 2015
  7. Junio C HamanoApr 16, 2015
  8. 0/3 UTF8 BOM follow-upJunio C Hamano, Apr 16, 2015
  9. 1/3 utf8-bom: introduce skip_utf8_bom() helperJunio C Hamano, Apr 16, 2015
  10. Jeff KingApr 16, 2015
  11. Junio C HamanoApr 16, 2015
  12. 2/3 config: use utf8_bom[] from utf.[ch] in git_parse_source()Junio C Hamano, Apr 16, 2015
  13. 3/3 attr: skip UTF8 BOM at the beginning of the input fileJunio C Hamano, Apr 16, 2015
  14. Carlos Martín NietoApr 16, 2015
  15. 0/4 UTF8 BOM follow-upJunio C Hamano, Apr 16, 2015
  16. 1/4 add_excludes_from_file: clarify the bom skipping logicJunio C Hamano, Apr 16, 2015
  17. 2/4 utf8-bom: introduce skip_utf8_bom() helperJunio C Hamano, Apr 16, 2015
  18. 3/4 config: use utf8_bom[] from utf.[ch] in git_parse_source()Junio C Hamano, Apr 16, 2015
  19. 4/4 attr: skip UTF8 BOM at the beginning of the input fileJunio C Hamano, Apr 16, 2015
  20. Jeff KingApr 16, 2015
  21. Karsten BleesApr 17, 2015
  22. Junio C HamanoApr 20, 2015
  23. Johannes SchindelinApr 16, 2015
  24. Torsten BögershausenApr 16, 2015

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.