git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH v9 4/8] utf8: add function to detect a missing UTF-16/32 BOM

From
Lars Schneider <larsxschneider@gmail.com>
Date
Mar 6, 2018, 23:12 UTC
Message-ID
<EC8DF4D3-CDF5-4CEB-BD18-585341BBB2C5@gmail.com>
In-Reply-To
<xmqq4lls3irx.fsf@gitster-ct.c.googlers.com>
Show 21 quoted lines
> On 07 Mar 2018, at 00:07, Junio C Hamano <gitster@pobox.com> wrote:
> 
> Junio C Hamano <gitster@pobox.com> writes:
> 
>> Lars Schneider <larsxschneider@gmail.com> writes:
>> 
>>>> Also "UTF16" or other spelling
>>>> the platform may support but this code fails to recognise will go
>>>> unchecked.
>>> 
>>> That is true. However, I would assume all iconv implementations use the
>>> same encoding names for UTF encodings, no? That means UTF16 would never be
>>> valid. Would you agree?
>> 
>> After seeing "UTF16" and others in "iconv -l | grep -i utf" output,
>> I am not sure what you mean by "Would you agree?"  People can say in
>> their .gitattributes file that this path is in "UTF16" without dash
>> and that is what will be fed to this code, no?
> 
> FWIW, "iconv -f utf8 -t utf16" seems not to complain, so I am
> reasonably sure that people expect downcased ones to work as well.

Sure! That's why I normalized it to upper case: https://public-inbox.org/git/CAPig+cQE0pKs-AMvh4GndyCXBMnx=70jPpDM6K4jJTe-74FecQ@mail.gmail.com/

After thinking about it I wonder if we should barf on "utf16" without dash. Your Linux iconv would handle this correctly. My macOS iconv would not. That means the repo would checkout correctly on your machine but not on mine.

What do you think?
- Lars
Previous: Junio C HamanoNext: Junio C Hamano
Message 11 of 26 in “convert: add support for different encodings”
  1. 0/8 convert: add support for different encodingslars.schneider@autodesk.com, Mar 4, 2018
  2. 1/8 strbuf: remove unnecessary NUL assignment in xstrdup_tolower()lars.schneider@autodesk.com, Mar 4, 2018
  3. 3/8 utf8: add function to detect prohibited UTF-16/32 BOMlars.schneider@autodesk.com, Mar 4, 2018
  4. 2/8 strbuf: add xstrdup_toupper()lars.schneider@autodesk.com, Mar 4, 2018
  5. 4/8 utf8: add function to detect a missing UTF-16/32 BOMlars.schneider@autodesk.com, Mar 4, 2018
  6. Junio C HamanoMar 6, 2018
  7. Lars SchneiderMar 6, 2018
  8. Junio C HamanoMar 6, 2018
  9. Lars SchneiderMar 6, 2018
  10. Junio C HamanoMar 6, 2018
  11. Lars SchneiderMar 6, 2018
  12. Junio C HamanoMar 6, 2018
  13. Torsten BögershausenMar 7, 2018
  14. 5/8 convert: add 'working-tree-encoding' attributelars.schneider@autodesk.com, Mar 4, 2018
  15. Eric SunshineMar 6, 2018
  16. Lars SchneiderMar 6, 2018
  17. Eric SunshineMar 6, 2018
  18. 8/8 convert: add round trip check based on 'core.checkRoundtripEncoding'lars.schneider@autodesk.com, Mar 4, 2018
  19. 6/8 convert: check for detectable errors in UTF encodingslars.schneider@autodesk.com, Mar 4, 2018
  20. Junio C HamanoMar 5, 2018
  21. Lars SchneiderMar 5, 2018
  22. Junio C HamanoMar 6, 2018
  23. Lars SchneiderMar 6, 2018
  24. Eric SunshineMar 6, 2018
  25. 7/8 convert: add tracing for 'working-tree-encoding' attributelars.schneider@autodesk.com, Mar 4, 2018
  26. Eric SunshineMar 6, 2018

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.