git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH v3 5/5] fast-export: do automatic reencoding of commit messages only if requested

From
Torsten Bögershausen <tboegi@web.de>
Date
May 13, 2019, 12:56 UTC
Message-ID
<20190513125635.tntxhjbar2eold2i@tb-raspi4>
In-Reply-To
<nycvar.QRO.7.76.6.1905131219490.44@tvgsbejvaqbjf.bet>
On Mon, May 13, 2019 at 12:23:29PM +0200, Johannes Schindelin wrote:
Show 24 quoted lines
> Hi Elijah,
>
> On Sat, 11 May 2019, Elijah Newren wrote:
>
> > [...] the craziness is based on how Windows behaves; it seems insane to
> > me that Windows decides to munge user data (in the form of the command
> > line provided), so much so that it makes me wonder if I really
> > understood Hannes' and Dscho's explanations of what it is doing.
>
> It is not the user data that is munged by *Windows*, but by *Git for
> Windows*. The user data on Windows is encoded in UTF-16 (or some slight
> variant thereof). Git *cannot* handle UTF-16. Git's test suite *cannot*
> handle UTF-16. So we convert. That's all there is to it.
>
> Ciao,
> Dscho
>
> P.S.: Of course it is not *all* there is to it. There is also a current
> code page which depends on the current user's current locale. We can
> definitely not rely on that, as Git has no idea about this and would quite
> positively produce incorrect output because of it. So we really just use
> the `*W()` functions of the Win32 API (i.e. the ones accepting wide
> Unicode characters and strings, i.e. UTF-16). I don't think we can do
> better than that.

We can actuall feed valid UTF-8 into a test case. (Remember that shell scripts need this octal numbering, see t/t0050)

See the "ä" code point: $ auml=$(printf '\303\244') $ printf $auml ä

Now we can feed those 2 bytes (wich are valid UTF) into Git and say "convert them from ISO-8859-1 into UTF-8, resulting in 4 bytes. Is my explanation clear enough ? If not, plese tell me.

Previous: Johannes SchindelinNext: Elijah Newren
Message 17 of 45 in “Fix and extend encoding handling in fast export/import”
  1. 0/5 Fix and extend encoding handling in fast export/importElijah Newren, Apr 30, 2019
  2. 1/5 t9350: fix encoding test to actually test reencodingElijah Newren, Apr 30, 2019
  3. 2/5 fast-import: support 'encoding' commit headerElijah Newren, Apr 30, 2019
  4. 4/5 fast-export: differentiate between explicitly utf-8 and implicitly utf-8Elijah Newren, Apr 30, 2019
  5. 3/5 fast-export: avoid stripping encoding header if we cannot reencodeElijah Newren, Apr 30, 2019
  6. 5/5 fast-export: do automatic reencoding of commit messages only if requestedElijah Newren, Apr 30, 2019
  7. 0/5 Fix and extend encoding handling in fast export/importElijah Newren, May 10, 2019
  8. 1/5 t9350: fix encoding test to actually test reencodingElijah Newren, May 10, 2019
  9. 2/5 fast-import: support 'encoding' commit headerElijah Newren, May 10, 2019
  10. 3/5 fast-export: avoid stripping encoding header if we cannot reencodeElijah Newren, May 10, 2019
  11. 5/5 fast-export: do automatic reencoding of commit messages only if requestedElijah Newren, May 10, 2019
  12. Torsten BögershausenMay 11, 2019
  13. Elijah NewrenMay 11, 2019
  14. Junio C HamanoMay 13, 2019
  15. Elijah NewrenMay 13, 2019
  16. Johannes SchindelinMay 13, 2019
  17. Torsten BögershausenMay 13, 2019
  18. Elijah NewrenMay 13, 2019
  19. Elijah NewrenMay 13, 2019
  20. 4/5 fast-export: differentiate between explicitly utf-8 and implicitly utf-8Elijah Newren, May 10, 2019
  21. Johannes SchindelinMay 13, 2019
  22. 0/5 Fix and extend encoding handling in fast export/importElijah Newren, May 13, 2019
  23. 1/5 t9350: fix encoding test to actually test reencodingElijah Newren, May 13, 2019
  24. 3/5 fast-export: avoid stripping encoding header if we cannot reencodeElijah Newren, May 13, 2019
  25. 4/5 fast-export: differentiate between explicitly utf-8 and implicitly utf-8Elijah Newren, May 13, 2019
  26. 5/5 fast-export: do automatic reencoding of commit messages only if requestedElijah Newren, May 13, 2019
  27. Junio C HamanoMay 13, 2019
  28. 2/5 fast-import: support 'encoding' commit headerElijah Newren, May 13, 2019
  29. 0/5 Fix and extend encoding handling in fast export/importElijah Newren, May 13, 2019
  30. 1/5 t9350: fix encoding test to actually test reencodingElijah Newren, May 13, 2019
  31. Torsten BögershausenMay 14, 2019
  32. 2/5 fast-import: support 'encoding' commit headerElijah Newren, May 13, 2019
  33. 3/5 fast-export: avoid stripping encoding header if we cannot reencodeElijah Newren, May 13, 2019
  34. Torsten BögershausenMay 14, 2019
  35. 4/5 fast-export: differentiate between explicitly utf-8 and implicitly utf-8Elijah Newren, May 13, 2019
  36. Torsten BögershausenMay 14, 2019
  37. 5/5 fast-export: do automatic reencoding of commit messages only if requestedElijah Newren, May 13, 2019
  38. Eric SunshineMay 14, 2019
  39. 0/5 Fix and extend encoding handling in fast export/importElijah Newren, May 14, 2019
  40. 1/5 t9350: fix encoding test to actually test reencodingElijah Newren, May 14, 2019
  41. 5/5 fast-export: do automatic reencoding of commit messages only if requestedElijah Newren, May 14, 2019
  42. 2/5 fast-import: support 'encoding' commit headerElijah Newren, May 14, 2019
  43. 4/5 fast-export: differentiate between explicitly UTF-8 and implicitly UTF-8Elijah Newren, May 14, 2019
  44. 3/5 fast-export: avoid stripping encoding header if we cannot reencodeElijah Newren, May 14, 2019
  45. Torsten BögershausenMay 16, 2019

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.