git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH v3 5/5] fast-export: do automatic reencoding of commit messages only if requested

From
Elijah Newren <newren@gmail.com>
Date
May 13, 2019, 13:29 UTC
Message-ID
<CABPp-BGbq3EJfpw-NykHM5TJdkpmqmttHYWonBP5kydV9PwTBg@mail.gmail.com>
In-Reply-To
<20190513125635.tntxhjbar2eold2i@tb-raspi4>
On Mon, May 13, 2019 at 5:56 AM Torsten Bögershausen <tboegi@web.de> wrote:
Show 30 quoted lines
>
> On Mon, May 13, 2019 at 12:23:29PM +0200, Johannes Schindelin wrote:
> > Hi Elijah,
> >
> > On Sat, 11 May 2019, Elijah Newren wrote:
> >
> > > [...] the craziness is based on how Windows behaves; it seems insane to
> > > me that Windows decides to munge user data (in the form of the command
> > > line provided), so much so that it makes me wonder if I really
> > > understood Hannes' and Dscho's explanations of what it is doing.
> >
> > It is not the user data that is munged by *Windows*, but by *Git for
> > Windows*. The user data on Windows is encoded in UTF-16 (or some slight
> > variant thereof). Git *cannot* handle UTF-16. Git's test suite *cannot*
> > handle UTF-16. So we convert. That's all there is to it.
> >
> > Ciao,
> > Dscho
> >
> > P.S.: Of course it is not *all* there is to it. There is also a current
> > code page which depends on the current user's current locale. We can
> > definitely not rely on that, as Git has no idea about this and would quite
> > positively produce incorrect output because of it. So we really just use
> > the `*W()` functions of the Win32 API (i.e. the ones accepting wide
> > Unicode characters and strings, i.e. UTF-16). I don't think we can do
> > better than that.
>
> We can actuall feed valid UTF-8 into a test case.
> (Remember that shell scripts need this octal numbering, see
> t/t0050)

Sure, but that's not useful here. I need to feed both valid and invalid ISO-8859-7 (or anything *other* than UTF-8) into a test case, in order to verify how git handles reencoding from something other than utf-8. I did something like what you proposed originally, but since it wasn't utf-8 it caused test failures on Windows.

Elijah
Previous: Torsten BögershausenNext: Elijah Newren
Message 18 of 45 in “Fix and extend encoding handling in fast export/import”
  1. 0/5 Fix and extend encoding handling in fast export/importElijah Newren, Apr 30, 2019
  2. 1/5 t9350: fix encoding test to actually test reencodingElijah Newren, Apr 30, 2019
  3. 2/5 fast-import: support 'encoding' commit headerElijah Newren, Apr 30, 2019
  4. 4/5 fast-export: differentiate between explicitly utf-8 and implicitly utf-8Elijah Newren, Apr 30, 2019
  5. 3/5 fast-export: avoid stripping encoding header if we cannot reencodeElijah Newren, Apr 30, 2019
  6. 5/5 fast-export: do automatic reencoding of commit messages only if requestedElijah Newren, Apr 30, 2019
  7. 0/5 Fix and extend encoding handling in fast export/importElijah Newren, May 10, 2019
  8. 1/5 t9350: fix encoding test to actually test reencodingElijah Newren, May 10, 2019
  9. 2/5 fast-import: support 'encoding' commit headerElijah Newren, May 10, 2019
  10. 3/5 fast-export: avoid stripping encoding header if we cannot reencodeElijah Newren, May 10, 2019
  11. 5/5 fast-export: do automatic reencoding of commit messages only if requestedElijah Newren, May 10, 2019
  12. Torsten BögershausenMay 11, 2019
  13. Elijah NewrenMay 11, 2019
  14. Junio C HamanoMay 13, 2019
  15. Elijah NewrenMay 13, 2019
  16. Johannes SchindelinMay 13, 2019
  17. Torsten BögershausenMay 13, 2019
  18. Elijah NewrenMay 13, 2019
  19. Elijah NewrenMay 13, 2019
  20. 4/5 fast-export: differentiate between explicitly utf-8 and implicitly utf-8Elijah Newren, May 10, 2019
  21. Johannes SchindelinMay 13, 2019
  22. 0/5 Fix and extend encoding handling in fast export/importElijah Newren, May 13, 2019
  23. 1/5 t9350: fix encoding test to actually test reencodingElijah Newren, May 13, 2019
  24. 3/5 fast-export: avoid stripping encoding header if we cannot reencodeElijah Newren, May 13, 2019
  25. 4/5 fast-export: differentiate between explicitly utf-8 and implicitly utf-8Elijah Newren, May 13, 2019
  26. 5/5 fast-export: do automatic reencoding of commit messages only if requestedElijah Newren, May 13, 2019
  27. Junio C HamanoMay 13, 2019
  28. 2/5 fast-import: support 'encoding' commit headerElijah Newren, May 13, 2019
  29. 0/5 Fix and extend encoding handling in fast export/importElijah Newren, May 13, 2019
  30. 1/5 t9350: fix encoding test to actually test reencodingElijah Newren, May 13, 2019
  31. Torsten BögershausenMay 14, 2019
  32. 2/5 fast-import: support 'encoding' commit headerElijah Newren, May 13, 2019
  33. 3/5 fast-export: avoid stripping encoding header if we cannot reencodeElijah Newren, May 13, 2019
  34. Torsten BögershausenMay 14, 2019
  35. 4/5 fast-export: differentiate between explicitly utf-8 and implicitly utf-8Elijah Newren, May 13, 2019
  36. Torsten BögershausenMay 14, 2019
  37. 5/5 fast-export: do automatic reencoding of commit messages only if requestedElijah Newren, May 13, 2019
  38. Eric SunshineMay 14, 2019
  39. 0/5 Fix and extend encoding handling in fast export/importElijah Newren, May 14, 2019
  40. 1/5 t9350: fix encoding test to actually test reencodingElijah Newren, May 14, 2019
  41. 5/5 fast-export: do automatic reencoding of commit messages only if requestedElijah Newren, May 14, 2019
  42. 2/5 fast-import: support 'encoding' commit headerElijah Newren, May 14, 2019
  43. 4/5 fast-export: differentiate between explicitly UTF-8 and implicitly UTF-8Elijah Newren, May 14, 2019
  44. 3/5 fast-export: avoid stripping encoding header if we cannot reencodeElijah Newren, May 14, 2019
  45. Torsten BögershausenMay 16, 2019

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.