git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: Cross-Platform Version Control

From
Linus Torvalds <torvalds@linux-foundation.org>
Date
May 13, 2009, 21:29 UTC
Message-ID
<alpine.LFD.2.01.0905131420500.3343@localhost.localdomain>
In-Reply-To
<alpine.LNX.2.00.0905131639580.2147@iabervon.org>
On Wed, 13 May 2009, Daniel Barkalow wrote:
Show 5 quoted lines
> > 
> > Now, the simple OS X case is not a huge problem, since the lstat will 
> > succeed with the fixed-up filename too.
> 
> I'm not seeing what the general case is, and how it could possibly behave.
Here's a simple example.

Let's say that your company uses Latin1 internally for your filesystems, because your tools really aren't utf-8 ready.

This is NOT AT ALL unnatural - it's how lots of people used to work with Linux over the years, and it's largely how people still use FAT, I suspect (except it's not latin1, it's some windows-specific 8-bits-per-character mapping).

IOW, if you have a file called 'åäö', it literally is encoded as '\xe5\xe4\xf6' (if you wonder why I picked those three letters, it's because they are the regular extra letters in Swedish - Swedish has 29 letters in its alphabet, and those three letters really are letters in their own right, they are NOT 'a' and 'o' with some dots/rings on top).

IOW, if you open such a file, you need to use those three bytes.

Now, even if you happen to have an OS and use Latin1 on disk, you may realize that you'd like to interact with others that use UTF-8, and would want to have your git archive that you export use nice portable UTF-8.

But you absolutely MUST NOT just do a conversion at "readdir()" time. If you do that, then your three-byte filename turns into a six-byte utf-8 sequence of '\xc3\xa5\xc3\xa4\xc3\xb6' and the thing is, now "lstat()" won't work on that sequence.

So obviously you could always turn things _back_ for lstat(), but quite frankly, that's (a) insane (b) incompetent and (c) not even always well-defined.

Show 5 quoted lines
> There's the "insensitive" behavior: if you create "foo" and look for 
> "FOO", it's there, but readdir() reports "foo".
> 
> There's the "converting" behavior: if you create "foo", readdir() reports 
> "FOO", but lstat("foo") returns it.

Then there's the behaviour above: you want your git repository to have utf-8, but your filesystem doesn't convert anything at all, and all your regular tools (think editors etc) are all Latin1.

Latin1 is going away, I hope, but I bet EUC-JP etc still exist. 
		Linus
Previous: Daniel BarkalowNext: Matthias Andree
Message 22 of 59 in “Re: Cross-Platform Version Control”
  1. Esko LuontolaMay 12, 2009
  2. Shawn O. PearceMay 12, 2009
  3. Johannes SchindelinMay 12, 2009
  4. Esko LuontolaMay 12, 2009
  5. Johannes SchindelinMay 12, 2009
  6. Esko LuontolaMay 12, 2009
  7. Johannes SchindelinMay 13, 2009
  8. Esko LuontolaMay 13, 2009
  9. Alex RiesenMay 13, 2009
  10. Johannes SchindelinMay 13, 2009
  11. Jeff KingMay 12, 2009
  12. Johannes SchindelinMay 12, 2009
  13. Linus TorvaldsMay 13, 2009
  14. Linus TorvaldsMay 13, 2009
  15. Andreas EricssonMay 13, 2009
  16. Linus TorvaldsMay 13, 2009
  17. Martin LanghoffMay 13, 2009
  18. Linus TorvaldsMay 13, 2009
  19. Theodore TsoMay 13, 2009
  20. Linus TorvaldsMay 13, 2009
  21. Daniel BarkalowMay 13, 2009
  22. Linus TorvaldsMay 13, 2009
  23. Matthias AndreeMay 13, 2009
  24. Linus TorvaldsMay 13, 2009
  25. Jay SoffianMay 13, 2009
  26. Matthias AndreeMay 13, 2009
  27. Dmitry PotapovMay 12, 2009
  28. Martin LanghoffMay 12, 2009
  29. Jakub NarebskiMay 12, 2009
  30. Extend sample pre-commit hook to check for non ascii file/usernamesHeiko Voigt, May 12, 2009
  31. Jakub NarebskiMay 12, 2009
  32. Extend sample pre-commit hook to check for non ascii filenamesHeiko Voigt, May 14, 2009
  33. Martin LanghoffMay 15, 2009
  34. Heiko VoigtMay 18, 2009
  35. Jakub NarebskiMay 18, 2009
  36. check for filenames that only differ in case to sample pre-commit hookHeiko Voigt, Jun 20, 2009
  37. Jakub NarebskiMay 15, 2009
  38. Extend sample pre-commit hook to check for non ascii filenamesHeiko Voigt, May 18, 2009
  39. Johannes SixtMay 18, 2009
  40. Heiko VoigtMay 18, 2009
  41. Johannes SixtMay 18, 2009
  42. Extend sample pre-commit hook to check for non ascii filenamesHeiko Voigt, May 19, 2009
  43. Junio C HamanoMay 18, 2009
  44. Julian PhillipsMay 18, 2009
  45. Junio C HamanoMay 15, 2009
  46. Peter KreftingMay 14, 2009
  47. Esko LuontolaMay 14, 2009
  48. Andreas EricssonMay 14, 2009
  49. Johannes SchindelinMay 14, 2009
  50. Dmitry PotapovMay 15, 2009
  51. Cross-Platform Version ControlJohn Tapsell, May 13, 2009
  52. Jay SoffianMay 13, 2009
  53. Alex RiesenMay 13, 2009
  54. Jay SoffianMay 13, 2009
  55. John TapsellMay 13, 2009
  56. Nicolas PitreMay 13, 2009
  57. Johannes SchindelinMay 13, 2009
  58. Andreas EricssonMay 13, 2009
  59. Miles BaderMay 14, 2009

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.