OSX, ZFS, UTF8, git - somebody hates me in this list

repeat by:
    0) download the OSX ZFS drivers
    1) Make a ZFS partition on OSX.
    2) Clone git.git to the ZFS partition
    3) immediately type git-status:

    # On branch master
    # Untracked files:
    #   (use "git add <file>..." to include in what will be committed)
    #
    #       "gitweb/test/Ma\314\210rchen"
    nothing added to commit but untracked files present (use "git add" to track)

    4) try "git-clean -f"
    Removing "gitweb/test/Ma\314\210rchen"

    5) try git-status now:
    # On branch master
    # Changed but not updated:
    #   (use "git add/rm <file>..." to update what will be committed)
    #
    #       deleted:    "gitweb/test/M\303\244rchen"
    #
    no changes added to commit (use "git add" and/or "git commit -a")

    6) still broken.  try git-reset --hard HEAD
    HEAD is now at c697ad1 Cleanup xread() loops to use read_in_full()
    7) and now git-status again:

    # On branch master
    # Untracked files:
    #   (use "git add <file>..." to include in what will be committed)
    #
    #       "gitweb/test/Ma\314\210rchen"
    nothing added to commit but untracked files present (use "git add" to track)

That's broke, and a show-stopper for me to use ZFS on OSX.  Ugh.

Can someone with ZFS on solaris try this, to try to figure out if this is a problem with git, ZFS, OSX, or some combination?

Thanks.

"Randal L. Schwartz" <merlyn@stonehenge.com> wrote:

Isn't this just the normal HFS+ name mangling?

Apparently the ZFS port to Mac OS X includes the same braindamaged UTF normalization that is applied to HFS+ volumes, which means they have reused the code or pushed it higher in their VFS layer. Linus has started to add support for teaching Git about this sort of brain damaged filesystem, but nobody has implemented the Mac OS X specific name hashing function required to make Git see that the name it got from the readdir(3) call is the same name Git passed to the OS, just encoded differently.

Shawn> "Randal L. Schwartz" <merlyn@stonehenge.com> wrote:

Shawn> Isn't this just the normal HFS+ name mangling?

I don't recall having this problem on HFS+.

But, oh, it's because my git.git is on a UFS partition. So UFS works fine. Why shouldn't ZFS (which will be closer to UFS than HFS+) be the same?

Shawn> Isn't this just the normal HFS+ name mangling?

Randal> I don't recall having this problem on HFS+.

Ahh, oddly enough, it does exactly the same on HFS+, but *not* UFS, which is why I hadn't noticed it yet, since all of my things I mirror are on my UFS partition, just in case they have case-colliding names.

I'm trying to get rid of my UFS partition because it's a pain in the rear, so I was hoping to upgrade to ZFS. Hence, the query.

On Sun, May 04, 2008 at 03:00:11PM -0700, Randal L. Schwartz wrote:

Anyone know if 'case sensitive' HFS+ does name mangling? Perhaps ZFS can also be formed as case sensitive.

David

El 5/5/2008, a las 8:52, David Brown escribió:

I believe case-sensitive HFS+ also mangles names.

I for one am pretty disappointed that Apple has chosen to bring across this undesirable behaviour from their legacy file system to a completely new one, when this would have been a perfect opportunity for them to break with the bad design decisions of the past. It's still pre-release, of course, so I guess there's still hope that their final version might not preserve this flaw.

Wincent

"Randal L. Schwartz" <merlyn@stonehenge.com> wrote:

Everyone has this problem on HFS+.  :-|
 

Because Apple is Apple and things must be done The Apple Way(tm), or no black turtleneck for you?

Shawn> Because Apple is Apple and things must be done The Apple Way(tm), Shawn> or no black turtleneck for you?

So apparently the problem is known. Is it also understood? Is the solution understood, and just waiting for implementor tuits?

"Randal L. Schwartz" <merlyn@stonehenge.com> wrote:

Yes, I think the issue has been beat to death here on the mailing list by many parties. Its more just an issue of someone writing the code and finding the little implementation details that weren't discussed yet.

Shawn> Isn't this just the normal HFS+ name mangling?

Randal> I don't recall having this problem on HFS+.

Also, ZFS is case-sensitive, which leads me to believe that out of the box, it's more like UFS.

Still don't know why ZFS breaks but UFS works.

On Mon, 5 May 2008, Randal L. Schwartz wrote:

UFS is a traditional Unix filesystem, and will not mangle your filenames.

ZFS apparently acts like a case-sensitive HFS+: it still tries to normalize to UTF-8 (and does it badly, at that - picking an Apple-specific almost-NFD form of normalization rather than the more sensible NFC form).

So ZFS may not corrupt cases, but it still corrupts UTF-8 filenames.

			Linus

On Mon, May 5, 2008 at 10:53 AM, Linus Torvalds <torvalds@linux-foundation.org> wrote:

Not sure if that is always the case. This thread suggests otherwise: http://opensolaris.org/jive/thread.jspa?threadID=53303&tstart=0

Case sensitivity and mangling look like they are configurable. See the "casesensitivity" and "normalization" options at http://docs.sun.com/app/docs/doc/819-2240/zfs-1m

ZFS on OSX may just default to "insensitive" and some form of normalization especially if they were/are thinking about making it the default fs and wanted backwards compatibility for broken apps. Randal, can you make a new FS with casesensitivity=sensitive and normalization=none and see how it behaves?

-Tarmigan

Tarmigan> ZFS on OSX may just default to "insensitive" and some form of Tarmigan> normalization especially if they were/are thinking about making it the Tarmigan> default fs and wanted backwards compatibility for broken apps. Tarmigan> Randal, can you make a new FS with casesensitivity=sensitive and Tarmigan> normalization=none and see how it behaves?

As I posted in a followup, without twiddling anything, ZFS on OSX is case *sensitive*, but apparently still "normalizes", which is causing the trip-up. I'll look to see if there are any parameters I can tweak to keep it from doing that, but I doubt it.

On Mon, 5 May 2008, Randal L. Schwartz wrote:

Well, the good news is that while the utf-8 normalization is crazy, you're pretty unlikely to actually hit it in practice. The git archive has the Märchen file on purpose to find these kinds of issues, but it's harmless appart from the annoyance.

Yes, some projects will have i18n support that may have files with special characters on purpose, and they can cause problems. Not least exactly because OS X does all the normalization exactly the wrong way (ie it chose the _unusual_ way to do things, expanding strings to their longest format rather than their shortest format), but I would expect it to be very rare in general, partly because it doesn't really matter what SCM you use to track things, and you'll have similar issues if you ever develop anything across OS's.

We can fix it. Junio hasn't merged my stupid infrastructure into core git yet, and I still don't really care enough or have a OS X box to test on (I've got a couple of mac minis, but they run that other OS ;), but we'll get the insane OS X crap handled *some* day.

			Linus

Hi,

On Mon, 5 May 2008, Linus Torvalds wrote:

FWIW I think you can reproduce on Linux. AFAIR there is some code in the hfs or hfsplus modules to perform the same UTF-8 normalization.

Ciao, Dscho

On May 5, 2008, at 02:00 PM, Tarmigan wrote:

Those are only valid in OpenSolaris at the moment, they do nothing on the Leopard zfs beta: macbookprov2# zfs set casesensitivity=sensitive zfs cannot set property for 'zfs': invalid property 'casesensitivity' macbookprov2# zfs set normalization=none zfs cannot set property for 'zfs': invalid property 'normalization' macbookprov2# uname -a Darwin macbookprov2.local 9.2.2 Darwin Kernel Version 9.2.2: Tue Mar 4 21:17:34 PST 2008; root:xnu-1228.4.31~1/RELEASE_I386 i386 macbookprov2# zfs get all zfs NAME PROPERTY VALUE SOURCE zfs type filesystem - zfs creation Sun May 4 22:03 2008 - zfs used 38.3M - zfs available 922M - zfs referenced 38.2M - zfs compressratio 1.00x - zfs mounted yes - zfs quota none default zfs reservation none default zfs recordsize 128K default zfs mountpoint /Volumes/zfs default zfs sharenfs off default zfs checksum on default zfs compression off default zfs atime on default zfs devices on default zfs exec on default zfs setuid on default zfs readonly off default zfs zoned off default zfs snapdir hidden default zfs aclmode groupmask default zfs aclinherit secure default zfs canmount on default zfs shareiscsi off default zfs xattr on default zfs copies 1 default zfs version 1 -

mitch