threads / patch / 11966

patch[WIP] git on MacOSX and files with decomposed utf-8 file names

Subject: [PATCH] [WIP] git on MacOSX and files with decomposed utf-8 file names

## tl;dr

7 messages between Feb 8, 2008 and Feb 26, 2008. Diffs are folded; open one to read it.

replies: 6people: 2as markdown or json

Steffen Prohaska· Feb 8, 2008, 07:09 UTC · lore

Add case-challenged file system tests

There have been discussions lately on case-challenging file systems and UTF normalization on Mac OS X. I like to see these problems fixed and would like to start working on a resolution. But I did not follow the recent discussions closely.

Is anyone actively orking on these issues? What is the current status?

The patch below adds two simple tests that currently fail on Mac and Windows. I also collected Mitch's test on utf-8 and will send it as a reply to this mail. These two patches contain test cases that should pass, but do currently fail.

    Steffen

-- >8 -- Git behaves strangely (from a user's point of view) on filesystems that preserve case but do not distinguish filenames that only differ by case. The two major examples are Windows and Mac OS X. Simple operations such as "git mv" or "git merge" can fail unexpectedly.

This commit adds two simple tests. Both tests currently fail on Windows and Mac, although they pass on Linux.

Signed-off-by: Steffen Prohaska <prohaska@zib.de>
---
 t/t0050-filesystems.sh |   36 ++++++++++++++++++++++++++++++++++++
 1 files changed, 36 insertions(+), 0 deletions(-)
 create mode 100755 t/t0050-filesystems.sh
Show changes to t/t0050-filesystems.sh +36 −0
diff --git a/t/t0050-filesystems.sh b/t/t0050-filesystems.sh
new file mode 100755
index 0000000..953b02b
--- /dev/null
+++ b/t/t0050-filesystems.sh
@@ -0,0 +1,36 @@
+#!/bin/sh
+
+test_description='Various filesystems issues'
+
+. ./test-lib.sh
+
+test_expect_success setup '
+
+	touch camelcase &&
+	git add camelcase &&
+	git commit -m "initial" &&
+	git tag initial &&
+	git checkout -b topic &&
+	git mv camelcase tmp &&
+	git mv tmp CamelCase &&
+	git commit -m "rename" &&
+	git checkout -f master
+
+'
+
+test_expect_success 'rename (case change)' '
+
+	git mv camelcase CamelCase &&
+	git commit -m "rename"
+
+'
+
+test_expect_success 'merge (case change)' '
+
+	git reset --hard initial &&
+	git merge topic
+
+'
+
+
+test_done

-- 
1.5.4.40.g4a680
Steffen Prohaska· Feb 8, 2008, 07:09 UTC · re: Steffen Prohaska · lore
From: Mitch Tishmack <mitcht.git@gmail.com>
[ spr: This patch contains Mitch's work with his original email
       as the commit message. ]

Apologies Steffen, I grabbed your CamelCase test and did a search/replace, wasn't sure what to call it though... But I am on lunch and wanted to be useful. Rip it apart all you want.

Fails on hfs* on OSX, works on ufs. I will bother with zfs when it can be used again.

On UFS:
$ /bin/sh ./t0060-normalization.sh
*   ok 1: setup
*   ok 2: rename (silent normalization)
*   ok 3: merge (silent normalization)
* passed all 3 test(s)
On HFS:
$ /bin/sh t0060-normalization.sh
*   ok 1: setup
* FAIL 2: rename (silent normalization)
     git mv ä ä &&
     git commit -m "rename"
* FAIL 3: merge (silent normalization)
     git reset --hard initial &&
     git merge topic
* failed 2 among 3 test(s)
The test case, it uses perl, assuming only 5.6.1+ will work with this:
Signed-off-by: Steffen Prohaska <prohaska@zib.de>
---
 t/t0060-normalization.sh |   36 ++++++++++++++++++++++++++++++++++++
 1 files changed, 36 insertions(+), 0 deletions(-)
 create mode 100755 t/t0060-normalization.sh
Show changes to t/t0060-normalization.sh +36 −0
diff --git a/t/t0060-normalization.sh b/t/t0060-normalization.sh
new file mode 100755
index 0000000..f754fe4
--- /dev/null
+++ b/t/t0060-normalization.sh
@@ -0,0 +1,36 @@
+#!/bin/sh
+
+test_description='Test for silent normalization issues'
+
+. ./test-lib.sh
+
+auml=`perl -CO -e 'print pack("U",0x00E4)'`
+aumlcdiar=`perl -CO -e 'print pack("U",0x0061).pack("U",0x0308)'`
+test_expect_success setup "
+  touch $aumlcdiar &&
+  git add $aumlcdiar &&
+  git commit -m \"initial\"
+  git tag initial &&
+  git checkout -b topic &&
+  git mv $aumlcdiar tmp &&
+  git mv tmp $auml &&
+  git commit -m \"rename\" &&
+  git checkout -f master
+
+"
+
+test_expect_success 'rename (silent normalization)' "
+
+ git mv $aumlcdiar $auml &&
+ git commit -m \"rename\"
+
+"
+
+test_expect_success 'merge (silent normalization)' '
+
+ git reset --hard initial &&
+ git merge topic
+
+'
+
+test_done

-- 
1.5.4.40.g4a680
Junio C Hamano· Feb 8, 2008, 18:02 UTC · re: Steffen Prohaska · lore

Re: Add case-challenged file system tests

Steffen Prohaska <prohaska@zib.de> writes:
> Is anyone actively orking on these issues?
> What is the current status?
Not that I know of.
> The patch below adds two simple tests that currently fail on Mac
> and Windows.  I also collected Mitch's test on utf-8 and will
> send it as a reply to this mail.  These two patches contain test
> cases that should pass, but do currently fail.
Could you fold them into one?  They are about the same topic.

Also, mark them with test_expect_failure if these are meant to be applied.

Steffen Prohaska· Feb 24, 2008, 17:19 UTC · re: Junio C Hamano · lore

[PATCH] Add tests for filesystem challenges (case and unicode normalization)

Unfortunately, I had no time to start the real work of fixing the issues that are tested below. But at least the tests should be in good shape now and could be applied.

    Steffen

-- 8< -- Git has difficulties on file systems that do not properly distinguish case or modify filenames in unexpected ways. The two major examples are Windows and Mac OS X. Both systems preserve case of file names but do not distinguish between filenames that differ only by case. Simple operations such as "git mv" or "git merge" can fail unexpectedly. In addition, Mac OS X normalizes unicode, which make git's life even harder.

This commit adds tests that currently fail but should pass if file system as decribed above are fully supported. The test need to be run on Windows and Mac X as they already pass on Linux.

Mitch Tishmack is the original author of the tests for unicode normalization.

Signed-off-by: Steffen Prohaska <prohaska@zib.de>
---
 t/t0050-filesystem.sh |   67 +++++++++++++++++++++++++++++++++++++++++++++++++
 1 files changed, 67 insertions(+), 0 deletions(-)
 create mode 100755 t/t0050-filesystem.sh
Show changes to t/t0050-filesystem.sh +67 −0
diff --git a/t/t0050-filesystem.sh b/t/t0050-filesystem.sh
new file mode 100755
index 0000000..a0ab02e
--- /dev/null
+++ b/t/t0050-filesystem.sh
@@ -0,0 +1,67 @@
+#!/bin/sh
+
+test_description='Various filesystem issues'
+
+. ./test-lib.sh
+
+test_expect_success "setup case tests" '
+
+	touch camelcase &&
+	git add camelcase &&
+	git commit -m "initial" &&
+	git tag initial &&
+	git checkout -b topic &&
+	git mv camelcase tmp &&
+	git mv tmp CamelCase &&
+	git commit -m "rename" &&
+	git checkout -f master
+
+'
+
+test_expect_failure 'rename (case change)' '
+
+	git mv camelcase CamelCase &&
+	git commit -m "rename"
+
+'
+
+test_expect_failure 'merge (case change)' '
+
+	git reset --hard initial &&
+	git merge topic
+
+'
+
+auml=`perl -CO -e 'print pack("U",0x00E4)'`
+aumlcdiar=`perl -CO -e 'print pack("U",0x0061).pack("U",0x0308)'`
+test_expect_success "setup unicode normalization tests" "
+
+  test_create_repo unicode &&
+  cd unicode &&
+  touch $aumlcdiar &&
+  git add $aumlcdiar &&
+  git commit -m \"initial\"
+  git tag initial &&
+  git checkout -b topic &&
+  git mv $aumlcdiar tmp &&
+  git mv tmp $auml &&
+  git commit -m \"rename\" &&
+  git checkout -f master
+
+"
+
+test_expect_failure 'rename (silent unicode normalization)' "
+
+ git mv $aumlcdiar $auml &&
+ git commit -m \"rename\"
+
+"
+
+test_expect_failure 'merge (silent unicode normalization)' '
+
+ git reset --hard initial &&
+ git merge topic
+
+'
+
+test_done

-- 
1.5.4.3.310.g78af
Junio C Hamano· Feb 25, 2008, 20:44 UTC · re: Steffen Prohaska · lore

Re: [PATCH] Add tests for filesystem challenges (case and unicode normalization)

Steffen Prohaska <prohaska@zib.de> writes:
> Unfortunately, I had no time to start the real work of fixing the
> issues that are tested below.
That's fine.  We are not in a hurry.
> But at least the tests should be in
> good shape now and could be applied.

I do not think we would want to see "FIXED" on systems that already behave sanely, so we would want a fix-up like this on top of your patch, and it would be a good to go.

I do not have a handy way to test this, though, so can you try it out and make sure test_case and test_unicode are set to test_expect_failure correctly on problematic filesystems?

Thanks.
--
 t/t0050-filesystem.sh |   38 ++++++++++++++++++++++++++++++++------
 1 files changed, 32 insertions(+), 6 deletions(-)
Show changes to t/t0050-filesystem.sh +32 −6
diff --git a/t/t0050-filesystem.sh b/t/t0050-filesystem.sh
index a0ab02e..b395c22 100755
--- a/t/t0050-filesystem.sh
+++ b/t/t0050-filesystem.sh
@@ -4,6 +4,34 @@ test_description='Various filesystem issues'
 
 . ./test-lib.sh
 
+auml=`perl -CO -e 'print pack("U",0x00E4)'`
+aumlcdiar=`perl -CO -e 'print pack("U",0x0061).pack("U",0x0308)'`
+
+test_expect_success 'see if we expect ' '
+
+	test_case=test_expect_success
+	test_unicode=test_expect_success
+	mkdir junk &&
+	echo good >junk/CamelCase &&
+	echo bad >junk/camelcase &&
+	if test "$(cat junk/CamelCase)" != good
+	then
+		test_camel=test_expect_failure
+		say "will test on a case insensitive filesystem"
+	fi &&
+	rm -fr junk &&
+	mkdir junk &&
+	>junk/"$auml" &&
+	case "$(cd junk && echo *)" in
+	"$aumlcdiar")
+		test_unicode=test_expect_failure
+		say "will test on a unicode corrupting filesystem"
+		;;
+	*)	;;
+	esac &&
+	rm -fr junk
+'
+
 test_expect_success "setup case tests" '
 
 	touch camelcase &&
@@ -18,22 +46,20 @@ test_expect_success "setup case tests" '
 
 '
 
-test_expect_failure 'rename (case change)' '
+$test_case 'rename (case change)' '
 
 	git mv camelcase CamelCase &&
 	git commit -m "rename"
 
 '
 
-test_expect_failure 'merge (case change)' '
+$test_case 'merge (case change)' '
 
 	git reset --hard initial &&
 	git merge topic
 
 '
 
-auml=`perl -CO -e 'print pack("U",0x00E4)'`
-aumlcdiar=`perl -CO -e 'print pack("U",0x0061).pack("U",0x0308)'`
 test_expect_success "setup unicode normalization tests" "
 
   test_create_repo unicode &&
@@ -50,14 +76,14 @@ test_expect_success "setup unicode normalization tests" "
 
 "
 
-test_expect_failure 'rename (silent unicode normalization)' "
+$test_unicode 'rename (silent unicode normalization)' "
 
  git mv $aumlcdiar $auml &&
  git commit -m \"rename\"
 
 "
 
-test_expect_failure 'merge (silent unicode normalization)' '
+$test_unicode 'merge (silent unicode normalization)' '
 
  git reset --hard initial &&
  git merge topic
Steffen Prohaska· Feb 25, 2008, 23:34 UTC · re: Junio C Hamano · lore

Re: [PATCH] Add tests for filesystem challenges (case and unicode normalization)

On Feb 25, 2008, at 9:44 PM, Junio C Hamano wrote:
Show 6 quoted lines
> Steffen Prohaska <prohaska@zib.de> writes:
>
>> Unfortunately, I had no time to start the real work of fixing the
>> issues that are tested below.
>
> That's fine.  We are not in a hurry.

Well, I am, ... kind of. git constantly stumbles over a filename whose case changed. But fortunately I do not need to wait much longer until history will eventually have grown sufficiently such that the rename will no longer surface ;-)

Show 10 quoted lines
>> But at least the tests should be in
>> good shape now and could be applied.
>
> I do not think we would want to see "FIXED" on systems that
> already behave sanely, so we would want a fix-up like this on
> top of your patch, and it would be a good to go.
>
> I do not have a handy way to test this, though, so can you try
> it out and make sure test_case and test_unicode are set to
> test_expect_failure correctly on problematic filesystems?

This is a sensible way of handling this. It works for me, except for ...

Show 26 quoted lines
> --
>
>  t/t0050-filesystem.sh |   38 ++++++++++++++++++++++++++++++++------
>  1 files changed, 32 insertions(+), 6 deletions(-)
>
> diff --git a/t/t0050-filesystem.sh b/t/t0050-filesystem.sh
> index a0ab02e..b395c22 100755
> --- a/t/t0050-filesystem.sh
> +++ b/t/t0050-filesystem.sh
> @@ -4,6 +4,34 @@ test_description='Various filesystem issues'
>
>  . ./test-lib.sh
>
> +auml=`perl -CO -e 'print pack("U",0x00E4)'`
> +aumlcdiar=`perl -CO -e 'print pack("U",0x0061).pack("U",0x0308)'`
> +
> +test_expect_success 'see if we expect ' '
> +
> +	test_case=test_expect_success
> +	test_unicode=test_expect_success
> +	mkdir junk &&
> +	echo good >junk/CamelCase &&
> +	echo bad >junk/camelcase &&
> +	if test "$(cat junk/CamelCase)" != good
> +	then
> +		test_camel=test_expect_failure
test_case (instead of test_camel).
		Steffen
Junio C Hamano· Feb 26, 2008, 00:28 UTC · re: Steffen Prohaska · lore

Re: [PATCH] Add tests for filesystem challenges (case and unicode normalization)

Steffen Prohaska <prohaska@zib.de> writes:
> This is a sensible way of handling this.  It works for me,
> except for ...
> ...
> test_case (instead of test_camel).
Thanks for testing.  Fixed.

← back to recent threads