threads / patch / 23863

patch, 7 partsAdd support for case insensitive directory and file lookups to git log

Subject: [PATCH 5/7] Add support for case insensitive directory and file lookups to git log

## tl;dr

9 messages between May 21, 2010 and May 26, 2010. Diffs are folded; open one to read it.

replies: 8people: 1as markdown or json

Joshua Jensen· May 21, 2010, 04:50 UTC · lore

[PATCH 0/7] Various updates to make core.ignorecase=true work better

This is a follow up patch to one I posted mid last year specifically for making .gitignore entries be processed in a case insensitive manner. I have used the patch with success since then, but I was unable to make the updates requested by list members.

Over the past month, I have been able to clean up the code and extend the core.ignorecase=true support to better handle git status, ls-files, log, add, and fast-import.

git status and add both use an update made to name-hash.c where directories, specifically names with a trailing slash, can be looked up in a case insensitive manner. After trying a myriad of solutions, this seemed to be the cleanest. Does anyone see a problem with embedding the directory names in the same hash as the file names? I couldn't find one, especially since I append a slash to each directory name.

The new directory_exists_in_index_icase() duplicates some of the functionality in directory_exists_in_index(). I don't understand some of the functionality I just mindlessly cloned. Can anyone clue me in on what directory_exists_in_index() achieves in the case of S_ISGITLINK()?

Updating match_one() bought case insensitive matching support for git ls-files? Are there other side effects I haven't found yet?

The git add path case folding functionality is a somewhat radical departure from what Git does now. It is described in detail in patch 6. Does anyone have any concerns?

Joshua Jensen (7):
  Add string comparison functions that respect the ignore_case
    variable.
  Case insensitivity support for .gitignore via core.ignorecase
  Add case insensitivity support for directories when using git status
  Add case insensitivity support when using git ls-files
  Add support for case insensitive directory and file lookups to git
    log
  Support case folding for git add when core.ignorecase=true
  Support case folding in git fast-import when core.ignorecase=true

 dir.c         |  110 +++++++++++++++++++++++++++++++++++++++++++++++----------
 dir.h         |    5 +++
 fast-import.c |    3 +-
 name-hash.c   |   44 ++++++++++++++++++++++-
 read-cache.c  |   16 ++++++++
 tree-diff.c   |    9 +++--
 6 files changed, 162 insertions(+), 25 deletions(-)

Joshua Jensen· May 21, 2010, 04:50 UTC · re: Joshua Jensen · lore

[PATCH 1/7] Add string comparison functions that respect the ignore_case variable.

Multiple locations within this patch series alter a case sensitive string comparison call such as strcmp() to be a call to a string comparison call that selects case comparison based on the global ignore_case variable. Behaviorally, when core.ignorecase=false, the *_icase() versions are functionally equivalent to their C runtime counterparts. When core.ignorecase=true, the *_icase() versions perform a case insensitive comparison.

Like Linus' earlier ignorecase patch, these may ignore filename conventions on certain file systems. By isolating filename comparisons to certain functions, support for those filename conventions may be more easily met.

Signed-off-by: Joshua Jensen <jjensen@workspacewhiz.com>
---
 dir.c |   35 +++++++++++++++++++++++++++++++++++
 dir.h |    5 +++++
 2 files changed, 40 insertions(+), 0 deletions(-)

Show changes to 2 files +40 −0

dir.c, dir.h

diff --git a/dir.c b/dir.c
index cb83332..21d2104 100644
--- a/dir.c
+++ b/dir.c
@@ -18,6 +18,41 @@ static int read_directory_recursive(struct dir_struct *dir, const char *path, in
 	int check_only, const struct path_simplify *simplify);
 static int get_dtype(struct dirent *de, const char *path, int len);
 
+/* helper string functions with support for the ignore_case flag */
+int strcmp_icase(const char *a, const char *b)
+{
+	return ignore_case ? strcasecmp(a, b) : strcmp(a, b);
+}
+
+int strncmp_icase(const char *a, const char *b, size_t count)
+{
+	return ignore_case ? strncasecmp(a, b, count) : strncmp(a, b, count);
+}
+
+int fnmatch_icase(const char *pattern, const char *string, int flags)
+{
+	return fnmatch(pattern, string, flags | (ignore_case ? FNM_CASEFOLD : 0));
+}
+
+int memcmp_icase(const char *a, const char *b, size_t count)
+{
+	if (ignore_case) {
+		int lowera = 0;
+		int lowerb = 0;
+		while (--count) {
+			lowera = tolower(*a++);
+			lowerb = tolower(*b++);
+			if (lowera != lowerb)
+				break;
+		}
+		return lowera - lowerb;
+
+		return 0;
+	} else {
+		return memcmp(a, b, count);
+	}
+}
+
 static int common_prefix(const char **pathspec)
 {
 	const char *path, *slash, *next;
diff --git a/dir.h b/dir.h
index 3bead5f..aced818 100644
--- a/dir.h
+++ b/dir.h
@@ -100,4 +100,9 @@ extern int remove_dir_recursively(struct strbuf *path, int flag);
 /* tries to remove the path with empty directories along it, ignores ENOENT */
 extern int remove_path(const char *path);
 
+extern int strcmp_icase(const char *a, const char *b);
+extern int strncmp_icase(const char *a, const char *b, size_t count);
+extern int fnmatch_icase(const char *pattern, const char *string, int flags);
+extern int memcmp_icase(const char *a, const char *b, size_t count);
+
 #endif

-- 
1.7.1.1930.gca7dd4

Joshua Jensen· May 21, 2010, 04:50 UTC · re: Joshua Jensen · lore

[PATCH 2/7] Case insensitivity support for .gitignore via core.ignorecase

This is especially beneficial when using Windows and Perforce and the git-p4 bridge. Internally, Perforce preserves a given file's full path including its case at the time it was added to the Perforce repository. When syncing a file down via Perforce, missing directories are created, if necessary, using the case as stored with the filename. Unfortunately, two files in the same directory can have differing cases for their respective paths, such as /diRa/file1.c and /DirA/file2.c. Depending on sync order, DirA/ may get created instead of diRa/.

It is possible to handle directory names in a case insensitive manner without this patch, but it is highly inconvenient, requiring each character to be specified like so: [Bb][Uu][Ii][Ll][Dd]. With this patch, the gitignore exclusions honor the core.ignorecase=true configuration setting and make the process less error prone. The above is specified like so: Build

Signed-off-by: Joshua Jensen <jjensen@workspacewhiz.com>
---
 dir.c |   12 ++++++------
 1 files changed, 6 insertions(+), 6 deletions(-)

Show changes to dir.c +6 −6

diff --git a/dir.c b/dir.c
index 21d2104..a19b7ab 100644
--- a/dir.c
+++ b/dir.c
@@ -409,14 +409,14 @@ int excluded_from_list(const char *pathname,
 			if (x->flags & EXC_FLAG_NODIR) {
 				/* match basename */
 				if (x->flags & EXC_FLAG_NOWILDCARD) {
-					if (!strcmp(exclude, basename))
+					if (!strcmp_icase(exclude, basename))
 						return to_exclude;
 				} else if (x->flags & EXC_FLAG_ENDSWITH) {
 					if (x->patternlen - 1 <= pathlen &&
-					    !strcmp(exclude + 1, pathname + pathlen - x->patternlen + 1))
+					    !strcmp_icase(exclude + 1, pathname + pathlen - x->patternlen + 1))
 						return to_exclude;
 				} else {
-					if (fnmatch(exclude, basename, 0) == 0)
+					if (fnmatch_icase(exclude, basename, 0) == 0)
 						return to_exclude;
 				}
 			}
@@ -431,14 +431,14 @@ int excluded_from_list(const char *pathname,
 
 				if (pathlen < baselen ||
 				    (baselen && pathname[baselen-1] != '/') ||
-				    strncmp(pathname, x->base, baselen))
+				    strncmp_icase(pathname, x->base, baselen))
 				    continue;
 
 				if (x->flags & EXC_FLAG_NOWILDCARD) {
-					if (!strcmp(exclude, pathname + baselen))
+					if (!strcmp_icase(exclude, pathname + baselen))
 						return to_exclude;
 				} else {
-					if (fnmatch(exclude, pathname+baselen,
+					if (fnmatch_icase(exclude, pathname+baselen,
 						    FNM_PATHNAME) == 0)
 					    return to_exclude;
 				}

-- 
1.7.1.1930.gca7dd4

Joshua Jensen· May 21, 2010, 04:50 UTC · re: Joshua Jensen · lore

[PATCH 3/7] Add case insensitivity support for directories when using git status

When using a case preserving but case insensitive file system, directory case can differ but still refer to the same physical directory. git status reports the directory with the alternate case as an Untracked file. (That is, when mydir/filea.txt is added to the repository and then the directory on disk is renamed from mydir/ to MyDir/, git status shows MyDir/ as being untracked.)

Support has been added in name-hash.c for hashing directories with a terminating slash into the name hash. When index_name_exists() is called with a directory (a name with a terminating slash), the name is not found via the normal cache_name_compare() call, but it is found in the slow_same_name() function.

Additionally, in dir.c, directory_exists_in_index_icase() allows newly added directories deeper in the directory chain to be identified.

Ultimately, it would be better if the file list was read in case insensitive alphabetical order from disk, but this change seems to suffice for now.

The end result is the directory is looked up in a case insensitive manner and does not show in the Untracked files list.

Signed-off-by: Joshua Jensen <jjensen@workspacewhiz.com>
---
 dir.c       |   25 ++++++++++++++++++++++++-
 name-hash.c |   44 +++++++++++++++++++++++++++++++++++++++++++-
 2 files changed, 67 insertions(+), 2 deletions(-)

Show changes to 2 files +67 −2

dir.c, name-hash.c

diff --git a/dir.c b/dir.c
index a19b7ab..c861f3e 100644
--- a/dir.c
+++ b/dir.c
@@ -503,6 +503,24 @@ enum exist_status {
 	index_gitdir,
 };
 
+static enum exist_status directory_exists_in_index_icase(const char *dirname, int len)
+{
+	struct cache_entry *ce = index_name_exists(&the_index, dirname, len + 1, ignore_case);
+	if (!ce)
+		return index_nonexistent;
+
+	if (!strncmp_icase(ce->name, dirname, len)) {
+		unsigned char endchar = ce->name[len];
+		if (endchar <= '/') {
+			if (endchar == '/')
+				return index_directory;
+			if (!endchar && S_ISGITLINK(ce->ce_mode))
+				return index_gitdir;
+		}
+	}
+	return index_nonexistent;
+}
+
 /*
  * The index sorts alphabetically by entry name, which
  * means that a gitlink sorts as '\0' at the end, while
@@ -512,7 +530,12 @@ enum exist_status {
  */
 static enum exist_status directory_exists_in_index(const char *dirname, int len)
 {
-	int pos = cache_name_pos(dirname, len);
+	int pos;
+
+	if (ignore_case)
+		return directory_exists_in_index_icase(dirname, len);
+
+	pos = cache_name_pos(dirname, len);
 	if (pos < 0)
 		pos = -pos-1;
 	while (pos < active_nr) {
diff --git a/name-hash.c b/name-hash.c
index 0031d78..b10b5b1 100644
--- a/name-hash.c
+++ b/name-hash.c
@@ -32,6 +32,30 @@ static unsigned int hash_name(const char *name, int namelen)
 	return hash;
 }
 
+static void hash_index_entry_directories(struct index_state *istate, struct cache_entry *ce)
+{
+	/* throw each directory component in the hash for quick lookup during a git status */
+	unsigned int hash;
+	void **pos;
+
+	const char *ptr = ce->name;
+	while (*ptr) {
+		while (*ptr && *ptr != '/')
+			++ptr;
+		if (*ptr == '/') {
+			++ptr;
+			hash = hash_name(ce->name, ptr - ce->name);
+			if (!lookup_hash(hash, &istate->name_hash)) {
+				pos = insert_hash(hash, ce, &istate->name_hash);
+				if (pos) {
+					ce->next = *pos;
+					*pos = ce;
+				}
+			}
+		}
+	}
+}
+
 static void hash_index_entry(struct index_state *istate, struct cache_entry *ce)
 {
 	void **pos;
@@ -47,6 +71,9 @@ static void hash_index_entry(struct index_state *istate, struct cache_entry *ce)
 		ce->next = *pos;
 		*pos = ce;
 	}
+
+	if (ignore_case)
+		hash_index_entry_directories(istate, ce);
 }
 
 static void lazy_init_name_hash(struct index_state *istate)
@@ -97,7 +124,22 @@ static int same_name(const struct cache_entry *ce, const char *name, int namelen
 	if (len == namelen && !cache_name_compare(name, namelen, ce->name, len))
 		return 1;
 
-	return icase && slow_same_name(name, namelen, ce->name, len);
+	if (!icase)
+		return 0;
+
+	/*
+	 * If the entry we're comparing is a filename (no trailing slash), then compare
+	 * the lengths exactly.
+	 */
+	if (name[namelen - 1] != '/') {
+		return slow_same_name(name, namelen, ce->name, len);
+	}
+
+	/*
+	 * For a directory, we point to an arbitrary cache_entry filename.  Just
+	 * make sure the directory portion matches.
+	 */
+	return slow_same_name(name, namelen, ce->name, namelen < len ? namelen : len);
 }
 
 struct cache_entry *index_name_exists(struct index_state *istate, const char *name, int namelen, int icase)

-- 
1.7.1.1930.gca7dd4

Joshua Jensen· May 21, 2010, 04:50 UTC · re: Joshua Jensen · lore

[PATCH 4/7] Add case insensitivity support when using git ls-files

When mydir/filea.txt is added, mydir/ is renamed to MyDir/, and MyDir/fileb.txt is added, running git ls-files mydir only shows mydir/filea.txt. Running git ls-files MyDir shows MyDir/fileb.txt. Running git ls-files mYdIR shows nothing.

With this patch running git ls-files for mydir, MyDir, and mYdIR shows mydir/filea.txt and MyDir/fileb.txt.

Wildcards are not handled case insensitively in this patch. Example: MyDir/aBc/file.txt is added. git ls-files MyDir/a* works fine, but git ls-files mydir/a* does not.

Signed-off-by: Joshua Jensen <jjensen@workspacewhiz.com>
---
 dir.c |   38 ++++++++++++++++++++++++++------------
 1 files changed, 26 insertions(+), 12 deletions(-)

Show changes to dir.c +26 −12

diff --git a/dir.c b/dir.c
index c861f3e..d67ec68 100644
--- a/dir.c
+++ b/dir.c
@@ -126,16 +126,30 @@ static int match_one(const char *match, const char *name, int namelen)
 	if (!*match)
 		return MATCHED_RECURSIVELY;
 
-	for (;;) {
-		unsigned char c1 = *match;
-		unsigned char c2 = *name;
-		if (c1 == '\0' || is_glob_special(c1))
-			break;
-		if (c1 != c2)
-			return 0;
-		match++;
-		name++;
-		namelen--;
+	if (ignore_case) {
+		for (;;) {
+			unsigned char c1 = tolower(*match);
+			unsigned char c2 = tolower(*name);
+			if (c1 == '\0' || is_glob_special(c1))
+				break;
+			if (c1 != c2)
+				return 0;
+			match++;
+			name++;
+			namelen--;
+		}
+	} else {
+		for (;;) {
+			unsigned char c1 = *match;
+			unsigned char c2 = *name;
+			if (c1 == '\0' || is_glob_special(c1))
+				break;
+			if (c1 != c2)
+				return 0;
+			match++;
+			name++;
+			namelen--;
+		}
 	}
 
 
@@ -144,8 +158,8 @@ static int match_one(const char *match, const char *name, int namelen)
 	 * we need to match by fnmatch
 	 */
 	matchlen = strlen(match);
-	if (strncmp(match, name, matchlen))
-		return !fnmatch(match, name, 0) ? MATCHED_FNMATCH : 0;
+	if (strncmp_icase(match, name, matchlen))
+		return !fnmatch_icase(match, name, 0) ? MATCHED_FNMATCH : 0;
 
 	if (namelen == matchlen)
 		return MATCHED_EXACTLY;

-- 
1.7.1.1930.gca7dd4

Joshua Jensen· May 21, 2010, 04:50 UTC · re: Joshua Jensen · lore

This patch also affects any other commands that use tree-diff.c.

Signed-off-by: Joshua Jensen <jjensen@workspacewhiz.com>
---
 tree-diff.c |    9 +++++----
 1 files changed, 5 insertions(+), 4 deletions(-)

Show changes to tree-diff.c +5 −4

diff --git a/tree-diff.c b/tree-diff.c
index fe9f52c..5110980 100644
--- a/tree-diff.c
+++ b/tree-diff.c
@@ -5,6 +5,7 @@
 #include "diff.h"
 #include "diffcore.h"
 #include "tree.h"
+#include "dir.h"
 
 static char *malloc_base(const char *base, int baselen, const char *path, int pathlen)
 {
@@ -114,7 +115,7 @@ static int tree_entry_interesting(struct tree_desc *desc, const char *base, int
 
 		if (baselen >= matchlen) {
 			/* If it doesn't match, move along... */
-			if (strncmp(base, match, matchlen))
+			if (strncmp_icase(base, match, matchlen))
 				continue;
 
 			/*
@@ -131,7 +132,7 @@ static int tree_entry_interesting(struct tree_desc *desc, const char *base, int
 		}
 
 		/* Does the base match? */
-		if (strncmp(base, match, baselen))
+		if (strncmp_icase(base, match, baselen))
 			continue;
 
 		match += baselen;
@@ -147,7 +148,7 @@ static int tree_entry_interesting(struct tree_desc *desc, const char *base, int
 			 * Does match sort strictly earlier than path
 			 * with their common parts?
 			 */
-			m = strncmp(match, path,
+			m = strncmp_icase(match, path,
 				    (matchlen < pathlen) ? matchlen : pathlen);
 			if (m < 0)
 				continue;
@@ -183,7 +184,7 @@ static int tree_entry_interesting(struct tree_desc *desc, const char *base, int
 			 * we cheated and did not do strncmp(), so we do
 			 * that here.
 			 */
-			m = strncmp(match, path, pathlen);
+			m = strncmp_icase(match, path, pathlen);
 
 		/*
 		 * If common part matched earlier then it is a hit,

-- 
1.7.1.1930.gca7dd4

Joshua Jensen· May 21, 2010, 04:50 UTC · re: Joshua Jensen · lore

[PATCH 6/7] Support case folding for git add when core.ignorecase=true

When MyDir/ABC/filea.txt is added to Git, the disk directory MyDir/ABC/ is renamed to mydir/aBc/, and then mydir/aBc/fileb.txt is added, the index will contain MyDir/ABC/filea.txt and mydir/aBc/fileb.txt. Although the earlier portions of this patch series account for those differences in case, this patch makes the pathing consistent by folding the case of newly added files against the first file added with that path.

In read-cache.c's add_to_index(), the index_name_exists() support used for git status's case insensitive directory lookups is used to find the proper directory case according to what the user already checked in. That is, MyDir/ABC/'s case is used to alter the stored path for fileb.txt to MyDir/ABC/fileb.txt (instead of mydir/aBc/fileb.txt).

This is especially important when cloning a repository to a case sensitive file system. MyDir/ABC/ and mydir/aBc/ exist in the same directory on a Windows machine, but on Linux, the files exist in two separate directories. The update to add_to_index(), in effect, treats a Windows file system as case sensitive by making path case consistent.

Signed-off-by: Joshua Jensen <jjensen@workspacewhiz.com>
---
 read-cache.c |   16 ++++++++++++++++
 1 files changed, 16 insertions(+), 0 deletions(-)

Show changes to read-cache.c +16 −0

diff --git a/read-cache.c b/read-cache.c
index f1f789b..b3954d5 100644
--- a/read-cache.c
+++ b/read-cache.c
@@ -608,6 +608,22 @@ int add_to_index(struct index_state *istate, const char *path, struct stat *st,
 		ce->ce_mode = ce_mode_from_stat(ent, st_mode);
 	}
 
+	if (ignore_case) {
+		const char *startPtr = ce->name;
+		const char *ptr = startPtr;
+		while (*ptr) {
+			while (*ptr && *ptr != '/')
+				++ptr;
+			if (*ptr == '/') {
+				struct cache_entry *foundce;
+				++ptr;
+				foundce = index_name_exists(&the_index, ce->name, ptr - ce->name, ignore_case);
+				if (foundce)
+					memcpy((void*)startPtr, foundce->name + (startPtr - ce->name), ptr - startPtr);
+			}
+		}
+	}
+
 	alias = index_name_exists(istate, ce->name, ce_namelen(ce), ignore_case);
 	if (alias && !ce_stage(alias) && !ie_match_stat(istate, alias, st, ce_option)) {
 		/* Nothing changed, really */

-- 
1.7.1.1930.gca7dd4

Joshua Jensen· May 21, 2010, 04:50 UTC · re: Joshua Jensen · lore

[PATCH 7/7] Support case folding in git fast-import when core.ignorecase=true

When core.ignorecase=true, imported file paths will be folded to match existing directory case.

Signed-off-by: Joshua Jensen <jjensen@workspacewhiz.com>
---
 fast-import.c |    3 ++-
 1 files changed, 2 insertions(+), 1 deletions(-)

Show changes to fast-import.c +2 −1

diff --git a/fast-import.c b/fast-import.c
index 352e2e3..2279276 100644
--- a/fast-import.c
+++ b/fast-import.c
@@ -156,6 +156,7 @@ Format of STDIN stream:
 #include "csum-file.h"
 #include "quote.h"
 #include "exec_cmd.h"
+#include "dir.h"
 
 #define PACK_ID_BITS 16
 #define MAX_PACK_ID ((1<<PACK_ID_BITS)-1)
@@ -1459,7 +1460,7 @@ static int tree_content_set(
 
 	for (i = 0; i < t->entry_count; i++) {
 		e = t->entries[i];
-		if (e->name->str_len == n && !strncmp(p, e->name->str_dat, n)) {
+		if (e->name->str_len == n && !strncmp_icase(p, e->name->str_dat, n)) {
 			if (!slash1) {
 				if (!S_ISDIR(mode)
 						&& e->versions[1].mode == mode

-- 
1.7.1.1930.gca7dd4

Joshua Jensen· May 26, 2010, 18:26 UTC · re: Joshua Jensen · lore

Re: [PATCH 0/7] Various updates to make core.ignorecase=true work better

  ----- Original Message -----
From: Joshua Jensen
Date: 5/20/2010 10:50 PM

Show 10 quoted lines

> Joshua Jensen (7):
>    Add string comparison functions that respect the ignore_case
>      variable.
>    Case insensitivity support for .gitignore via core.ignorecase
>    Add case insensitivity support for directories when using git status
>    Add case insensitivity support when using git ls-files
>    Add support for case insensitive directory and file lookups to git
>      log
>    Support case folding for git add when core.ignorecase=true
>    Support case folding in git fast-import when core.ignorecase=true

Would this patch series be better sent to the msysGit mailing list, given that it addresses Windows (and Mac OS X, I suppose) case preserving but case insensitive file system issues? I posted here first, because it builds on some core.ignorecase functionality Linus wrote.

Thanks.

Josh