{"thread":{"id":"23595","subject":"[PATCH RFC v2 0/4] Patches to avoid reporting conversion changes.","startedAt":"2010-04-25T16:29:04Z","lastAt":"2010-04-25T16:29:08Z","messageCount":5,"participants":["Henrik Grubbström (Grubba)"],"isPatch":true,"patchVersion":2,"patchTotal":4},"messages":[{"id":"140346","messageId":"cover.1272210580.git.grubba@grubba.org","threadId":"23595","inReplyTo":null,"subject":"[PATCH RFC v2 0/4] Patches to avoid reporting conversion changes.","fromName":"Henrik Grubbström (Grubba)","fromEmail":"grubba@grubba.org","sentAt":"2010-04-25T16:29:04Z","receivedAt":"2010-04-25T16:29:04Z","isPatch":true,"sender":{"key":"grubba@grubba.org","avatar":"https://avatars.githubusercontent.com/u/1169458?v=4"},"body":"This is the second go at having the git index keep track of the\nconversion mode and corresponding normalized blob sha1 for files.\n\nSince last time the diff behaviour patch has been removed, and\nwith it the need for storing of the normalized blob and thus\nthe gc patch. Some basic test cases have been added.\n\nThanks to Junio C Hamano for the suggestion and some of the tests.\n\nHenrik Grubbström (Grubba) (4):\n  sha1_file: Added index_blob().\n  cache: Added ce_norm_sha1() and related cache_entry fields.\n  cache: Added index extension \"NORM\".\n  t/t0021: Test that conversion changes are detected.\n\n cache.h               |   22 ++++++++++++++++\n convert.c             |   31 +++++++++++++++++++++++\n read-cache.c          |   66 +++++++++++++++++++++++++++++++++++++++++++------\n sha1_file.c           |   19 ++++++++++++++\n t/t0021-conversion.sh |   50 +++++++++++++++++++++++++++++++++++++\n 5 files changed, 180 insertions(+), 8 deletions(-)\n"},{"id":"140347","messageId":"73348f7cf4878413eaf97d8d9a88fce8be7c24df.1272210580.git.grubba@grubba.org","threadId":"23595","inReplyTo":"cover.1272210580.git.grubba@grubba.org","subject":"[PATCH RFC v2 1/4] sha1_file: Added index_blob().","fromName":"Henrik Grubbström (Grubba)","fromEmail":"grubba@grubba.org","sentAt":"2010-04-25T16:29:05Z","receivedAt":"2010-04-25T16:29:05Z","isPatch":true,"sender":{"key":"grubba@grubba.org","avatar":"https://avatars.githubusercontent.com/u/1169458?v=4"},"body":"When conversion attributes have changed, it\nis useful to be able to easily reconvert an\nexisting blob.\n\nSigned-off-by: Henrik Grubbström <grubba@grubba.org>\n---\nNo changes since v1.\n\n cache.h     |    1 +\n sha1_file.c |   19 +++++++++++++++++++\n 2 files changed, 20 insertions(+), 0 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 5eb0573..1fe2d7d 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -494,6 +494,7 @@ extern int ie_match_stat(const struct index_state *, struct cache_entry *, struc\n extern int ie_modified(const struct index_state *, struct cache_entry *, struct stat *, unsigned int);\n \n extern int ce_path_match(const struct cache_entry *ce, const char **pathspec);\n+extern int index_blob(unsigned char *dst_sha1, const unsigned char *src_sha1, int write_object, const char *path);\n extern int index_fd(unsigned char *sha1, int fd, struct stat *st, int write_object, enum object_type type, const char *path);\n extern int index_path(unsigned char *sha1, const char *path, struct stat *st, int write_object);\n extern void fill_stat_cache_info(struct cache_entry *ce, struct stat *st);\ndiff --git a/sha1_file.c b/sha1_file.c\nindex ff65328..c162321 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -2434,6 +2434,25 @@ static int index_mem(unsigned char *sha1, void *buf, size_t size,\n \n #define SMALL_FILE_SIZE (32*1024)\n \n+int index_blob(unsigned char *dst_sha1, const unsigned char *src_sha1,\n+\t       int write_object, const char *path)\n+{\n+\tvoid *buf;\n+\tunsigned long buflen = 0;\n+\tint ret;\n+\n+\tmemcpy(dst_sha1, src_sha1, 20);\n+\tbuf = read_object_with_reference(src_sha1, typename(OBJ_BLOB),\n+\t\t\t\t\t &buflen, dst_sha1);\n+\tif (!buf)\n+\t\treturn 0;\n+\n+\tret = index_mem(dst_sha1, buf, buflen, write_object, OBJ_BLOB, path);\n+\tfree(buf);\n+\n+\treturn ret;\n+}\n+\n int index_fd(unsigned char *sha1, int fd, struct stat *st, int write_object,\n \t     enum object_type type, const char *path)\n {\n-- \n1.7.0.4.369.g81e89\n"},{"id":"140350","messageId":"871d5dbed4353d86854ef2705bb14bc352ea57c2.1272210580.git.grubba@grubba.org","threadId":"23595","inReplyTo":"cover.1272210580.git.grubba@grubba.org","subject":"[PATCH RFC v2 2/4] cache: Added ce_norm_sha1() and related cache_entry fields.","fromName":"Henrik Grubbström (Grubba)","fromEmail":"grubba@grubba.org","sentAt":"2010-04-25T16:29:06Z","receivedAt":"2010-04-25T16:29:06Z","isPatch":true,"sender":{"key":"grubba@grubba.org","avatar":"https://avatars.githubusercontent.com/u/1169458?v=4"},"body":"The index now keeps track of the conversion mode that was active\nwhen the entry was created. This can be used to detect the most\ncommon cases of when the conversion mode has changed.\n\nSigned-off-by: Henrik Grubbström <grubba@grubba.org>\n---\ngit_norm_flags has been extended with one flag (NORM_CONV_CRLF_WT)\nto be able to keep track of the working tree state as well as the\nrepository state.\n\ngit_norm_flags() now takes account of the auto_crlf state.\n\nce_match_stat_basic() now knows that a normalization change may\naffect the working tree file size.\n\nUpdating of the normalization state is now done in ce_compare_data().\n\n cache.h      |   14 ++++++++++++++\n convert.c    |   31 +++++++++++++++++++++++++++++++\n read-cache.c |   17 +++++++++++++++--\n 3 files changed, 60 insertions(+), 2 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 1fe2d7d..3e70bef 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -151,10 +151,18 @@ struct cache_entry {\n \tunsigned int ce_size;\n \tunsigned int ce_flags;\n \tunsigned char sha1[20];\n+\tunsigned int norm_flags;\n+\tunsigned char norm_sha1[20];\n \tstruct cache_entry *next;\n \tchar name[FLEX_ARRAY]; /* more */\n };\n \n+#define NORM_CONV_CRLF_GIT\t0x0001\n+#define NORM_CONV_CRLF_WT\t0x0002\n+#define NORM_CONV_CRLF_GUESS\t0x0004\n+#define NORM_CONV_IDENT\t\t0x0008\n+#define NORM_CONV_FILT\t\t0x0010\n+\n #define CE_NAMEMASK  (0x0fff)\n #define CE_STAGEMASK (0x3000)\n #define CE_EXTENDED  (0x4000)\n@@ -278,6 +286,11 @@ static inline int ce_to_dtype(const struct cache_entry *ce)\n \telse\n \t\treturn DT_UNKNOWN;\n }\n+static inline unsigned char *ce_norm_sha1(struct cache_entry *ce)\n+{\n+\treturn ce->norm_flags?ce->norm_sha1:ce->sha1;\n+}\n+\n #define canon_mode(mode) \\\n \t(S_ISREG(mode) ? (S_IFREG | ce_permissions(mode)) : \\\n \tS_ISLNK(mode) ? S_IFLNK : S_ISDIR(mode) ? S_IFDIR : S_IFGITLINK)\n@@ -1014,6 +1027,7 @@ extern void trace_argv_printf(const char **argv, const char *format, ...);\n \n /* convert.c */\n /* returns 1 if *dst was used */\n+extern unsigned int git_norm_flags(const char *path);\n extern int convert_to_git(const char *path, const char *src, size_t len,\n                           struct strbuf *dst, enum safe_crlf checksafe);\n extern int convert_to_working_tree(const char *path, const char *src, size_t len, struct strbuf *dst);\ndiff --git a/convert.c b/convert.c\nindex 4f8fcb7..5f36669 100644\n--- a/convert.c\n+++ b/convert.c\n@@ -568,6 +568,37 @@ static int git_path_check_ident(const char *path, struct git_attr_check *check)\n \treturn !!ATTR_TRUE(value);\n }\n \n+unsigned int git_norm_flags(const char *path)\n+{\n+\tstruct git_attr_check check[3];\n+\tint crlf = CRLF_GUESS;\n+\tint ident = 0;\n+\tunsigned ret = 0;\n+\tstruct convert_driver *drv = NULL;\n+\n+\tsetup_convert_check(check);\n+\tif (!git_checkattr(path, ARRAY_SIZE(check), check)) {\n+\t\tcrlf = git_path_check_crlf(path, check + 0);\n+\t\tident = git_path_check_ident(path, check + 1);\n+\t\tdrv = git_path_check_convert(path, check + 2);\n+\t}\n+\n+\tif (auto_crlf && (crlf != CRLF_BINARY)) {\n+\t\tret |= NORM_CONV_CRLF_GIT;\n+\t\tif (crlf != CRLF_INPUT && auto_crlf > 0)\n+\t\t\tret |= NORM_CONV_CRLF_WT;\n+\t\tif (crlf == CRLF_GUESS)\n+\t\t\tret |= NORM_CONV_CRLF_GUESS;\n+\t}\n+\tif (ident) {\n+\t\tret |= NORM_CONV_IDENT;\n+\t}\n+\tif (drv) {\n+\t\tret |= NORM_CONV_FILT;\n+\t}\n+\treturn ret;\n+}\n+\n int convert_to_git(const char *path, const char *src, size_t len,\n                    struct strbuf *dst, enum safe_crlf checksafe)\n {\ndiff --git a/read-cache.c b/read-cache.c\nindex f1f789b..1a698bf 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -88,12 +88,20 @@ void fill_stat_cache_info(struct cache_entry *ce, struct stat *st)\n static int ce_compare_data(struct cache_entry *ce, struct stat *st)\n {\n \tint match = -1;\n-\tint fd = open(ce->name, O_RDONLY);\n+\tint fd;\n+\tunsigned int norm_flags = git_norm_flags(ce->name);\n \n+\tif (norm_flags != ce->norm_flags) {\n+\t\tce->norm_flags = norm_flags;\n+\t\tif (norm_flags)\n+\t\t\tindex_blob(ce->norm_sha1, ce->sha1, 0, ce->name);\n+\t}\n+\n+\tfd = open(ce->name, O_RDONLY);\n \tif (fd >= 0) {\n \t\tunsigned char sha1[20];\n \t\tif (!index_fd(sha1, fd, st, 0, OBJ_BLOB, ce->name))\n-\t\t\tmatch = hashcmp(sha1, ce->sha1);\n+\t\t\tmatch = hashcmp(sha1, ce_norm_sha1(ce));\n \t\t/* index_fd() closed the file descriptor already */\n \t}\n \treturn match;\n@@ -227,6 +235,11 @@ static int ce_match_stat_basic(struct cache_entry *ce, struct stat *st)\n \t\tchanged |= INODE_CHANGED;\n #endif\n \n+\t/* ce_size can not be trusted if the conversion mode has changed. */\n+\tif ((ce->ce_mode & S_IFMT) == S_IFREG &&\n+\t    ce->norm_flags != git_norm_flags(ce->name))\n+\t\treturn changed;\n+\n \tif (ce->ce_size != (unsigned int) st->st_size)\n \t\tchanged |= DATA_CHANGED;\n \n-- \n1.7.0.4.369.g81e89\n"},{"id":"140349","messageId":"2c51a366fcf74e24a94002524a41eb8e90e2cf91.1272210580.git.grubba@grubba.org","threadId":"23595","inReplyTo":"cover.1272210580.git.grubba@grubba.org","subject":"[PATCH RFC v2 3/4] cache: Added index extension \"NORM\".","fromName":"Henrik Grubbström (Grubba)","fromEmail":"grubba@grubba.org","sentAt":"2010-04-25T16:29:07Z","receivedAt":"2010-04-25T16:29:07Z","isPatch":true,"sender":{"key":"grubba@grubba.org","avatar":"https://avatars.githubusercontent.com/u/1169458?v=4"},"body":"The index can now store and retrieve the ce_norm_sha1 data.\n\nSigned-off-by: Henrik Grubbström <grubba@grubba.org>\n---\nUnchanged since v1.\n\n cache.h      |    7 +++++++\n read-cache.c |   49 +++++++++++++++++++++++++++++++++++++++++++------\n 2 files changed, 50 insertions(+), 6 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 3e70bef..9aa031b 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -157,6 +157,13 @@ struct cache_entry {\n \tchar name[FLEX_ARRAY]; /* more */\n };\n \n+struct ondisk_norm_sha1 {\n+\tunsigned int entry_no;\n+\tunsigned int norm_flags;\n+\tunsigned int norm_size;\n+\tunsigned char norm_sha1[20];\n+};\n+\n #define NORM_CONV_CRLF_GIT\t0x0001\n #define NORM_CONV_CRLF_WT\t0x0002\n #define NORM_CONV_CRLF_GUESS\t0x0004\ndiff --git a/read-cache.c b/read-cache.c\nindex 1a698bf..5abb59d 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -27,6 +27,7 @@ static struct cache_entry *refresh_cache_entry(struct cache_entry *ce, int reall\n #define CACHE_EXT(s) ( (s[0]<<24)|(s[1]<<16)|(s[2]<<8)|(s[3]) )\n #define CACHE_EXT_TREE 0x54524545\t/* \"TREE\" */\n #define CACHE_EXT_RESOLVE_UNDO 0x52455543 /* \"REUC\" */\n+#define CACHE_EXT_NORM_SHA1 0x4e4f524d\t/* \"NORM\" */\n \n struct index_state the_index;\n \n@@ -1191,6 +1192,21 @@ static int verify_hdr(struct cache_header *hdr, unsigned long size)\n \treturn 0;\n }\n \n+static int norm_sha1_read(struct cache_entry **cache, unsigned int entries,\n+\t\t\t  const struct ondisk_norm_sha1 *data, unsigned long sz)\n+{\n+\twhile (sz >= sizeof(*data)) {\n+\t\tunsigned int entry_no = ntohl(data->entry_no);\n+\t\tif (entry_no < entries) {\n+\t\t\tcache[entry_no]->norm_flags = ntohl(data->norm_flags);\n+\t\t\tmemcpy(cache[entry_no]->norm_sha1, data->norm_sha1, 20);\n+\t\t}\n+\t\tsz -= sizeof(*data);\n+\t\tdata++;\n+\t}\n+\treturn 0;\n+}\n+\n static int read_index_extension(struct index_state *istate,\n \t\t\t\tconst char *ext, void *data, unsigned long sz)\n {\n@@ -1201,6 +1217,9 @@ static int read_index_extension(struct index_state *istate,\n \tcase CACHE_EXT_RESOLVE_UNDO:\n \t\tistate->resolve_undo = resolve_undo_read(data, sz);\n \t\tbreak;\n+\tcase CACHE_EXT_NORM_SHA1:\n+\t\treturn norm_sha1_read(istate->cache, istate->cache_nr, data, sz);\n+\t\tbreak;\n \tdefault:\n \t\tif (*ext < 'A' || 'Z' < *ext)\n \t\t\treturn error(\"index uses %.4s extension, which we do not understand\",\n@@ -1524,6 +1543,16 @@ static void ce_smudge_racily_clean_entry(struct cache_entry *ce)\n \t}\n }\n \n+static void norm_sha1_write(struct strbuf *sb, const struct cache_entry *ce,\n+\t\t\t    int entry_no)\n+{\n+\tstruct ondisk_norm_sha1 entry;\n+\tentry.entry_no = htonl(entry_no);\n+\tentry.norm_flags = htonl(ce->norm_flags);\n+\tmemcpy(entry.norm_sha1, ce->norm_sha1, 20);\n+\tstrbuf_add(sb, &entry, sizeof(entry));\n+}\n+\n static int ce_write_entry(git_SHA_CTX *c, int fd, struct cache_entry *ce)\n {\n \tint size = ondisk_ce_size(ce);\n@@ -1559,10 +1588,11 @@ int write_index(struct index_state *istate, int newfd)\n {\n \tgit_SHA_CTX c;\n \tstruct cache_header hdr;\n-\tint i, err, removed, extended;\n+\tint i, j, err, removed, extended;\n \tstruct cache_entry **cache = istate->cache;\n \tint entries = istate->cache_nr;\n \tstruct stat st;\n+\tstruct strbuf sb = STRBUF_INIT;\n \n \tfor (i = removed = extended = 0; i < entries; i++) {\n \t\tif (cache[i]->ce_flags & CE_REMOVE)\n@@ -1585,7 +1615,7 @@ int write_index(struct index_state *istate, int newfd)\n \tif (ce_write(&c, newfd, &hdr, sizeof(hdr)) < 0)\n \t\treturn -1;\n \n-\tfor (i = 0; i < entries; i++) {\n+\tfor (i = j = 0; i < entries; i++) {\n \t\tstruct cache_entry *ce = cache[i];\n \t\tif (ce->ce_flags & CE_REMOVE)\n \t\t\tcontinue;\n@@ -1593,12 +1623,21 @@ int write_index(struct index_state *istate, int newfd)\n \t\t\tce_smudge_racily_clean_entry(ce);\n \t\tif (ce_write_entry(&c, newfd, ce) < 0)\n \t\t\treturn -1;\n+\t\tif (ce->norm_flags)\n+\t\t\tnorm_sha1_write(&sb, ce, j);\n+\t\tj++;\n \t}\n \n \t/* Write extension data here */\n+\tif (sb.len) {\n+\t\terr = write_index_ext_header(&c, newfd, CACHE_EXT_NORM_SHA1,\n+\t\t\t\t\t     sb.len) < 0\n+\t\t\t|| ce_write(&c, newfd, sb.buf, sb.len) < 0;\n+\t\tstrbuf_release(&sb);\n+\t\tif (err)\n+\t\t\treturn -1;\n+\t}\n \tif (istate->cache_tree) {\n-\t\tstruct strbuf sb = STRBUF_INIT;\n-\n \t\tcache_tree_write(&sb, istate->cache_tree);\n \t\terr = write_index_ext_header(&c, newfd, CACHE_EXT_TREE, sb.len) < 0\n \t\t\t|| ce_write(&c, newfd, sb.buf, sb.len) < 0;\n@@ -1607,8 +1646,6 @@ int write_index(struct index_state *istate, int newfd)\n \t\t\treturn -1;\n \t}\n \tif (istate->resolve_undo) {\n-\t\tstruct strbuf sb = STRBUF_INIT;\n-\n \t\tresolve_undo_write(&sb, istate->resolve_undo);\n \t\terr = write_index_ext_header(&c, newfd, CACHE_EXT_RESOLVE_UNDO,\n \t\t\t\t\t     sb.len) < 0\n-- \n1.7.0.4.369.g81e89\n"},{"id":"140348","messageId":"2be6dfd1358e8eda4d2ebb9b8a763602b46b60d8.1272210580.git.grubba@grubba.org","threadId":"23595","inReplyTo":"cover.1272210580.git.grubba@grubba.org","subject":"[PATCH RFC v2 4/4] t/t0021: Test that conversion changes are detected.","fromName":"Henrik Grubbström (Grubba)","fromEmail":"grubba@grubba.org","sentAt":"2010-04-25T16:29:08Z","receivedAt":"2010-04-25T16:29:08Z","isPatch":true,"sender":{"key":"grubba@grubba.org","avatar":"https://avatars.githubusercontent.com/u/1169458?v=4"},"body":"Signed-off-by Henrik Grubbström <grubba@grubba.org>\n---\nThanks to Junio C Hamano for some of the tests.\n\n t/t0021-conversion.sh |   50 +++++++++++++++++++++++++++++++++++++++++++++++++\n 1 files changed, 50 insertions(+), 0 deletions(-)\n\ndiff --git a/t/t0021-conversion.sh b/t/t0021-conversion.sh\nindex 6cb8d60..b6de203 100755\n--- a/t/t0021-conversion.sh\n+++ b/t/t0021-conversion.sh\n@@ -89,4 +89,54 @@ test_expect_success expanded_in_repo '\n \tcmp expanded-keywords expected-output\n '\n \n+# Check that files that have had their canonical representation\n+# changed since being checked in aren't reported as modified\n+# directly after being checked out.\n+test_expect_success keywords_not_modified '\n+\t{\n+\t\techo \"File with foreign keywords\"\n+\t\techo \"\\$Id\\$\"\n+\t\techo \"\\$Id: NoTerminatingSymbol\"\n+\t\techo \"\\$Id: Foreign Commit With Spaces \\$\"\n+\t\techo \"\\$Id: GitCommitId \\$\"\n+\t\techo \"\\$Id: NoTerminatingSymbolAtEOF\"\n+\t} > expanded-keywords2 &&\n+\n+\tgit add expanded-keywords2 &&\n+\tgit commit -m \"File with keywords expanded\" &&\n+\n+\techo \"expanded-keywords2 ident\" >> .gitattributes &&\n+\n+\trm -f expanded-keywords2 &&\n+\tgit checkout -- expanded-keywords2 &&\n+\n+\ttest \"x`git status --porcelain -- expanded-keywords2`\" = x\n+'\n+\n+# Test detection of CRLF conversion changes CRLF ==> LF.\n+test_expect_success crlf_conversion_change_crlf_to_lf '\n+\t# step 0. a blob with CRLF\n+\tgit init one && cd one &&\n+\techo -e \"a quick brown fox\\015\" >kuzu &&\n+\tgit add kuzu && git commit -m kuzu &&\n+\t# step 1. you want CRLF in work area, LF in repository\n+\tgit config core.autocrlf true &&\n+\t# step 2. user edit and revert.\n+\ttouch kuzu &&\n+\tgit update-index --refresh\n+'\n+\n+# Test detection of CRLF conversion changes LF ==> CRLF.\n+test_expect_success crlf_conversion_change_lf_to_crlf '\n+\t# step 0 & 1. a project with LF ending\n+\tgit init two && cd two &&\n+\techo a quick brown fox >kuzu &&\n+\tgit add kuzu && git commit -m kuzu &&\n+\t# step 2. you want CRLF in your work area\n+\techo -e \"a quick brown fox\\015\" >kuzu &&\n+\tgit config core.autocrlf true &&\n+\t# step 3. oops, refresh\n+\tgit update-index --refresh\n+'\n+\n test_done\n-- \n1.7.0.4.369.g81e89\n"}]}