{"thread":{"id":"34419","subject":"[PATCH v2 00/19] Index-v5","startedAt":"2013-07-12T17:26:45Z","lastAt":"2013-08-09T13:10:31Z","messageCount":51,"participants":["Thomas Gummerer","Duy Nguyen","Ramsay Jones","Junio C Hamano"],"isPatch":true,"patchVersion":2,"patchTotal":19},"messages":[{"id":"223174","messageId":"1373650024-3001-1-git-send-email-t.gummerer@gmail.com","threadId":"34419","inReplyTo":null,"subject":"[PATCH v2 00/19] Index-v5","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2013-07-12T17:26:45Z","receivedAt":"2013-07-12T17:26:45Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"Hi,\n\nprevious rounds (without api) are at $gmane/202752, $gmane/202923,\n$gmane/203088 and $gmane/203517, the previous round with api was at\n$gmane/229732. Thanks to Junio, Duy and Eric for their comments on\nthe previous round.\n\nChanges since the previous round:\n\nAdd documentation for the index reading api (was sent later in the\nprevious series as 5.5/22).\n\nread-cache: add index reading api\n\n - Remove the next_ce pointer, and use index_state->cache[] as part\n of the api, making a few functions unnecessary (next_index_entry,\n get_index_entry_by_name, sort_index). Those were removed.\n\n - Replace const char **pathspec with const struct pathspec\n\n - remove the index_change_filter_opts function. For better\n performance the caller should make up its mind before reading\n anything which part of the index file should be read. Changing\n the filter_opts later makes us read part of the index again, which\n is sub-optimal.\n\n - set the filter_opts to NULL when discarding the cache.\n\n - discussed and didn't change: replacing the pathspec with\n tree_entry_interesting doesn't work well. Using common_prefix()\n to calculate the adjusted_pathspec would work, but has some\n disadvantages when glob is used in the pathspec.\n\nmake sure partially read index is not changed\n\n - instead of using the index_change_filter_opts function to set the\n filter_opts to NULL before changing the index, die() when the\n caller tries that. \n\n - use filter_opts to check if the index was partially read instead\n of an extra flag.\n\n - die() when trying to re-read a partially read index.\n\npatches (6,7,8,9,12/22) were removed, as they are no longer needed\nwhen keeping index_state->cache[] as part of the api.\n\nls-files: use index api\n\n - only use the new api for loading the filtered index, but don't use\n the for_each_index_entry function, as it would need to change the\n filter_opts after reading the index. That will be made obsolete\n by a series that Duy has cooking [1], after which the loops can be\n switched out by using for_each_index_entry.\n\nOther than that, the typos found by Eric were fixed.\n\n[1] http://thread.gmane.org/gmane.comp.version-control.git/229732/focus=230023\n\nThomas Gummerer (18):\n t2104: Don't fail for index versions other than [23]\n read-cache: split index file version specific functionality\n read-cache: move index v2 specific functions to their own file\n read-cache: Re-read index if index file changed\n Add documentation for the index api\n read-cache: add index reading api\n make sure partially read index is not changed\n grep.c: Use index api\n ls-files.c: use index api\n documentation: add documentation of the index-v5 file format\n read-cache: make in-memory format aware of stat_crc\n read-cache: read index-v5\n read-cache: read resolve-undo data\n read-cache: read cache-tree in index-v5\n read-cache: write index-v5\n read-cache: write index-v5 cache-tree data\n read-cache: write resolve-undo data for index-v5\n update-index.c: rewrite index when index-version is given\n\nThomas Rast (1):\n p0003-index.sh: add perf test for the index formats\n\n Documentation/technical/api-in-core-index.txt | 54 +-\n Documentation/technical/index-file-format-v5.txt | 296 +++++\n Makefile | 3 +\n builtin/grep.c | 71 +-\n builtin/ls-files.c | 31 +-\n builtin/update-index.c | 8 +-\n cache-tree.c | 2 +-\n cache-tree.h | 6 +\n cache.h | 140 +-\n read-cache-v2.c | 589 +++++++++\n read-cache-v5.c | 1516 ++++++++++++++++++++++\n read-cache.c | 676 +++-------\n read-cache.h | 65 +\n t/perf/p0003-index.sh | 59 +\n t/t2104-update-index-skip-worktree.sh | 1 +\n test-index-version.c | 7 +-\n 16 files changed, 2942 insertions(+), 582 deletions(-)\n create mode 100644 Documentation/technical/index-file-format-v5.txt\n create mode 100644 read-cache-v2.c\n create mode 100644 read-cache-v5.c\n create mode 100644 read-cache.h\n create mode 100755 t/perf/p0003-index.sh\n\n-- \n1.8.3.453.g1dfc63d\n"},{"id":"223175","messageId":"1373650024-3001-2-git-send-email-t.gummerer@gmail.com","threadId":"34419","inReplyTo":"1373650024-3001-1-git-send-email-t.gummerer@gmail.com","subject":"[PATCH v2 01/19] t2104: Don't fail for index versions other than [23]","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2013-07-12T17:26:46Z","receivedAt":"2013-07-12T17:26:46Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"t2104 currently checks for the exact index version 2 or 3,\ndepending if there is a skip-worktree flag or not. Other\nindex versions do not use extended flags and thus cannot\nbe tested for version changes.\n\nMake this test update the index to version 2 at the beginning\nof the test. Testing the skip-worktree flags for the default\nindex format is still covered by t7011 and t7012.\n\nSigned-off-by: Thomas Gummerer \n---\n t/t2104-update-index-skip-worktree.sh | 1 +\n 1 file changed, 1 insertion(+)\n\ndiff --git a/t/t2104-update-index-skip-worktree.sh b/t/t2104-update-index-skip-worktree.sh\nindex 1d0879b..bd9644f 100755\n--- a/t/t2104-update-index-skip-worktree.sh\n+++ b/t/t2104-update-index-skip-worktree.sh\n@@ -22,6 +22,7 @@ H sub/2\n EOF\n \n test_expect_success 'setup' '\n+\tgit update-index --index-version=2 &&\n \tmkdir sub &&\n \ttouch ./1 ./2 sub/1 sub/2 &&\n \tgit add 1 2 sub/1 sub/2 &&\n-- \n1.8.3.453.g1dfc63d\n"},{"id":"223176","messageId":"1373650024-3001-3-git-send-email-t.gummerer@gmail.com","threadId":"34419","inReplyTo":"1373650024-3001-1-git-send-email-t.gummerer@gmail.com","subject":"[PATCH v2 02/19] read-cache: split index file version specific functionality","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2013-07-12T17:26:47Z","receivedAt":"2013-07-12T17:26:47Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"Split index file version specific functionality to their own functions,\nto prepare for moving the index file version specific parts to their own\nfile. This makes it easier to add a new index file format later.\n\nSigned-off-by: Thomas Gummerer \n---\n cache.h | 5 +-\n read-cache.c | 130 +++++++++++++++++++++++++++++++++------------------\n test-index-version.c | 2 +-\n 3 files changed, 90 insertions(+), 47 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex c288678..7af853b 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -100,9 +100,12 @@ unsigned long git_deflate_bound(git_zstream *, unsigned long);\n */\n \n #define CACHE_SIGNATURE 0x44495243\t/* \"DIRC\" */\n-struct cache_header {\n+struct cache_version_header {\n \tunsigned int hdr_signature;\n \tunsigned int hdr_version;\n+};\n+\n+struct cache_header {\n \tunsigned int hdr_entries;\n };\n \ndiff --git a/read-cache.c b/read-cache.c\nindex d5201f9..93947bf 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -1268,10 +1268,8 @@ struct ondisk_cache_entry_extended {\n \t\t\t ondisk_cache_entry_extended_size(ce_namelen(ce)) : \\\n \t\t\t ondisk_cache_entry_size(ce_namelen(ce)))\n \n-static int verify_hdr(struct cache_header *hdr, unsigned long size)\n+static int verify_hdr_version(struct cache_version_header *hdr, unsigned long size)\n {\n-\tgit_SHA_CTX c;\n-\tunsigned char sha1[20];\n \tint hdr_version;\n \n \tif (hdr->hdr_signature != htonl(CACHE_SIGNATURE))\n@@ -1279,10 +1277,22 @@ static int verify_hdr(struct cache_header *hdr, unsigned long size)\n \thdr_version = ntohl(hdr->hdr_version);\n \tif (hdr_version < INDEX_FORMAT_LB || INDEX_FORMAT_UB < hdr_version)\n \t\treturn error(\"bad index version %d\", hdr_version);\n+\treturn 0;\n+}\n+\n+static int verify_hdr(void *mmap, unsigned long size)\n+{\n+\tgit_SHA_CTX c;\n+\tunsigned char sha1[20];\n+\n+\tif (size < sizeof(struct cache_version_header)\n+\t + sizeof(struct cache_header) + 20)\n+\t\tdie(\"index file smaller than expected\");\n+\n \tgit_SHA1_Init(&c);\n-\tgit_SHA1_Update(&c, hdr, size - 20);\n+\tgit_SHA1_Update(&c, mmap, size - 20);\n \tgit_SHA1_Final(sha1, &c);\n-\tif (hashcmp(sha1, (unsigned char *)hdr + size - 20))\n+\tif (hashcmp(sha1, (unsigned char *)mmap + size - 20))\n \t\treturn error(\"bad index file sha1 signature\");\n \treturn 0;\n }\n@@ -1424,47 +1434,19 @@ static struct cache_entry *create_from_disk(struct ondisk_cache_entry *ondisk,\n \treturn ce;\n }\n \n-/* remember to discard_cache() before reading a different cache! */\n-int read_index_from(struct index_state *istate, const char *path)\n+static int read_index_v2(struct index_state *istate, void *mmap, unsigned long mmap_size)\n {\n-\tint fd, i;\n-\tstruct stat st;\n+\tint i;\n \tunsigned long src_offset;\n-\tstruct cache_header *hdr;\n-\tvoid *mmap;\n-\tsize_t mmap_size;\n+\tstruct cache_version_header *hdr;\n+\tstruct cache_header *hdr_v2;\n \tstruct strbuf previous_name_buf = STRBUF_INIT, *previous_name;\n \n-\tif (istate->initialized)\n-\t\treturn istate->cache_nr;\n-\n-\tistate->timestamp.sec = 0;\n-\tistate->timestamp.nsec = 0;\n-\tfd = open(path, O_RDONLY);\n-\tif (fd < 0) {\n-\t\tif (errno == ENOENT)\n-\t\t\treturn 0;\n-\t\tdie_errno(\"index file open failed\");\n-\t}\n-\n-\tif (fstat(fd, &st))\n-\t\tdie_errno(\"cannot stat the open index\");\n-\n-\tmmap_size = xsize_t(st.st_size);\n-\tif (mmap_size < sizeof(struct cache_header) + 20)\n-\t\tdie(\"index file smaller than expected\");\n-\n-\tmmap = xmmap(NULL, mmap_size, PROT_READ | PROT_WRITE, MAP_PRIVATE, fd, 0);\n-\tif (mmap == MAP_FAILED)\n-\t\tdie_errno(\"unable to map index file\");\n-\tclose(fd);\n-\n \thdr = mmap;\n-\tif (verify_hdr(hdr, mmap_size) < 0)\n-\t\tgoto unmap;\n+\thdr_v2 = (struct cache_header *)((char *)mmap + sizeof(*hdr));\n \n \tistate->version = ntohl(hdr->hdr_version);\n-\tistate->cache_nr = ntohl(hdr->hdr_entries);\n+\tistate->cache_nr = ntohl(hdr_v2->hdr_entries);\n \tistate->cache_alloc = alloc_nr(istate->cache_nr);\n \tistate->cache = xcalloc(istate->cache_alloc, sizeof(*istate->cache));\n \tistate->initialized = 1;\n@@ -1474,7 +1456,7 @@ int read_index_from(struct index_state *istate, const char *path)\n \telse\n \t\tprevious_name = NULL;\n \n-\tsrc_offset = sizeof(*hdr);\n+\tsrc_offset = sizeof(*hdr) + sizeof(*hdr_v2);\n \tfor (i = 0; i < istate->cache_nr; i++) {\n \t\tstruct ondisk_cache_entry *disk_ce;\n \t\tstruct cache_entry *ce;\n@@ -1487,8 +1469,6 @@ int read_index_from(struct index_state *istate, const char *path)\n \t\tsrc_offset += consumed;\n \t}\n \tstrbuf_release(&previous_name_buf);\n-\tistate->timestamp.sec = st.st_mtime;\n-\tistate->timestamp.nsec = ST_MTIME_NSEC(st);\n \n \twhile (src_offset <= mmap_size - 20 - 8) {\n \t\t/* After an array of active_nr index entries,\n@@ -1508,6 +1488,58 @@ int read_index_from(struct index_state *istate, const char *path)\n \t\tsrc_offset += 8;\n \t\tsrc_offset += extsize;\n \t}\n+\treturn 0;\n+unmap:\n+\tmunmap(mmap, mmap_size);\n+\tdie(\"index file corrupt\");\n+}\n+\n+/* remember to discard_cache() before reading a different cache! */\n+int read_index_from(struct index_state *istate, const char *path)\n+{\n+\tint fd;\n+\tstruct stat st;\n+\tstruct cache_version_header *hdr;\n+\tvoid *mmap;\n+\tsize_t mmap_size;\n+\n+\terrno = EBUSY;\n+\tif (istate->initialized)\n+\t\treturn istate->cache_nr;\n+\n+\terrno = ENOENT;\n+\tistate->timestamp.sec = 0;\n+\tistate->timestamp.nsec = 0;\n+\tfd = open(path, O_RDONLY);\n+\tif (fd < 0) {\n+\t\tif (errno == ENOENT)\n+\t\t\treturn 0;\n+\t\tdie_errno(\"index file open failed\");\n+\t}\n+\n+\tif (fstat(fd, &st))\n+\t\tdie_errno(\"cannot stat the open index\");\n+\n+\terrno = EINVAL;\n+\tmmap_size = xsize_t(st.st_size);\n+\tif (mmap_size < sizeof(struct cache_header) + 20)\n+\t\tdie(\"index file smaller than expected\");\n+\n+\tmmap = xmmap(NULL, mmap_size, PROT_READ | PROT_WRITE, MAP_PRIVATE, fd, 0);\n+\tclose(fd);\n+\tif (mmap == MAP_FAILED)\n+\t\tdie_errno(\"unable to map index file\");\n+\n+\thdr = mmap;\n+\tif (verify_hdr_version(hdr, mmap_size) < 0)\n+\t\tgoto unmap;\n+\n+\tif (verify_hdr(mmap, mmap_size) < 0)\n+\t\tgoto unmap;\n+\n+\tread_index_v2(istate, mmap, mmap_size);\n+\tistate->timestamp.sec = st.st_mtime;\n+\tistate->timestamp.nsec = ST_MTIME_NSEC(st);\n \tmunmap(mmap, mmap_size);\n \treturn istate->cache_nr;\n \n@@ -1771,10 +1803,11 @@ void update_index_if_able(struct index_state *istate, struct lock_file *lockfile\n \t\trollback_lock_file(lockfile);\n }\n \n-int write_index(struct index_state *istate, int newfd)\n+static int write_index_v2(struct index_state *istate, int newfd)\n {\n \tgit_SHA_CTX c;\n-\tstruct cache_header hdr;\n+\tstruct cache_version_header hdr;\n+\tstruct cache_header hdr_v2;\n \tint i, err, removed, extended, hdr_version;\n \tstruct cache_entry **cache = istate->cache;\n \tint entries = istate->cache_nr;\n@@ -1804,11 +1837,13 @@ int write_index(struct index_state *istate, int newfd)\n \n \thdr.hdr_signature = htonl(CACHE_SIGNATURE);\n \thdr.hdr_version = htonl(hdr_version);\n-\thdr.hdr_entries = htonl(entries - removed);\n+\thdr_v2.hdr_entries = htonl(entries - removed);\n \n \tgit_SHA1_Init(&c);\n \tif (ce_write(&c, newfd, &hdr, sizeof(hdr)) < 0)\n \t\treturn -1;\n+\tif (ce_write(&c, newfd, &hdr_v2, sizeof(hdr_v2)) < 0)\n+\t\treturn -1;\n \n \tprevious_name = (hdr_version == 4) ? &previous_name_buf : NULL;\n \tfor (i = 0; i < entries; i++) {\n@@ -1854,6 +1889,11 @@ int write_index(struct index_state *istate, int newfd)\n \treturn 0;\n }\n \n+int write_index(struct index_state *istate, int newfd)\n+{\n+\treturn write_index_v2(istate, newfd);\n+}\n+\n /*\n * Read the index file that is potentially unmerged into given\n * index_state, dropping any unmerged entries. Returns true if\ndiff --git a/test-index-version.c b/test-index-version.c\nindex 05d4699..4c0386f 100644\n--- a/test-index-version.c\n+++ b/test-index-version.c\n@@ -2,7 +2,7 @@\n \n int main(int argc, char **argv)\n {\n-\tstruct cache_header hdr;\n+\tstruct cache_version_header hdr;\n \tint version;\n \n \tmemset(&hdr,0,sizeof(hdr));\n-- \n1.8.3.453.g1dfc63d\n"},{"id":"223177","messageId":"1373650024-3001-4-git-send-email-t.gummerer@gmail.com","threadId":"34419","inReplyTo":"1373650024-3001-1-git-send-email-t.gummerer@gmail.com","subject":"[PATCH v2 03/19] read-cache: move index v2 specific functions to their own file","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2013-07-12T17:26:48Z","receivedAt":"2013-07-12T17:26:48Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"Move index version 2 specific functions to their own file. The non-index\nspecific functions will be in read-cache.c, while the index version 2\nspecific functions will be in read-cache-v2.c.\n\nHelped-by: Nguyen Thai Ngoc Duy \nSigned-off-by: Thomas Gummerer \n---\n Makefile | 2 +\n cache.h | 16 +-\n read-cache-v2.c | 556 +++++++++++++++++++++++++++++++++++++++++++++++++\n read-cache.c | 575 ++++-----------------------------------------------\n read-cache.h | 57 +++++\n test-index-version.c | 5 +\n 6 files changed, 661 insertions(+), 550 deletions(-)\n create mode 100644 read-cache-v2.c\n create mode 100644 read-cache.h\n\ndiff --git a/Makefile b/Makefile\nindex 5a68fe5..73369ae 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -711,6 +711,7 @@ LIB_H += progress.h\n LIB_H += prompt.h\n LIB_H += quote.h\n LIB_H += reachable.h\n+LIB_H += read-cache.h\n LIB_H += reflog-walk.h\n LIB_H += refs.h\n LIB_H += remote.h\n@@ -854,6 +855,7 @@ LIB_OBJS += prompt.o\n LIB_OBJS += quote.o\n LIB_OBJS += reachable.o\n LIB_OBJS += read-cache.o\n+LIB_OBJS += read-cache-v2.o\n LIB_OBJS += reflog-walk.o\n LIB_OBJS += refs.o\n LIB_OBJS += remote.o\ndiff --git a/cache.h b/cache.h\nindex 7af853b..5082b34 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -95,19 +95,8 @@ unsigned long git_deflate_bound(git_zstream *, unsigned long);\n */\n #define DEFAULT_GIT_PORT 9418\n \n-/*\n- * Basic data structures for the directory cache\n- */\n \n #define CACHE_SIGNATURE 0x44495243\t/* \"DIRC\" */\n-struct cache_version_header {\n-\tunsigned int hdr_signature;\n-\tunsigned int hdr_version;\n-};\n-\n-struct cache_header {\n-\tunsigned int hdr_entries;\n-};\n \n #define INDEX_FORMAT_LB 2\n #define INDEX_FORMAT_UB 4\n@@ -280,6 +269,7 @@ struct index_state {\n \t\t initialized : 1;\n \tstruct hash_table name_hash;\n \tstruct hash_table dir_hash;\n+\tstruct index_ops *ops;\n };\n \n extern struct index_state the_index;\n@@ -489,8 +479,8 @@ extern void *read_blob_data_from_index(struct index_state *, const char *, unsig\n #define CE_MATCH_RACY_IS_DIRTY\t\t02\n /* do stat comparison even if CE_SKIP_WORKTREE is true */\n #define CE_MATCH_IGNORE_SKIP_WORKTREE\t04\n-extern int ie_match_stat(const struct index_state *, const struct cache_entry *, struct stat *, unsigned int);\n-extern int ie_modified(const struct index_state *, const struct cache_entry *, struct stat *, unsigned int);\n+extern int ie_match_stat(struct index_state *, const struct cache_entry *, struct stat *, unsigned int);\n+extern int ie_modified(struct index_state *, const struct cache_entry *, struct stat *, unsigned int);\n \n #define PATHSPEC_ONESTAR 1\t/* the pathspec pattern sastisfies GFNM_ONESTAR */\n \ndiff --git a/read-cache-v2.c b/read-cache-v2.c\nnew file mode 100644\nindex 0000000..a6883c3\n--- /dev/null\n+++ b/read-cache-v2.c\n@@ -0,0 +1,556 @@\n+#include \"cache.h\"\n+#include \"read-cache.h\"\n+#include \"resolve-undo.h\"\n+#include \"cache-tree.h\"\n+#include \"varint.h\"\n+\n+/* Mask for the name length in ce_flags in the on-disk index */\n+#define CE_NAMEMASK (0x0fff)\n+\n+struct cache_header {\n+\tunsigned int hdr_entries;\n+};\n+\n+/*****************************************************************\n+ * Index File I/O\n+ *****************************************************************/\n+\n+/*\n+ * dev/ino/uid/gid/size are also just tracked to the low 32 bits\n+ * Again - this is just a (very strong in practice) heuristic that\n+ * the inode hasn't changed.\n+ *\n+ * We save the fields in big-endian order to allow using the\n+ * index file over NFS transparently.\n+ */\n+struct ondisk_cache_entry {\n+\tstruct cache_time ctime;\n+\tstruct cache_time mtime;\n+\tunsigned int dev;\n+\tunsigned int ino;\n+\tunsigned int mode;\n+\tunsigned int uid;\n+\tunsigned int gid;\n+\tunsigned int size;\n+\tunsigned char sha1[20];\n+\tunsigned short flags;\n+\tchar name[FLEX_ARRAY]; /* more */\n+};\n+\n+/*\n+ * This struct is used when CE_EXTENDED bit is 1\n+ * The struct must match ondisk_cache_entry exactly from\n+ * ctime till flags\n+ */\n+struct ondisk_cache_entry_extended {\n+\tstruct cache_time ctime;\n+\tstruct cache_time mtime;\n+\tunsigned int dev;\n+\tunsigned int ino;\n+\tunsigned int mode;\n+\tunsigned int uid;\n+\tunsigned int gid;\n+\tunsigned int size;\n+\tunsigned char sha1[20];\n+\tunsigned short flags;\n+\tunsigned short flags2;\n+\tchar name[FLEX_ARRAY]; /* more */\n+};\n+\n+/* These are only used for v3 or lower */\n+#define align_flex_name(STRUCT,len) ((offsetof(struct STRUCT,name) + (len) + 8) & ~7)\n+#define ondisk_cache_entry_size(len) align_flex_name(ondisk_cache_entry,len)\n+#define ondisk_cache_entry_extended_size(len) align_flex_name(ondisk_cache_entry_extended,len)\n+#define ondisk_ce_size(ce) (((ce)->ce_flags & CE_EXTENDED) ? \\\n+\t\t\t ondisk_cache_entry_extended_size(ce_namelen(ce)) : \\\n+\t\t\t ondisk_cache_entry_size(ce_namelen(ce)))\n+\n+static int verify_hdr(void *mmap, unsigned long size)\n+{\n+\tgit_SHA_CTX c;\n+\tunsigned char sha1[20];\n+\n+\tif (size < sizeof(struct cache_version_header)\n+\t\t\t+ sizeof(struct cache_header) + 20)\n+\t\tdie(\"index file smaller than expected\");\n+\n+\tgit_SHA1_Init(&c);\n+\tgit_SHA1_Update(&c, mmap, size - 20);\n+\tgit_SHA1_Final(sha1, &c);\n+\tif (hashcmp(sha1, (unsigned char *)mmap + size - 20))\n+\t\treturn error(\"bad index file sha1 signature\");\n+\treturn 0;\n+}\n+\n+static int match_stat_basic(const struct cache_entry *ce,\n+\t\t\t struct stat *st, int changed)\n+{\n+\tchanged |= match_stat_data(&ce->ce_stat_data, st);\n+\n+\t/* Racily smudged entry? */\n+\tif (!ce->ce_stat_data.sd_size) {\n+\t\tif (!is_empty_blob_sha1(ce->sha1))\n+\t\t\tchanged |= DATA_CHANGED;\n+\t}\n+\treturn changed;\n+}\n+\n+static struct cache_entry *cache_entry_from_ondisk(struct ondisk_cache_entry *ondisk,\n+\t\t\t\t\t\t unsigned int flags,\n+\t\t\t\t\t\t const char *name,\n+\t\t\t\t\t\t size_t len)\n+{\n+\tstruct cache_entry *ce = xmalloc(cache_entry_size(len));\n+\n+\tce->ce_stat_data.sd_ctime.sec = ntoh_l(ondisk->ctime.sec);\n+\tce->ce_stat_data.sd_mtime.sec = ntoh_l(ondisk->mtime.sec);\n+\tce->ce_stat_data.sd_ctime.nsec = ntoh_l(ondisk->ctime.nsec);\n+\tce->ce_stat_data.sd_mtime.nsec = ntoh_l(ondisk->mtime.nsec);\n+\tce->ce_stat_data.sd_dev = ntoh_l(ondisk->dev);\n+\tce->ce_stat_data.sd_ino = ntoh_l(ondisk->ino);\n+\tce->ce_mode = ntoh_l(ondisk->mode);\n+\tce->ce_stat_data.sd_uid = ntoh_l(ondisk->uid);\n+\tce->ce_stat_data.sd_gid = ntoh_l(ondisk->gid);\n+\tce->ce_stat_data.sd_size = ntoh_l(ondisk->size);\n+\tce->ce_flags = flags & ~CE_NAMEMASK;\n+\tce->ce_namelen = len;\n+\thashcpy(ce->sha1, ondisk->sha1);\n+\tmemcpy(ce->name, name, len);\n+\tce->name[len] = '\\0';\n+\treturn ce;\n+}\n+\n+/*\n+ * Adjacent cache entries tend to share the leading paths, so it makes\n+ * sense to only store the differences in later entries. In the v4\n+ * on-disk format of the index, each on-disk cache entry stores the\n+ * number of bytes to be stripped from the end of the previous name,\n+ * and the bytes to append to the result, to come up with its name.\n+ */\n+static unsigned long expand_name_field(struct strbuf *name, const char *cp_)\n+{\n+\tconst unsigned char *ep, *cp = (const unsigned char *)cp_;\n+\tsize_t len = decode_varint(&cp);\n+\n+\tif (name->len < len)\n+\t\tdie(\"malformed name field in the index\");\n+\tstrbuf_remove(name, name->len - len, len);\n+\tfor (ep = cp; *ep; ep++)\n+\t\t; /* find the end */\n+\tstrbuf_add(name, cp, ep - cp);\n+\treturn (const char *)ep + 1 - cp_;\n+}\n+\n+static struct cache_entry *create_from_disk(struct ondisk_cache_entry *ondisk,\n+\t\t\t\t\t unsigned long *ent_size,\n+\t\t\t\t\t struct strbuf *previous_name)\n+{\n+\tstruct cache_entry *ce;\n+\tsize_t len;\n+\tconst char *name;\n+\tunsigned int flags;\n+\n+\t/* On-disk flags are just 16 bits */\n+\tflags = ntoh_s(ondisk->flags);\n+\tlen = flags & CE_NAMEMASK;\n+\n+\tif (flags & CE_EXTENDED) {\n+\t\tstruct ondisk_cache_entry_extended *ondisk2;\n+\t\tint extended_flags;\n+\t\tondisk2 = (struct ondisk_cache_entry_extended *)ondisk;\n+\t\textended_flags = ntoh_s(ondisk2->flags2) << 16;\n+\t\t/* We do not yet understand any bit out of CE_EXTENDED_FLAGS */\n+\t\tif (extended_flags & ~CE_EXTENDED_FLAGS)\n+\t\t\tdie(\"Unknown index entry format %08x\", extended_flags);\n+\t\tflags |= extended_flags;\n+\t\tname = ondisk2->name;\n+\t}\n+\telse\n+\t\tname = ondisk->name;\n+\n+\tif (!previous_name) {\n+\t\t/* v3 and earlier */\n+\t\tif (len == CE_NAMEMASK)\n+\t\t\tlen = strlen(name);\n+\t\tce = cache_entry_from_ondisk(ondisk, flags, name, len);\n+\n+\t\t*ent_size = ondisk_ce_size(ce);\n+\t} else {\n+\t\tunsigned long consumed;\n+\t\tconsumed = expand_name_field(previous_name, name);\n+\t\tce = cache_entry_from_ondisk(ondisk, flags,\n+\t\t\t\t\t previous_name->buf,\n+\t\t\t\t\t previous_name->len);\n+\n+\t\t*ent_size = (name - ((char *)ondisk)) + consumed;\n+\t}\n+\treturn ce;\n+}\n+\n+static int read_index_extension(struct index_state *istate,\n+\t\t\t\tconst char *ext, void *data, unsigned long sz)\n+{\n+\tswitch (CACHE_EXT(ext)) {\n+\tcase CACHE_EXT_TREE:\n+\t\tistate->cache_tree = cache_tree_read(data, sz);\n+\t\tbreak;\n+\tcase CACHE_EXT_RESOLVE_UNDO:\n+\t\tistate->resolve_undo = resolve_undo_read(data, sz);\n+\t\tbreak;\n+\tdefault:\n+\t\tif (*ext < 'A' || 'Z' < *ext)\n+\t\t\treturn error(\"index uses %.4s extension, which we do not understand\",\n+\t\t\t\t ext);\n+\t\tfprintf(stderr, \"ignoring %.4s extension\\n\", ext);\n+\t\tbreak;\n+\t}\n+\treturn 0;\n+}\n+\n+static int read_index_v2(struct index_state *istate, void *mmap,\n+\t\t\t unsigned long mmap_size)\n+{\n+\tint i;\n+\tunsigned long src_offset;\n+\tstruct cache_version_header *hdr;\n+\tstruct cache_header *hdr_v2;\n+\tstruct strbuf previous_name_buf = STRBUF_INIT, *previous_name;\n+\n+\thdr = mmap;\n+\thdr_v2 = (struct cache_header *)((char *)mmap + sizeof(*hdr));\n+\tistate->version = ntohl(hdr->hdr_version);\n+\tistate->cache_nr = ntohl(hdr_v2->hdr_entries);\n+\tistate->cache_alloc = alloc_nr(istate->cache_nr);\n+\tistate->cache = xcalloc(istate->cache_alloc, sizeof(struct cache_entry *));\n+\tistate->initialized = 1;\n+\n+\tif (istate->version == 4)\n+\t\tprevious_name = &previous_name_buf;\n+\telse\n+\t\tprevious_name = NULL;\n+\n+\tsrc_offset = sizeof(*hdr) + sizeof(*hdr_v2);\n+\tfor (i = 0; i < istate->cache_nr; i++) {\n+\t\tstruct ondisk_cache_entry *disk_ce;\n+\t\tstruct cache_entry *ce;\n+\t\tunsigned long consumed;\n+\n+\t\tdisk_ce = (struct ondisk_cache_entry *)((char *)mmap + src_offset);\n+\t\tce = create_from_disk(disk_ce, &consumed, previous_name);\n+\t\tset_index_entry(istate, i, ce);\n+\n+\t\tsrc_offset += consumed;\n+\t}\n+\tstrbuf_release(&previous_name_buf);\n+\n+\twhile (src_offset <= mmap_size - 20 - 8) {\n+\t\t/* After an array of active_nr index entries,\n+\t\t * there can be arbitrary number of extended\n+\t\t * sections, each of which is prefixed with\n+\t\t * extension name (4-byte) and section length\n+\t\t * in 4-byte network byte order.\n+\t\t */\n+\t\tuint32_t extsize;\n+\t\tmemcpy(&extsize, (char *)mmap + src_offset + 4, 4);\n+\t\textsize = ntohl(extsize);\n+\t\tif (read_index_extension(istate,\n+\t\t\t\t\t(const char *) mmap + src_offset,\n+\t\t\t\t\t(char *) mmap + src_offset + 8,\n+\t\t\t\t\textsize) < 0)\n+\t\t\tgoto unmap;\n+\t\tsrc_offset += 8;\n+\t\tsrc_offset += extsize;\n+\t}\n+\treturn 0;\n+unmap:\n+\tmunmap(mmap, mmap_size);\n+\tdie(\"index file corrupt\");\n+}\n+\n+#define WRITE_BUFFER_SIZE 8192\n+static unsigned char write_buffer[WRITE_BUFFER_SIZE];\n+static unsigned long write_buffer_len;\n+\n+static int ce_write_flush(git_SHA_CTX *context, int fd)\n+{\n+\tunsigned int buffered = write_buffer_len;\n+\tif (buffered) {\n+\t\tgit_SHA1_Update(context, write_buffer, buffered);\n+\t\tif (write_in_full(fd, write_buffer, buffered) != buffered)\n+\t\t\treturn -1;\n+\t\twrite_buffer_len = 0;\n+\t}\n+\treturn 0;\n+}\n+\n+static int ce_write(git_SHA_CTX *context, int fd, void *data, unsigned int len)\n+{\n+\twhile (len) {\n+\t\tunsigned int buffered = write_buffer_len;\n+\t\tunsigned int partial = WRITE_BUFFER_SIZE - buffered;\n+\t\tif (partial > len)\n+\t\t\tpartial = len;\n+\t\tmemcpy(write_buffer + buffered, data, partial);\n+\t\tbuffered += partial;\n+\t\tif (buffered == WRITE_BUFFER_SIZE) {\n+\t\t\twrite_buffer_len = buffered;\n+\t\t\tif (ce_write_flush(context, fd))\n+\t\t\t\treturn -1;\n+\t\t\tbuffered = 0;\n+\t\t}\n+\t\twrite_buffer_len = buffered;\n+\t\tlen -= partial;\n+\t\tdata = (char *) data + partial;\n+\t}\n+\treturn 0;\n+}\n+\n+static int write_index_ext_header(git_SHA_CTX *context, int fd,\n+\t\t\t\t unsigned int ext, unsigned int sz)\n+{\n+\text = htonl(ext);\n+\tsz = htonl(sz);\n+\treturn ((ce_write(context, fd, &ext, 4) < 0) ||\n+\t\t(ce_write(context, fd, &sz, 4) < 0)) ? -1 : 0;\n+}\n+\n+static int ce_flush(git_SHA_CTX *context, int fd)\n+{\n+\tunsigned int left = write_buffer_len;\n+\n+\tif (left) {\n+\t\twrite_buffer_len = 0;\n+\t\tgit_SHA1_Update(context, write_buffer, left);\n+\t}\n+\n+\t/* Flush first if not enough space for SHA1 signature */\n+\tif (left + 20 > WRITE_BUFFER_SIZE) {\n+\t\tif (write_in_full(fd, write_buffer, left) != left)\n+\t\t\treturn -1;\n+\t\tleft = 0;\n+\t}\n+\n+\t/* Append the SHA1 signature at the end */\n+\tgit_SHA1_Final(write_buffer + left, context);\n+\tleft += 20;\n+\treturn (write_in_full(fd, write_buffer, left) != left) ? -1 : 0;\n+}\n+\n+static void ce_smudge_racily_clean_entry(struct index_state *istate, struct cache_entry *ce)\n+{\n+\t/*\n+\t * The only thing we care about in this function is to smudge the\n+\t * falsely clean entry due to touch-update-touch race, so we leave\n+\t * everything else as they are. We are called for entries whose\n+\t * ce_stat_data.sd_mtime match the index file mtime.\n+\t *\n+\t * Note that this actually does not do much for gitlinks, for\n+\t * which ce_match_stat_basic() always goes to the actual\n+\t * contents. The caller checks with is_racy_timestamp() which\n+\t * always says \"no\" for gitlinks, so we are not called for them ;-)\n+\t */\n+\tstruct stat st;\n+\n+\tif (lstat(ce->name, &st) < 0)\n+\t\treturn;\n+\tif (ce_match_stat_basic(istate, ce, &st))\n+\t\treturn;\n+\tif (ce_modified_check_fs(ce, &st)) {\n+\t\t/* This is \"racily clean\"; smudge it. Note that this\n+\t\t * is a tricky code. At first glance, it may appear\n+\t\t * that it can break with this sequence:\n+\t\t *\n+\t\t * $ echo xyzzy >frotz\n+\t\t * $ git-update-index --add frotz\n+\t\t * $ : >frotz\n+\t\t * $ sleep 3\n+\t\t * $ echo filfre >nitfol\n+\t\t * $ git-update-index --add nitfol\n+\t\t *\n+\t\t * but it does not. When the second update-index runs,\n+\t\t * it notices that the entry \"frotz\" has the same timestamp\n+\t\t * as index, and if we were to smudge it by resetting its\n+\t\t * size to zero here, then the object name recorded\n+\t\t * in index is the 6-byte file but the cached stat information\n+\t\t * becomes zero --- which would then match what we would\n+\t\t * obtain from the filesystem next time we stat(\"frotz\").\n+\t\t *\n+\t\t * However, the second update-index, before calling\n+\t\t * this function, notices that the cached size is 6\n+\t\t * bytes and what is on the filesystem is an empty\n+\t\t * file, and never calls us, so the cached size information\n+\t\t * for \"frotz\" stays 6 which does not match the filesystem.\n+\t\t */\n+\t\tce->ce_stat_data.sd_size = 0;\n+\t}\n+}\n+\n+/* Copy miscellaneous fields but not the name */\n+static char *copy_cache_entry_to_ondisk(struct ondisk_cache_entry *ondisk,\n+\t\t\t\t struct cache_entry *ce)\n+{\n+\tshort flags;\n+\n+\tondisk->ctime.sec = htonl(ce->ce_stat_data.sd_ctime.sec);\n+\tondisk->mtime.sec = htonl(ce->ce_stat_data.sd_mtime.sec);\n+\tondisk->ctime.nsec = htonl(ce->ce_stat_data.sd_ctime.nsec);\n+\tondisk->mtime.nsec = htonl(ce->ce_stat_data.sd_mtime.nsec);\n+\tondisk->dev = htonl(ce->ce_stat_data.sd_dev);\n+\tondisk->ino = htonl(ce->ce_stat_data.sd_ino);\n+\tondisk->mode = htonl(ce->ce_mode);\n+\tondisk->uid = htonl(ce->ce_stat_data.sd_uid);\n+\tondisk->gid = htonl(ce->ce_stat_data.sd_gid);\n+\tondisk->size = htonl(ce->ce_stat_data.sd_size);\n+\thashcpy(ondisk->sha1, ce->sha1);\n+\n+\tflags = ce->ce_flags;\n+\tflags |= (ce_namelen(ce) >= CE_NAMEMASK ? CE_NAMEMASK : ce_namelen(ce));\n+\tondisk->flags = htons(flags);\n+\tif (ce->ce_flags & CE_EXTENDED) {\n+\t\tstruct ondisk_cache_entry_extended *ondisk2;\n+\t\tondisk2 = (struct ondisk_cache_entry_extended *)ondisk;\n+\t\tondisk2->flags2 = htons((ce->ce_flags & CE_EXTENDED_FLAGS) >> 16);\n+\t\treturn ondisk2->name;\n+\t}\n+\telse {\n+\t\treturn ondisk->name;\n+\t}\n+}\n+\n+static int ce_write_entry(git_SHA_CTX *c, int fd, struct cache_entry *ce,\n+\t\t\t struct strbuf *previous_name)\n+{\n+\tint size;\n+\tstruct ondisk_cache_entry *ondisk;\n+\tchar *name;\n+\tint result;\n+\n+\tif (!previous_name) {\n+\t\tsize = ondisk_ce_size(ce);\n+\t\tondisk = xcalloc(1, size);\n+\t\tname = copy_cache_entry_to_ondisk(ondisk, ce);\n+\t\tmemcpy(name, ce->name, ce_namelen(ce));\n+\t} else {\n+\t\tint common, to_remove, prefix_size;\n+\t\tunsigned char to_remove_vi[16];\n+\t\tfor (common = 0;\n+\t\t (ce->name[common] &&\n+\t\t common < previous_name->len &&\n+\t\t ce->name[common] == previous_name->buf[common]);\n+\t\t common++)\n+\t\t\t; /* still matching */\n+\t\tto_remove = previous_name->len - common;\n+\t\tprefix_size = encode_varint(to_remove, to_remove_vi);\n+\n+\t\tif (ce->ce_flags & CE_EXTENDED)\n+\t\t\tsize = offsetof(struct ondisk_cache_entry_extended, name);\n+\t\telse\n+\t\t\tsize = offsetof(struct ondisk_cache_entry, name);\n+\t\tsize += prefix_size + (ce_namelen(ce) - common + 1);\n+\n+\t\tondisk = xcalloc(1, size);\n+\t\tname = copy_cache_entry_to_ondisk(ondisk, ce);\n+\t\tmemcpy(name, to_remove_vi, prefix_size);\n+\t\tmemcpy(name + prefix_size, ce->name + common, ce_namelen(ce) - common);\n+\n+\t\tstrbuf_splice(previous_name, common, to_remove,\n+\t\t\t ce->name + common, ce_namelen(ce) - common);\n+\t}\n+\n+\tresult = ce_write(c, fd, ondisk, size);\n+\tfree(ondisk);\n+\treturn result;\n+}\n+\n+static int write_index_v2(struct index_state *istate, int newfd)\n+{\n+\tgit_SHA_CTX c;\n+\tstruct cache_version_header hdr;\n+\tstruct cache_header hdr_v2;\n+\tint i, err, removed, extended, hdr_version;\n+\tstruct cache_entry **cache = istate->cache;\n+\tint entries = istate->cache_nr;\n+\tstruct stat st;\n+\tstruct strbuf previous_name_buf = STRBUF_INIT, *previous_name;\n+\n+\tfor (i = removed = extended = 0; i < entries; i++) {\n+\t\tif (cache[i]->ce_flags & CE_REMOVE)\n+\t\t\tremoved++;\n+\n+\t\t/* reduce extended entries if possible */\n+\t\tcache[i]->ce_flags &= ~CE_EXTENDED;\n+\t\tif (cache[i]->ce_flags & CE_EXTENDED_FLAGS) {\n+\t\t\textended++;\n+\t\t\tcache[i]->ce_flags |= CE_EXTENDED;\n+\t\t}\n+\t}\n+\n+\tif (!istate->version)\n+\t\tistate->version = INDEX_FORMAT_DEFAULT;\n+\n+\t/* demote version 3 to version 2 when the latter suffices */\n+\tif (istate->version == 3 || istate->version == 2)\n+\t\tistate->version = extended ? 3 : 2;\n+\n+\thdr_version = istate->version;\n+\n+\thdr.hdr_signature = htonl(CACHE_SIGNATURE);\n+\thdr.hdr_version = htonl(hdr_version);\n+\thdr_v2.hdr_entries = htonl(entries - removed);\n+\n+\tgit_SHA1_Init(&c);\n+\tif (ce_write(&c, newfd, &hdr, sizeof(hdr)) < 0)\n+\t\treturn -1;\n+\tif (ce_write(&c, newfd, &hdr_v2, sizeof(hdr_v2)) < 0)\n+\t\treturn -1;\n+\n+\tprevious_name = (hdr_version == 4) ? &previous_name_buf : NULL;\n+\tfor (i = 0; i < entries; i++) {\n+\t\tstruct cache_entry *ce = cache[i];\n+\t\tif (ce->ce_flags & CE_REMOVE)\n+\t\t\tcontinue;\n+\t\tif (!ce_uptodate(ce) && is_racy_timestamp(istate, ce))\n+\t\t\tce_smudge_racily_clean_entry(istate, ce);\n+\t\tif (is_null_sha1(ce->sha1))\n+\t\t\treturn error(\"cache entry has null sha1: %s\", ce->name);\n+\t\tif (ce_write_entry(&c, newfd, ce, previous_name) < 0)\n+\t\t\treturn -1;\n+\t}\n+\tstrbuf_release(&previous_name_buf);\n+\n+\t/* Write extension data here */\n+\tif (istate->cache_tree) {\n+\t\tstruct strbuf sb = STRBUF_INIT;\n+\n+\t\tcache_tree_write(&sb, istate->cache_tree);\n+\t\terr = write_index_ext_header(&c, newfd, CACHE_EXT_TREE, sb.len) < 0\n+\t\t\t|| ce_write(&c, newfd, sb.buf, sb.len) < 0;\n+\t\tstrbuf_release(&sb);\n+\t\tif (err)\n+\t\t\treturn -1;\n+\t}\n+\tif (istate->resolve_undo) {\n+\t\tstruct strbuf sb = STRBUF_INIT;\n+\n+\t\tresolve_undo_write(&sb, istate->resolve_undo);\n+\t\terr = write_index_ext_header(&c, newfd, CACHE_EXT_RESOLVE_UNDO,\n+\t\t\t\t\t sb.len) < 0\n+\t\t\t|| ce_write(&c, newfd, sb.buf, sb.len) < 0;\n+\t\tstrbuf_release(&sb);\n+\t\tif (err)\n+\t\t\treturn -1;\n+\t}\n+\n+\tif (ce_flush(&c, newfd) || fstat(newfd, &st))\n+\t\treturn -1;\n+\tistate->timestamp.sec = (unsigned int)st.st_mtime;\n+\tistate->timestamp.nsec = ST_MTIME_NSEC(st);\n+\treturn 0;\n+}\n+\n+struct index_ops v2_ops = {\n+\tmatch_stat_basic,\n+\tverify_hdr,\n+\tread_index_v2,\n+\twrite_index_v2\n+};\ndiff --git a/read-cache.c b/read-cache.c\nindex 93947bf..1e7ffc2 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -5,6 +5,7 @@\n */\n #define NO_THE_INDEX_COMPATIBILITY_MACROS\n #include \"cache.h\"\n+#include \"read-cache.h\"\n #include \"cache-tree.h\"\n #include \"refs.h\"\n #include \"dir.h\"\n@@ -17,26 +18,9 @@\n \n static struct cache_entry *refresh_cache_entry(struct cache_entry *ce, int really);\n \n-/* Mask for the name length in ce_flags in the on-disk index */\n-\n-#define CE_NAMEMASK (0x0fff)\n-\n-/* Index extensions.\n- *\n- * The first letter should be 'A'..'Z' for extensions that are not\n- * necessary for a correct operation (i.e. optimization data).\n- * When new extensions are added that _needs_ to be understood in\n- * order to correctly interpret the index file, pick character that\n- * is outside the range, to cause the reader to abort.\n- */\n-\n-#define CACHE_EXT(s) ( (s[0]<<24)|(s[1]<<16)|(s[2]<<8)|(s[3]) )\n-#define CACHE_EXT_TREE 0x54524545\t/* \"TREE\" */\n-#define CACHE_EXT_RESOLVE_UNDO 0x52455543 /* \"REUC\" */\n-\n struct index_state the_index;\n \n-static void set_index_entry(struct index_state *istate, int nr, struct cache_entry *ce)\n+void set_index_entry(struct index_state *istate, int nr, struct cache_entry *ce)\n {\n \tistate->cache[nr] = ce;\n \tadd_name_hash(istate, ce);\n@@ -190,7 +174,7 @@ static int ce_compare_gitlink(const struct cache_entry *ce)\n \treturn hashcmp(sha1, ce->sha1);\n }\n \n-static int ce_modified_check_fs(const struct cache_entry *ce, struct stat *st)\n+int ce_modified_check_fs(const struct cache_entry *ce, struct stat *st)\n {\n \tswitch (st->st_mode & S_IFMT) {\n \tcase S_IFREG:\n@@ -210,7 +194,21 @@ static int ce_modified_check_fs(const struct cache_entry *ce, struct stat *st)\n \treturn 0;\n }\n \n-static int ce_match_stat_basic(const struct cache_entry *ce, struct stat *st)\n+/*\n+ * Check if the reading/writing operations are set and set them\n+ * to the correct version\n+ */\n+static void set_istate_ops(struct index_state *istate)\n+{\n+\tif (!istate->version)\n+\t\tistate->version = INDEX_FORMAT_DEFAULT;\n+\n+\tif (istate->version >= 2 && istate->version <= 4)\n+\t\tistate->ops = &v2_ops;\n+}\n+\n+int ce_match_stat_basic(struct index_state *istate,\n+\t\t\tconst struct cache_entry *ce, struct stat *st)\n {\n \tunsigned int changed = 0;\n \n@@ -243,19 +241,14 @@ static int ce_match_stat_basic(const struct cache_entry *ce, struct stat *st)\n \t\tdie(\"internal error: ce_mode is %o\", ce->ce_mode);\n \t}\n \n-\tchanged |= match_stat_data(&ce->ce_stat_data, st);\n-\n-\t/* Racily smudged entry? */\n-\tif (!ce->ce_stat_data.sd_size) {\n-\t\tif (!is_empty_blob_sha1(ce->sha1))\n-\t\t\tchanged |= DATA_CHANGED;\n-\t}\n-\n+\tset_istate_ops(istate);\n+\tchanged = istate->ops->match_stat_basic(ce, st, changed);\n \treturn changed;\n }\n \n-static int is_racy_timestamp(const struct index_state *istate,\n-\t\t\t const struct cache_entry *ce)\n+\n+int is_racy_timestamp(const struct index_state *istate,\n+\t\t const struct cache_entry *ce)\n {\n \treturn (!S_ISGITLINK(ce->ce_mode) &&\n \t\tistate->timestamp.sec &&\n@@ -270,9 +263,8 @@ static int is_racy_timestamp(const struct index_state *istate,\n \t\t );\n }\n \n-int ie_match_stat(const struct index_state *istate,\n-\t\t const struct cache_entry *ce, struct stat *st,\n-\t\t unsigned int options)\n+int ie_match_stat(struct index_state *istate, const struct cache_entry *ce,\n+\t\t struct stat *st, unsigned int options)\n {\n \tunsigned int changed;\n \tint ignore_valid = options & CE_MATCH_IGNORE_VALID;\n@@ -298,7 +290,7 @@ int ie_match_stat(const struct index_state *istate,\n \tif (ce->ce_flags & CE_INTENT_TO_ADD)\n \t\treturn DATA_CHANGED | TYPE_CHANGED | MODE_CHANGED;\n \n-\tchanged = ce_match_stat_basic(ce, st);\n+\tchanged = ce_match_stat_basic(istate, ce, st);\n \n \t/*\n \t * Within 1 second of this sequence:\n@@ -326,8 +318,7 @@ int ie_match_stat(const struct index_state *istate,\n \treturn changed;\n }\n \n-int ie_modified(const struct index_state *istate,\n-\t\tconst struct cache_entry *ce,\n+int ie_modified(struct index_state *istate, const struct cache_entry *ce,\n \t\tstruct stat *st, unsigned int options)\n {\n \tint changed, changed_fs;\n@@ -1211,13 +1202,10 @@ static struct cache_entry *refresh_cache_entry(struct cache_entry *ce, int reall\n \treturn refresh_cache_ent(&the_index, ce, really, NULL, NULL);\n }\n \n-\n /*****************************************************************\n * Index File I/O\n *****************************************************************/\n \n-#define INDEX_FORMAT_DEFAULT 3\n-\n /*\n * dev/ino/uid/gid/size are also just tracked to the low 32 bits\n * Again - this is just a (very strong in practice) heuristic that\n@@ -1268,7 +1256,8 @@ struct ondisk_cache_entry_extended {\n \t\t\t ondisk_cache_entry_extended_size(ce_namelen(ce)) : \\\n \t\t\t ondisk_cache_entry_size(ce_namelen(ce)))\n \n-static int verify_hdr_version(struct cache_version_header *hdr, unsigned long size)\n+static int verify_hdr_version(struct index_state *istate,\n+\t\t\t struct cache_version_header *hdr, unsigned long size)\n {\n \tint hdr_version;\n \n@@ -1277,43 +1266,7 @@ static int verify_hdr_version(struct cache_version_header *hdr, unsigned long si\n \thdr_version = ntohl(hdr->hdr_version);\n \tif (hdr_version < INDEX_FORMAT_LB || INDEX_FORMAT_UB < hdr_version)\n \t\treturn error(\"bad index version %d\", hdr_version);\n-\treturn 0;\n-}\n-\n-static int verify_hdr(void *mmap, unsigned long size)\n-{\n-\tgit_SHA_CTX c;\n-\tunsigned char sha1[20];\n-\n-\tif (size < sizeof(struct cache_version_header)\n-\t + sizeof(struct cache_header) + 20)\n-\t\tdie(\"index file smaller than expected\");\n-\n-\tgit_SHA1_Init(&c);\n-\tgit_SHA1_Update(&c, mmap, size - 20);\n-\tgit_SHA1_Final(sha1, &c);\n-\tif (hashcmp(sha1, (unsigned char *)mmap + size - 20))\n-\t\treturn error(\"bad index file sha1 signature\");\n-\treturn 0;\n-}\n-\n-static int read_index_extension(struct index_state *istate,\n-\t\t\t\tconst char *ext, void *data, unsigned long sz)\n-{\n-\tswitch (CACHE_EXT(ext)) {\n-\tcase CACHE_EXT_TREE:\n-\t\tistate->cache_tree = cache_tree_read(data, sz);\n-\t\tbreak;\n-\tcase CACHE_EXT_RESOLVE_UNDO:\n-\t\tistate->resolve_undo = resolve_undo_read(data, sz);\n-\t\tbreak;\n-\tdefault:\n-\t\tif (*ext < 'A' || 'Z' < *ext)\n-\t\t\treturn error(\"index uses %.4s extension, which we do not understand\",\n-\t\t\t\t ext);\n-\t\tfprintf(stderr, \"ignoring %.4s extension\\n\", ext);\n-\t\tbreak;\n-\t}\n+\tistate->ops = &v2_ops;\n \treturn 0;\n }\n \n@@ -1322,178 +1275,6 @@ int read_index(struct index_state *istate)\n \treturn read_index_from(istate, get_index_file());\n }\n \n-#ifndef NEEDS_ALIGNED_ACCESS\n-#define ntoh_s(var) ntohs(var)\n-#define ntoh_l(var) ntohl(var)\n-#else\n-static inline uint16_t ntoh_s_force_align(void *p)\n-{\n-\tuint16_t x;\n-\tmemcpy(&x, p, sizeof(x));\n-\treturn ntohs(x);\n-}\n-static inline uint32_t ntoh_l_force_align(void *p)\n-{\n-\tuint32_t x;\n-\tmemcpy(&x, p, sizeof(x));\n-\treturn ntohl(x);\n-}\n-#define ntoh_s(var) ntoh_s_force_align(&(var))\n-#define ntoh_l(var) ntoh_l_force_align(&(var))\n-#endif\n-\n-static struct cache_entry *cache_entry_from_ondisk(struct ondisk_cache_entry *ondisk,\n-\t\t\t\t\t\t unsigned int flags,\n-\t\t\t\t\t\t const char *name,\n-\t\t\t\t\t\t size_t len)\n-{\n-\tstruct cache_entry *ce = xmalloc(cache_entry_size(len));\n-\n-\tce->ce_stat_data.sd_ctime.sec = ntoh_l(ondisk->ctime.sec);\n-\tce->ce_stat_data.sd_mtime.sec = ntoh_l(ondisk->mtime.sec);\n-\tce->ce_stat_data.sd_ctime.nsec = ntoh_l(ondisk->ctime.nsec);\n-\tce->ce_stat_data.sd_mtime.nsec = ntoh_l(ondisk->mtime.nsec);\n-\tce->ce_stat_data.sd_dev = ntoh_l(ondisk->dev);\n-\tce->ce_stat_data.sd_ino = ntoh_l(ondisk->ino);\n-\tce->ce_mode = ntoh_l(ondisk->mode);\n-\tce->ce_stat_data.sd_uid = ntoh_l(ondisk->uid);\n-\tce->ce_stat_data.sd_gid = ntoh_l(ondisk->gid);\n-\tce->ce_stat_data.sd_size = ntoh_l(ondisk->size);\n-\tce->ce_flags = flags & ~CE_NAMEMASK;\n-\tce->ce_namelen = len;\n-\thashcpy(ce->sha1, ondisk->sha1);\n-\tmemcpy(ce->name, name, len);\n-\tce->name[len] = '\\0';\n-\treturn ce;\n-}\n-\n-/*\n- * Adjacent cache entries tend to share the leading paths, so it makes\n- * sense to only store the differences in later entries. In the v4\n- * on-disk format of the index, each on-disk cache entry stores the\n- * number of bytes to be stripped from the end of the previous name,\n- * and the bytes to append to the result, to come up with its name.\n- */\n-static unsigned long expand_name_field(struct strbuf *name, const char *cp_)\n-{\n-\tconst unsigned char *ep, *cp = (const unsigned char *)cp_;\n-\tsize_t len = decode_varint(&cp);\n-\n-\tif (name->len < len)\n-\t\tdie(\"malformed name field in the index\");\n-\tstrbuf_remove(name, name->len - len, len);\n-\tfor (ep = cp; *ep; ep++)\n-\t\t; /* find the end */\n-\tstrbuf_add(name, cp, ep - cp);\n-\treturn (const char *)ep + 1 - cp_;\n-}\n-\n-static struct cache_entry *create_from_disk(struct ondisk_cache_entry *ondisk,\n-\t\t\t\t\t unsigned long *ent_size,\n-\t\t\t\t\t struct strbuf *previous_name)\n-{\n-\tstruct cache_entry *ce;\n-\tsize_t len;\n-\tconst char *name;\n-\tunsigned int flags;\n-\n-\t/* On-disk flags are just 16 bits */\n-\tflags = ntoh_s(ondisk->flags);\n-\tlen = flags & CE_NAMEMASK;\n-\n-\tif (flags & CE_EXTENDED) {\n-\t\tstruct ondisk_cache_entry_extended *ondisk2;\n-\t\tint extended_flags;\n-\t\tondisk2 = (struct ondisk_cache_entry_extended *)ondisk;\n-\t\textended_flags = ntoh_s(ondisk2->flags2) << 16;\n-\t\t/* We do not yet understand any bit out of CE_EXTENDED_FLAGS */\n-\t\tif (extended_flags & ~CE_EXTENDED_FLAGS)\n-\t\t\tdie(\"Unknown index entry format %08x\", extended_flags);\n-\t\tflags |= extended_flags;\n-\t\tname = ondisk2->name;\n-\t}\n-\telse\n-\t\tname = ondisk->name;\n-\n-\tif (!previous_name) {\n-\t\t/* v3 and earlier */\n-\t\tif (len == CE_NAMEMASK)\n-\t\t\tlen = strlen(name);\n-\t\tce = cache_entry_from_ondisk(ondisk, flags, name, len);\n-\n-\t\t*ent_size = ondisk_ce_size(ce);\n-\t} else {\n-\t\tunsigned long consumed;\n-\t\tconsumed = expand_name_field(previous_name, name);\n-\t\tce = cache_entry_from_ondisk(ondisk, flags,\n-\t\t\t\t\t previous_name->buf,\n-\t\t\t\t\t previous_name->len);\n-\n-\t\t*ent_size = (name - ((char *)ondisk)) + consumed;\n-\t}\n-\treturn ce;\n-}\n-\n-static int read_index_v2(struct index_state *istate, void *mmap, unsigned long mmap_size)\n-{\n-\tint i;\n-\tunsigned long src_offset;\n-\tstruct cache_version_header *hdr;\n-\tstruct cache_header *hdr_v2;\n-\tstruct strbuf previous_name_buf = STRBUF_INIT, *previous_name;\n-\n-\thdr = mmap;\n-\thdr_v2 = (struct cache_header *)((char *)mmap + sizeof(*hdr));\n-\n-\tistate->version = ntohl(hdr->hdr_version);\n-\tistate->cache_nr = ntohl(hdr_v2->hdr_entries);\n-\tistate->cache_alloc = alloc_nr(istate->cache_nr);\n-\tistate->cache = xcalloc(istate->cache_alloc, sizeof(*istate->cache));\n-\tistate->initialized = 1;\n-\n-\tif (istate->version == 4)\n-\t\tprevious_name = &previous_name_buf;\n-\telse\n-\t\tprevious_name = NULL;\n-\n-\tsrc_offset = sizeof(*hdr) + sizeof(*hdr_v2);\n-\tfor (i = 0; i < istate->cache_nr; i++) {\n-\t\tstruct ondisk_cache_entry *disk_ce;\n-\t\tstruct cache_entry *ce;\n-\t\tunsigned long consumed;\n-\n-\t\tdisk_ce = (struct ondisk_cache_entry *)((char *)mmap + src_offset);\n-\t\tce = create_from_disk(disk_ce, &consumed, previous_name);\n-\t\tset_index_entry(istate, i, ce);\n-\n-\t\tsrc_offset += consumed;\n-\t}\n-\tstrbuf_release(&previous_name_buf);\n-\n-\twhile (src_offset <= mmap_size - 20 - 8) {\n-\t\t/* After an array of active_nr index entries,\n-\t\t * there can be arbitrary number of extended\n-\t\t * sections, each of which is prefixed with\n-\t\t * extension name (4-byte) and section length\n-\t\t * in 4-byte network byte order.\n-\t\t */\n-\t\tuint32_t extsize;\n-\t\tmemcpy(&extsize, (char *)mmap + src_offset + 4, 4);\n-\t\textsize = ntohl(extsize);\n-\t\tif (read_index_extension(istate,\n-\t\t\t\t\t (const char *) mmap + src_offset,\n-\t\t\t\t\t (char *) mmap + src_offset + 8,\n-\t\t\t\t\t extsize) < 0)\n-\t\t\tgoto unmap;\n-\t\tsrc_offset += 8;\n-\t\tsrc_offset += extsize;\n-\t}\n-\treturn 0;\n-unmap:\n-\tmunmap(mmap, mmap_size);\n-\tdie(\"index file corrupt\");\n-}\n-\n /* remember to discard_cache() before reading a different cache! */\n int read_index_from(struct index_state *istate, const char *path)\n {\n@@ -1510,6 +1291,7 @@ int read_index_from(struct index_state *istate, const char *path)\n \terrno = ENOENT;\n \tistate->timestamp.sec = 0;\n \tistate->timestamp.nsec = 0;\n+\n \tfd = open(path, O_RDONLY);\n \tif (fd < 0) {\n \t\tif (errno == ENOENT)\n@@ -1522,24 +1304,23 @@ int read_index_from(struct index_state *istate, const char *path)\n \n \terrno = EINVAL;\n \tmmap_size = xsize_t(st.st_size);\n-\tif (mmap_size < sizeof(struct cache_header) + 20)\n-\t\tdie(\"index file smaller than expected\");\n-\n \tmmap = xmmap(NULL, mmap_size, PROT_READ | PROT_WRITE, MAP_PRIVATE, fd, 0);\n \tclose(fd);\n \tif (mmap == MAP_FAILED)\n \t\tdie_errno(\"unable to map index file\");\n \n \thdr = mmap;\n-\tif (verify_hdr_version(hdr, mmap_size) < 0)\n+\tif (verify_hdr_version(istate, hdr, mmap_size) < 0)\n \t\tgoto unmap;\n \n-\tif (verify_hdr(mmap, mmap_size) < 0)\n+\tif (istate->ops->verify_hdr(mmap, mmap_size) < 0)\n \t\tgoto unmap;\n \n-\tread_index_v2(istate, mmap, mmap_size);\n+\tif (istate->ops->read_index(istate, mmap, mmap_size) < 0)\n+\t\tgoto unmap;\n \tistate->timestamp.sec = st.st_mtime;\n \tistate->timestamp.nsec = ST_MTIME_NSEC(st);\n+\n \tmunmap(mmap, mmap_size);\n \treturn istate->cache_nr;\n \n@@ -1583,201 +1364,6 @@ int unmerged_index(const struct index_state *istate)\n \treturn 0;\n }\n \n-#define WRITE_BUFFER_SIZE 8192\n-static unsigned char write_buffer[WRITE_BUFFER_SIZE];\n-static unsigned long write_buffer_len;\n-\n-static int ce_write_flush(git_SHA_CTX *context, int fd)\n-{\n-\tunsigned int buffered = write_buffer_len;\n-\tif (buffered) {\n-\t\tgit_SHA1_Update(context, write_buffer, buffered);\n-\t\tif (write_in_full(fd, write_buffer, buffered) != buffered)\n-\t\t\treturn -1;\n-\t\twrite_buffer_len = 0;\n-\t}\n-\treturn 0;\n-}\n-\n-static int ce_write(git_SHA_CTX *context, int fd, void *data, unsigned int len)\n-{\n-\twhile (len) {\n-\t\tunsigned int buffered = write_buffer_len;\n-\t\tunsigned int partial = WRITE_BUFFER_SIZE - buffered;\n-\t\tif (partial > len)\n-\t\t\tpartial = len;\n-\t\tmemcpy(write_buffer + buffered, data, partial);\n-\t\tbuffered += partial;\n-\t\tif (buffered == WRITE_BUFFER_SIZE) {\n-\t\t\twrite_buffer_len = buffered;\n-\t\t\tif (ce_write_flush(context, fd))\n-\t\t\t\treturn -1;\n-\t\t\tbuffered = 0;\n-\t\t}\n-\t\twrite_buffer_len = buffered;\n-\t\tlen -= partial;\n-\t\tdata = (char *) data + partial;\n-\t}\n-\treturn 0;\n-}\n-\n-static int write_index_ext_header(git_SHA_CTX *context, int fd,\n-\t\t\t\t unsigned int ext, unsigned int sz)\n-{\n-\text = htonl(ext);\n-\tsz = htonl(sz);\n-\treturn ((ce_write(context, fd, &ext, 4) < 0) ||\n-\t\t(ce_write(context, fd, &sz, 4) < 0)) ? -1 : 0;\n-}\n-\n-static int ce_flush(git_SHA_CTX *context, int fd)\n-{\n-\tunsigned int left = write_buffer_len;\n-\n-\tif (left) {\n-\t\twrite_buffer_len = 0;\n-\t\tgit_SHA1_Update(context, write_buffer, left);\n-\t}\n-\n-\t/* Flush first if not enough space for SHA1 signature */\n-\tif (left + 20 > WRITE_BUFFER_SIZE) {\n-\t\tif (write_in_full(fd, write_buffer, left) != left)\n-\t\t\treturn -1;\n-\t\tleft = 0;\n-\t}\n-\n-\t/* Append the SHA1 signature at the end */\n-\tgit_SHA1_Final(write_buffer + left, context);\n-\tleft += 20;\n-\treturn (write_in_full(fd, write_buffer, left) != left) ? -1 : 0;\n-}\n-\n-static void ce_smudge_racily_clean_entry(struct cache_entry *ce)\n-{\n-\t/*\n-\t * The only thing we care about in this function is to smudge the\n-\t * falsely clean entry due to touch-update-touch race, so we leave\n-\t * everything else as they are. We are called for entries whose\n-\t * ce_stat_data.sd_mtime match the index file mtime.\n-\t *\n-\t * Note that this actually does not do much for gitlinks, for\n-\t * which ce_match_stat_basic() always goes to the actual\n-\t * contents. The caller checks with is_racy_timestamp() which\n-\t * always says \"no\" for gitlinks, so we are not called for them ;-)\n-\t */\n-\tstruct stat st;\n-\n-\tif (lstat(ce->name, &st) < 0)\n-\t\treturn;\n-\tif (ce_match_stat_basic(ce, &st))\n-\t\treturn;\n-\tif (ce_modified_check_fs(ce, &st)) {\n-\t\t/* This is \"racily clean\"; smudge it. Note that this\n-\t\t * is a tricky code. At first glance, it may appear\n-\t\t * that it can break with this sequence:\n-\t\t *\n-\t\t * $ echo xyzzy >frotz\n-\t\t * $ git-update-index --add frotz\n-\t\t * $ : >frotz\n-\t\t * $ sleep 3\n-\t\t * $ echo filfre >nitfol\n-\t\t * $ git-update-index --add nitfol\n-\t\t *\n-\t\t * but it does not. When the second update-index runs,\n-\t\t * it notices that the entry \"frotz\" has the same timestamp\n-\t\t * as index, and if we were to smudge it by resetting its\n-\t\t * size to zero here, then the object name recorded\n-\t\t * in index is the 6-byte file but the cached stat information\n-\t\t * becomes zero --- which would then match what we would\n-\t\t * obtain from the filesystem next time we stat(\"frotz\").\n-\t\t *\n-\t\t * However, the second update-index, before calling\n-\t\t * this function, notices that the cached size is 6\n-\t\t * bytes and what is on the filesystem is an empty\n-\t\t * file, and never calls us, so the cached size information\n-\t\t * for \"frotz\" stays 6 which does not match the filesystem.\n-\t\t */\n-\t\tce->ce_stat_data.sd_size = 0;\n-\t}\n-}\n-\n-/* Copy miscellaneous fields but not the name */\n-static char *copy_cache_entry_to_ondisk(struct ondisk_cache_entry *ondisk,\n-\t\t\t\t struct cache_entry *ce)\n-{\n-\tshort flags;\n-\n-\tondisk->ctime.sec = htonl(ce->ce_stat_data.sd_ctime.sec);\n-\tondisk->mtime.sec = htonl(ce->ce_stat_data.sd_mtime.sec);\n-\tondisk->ctime.nsec = htonl(ce->ce_stat_data.sd_ctime.nsec);\n-\tondisk->mtime.nsec = htonl(ce->ce_stat_data.sd_mtime.nsec);\n-\tondisk->dev = htonl(ce->ce_stat_data.sd_dev);\n-\tondisk->ino = htonl(ce->ce_stat_data.sd_ino);\n-\tondisk->mode = htonl(ce->ce_mode);\n-\tondisk->uid = htonl(ce->ce_stat_data.sd_uid);\n-\tondisk->gid = htonl(ce->ce_stat_data.sd_gid);\n-\tondisk->size = htonl(ce->ce_stat_data.sd_size);\n-\thashcpy(ondisk->sha1, ce->sha1);\n-\n-\tflags = ce->ce_flags;\n-\tflags |= (ce_namelen(ce) >= CE_NAMEMASK ? CE_NAMEMASK : ce_namelen(ce));\n-\tondisk->flags = htons(flags);\n-\tif (ce->ce_flags & CE_EXTENDED) {\n-\t\tstruct ondisk_cache_entry_extended *ondisk2;\n-\t\tondisk2 = (struct ondisk_cache_entry_extended *)ondisk;\n-\t\tondisk2->flags2 = htons((ce->ce_flags & CE_EXTENDED_FLAGS) >> 16);\n-\t\treturn ondisk2->name;\n-\t}\n-\telse {\n-\t\treturn ondisk->name;\n-\t}\n-}\n-\n-static int ce_write_entry(git_SHA_CTX *c, int fd, struct cache_entry *ce,\n-\t\t\t struct strbuf *previous_name)\n-{\n-\tint size;\n-\tstruct ondisk_cache_entry *ondisk;\n-\tchar *name;\n-\tint result;\n-\n-\tif (!previous_name) {\n-\t\tsize = ondisk_ce_size(ce);\n-\t\tondisk = xcalloc(1, size);\n-\t\tname = copy_cache_entry_to_ondisk(ondisk, ce);\n-\t\tmemcpy(name, ce->name, ce_namelen(ce));\n-\t} else {\n-\t\tint common, to_remove, prefix_size;\n-\t\tunsigned char to_remove_vi[16];\n-\t\tfor (common = 0;\n-\t\t (ce->name[common] &&\n-\t\t common < previous_name->len &&\n-\t\t ce->name[common] == previous_name->buf[common]);\n-\t\t common++)\n-\t\t\t; /* still matching */\n-\t\tto_remove = previous_name->len - common;\n-\t\tprefix_size = encode_varint(to_remove, to_remove_vi);\n-\n-\t\tif (ce->ce_flags & CE_EXTENDED)\n-\t\t\tsize = offsetof(struct ondisk_cache_entry_extended, name);\n-\t\telse\n-\t\t\tsize = offsetof(struct ondisk_cache_entry, name);\n-\t\tsize += prefix_size + (ce_namelen(ce) - common + 1);\n-\n-\t\tondisk = xcalloc(1, size);\n-\t\tname = copy_cache_entry_to_ondisk(ondisk, ce);\n-\t\tmemcpy(name, to_remove_vi, prefix_size);\n-\t\tmemcpy(name + prefix_size, ce->name + common, ce_namelen(ce) - common);\n-\n-\t\tstrbuf_splice(previous_name, common, to_remove,\n-\t\t\t ce->name + common, ce_namelen(ce) - common);\n-\t}\n-\n-\tresult = ce_write(c, fd, ondisk, size);\n-\tfree(ondisk);\n-\treturn result;\n-}\n-\n static int has_racy_timestamp(struct index_state *istate)\n {\n \tint entries = istate->cache_nr;\n@@ -1803,95 +1389,10 @@ void update_index_if_able(struct index_state *istate, struct lock_file *lockfile\n \t\trollback_lock_file(lockfile);\n }\n \n-static int write_index_v2(struct index_state *istate, int newfd)\n-{\n-\tgit_SHA_CTX c;\n-\tstruct cache_version_header hdr;\n-\tstruct cache_header hdr_v2;\n-\tint i, err, removed, extended, hdr_version;\n-\tstruct cache_entry **cache = istate->cache;\n-\tint entries = istate->cache_nr;\n-\tstruct stat st;\n-\tstruct strbuf previous_name_buf = STRBUF_INIT, *previous_name;\n-\n-\tfor (i = removed = extended = 0; i < entries; i++) {\n-\t\tif (cache[i]->ce_flags & CE_REMOVE)\n-\t\t\tremoved++;\n-\n-\t\t/* reduce extended entries if possible */\n-\t\tcache[i]->ce_flags &= ~CE_EXTENDED;\n-\t\tif (cache[i]->ce_flags & CE_EXTENDED_FLAGS) {\n-\t\t\textended++;\n-\t\t\tcache[i]->ce_flags |= CE_EXTENDED;\n-\t\t}\n-\t}\n-\n-\tif (!istate->version)\n-\t\tistate->version = INDEX_FORMAT_DEFAULT;\n-\n-\t/* demote version 3 to version 2 when the latter suffices */\n-\tif (istate->version == 3 || istate->version == 2)\n-\t\tistate->version = extended ? 3 : 2;\n-\n-\thdr_version = istate->version;\n-\n-\thdr.hdr_signature = htonl(CACHE_SIGNATURE);\n-\thdr.hdr_version = htonl(hdr_version);\n-\thdr_v2.hdr_entries = htonl(entries - removed);\n-\n-\tgit_SHA1_Init(&c);\n-\tif (ce_write(&c, newfd, &hdr, sizeof(hdr)) < 0)\n-\t\treturn -1;\n-\tif (ce_write(&c, newfd, &hdr_v2, sizeof(hdr_v2)) < 0)\n-\t\treturn -1;\n-\n-\tprevious_name = (hdr_version == 4) ? &previous_name_buf : NULL;\n-\tfor (i = 0; i < entries; i++) {\n-\t\tstruct cache_entry *ce = cache[i];\n-\t\tif (ce->ce_flags & CE_REMOVE)\n-\t\t\tcontinue;\n-\t\tif (!ce_uptodate(ce) && is_racy_timestamp(istate, ce))\n-\t\t\tce_smudge_racily_clean_entry(ce);\n-\t\tif (is_null_sha1(ce->sha1))\n-\t\t\treturn error(\"cache entry has null sha1: %s\", ce->name);\n-\t\tif (ce_write_entry(&c, newfd, ce, previous_name) < 0)\n-\t\t\treturn -1;\n-\t}\n-\tstrbuf_release(&previous_name_buf);\n-\n-\t/* Write extension data here */\n-\tif (istate->cache_tree) {\n-\t\tstruct strbuf sb = STRBUF_INIT;\n-\n-\t\tcache_tree_write(&sb, istate->cache_tree);\n-\t\terr = write_index_ext_header(&c, newfd, CACHE_EXT_TREE, sb.len) < 0\n-\t\t\t|| ce_write(&c, newfd, sb.buf, sb.len) < 0;\n-\t\tstrbuf_release(&sb);\n-\t\tif (err)\n-\t\t\treturn -1;\n-\t}\n-\tif (istate->resolve_undo) {\n-\t\tstruct strbuf sb = STRBUF_INIT;\n-\n-\t\tresolve_undo_write(&sb, istate->resolve_undo);\n-\t\terr = write_index_ext_header(&c, newfd, CACHE_EXT_RESOLVE_UNDO,\n-\t\t\t\t\t sb.len) < 0\n-\t\t\t|| ce_write(&c, newfd, sb.buf, sb.len) < 0;\n-\t\tstrbuf_release(&sb);\n-\t\tif (err)\n-\t\t\treturn -1;\n-\t}\n-\n-\tif (ce_flush(&c, newfd) || fstat(newfd, &st))\n-\t\treturn -1;\n-\tistate->timestamp.sec = (unsigned int)st.st_mtime;\n-\tistate->timestamp.nsec = ST_MTIME_NSEC(st);\n-\treturn 0;\n-}\n-\n int write_index(struct index_state *istate, int newfd)\n {\n-\treturn write_index_v2(istate, newfd);\n+\tset_istate_ops(istate);\n+\treturn istate->ops->write_index(istate, newfd);\n }\n \n /*\ndiff --git a/read-cache.h b/read-cache.h\nnew file mode 100644\nindex 0000000..f31a38b\n--- /dev/null\n+++ b/read-cache.h\n@@ -0,0 +1,57 @@\n+/* Index extensions.\n+ *\n+ * The first letter should be 'A'..'Z' for extensions that are not\n+ * necessary for a correct operation (i.e. optimization data).\n+ * When new extensions are added that _needs_ to be understood in\n+ * order to correctly interpret the index file, pick character that\n+ * is outside the range, to cause the reader to abort.\n+ */\n+\n+#define CACHE_EXT(s) ( (s[0]<<24)|(s[1]<<16)|(s[2]<<8)|(s[3]) )\n+#define CACHE_EXT_TREE 0x54524545\t/* \"TREE\" */\n+#define CACHE_EXT_RESOLVE_UNDO 0x52455543 /* \"REUC\" */\n+\n+#define INDEX_FORMAT_DEFAULT 3\n+\n+/*\n+ * Basic data structures for the directory cache\n+ */\n+struct cache_version_header {\n+\tunsigned int hdr_signature;\n+\tunsigned int hdr_version;\n+};\n+\n+struct index_ops {\n+\tint (*match_stat_basic)(const struct cache_entry *ce, struct stat *st, int changed);\n+\tint (*verify_hdr)(void *mmap, unsigned long size);\n+\tint (*read_index)(struct index_state *istate, void *mmap, unsigned long mmap_size);\n+\tint (*write_index)(struct index_state *istate, int newfd);\n+};\n+\n+extern struct index_ops v2_ops;\n+\n+#ifndef NEEDS_ALIGNED_ACCESS\n+#define ntoh_s(var) ntohs(var)\n+#define ntoh_l(var) ntohl(var)\n+#else\n+static inline uint16_t ntoh_s_force_align(void *p)\n+{\n+\tuint16_t x;\n+\tmemcpy(&x, p, sizeof(x));\n+\treturn ntohs(x);\n+}\n+static inline uint32_t ntoh_l_force_align(void *p)\n+{\n+\tuint32_t x;\n+\tmemcpy(&x, p, sizeof(x));\n+\treturn ntohl(x);\n+}\n+#define ntoh_s(var) ntoh_s_force_align(&(var))\n+#define ntoh_l(var) ntoh_l_force_align(&(var))\n+#endif\n+\n+extern int ce_modified_check_fs(const struct cache_entry *ce, struct stat *st);\n+extern int ce_match_stat_basic(struct index_state *istate, const struct cache_entry *ce,\n+\t\t\t struct stat *st);\n+extern int is_racy_timestamp(const struct index_state *istate, const struct cache_entry *ce);\n+extern void set_index_entry(struct index_state *istate, int nr, struct cache_entry *ce);\ndiff --git a/test-index-version.c b/test-index-version.c\nindex 4c0386f..65545a7 100644\n--- a/test-index-version.c\n+++ b/test-index-version.c\n@@ -1,5 +1,10 @@\n #include \"cache.h\"\n \n+struct cache_version_header {\n+\tunsigned int hdr_signature;\n+\tunsigned int hdr_version;\n+};\n+\n int main(int argc, char **argv)\n {\n \tstruct cache_version_header hdr;\n-- \n1.8.3.453.g1dfc63d\n"},{"id":"223178","messageId":"1373650024-3001-5-git-send-email-t.gummerer@gmail.com","threadId":"34419","inReplyTo":"1373650024-3001-1-git-send-email-t.gummerer@gmail.com","subject":"[PATCH v2 04/19] read-cache: Re-read index if index file changed","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2013-07-12T17:26:49Z","receivedAt":"2013-07-12T17:26:49Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"Add the possibility of re-reading the index file, if it changed\nwhile reading.\n\nThe index file might change during the read, causing outdated\ninformation to be displayed. We check if the index file changed\nby using its stat data as heuristic.\n\nHelped-by: Ramsay Jones \nSigned-off-by: Thomas Gummerer \n---\n read-cache.c | 91 +++++++++++++++++++++++++++++++++++++-----------------------\n 1 file changed, 57 insertions(+), 34 deletions(-)\n\ndiff --git a/read-cache.c b/read-cache.c\nindex 1e7ffc2..3e3a0e2 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -1275,11 +1275,31 @@ int read_index(struct index_state *istate)\n \treturn read_index_from(istate, get_index_file());\n }\n \n+static int index_changed(struct stat *st_old, struct stat *st_new)\n+{\n+\tif (st_old->st_mtime != st_new->st_mtime ||\n+#if !defined (__CYGWIN__)\n+\t st_old->st_uid != st_new->st_uid ||\n+\t st_old->st_gid != st_new->st_gid ||\n+\t st_old->st_ino != st_new->st_ino ||\n+#endif\n+#if USE_NSEC\n+\t ST_MTIME_NSEC(*st_old) != ST_MTIME_NSEC(*st_new) ||\n+#endif\n+#if USE_STDEV\n+\t st_old->st_dev != st_new->st_dev ||\n+#endif\n+\t st_old->st_size != st_new->st_size)\n+\t\treturn 1;\n+\n+\treturn 0;\n+}\n+\n /* remember to discard_cache() before reading a different cache! */\n int read_index_from(struct index_state *istate, const char *path)\n {\n-\tint fd;\n-\tstruct stat st;\n+\tint fd, err, i;\n+\tstruct stat st_old, st_new;\n \tstruct cache_version_header *hdr;\n \tvoid *mmap;\n \tsize_t mmap_size;\n@@ -1291,41 +1311,44 @@ int read_index_from(struct index_state *istate, const char *path)\n \terrno = ENOENT;\n \tistate->timestamp.sec = 0;\n \tistate->timestamp.nsec = 0;\n+\tfor (i = 0; i < 50; i++) {\n+\t\terr = 0;\n+\t\tfd = open(path, O_RDONLY);\n+\t\tif (fd < 0) {\n+\t\t\tif (errno == ENOENT)\n+\t\t\t\treturn 0;\n+\t\t\tdie_errno(\"index file open failed\");\n+\t\t}\n \n-\tfd = open(path, O_RDONLY);\n-\tif (fd < 0) {\n-\t\tif (errno == ENOENT)\n-\t\t\treturn 0;\n-\t\tdie_errno(\"index file open failed\");\n+\t\tif (fstat(fd, &st_old))\n+\t\t\tdie_errno(\"cannot stat the open index\");\n+\n+\t\terrno = EINVAL;\n+\t\tmmap_size = xsize_t(st_old.st_size);\n+\t\tmmap = xmmap(NULL, mmap_size, PROT_READ | PROT_WRITE, MAP_PRIVATE, fd, 0);\n+\t\tclose(fd);\n+\t\tif (mmap == MAP_FAILED)\n+\t\t\tdie_errno(\"unable to map index file\");\n+\n+\t\thdr = mmap;\n+\t\tif (verify_hdr_version(istate, hdr, mmap_size) < 0)\n+\t\t\terr = 1;\n+\n+\t\tif (istate->ops->verify_hdr(mmap, mmap_size) < 0)\n+\t\t\terr = 1;\n+\n+\t\tif (istate->ops->read_index(istate, mmap, mmap_size) < 0)\n+\t\t\terr = 1;\n+\t\tistate->timestamp.sec = st_old.st_mtime;\n+\t\tistate->timestamp.nsec = ST_MTIME_NSEC(st_old);\n+\t\tif (lstat(path, &st_new))\n+\t\t\tdie_errno(\"cannot stat the open index\");\n+\n+\t\tmunmap(mmap, mmap_size);\n+\t\tif (!index_changed(&st_old, &st_new) && !err)\n+\t\t\treturn istate->cache_nr;\n \t}\n \n-\tif (fstat(fd, &st))\n-\t\tdie_errno(\"cannot stat the open index\");\n-\n-\terrno = EINVAL;\n-\tmmap_size = xsize_t(st.st_size);\n-\tmmap = xmmap(NULL, mmap_size, PROT_READ | PROT_WRITE, MAP_PRIVATE, fd, 0);\n-\tclose(fd);\n-\tif (mmap == MAP_FAILED)\n-\t\tdie_errno(\"unable to map index file\");\n-\n-\thdr = mmap;\n-\tif (verify_hdr_version(istate, hdr, mmap_size) < 0)\n-\t\tgoto unmap;\n-\n-\tif (istate->ops->verify_hdr(mmap, mmap_size) < 0)\n-\t\tgoto unmap;\n-\n-\tif (istate->ops->read_index(istate, mmap, mmap_size) < 0)\n-\t\tgoto unmap;\n-\tistate->timestamp.sec = st.st_mtime;\n-\tistate->timestamp.nsec = ST_MTIME_NSEC(st);\n-\n-\tmunmap(mmap, mmap_size);\n-\treturn istate->cache_nr;\n-\n-unmap:\n-\tmunmap(mmap, mmap_size);\n \tdie(\"index file corrupt\");\n }\n \n-- \n1.8.3.453.g1dfc63d\n"},{"id":"223179","messageId":"1373650024-3001-6-git-send-email-t.gummerer@gmail.com","threadId":"34419","inReplyTo":"1373650024-3001-1-git-send-email-t.gummerer@gmail.com","subject":"[PATCH v2 05/19] Add documentation for the index api","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2013-07-12T17:26:50Z","receivedAt":"2013-07-12T17:26:50Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"Add documentation for the index reading api. This also includes\ndocumentation for the new api functions introduced in the next patch.\n\nHelped-by: Nguyễn Thái Ngọc Duy \nSigned-off-by: Thomas Gummerer \n---\n Documentation/technical/api-in-core-index.txt | 54 +++++++++++++++++++++++++--\n 1 file changed, 50 insertions(+), 4 deletions(-)\n\ndiff --git a/Documentation/technical/api-in-core-index.txt b/Documentation/technical/api-in-core-index.txt\nindex adbdbf5..9b8c37c 100644\n--- a/Documentation/technical/api-in-core-index.txt\n+++ b/Documentation/technical/api-in-core-index.txt\n@@ -1,14 +1,60 @@\n in-core index API\n =================\n \n+Reading API\n+-----------\n+\n+`cache`::\n+\n+\tAn array of cache entries. This is used to access the cache\n+\tentries directly. Use `index_name_pos` to search for the\n+\tindex of a specific cache entry.\n+\n+`read_index_filtered`::\n+\n+\tRead a part of the index, filtered by the pathspec given in\n+\tthe opts. The function may load more than necessary, so the\n+\tcaller still responsible to apply filters appropriately. The\n+\tfiltering is only done for performance reasons, as it's\n+\tpossible to only read part of the index when the on-disk\n+\tformat is index-v5.\n+\n+\tTo iterate only over the entries that match the pathspec, use\n+\tthe for_each_index_entry function.\n+\n+`read_index`::\n+\n+\tRead the whole index file from disk.\n+\n+`index_name_pos`::\n+\n+\tFind a cache_entry with name in the index. Returns pos if an\n+\tentry is matched exactly and -1-pos if an entry is matched\n+\tpartially.\n+\te.g.\n+\tindex:\n+\tfile1\n+\tfile2\n+\tpath/file1\n+\tzzz\n+\n+\tindex_name_pos(\"path/file1\", 10) returns 2, while\n+\tindex_name_pos(\"path\", 4) returns -3\n+\n+`for_each_index_entry`::\n+\n+\tIterates over all cache_entries in the index filtered by\n+\tfilter_opts in the index_state. For each cache entry fn is\n+\texecuted with cb_data as callback data. From within the loop\n+\tdo `return 0` to continue, or `return 1` to break the loop.\n+\n+TODO\n+----\n Talk about and , things like:\n \n-* cache -> the_index macros\n-* read_index()\n * write_index()\n * ie_match_stat() and ie_modified(); how they are different and when to\n use which.\n-* index_name_pos()\n * remove_index_entry_at()\n * remove_file_from_index()\n * add_file_to_index()\n@@ -18,4 +64,4 @@ Talk about and , things like:\n * cache_tree_invalidate_path()\n * cache_tree_update()\n \n-(JC, Linus)\n+(JC, Linus, Thomas Gummerer)\n-- \n1.8.3.453.g1dfc63d\n"},{"id":"223180","messageId":"1373650024-3001-7-git-send-email-t.gummerer@gmail.com","threadId":"34419","inReplyTo":"1373650024-3001-1-git-send-email-t.gummerer@gmail.com","subject":"[PATCH v2 06/19] read-cache: add index reading api","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2013-07-12T17:26:51Z","receivedAt":"2013-07-12T17:26:51Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"Add an api for access to the index file. Currently there is only a very\nbasic api for accessing the index file, which only allows a full read of\nthe index, and lets the users of the data filter it. The new index api\ngives the users the possibility to use only part of the index and\nprovides functions for iterating over and accessing cache entries.\n\nThis simplifies future improvements to the in-memory format, as changes\nwill be concentrated on one file, instead of the whole git source code.\n\nHelped-by: Nguyễn Thái Ngọc Duy \nSigned-off-by: Thomas Gummerer \n---\n cache.h | 42 +++++++++++++++++++++++++++++++++++++++++-\n read-cache-v2.c | 35 +++++++++++++++++++++++++++++++++--\n read-cache.c | 44 ++++++++++++++++++++++++++++++++++++++++----\n read-cache.h | 8 +++++++-\n 4 files changed, 121 insertions(+), 8 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 5082b34..d305d21 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -127,7 +127,7 @@ struct cache_entry {\n \tunsigned int ce_flags;\n \tunsigned int ce_namelen;\n \tunsigned char sha1[20];\n-\tstruct cache_entry *next;\n+\tstruct cache_entry *next; /* used by name_hash */\n \tchar name[FLEX_ARRAY]; /* more */\n };\n \n@@ -258,6 +258,29 @@ static inline unsigned int canon_mode(unsigned int mode)\n \n #define cache_entry_size(len) (offsetof(struct cache_entry,name) + (len) + 1)\n \n+/*\n+ * Options by which the index should be filtered when read partially.\n+ *\n+ * pathspec: The pathspec which the index entries have to match\n+ * seen: Used to return the seen parameter from match_pathspec()\n+ * max_prefix_len: The common prefix length of the pathspecs\n+ *\n+ * read_staged: used to indicate if the conflicted entries (entries\n+ * with a stage) should be included\n+ * read_cache_tree: used to indicate if the cache-tree should be read\n+ * read_resolve_undo: used to indicate if the resolve undo data should\n+ * be read\n+ */\n+struct filter_opts {\n+\tconst struct pathspec *pathspec;\n+\tchar *seen;\n+\tint max_prefix_len;\n+\n+\tint read_staged;\n+\tint read_cache_tree;\n+\tint read_resolve_undo;\n+};\n+\n struct index_state {\n \tstruct cache_entry **cache;\n \tunsigned int version;\n@@ -270,6 +293,8 @@ struct index_state {\n \tstruct hash_table name_hash;\n \tstruct hash_table dir_hash;\n \tstruct index_ops *ops;\n+\tstruct internal_ops *internal_ops;\n+\tstruct filter_opts *filter_opts;\n };\n \n extern struct index_state the_index;\n@@ -311,6 +336,12 @@ extern void free_name_hash(struct index_state *istate);\n #define unmerge_cache_entry_at(at) unmerge_index_entry_at(&the_index, at)\n #define unmerge_cache(pathspec) unmerge_index(&the_index, pathspec)\n #define read_blob_data_from_cache(path, sz) read_blob_data_from_index(&the_index, (path), (sz))\n+\n+/* index api */\n+#define read_cache_filtered(opts) read_index_filtered(&the_index, (opts))\n+#define read_cache_filtered_from(path, opts) read_index_filtered_from(&the_index, (path), (opts))\n+#define for_each_cache_entry(fn, cb_data) \\\n+\tfor_each_index_entry(&the_index, (fn), (cb_data))\n #endif\n \n enum object_type {\n@@ -438,6 +469,15 @@ extern int init_db(const char *template_dir, unsigned int flags);\n \t\t} \\\n \t} while (0)\n \n+/* index api */\n+extern int read_index_filtered(struct index_state *, struct filter_opts *opts);\n+extern int read_index_filtered_from(struct index_state *, const char *path, struct filter_opts *opts);\n+\n+typedef int each_cache_entry_fn(struct cache_entry *ce, void *);\n+extern int for_each_index_entry(struct index_state *istate,\n+\t\t\t\teach_cache_entry_fn, void *);\n+\n+\n /* Initialize and use the cache information */\n extern int read_index(struct index_state *);\n extern int read_index_preload(struct index_state *, const char **pathspec);\ndiff --git a/read-cache-v2.c b/read-cache-v2.c\nindex a6883c3..51b618f 100644\n--- a/read-cache-v2.c\n+++ b/read-cache-v2.c\n@@ -3,6 +3,7 @@\n #include \"resolve-undo.h\"\n #include \"cache-tree.h\"\n #include \"varint.h\"\n+#include \"dir.h\"\n \n /* Mask for the name length in ce_flags in the on-disk index */\n #define CE_NAMEMASK (0x0fff)\n@@ -207,8 +208,14 @@ static int read_index_extension(struct index_state *istate,\n \treturn 0;\n }\n \n+/*\n+ * The performance is the same if we read the whole index or only\n+ * part of it, therefore we always read the whole index to avoid\n+ * having to re-read it later. The filter_opts will determine\n+ * what part of the index is used when retrieving the cache-entries.\n+ */\n static int read_index_v2(struct index_state *istate, void *mmap,\n-\t\t\t unsigned long mmap_size)\n+\t\t\t unsigned long mmap_size, struct filter_opts *opts)\n {\n \tint i;\n \tunsigned long src_offset;\n@@ -238,7 +245,6 @@ static int read_index_v2(struct index_state *istate, void *mmap,\n \t\tdisk_ce = (struct ondisk_cache_entry *)((char *)mmap + src_offset);\n \t\tce = create_from_disk(disk_ce, &consumed, previous_name);\n \t\tset_index_entry(istate, i, ce);\n-\n \t\tsrc_offset += consumed;\n \t}\n \tstrbuf_release(&previous_name_buf);\n@@ -548,9 +554,34 @@ static int write_index_v2(struct index_state *istate, int newfd)\n \treturn 0;\n }\n \n+int for_each_index_entry_v2(struct index_state *istate, each_cache_entry_fn fn, void *cb_data)\n+{\n+\tint i, ret = 0;\n+\tstruct filter_opts *opts= istate->filter_opts;\n+\n+\tfor (i = 0; i < istate->cache_nr; i++) {\n+\t\tstruct cache_entry *ce = istate->cache[i];\n+\n+\t\tif (opts && !opts->read_staged && ce_stage(ce))\n+\t\t\tcontinue;\n+\n+\t\tif (opts && !match_pathspec_depth(opts->pathspec, ce->name, ce_namelen(ce),\n+\t\t\t\t\t\t opts->max_prefix_len, opts->seen))\n+\t\t\tcontinue;\n+\n+\t\tif ((ret = fn(istate->cache[i], cb_data)))\n+\t\t\tbreak;\n+\t}\n+\treturn ret;\n+}\n+\n struct index_ops v2_ops = {\n \tmatch_stat_basic,\n \tverify_hdr,\n \tread_index_v2,\n \twrite_index_v2\n };\n+\n+struct internal_ops v2_internal_ops = {\n+\tfor_each_index_entry_v2,\n+};\ndiff --git a/read-cache.c b/read-cache.c\nindex 3e3a0e2..9053d43 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -996,6 +996,7 @@ int add_index_entry(struct index_state *istate, struct cache_entry *ce, int opti\n \t\tmemmove(istate->cache + pos + 1,\n \t\t\tistate->cache + pos,\n \t\t\t(istate->cache_nr - pos - 1) * sizeof(ce));\n+\n \tset_index_entry(istate, pos, ce);\n \tistate->cache_changed = 1;\n \treturn 0;\n@@ -1272,7 +1273,32 @@ static int verify_hdr_version(struct index_state *istate,\n \n int read_index(struct index_state *istate)\n {\n-\treturn read_index_from(istate, get_index_file());\n+\treturn read_index_filtered_from(istate, get_index_file(), NULL);\n+}\n+\n+int read_index_filtered(struct index_state *istate, struct filter_opts *opts)\n+{\n+\treturn read_index_filtered_from(istate, get_index_file(), opts);\n+}\n+\n+int set_internal_ops(struct index_state *istate)\n+{\n+\tif (!istate->internal_ops && istate->cache)\n+\t\tistate->internal_ops = &v2_internal_ops;\n+\tif (!istate->internal_ops)\n+\t\treturn 0;\n+\treturn 1;\n+}\n+\n+/*\n+ * Execute fn for each index entry which is currently in istate. Data\n+ * can be given to the function using the cb_data parameter.\n+ */\n+int for_each_index_entry(struct index_state *istate, each_cache_entry_fn fn, void *cb_data)\n+{\n+\tif (!set_internal_ops(istate))\n+\t\treturn 0;\n+\treturn istate->internal_ops->for_each_index_entry(istate, fn, cb_data);\n }\n \n static int index_changed(struct stat *st_old, struct stat *st_new)\n@@ -1295,8 +1321,9 @@ static int index_changed(struct stat *st_old, struct stat *st_new)\n \treturn 0;\n }\n \n-/* remember to discard_cache() before reading a different cache! */\n-int read_index_from(struct index_state *istate, const char *path)\n+\n+int read_index_filtered_from(struct index_state *istate, const char *path,\n+\t\t\t struct filter_opts *opts)\n {\n \tint fd, err, i;\n \tstruct stat st_old, st_new;\n@@ -1337,7 +1364,7 @@ int read_index_from(struct index_state *istate, const char *path)\n \t\tif (istate->ops->verify_hdr(mmap, mmap_size) < 0)\n \t\t\terr = 1;\n \n-\t\tif (istate->ops->read_index(istate, mmap, mmap_size) < 0)\n+\t\tif (istate->ops->read_index(istate, mmap, mmap_size, opts) < 0)\n \t\t\terr = 1;\n \t\tistate->timestamp.sec = st_old.st_mtime;\n \t\tistate->timestamp.nsec = ST_MTIME_NSEC(st_old);\n@@ -1345,6 +1372,7 @@ int read_index_from(struct index_state *istate, const char *path)\n \t\t\tdie_errno(\"cannot stat the open index\");\n \n \t\tmunmap(mmap, mmap_size);\n+\t\tistate->filter_opts = opts;\n \t\tif (!index_changed(&st_old, &st_new) && !err)\n \t\t\treturn istate->cache_nr;\n \t}\n@@ -1352,6 +1380,13 @@ int read_index_from(struct index_state *istate, const char *path)\n \tdie(\"index file corrupt\");\n }\n \n+\n+/* remember to discard_cache() before reading a different cache! */\n+int read_index_from(struct index_state *istate, const char *path)\n+{\n+\treturn read_index_filtered_from(istate, path, NULL);\n+}\n+\n int is_index_unborn(struct index_state *istate)\n {\n \treturn (!istate->cache_nr && !istate->timestamp.sec);\n@@ -1374,6 +1409,7 @@ int discard_index(struct index_state *istate)\n \tfree(istate->cache);\n \tistate->cache = NULL;\n \tistate->cache_alloc = 0;\n+\tistate->filter_opts = NULL;\n \treturn 0;\n }\n \ndiff --git a/read-cache.h b/read-cache.h\nindex f31a38b..e72a585 100644\n--- a/read-cache.h\n+++ b/read-cache.h\n@@ -24,11 +24,17 @@ struct cache_version_header {\n struct index_ops {\n \tint (*match_stat_basic)(const struct cache_entry *ce, struct stat *st, int changed);\n \tint (*verify_hdr)(void *mmap, unsigned long size);\n-\tint (*read_index)(struct index_state *istate, void *mmap, unsigned long mmap_size);\n+\tint (*read_index)(struct index_state *istate, void *mmap, unsigned long mmap_size,\n+\t\t\t struct filter_opts *opts);\n \tint (*write_index)(struct index_state *istate, int newfd);\n };\n \n+struct internal_ops {\n+\tint (*for_each_index_entry)(struct index_state *istate, each_cache_entry_fn fn, void *cb_data);\n+};\n+\n extern struct index_ops v2_ops;\n+extern struct internal_ops v2_internal_ops;\n \n #ifndef NEEDS_ALIGNED_ACCESS\n #define ntoh_s(var) ntohs(var)\n-- \n1.8.3.453.g1dfc63d\n"},{"id":"223181","messageId":"1373650024-3001-8-git-send-email-t.gummerer@gmail.com","threadId":"34419","inReplyTo":"1373650024-3001-1-git-send-email-t.gummerer@gmail.com","subject":"[PATCH v2 07/19] make sure partially read index is not changed","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2013-07-12T17:26:52Z","receivedAt":"2013-07-12T17:26:52Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"A partially read index file currently cannot be written to disk. Make\nsure that never happens, by erroring out when a caller tries to change a\npartially read index. The caller is responsible for reading the whole\nindex when it's trying to change it later.\n\nForcing the caller to load the right part of the index file instead of\nre-reading it when changing it, gives a bit of a performance advantage,\nby avoiding to read parts of the index twice.\n\nSigned-off-by: Thomas Gummerer \n---\n builtin/update-index.c | 4 ++++\n cache.h | 1 +\n read-cache-v2.c | 2 ++\n read-cache.c | 10 ++++++++++\n 4 files changed, 17 insertions(+)\n\ndiff --git a/builtin/update-index.c b/builtin/update-index.c\nindex 5c7762e..4c6e3a6 100644\n--- a/builtin/update-index.c\n+++ b/builtin/update-index.c\n@@ -49,6 +49,8 @@ static int mark_ce_flags(const char *path, int flag, int mark)\n \tint namelen = strlen(path);\n \tint pos = cache_name_pos(path, namelen);\n \tif (0 <= pos) {\n+\t\tif (active_cache_partially_read)\n+\t\t\tdie(\"BUG: Can't change a partially read index\");\n \t\tif (mark)\n \t\t\tactive_cache[pos]->ce_flags |= flag;\n \t\telse\n@@ -253,6 +255,8 @@ static void chmod_path(int flip, const char *path)\n \tpos = cache_name_pos(path, strlen(path));\n \tif (pos < 0)\n \t\tgoto fail;\n+\tif (active_cache_partially_read)\n+\t\tdie(\"BUG: Can't change a partially read index\");\n \tce = active_cache[pos];\n \tmode = ce->ce_mode;\n \tif (!S_ISREG(mode))\ndiff --git a/cache.h b/cache.h\nindex d305d21..455b772 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -311,6 +311,7 @@ extern void free_name_hash(struct index_state *istate);\n #define active_alloc (the_index.cache_alloc)\n #define active_cache_changed (the_index.cache_changed)\n #define active_cache_tree (the_index.cache_tree)\n+#define active_cache_partially_read (the_index.filter_opts)\n \n #define read_cache() read_index(&the_index)\n #define read_cache_from(path) read_index_from(&the_index, (path))\ndiff --git a/read-cache-v2.c b/read-cache-v2.c\nindex 51b618f..f3c0685 100644\n--- a/read-cache-v2.c\n+++ b/read-cache-v2.c\n@@ -479,6 +479,8 @@ static int write_index_v2(struct index_state *istate, int newfd)\n \tstruct stat st;\n \tstruct strbuf previous_name_buf = STRBUF_INIT, *previous_name;\n \n+\tif (istate->filter_opts)\n+\t\tdie(\"BUG: index: cannot write a partially read index\");\n \tfor (i = removed = extended = 0; i < entries; i++) {\n \t\tif (cache[i]->ce_flags & CE_REMOVE)\n \t\t\tremoved++;\ndiff --git a/read-cache.c b/read-cache.c\nindex 9053d43..ab716ed 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -30,6 +30,8 @@ static void replace_index_entry(struct index_state *istate, int nr, struct cache\n {\n \tstruct cache_entry *old = istate->cache[nr];\n \n+\tif (istate->filter_opts)\n+\t\tdie(\"BUG: Can't change a partially read index\");\n \tremove_name_hash(istate, old);\n \tset_index_entry(istate, nr, ce);\n \tistate->cache_changed = 1;\n@@ -467,6 +469,8 @@ int remove_index_entry_at(struct index_state *istate, int pos)\n {\n \tstruct cache_entry *ce = istate->cache[pos];\n \n+\tif (istate->filter_opts)\n+\t\tdie(\"BUG: Can't change a partially read index\");\n \trecord_resolve_undo(istate, ce);\n \tremove_name_hash(istate, ce);\n \tistate->cache_changed = 1;\n@@ -973,6 +977,8 @@ int add_index_entry(struct index_state *istate, struct cache_entry *ce, int opti\n {\n \tint pos;\n \n+\tif (istate->filter_opts)\n+\t\tdie(\"BUG: Can't change a partially read index\");\n \tif (option & ADD_CACHE_JUST_APPEND)\n \t\tpos = istate->cache_nr;\n \telse {\n@@ -1173,6 +1179,8 @@ int refresh_index(struct index_state *istate, unsigned int flags, const char **p\n \t\t\t\t/* If we are doing --really-refresh that\n \t\t\t\t * means the index is not valid anymore.\n \t\t\t\t */\n+\t\t\t\tif (istate->filter_opts)\n+\t\t\t\t\tdie(\"BUG: Can't change a partially read index\");\n \t\t\t\tce->ce_flags &= ~CE_VALID;\n \t\t\t\tistate->cache_changed = 1;\n \t\t\t}\n@@ -1331,6 +1339,8 @@ int read_index_filtered_from(struct index_state *istate, const char *path,\n \tvoid *mmap;\n \tsize_t mmap_size;\n \n+\tif (istate->filter_opts)\n+\t\tdie(\"BUG: Can't re-read partially read index\");\n \terrno = EBUSY;\n \tif (istate->initialized)\n \t\treturn istate->cache_nr;\n-- \n1.8.3.453.g1dfc63d\n"},{"id":"223182","messageId":"1373650024-3001-9-git-send-email-t.gummerer@gmail.com","threadId":"34419","inReplyTo":"1373650024-3001-1-git-send-email-t.gummerer@gmail.com","subject":"[PATCH v2 08/19] grep.c: Use index api","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2013-07-12T17:26:53Z","receivedAt":"2013-07-12T17:26:53Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"Signed-off-by: Thomas Gummerer \n---\n builtin/grep.c | 71 ++++++++++++++++++++++++++++++----------------------------\n 1 file changed, 37 insertions(+), 34 deletions(-)\n\ndiff --git a/builtin/grep.c b/builtin/grep.c\nindex a419cda..8b02644 100644\n--- a/builtin/grep.c\n+++ b/builtin/grep.c\n@@ -368,41 +368,33 @@ static void run_pager(struct grep_opt *opt, const char *prefix)\n \tfree(argv);\n }\n \n-static int grep_cache(struct grep_opt *opt, const struct pathspec *pathspec, int cached)\n+struct grep_opts {\n+\tstruct grep_opt *opt;\n+\tconst struct pathspec *pathspec;\n+\tint cached;\n+\tint hit;\n+};\n+\n+static int grep_cache(struct cache_entry *ce, void *cb_data)\n {\n-\tint hit = 0;\n-\tint nr;\n-\tread_cache();\n+\tstruct grep_opts *opts = cb_data;\n \n-\tfor (nr = 0; nr < active_nr; nr++) {\n-\t\tstruct cache_entry *ce = active_cache[nr];\n-\t\tif (!S_ISREG(ce->ce_mode))\n-\t\t\tcontinue;\n-\t\tif (!match_pathspec_depth(pathspec, ce->name, ce_namelen(ce), 0, NULL))\n-\t\t\tcontinue;\n-\t\t/*\n-\t\t * If CE_VALID is on, we assume worktree file and its cache entry\n-\t\t * are identical, even if worktree file has been modified, so use\n-\t\t * cache version instead\n-\t\t */\n-\t\tif (cached || (ce->ce_flags & CE_VALID) || ce_skip_worktree(ce)) {\n-\t\t\tif (ce_stage(ce))\n-\t\t\t\tcontinue;\n-\t\t\thit |= grep_sha1(opt, ce->sha1, ce->name, 0, ce->name);\n-\t\t}\n-\t\telse\n-\t\t\thit |= grep_file(opt, ce->name);\n-\t\tif (ce_stage(ce)) {\n-\t\t\tdo {\n-\t\t\t\tnr++;\n-\t\t\t} while (nr < active_nr &&\n-\t\t\t\t !strcmp(ce->name, active_cache[nr]->name));\n-\t\t\tnr--; /* compensate for loop control */\n-\t\t}\n-\t\tif (hit && opt->status_only)\n-\t\t\tbreak;\n-\t}\n-\treturn hit;\n+\tif (!S_ISREG(ce->ce_mode))\n+\t\treturn 0;\n+\tif (!match_pathspec_depth(opts->pathspec, ce->name, ce_namelen(ce), 0, NULL))\n+\t\treturn 0;\n+\t/*\n+\t * If CE_VALID is on, we assume worktree file and its cache entry\n+\t * are identical, even if worktree file has been modified, so use\n+\t * cache version instead\n+\t */\n+\tif (opts->cached || (ce->ce_flags & CE_VALID) || ce_skip_worktree(ce))\n+\t\topts->hit |= grep_sha1(opts->opt, ce->sha1, ce->name, 0, ce->name);\n+\telse\n+\t\topts->hit |= grep_file(opts->opt, ce->name);\n+\tif (opts->hit && opts->opt->status_only)\n+\t\treturn 1;\n+\treturn 0;\n }\n \n static int grep_tree(struct grep_opt *opt, const struct pathspec *pathspec,\n@@ -895,10 +887,21 @@ int cmd_grep(int argc, const char **argv, const char *prefix)\n \t} else if (0 <= opt_exclude) {\n \t\tdie(_(\"--[no-]exclude-standard cannot be used for tracked contents.\"));\n \t} else if (!list.nr) {\n+\t\tstruct grep_opts opts;\n+\t\tstruct filter_opts *filter_opts = xmalloc(sizeof(*filter_opts));\n+\n \t\tif (!cached)\n \t\t\tsetup_work_tree();\n \n-\t\thit = grep_cache(&opt, &pathspec, cached);\n+\t\tmemset(filter_opts, 0, sizeof(*filter_opts));\n+\t\tfilter_opts->pathspec = &pathspec;\n+\t\topts.opt = &opt;\n+\t\topts.pathspec = &pathspec;\n+\t\topts.cached = cached;\n+\t\topts.hit = 0;\n+\t\tread_cache_filtered(filter_opts);\n+\t\tfor_each_cache_entry(grep_cache, &opts);\n+\t\thit = opts.hit;\n \t} else {\n \t\tif (cached)\n \t\t\tdie(_(\"both --cached and trees are given.\"));\n-- \n1.8.3.453.g1dfc63d\n"},{"id":"223183","messageId":"1373650024-3001-10-git-send-email-t.gummerer@gmail.com","threadId":"34419","inReplyTo":"1373650024-3001-1-git-send-email-t.gummerer@gmail.com","subject":"[PATCH v2 09/19] ls-files.c: use index api","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2013-07-12T17:26:54Z","receivedAt":"2013-07-12T17:26:54Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"Use the index api to read only part of the index, if the on-disk version\nof the index is index-v5.\n\nSigned-off-by: Thomas Gummerer \n---\n builtin/ls-files.c | 31 ++++++++++++++++++++-----------\n 1 file changed, 20 insertions(+), 11 deletions(-)\n\ndiff --git a/builtin/ls-files.c b/builtin/ls-files.c\nindex 08d9786..80cc398 100644\n--- a/builtin/ls-files.c\n+++ b/builtin/ls-files.c\n@@ -31,6 +31,7 @@ static const char *prefix;\n static int max_prefix_len;\n static int prefix_len;\n static const char **pathspec;\n+static struct pathspec pathspec_struct;\n static int error_unmatch;\n static char *ps_matched;\n static const char *with_tree;\n@@ -457,6 +458,7 @@ int cmd_ls_files(int argc, const char **argv, const char *cmd_prefix)\n \tstruct dir_struct dir;\n \tstruct exclude_list *el;\n \tstruct string_list exclude_list = STRING_LIST_INIT_NODUP;\n+\tstruct filter_opts *opts = xmalloc(sizeof(*opts));\n \tstruct option builtin_ls_files_options[] = {\n \t\t{ OPTION_CALLBACK, 'z', NULL, NULL, NULL,\n \t\t\tN_(\"paths are separated with NUL character\"),\n@@ -522,9 +524,6 @@ int cmd_ls_files(int argc, const char **argv, const char *cmd_prefix)\n \t\tprefix_len = strlen(prefix);\n \tgit_config(git_default_config, NULL);\n \n-\tif (read_cache() < 0)\n-\t\tdie(\"index file corrupt\");\n-\n \targc = parse_options(argc, argv, prefix, builtin_ls_files_options,\n \t\t\tls_files_usage, 0);\n \tel = add_exclude_list(&dir, EXC_CMDL, \"--exclude option\");\n@@ -556,14 +555,7 @@ int cmd_ls_files(int argc, const char **argv, const char *cmd_prefix)\n \t\tsetup_work_tree();\n \n \tpathspec = get_pathspec(prefix, argv);\n-\n-\t/* be nice with submodule paths ending in a slash */\n-\tif (pathspec)\n-\t\tstrip_trailing_slash_from_submodules();\n-\n-\t/* Find common prefix for all pathspec's */\n-\tmax_prefix = common_prefix(pathspec);\n-\tmax_prefix_len = max_prefix ? strlen(max_prefix) : 0;\n+\tinit_pathspec(&pathspec_struct, pathspec);\n \n \t/* Treat unmatching pathspec elements as errors */\n \tif (pathspec && error_unmatch) {\n@@ -573,6 +565,23 @@ int cmd_ls_files(int argc, const char **argv, const char *cmd_prefix)\n \t\tps_matched = xcalloc(1, num);\n \t}\n \n+\tif (!with_tree) {\n+\t\tmemset(opts, 0, sizeof(*opts));\n+\t\topts->pathspec = &pathspec_struct;\n+\t\topts->read_staged = 1;\n+\t\tif (show_resolve_undo)\n+\t\t\topts->read_resolve_undo = 1;\n+\t\tread_cache_filtered(opts);\n+\t} else {\n+\t\tread_cache();\n+\t}\n+\t/* be nice with submodule paths ending in a slash */\n+\tif (pathspec)\n+\t\tstrip_trailing_slash_from_submodules();\n+\n+\tmax_prefix = common_prefix(pathspec);\n+\tmax_prefix_len = max_prefix ? strlen(max_prefix) : 0;\n+\n \tif ((dir.flags & DIR_SHOW_IGNORED) && !exc_given)\n \t\tdie(\"ls-files --ignored needs some exclude pattern\");\n \n-- \n1.8.3.453.g1dfc63d\n"},{"id":"223186","messageId":"1373650024-3001-11-git-send-email-t.gummerer@gmail.com","threadId":"34419","inReplyTo":"1373650024-3001-1-git-send-email-t.gummerer@gmail.com","subject":"[PATCH v2 10/19] documentation: add documentation of the index-v5 file format","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2013-07-12T17:26:55Z","receivedAt":"2013-07-12T17:26:55Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"Add a documentation of the index file format version 5 to\nDocumentation/technical.\n\nHelped-by: Michael Haggerty \nHelped-by: Junio C Hamano \nHelped-by: Thomas Rast \nHelped-by: Nguyen Thai Ngoc Duy \nHelped-by: Robin Rosenberg \nSigned-off-by: Thomas Gummerer \n---\n Documentation/technical/index-file-format-v5.txt | 296 +++++++++++++++++++++++\n 1 file changed, 296 insertions(+)\n create mode 100644 Documentation/technical/index-file-format-v5.txt\n\ndiff --git a/Documentation/technical/index-file-format-v5.txt b/Documentation/technical/index-file-format-v5.txt\nnew file mode 100644\nindex 0000000..4213087\n--- /dev/null\n+++ b/Documentation/technical/index-file-format-v5.txt\n@@ -0,0 +1,296 @@\n+GIT index format\n+================\n+\n+== The git index\n+\n+ The git index file (.git/index) documents the status of the files\n+ in the git staging area.\n+\n+ The staging area is used for preparing commits, merging, etc.\n+\n+== The git index file format\n+\n+ All binary numbers are in network byte order. Version 5 is described\n+ here. The index file consists of various sections. They appear in\n+ the following order in the file.\n+\n+ - header: the description of the index format, including it's signature,\n+ version and various other fields that are used internally.\n+\n+ - diroffsets (ndir entries of \"direcotry offset\"): A 4-byte offset\n+ relative to the beginning of the \"direntries block\" (see below)\n+ for each of the ndir directories in the index, sorted by pathname\n+ (of the directory it's pointing to). [1]\n+\n+ - direntries (ndir entries of \"directory offset\"): A directory entry\n+ for each of the ndir directories in the index, sorted by pathname\n+ (see below). [2]\n+\n+ - fileoffsets (nfile entries of \"file offset\"): A 4-byte offset\n+ relative to the beginning of the fileentries block (see below)\n+ for each of the nfile files in the index. [1]\n+\n+ - fileentries (nfile entries of \"file entry\"): A file entry for\n+ each of the nfile files in the index (see below).\n+\n+ - crdata: A number of entries for conflicted data/resolved conflicts\n+ (see below).\n+\n+ - Extensions (Currently none, see below in the future)\n+\n+ Extensions are identified by signature. Optional extensions can\n+ be ignored if GIT does not understand them.\n+\n+ GIT supports an arbitrary number of extension, but currently none\n+ is implemented. [3]\n+\n+ extsig (32-bits): extension signature. If the first byte is 'A'..'Z'\n+ the extension is optional and can be ignored.\n+\n+ extsize (32-bits): size of the extension, excluding the header\n+ (extsig, extsize, extchecksum).\n+\n+ extchecksum (32-bits): crc32 checksum of the extension signature\n+ and size.\n+\n+ - Extension data.\n+\n+== Header\n+ sig (32-bits): Signature:\n+ The signature is { 'D', 'I', 'R', 'C' } (stands for \"dircache\")\n+\n+ vnr (32-bits): Version number:\n+ The current supported versions are 2, 3, 4 and 5.\n+\n+ ndir (32-bits): number of directories in the index.\n+\n+ nfile (32-bits): number of file entries in the index.\n+\n+ fblockoffset (32-bits): offset to the file block, relative to the\n+ beginning of the file.\n+\n+ - Offset to the extensions.\n+\n+ nextensions (32-bits): number of extensions.\n+\n+ extoffset (32-bits): offset to the extension. (Possibly none, as\n+ many as indicated in the 4-byte number of extensions)\n+\n+ headercrc (32-bits): crc checksum including the header and the\n+ offsets to the extensions.\n+\n+\n+== Directory offsets (diroffsets)\n+\n+ diroffset (32-bits): offset to the directory relative to the beginning\n+ of the index file. There are ndir + 1 offsets in the diroffset table,\n+ the last is pointing to the end of the last direntry. With this last\n+ entry, we are able to replace the strlen of when reading the directory\n+ name, by calculating it from diroffset[n+1]-diroffset[n]-61. 61 is the\n+ size of the directory data, which follows each each directory + the\n+ crc sum + the NUL byte.\n+\n+ This part is needed for making the directory entries bisectable and\n+ thus allowing a binary search.\n+\n+== Directory entry (direntries)\n+\n+ Directory entries are sorted in lexicographic order by the name\n+ of their path starting with the root.\n+\n+ pathname (variable length, nul terminated): relative to top level\n+ directory (without the leading slash). '/' is used as path\n+ separator. A string of length 0 ('') indicates the root directory.\n+ The special path components \".\", and \"..\" (without quotes) are\n+ disallowed. The path also includes a trailing slash. [9]\n+\n+ foffset (32-bits): offset to the lexicographically first file in\n+ the file offsets (fileoffsets), relative to the beginning of\n+ the fileoffset block.\n+\n+ cr (32-bits): offset to conflicted/resolved data at the end of the\n+ index. 0 if there is no such data. [4]\n+\n+ ncr (32-bits): number of conflicted/resolved data entries at the\n+ end of the index if the offset is non 0. If cr is 0, ncr is\n+ also 0.\n+\n+ nsubtrees (32-bits): number of subtrees this tree has in the index.\n+\n+ nfiles (32-bits): number of files in the directory, that are in\n+ the index.\n+\n+ nentries (32-bits): number of entries in the index that is covered\n+ by the tree this entry represents. (-1 if the entry is invalid).\n+ This number includes all the files in this tree, recursively.\n+\n+ objname (160-bits): object name for the object that would result\n+ from writing this span of index as a tree. This is only valid\n+ if nentries is valid, meaning the cache-tree is valid.\n+\n+ flags (16-bits): 'flags' field split into (high to low bits) (For\n+ D/F conflicts)\n+\n+ stage (2-bits): stage of the directory during merge\n+\n+ 14-bit unused\n+\n+ dircrc (32-bits): crc32 checksum for each directory entry.\n+\n+ The last 24 bytes (4-byte number of entries + 160-bit object name) are\n+ for the cache tree. An entry can be in an invalidated state which is\n+ represented by having -1 in the entry_count field.\n+\n+ The entries are written out in the top-down, depth-first order. The\n+ first entry represents the root level of the repository, followed by\n+ the first subtree - let's call it A - of the root level, followed by\n+ the first subtree of A, ... There is no prefix compression for\n+ directories.\n+\n+== File offsets (fileoffsets)\n+\n+ fileoffset (32-bits): offset to the file relative to the beginning of\n+ the fileentries block.\n+\n+ This part is needed for making the file entries bisectable and\n+ thus allowing a binary search. There are nfile + 1 offsets in the\n+ fileoffset table, the last is pointing to the end of the last\n+ fileentry. With this last entry, we can replace the strlen when\n+ reading each filename, by calculating its length with the offsets.\n+\n+== File entry (fileentries)\n+\n+ File entries are sorted in ascending order on the name field, after the\n+ respective offset given by the directory entries. All file names are\n+ prefix compressed, meaning the file name is relative to the directory.\n+\n+ filename (variable length, nul terminated). The exact encoding is\n+ undefined, but the filename cannot contain a NUL byte (iow, the same\n+ encoding as a UNIX pathname).\n+\n+ flags (16-bits): 'flags' field split into (high to low bits)\n+\n+ assumevalid (1-bit): assume-valid flag\n+\n+ intenttoadd (1-bit): intent-to-add flag, used by \"git add -N\".\n+ Extended flag in index v3.\n+\n+ stage (2-bit): stage of the file during merge\n+\n+ skipworktree (1-bit): skip-worktree flag, used by sparse checkout.\n+ Extended flag in index v3.\n+\n+ smudged (1-bit): indicates if the file is racily smudged.\n+\n+ 10-bit unused, must be zero [6]\n+\n+ mode (16-bits): file mode, split into (high to low bits)\n+\n+ objtype (4-bits): object type\n+ valid values in binary are 1000 (regular file), 1010 (symbolic\n+ link) and 1110 (gitlink)\n+\n+ 3-bit unused\n+\n+ permission (9-bits): unix permission. Only 0755 and 0644 are valid\n+ for regular files. Symbolic links and gitlinks have value 0 in\n+ this field.\n+\n+ mtimes (32-bits): mtime seconds, the last time a file's data changed\n+ this is stat(2) data\n+\n+ mtimens (32-bits): mtime nanosecond fractions\n+ this is stat(2) data\n+\n+ file size (32-bits): The on-disk size, trucated to 32-bit.\n+ this is stat(2) data\n+\n+ statcrc (32-bits): crc32 checksum over ctime seconds, ctime\n+ nanoseconds, ino, dev, uid, gid (All stat(2) data\n+ except mtime and file size). If the statcrc is 0 it will\n+ be ignored. [7]\n+\n+ objhash (160-bits): SHA-1 for the represented object\n+\n+ entrycrc (32-bits): crc32 checksum for the file entry. The crc code\n+ includes the offset to the offset to the file, relative to the\n+ beginning of the file.\n+\n+== Conflict data\n+\n+ A conflict is represented in the index as a set of higher stage entries.\n+ These entries are stored at the end of the index. When a conflict is\n+ resolved (e.g. with \"git add path\"). A bit is flipped, to indicate that\n+ the conflict is resolved, but the entries will be kept, so that\n+ conflicts can be recreated (e.g. with \"git checkout -m\", in case users\n+ want to redo a conflict resolution from scratch.\n+\n+ The first part of a conflict (usually stage 1) will be stored both in\n+ the entries part of the index and in the conflict part. All other parts\n+ will only be stored in the conflict part.\n+\n+ filename (variable length, nul terminated): filename of the entry,\n+ relative to its containing directory).\n+\n+ nfileconflicts (32-bits): number of conflicts for the file [8]\n+\n+ flags (nfileconflicts entries of \"flags\") (16-bits): 'flags' field\n+ split into:\n+\n+ conflicted (1-bit): conflicted state (conflicted/resolved) (1 if\n+ conflicted)\n+\n+ stage (2-bits): stage during merge.\n+\n+ 13-bit unused\n+\n+ entry_mode (nfileconflicts entries of \"entry mode\") (16-bits):\n+ octal numbers, entry mode of eache entry in the different stages.\n+ (How many is defined by the 4-byte number before)\n+\n+ objectnames (nfileconflicts entries of \"object name\") (160-bits):\n+ object names of the different stages.\n+\n+ conflictcrc (32-bits): crc32 checksum over conflict data.\n+\n+== Design explanations\n+\n+[1] The directory and file offsets are included in the index format\n+ to enable bisectability of the index, for binary searches.Updating\n+ a single entry and partial reading will benefit from this.\n+\n+[2] The directories are saved in their own block, to be able to\n+ quickly search for a directory in the index. They include a\n+ offset to the (lexically) first file in the directory.\n+\n+[3] The data of the cache-tree extension and the resolve undo\n+ extension is now part of the index itself, but if other extensions\n+ come up in the future, there is no need to change the index, they\n+ can simply be added at the end.\n+\n+[4] To avoid rewrites of the whole index when there are conflicts or\n+ conflicts are being resolved, conflicted data will be stored at\n+ the end of the index. To mark the conflict resolved, just a bit\n+ has to be flipped. The data will still be there, if a user wants\n+ to redo the conflict resolution.\n+\n+[5] Since only 4 modes are effectively allowed in git but 32-bit are\n+ used to store them, having a two bit flag for the mode is enough\n+ and saves 4 byte per entry.\n+\n+[6] The length of the file name was dropped, since each file name is\n+ nul terminated anyway.\n+\n+[7] Since all stat data (except mtime and ctime) is just used for\n+ checking if a file has changed a checksum of the data is enough.\n+ In addition to that Thomas Rast suggested ctime could be ditched\n+ completely (core.trustctime=false) and thus included in the\n+ checksum. This would save 24 bytes per index entry, which would\n+ be about 4 MB on the Webkit index.\n+ (Thanks for the suggestion to Michael Haggerty)\n+\n+[8] Since there can be more stage #1 entries, it is necessary to know\n+ the number of conflict data entries there are.\n+\n+[9] As Michael Haggerty pointed out on the mailing list, storing the\n+ trailing slash will simplify a few operations.\n-- \n1.8.3.453.g1dfc63d\n"},{"id":"223187","messageId":"1373650024-3001-12-git-send-email-t.gummerer@gmail.com","threadId":"34419","inReplyTo":"1373650024-3001-1-git-send-email-t.gummerer@gmail.com","subject":"[PATCH v2 11/19] read-cache: make in-memory format aware of stat_crc","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2013-07-12T17:26:56Z","receivedAt":"2013-07-12T17:26:56Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"Make the in-memory format aware of the stat_crc used by index-v5.\nIt is simply ignored by index version prior to v5.\n\nSigned-off-by: Thomas Gummerer \n---\n cache.h | 1 +\n read-cache.c | 25 +++++++++++++++++++++++++\n 2 files changed, 26 insertions(+)\n\ndiff --git a/cache.h b/cache.h\nindex 455b772..2097105 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -127,6 +127,7 @@ struct cache_entry {\n \tunsigned int ce_flags;\n \tunsigned int ce_namelen;\n \tunsigned char sha1[20];\n+\tuint32_t ce_stat_crc;\n \tstruct cache_entry *next; /* used by name_hash */\n \tchar name[FLEX_ARRAY]; /* more */\n };\ndiff --git a/read-cache.c b/read-cache.c\nindex ab716ed..9bfbb4f 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -108,6 +108,29 @@ int match_stat_data(const struct stat_data *sd, struct stat *st)\n \treturn changed;\n }\n \n+static uint32_t calculate_stat_crc(struct cache_entry *ce)\n+{\n+\tunsigned int ctimens = 0;\n+\tuint32_t stat, stat_crc;\n+\n+\tstat = htonl(ce->ce_stat_data.sd_ctime.sec);\n+\tstat_crc = crc32(0, (Bytef*)&stat, 4);\n+#ifdef USE_NSEC\n+\tctimens = ce->ce_stat_data.sd_ctime.nsec;\n+#endif\n+\tstat = htonl(ctimens);\n+\tstat_crc = crc32(stat_crc, (Bytef*)&stat, 4);\n+\tstat = htonl(ce->ce_stat_data.sd_ino);\n+\tstat_crc = crc32(stat_crc, (Bytef*)&stat, 4);\n+\tstat = htonl(ce->ce_stat_data.sd_dev);\n+\tstat_crc = crc32(stat_crc, (Bytef*)&stat, 4);\n+\tstat = htonl(ce->ce_stat_data.sd_uid);\n+\tstat_crc = crc32(stat_crc, (Bytef*)&stat, 4);\n+\tstat = htonl(ce->ce_stat_data.sd_gid);\n+\tstat_crc = crc32(stat_crc, (Bytef*)&stat, 4);\n+\treturn stat_crc;\n+}\n+\n /*\n * This only updates the \"non-critical\" parts of the directory\n * cache, ie the parts that aren't tracked by GIT, and only used\n@@ -122,6 +145,8 @@ void fill_stat_cache_info(struct cache_entry *ce, struct stat *st)\n \n \tif (S_ISREG(st->st_mode))\n \t\tce_mark_uptodate(ce);\n+\n+\tce->ce_stat_crc = calculate_stat_crc(ce);\n }\n \n static int ce_compare_data(const struct cache_entry *ce, struct stat *st)\n-- \n1.8.3.453.g1dfc63d\n"},{"id":"223188","messageId":"1373650024-3001-13-git-send-email-t.gummerer@gmail.com","threadId":"34419","inReplyTo":"1373650024-3001-1-git-send-email-t.gummerer@gmail.com","subject":"[PATCH v2 12/19] read-cache: read index-v5","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2013-07-12T17:26:57Z","receivedAt":"2013-07-12T17:26:57Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"Make git read the index file version 5 without complaining.\n\nThis version of the reader doesn't read neither the cache-tree\nnor the resolve undo data, but doesn't choke on an index that\nincludes such data.\n\nHelped-by: Junio C Hamano \nHelped-by: Nguyen Thai Ngoc Duy \nHelped-by: Thomas Rast \nSigned-off-by: Thomas Gummerer \n---\n Makefile | 1 +\n cache.h | 75 ++++++-\n read-cache-v5.c | 638 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n read-cache.h | 1 +\n 4 files changed, 714 insertions(+), 1 deletion(-)\n create mode 100644 read-cache-v5.c\n\ndiff --git a/Makefile b/Makefile\nindex 73369ae..80e35f5 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -856,6 +856,7 @@ LIB_OBJS += quote.o\n LIB_OBJS += reachable.o\n LIB_OBJS += read-cache.o\n LIB_OBJS += read-cache-v2.o\n+LIB_OBJS += read-cache-v5.o\n LIB_OBJS += reflog-walk.o\n LIB_OBJS += refs.o\n LIB_OBJS += remote.o\ndiff --git a/cache.h b/cache.h\nindex 2097105..1e5cc77 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -99,7 +99,7 @@ unsigned long git_deflate_bound(git_zstream *, unsigned long);\n #define CACHE_SIGNATURE 0x44495243\t/* \"DIRC\" */\n \n #define INDEX_FORMAT_LB 2\n-#define INDEX_FORMAT_UB 4\n+#define INDEX_FORMAT_UB 5\n \n /*\n * The \"cache_time\" is just the low 32 bits of the\n@@ -121,6 +121,15 @@ struct stat_data {\n \tunsigned int sd_size;\n };\n \n+/*\n+ * The *next pointer is used in read_entries_v5 for holding\n+ * all the elements of a directory, and points to the next\n+ * cache_entry in a directory.\n+ *\n+ * It is reset by the add_name_hash call in set_index_entry\n+ * to set it to point to the next cache_entry in the\n+ * correct in-memory format ordering.\n+ */\n struct cache_entry {\n \tstruct stat_data ce_stat_data;\n \tunsigned int ce_mode;\n@@ -132,11 +141,59 @@ struct cache_entry {\n \tchar name[FLEX_ARRAY]; /* more */\n };\n \n+struct directory_entry {\n+\tstruct directory_entry *next;\n+\tstruct directory_entry *next_hash;\n+\tstruct cache_entry *ce;\n+\tstruct cache_entry *ce_last;\n+\tstruct conflict_entry *conflict;\n+\tstruct conflict_entry *conflict_last;\n+\tunsigned int conflict_size;\n+\tunsigned int de_foffset;\n+\tunsigned int de_cr;\n+\tunsigned int de_ncr;\n+\tunsigned int de_nsubtrees;\n+\tunsigned int de_nfiles;\n+\tunsigned int de_nentries;\n+\tunsigned char sha1[20];\n+\tunsigned short de_flags;\n+\tunsigned int de_pathlen;\n+\tchar pathname[FLEX_ARRAY];\n+};\n+\n+struct conflict_part {\n+\tstruct conflict_part *next;\n+\tunsigned short flags;\n+\tunsigned short entry_mode;\n+\tunsigned char sha1[20];\n+};\n+\n+struct conflict_entry {\n+\tstruct conflict_entry *next;\n+\tunsigned int nfileconflicts;\n+\tstruct conflict_part *entries;\n+\tunsigned int namelen;\n+\tunsigned int pathlen;\n+\tchar name[FLEX_ARRAY];\n+};\n+\n+struct ondisk_conflict_part {\n+\tunsigned short flags;\n+\tunsigned short entry_mode;\n+\tunsigned char sha1[20];\n+};\n+\n+#define CE_NAMEMASK (0x0fff)\n #define CE_STAGEMASK (0x3000)\n #define CE_EXTENDED (0x4000)\n #define CE_VALID (0x8000)\n+#define CE_SMUDGED (0x0400) /* index v5 only flag */\n #define CE_STAGESHIFT 12\n \n+#define CONFLICT_CONFLICTED (0x8000)\n+#define CONFLICT_STAGESHIFT 13\n+#define CONFLICT_STAGEMASK (0x6000)\n+\n /*\n * Range 0xFFFF0000 in ce_flags is divided into\n * two parts: in-memory flags and on-disk ones.\n@@ -173,6 +230,18 @@ struct cache_entry {\n #define CE_EXTENDED_FLAGS (CE_INTENT_TO_ADD | CE_SKIP_WORKTREE)\n \n /*\n+ * Representation of the extended on-disk flags in the v5 format.\n+ * They must not collide with the ordinary on-disk flags, and need to\n+ * fit in 16 bits. Note however that v5 does not save the name\n+ * length.\n+ */\n+#define CE_INTENT_TO_ADD_V5 (0x4000)\n+#define CE_SKIP_WORKTREE_V5 (0x0800)\n+#if (CE_VALID|CE_STAGEMASK) & (CE_INTENTTOADD_V5|CE_SKIPWORKTREE_V5)\n+#error \"v5 on-disk flags collide with ordinary on-disk flags\"\n+#endif\n+\n+/*\n * Safeguard to avoid saving wrong flags:\n * - CE_EXTENDED2 won't get saved until its semantic is known\n * - Bits in 0x0000FFFF have been saved in ce_flags already\n@@ -211,6 +280,8 @@ static inline unsigned create_ce_flags(unsigned stage)\n #define ce_skip_worktree(ce) ((ce)->ce_flags & CE_SKIP_WORKTREE)\n #define ce_mark_uptodate(ce) ((ce)->ce_flags |= CE_UPTODATE)\n \n+#define conflict_stage(c) ((CONFLICT_STAGEMASK & (c)->flags) >> CONFLICT_STAGESHIFT)\n+\n #define ce_permissions(mode) (((mode) & 0100) ? 0755 : 0644)\n static inline unsigned int create_ce_mode(unsigned int mode)\n {\n@@ -258,6 +329,8 @@ static inline unsigned int canon_mode(unsigned int mode)\n }\n \n #define cache_entry_size(len) (offsetof(struct cache_entry,name) + (len) + 1)\n+#define directory_entry_size(len) (offsetof(struct directory_entry,pathname) + (len) + 1)\n+#define conflict_entry_size(len) (offsetof(struct conflict_entry,name) + (len) + 1)\n \n /*\n * Options by which the index should be filtered when read partially.\ndiff --git a/read-cache-v5.c b/read-cache-v5.c\nnew file mode 100644\nindex 0000000..00112ea\n--- /dev/null\n+++ b/read-cache-v5.c\n@@ -0,0 +1,638 @@\n+#include \"cache.h\"\n+#include \"read-cache.h\"\n+#include \"resolve-undo.h\"\n+#include \"cache-tree.h\"\n+#include \"dir.h\"\n+\n+#define ptr_add(x,y) ((void *)(((char *)(x)) + (y)))\n+\n+struct cache_header {\n+\tunsigned int hdr_ndir;\n+\tunsigned int hdr_nfile;\n+\tunsigned int hdr_fblockoffset;\n+\tunsigned int hdr_nextension;\n+};\n+\n+/*****************************************************************\n+ * Index File I/O\n+ *****************************************************************/\n+\n+struct ondisk_cache_entry {\n+\tunsigned short flags;\n+\tunsigned short mode;\n+\tstruct cache_time mtime;\n+\tunsigned int size;\n+\tint stat_crc;\n+\tunsigned char sha1[20];\n+};\n+\n+struct ondisk_directory_entry {\n+\tunsigned int foffset;\n+\tunsigned int cr;\n+\tunsigned int ncr;\n+\tunsigned int nsubtrees;\n+\tunsigned int nfiles;\n+\tunsigned int nentries;\n+\tunsigned char sha1[20];\n+\tunsigned short flags;\n+};\n+\n+static int check_crc32(int initialcrc,\n+\t\t\tvoid *data,\n+\t\t\tsize_t len,\n+\t\t\tunsigned int expected_crc)\n+{\n+\tint crc;\n+\n+\tcrc = crc32(initialcrc, (Bytef*)data, len);\n+\treturn crc == expected_crc;\n+}\n+\n+static int match_stat_crc(struct stat *st, uint32_t expected_crc)\n+{\n+\tuint32_t data, stat_crc = 0;\n+\tunsigned int ctimens = 0;\n+\n+\tdata = htonl(st->st_ctime);\n+\tstat_crc = crc32(0, (Bytef*)&data, 4);\n+#ifdef USE_NSEC\n+\tctimens = ST_CTIME_NSEC(*st);\n+#endif\n+\tdata = htonl(ctimens);\n+\tstat_crc = crc32(stat_crc, (Bytef*)&data, 4);\n+\tdata = htonl(st->st_ino);\n+\tstat_crc = crc32(stat_crc, (Bytef*)&data, 4);\n+\tdata = htonl(st->st_dev);\n+\tstat_crc = crc32(stat_crc, (Bytef*)&data, 4);\n+\tdata = htonl(st->st_uid);\n+\tstat_crc = crc32(stat_crc, (Bytef*)&data, 4);\n+\tdata = htonl(st->st_gid);\n+\tstat_crc = crc32(stat_crc, (Bytef*)&data, 4);\n+\n+\treturn stat_crc == expected_crc;\n+}\n+\n+static int match_stat_basic(const struct cache_entry *ce,\n+\t\t\t struct stat *st,\n+\t\t\t int changed)\n+{\n+\n+\tif (ce->ce_stat_data.sd_mtime.sec != (unsigned int)st->st_mtime)\n+\t\tchanged |= MTIME_CHANGED;\n+#ifdef USE_NSEC\n+\tif (ce->ce_stat_data.sd_mtime.nsec != ST_MTIME_NSEC(*st))\n+\t\tchanged |= MTIME_CHANGED;\n+#endif\n+\tif (ce->ce_stat_data.sd_size != (unsigned int)st->st_size)\n+\t\tchanged |= DATA_CHANGED;\n+\n+\tif (trust_ctime && ce->ce_stat_crc != 0 && !match_stat_crc(st, ce->ce_stat_crc)) {\n+\t\tchanged |= OWNER_CHANGED;\n+\t\tchanged |= INODE_CHANGED;\n+\t}\n+\t/* Racily smudged entry? */\n+\tif (ce->ce_flags & CE_SMUDGED) {\n+\t\tif (!changed && !is_empty_blob_sha1(ce->sha1) && ce_modified_check_fs(ce, st))\n+\t\t\tchanged |= DATA_CHANGED;\n+\t}\n+\treturn changed;\n+}\n+\n+static int verify_hdr(void *mmap, unsigned long size)\n+{\n+\tuint32_t *filecrc;\n+\tunsigned int header_size;\n+\tstruct cache_version_header *hdr;\n+\tstruct cache_header *hdr_v5;\n+\n+\tif (size < sizeof(struct cache_version_header)\n+\t\t\t+ sizeof (struct cache_header) + 4)\n+\t\tdie(\"index file smaller than expected\");\n+\n+\thdr = mmap;\n+\thdr_v5 = ptr_add(mmap, sizeof(*hdr));\n+\t/* Size of the header + the size of the extensionoffsets */\n+\theader_size = sizeof(*hdr) + sizeof(*hdr_v5) + hdr_v5->hdr_nextension * 4;\n+\t/* Initialize crc */\n+\tfilecrc = ptr_add(mmap, header_size);\n+\tif (!check_crc32(0, hdr, header_size, ntohl(*filecrc)))\n+\t\treturn error(\"bad index file header crc signature\");\n+\treturn 0;\n+}\n+\n+static struct cache_entry *cache_entry_from_ondisk(struct ondisk_cache_entry *ondisk,\n+\t\t\t\t\t\t struct directory_entry *de,\n+\t\t\t\t\t\t char *name,\n+\t\t\t\t\t\t size_t len,\n+\t\t\t\t\t\t size_t prefix_len)\n+{\n+\tstruct cache_entry *ce = xmalloc(cache_entry_size(len + de->de_pathlen));\n+\tint flags;\n+\n+\tflags = ntoh_s(ondisk->flags);\n+\tce->ce_stat_data.sd_ctime.sec = 0;\n+\tce->ce_stat_data.sd_mtime.sec = ntoh_l(ondisk->mtime.sec);\n+\tce->ce_stat_data.sd_ctime.nsec = 0;\n+\tce->ce_stat_data.sd_mtime.nsec = ntoh_l(ondisk->mtime.nsec);\n+\tce->ce_stat_data.sd_dev = 0;\n+\tce->ce_stat_data.sd_ino = 0;\n+\tce->ce_stat_data.sd_uid = 0;\n+\tce->ce_stat_data.sd_gid = 0;\n+\tce->ce_stat_data.sd_size = ntoh_l(ondisk->size);\n+\tce->ce_mode = ntoh_s(ondisk->mode);\n+\tce->ce_flags = flags & CE_STAGEMASK;\n+\tce->ce_flags |= flags & CE_VALID;\n+\tce->ce_flags |= flags & CE_SMUDGED;\n+\tif (flags & CE_INTENT_TO_ADD_V5)\n+\t\tce->ce_flags |= CE_INTENT_TO_ADD;\n+\tif (flags & CE_SKIP_WORKTREE_V5)\n+\t\tce->ce_flags |= CE_SKIP_WORKTREE;\n+\tce->ce_stat_crc = ntoh_l(ondisk->stat_crc);\n+\tce->ce_namelen = len + de->de_pathlen;\n+\thashcpy(ce->sha1, ondisk->sha1);\n+\tmemcpy(ce->name, de->pathname, de->de_pathlen);\n+\tmemcpy(ce->name + de->de_pathlen, name, len);\n+\tce->name[len + de->de_pathlen] = '\\0';\n+\treturn ce;\n+}\n+\n+static struct directory_entry *directory_entry_from_ondisk(struct ondisk_directory_entry *ondisk,\n+\t\t\t\t\t\t const char *name,\n+\t\t\t\t\t\t size_t len)\n+{\n+\tstruct directory_entry *de = xmalloc(directory_entry_size(len));\n+\n+\n+\tmemcpy(de->pathname, name, len);\n+\tde->pathname[len] = '\\0';\n+\tde->de_flags = ntoh_s(ondisk->flags);\n+\tde->de_foffset = ntoh_l(ondisk->foffset);\n+\tde->de_cr = ntoh_l(ondisk->cr);\n+\tde->de_ncr = ntoh_l(ondisk->ncr);\n+\tde->de_nsubtrees = ntoh_l(ondisk->nsubtrees);\n+\tde->de_nfiles = ntoh_l(ondisk->nfiles);\n+\tde->de_nentries = ntoh_l(ondisk->nentries);\n+\tde->de_pathlen = len;\n+\thashcpy(de->sha1, ondisk->sha1);\n+\treturn de;\n+}\n+\n+static struct conflict_part *conflict_part_from_ondisk(struct ondisk_conflict_part *ondisk)\n+{\n+\tstruct conflict_part *cp = xmalloc(sizeof(struct conflict_part));\n+\n+\tcp->flags = ntoh_s(ondisk->flags);\n+\tcp->entry_mode = ntoh_s(ondisk->entry_mode);\n+\thashcpy(cp->sha1, ondisk->sha1);\n+\treturn cp;\n+}\n+\n+static struct cache_entry *convert_conflict_part(struct conflict_part *cp,\n+\t\t\t\t\t\tchar * name,\n+\t\t\t\t\t\tunsigned int len)\n+{\n+\n+\tstruct cache_entry *ce = xmalloc(cache_entry_size(len));\n+\n+\tce->ce_stat_data.sd_ctime.sec = 0;\n+\tce->ce_stat_data.sd_mtime.sec = 0;\n+\tce->ce_stat_data.sd_ctime.nsec = 0;\n+\tce->ce_stat_data.sd_mtime.nsec = 0;\n+\tce->ce_stat_data.sd_dev = 0;\n+\tce->ce_stat_data.sd_ino = 0;\n+\tce->ce_stat_data.sd_uid = 0;\n+\tce->ce_stat_data.sd_gid = 0;\n+\tce->ce_stat_data.sd_size = 0;\n+\tce->ce_mode = cp->entry_mode;\n+\tce->ce_flags = conflict_stage(cp) << CE_STAGESHIFT;\n+\tce->ce_stat_crc = 0;\n+\tce->ce_namelen = len;\n+\thashcpy(ce->sha1, cp->sha1);\n+\tmemcpy(ce->name, name, len);\n+\tce->name[len] = '\\0';\n+\treturn ce;\n+}\n+\n+static struct directory_entry *read_directories(unsigned int *dir_offset,\n+\t\t\t\tunsigned int *dir_table_offset,\n+\t\t\t\tvoid *mmap,\n+\t\t\t\tint mmap_size)\n+{\n+\tint i, ondisk_directory_size;\n+\tuint32_t *filecrc, *beginning, *end;\n+\tstruct directory_entry *current = NULL;\n+\tstruct ondisk_directory_entry *disk_de;\n+\tstruct directory_entry *de;\n+\tunsigned int data_len, len;\n+\tchar *name;\n+\n+\t/*\n+\t * Length of pathname + nul byte for termination + size of\n+\t * members of ondisk_directory_entry. (Just using the size\n+\t * of the struct doesn't work, because there may be padding\n+\t * bytes for the struct)\n+\t */\n+\tondisk_directory_size = sizeof(disk_de->flags)\n+\t\t+ sizeof(disk_de->foffset)\n+\t\t+ sizeof(disk_de->cr)\n+\t\t+ sizeof(disk_de->ncr)\n+\t\t+ sizeof(disk_de->nsubtrees)\n+\t\t+ sizeof(disk_de->nfiles)\n+\t\t+ sizeof(disk_de->nentries)\n+\t\t+ sizeof(disk_de->sha1);\n+\tname = ptr_add(mmap, *dir_offset);\n+\tbeginning = ptr_add(mmap, *dir_table_offset);\n+\tend = ptr_add(mmap, *dir_table_offset + 4);\n+\tlen = ntoh_l(*end) - ntoh_l(*beginning) - ondisk_directory_size - 5;\n+\tdisk_de = ptr_add(mmap, *dir_offset + len + 1);\n+\tde = directory_entry_from_ondisk(disk_de, name, len);\n+\tde->next = NULL;\n+\n+\tdata_len = len + 1 + ondisk_directory_size;\n+\tfilecrc = ptr_add(mmap, *dir_offset + data_len);\n+\tif (!check_crc32(0, ptr_add(mmap, *dir_offset), data_len, ntoh_l(*filecrc)))\n+\t\tgoto unmap;\n+\n+\t*dir_table_offset += 4;\n+\t*dir_offset += data_len + 4; /* crc code */\n+\n+\tcurrent = de;\n+\tfor (i = 0; i < de->de_nsubtrees; i++) {\n+\t\tcurrent->next = read_directories(dir_offset, dir_table_offset,\n+\t\t\t\t\t\tmmap, mmap_size);\n+\t\twhile (current->next)\n+\t\t\tcurrent = current->next;\n+\t}\n+\n+\treturn de;\n+unmap:\n+\tmunmap(mmap, mmap_size);\n+\tdie(\"directory crc doesn't match for '%s'\", de->pathname);\n+}\n+\n+static int read_entry(struct cache_entry **ce, struct directory_entry *de,\n+\t\t unsigned int *entry_offset,\n+\t\t void **mmap, unsigned long mmap_size,\n+\t\t unsigned int *foffsetblock)\n+{\n+\tint len, offset_to_offset;\n+\tchar *name;\n+\tuint32_t foffsetblockcrc;\n+\tuint32_t *filecrc, *beginning, *end;\n+\tstruct ondisk_cache_entry *disk_ce;\n+\n+\tname = ptr_add(*mmap, *entry_offset);\n+\tbeginning = ptr_add(*mmap, *foffsetblock);\n+\tend = ptr_add(*mmap, *foffsetblock + 4);\n+\tlen = ntoh_l(*end) - ntoh_l(*beginning) - sizeof(struct ondisk_cache_entry) - 5;\n+\tdisk_ce = ptr_add(*mmap, *entry_offset + len + 1);\n+\t*ce = cache_entry_from_ondisk(disk_ce, de, name, len, de->de_pathlen);\n+\tfilecrc = ptr_add(*mmap, *entry_offset + len + 1 + sizeof(*disk_ce));\n+\toffset_to_offset = htonl(*foffsetblock);\n+\tfoffsetblockcrc = crc32(0, (Bytef*)&offset_to_offset, 4);\n+\tif (!check_crc32(foffsetblockcrc,\n+\t\tptr_add(*mmap, *entry_offset), len + 1 + sizeof(*disk_ce),\n+\t\tntoh_l(*filecrc)))\n+\t\treturn -1;\n+\n+\t*entry_offset += len + 1 + sizeof(*disk_ce) + 4;\n+\treturn 0;\n+}\n+\n+static void ce_queue_push(struct cache_entry **head,\n+\t\t\t struct cache_entry **tail,\n+\t\t\t struct cache_entry *ce)\n+{\n+\tif (!*head) {\n+\t\t*head = *tail = ce;\n+\t\t(*tail)->next = NULL;\n+\t\treturn;\n+\t}\n+\n+\t(*tail)->next = ce;\n+\tce->next = NULL;\n+\t*tail = (*tail)->next;\n+}\n+\n+static void conflict_entry_push(struct conflict_entry **head,\n+\t\t\t\tstruct conflict_entry **tail,\n+\t\t\t\tstruct conflict_entry *conflict_entry)\n+{\n+\tif (!*head) {\n+\t\t*head = *tail = conflict_entry;\n+\t\t(*tail)->next = NULL;\n+\t\treturn;\n+\t}\n+\n+\t(*tail)->next = conflict_entry;\n+\tconflict_entry->next = NULL;\n+\t*tail = (*tail)->next;\n+}\n+\n+static struct cache_entry *ce_queue_pop(struct cache_entry **head)\n+{\n+\tstruct cache_entry *ce;\n+\n+\tce = *head;\n+\t*head = (*head)->next;\n+\treturn ce;\n+}\n+\n+static void conflict_part_head_remove(struct conflict_part **head)\n+{\n+\tstruct conflict_part *to_free;\n+\n+\tto_free = *head;\n+\t*head = (*head)->next;\n+\tfree(to_free);\n+}\n+\n+static void conflict_entry_head_remove(struct conflict_entry **head)\n+{\n+\tstruct conflict_entry *to_free;\n+\n+\tto_free = *head;\n+\t*head = (*head)->next;\n+\tfree(to_free);\n+}\n+\n+struct conflict_entry *create_new_conflict(char *name, int len, int pathlen)\n+{\n+\tstruct conflict_entry *conflict_entry;\n+\n+\tif (pathlen)\n+\t\tpathlen++;\n+\tconflict_entry = xmalloc(conflict_entry_size(len));\n+\tconflict_entry->entries = NULL;\n+\tconflict_entry->nfileconflicts = 0;\n+\tconflict_entry->namelen = len;\n+\tmemcpy(conflict_entry->name, name, len);\n+\tconflict_entry->name[len] = '\\0';\n+\tconflict_entry->pathlen = pathlen;\n+\tconflict_entry->next = NULL;\n+\n+\treturn conflict_entry;\n+}\n+\n+void add_part_to_conflict_entry(struct directory_entry *de,\n+\t\t\t\t\tstruct conflict_entry *entry,\n+\t\t\t\t\tstruct conflict_part *conflict_part)\n+{\n+\n+\tstruct conflict_part *conflict_search;\n+\n+\tentry->nfileconflicts++;\n+\tde->conflict_size += sizeof(struct ondisk_conflict_part);\n+\tif (!entry->entries)\n+\t\tentry->entries = conflict_part;\n+\telse {\n+\t\tconflict_search = entry->entries;\n+\t\twhile (conflict_search->next)\n+\t\t\tconflict_search = conflict_search->next;\n+\t\tconflict_search->next = conflict_part;\n+\t}\n+}\n+\n+static int read_conflicts(struct conflict_entry **head,\n+\t\t\t struct directory_entry *de,\n+\t\t\t void **mmap, unsigned long mmap_size)\n+{\n+\tstruct conflict_entry *tail;\n+\tunsigned int croffset, i;\n+\tchar *full_name;\n+\n+\tcroffset = de->de_cr;\n+\ttail = NULL;\n+\tfor (i = 0; i < de->de_ncr; i++) {\n+\t\tstruct conflict_entry *conflict_new;\n+\t\tunsigned int len, *nfileconflicts;\n+\t\tchar *name;\n+\t\tvoid *crc_start;\n+\t\tint k, offset;\n+\t\tuint32_t *filecrc;\n+\n+\t\toffset = croffset;\n+\t\tcrc_start = ptr_add(*mmap, offset);\n+\t\tname = ptr_add(*mmap, offset);\n+\t\tlen = strlen(name);\n+\t\toffset += len + 1;\n+\t\tnfileconflicts = ptr_add(*mmap, offset);\n+\t\toffset += 4;\n+\n+\t\tfull_name = xmalloc(sizeof(char) * (len + de->de_pathlen));\n+\t\tmemcpy(full_name, de->pathname, de->de_pathlen);\n+\t\tmemcpy(full_name + de->de_pathlen, name, len);\n+\t\tconflict_new = create_new_conflict(full_name,\n+\t\t\t\tlen + de->de_pathlen, de->de_pathlen);\n+\t\tfor (k = 0; k < ntoh_l(*nfileconflicts); k++) {\n+\t\t\tstruct ondisk_conflict_part *ondisk;\n+\t\t\tstruct conflict_part *cp;\n+\n+\t\t\tondisk = ptr_add(*mmap, offset);\n+\t\t\tcp = conflict_part_from_ondisk(ondisk);\n+\t\t\tcp->next = NULL;\n+\t\t\tadd_part_to_conflict_entry(de, conflict_new, cp);\n+\t\t\toffset += sizeof(struct ondisk_conflict_part);\n+\t\t}\n+\t\tfilecrc = ptr_add(*mmap, offset);\n+\t\tfree(full_name);\n+\t\tif (!check_crc32(0, crc_start,\n+\t\t\tlen + 1 + 4 + conflict_new->nfileconflicts\n+\t\t\t* sizeof(struct ondisk_conflict_part),\n+\t\t\tntoh_l(*filecrc)))\n+\t\t\treturn -1;\n+\t\tcroffset = offset + 4;\n+\t\tconflict_entry_push(head, &tail, conflict_new);\n+\t}\n+\treturn 0;\n+}\n+\n+static int read_entries(struct index_state *istate, struct directory_entry **de,\n+\t\t\tunsigned int *entry_offset, void **mmap,\n+\t\t\tunsigned long mmap_size, unsigned int *nr,\n+\t\t\tunsigned int *foffsetblock)\n+{\n+\tstruct cache_entry *head = NULL, *tail = NULL;\n+\tstruct conflict_entry *conflict_queue;\n+\tstruct cache_entry *ce;\n+\tint i;\n+\n+\tconflict_queue = NULL;\n+\tif (read_conflicts(&conflict_queue, *de, mmap, mmap_size) < 0)\n+\t\treturn -1;\n+\tfor (i = 0; i < (*de)->de_nfiles; i++) {\n+\t\tif (read_entry(&ce,\n+\t\t\t *de,\n+\t\t\t entry_offset,\n+\t\t\t mmap,\n+\t\t\t mmap_size,\n+\t\t\t foffsetblock) < 0)\n+\t\t\treturn -1;\n+\t\tce_queue_push(&head, &tail, ce);\n+\t\t*foffsetblock += 4;\n+\n+\t\t/*\n+\t\t * Add the conflicted entries at the end of the index file\n+\t\t * to the in memory format\n+\t\t */\n+\t\tif (conflict_queue &&\n+\t\t (conflict_queue->entries->flags & CONFLICT_CONFLICTED) != 0 &&\n+\t\t !cache_name_compare(conflict_queue->name, conflict_queue->namelen,\n+\t\t\t\t\tce->name, ce_namelen(ce))) {\n+\t\t\tstruct conflict_part *cp;\n+\t\t\tcp = conflict_queue->entries;\n+\t\t\tcp = cp->next;\n+\t\t\twhile (cp) {\n+\t\t\t\tce = convert_conflict_part(cp,\n+\t\t\t\t\t\tconflict_queue->name,\n+\t\t\t\t\t\tconflict_queue->namelen);\n+\t\t\t\tce_queue_push(&head, &tail, ce);\n+\t\t\t\tconflict_part_head_remove(&cp);\n+\t\t\t}\n+\t\t\tconflict_entry_head_remove(&conflict_queue);\n+\t\t}\n+\t}\n+\n+\t*de = (*de)->next;\n+\n+\twhile (head) {\n+\t\tif (*de != NULL\n+\t\t && strcmp(head->name, (*de)->pathname) > 0) {\n+\t\t\tread_entries(istate,\n+\t\t\t\t de,\n+\t\t\t\t entry_offset,\n+\t\t\t\t mmap,\n+\t\t\t\t mmap_size,\n+\t\t\t\t nr,\n+\t\t\t\t foffsetblock);\n+\t\t} else {\n+\t\t\tce = ce_queue_pop(&head);\n+\t\t\tset_index_entry(istate, *nr, ce);\n+\t\t\t(*nr)++;\n+\t\t\tce->next = NULL;\n+\t\t}\n+\t}\n+\treturn 0;\n+}\n+\n+static struct directory_entry *read_head_directories(struct index_state *istate,\n+\t\t\t\t\t\t unsigned int *entry_offset,\n+\t\t\t\t\t\t unsigned int *foffsetblock,\n+\t\t\t\t\t\t unsigned int *ndirs,\n+\t\t\t\t\t\t void *mmap, unsigned long mmap_size)\n+{\n+\tunsigned int dir_offset, dir_table_offset;\n+\tstruct cache_version_header *hdr;\n+\tstruct cache_header *hdr_v5;\n+\tstruct directory_entry *root_directory;\n+\n+\thdr = mmap;\n+\thdr_v5 = ptr_add(mmap, sizeof(*hdr));\n+\tistate->version = ntohl(hdr->hdr_version);\n+\tistate->cache_alloc = alloc_nr(ntohl(hdr_v5->hdr_nfile));\n+\tistate->cache = xcalloc(istate->cache_alloc, sizeof(struct cache_entry *));\n+\tistate->initialized = 1;\n+\n+\t/* Skip size of the header + crc sum + size of offsets */\n+\tdir_offset = sizeof(*hdr) + sizeof(*hdr_v5) + 4 + (ntohl(hdr_v5->hdr_ndir) + 1) * 4;\n+\tdir_table_offset = sizeof(*hdr) + sizeof(*hdr_v5) + 4;\n+\troot_directory = read_directories(&dir_offset, &dir_table_offset,\n+\t\t\t\t\t mmap, mmap_size);\n+\n+\t*entry_offset = ntohl(hdr_v5->hdr_fblockoffset);\n+\t*foffsetblock = dir_offset;\n+\t*ndirs = ntohl(hdr_v5->hdr_ndir);\n+\treturn root_directory;\n+}\n+\n+static int read_index_filtered_v5(struct index_state *istate, void *mmap,\n+\t\t\t\t unsigned long mmap_size, struct filter_opts *opts)\n+{\n+\tunsigned int entry_offset, ndirs, foffsetblock, nr = 0;\n+\tstruct directory_entry *root_directory, *de;\n+\tconst char **adjusted_pathspec = NULL;\n+\tint need_root = 0, i, n;\n+\tchar *oldpath, *seen;\n+\n+\troot_directory = read_head_directories(istate, &entry_offset,\n+\t\t\t\t\t &foffsetblock, &ndirs,\n+\t\t\t\t\t mmap, mmap_size);\n+\n+\tif (opts->pathspec->raw) {\n+\t\tneed_root = 0;\n+\t\tseen = xcalloc(1, ndirs);\n+\t\tfor (de = root_directory; de; de = de->next)\n+\t\t\tmatch_pathspec(opts->pathspec->raw, de->pathname, de->de_pathlen, 0, seen);\n+\t\tfor (n = 0; opts->pathspec->raw[n]; n++)\n+\t\t\t/* just count */;\n+\t\tadjusted_pathspec = xmalloc((n+1)*sizeof(char *));\n+\t\tadjusted_pathspec[n] = NULL;\n+\t\tfor (i = 0; i < n; i++) {\n+\t\t\tif (seen[i] == MATCHED_EXACTLY)\n+\t\t\t\tadjusted_pathspec[i] = opts->pathspec->raw[i];\n+\t\t\telse {\n+\t\t\t\tchar *super = strdup(opts->pathspec->raw[i]);\n+\t\t\t\tint len = strlen(super);\n+\t\t\t\twhile (len && super[len - 1] == '/')\n+\t\t\t\t\tsuper[--len] = '\\0'; /* strip trailing / */\n+\t\t\t\twhile (len && super[--len] != '/')\n+\t\t\t\t\t; /* scan backwards to next / */\n+\t\t\t\tif (len >= 0)\n+\t\t\t\t\tsuper[len--] = '\\0';\n+\t\t\t\tif (len <= 0) {\n+\t\t\t\t\tneed_root = 1;\n+\t\t\t\t\tbreak;\n+\t\t\t\t}\n+\t\t\t\tadjusted_pathspec[i] = super;\n+\t\t\t}\n+\t\t}\n+\t}\n+\n+\tde = root_directory;\n+\twhile (de) {\n+\t\tif (need_root ||\n+\t\t match_pathspec(adjusted_pathspec, de->pathname, de->de_pathlen, 0, NULL)) {\n+\t\t\tunsigned int subdir_foffsetblock = de->de_foffset + foffsetblock;\n+\t\t\tunsigned int *off = mmap + subdir_foffsetblock;\n+\t\t\tunsigned int subdir_entry_offset = entry_offset + ntoh_l(*off);\n+\t\t\toldpath = de->pathname;\n+\t\t\tdo {\n+\t\t\t\tif (read_entries(istate, &de, &subdir_entry_offset,\n+\t\t\t\t\t\t &mmap, mmap_size, &nr,\n+\t\t\t\t\t\t &subdir_foffsetblock) < 0)\n+\t\t\t\t\treturn -1;\n+\t\t\t} while (de && !prefixcmp(de->pathname, oldpath));\n+\t\t} else\n+\t\t\tde = de->next;\n+\t}\n+\tistate->cache_nr = nr;\n+\treturn 0;\n+}\n+\n+static int read_index_v5(struct index_state *istate, void *mmap,\n+\t\t\t unsigned long mmap_size, struct filter_opts *opts)\n+{\n+\tunsigned int entry_offset, ndirs, foffsetblock, nr = 0;\n+\tstruct directory_entry *root_directory, *de;\n+\n+\tif (opts != NULL)\n+\t\treturn read_index_filtered_v5(istate, mmap, mmap_size, opts);\n+\n+\troot_directory = read_head_directories(istate, &entry_offset,\n+\t\t\t\t\t &foffsetblock, &ndirs,\n+\t\t\t\t\t mmap, mmap_size);\n+\tde = root_directory;\n+\twhile (de)\n+\t\tif (read_entries(istate, &de, &entry_offset, &mmap,\n+\t\t\t\t mmap_size, &nr, &foffsetblock) < 0)\n+\t\t\treturn -1;\n+\tistate->cache_nr = nr;\n+\treturn 0;\n+}\n+\n+struct index_ops v5_ops = {\n+\tmatch_stat_basic,\n+\tverify_hdr,\n+\tread_index_v5,\n+\tNULL\n+};\ndiff --git a/read-cache.h b/read-cache.h\nindex e72a585..869c562 100644\n--- a/read-cache.h\n+++ b/read-cache.h\n@@ -35,6 +35,7 @@ struct internal_ops {\n \n extern struct index_ops v2_ops;\n extern struct internal_ops v2_internal_ops;\n+extern struct index_ops v5_ops;\n \n #ifndef NEEDS_ALIGNED_ACCESS\n #define ntoh_s(var) ntohs(var)\n-- \n1.8.3.453.g1dfc63d\n"},{"id":"223189","messageId":"1373650024-3001-14-git-send-email-t.gummerer@gmail.com","threadId":"34419","inReplyTo":"1373650024-3001-1-git-send-email-t.gummerer@gmail.com","subject":"[PATCH v2 13/19] read-cache: read resolve-undo data","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2013-07-12T17:26:58Z","receivedAt":"2013-07-12T17:26:58Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"Make git read the resolve-undo data from the index.\n\nSince the resolve-undo data is joined with the conflicts in\nthe ondisk format of the index file version 5, conflicts and\nresolved data is read at the same time, and the resolve-undo\ndata is then converted to the in-memory format.\n\nHelped-by: Thomas Rast \nSigned-off-by: Thomas Gummerer \n---\n read-cache-v5.c | 39 +++++++++++++++++++++++++++++++++++++++\n 1 file changed, 39 insertions(+)\n\ndiff --git a/read-cache-v5.c b/read-cache-v5.c\nindex 00112ea..853b97d 100644\n--- a/read-cache-v5.c\n+++ b/read-cache-v5.c\n@@ -1,5 +1,6 @@\n #include \"cache.h\"\n #include \"read-cache.h\"\n+#include \"string-list.h\"\n #include \"resolve-undo.h\"\n #include \"cache-tree.h\"\n #include \"dir.h\"\n@@ -447,6 +448,43 @@ static int read_conflicts(struct conflict_entry **head,\n \treturn 0;\n }\n \n+static void resolve_undo_convert_v5(struct index_state *istate,\n+\t\t\t\t struct conflict_entry *conflict)\n+{\n+\tint i;\n+\n+\twhile (conflict) {\n+\t\tstruct string_list_item *lost;\n+\t\tstruct resolve_undo_info *ui;\n+\t\tstruct conflict_part *cp;\n+\n+\t\tif (conflict->entries &&\n+\t\t (conflict->entries->flags & CONFLICT_CONFLICTED) != 0) {\n+\t\t\tconflict = conflict->next;\n+\t\t\tcontinue;\n+\t\t}\n+\t\tif (!istate->resolve_undo) {\n+\t\t\tistate->resolve_undo = xcalloc(1, sizeof(struct string_list));\n+\t\t\tistate->resolve_undo->strdup_strings = 1;\n+\t\t}\n+\n+\t\tlost = string_list_insert(istate->resolve_undo, conflict->name);\n+\t\tif (!lost->util)\n+\t\t\tlost->util = xcalloc(1, sizeof(*ui));\n+\t\tui = lost->util;\n+\n+\t\tcp = conflict->entries;\n+\t\tfor (i = 0; i < 3; i++)\n+\t\t\tui->mode[i] = 0;\n+\t\twhile (cp) {\n+\t\t\tui->mode[conflict_stage(cp) - 1] = cp->entry_mode;\n+\t\t\thashcpy(ui->sha1[conflict_stage(cp) - 1], cp->sha1);\n+\t\t\tcp = cp->next;\n+\t\t}\n+\t\tconflict = conflict->next;\n+\t}\n+}\n+\n static int read_entries(struct index_state *istate, struct directory_entry **de,\n \t\t\tunsigned int *entry_offset, void **mmap,\n \t\t\tunsigned long mmap_size, unsigned int *nr,\n@@ -460,6 +498,7 @@ static int read_entries(struct index_state *istate, struct directory_entry **de,\n \tconflict_queue = NULL;\n \tif (read_conflicts(&conflict_queue, *de, mmap, mmap_size) < 0)\n \t\treturn -1;\n+\tresolve_undo_convert_v5(istate, conflict_queue);\n \tfor (i = 0; i < (*de)->de_nfiles; i++) {\n \t\tif (read_entry(&ce,\n \t\t\t *de,\n-- \n1.8.3.453.g1dfc63d\n"},{"id":"223190","messageId":"1373650024-3001-15-git-send-email-t.gummerer@gmail.com","threadId":"34419","inReplyTo":"1373650024-3001-1-git-send-email-t.gummerer@gmail.com","subject":"[PATCH v2 14/19] read-cache: read cache-tree in index-v5","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2013-07-12T17:26:59Z","receivedAt":"2013-07-12T17:26:59Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"Since the cache-tree data is saved as part of the directory data,\nwe already read it at the beginning of the index. The cache-tree\nis only converted from this directory data.\n\nThe cache-tree data is arranged in a tree, with the children sorted by\npathlen at each node, while the ondisk format is sorted lexically.\nSo we have to rebuild this format from the on-disk directory list.\n\nSigned-off-by: Thomas Gummerer \n---\n cache-tree.c | 2 +-\n cache-tree.h | 6 ++++\n read-cache-v5.c | 100 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n 3 files changed, 107 insertions(+), 1 deletion(-)\n\ndiff --git a/cache-tree.c b/cache-tree.c\nindex 37e4d00..f4b0917 100644\n--- a/cache-tree.c\n+++ b/cache-tree.c\n@@ -31,7 +31,7 @@ void cache_tree_free(struct cache_tree **it_p)\n \t*it_p = NULL;\n }\n \n-static int subtree_name_cmp(const char *one, int onelen,\n+int subtree_name_cmp(const char *one, int onelen,\n \t\t\t const char *two, int twolen)\n {\n \tif (onelen < twolen)\ndiff --git a/cache-tree.h b/cache-tree.h\nindex 55d0f59..9aac493 100644\n--- a/cache-tree.h\n+++ b/cache-tree.h\n@@ -21,10 +21,16 @@ struct cache_tree {\n \tstruct cache_tree_sub **down;\n };\n \n+struct directory_queue {\n+\tstruct directory_queue *down;\n+\tstruct directory_entry *de;\n+};\n+\n struct cache_tree *cache_tree(void);\n void cache_tree_free(struct cache_tree **);\n void cache_tree_invalidate_path(struct cache_tree *, const char *);\n struct cache_tree_sub *cache_tree_sub(struct cache_tree *, const char *);\n+int subtree_name_cmp(const char *, int, const char *, int);\n \n void cache_tree_write(struct strbuf *, struct cache_tree *root);\n struct cache_tree *cache_tree_read(const char *buffer, unsigned long size);\ndiff --git a/read-cache-v5.c b/read-cache-v5.c\nindex 853b97d..0b9c320 100644\n--- a/read-cache-v5.c\n+++ b/read-cache-v5.c\n@@ -448,6 +448,103 @@ static int read_conflicts(struct conflict_entry **head,\n \treturn 0;\n }\n \n+static struct cache_tree *convert_one(struct directory_queue *queue, int dirnr)\n+{\n+\tint i, subtree_nr;\n+\tstruct cache_tree *it;\n+\tstruct directory_queue *down;\n+\n+\tit = cache_tree();\n+\tit->entry_count = queue[dirnr].de->de_nentries;\n+\tsubtree_nr = queue[dirnr].de->de_nsubtrees;\n+\tif (0 <= it->entry_count)\n+\t\thashcpy(it->sha1, queue[dirnr].de->sha1);\n+\n+\t/*\n+\t * Just a heuristic -- we do not add directories that often but\n+\t * we do not want to have to extend it immediately when we do,\n+\t * hence +2.\n+\t */\n+\tit->subtree_alloc = subtree_nr + 2;\n+\tit->down = xcalloc(it->subtree_alloc, sizeof(struct cache_tree_sub *));\n+\tdown = queue[dirnr].down;\n+\tfor (i = 0; i < subtree_nr; i++) {\n+\t\tstruct cache_tree *sub;\n+\t\tstruct cache_tree_sub *subtree;\n+\t\tchar *buf, *name;\n+\n+\t\tname = \"\";\n+\t\tbuf = strtok(down[i].de->pathname, \"/\");\n+\t\twhile (buf) {\n+\t\t\tname = buf;\n+\t\t\tbuf = strtok(NULL, \"/\");\n+\t\t}\n+\t\tsub = convert_one(down, i);\n+\t\tif(!sub)\n+\t\t\tgoto free_return;\n+\t\tsubtree = cache_tree_sub(it, name);\n+\t\tsubtree->cache_tree = sub;\n+\t}\n+\tif (subtree_nr != it->subtree_nr)\n+\t\tdie(\"cache-tree: internal error\");\n+\treturn it;\n+ free_return:\n+\tcache_tree_free(&it);\n+\treturn NULL;\n+}\n+\n+static int compare_cache_tree_elements(const void *a, const void *b)\n+{\n+\tconst struct directory_entry *de1, *de2;\n+\n+\tde1 = ((const struct directory_queue *)a)->de;\n+\tde2 = ((const struct directory_queue *)b)->de;\n+\treturn subtree_name_cmp(de1->pathname, de1->de_pathlen,\n+\t\t\t\tde2->pathname, de2->de_pathlen);\n+}\n+\n+static struct directory_entry *sort_directories(struct directory_entry *de,\n+\t\t\t\t\t\tstruct directory_queue *queue)\n+{\n+\tint i, nsubtrees;\n+\n+\tnsubtrees = de->de_nsubtrees;\n+\tfor (i = 0; i < nsubtrees; i++) {\n+\t\tstruct directory_entry *new_de;\n+\t\tde = de->next;\n+\t\tnew_de = xmalloc(directory_entry_size(de->de_pathlen));\n+\t\tmemcpy(new_de, de, directory_entry_size(de->de_pathlen));\n+\t\tqueue[i].de = new_de;\n+\t\tif (de->de_nsubtrees) {\n+\t\t\tqueue[i].down = xcalloc(de->de_nsubtrees,\n+\t\t\t\t\tsizeof(struct directory_queue));\n+\t\t\tde = sort_directories(de,\n+\t\t\t\t\tqueue[i].down);\n+\t\t}\n+\t}\n+\tqsort(queue, nsubtrees, sizeof(struct directory_queue),\n+\t\t\tcompare_cache_tree_elements);\n+\treturn de;\n+}\n+\n+/*\n+ * This function modifies the directory argument that is given to it.\n+ * Don't use it if the directory entries are still needed after.\n+ */\n+static struct cache_tree *cache_tree_convert_v5(struct directory_entry *de)\n+{\n+\tstruct directory_queue *queue;\n+\n+\tif (!de->de_nentries)\n+\t\treturn NULL;\n+\tqueue = xcalloc(1, sizeof(struct directory_queue));\n+\tqueue[0].de = de;\n+\tqueue[0].down = xcalloc(de->de_nsubtrees, sizeof(struct directory_queue));\n+\n+\tsort_directories(de, queue[0].down);\n+\treturn convert_one(queue, 0);\n+}\n+\n static void resolve_undo_convert_v5(struct index_state *istate,\n \t\t\t\t struct conflict_entry *conflict)\n {\n@@ -644,6 +741,7 @@ static int read_index_filtered_v5(struct index_state *istate, void *mmap,\n \t\t} else\n \t\t\tde = de->next;\n \t}\n+\tistate->cache_tree = cache_tree_convert_v5(root_directory);\n \tistate->cache_nr = nr;\n \treturn 0;\n }\n@@ -665,6 +763,8 @@ static int read_index_v5(struct index_state *istate, void *mmap,\n \t\tif (read_entries(istate, &de, &entry_offset, &mmap,\n \t\t\t\t mmap_size, &nr, &foffsetblock) < 0)\n \t\t\treturn -1;\n+\n+\tistate->cache_tree = cache_tree_convert_v5(root_directory);\n \tistate->cache_nr = nr;\n \treturn 0;\n }\n-- \n1.8.3.453.g1dfc63d\n"},{"id":"223191","messageId":"1373650024-3001-16-git-send-email-t.gummerer@gmail.com","threadId":"34419","inReplyTo":"1373650024-3001-1-git-send-email-t.gummerer@gmail.com","subject":"[PATCH v2 15/19] read-cache: write index-v5","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2013-07-12T17:27:00Z","receivedAt":"2013-07-12T17:27:00Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"Write the index version 5 file format to disk. This version doesn't\nwrite the cache-tree data and resolve-undo data to the file.\n\nThe main work is done when filtering out the directories from the\ncurrent in-memory format, where in the same turn also the conflicts\nand the file data is calculated.\n\nHelped-by: Nguyen Thai Ngoc Duy \nHelped-by: Thomas Rast \nSigned-off-by: Thomas Gummerer \n---\n cache.h | 8 +\n read-cache-v5.c | 594 +++++++++++++++++++++++++++++++++++++++++++++++++++++++-\n read-cache.c | 11 +-\n read-cache.h | 1 +\n 4 files changed, 611 insertions(+), 3 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 1e5cc77..f3685f8 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -565,6 +565,7 @@ extern int unmerged_index(const struct index_state *);\n extern int verify_path(const char *path);\n extern struct cache_entry *index_name_exists(struct index_state *istate, const char *name, int namelen, int igncase);\n extern int index_name_pos(const struct index_state *, const char *name, int namelen);\n+extern struct directory_entry *init_directory_entry(char *pathname, int len);\n #define ADD_CACHE_OK_TO_ADD 1\t\t/* Ok to add */\n #define ADD_CACHE_OK_TO_REPLACE 2\t/* Ok to replace file/directory */\n #define ADD_CACHE_SKIP_DFCHECK 4\t/* Ok to skip DF conflict checks */\n@@ -1363,6 +1364,13 @@ static inline ssize_t write_str_in_full(int fd, const char *str)\n \treturn write_in_full(fd, str, strlen(str));\n }\n \n+/* index-v5 helper functions */\n+extern char *super_directory(const char *filename);\n+extern void insert_directory_entry(struct directory_entry *, struct hash_table *, int *, unsigned int *, uint32_t);\n+extern void add_conflict_to_directory_entry(struct directory_entry *, struct conflict_entry *);\n+extern void add_part_to_conflict_entry(struct directory_entry *, struct conflict_entry *, struct conflict_part *);\n+extern struct conflict_entry *create_new_conflict(char *, int, int);\n+\n /* pager.c */\n extern void setup_pager(void);\n extern const char *pager_program;\ndiff --git a/read-cache-v5.c b/read-cache-v5.c\nindex 0b9c320..33667d7 100644\n--- a/read-cache-v5.c\n+++ b/read-cache-v5.c\n@@ -769,9 +769,601 @@ static int read_index_v5(struct index_state *istate, void *mmap,\n \treturn 0;\n }\n \n+#define WRITE_BUFFER_SIZE 8192\n+static unsigned char write_buffer[WRITE_BUFFER_SIZE];\n+static unsigned long write_buffer_len;\n+\n+static int ce_write_flush(int fd)\n+{\n+\tunsigned int buffered = write_buffer_len;\n+\tif (buffered) {\n+\t\tif (write_in_full(fd, write_buffer, buffered) != buffered)\n+\t\t\treturn -1;\n+\t\twrite_buffer_len = 0;\n+\t}\n+\treturn 0;\n+}\n+\n+static int ce_write(uint32_t *crc, int fd, void *data, unsigned int len)\n+{\n+\tif (crc)\n+\t\t*crc = crc32(*crc, (Bytef*)data, len);\n+\twhile (len) {\n+\t\tunsigned int buffered = write_buffer_len;\n+\t\tunsigned int partial = WRITE_BUFFER_SIZE - buffered;\n+\t\tif (partial > len)\n+\t\t\tpartial = len;\n+\t\tmemcpy(write_buffer + buffered, data, partial);\n+\t\tbuffered += partial;\n+\t\tif (buffered == WRITE_BUFFER_SIZE) {\n+\t\t\twrite_buffer_len = buffered;\n+\t\t\tif (ce_write_flush(fd))\n+\t\t\t\treturn -1;\n+\t\t\tbuffered = 0;\n+\t\t}\n+\t\twrite_buffer_len = buffered;\n+\t\tlen -= partial;\n+\t\tdata = (char *) data + partial;\n+\t}\n+\treturn 0;\n+}\n+\n+static int ce_flush(int fd)\n+{\n+\tunsigned int left = write_buffer_len;\n+\n+\tif (left)\n+\t\twrite_buffer_len = 0;\n+\n+\tif (write_in_full(fd, write_buffer, left) != left)\n+\t\treturn -1;\n+\n+\treturn 0;\n+}\n+\n+static void ce_smudge_racily_clean_entry(struct cache_entry *ce)\n+{\n+\t/*\n+\t * This method shall only be called if the timestamp of ce\n+\t * is racy (check with is_racy_timestamp). If the timestamp\n+\t * is racy, the writer will set the CE_SMUDGED flag.\n+\t *\n+\t * The reader (match_stat_basic) will then take care\n+\t * of checking if the entry is really changed or not, by\n+\t * taking into account the size and the stat_crc and if\n+\t * that hasn't changed checking the sha1.\n+\t */\n+\tce->ce_flags |= CE_SMUDGED;\n+}\n+\n+char *super_directory(const char *filename)\n+{\n+\tchar *slash;\n+\n+\tslash = strrchr(filename, '/');\n+\tif (slash)\n+\t\treturn xmemdupz(filename, slash-filename);\n+\treturn NULL;\n+}\n+\n+struct directory_entry *init_directory_entry(char *pathname, int len)\n+{\n+\tstruct directory_entry *de = xmalloc(directory_entry_size(len));\n+\n+\tmemcpy(de->pathname, pathname, len);\n+\tde->pathname[len] = '\\0';\n+\tde->de_flags = 0;\n+\tde->de_foffset = 0;\n+\tde->de_cr = 0;\n+\tde->de_ncr = 0;\n+\tde->de_nsubtrees = 0;\n+\tde->de_nfiles = 0;\n+\tde->de_nentries = 0;\n+\tmemset(de->sha1, 0, 20);\n+\tde->de_pathlen = len;\n+\tde->next = NULL;\n+\tde->next_hash = NULL;\n+\tde->ce = NULL;\n+\tde->ce_last = NULL;\n+\tde->conflict = NULL;\n+\tde->conflict_last = NULL;\n+\tde->conflict_size = 0;\n+\treturn de;\n+}\n+\n+static void ondisk_from_directory_entry(struct directory_entry *de,\n+\t\t\t\t\tstruct ondisk_directory_entry *ondisk)\n+{\n+\tondisk->foffset = htonl(de->de_foffset);\n+\tondisk->cr = htonl(de->de_cr);\n+\tondisk->ncr = htonl(de->de_ncr);\n+\tondisk->nsubtrees = htonl(de->de_nsubtrees);\n+\tondisk->nfiles = htonl(de->de_nfiles);\n+\tondisk->nentries = htonl(de->de_nentries);\n+\thashcpy(ondisk->sha1, de->sha1);\n+\tondisk->flags = htons(de->de_flags);\n+}\n+\n+static struct conflict_part *conflict_part_from_inmemory(struct cache_entry *ce)\n+{\n+\tstruct conflict_part *conflict;\n+\tint flags;\n+\n+\tconflict = xmalloc(sizeof(struct conflict_part));\n+\tflags = CONFLICT_CONFLICTED;\n+\tflags |= ce_stage(ce) << CONFLICT_STAGESHIFT;\n+\tconflict->flags = flags;\n+\tconflict->entry_mode = ce->ce_mode;\n+\tconflict->next = NULL;\n+\thashcpy(conflict->sha1, ce->sha1);\n+\treturn conflict;\n+}\n+\n+static void conflict_to_ondisk(struct conflict_part *cp,\n+\t\t\t\tstruct ondisk_conflict_part *ondisk)\n+{\n+\tondisk->flags = htons(cp->flags);\n+\tondisk->entry_mode = htons(cp->entry_mode);\n+\thashcpy(ondisk->sha1, cp->sha1);\n+}\n+\n+void add_conflict_to_directory_entry(struct directory_entry *de,\n+\t\t\t\t\tstruct conflict_entry *conflict_entry)\n+{\n+\tde->de_ncr++;\n+\tde->conflict_size += conflict_entry->namelen + 1 + 8 - conflict_entry->pathlen;\n+\tconflict_entry_push(&de->conflict, &de->conflict_last, conflict_entry);\n+}\n+\n+void insert_directory_entry(struct directory_entry *de,\n+\t\t\tstruct hash_table *table,\n+\t\t\tint *total_dir_len,\n+\t\t\tunsigned int *ndir,\n+\t\t\tuint32_t crc)\n+{\n+\tstruct directory_entry *insert;\n+\n+\tinsert = (struct directory_entry *)insert_hash(crc, de, table);\n+\tif (insert) {\n+\t\tde->next_hash = insert->next_hash;\n+\t\tinsert->next_hash = de;\n+\t}\n+\t(*ndir)++;\n+\tif (de->de_pathlen == 0)\n+\t\t(*total_dir_len)++;\n+\telse\n+\t\t*total_dir_len += de->de_pathlen + 2;\n+}\n+\n+static struct conflict_entry *create_conflict_entry_from_ce(struct cache_entry *ce,\n+\t\t\t\t\t\t\t\tint pathlen)\n+{\n+\treturn create_new_conflict(ce->name, ce_namelen(ce), pathlen);\n+}\n+\n+static struct directory_entry *compile_directory_data(struct index_state *istate,\n+\t\t\t\t\t\tint nfile,\n+\t\t\t\t\t\tunsigned int *ndir,\n+\t\t\t\t\t\tint *non_conflicted,\n+\t\t\t\t\t\tint *total_dir_len,\n+\t\t\t\t\t\tint *total_file_len)\n+{\n+\tint i, dir_len = -1;\n+\tchar *dir;\n+\tstruct directory_entry *de, *current, *search, *found, *new, *previous_entry;\n+\tstruct cache_entry **cache = istate->cache;\n+\tstruct conflict_entry *conflict_entry;\n+\tstruct hash_table table;\n+\tuint32_t crc;\n+\n+\tinit_hash(&table);\n+\tde = init_directory_entry(\"\", 0);\n+\tcurrent = de;\n+\t*ndir = 1;\n+\t*total_dir_len = 1;\n+\tcrc = crc32(0, (Bytef*)de->pathname, de->de_pathlen);\n+\tinsert_hash(crc, de, &table);\n+\tconflict_entry = NULL;\n+\tfor (i = 0; i < nfile; i++) {\n+\t\tint new_entry;\n+\t\tif (cache[i]->ce_flags & CE_REMOVE)\n+\t\t\tcontinue;\n+\n+\t\tnew_entry = !ce_stage(cache[i]) || !conflict_entry\n+\t\t || cache_name_compare(conflict_entry->name, conflict_entry->namelen,\n+\t\t\t\t\tcache[i]->name, ce_namelen(cache[i]));\n+\t\tif (new_entry)\n+\t\t\t(*non_conflicted)++;\n+\t\tif (dir_len < 0 || strncmp(cache[i]->name, dir, dir_len)\n+\t\t || cache[i]->name[dir_len] != '/'\n+\t\t || strchr(cache[i]->name + dir_len + 1, '/')) {\n+\t\t\tdir = super_directory(cache[i]->name);\n+\t\t\tif (!dir)\n+\t\t\t\tdir_len = 0;\n+\t\t\telse\n+\t\t\t\tdir_len = strlen(dir);\n+\t\t\tcrc = crc32(0, (Bytef*)dir, dir_len);\n+\t\t\tfound = lookup_hash(crc, &table);\n+\t\t\tsearch = found;\n+\t\t\twhile (search && dir_len != 0 && strcmp(dir, search->pathname) != 0)\n+\t\t\t\tsearch = search->next_hash;\n+\t\t}\n+\t\tprevious_entry = current;\n+\t\tif (!search || !found) {\n+\t\t\tnew = init_directory_entry(dir, dir_len);\n+\t\t\tcurrent->next = new;\n+\t\t\tcurrent = current->next;\n+\t\t\tinsert_directory_entry(new, &table, total_dir_len, ndir, crc);\n+\t\t\tsearch = current;\n+\t\t}\n+\t\tif (new_entry) {\n+\t\t\tsearch->de_nfiles++;\n+\t\t\t*total_file_len += ce_namelen(cache[i]) + 1;\n+\t\t\tif (search->de_pathlen)\n+\t\t\t\t*total_file_len -= search->de_pathlen + 1;\n+\t\t\tce_queue_push(&(search->ce), &(search->ce_last), cache[i]);\n+\t\t}\n+\t\tif (ce_stage(cache[i]) > 0) {\n+\t\t\tstruct conflict_part *conflict_part;\n+\t\t\tif (new_entry) {\n+\t\t\t\tconflict_entry = create_conflict_entry_from_ce(cache[i], search->de_pathlen);\n+\t\t\t\tadd_conflict_to_directory_entry(search, conflict_entry);\n+\t\t\t}\n+\t\t\tconflict_part = conflict_part_from_inmemory(cache[i]);\n+\t\t\tadd_part_to_conflict_entry(search, conflict_entry, conflict_part);\n+\t\t}\n+\t\tif (dir && !found) {\n+\t\t\tstruct directory_entry *no_subtrees;\n+\n+\t\t\tno_subtrees = current;\n+\t\t\tdir = super_directory(dir);\n+\t\t\tif (dir)\n+\t\t\t\tdir_len = strlen(dir);\n+\t\t\telse\n+\t\t\t\tdir_len = 0;\n+\t\t\tcrc = crc32(0, (Bytef*)dir, dir_len);\n+\t\t\tfound = lookup_hash(crc, &table);\n+\t\t\twhile (!found) {\n+\t\t\t\tnew = init_directory_entry(dir, dir_len);\n+\t\t\t\tnew->de_nsubtrees = 1;\n+\t\t\t\tnew->next = no_subtrees;\n+\t\t\t\tno_subtrees = new;\n+\t\t\t\tinsert_directory_entry(new, &table, total_dir_len, ndir, crc);\n+\t\t\t\tdir = super_directory(dir);\n+\t\t\t\tif (!dir)\n+\t\t\t\t\tdir_len = 0;\n+\t\t\t\telse\n+\t\t\t\t\tdir_len = strlen(dir);\n+\t\t\t\tcrc = crc32(0, (Bytef*)dir, dir_len);\n+\t\t\t\tfound = lookup_hash(crc, &table);\n+\t\t\t}\n+\t\t\tsearch = found;\n+\t\t\twhile (search->next_hash && strcmp(dir, search->pathname) != 0)\n+\t\t\t\tsearch = search->next_hash;\n+\t\t\tif (search)\n+\t\t\t\tfound = search;\n+\t\t\tfound->de_nsubtrees++;\n+\t\t\tprevious_entry->next = no_subtrees;\n+\t\t}\n+\t}\n+\treturn de;\n+}\n+\n+static void ondisk_from_cache_entry(struct cache_entry *ce,\n+\t\t\t\t struct ondisk_cache_entry *ondisk)\n+{\n+\tunsigned int flags;\n+\n+\tflags = ce->ce_flags & CE_STAGEMASK;\n+\tflags |= ce->ce_flags & CE_VALID;\n+\tflags |= ce->ce_flags & CE_SMUDGED;\n+\tif (ce->ce_flags & CE_INTENT_TO_ADD)\n+\t\tflags |= CE_INTENT_TO_ADD_V5;\n+\tif (ce->ce_flags & CE_SKIP_WORKTREE)\n+\t\tflags |= CE_SKIP_WORKTREE_V5;\n+\tondisk->flags = htons(flags);\n+\tondisk->mode = htons(ce->ce_mode);\n+\tondisk->mtime.sec = htonl(ce->ce_stat_data.sd_mtime.sec);\n+#ifdef USE_NSEC\n+\tondisk->mtime.nsec = htonl(ce->ce_stat_data.sd_mtime.nsec);\n+#else\n+\tondisk->mtime.nsec = 0;\n+#endif\n+\tondisk->size = htonl(ce->ce_stat_data.sd_size);\n+\tif (!ce->ce_stat_crc)\n+\t\tce->ce_stat_crc = calculate_stat_crc(ce);\n+\tondisk->stat_crc = htonl(ce->ce_stat_crc);\n+\thashcpy(ondisk->sha1, ce->sha1);\n+}\n+\n+static int write_directories(struct directory_entry *de, int fd, int conflict_offset)\n+{\n+\tstruct directory_entry *current;\n+\tstruct ondisk_directory_entry ondisk;\n+\tint current_offset, offset_write, ondisk_size, foffset;\n+\tuint32_t crc;\n+\n+\t/*\n+\t * This is needed because the compiler aligns structs to sizes multiple\n+\t * of 4\n+\t */\n+\tondisk_size = sizeof(ondisk.flags)\n+\t\t+ sizeof(ondisk.foffset)\n+\t\t+ sizeof(ondisk.cr)\n+\t\t+ sizeof(ondisk.ncr)\n+\t\t+ sizeof(ondisk.nsubtrees)\n+\t\t+ sizeof(ondisk.nfiles)\n+\t\t+ sizeof(ondisk.nentries)\n+\t\t+ sizeof(ondisk.sha1);\n+\tcurrent = de;\n+\tcurrent_offset = 0;\n+\tfoffset = 0;\n+\twhile (current) {\n+\t\tint pathlen;\n+\n+\t\toffset_write = htonl(current_offset);\n+\t\tif (ce_write(NULL, fd, &offset_write, 4) < 0)\n+\t\t\treturn -1;\n+\t\tif (current->de_pathlen == 0)\n+\t\t\tpathlen = 0;\n+\t\telse\n+\t\t\tpathlen = current->de_pathlen + 1;\n+\t\tcurrent_offset += pathlen + 1 + ondisk_size + 4;\n+\t\tcurrent = current->next;\n+\t}\n+\t/*\n+\t * Write one more offset, which points to the end of the entries,\n+\t * because we use it for calculating the dir length, instead of\n+\t * using strlen.\n+\t */\n+\toffset_write = htonl(current_offset);\n+\tif (ce_write(NULL, fd, &offset_write, 4) < 0)\n+\t\treturn -1;\n+\tcurrent = de;\n+\twhile (current) {\n+\t\tcrc = 0;\n+\t\tif (current->de_pathlen == 0) {\n+\t\t\tif (ce_write(&crc, fd, current->pathname, 1) < 0)\n+\t\t\t\treturn -1;\n+\t\t} else {\n+\t\t\tchar *path;\n+\t\t\tpath = xmalloc(sizeof(char) * (current->de_pathlen + 2));\n+\t\t\tmemcpy(path, current->pathname, current->de_pathlen);\n+\t\t\tmemcpy(path + current->de_pathlen, \"/\\0\", 2);\n+\t\t\tif (ce_write(&crc, fd, path, current->de_pathlen + 2) < 0)\n+\t\t\t\treturn -1;\n+\t\t}\n+\t\tcurrent->de_foffset = foffset;\n+\t\tcurrent->de_cr = conflict_offset;\n+\t\tondisk_from_directory_entry(current, &ondisk);\n+\t\tif (ce_write(&crc, fd, &ondisk, ondisk_size) < 0)\n+\t\t\treturn -1;\n+\t\tcrc = htonl(crc);\n+\t\tif (ce_write(NULL, fd, &crc, 4) < 0)\n+\t\t\treturn -1;\n+\t\tconflict_offset += current->conflict_size;\n+\t\tfoffset += current->de_nfiles * 4;\n+\t\tcurrent = current->next;\n+\t}\n+\treturn 0;\n+}\n+\n+static int write_entries(struct index_state *istate,\n+\t\t\t struct directory_entry *de,\n+\t\t\t int entries,\n+\t\t\t int fd,\n+\t\t\t int offset_to_offset)\n+{\n+\tint offset, offset_write, ondisk_size;\n+\tstruct directory_entry *current;\n+\n+\toffset = 0;\n+\tondisk_size = sizeof(struct ondisk_cache_entry);\n+\tcurrent = de;\n+\twhile (current) {\n+\t\tint pathlen;\n+\t\tstruct cache_entry *ce = current->ce;\n+\n+\t\tif (current->de_pathlen == 0)\n+\t\t\tpathlen = 0;\n+\t\telse\n+\t\t\tpathlen = current->de_pathlen + 1;\n+\t\twhile (ce) {\n+\t\t\tif (ce->ce_flags & CE_REMOVE)\n+\t\t\t\tcontinue;\n+\t\t\tif (!ce_uptodate(ce) && is_racy_timestamp(istate, ce))\n+\t\t\t\tce_smudge_racily_clean_entry(ce);\n+\t\t\tif (is_null_sha1(ce->sha1))\n+\t\t\t\treturn error(\"cache entry has null sha1: %s\", ce->name);\n+\n+\t\t\toffset_write = htonl(offset);\n+\t\t\tif (ce_write(NULL, fd, &offset_write, 4) < 0)\n+\t\t\t\treturn -1;\n+\t\t\toffset += ce_namelen(ce) - pathlen + 1 + ondisk_size + 4;\n+\t\t\tce = ce->next;\n+\t\t}\n+\t\tcurrent = current->next;\n+\t}\n+\t/*\n+\t * Write one more offset, which points to the end of the entries,\n+\t * because we use it for calculating the file length, instead of\n+\t * using strlen.\n+\t */\n+\toffset_write = htonl(offset);\n+\tif (ce_write(NULL, fd, &offset_write, 4) < 0)\n+\t\treturn -1;\n+\n+\toffset = offset_to_offset;\n+\tcurrent = de;\n+\twhile (current) {\n+\t\tint pathlen;\n+\t\tstruct cache_entry *ce = current->ce;\n+\n+\t\tif (current->de_pathlen == 0)\n+\t\t\tpathlen = 0;\n+\t\telse\n+\t\t\tpathlen = current->de_pathlen + 1;\n+\t\twhile (ce) {\n+\t\t\tstruct ondisk_cache_entry ondisk;\n+\t\t\tuint32_t crc, calc_crc;\n+\n+\t\t\tif (ce->ce_flags & CE_REMOVE)\n+\t\t\t\tcontinue;\n+\t\t\tcalc_crc = htonl(offset);\n+\t\t\tcrc = crc32(0, (Bytef*)&calc_crc, 4);\n+\t\t\tif (ce_write(&crc, fd, ce->name + pathlen,\n+\t\t\t\t\tce_namelen(ce) - pathlen + 1) < 0)\n+\t\t\t\treturn -1;\n+\t\t\tondisk_from_cache_entry(ce, &ondisk);\n+\t\t\tif (ce_write(&crc, fd, &ondisk, ondisk_size) < 0)\n+\t\t\t\treturn -1;\n+\t\t\tcrc = htonl(crc);\n+\t\t\tif (ce_write(NULL, fd, &crc, 4) < 0)\n+\t\t\t\treturn -1;\n+\t\t\toffset += 4;\n+\t\t\tce = ce->next;\n+\t\t}\n+\t\tcurrent = current->next;\n+\t}\n+\treturn 0;\n+}\n+\n+static int write_conflict(struct conflict_entry *conflict, int fd)\n+{\n+\tstruct conflict_entry *current;\n+\tstruct conflict_part *current_part;\n+\tuint32_t crc;\n+\n+\tcurrent = conflict;\n+\twhile (current) {\n+\t\tunsigned int to_write;\n+\n+\t\tcrc = 0;\n+\t\tif (ce_write(&crc, fd,\n+\t\t (Bytef*)(current->name + current->pathlen),\n+\t\t current->namelen - current->pathlen) < 0)\n+\t\t\treturn -1;\n+\t\tif (ce_write(&crc, fd, (Bytef*)\"\\0\", 1) < 0)\n+\t\t\treturn -1;\n+\t\tto_write = htonl(current->nfileconflicts);\n+\t\tif (ce_write(&crc, fd, (Bytef*)&to_write, 4) < 0)\n+\t\t\treturn -1;\n+\t\tcurrent_part = current->entries;\n+\t\twhile (current_part) {\n+\t\t\tstruct ondisk_conflict_part ondisk;\n+\n+\t\t\tconflict_to_ondisk(current_part, &ondisk);\n+\t\t\tif (ce_write(&crc, fd, (Bytef*)&ondisk, sizeof(struct ondisk_conflict_part)) < 0)\n+\t\t\t\treturn 0;\n+\t\t\tcurrent_part = current_part->next;\n+\t\t}\n+\t\tto_write = htonl(crc);\n+\t\tif (ce_write(NULL, fd, (Bytef*)&to_write, 4) < 0)\n+\t\t\treturn -1;\n+\t\tcurrent = current->next;\n+\t}\n+\treturn 0;\n+}\n+\n+static int write_conflicts(struct index_state *istate,\n+\t\t\t struct directory_entry *de,\n+\t\t\t int fd)\n+{\n+\tstruct directory_entry *current;\n+\n+\tcurrent = de;\n+\twhile (current) {\n+\t\tif (current->de_ncr != 0) {\n+\t\t\tif (write_conflict(current->conflict, fd) < 0)\n+\t\t\t\treturn -1;\n+\t\t}\n+\t\tcurrent = current->next;\n+\t}\n+\treturn 0;\n+}\n+\n+static int write_index_v5(struct index_state *istate, int newfd)\n+{\n+\tstruct cache_version_header hdr;\n+\tstruct cache_header hdr_v5;\n+\tstruct cache_entry **cache = istate->cache;\n+\tstruct directory_entry *de;\n+\tstruct ondisk_directory_entry *ondisk;\n+\tint entries = istate->cache_nr;\n+\tint i, removed, non_conflicted, total_dir_len, ondisk_directory_size;\n+\tint total_file_len, conflict_offset, offset_to_offset;\n+\tunsigned int ndir;\n+\tuint32_t crc;\n+\n+\tif (istate->filter_opts)\n+\t\tdie(\"BUG: index: cannot write a partially read index\");\n+\n+\tfor (i = removed = 0; i < entries; i++) {\n+\t\tif (cache[i]->ce_flags & CE_REMOVE)\n+\t\t\tremoved++;\n+\t}\n+\thdr.hdr_signature = htonl(CACHE_SIGNATURE);\n+\thdr.hdr_version = htonl(istate->version);\n+\thdr_v5.hdr_nfile = htonl(entries - removed);\n+\thdr_v5.hdr_nextension = htonl(0); /* Currently no extensions are supported */\n+\n+\tnon_conflicted = 0;\n+\ttotal_dir_len = 0;\n+\ttotal_file_len = 0;\n+\tde = compile_directory_data(istate, entries, &ndir, &non_conflicted,\n+\t\t\t&total_dir_len, &total_file_len);\n+\thdr_v5.hdr_ndir = htonl(ndir);\n+\n+\t/*\n+\t * This is needed because the compiler aligns structs to sizes multipe\n+\t * of 4\n+\t */\n+\tondisk_directory_size = sizeof(ondisk->flags)\n+\t\t+ sizeof(ondisk->foffset)\n+\t\t+ sizeof(ondisk->cr)\n+\t\t+ sizeof(ondisk->ncr)\n+\t\t+ sizeof(ondisk->nsubtrees)\n+\t\t+ sizeof(ondisk->nfiles)\n+\t\t+ sizeof(ondisk->nentries)\n+\t\t+ sizeof(ondisk->sha1);\n+\thdr_v5.hdr_fblockoffset = htonl(sizeof(hdr) + sizeof(hdr_v5) + 4\n+\t\t+ (ndir + 1) * 4\n+\t\t+ total_dir_len\n+\t\t+ ndir * (ondisk_directory_size + 4)\n+\t\t+ (non_conflicted + 1) * 4);\n+\n+\tcrc = 0;\n+\tif (ce_write(&crc, newfd, &hdr, sizeof(hdr)) < 0)\n+\t\treturn -1;\n+\tif (ce_write(&crc, newfd, &hdr_v5, sizeof(hdr_v5)) < 0)\n+\t\treturn -1;\n+\tcrc = htonl(crc);\n+\tif (ce_write(NULL, newfd, &crc, 4) < 0)\n+\t\treturn -1;\n+\n+\tconflict_offset = sizeof(hdr) + sizeof(hdr_v5) + 4\n+\t\t+ (ndir + 1) * 4\n+\t\t+ total_dir_len\n+\t\t+ ndir * (ondisk_directory_size + 4)\n+\t\t+ (non_conflicted + 1) * 4\n+\t\t+ total_file_len\n+\t\t+ non_conflicted * (sizeof(struct ondisk_cache_entry) + 4);\n+\tif (write_directories(de, newfd, conflict_offset) < 0)\n+\t\treturn -1;\n+\toffset_to_offset = sizeof(hdr) + sizeof(hdr_v5) + 4\n+\t\t+ (ndir + 1) * 4\n+\t\t+ total_dir_len\n+\t\t+ ndir * (ondisk_directory_size + 4);\n+\tif (write_entries(istate, de, entries, newfd, offset_to_offset) < 0)\n+\t\treturn -1;\n+\tif (write_conflicts(istate, de, newfd) < 0)\n+\t\treturn -1;\n+\treturn ce_flush(newfd);\n+}\n+\n struct index_ops v5_ops = {\n \tmatch_stat_basic,\n \tverify_hdr,\n \tread_index_v5,\n-\tNULL\n+\twrite_index_v5\n };\ndiff --git a/read-cache.c b/read-cache.c\nindex 9bfbb4f..2fcca7f 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -108,7 +108,7 @@ int match_stat_data(const struct stat_data *sd, struct stat *st)\n \treturn changed;\n }\n \n-static uint32_t calculate_stat_crc(struct cache_entry *ce)\n+uint32_t calculate_stat_crc(struct cache_entry *ce)\n {\n \tunsigned int ctimens = 0;\n \tuint32_t stat, stat_crc;\n@@ -232,6 +232,8 @@ static void set_istate_ops(struct index_state *istate)\n \n \tif (istate->version >= 2 && istate->version <= 4)\n \t\tistate->ops = &v2_ops;\n+\tif (istate->version == 5)\n+\t\tistate->ops = &v5_ops;\n }\n \n int ce_match_stat_basic(struct index_state *istate,\n@@ -1300,7 +1302,12 @@ static int verify_hdr_version(struct index_state *istate,\n \thdr_version = ntohl(hdr->hdr_version);\n \tif (hdr_version < INDEX_FORMAT_LB || INDEX_FORMAT_UB < hdr_version)\n \t\treturn error(\"bad index version %d\", hdr_version);\n-\tistate->ops = &v2_ops;\n+\n+\tif (hdr_version >= 2 && hdr_version <= 4)\n+\t\tistate->ops = &v2_ops;\n+\telse if (hdr_version == 5)\n+\t\tistate->ops = &v5_ops;\n+\n \treturn 0;\n }\n \ndiff --git a/read-cache.h b/read-cache.h\nindex 869c562..0cb3b8b 100644\n--- a/read-cache.h\n+++ b/read-cache.h\n@@ -62,3 +62,4 @@ extern int ce_match_stat_basic(struct index_state *istate, const struct cache_en\n \t\t\t struct stat *st);\n extern int is_racy_timestamp(const struct index_state *istate, const struct cache_entry *ce);\n extern void set_index_entry(struct index_state *istate, int nr, struct cache_entry *ce);\n+extern uint32_t calculate_stat_crc(struct cache_entry *ce);\n-- \n1.8.3.453.g1dfc63d\n"},{"id":"223192","messageId":"1373650024-3001-17-git-send-email-t.gummerer@gmail.com","threadId":"34419","inReplyTo":"1373650024-3001-1-git-send-email-t.gummerer@gmail.com","subject":"[PATCH v2 16/19] read-cache: write index-v5 cache-tree data","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2013-07-12T17:27:01Z","receivedAt":"2013-07-12T17:27:01Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"Write the cache-tree data for the index version 5 file format. The\nin-memory cache-tree data is converted to the ondisk format, by adding\nit to the directory entries, that were compiled from the cache-entries\nin the step before.\n\nSigned-off-by: Thomas Gummerer \n---\n read-cache-v5.c | 53 +++++++++++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 53 insertions(+)\n\ndiff --git a/read-cache-v5.c b/read-cache-v5.c\nindex 33667d7..cd819b4 100644\n--- a/read-cache-v5.c\n+++ b/read-cache-v5.c\n@@ -941,6 +941,57 @@ static struct conflict_entry *create_conflict_entry_from_ce(struct cache_entry *\n \treturn create_new_conflict(ce->name, ce_namelen(ce), pathlen);\n }\n \n+static void convert_one_to_ondisk_v5(struct hash_table *table, struct cache_tree *it,\n+\t\t\t\tconst char *path, int pathlen, uint32_t crc)\n+{\n+\tint i;\n+\tstruct directory_entry *found, *search;\n+\n+\tcrc = crc32(crc, (Bytef*)path, pathlen);\n+\tfound = lookup_hash(crc, table);\n+\tsearch = found;\n+\twhile (search && strcmp(path, search->pathname + search->de_pathlen - strlen(path)) != 0)\n+\t\tsearch = search->next_hash;\n+\tif (!search)\n+\t\treturn;\n+\t/*\n+\t * The number of subtrees is already calculated by\n+\t * compile_directory_data, therefore we only need to\n+\t * add the entry_count\n+\t */\n+\tsearch->de_nentries = it->entry_count;\n+\tif (0 <= it->entry_count)\n+\t\thashcpy(search->sha1, it->sha1);\n+\tif (strcmp(path, \"\") != 0)\n+\t\tcrc = crc32(crc, (Bytef*)\"/\", 1);\n+\n+#if DEBUG\n+\tif (0 <= it->entry_count)\n+\t\tfprintf(stderr, \"cache-tree <%.*s> (%d ent, %d subtree) %s\\n\",\n+\t\t\tpathlen, path, it->entry_count, it->subtree_nr,\n+\t\t\tsha1_to_hex(it->sha1));\n+\telse\n+\t\tfprintf(stderr, \"cache-tree <%.*s> (%d subtree) invalid\\n\",\n+\t\t\tpathlen, path, it->subtree_nr);\n+#endif\n+\n+\tfor (i = 0; i < it->subtree_nr; i++) {\n+\t\tstruct cache_tree_sub *down = it->down[i];\n+\t\tif (i) {\n+\t\t\tstruct cache_tree_sub *prev = it->down[i-1];\n+\t\t\tif (subtree_name_cmp(down->name, down->namelen,\n+\t\t\t\t\t prev->name, prev->namelen) <= 0)\n+\t\t\t\tdie(\"fatal - unsorted cache subtree\");\n+\t\t}\n+\t\tconvert_one_to_ondisk_v5(table, down->cache_tree, down->name, down->namelen, crc);\n+\t}\n+}\n+\n+static void cache_tree_to_ondisk_v5(struct hash_table *table, struct cache_tree *root)\n+{\n+\tconvert_one_to_ondisk_v5(table, root, \"\", 0, 0);\n+}\n+\n static struct directory_entry *compile_directory_data(struct index_state *istate,\n \t\t\t\t\t\tint nfile,\n \t\t\t\t\t\tunsigned int *ndir,\n@@ -1046,6 +1097,8 @@ static struct directory_entry *compile_directory_data(struct index_state *istate\n \t\t\tprevious_entry->next = no_subtrees;\n \t\t}\n \t}\n+\tif (istate->cache_tree)\n+\t\tcache_tree_to_ondisk_v5(&table, istate->cache_tree);\n \treturn de;\n }\n \n-- \n1.8.3.453.g1dfc63d\n"},{"id":"223193","messageId":"1373650024-3001-18-git-send-email-t.gummerer@gmail.com","threadId":"34419","inReplyTo":"1373650024-3001-1-git-send-email-t.gummerer@gmail.com","subject":"[PATCH v2 17/19] read-cache: write resolve-undo data for index-v5","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2013-07-12T17:27:02Z","receivedAt":"2013-07-12T17:27:02Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"Make git read the resolve-undo data from the index.\n\nSince the resolve-undo data is joined with the conflicts in\nthe ondisk format of the index file version 5, conflicts and\nresolved data is read at the same time, and the resolve-undo\ndata is then converted to the in-memory format.\n\nHelped-by: Thomas Rast \nSigned-off-by: Thomas Gummerer \n---\n read-cache-v5.c | 94 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 94 insertions(+)\n\ndiff --git a/read-cache-v5.c b/read-cache-v5.c\nindex cd819b4..093ee1a 100644\n--- a/read-cache-v5.c\n+++ b/read-cache-v5.c\n@@ -992,6 +992,99 @@ static void cache_tree_to_ondisk_v5(struct hash_table *table, struct cache_tree\n \tconvert_one_to_ondisk_v5(table, root, \"\", 0, 0);\n }\n \n+static void resolve_undo_to_ondisk_v5(struct hash_table *table,\n+\t\t\t\t struct string_list *resolve_undo,\n+\t\t\t\t unsigned int *ndir, int *total_dir_len,\n+\t\t\t\t struct directory_entry *de)\n+{\n+\tstruct string_list_item *item;\n+\tstruct directory_entry *search;\n+\n+\tif (!resolve_undo)\n+\t\treturn;\n+\tfor_each_string_list_item(item, resolve_undo) {\n+\t\tstruct conflict_entry *conflict_entry;\n+\t\tstruct resolve_undo_info *ui = item->util;\n+\t\tchar *super;\n+\t\tint i, dir_len, len;\n+\t\tuint32_t crc;\n+\t\tstruct directory_entry *found, *current, *new_tree;\n+\n+\t\tif (!ui)\n+\t\t\tcontinue;\n+\n+\t\tsuper = super_directory(item->string);\n+\t\tif (!super)\n+\t\t\tdir_len = 0;\n+\t\telse\n+\t\t\tdir_len = strlen(super);\n+\t\tcrc = crc32(0, (Bytef*)super, dir_len);\n+\t\tfound = lookup_hash(crc, table);\n+\t\tcurrent = NULL;\n+\t\tnew_tree = NULL;\n+\n+\t\twhile (!found) {\n+\t\t\tstruct directory_entry *new;\n+\n+\t\t\tnew = init_directory_entry(super, dir_len);\n+\t\t\tif (!current)\n+\t\t\t\tcurrent = new;\n+\t\t\tinsert_directory_entry(new, table, total_dir_len, ndir, crc);\n+\t\t\tif (new_tree != NULL)\n+\t\t\t\tnew->de_nsubtrees = 1;\n+\t\t\tnew->next = new_tree;\n+\t\t\tnew_tree = new;\n+\t\t\tsuper = super_directory(super);\n+\t\t\tif (!super)\n+\t\t\t\tdir_len = 0;\n+\t\t\telse\n+\t\t\t\tdir_len = strlen(super);\n+\t\t\tcrc = crc32(0, (Bytef*)super, dir_len);\n+\t\t\tfound = lookup_hash(crc, table);\n+\t\t}\n+\t\tsearch = found;\n+\t\twhile (search->next_hash && strcmp(super, search->pathname) != 0)\n+\t\t\tsearch = search->next_hash;\n+\t\tif (search && !current)\n+\t\t\tcurrent = search;\n+\t\tif (!search && !current)\n+\t\t\tcurrent = new_tree;\n+\t\tif (!super && new_tree) {\n+\t\t\tnew_tree->next = de->next;\n+\t\t\tde->next = new_tree;\n+\t\t\tde->de_nsubtrees++;\n+\t\t} else if (new_tree) {\n+\t\t\tstruct directory_entry *temp;\n+\n+\t\t\tsearch = de->next;\n+\t\t\twhile (strcmp(super, search->pathname))\n+\t\t\t\tsearch = search->next;\n+\t\t\ttemp = new_tree;\n+\t\t\twhile (temp->next)\n+\t\t\t\ttemp = temp->next;\n+\t\t\tsearch->de_nsubtrees++;\n+\t\t\ttemp->next = search->next;\n+\t\t\tsearch->next = new_tree;\n+\t\t}\n+\n+\t\tlen = strlen(item->string);\n+\t\tconflict_entry = create_new_conflict(item->string, len, current->de_pathlen);\n+\t\tadd_conflict_to_directory_entry(current, conflict_entry);\n+\t\tfor (i = 0; i < 3; i++) {\n+\t\t\tif (ui->mode[i]) {\n+\t\t\t\tstruct conflict_part *cp;\n+\n+\t\t\t\tcp = xmalloc(sizeof(struct conflict_part));\n+\t\t\t\tcp->flags = (i + 1) << CONFLICT_STAGESHIFT;\n+\t\t\t\tcp->entry_mode = ui->mode[i];\n+\t\t\t\tcp->next = NULL;\n+\t\t\t\thashcpy(cp->sha1, ui->sha1[i]);\n+\t\t\t\tadd_part_to_conflict_entry(current, conflict_entry, cp);\n+\t\t\t}\n+\t\t}\n+\t}\n+}\n+\n static struct directory_entry *compile_directory_data(struct index_state *istate,\n \t\t\t\t\t\tint nfile,\n \t\t\t\t\t\tunsigned int *ndir,\n@@ -1099,6 +1192,7 @@ static struct directory_entry *compile_directory_data(struct index_state *istate\n \t}\n \tif (istate->cache_tree)\n \t\tcache_tree_to_ondisk_v5(&table, istate->cache_tree);\n+\tresolve_undo_to_ondisk_v5(&table, istate->resolve_undo, ndir, total_dir_len, de);\n \treturn de;\n }\n \n-- \n1.8.3.453.g1dfc63d\n"},{"id":"223194","messageId":"1373650024-3001-19-git-send-email-t.gummerer@gmail.com","threadId":"34419","inReplyTo":"1373650024-3001-1-git-send-email-t.gummerer@gmail.com","subject":"[PATCH v2 18/19] update-index.c: rewrite index when index-version is given","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2013-07-12T17:27:03Z","receivedAt":"2013-07-12T17:27:03Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"Make update-index always rewrite the index when a index-version\nis given, even if the index already has the right version.\nThis option is used for performance testing the writer and\nreader.\n\nSigned-off-by: Thomas Gummerer \n---\n builtin/update-index.c | 4 ++--\n 1 file changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/update-index.c b/builtin/update-index.c\nindex 4c6e3a6..7e723c0 100644\n--- a/builtin/update-index.c\n+++ b/builtin/update-index.c\n@@ -6,6 +6,7 @@\n #include \"cache.h\"\n #include \"quote.h\"\n #include \"cache-tree.h\"\n+#include \"read-cache.h\"\n #include \"tree-walk.h\"\n #include \"builtin.h\"\n #include \"refs.h\"\n@@ -863,8 +864,7 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)\n \t\t\t preferred_index_format,\n \t\t\t INDEX_FORMAT_LB, INDEX_FORMAT_UB);\n \n-\t\tif (the_index.version != preferred_index_format)\n-\t\t\tactive_cache_changed = 1;\n+\t\tactive_cache_changed = 1;\n \t\tthe_index.version = preferred_index_format;\n \t}\n \n-- \n1.8.3.453.g1dfc63d\n"},{"id":"223195","messageId":"1373650024-3001-20-git-send-email-t.gummerer@gmail.com","threadId":"34419","inReplyTo":"1373650024-3001-1-git-send-email-t.gummerer@gmail.com","subject":"[PATCH v2 19/19] p0003-index.sh: add perf test for the index formats","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2013-07-12T17:27:04Z","receivedAt":"2013-07-12T17:27:04Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"From: Thomas Rast \n\nAdd a performance test for index version [23]/4/5 by using\ngit update-index --index-version=x, thus testing both the reader\nand the writer speed of all index formats.\n\nSigned-off-by: Thomas Rast \nSigned-off-by: Thomas Gummerer \n---\n t/perf/p0003-index.sh | 59 +++++++++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 59 insertions(+)\n create mode 100755 t/perf/p0003-index.sh\n\ndiff --git a/t/perf/p0003-index.sh b/t/perf/p0003-index.sh\nnew file mode 100755\nindex 0000000..3e02868\n--- /dev/null\n+++ b/t/perf/p0003-index.sh\n@@ -0,0 +1,59 @@\n+#!/bin/sh\n+\n+test_description=\"Tests index versions [23]/4/5\"\n+\n+. ./perf-lib.sh\n+\n+test_perf_large_repo\n+\n+test_expect_success \"convert to v3\" \"\n+\tgit update-index --index-version=2\n+\"\n+\n+test_perf \"v[23]: update-index\" \"\n+\tgit update-index --index-version=2 >/dev/null\n+\"\n+\n+subdir=$(git ls-files | sed 's#/[^/]*$##' | grep -v '^$' | uniq | tail -n 30 | head -1)\n+\n+test_perf \"v[23]: grep nonexistent -- subdir\" \"\n+\ttest_must_fail git grep nonexistent -- $subdir >/dev/null\n+\"\n+\n+test_perf \"v[23]: ls-files -- subdir\" \"\n+\tgit ls-files $subdir >/dev/null\n+\"\n+\n+test_expect_success \"convert to v4\" \"\n+\tgit update-index --index-version=4\n+\"\n+\n+test_perf \"v4: update-index\" \"\n+\tgit update-index --index-version=4 >/dev/null\n+\"\n+\n+test_perf \"v4: grep nonexistent -- subdir\" \"\n+\ttest_must_fail git grep nonexistent -- $subdir >/dev/null\n+\"\n+\n+test_perf \"v4: ls-files -- subdir\" \"\n+\tgit ls-files $subdir >/dev/null\n+\"\n+\n+test_expect_success \"convert to v5\" \"\n+\tgit update-index --index-version=5\n+\"\n+\n+test_perf \"v5: update-index\" \"\n+\tgit update-index --index-version=5 >/dev/null\n+\"\n+\n+test_perf \"v5: grep nonexistent -- subdir\" \"\n+\ttest_must_fail git grep nonexistent -- $subdir >/dev/null\n+\"\n+\n+test_perf \"v5: ls-files -- subdir\" \"\n+\tgit ls-files $subdir >/dev/null\n+\"\n+\n+test_done\n-- \n1.8.3.453.g1dfc63d\n"},{"id":"223262","messageId":"CACsJy8AhmYBjzqPtF3f9Gk8hq2bk8-PtFCto9_4AkRePomvhAQ@mail.gmail.com","threadId":"34419","inReplyTo":"1373650024-3001-1-git-send-email-t.gummerer@gmail.com","subject":"Re: [PATCH v2 00/19] Index-v5","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2013-07-14T02:59:00Z","receivedAt":"2013-07-14T02:59:00Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Sat, Jul 13, 2013 at 12:26 AM, Thomas Gummerer wrote:\n> t/perf/p0003-index.sh | 59 +\n> t/t2104-update-index-skip-worktree.sh | 1 +\n\nFor such a big code addition, the test part seems modest. Speaking\nfrom my experience, I rarely run perf tests and \"make test\" does not\nactivate v5 code at all. A few more tests would be nice. The good news\nis I changed default index version to 5 and ran \"make test\", all\npassed.\n--\nDuy\n"},{"id":"223263","messageId":"CACsJy8A9E5yEQZTfw6sT+2VZfDHLx5iC-mb+otRkez7f-5Mq0Q@mail.gmail.com","threadId":"34419","inReplyTo":"1373650024-3001-4-git-send-email-t.gummerer@gmail.com","subject":"Re: [PATCH v2 03/19] read-cache: move index v2 specific functions to their own file","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2013-07-14T03:10:35Z","receivedAt":"2013-07-14T03:10:35Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Sat, Jul 13, 2013 at 12:26 AM, Thomas Gummerer wrote:\n> @@ -489,8 +479,8 @@ extern void *read_blob_data_from_index(struct index_state *, const char *, unsig\n> #define CE_MATCH_RACY_IS_DIRTY 02\n> /* do stat comparison even if CE_SKIP_WORKTREE is true */\n> #define CE_MATCH_IGNORE_SKIP_WORKTREE 04\n> -extern int ie_match_stat(const struct index_state *, const struct cache_entry *, struct stat *, unsigned int);\n> -extern int ie_modified(const struct index_state *, const struct cache_entry *, struct stat *, unsigned int);\n> +extern int ie_match_stat(struct index_state *, const struct cache_entry *, struct stat *, unsigned int);\n> +extern int ie_modified(struct index_state *, const struct cache_entry *, struct stat *, unsigned int);\n>\n\nI would rather we keep \"const struct index_state*\" if we could. I\ntried putting \"const\" back and found that ce_match_stat_basic() calls\nset_istate_ops(), which writes to \"struct index_state\". Putting\nset_istate_ops() in ce_match_stat_basic() may seem convenient, but\ndoes not make much sense (why would a match_stat function update\nindex_ops?). I think you could move it out and\n\n - read_index calls set_istate_ops\n - (bonus) discard_index probably should reset \"version\" field to zero\nand clear internal_ops\n - the callers that use index without read_index must call\ninitialize_index() or something, which in turn calls set_istate_ops.\ninitialize_index() may take the preferred index version\n - do not let update-index modifies version field directly when\n--index-version is given. Wrap it with set_index_version() or\nsomething, so we can do internal conversion from one version to\nanother if needed\n - remove set_istate_ops in write_index(), we may need internal_ops\nlong before writing. When write_index is called, internal_ops should\nbe already initialized\n--\nDuy\n"},{"id":"223264","messageId":"CACsJy8A7T8KRuTRoC9-2RXR21gWsgPt1uA6xa6BhjK2k3gF2CA@mail.gmail.com","threadId":"34419","inReplyTo":"1373650024-3001-7-git-send-email-t.gummerer@gmail.com","subject":"Re: [PATCH v2 06/19] read-cache: add index reading api","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2013-07-14T03:21:50Z","receivedAt":"2013-07-14T03:21:50Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Sat, Jul 13, 2013 at 12:26 AM, Thomas Gummerer wrote:\n> @@ -238,7 +245,6 @@ static int read_index_v2(struct index_state *istate, void *mmap,\n> disk_ce = (struct ondisk_cache_entry *)((char *)mmap + src_offset);\n> ce = create_from_disk(disk_ce, &consumed, previous_name);\n> set_index_entry(istate, i, ce);\n> -\n> src_offset += consumed;\n> }\n> strbuf_release(&previous_name_buf);\n\nNit pick. This is unnecessary.\n\n> +int for_each_index_entry_v2(struct index_state *istate, each_cache_entry_fn fn, void *cb_data)\n> +{\n> + int i, ret = 0;\n> + struct filter_opts *opts= istate->filter_opts;\n\nNitpick. space before \"=\".\n\n> --- a/read-cache.c\n> +++ b/read-cache.c\n> @@ -996,6 +996,7 @@ int add_index_entry(struct index_state *istate, struct cache_entry *ce, int opti\n> memmove(istate->cache + pos + 1,\n> istate->cache + pos,\n> (istate->cache_nr - pos - 1) * sizeof(ce));\n> +\n> set_index_entry(istate, pos, ce);\n> istate->cache_changed = 1;\n> return 0;\n\nNP: unnecessary change.\n\n> +int set_internal_ops(struct index_state *istate)\n> +{\n> + if (!istate->internal_ops && istate->cache)\n> + istate->internal_ops = &v2_internal_ops;\n> + if (!istate->internal_ops)\n> + return 0;\n> + return 1;\n> +}\n> +\n> +/*\n> + * Execute fn for each index entry which is currently in istate. Data\n> + * can be given to the function using the cb_data parameter.\n> + */\n> +int for_each_index_entry(struct index_state *istate, each_cache_entry_fn fn, void *cb_data)\n> +{\n> + if (!set_internal_ops(istate))\n> + return 0;\n> + return istate->internal_ops->for_each_index_entry(istate, fn, cb_data);\n> }\n\nset_internal_ops should have been called in read_index and initialize_index.\n\n> @@ -1374,6 +1409,7 @@ int discard_index(struct index_state *istate)\n> free(istate->cache);\n> istate->cache = NULL;\n> istate->cache_alloc = 0;\n> + istate->filter_opts = NULL;\n> return 0;\n> }\n\nWhy is internal_ops not reset here?\n\n> --- a/read-cache.h\n> +++ b/read-cache.h\n> @@ -24,11 +24,17 @@ struct cache_version_header {\n> struct index_ops {\n> int (*match_stat_basic)(const struct cache_entry *ce, struct stat *st, int changed);\n> int (*verify_hdr)(void *mmap, unsigned long size);\n> - int (*read_index)(struct index_state *istate, void *mmap, unsigned long mmap_size);\n> + int (*read_index)(struct index_state *istate, void *mmap, unsigned long mmap_size,\n> + struct filter_opts *opts);\n> int (*write_index)(struct index_state *istate, int newfd);\n> };\n>\n> +struct internal_ops {\n> + int (*for_each_index_entry)(struct index_state *istate, each_cache_entry_fn fn, void *cb_data);\n> +};\n> +\n\nI wonder if we do need separate internal_ops from index_ops. Why not\nmerge internal_ops in index_ops?\n--\nDuy\n"},{"id":"223265","messageId":"CACsJy8A61gfWsMecJha-3GoWXFT0bb3zAMJ5YT-a=vs=_3WghQ@mail.gmail.com","threadId":"34419","inReplyTo":"1373650024-3001-8-git-send-email-t.gummerer@gmail.com","subject":"Re: [PATCH v2 07/19] make sure partially read index is not changed","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2013-07-14T03:29:02Z","receivedAt":"2013-07-14T03:29:02Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Sat, Jul 13, 2013 at 12:26 AM, Thomas Gummerer wrote:\n> A partially read index file currently cannot be written to disk. Make\n> sure that never happens, by erroring out when a caller tries to change a\n> partially read index. The caller is responsible for reading the whole\n> index when it's trying to change it later.\n>\n> Forcing the caller to load the right part of the index file instead of\n> re-reading it when changing it, gives a bit of a performance advantage,\n> by avoiding to read parts of the index twice.\n\nIf you want to be strict about this, put similar check in\nfill_stat_cache_info (used by entry.c), cache_tree_invalidate_path and\nconvert the code base (maybe except unpack-trees.c, just put a check\nin the beginning of unpack_trees()) to use wrapper function to change\nce_flags (some still update ce_flags directly, grep them). Some flags\nare in memory only and should be allowed to change in partial index,\nsome are to be written on disk and should not be allowed.\n-- \nDuy\n"},{"id":"223266","messageId":"CACsJy8CMqJ5TF1V7CMTQjnAUp=zXPYxotq5uLqvaF0Sikzv+uw@mail.gmail.com","threadId":"34419","inReplyTo":"1373650024-3001-9-git-send-email-t.gummerer@gmail.com","subject":"Re: [PATCH v2 08/19] grep.c: Use index api","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2013-07-14T03:32:57Z","receivedAt":"2013-07-14T03:32:57Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Sat, Jul 13, 2013 at 12:26 AM, Thomas Gummerer wrote:\n> +static int grep_cache(struct cache_entry *ce, void *cb_data)\n> {\n> - int hit = 0;\n> - int nr;\n> - read_cache();\n> + struct grep_opts *opts = cb_data;\n>\n> - for (nr = 0; nr < active_nr; nr++) {\n> - struct cache_entry *ce = active_cache[nr];\n> - if (!S_ISREG(ce->ce_mode))\n> - continue;\n> - if (!match_pathspec_depth(pathspec, ce->name, ce_namelen(ce), 0, NULL))\n> - continue;\n> - /*\n> - * If CE_VALID is on, we assume worktree file and its cache entry\n> - * are identical, even if worktree file has been modified, so use\n> - * cache version instead\n> - */\n> - if (cached || (ce->ce_flags & CE_VALID) || ce_skip_worktree(ce)) {\n> - if (ce_stage(ce))\n> - continue;\n> - hit |= grep_sha1(opt, ce->sha1, ce->name, 0, ce->name);\n> - }\n> - else\n> - hit |= grep_file(opt, ce->name);\n> - if (ce_stage(ce)) {\n> - do {\n> - nr++;\n> - } while (nr < active_nr &&\n> - !strcmp(ce->name, active_cache[nr]->name));\n> - nr--; /* compensate for loop control */\n> - }\n> - if (hit && opt->status_only)\n> - break;\n> - }\n> - return hit;\n> + if (!S_ISREG(ce->ce_mode))\n> + return 0;\n> + if (!match_pathspec_depth(opts->pathspec, ce->name, ce_namelen(ce), 0, NULL))\n> + return 0;\n\nYou do a match_pathspec_depth here..\n\n> @@ -895,10 +887,21 @@ int cmd_grep(int argc, const char **argv, const char *prefix)\n> } else if (0 <= opt_exclude) {\n> die(_(\"--[no-]exclude-standard cannot be used for tracked contents.\"));\n> } else if (!list.nr) {\n> + struct grep_opts opts;\n> + struct filter_opts *filter_opts = xmalloc(sizeof(*filter_opts));\n> +\n> if (!cached)\n> setup_work_tree();\n>\n> - hit = grep_cache(&opt, &pathspec, cached);\n> + memset(filter_opts, 0, sizeof(*filter_opts));\n> + filter_opts->pathspec = &pathspec;\n> + opts.opt = &opt;\n> + opts.pathspec = &pathspec;\n> + opts.cached = cached;\n> + opts.hit = 0;\n> + read_cache_filtered(filter_opts);\n> + for_each_cache_entry(grep_cache, &opts);\n\nAnd here again inside for_each_cache_entry. In the worst case that\ncould turn into 2 expensive fnmatch instead of one. Is this conversion\nworth it? Note that match_pathspec is just a deprecated version of\nmatch_pathspec_depth. They basically do the same thing.\n-- \nDuy\n"},{"id":"223267","messageId":"CACsJy8DA=AD8=_kVYLNMS53fb8QwqKG4=8fA1cYAQBTn1gcj0Q@mail.gmail.com","threadId":"34419","inReplyTo":"1373650024-3001-10-git-send-email-t.gummerer@gmail.com","subject":"Re: [PATCH v2 09/19] ls-files.c: use index api","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2013-07-14T03:39:46Z","receivedAt":"2013-07-14T03:39:46Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Sat, Jul 13, 2013 at 12:26 AM, Thomas Gummerer wrote:\n> + if (!with_tree) {\n> + memset(opts, 0, sizeof(*opts));\n> + opts->pathspec = &pathspec_struct;\n> + opts->read_staged = 1;\n> + if (show_resolve_undo)\n> + opts->read_resolve_undo = 1;\n> + read_cache_filtered(opts);\n\nSo you load partial index here.\n\n> + } else {\n> + read_cache();\n> + }\n> + /* be nice with submodule paths ending in a slash */\n> + if (pathspec)\n> + strip_trailing_slash_from_submodules();\n\nThen strip_trailing_slash_from_submodules will attempt to convert\npathspec \"foo/\" to \"foo\" if \"foo\" exists in the index and is a\ngitlink. But becaues \"foo/\" is used to load the partial index, \"foo\"\nis not loaded (is it?) and this could become incorrect no-op. I\nsuggest you go through the pathspec once checking for ones ending with\n'/'. If so strip_trailing_... may potentially update pathspec, just\nload full index. If no pathspec ends with '/', strip_trail... is no-op\nand we can do partial loading safely.\n-- \nDuy\n"},{"id":"223268","messageId":"CACsJy8Ckcuk11Vmhyo-2wk6bMtNo17=Q9KXU9B-JAi0j7Cmi+A@mail.gmail.com","threadId":"34419","inReplyTo":"1373650024-3001-11-git-send-email-t.gummerer@gmail.com","subject":"Re: [PATCH v2 10/19] documentation: add documentation of the index-v5 file format","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2013-07-14T03:59:37Z","receivedAt":"2013-07-14T03:59:37Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Sat, Jul 13, 2013 at 12:26 AM, Thomas Gummerer wrote:\n> +== Directory offsets (diroffsets)\n> +\n> + diroffset (32-bits): offset to the directory relative to the beginning\n> + of the index file. There are ndir + 1 offsets in the diroffset table,\n> + the last is pointing to the end of the last direntry. With this last\n> + entry, we are able to replace the strlen of when reading the directory\n\nstrlen of what?\n\n> + name, by calculating it from diroffset[n+1]-diroffset[n]-61. 61 is the\n> + size of the directory data, which follows each each directory + the\n> + crc sum + the NUL byte.\n> +\n> + This part is needed for making the directory entries bisectable and\n> + thus allowing a binary search.\n\nJust thinking out loud. Maybe this section and fileoffsets should be\nmade optional extensions. So far I see no use for them. It's nice for\na program to occasionally look at a single entry, but such programs do\nnot exist (yet). For inotify monitor that may want to update a single\nfile's stat, it could regenerate the index with {dir,file}offsets\nextensions the first time it attempts to update the index, then it\ncould do bsearch.\n--\nDuy\n"},{"id":"223269","messageId":"CACsJy8C6HRCYMR3Q=j-D=2kgzvA7=0tauSnwrjpXzSPZWe+VZw@mail.gmail.com","threadId":"34419","inReplyTo":"1373650024-3001-13-git-send-email-t.gummerer@gmail.com","subject":"Re: [PATCH v2 12/19] read-cache: read index-v5","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2013-07-14T04:42:28Z","receivedAt":"2013-07-14T04:42:28Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Sat, Jul 13, 2013 at 12:26 AM, Thomas Gummerer wrote:\n> +struct directory_entry {\n> + struct directory_entry *next;\n> + struct directory_entry *next_hash;\n> + struct cache_entry *ce;\n> + struct cache_entry *ce_last;\n> + struct conflict_entry *conflict;\n> + struct conflict_entry *conflict_last;\n> + unsigned int conflict_size;\n> + unsigned int de_foffset;\n> + unsigned int de_cr;\n> + unsigned int de_ncr;\n> + unsigned int de_nsubtrees;\n> + unsigned int de_nfiles;\n> + unsigned int de_nentries;\n> + unsigned char sha1[20];\n> + unsigned short de_flags;\n> + unsigned int de_pathlen;\n> + char pathname[FLEX_ARRAY];\n> +};\n> +\n> +struct conflict_part {\n> + struct conflict_part *next;\n> + unsigned short flags;\n> + unsigned short entry_mode;\n> + unsigned char sha1[20];\n> +};\n> +\n> +struct conflict_entry {\n> + struct conflict_entry *next;\n> + unsigned int nfileconflicts;\n> + struct conflict_part *entries;\n> + unsigned int namelen;\n> + unsigned int pathlen;\n> + char name[FLEX_ARRAY];\n> +};\n> +\n> +struct ondisk_conflict_part {\n> + unsigned short flags;\n> + unsigned short entry_mode;\n> + unsigned char sha1[20];\n> +};\n\nThese new structs should probably be in read-cache-v5.c, or read-cache.h\n\n> #define cache_entry_size(len) (offsetof(struct cache_entry,name) + (len) + 1)\n> +#define directory_entry_size(len) (offsetof(struct directory_entry,pathname) + (len) + 1)\n> +#define conflict_entry_size(len) (offsetof(struct conflict_entry,name) + (len) + 1)\n\nThese are used by read-cache-v5.c only so far. I'd say move them to\nread-cache.h or read-cache-v5.c together with the new structs.\n\n> +struct ondisk_cache_entry {\n> + unsigned short flags;\n> + unsigned short mode;\n> + struct cache_time mtime;\n> + unsigned int size;\n> + int stat_crc;\n> + unsigned char sha1[20];\n> +};\n> +\n> +struct ondisk_directory_entry {\n> + unsigned int foffset;\n> + unsigned int cr;\n> + unsigned int ncr;\n> + unsigned int nsubtrees;\n> + unsigned int nfiles;\n> + unsigned int nentries;\n> + unsigned char sha1[20];\n> + unsigned short flags;\n> +};\n\nPerhaps use uint32_t, uint16_t and friends for all on-disk structures?\n\n> +static struct directory_entry *read_directories(unsigned int *dir_offset,\n> + unsigned int *dir_table_offset,\n> + void *mmap,\n> + int mmap_size)\n> +{\n> + int i, ondisk_directory_size;\n> + uint32_t *filecrc, *beginning, *end;\n> + struct directory_entry *current = NULL;\n> + struct ondisk_directory_entry *disk_de;\n> + struct directory_entry *de;\n> + unsigned int data_len, len;\n> + char *name;\n> +\n> + /*\n> + * Length of pathname + nul byte for termination + size of\n> + * members of ondisk_directory_entry. (Just using the size\n> + * of the struct doesn't work, because there may be padding\n> + * bytes for the struct)\n> + */\n> + ondisk_directory_size = sizeof(disk_de->flags)\n> + + sizeof(disk_de->foffset)\n> + + sizeof(disk_de->cr)\n> + + sizeof(disk_de->ncr)\n> + + sizeof(disk_de->nsubtrees)\n> + + sizeof(disk_de->nfiles)\n> + + sizeof(disk_de->nentries)\n> + + sizeof(disk_de->sha1);\n> + name = ptr_add(mmap, *dir_offset);\n> + beginning = ptr_add(mmap, *dir_table_offset);\n> + end = ptr_add(mmap, *dir_table_offset + 4);\n> + len = ntoh_l(*end) - ntoh_l(*beginning) - ondisk_directory_size - 5;\n> + disk_de = ptr_add(mmap, *dir_offset + len + 1);\n> + de = directory_entry_from_ondisk(disk_de, name, len);\n> + de->next = NULL;\n> +\n> + data_len = len + 1 + ondisk_directory_size;\n> + filecrc = ptr_add(mmap, *dir_offset + data_len);\n> + if (!check_crc32(0, ptr_add(mmap, *dir_offset), data_len, ntoh_l(*filecrc)))\n> + goto unmap;\n> +\n> + *dir_table_offset += 4;\n> + *dir_offset += data_len + 4; /* crc code */\n> +\n> + current = de;\n> + for (i = 0; i < de->de_nsubtrees; i++) {\n> + current->next = read_directories(dir_offset, dir_table_offset,\n> + mmap, mmap_size);\n> + while (current->next)\n> + current = current->next;\n> + }\n> +\n> + return de;\n> +unmap:\n> + munmap(mmap, mmap_size);\n> + die(\"directory crc doesn't match for '%s'\", de->pathname);\n> +}\n\nYou don't have to munmap when you die() anway. I'm not sure if flatten\nthe directory hierarchy into a list (linked by next pointer) is a good\nidea, or we should maintain the tree structure in memory. Still a lot\nof reading to figure that out..\n\nI skipped from here..\n\n> +static void ce_queue_push(struct cache_entry **head,\n> + struct cache_entry **tail,\n> + struct cache_entry *ce)\n> +{\n\n...\n\n> +static int read_conflicts(struct conflict_entry **head,\n> + struct directory_entry *de,\n> + void **mmap, unsigned long mmap_size)\n> +{\n\ntill the end of this function. Not interested in conflict stuff yet.\n\n\n> +static struct directory_entry *read_head_directories(struct index_state *istate,\n> + unsigned int *entry_offset,\n> + unsigned int *foffsetblock,\n> + unsigned int *ndirs,\n> + void *mmap, unsigned long mmap_size)\n> +{\n\nMaybe read_all_directories is a better nam.\n\n> +static int read_index_filtered_v5(struct index_state *istate, void *mmap,\n> + unsigned long mmap_size, struct filter_opts *opts)\n> +{\n> + unsigned int entry_offset, ndirs, foffsetblock, nr = 0;\n> + struct directory_entry *root_directory, *de;\n> + const char **adjusted_pathspec = NULL;\n> + int need_root = 0, i, n;\n> + char *oldpath, *seen;\n> +\n> + ...\n> +\n> + de = root_directory;\n> + while (de) {\n> + if (need_root ||\n> + match_pathspec(adjusted_pathspec, de->pathname, de->de_pathlen, 0, NULL)) {\n> + unsigned int subdir_foffsetblock = de->de_foffset + foffsetblock;\n> + unsigned int *off = mmap + subdir_foffsetblock;\n> + unsigned int subdir_entry_offset = entry_offset + ntoh_l(*off);\n> + oldpath = de->pathname;\n> + do {\n> + if (read_entries(istate, &de, &subdir_entry_offset,\n> + &mmap, mmap_size, &nr,\n> + &subdir_foffsetblock) < 0)\n> + return -1;\n> + } while (de && !prefixcmp(de->pathname, oldpath));\n> + } else\n> + de = de->next;\n> + }\n\nHm.. if we maintain a tree structure here (one link to the first\nsubdir, one link to the next sibling), I think the \"do\" loop could be\ndone without prefixcmp. Just check if \"de\" returned by read_entries is\nthe next sibling \"de\" (iow the end of current directory recursively).\n\n> + istate->cache_nr = nr;\n> + return 0;\n> +}\n> +\n> +static int read_index_v5(struct index_state *istate, void *mmap,\n> + unsigned long mmap_size, struct filter_opts *opts)\n> +{\n> + unsigned int entry_offset, ndirs, foffsetblock, nr = 0;\n> + struct directory_entry *root_directory, *de;\n> +\n> + if (opts != NULL)\n> + return read_index_filtered_v5(istate, mmap, mmap_size, opts);\n> +\n> + root_directory = read_head_directories(istate, &entry_offset,\n> + &foffsetblock, &ndirs,\n> + mmap, mmap_size);\n> + de = root_directory;\n> + while (de)\n> + if (read_entries(istate, &de, &entry_offset, &mmap,\n> + mmap_size, &nr, &foffsetblock) < 0)\n> + return -1;\n> + istate->cache_nr = nr;\n> + return 0;\n> +}\n\nMake it call read_index_filtered_v5 with an empty pathspec instead.\nmatch_pathspec* returns true immediately if pathspec is empty. Without\nthe removal of prefixcmp() in the \"do\" loop mentioned above,\nread_index_filtered_v5 can't be more expensive than this version.\n\nThat was it! Lunch time! Maybe I'll read the rest in the afternoon, or\nsomeday next week.\n--\nDuy\n"},{"id":"223406","messageId":"87ehb0cgqt.fsf@gmail.com","threadId":"34419","inReplyTo":"CACsJy8AhmYBjzqPtF3f9Gk8hq2bk8-PtFCto9_4AkRePomvhAQ@mail.gmail.com","subject":"Re: [PATCH v2 00/19] Index-v5","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2013-07-15T09:30:34Z","receivedAt":"2013-07-15T09:30:34Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"Duy Nguyen writes:\n\n> On Sat, Jul 13, 2013 at 12:26 AM, Thomas Gummerer wrote:\n>> t/perf/p0003-index.sh | 59 +\n>> t/t2104-update-index-skip-worktree.sh | 1 +\n>\n> For such a big code addition, the test part seems modest. Speaking\n> from my experience, I rarely run perf tests and \"make test\" does not\n> activate v5 code at all. A few more tests would be nice. The good news\n> is I changed default index version to 5 and ran \"make test\", all\n> passed.\n\nI was using the test suite with index version 5 as default index version\nfor testing of the new index file format. I think that's the best way\nto test the index, as it covers all aspects. Maybe we should add a test\nthat covers the basic functionality, just to make sure nothing obvious\nis broken when running the test suite with index-v2? Something like\nthis maybe:\n\n--->8---\n\n>From c476b521c94f1a9b0b4fcfe92d63321442d79c9a Mon Sep 17 00:00:00 2001\nFrom: Thomas Gummerer \nDate: Mon, 15 Jul 2013 11:21:06 +0200\nSubject: [PATCH] t1600: add basic test for index-v5\n\nAdd a test that checks the index-v5 file format when running the\ntest-suite with index-v2 as default index format. When making changes\nto the index, the test suite still should be run with both index v2 and\nindex v5 as default index format, for better coverage of all aspects of\nthe index.\n\nSigned-off-by: Thomas Gummerer \n---\n t/t1600-index-v5.sh | 133 ++++++++++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 133 insertions(+)\n create mode 100755 t/t1600-index-v5.sh\n\ndiff --git a/t/t1600-index-v5.sh b/t/t1600-index-v5.sh\nnew file mode 100755\nindex 0000000..528c17e\n--- /dev/null\n+++ b/t/t1600-index-v5.sh\n@@ -0,0 +1,133 @@\n+#!/bin/sh\n+#\n+# Copyright (c) 2013 Thomas Gummerer\n+\n+test_description='Test basic functionaltiy of index-v5.\n+\n+This test just covers the basics, to make sure normal runs of the test\n+suite cover this version of the index file format too. For better\n+testing of the index-v5 format, the default index version should be\n+changed to 5 and the test suite should be re-run'\n+\n+. ./test-lib.sh\n+\n+check_resolve_undo () {\n+\tmsg=$1\n+\tshift\n+\twhile case $# in\n+\t0)\tbreak ;;\n+\t1|2|3)\tdie \"Bug in check-resolve-undo test\" ;;\n+\tesac\n+\tdo\n+\t\tpath=$1\n+\t\tshift\n+\t\tfor stage in 1 2 3\n+\t\tdo\n+\t\t\tsha1=$1\n+\t\t\tshift\n+\t\t\tcase \"$sha1\" in\n+\t\t\t'') continue ;;\n+\t\t\tesac\n+\t\t\tsha1=$(git rev-parse --verify \"$sha1\")\n+\t\t\tprintf \"100644 %s %s\\t%s\\n\" $sha1 $stage $path\n+\t\tdone\n+\tdone >\"$msg.expect\" &&\n+\tgit ls-files --resolve-undo >\"$msg.actual\" &&\n+\ttest_cmp \"$msg.expect\" \"$msg.actual\"\n+}\n+\n+prime_resolve_undo () {\n+\tgit reset --hard &&\n+\tgit checkout second^0 &&\n+\ttest_tick &&\n+\ttest_must_fail git merge third^0 &&\n+\techo merge does not leave anything &&\n+\tcheck_resolve_undo empty &&\n+\techo different >fi/le &&\n+\tgit add fi/le &&\n+\techo resolving records &&\n+\tcheck_resolve_undo recorded fi/le initial:fi/le second:fi/le third:fi/le\n+}\n+\n+test_expect_success 'setup' '\n+\tgit update-index --index-version=5 &&\n+\techo file1 >file1 &&\n+\techo file2 >file2 &&\n+\tmkdir -p top/sub &&\n+\techo x >top/x &&\n+\techo xy >top/xy &&\n+\techo y >top/y &&\n+\techo yx >top/yx &&\n+\techo sub1 >top/sub/sub1 &&\n+\tgit add . &&\n+\tgit commit -m \"initial import\"\n+'\n+\n+test_expect_success 'ls-files shows all files' '\n+\tcat >expected <<-EOF &&\n+\t100644 e2129701f1a4d54dc44f03c93bca0a2aec7c5449 0\tfile1\n+\t100644 6c493ff740f9380390d5c9ddef4af18697ac9375 0\tfile2\n+\t100644 48df0cb83fee5d667537343f60a6057a63dd3c9b 0\ttop/sub/sub1\n+\t100644 587be6b4c3f93f93c489c0111bba5596147a26cb 0\ttop/x\n+\t100644 5aad9376af82d7b98a34f95fb0f298a162f52656 0\ttop/xy\n+\t100644 975fbec8256d3e8a3797e7a3611380f27c49f4ac 0\ttop/y\n+\t100644 ba1575927fa5b1f4bce72ad0c349566f1b02508e 0\ttop/yx\n+\tEOF\n+\tgit ls-files --stage >actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'ls-files with pathspec in subdir' '\n+\tcd top/sub &&\n+\tcat >expected <<-EOF &&\n+\t../x\n+\t../xy\n+\tEOF\n+\tgit ls-files --error-unmatch ../x* >actual &&\n+\ttest_cmp expected actual &&\n+\tcd ../..\n+'\n+\n+test_expect_success 'read-tree HEAD establishes cache-tree' '\n+\tgit read-tree HEAD &&\n+\tcat >expected <<-EOF &&\n+\t84e73410ea7864ccada24d897462e8ce1e1b872b (7 entries, 1 subtrees)\n+\t602482536bd3852e8ac2977ed1a9913a8c244aa0 top/ (5 entries, 1 subtrees)\n+\t20bb0109200f37a7e19283b4abc4a672be3f0adb top/sub/ (1 entries, 0 subtrees)\n+\tEOF\n+\ttest-dump-cache-tree >actual &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'setup resolve-undo' '\n+\tmkdir fi &&\n+\tprintf \"a\\0a\" >binary &&\n+\tgit add binary &&\n+\ttest_commit initial fi/le first &&\n+\tgit branch side &&\n+\tgit branch another &&\n+\tprintf \"a\\0b\" >binary &&\n+\tgit add binary &&\n+\ttest_commit second fi/le second &&\n+\tgit checkout side &&\n+\ttest_commit third fi/le third &&\n+\tgit branch add-add &&\n+\tgit checkout another &&\n+\ttest_commit fourth fi/le fourth &&\n+\tgit checkout add-add &&\n+\ttest_commit fifth add-differently &&\n+\tgit checkout master\n+'\n+\n+test_expect_success 'add records switch clears' '\n+\tprime_resolve_undo &&\n+\ttest_tick &&\n+\tgit commit -m merged &&\n+\techo committing keeps &&\n+\tcheck_resolve_undo kept fi/le initial:fi/le second:fi/le third:fi/le &&\n+\tgit checkout second^0 &&\n+\techo switching clears &&\n+\tcheck_resolve_undo cleared\n+'\n+\n+test_done\n--\n1.8.3.453.g1dfc63d\n"},{"id":"223407","messageId":"CACsJy8AiRCRvGmj4ZV+sc68d1z=S7YrRgPtPgEK+-zzNg7HcsA@mail.gmail.com","threadId":"34419","inReplyTo":"87ehb0cgqt.fsf@gmail.com","subject":"Re: [PATCH v2 00/19] Index-v5","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2013-07-15T09:38:55Z","receivedAt":"2013-07-15T09:38:55Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Mon, Jul 15, 2013 at 4:30 PM, Thomas Gummerer wrote:\n> Duy Nguyen writes:\n>\n>> On Sat, Jul 13, 2013 at 12:26 AM, Thomas Gummerer wrote:\n>>> t/perf/p0003-index.sh | 59 +\n>>> t/t2104-update-index-skip-worktree.sh | 1 +\n>>\n>> For such a big code addition, the test part seems modest. Speaking\n>> from my experience, I rarely run perf tests and \"make test\" does not\n>> activate v5 code at all. A few more tests would be nice. The good news\n>> is I changed default index version to 5 and ran \"make test\", all\n>> passed.\n>\n> I was using the test suite with index version 5 as default index version\n> for testing of the new index file format. I think that's the best way\n> to test the index, as it covers all aspects.\n\nWe need other people to run the test suite with v5 to catch bugs that\nonly appear on other platforms. Perhaps an env variable to allow the\ntest suite to override the binary's default index version and a new\nmake target \"test-v5\" maybe.\n\n> Maybe we should add a test\n> that covers the basic functionality, just to make sure nothing obvious\n> is broken when running the test suite with index-v2? Something like\n> this maybe:\n\nYep. v5 specfic test cases can cover some corner cases that the rest\nof the test suite does not care. Haven't read your t1600 though.\n--\nDuy\n"},{"id":"223415","messageId":"87bo64cfro.fsf@gmail.com","threadId":"34419","inReplyTo":"CACsJy8CMqJ5TF1V7CMTQjnAUp=zXPYxotq5uLqvaF0Sikzv+uw@mail.gmail.com","subject":"Re: [PATCH v2 08/19] grep.c: Use index api","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2013-07-15T09:51:39Z","receivedAt":"2013-07-15T09:51:39Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"Duy Nguyen writes:\n\n> On Sat, Jul 13, 2013 at 12:26 AM, Thomas Gummerer wrote:\n>> +static int grep_cache(struct cache_entry *ce, void *cb_data)\n>> {\n>> - int hit = 0;\n>> - int nr;\n>> - read_cache();\n>> + struct grep_opts *opts = cb_data;\n>>\n>> - for (nr = 0; nr < active_nr; nr++) {\n>> - struct cache_entry *ce = active_cache[nr];\n>> - if (!S_ISREG(ce->ce_mode))\n>> - continue;\n>> - if (!match_pathspec_depth(pathspec, ce->name, ce_namelen(ce), 0, NULL))\n>> - continue;\n>> - /*\n>> - * If CE_VALID is on, we assume worktree file and its cache entry\n>> - * are identical, even if worktree file has been modified, so use\n>> - * cache version instead\n>> - */\n>> - if (cached || (ce->ce_flags & CE_VALID) || ce_skip_worktree(ce)) {\n>> - if (ce_stage(ce))\n>> - continue;\n>> - hit |= grep_sha1(opt, ce->sha1, ce->name, 0, ce->name);\n>> - }\n>> - else\n>> - hit |= grep_file(opt, ce->name);\n>> - if (ce_stage(ce)) {\n>> - do {\n>> - nr++;\n>> - } while (nr < active_nr &&\n>> - !strcmp(ce->name, active_cache[nr]->name));\n>> - nr--; /* compensate for loop control */\n>> - }\n>> - if (hit && opt->status_only)\n>> - break;\n>> - }\n>> - return hit;\n>> + if (!S_ISREG(ce->ce_mode))\n>> + return 0;\n>> + if (!match_pathspec_depth(opts->pathspec, ce->name, ce_namelen(ce), 0, NULL))\n>> + return 0;\n>\n> You do a match_pathspec_depth here..\n>\n>> @@ -895,10 +887,21 @@ int cmd_grep(int argc, const char **argv, const char *prefix)\n>> } else if (0 <= opt_exclude) {\n>> die(_(\"--[no-]exclude-standard cannot be used for tracked contents.\"));\n>> } else if (!list.nr) {\n>> + struct grep_opts opts;\n>> + struct filter_opts *filter_opts = xmalloc(sizeof(*filter_opts));\n>> +\n>> if (!cached)\n>> setup_work_tree();\n>>\n>> - hit = grep_cache(&opt, &pathspec, cached);\n>> + memset(filter_opts, 0, sizeof(*filter_opts));\n>> + filter_opts->pathspec = &pathspec;\n>> + opts.opt = &opt;\n>> + opts.pathspec = &pathspec;\n>> + opts.cached = cached;\n>> + opts.hit = 0;\n>> + read_cache_filtered(filter_opts);\n>> + for_each_cache_entry(grep_cache, &opts);\n>\n> And here again inside for_each_cache_entry. In the worst case that\n> could turn into 2 expensive fnmatch instead of one. Is this conversion\n> worth it? Note that match_pathspec is just a deprecated version of\n> match_pathspec_depth. They basically do the same thing.\n\nRight, the match_pathspec_depth should in builtin/grep.c should be\nremoved, it's unnecessary when using for_each_index_entry. Thanks for\nspotting it. Other than that I still think the change makes sense.\n"},{"id":"223416","messageId":"CACsJy8DDRN5aTOzxuiGSa6UjbBiVUYwHjPcnN+8mw4xKmvkx-A@mail.gmail.com","threadId":"34419","inReplyTo":"1373650024-3001-13-git-send-email-t.gummerer@gmail.com","subject":"Re: [PATCH v2 12/19] read-cache: read index-v5","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2013-07-15T10:12:45Z","receivedAt":"2013-07-15T10:12:45Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"A little bit more..\n\nOn Sat, Jul 13, 2013 at 12:26 AM, Thomas Gummerer wrote:\n> +static void ce_queue_push(struct cache_entry **head,\n> + struct cache_entry **tail,\n> + struct cache_entry *ce)\n> +{\n> + if (!*head) {\n> + *head = *tail = ce;\n> + (*tail)->next = NULL;\n> + return;\n> + }\n> +\n> + (*tail)->next = ce;\n> + ce->next = NULL;\n> + *tail = (*tail)->next;\n\nNo no no. \"next\" is for name-hash.c don't \"reuse\" it here. And I don't\nthink you really need to maintain a linked list of cache_entries by\nthe way. More on read_entries comments below..\n\n> +}\n> +\n> +static struct cache_entry *ce_queue_pop(struct cache_entry **head)\n> +{\n> + struct cache_entry *ce;\n> +\n> + ce = *head;\n> + *head = (*head)->next;\n> + return ce;\n> +}\n\nSame as ce_queue_push.\n\n> +static int read_entries(struct index_state *istate, struct directory_entry **de,\n> + unsigned int *entry_offset, void **mmap,\n> + unsigned long mmap_size, unsigned int *nr,\n> + unsigned int *foffsetblock)\n> +{\n> + struct cache_entry *head = NULL, *tail = NULL;\n> + struct conflict_entry *conflict_queue;\n> + struct cache_entry *ce;\n> + int i;\n> +\n> + conflict_queue = NULL;\n> + if (read_conflicts(&conflict_queue, *de, mmap, mmap_size) < 0)\n> + return -1;\n> + for (i = 0; i < (*de)->de_nfiles; i++) {\n> + if (read_entry(&ce,\n> + *de,\n> + entry_offset,\n> + mmap,\n> + mmap_size,\n> + foffsetblock) < 0)\n> + return -1;\n> + ce_queue_push(&head, &tail, ce);\n> + *foffsetblock += 4;\n> +\n> + /*\n> + * Add the conflicted entries at the end of the index file\n> + * to the in memory format\n> + */\n> + if (conflict_queue &&\n> + (conflict_queue->entries->flags & CONFLICT_CONFLICTED) != 0 &&\n> + !cache_name_compare(conflict_queue->name, conflict_queue->namelen,\n> + ce->name, ce_namelen(ce))) {\n> + struct conflict_part *cp;\n> + cp = conflict_queue->entries;\n> + cp = cp->next;\n> + while (cp) {\n> + ce = convert_conflict_part(cp,\n> + conflict_queue->name,\n> + conflict_queue->namelen);\n> + ce_queue_push(&head, &tail, ce);\n> + conflict_part_head_remove(&cp);\n> + }\n> + conflict_entry_head_remove(&conflict_queue);\n> + }\n\nI start to wonder if separating staged entries is a good idea. It\nseems to make the code more complicated. The good point about conflict\nsection at the end of the file is you can just truncate() it out.\nAnother way is putting staged entries in fileentries, sorted\nalphabetically then by stage number, and a flag indicating if the\nentry is valid. When you remove resolve an entry, just set the flag to\ninvalid (partial write), so that read code will skip it.\n\nI think this approach is reasonably cheap (unless there are a lot of\nconflicts) and it simplifies this piece of code. truncate() may be\noverrated anyway. In my experience, I \"git add \" as soon as I\nresolve (so that \"git diff\" shrinks). One entry at a time, one\nindex write at a time. I don't think I ever resolve everything then\n\"git add -u .\", which is where truncate() shines because staged\nentries are removed all at once. We should optimize for one file\nresolution at a time, imo.\n\n> + }\n> +\n> + *de = (*de)->next;\n> +\n> + while (head) {\n> + if (*de != NULL\n> + && strcmp(head->name, (*de)->pathname) > 0) {\n> + read_entries(istate,\n> + de,\n> + entry_offset,\n> + mmap,\n> + mmap_size,\n> + nr,\n> + foffsetblock);\n> + } else {\n> + ce = ce_queue_pop(&head);\n> + set_index_entry(istate, *nr, ce);\n> + (*nr)++;\n> + ce->next = NULL;\n> + }\n\nIn this loop, you do some sort of merge sort combining a list of\nsorted files and a list of sorted directories (which will be expanded\nto bunches of files by the recursive read_entries). strcmp() does two\nthings. Assume we're in directory 'a', it makes sure that subdirectory\n'a/b' is only inserted after file 'a/a' is inserted to maintain index\nsort order. It also makes sure that 'de' of an outside directory 'b'\nwill not be expanded here. strcmp will be called for every file,\nchecking _full_ path. Sounds expensive.\n\nIf you maintain two pointers first_subdir and next_sibling in \"de\" as\nin my previous mail, you know \"de\" 'b' is out without string\ncomparison (de->next_sibling would be NULL). With that, the purpose of\nstrcmp is simply making sure that 'a/b' is inserted after 'a/a'.\nBecause we know all possible \"de\" is a subdirectory of 'a', we don't\nneed to compare the prefix 'a/'. We still need to strcmp on every\nfile, but we only need to compare the file name, not the leading\ndirectory anymore. Cheaper.\n\nGoing further, I wonder if we could eliminate strcmp completely. We\ncould store dummy entries in fileentries to mark the positions of the\nsubdirectories. When we encounter a dummy entry, we call\nread_entries() instead of set_index_entry.\n\nIf you merge the \"read_entry\" for loop in this while loop, you don't\nneed to maintain a linked list of ce to pop out one by one here.\n\n> + }\n> + return 0;\n> +}\n> +\n-- \nDuy\n"},{"id":"223547","messageId":"51E5B52C.7050603@ramsay1.demon.co.uk","threadId":"34419","inReplyTo":"1373650024-3001-1-git-send-email-t.gummerer@gmail.com","subject":"Re: [PATCH v2 00/19] Index-v5","fromName":"Ramsay Jones","fromEmail":"ramsay@ramsay1.demon.co.uk","sentAt":"2013-07-16T21:03:40Z","receivedAt":"2013-07-16T21:03:40Z","isPatch":true,"sender":{"key":"ramsay@ramsayjones.plus.com","avatar":"https://avatars.githubusercontent.com/u/33702710?v=4"},"body":"Thomas Gummerer wrote:\n> Hi,\n> \n> previous rounds (without api) are at $gmane/202752, $gmane/202923,\n> $gmane/203088 and $gmane/203517, the previous round with api was at\n> $gmane/229732. Thanks to Junio, Duy and Eric for their comments on\n> the previous round.\n\nIf I remember correctly, the original version of this series had the\nsame problem as Michael's \"Fix some reference-related races\" series\n(now in master). In particular, you had introduced an 'index_changed()'\nfunction which does essentially the same job as 'stat_validity_check()'\nin the new reference handling API. I seem to remember advising you\nnot to compare st_uid, st_gid and st_ino on __CYGWIN__.\n\nI haven't had time to look at this version of your series yet, but it\nmay be worth taking a look at stat_validity_check(). (although that is\ncausing failures on cygwin at the moment! ;-)\n\nAlso, I can't recall if I mentioned it to you at the time, but your\nindex reading code was (unnecessarily) calling munmap() twice on the\nsame buffer (without an intervening mmap()). This causes problems for\nsystems that have the NO_MMAP build variable set. In particular, the\ncompat/mmap.c code will attempt to free() the allocated memory block\ntwice, with unpredictable results.\n\nI wrote a patch to address this at the time (Hmm, seems to be built\non v1.8.1), but didn't submit it since your patch didn't progress. :-D\nI have included the patch below.\n\nATB,\nRamsay Jones\n\n-- >8 --\nFrom: Ramsay Jones \nDate: Sun, 9 Sep 2012 20:50:32 +0100\nSubject: [PATCH] mmap.c: Keep log of mmap() blocks to avoid double-delete bug\n\nWhen compiling with the NO_MMAP build variable set, the built-in\n'git_mmap()' and 'git_munmap()' compatibility routines use simple\nmemory allocation and file I/O to emulate the required behaviour.\nThe current implementation is vulnerable to the \"double-delete\" bug\n(where the pointer returned by malloc() is passed to free() two or\nmore times), should the mapped memory block address be passed to\nmunmap() multiple times.\n\nIn order to guard the implementation from such a calling sequence,\nwe keep a list of mmap-block descriptors, which we then consult to\ndetermine the validity of the input pointer to munmap(). This then\nallows 'git_munmap()' to return -1 on error, as required, with\nerrno set to EINVAL.\n\nUsing a list in the log of mmap-ed blocks, along with the resulting\nlinear search, means that the performance of the code is directly\nproportional to the number of concurrently active memory mapped\nfile regions. The number of such regions is not expected to be\nexcessive.\n\nSigned-off-by: Ramsay Jones \n---\n compat/mmap.c | 57 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++-\n 1 file changed, 56 insertions(+), 1 deletion(-)\n\ndiff --git a/compat/mmap.c b/compat/mmap.c\nindex c9d46d1..400e034 100644\n--- a/compat/mmap.c\n+++ b/compat/mmap.c\n@@ -1,14 +1,61 @@\n #include \"../git-compat-util.h\"\n \n+struct mmbd { /* memory mapped block descriptor */\n+\tstruct mmbd *next; /* next in list */\n+\tvoid *start; /* pointer to memory mapped block */\n+\tsize_t length; /* length of memory mapped block */\n+};\n+\n+static struct mmbd *head; /* head of mmb descriptor list */\n+\n+\n+static void add_desc(struct mmbd *desc, void *start, size_t length)\n+{\n+\tdesc->start = start;\n+\tdesc->length = length;\n+\tdesc->next = head;\n+\thead = desc;\n+}\n+\n+static void free_desc(struct mmbd *desc)\n+{\n+\tif (head == desc)\n+\t\thead = head->next;\n+\telse {\n+\t\tstruct mmbd *d = head;\n+\t\tfor (; d; d = d->next) {\n+\t\t\tif (d->next == desc) {\n+\t\t\t\td->next = desc->next;\n+\t\t\t\tbreak;\n+\t\t\t}\n+\t\t}\n+\t}\n+\tfree(desc);\n+}\n+\n+static struct mmbd *find_desc(void *start)\n+{\n+\tstruct mmbd *d = head;\n+\tfor (; d; d = d->next) {\n+\t\tif (d->start == start)\n+\t\t\treturn d;\n+\t}\n+\treturn NULL;\n+}\n+\n void *git_mmap(void *start, size_t length, int prot, int flags, int fd, off_t offset)\n {\n \tsize_t n = 0;\n+\tstruct mmbd *desc = NULL;\n \n \tif (start != NULL || !(flags & MAP_PRIVATE))\n \t\tdie(\"Invalid usage of mmap when built with NO_MMAP\");\n \n \tstart = xmalloc(length);\n-\tif (start == NULL) {\n+\tdesc = xmalloc(sizeof(*desc));\n+\tif (!start || !desc) {\n+\t\tfree(start);\n+\t\tfree(desc);\n \t\terrno = ENOMEM;\n \t\treturn MAP_FAILED;\n \t}\n@@ -25,18 +72,26 @@ void *git_mmap(void *start, size_t length, int prot, int flags, int fd, off_t of\n \t\t\tif (errno == EAGAIN || errno == EINTR)\n \t\t\t\tcontinue;\n \t\t\tfree(start);\n+\t\t\tfree(desc);\n \t\t\terrno = EACCES;\n \t\t\treturn MAP_FAILED;\n \t\t}\n \n \t\tn += count;\n \t}\n+\tadd_desc(desc, start, length);\n \n \treturn start;\n }\n \n int git_munmap(void *start, size_t length)\n {\n+\tstruct mmbd *d = find_desc(start);\n+\tif (!d) {\n+\t\terrno = EINVAL;\n+\t\treturn -1;\n+\t}\n+\tfree_desc(d);\n \tfree(start);\n \treturn 0;\n }\n-- \n1.8.3\n"},{"id":"223565","messageId":"87zjtlsjcw.fsf@gmail.com","threadId":"34419","inReplyTo":"51E5B52C.7050603@ramsay1.demon.co.uk","subject":"Re: [PATCH v2 00/19] Index-v5","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2013-07-17T08:04:15Z","receivedAt":"2013-07-17T08:04:15Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"Ramsay Jones writes:\n\n> Thomas Gummerer wrote:\n>> Hi,\n>>\n>> previous rounds (without api) are at $gmane/202752, $gmane/202923,\n>> $gmane/203088 and $gmane/203517, the previous round with api was at\n>> $gmane/229732. Thanks to Junio, Duy and Eric for their comments on\n>> the previous round.\n>\n> If I remember correctly, the original version of this series had the\n> same problem as Michael's \"Fix some reference-related races\" series\n> (now in master). In particular, you had introduced an 'index_changed()'\n> function which does essentially the same job as 'stat_validity_check()'\n> in the new reference handling API. I seem to remember advising you\n> not to compare st_uid, st_gid and st_ino on __CYGWIN__.\n\nYes, you provided a patch that simply wrapped those checks in a #if\n!defined (__CYGWIN__), which is included in the new series too.\n\n\n> I haven't had time to look at this version of your series yet, but it\n> may be worth taking a look at stat_validity_check(). (although that is\n> causing failures on cygwin at the moment! ;-)\n\nI took a quick look, that function makes sense I think. I'll use it in\nthe re-roll. It makes probably sense to wrap the uid, gid and ino\nfields as in the index_changed function.\n\n\n> Also, I can't recall if I mentioned it to you at the time, but your\n> index reading code was (unnecessarily) calling munmap() twice on the\n> same buffer (without an intervening mmap()). This causes problems for\n> systems that have the NO_MMAP build variable set. In particular, the\n> compat/mmap.c code will attempt to free() the allocated memory block\n> twice, with unpredictable results.\n>\n> I wrote a patch to address this at the time (Hmm, seems to be built\n> on v1.8.1), but didn't submit it since your patch didn't progress. :-D\n> I have included the patch below.\n\nI can't recall this either. From a quick check I don't call munmap() on\na already unmapped mmap, so I think this is fine as it is and your patch\nis independent from it. Not sure if it makes sense as safeguard for\nfuture changes.\n\n> -- >8 --\n> From: Ramsay Jones \n> Date: Sun, 9 Sep 2012 20:50:32 +0100\n> Subject: [PATCH] mmap.c: Keep log of mmap() blocks to avoid double-delete bug\n>\n> When compiling with the NO_MMAP build variable set, the built-in\n> 'git_mmap()' and 'git_munmap()' compatibility routines use simple\n> memory allocation and file I/O to emulate the required behaviour.\n> The current implementation is vulnerable to the \"double-delete\" bug\n> (where the pointer returned by malloc() is passed to free() two or\n> more times), should the mapped memory block address be passed to\n> munmap() multiple times.\n>\n> In order to guard the implementation from such a calling sequence,\n> we keep a list of mmap-block descriptors, which we then consult to\n> determine the validity of the input pointer to munmap(). This then\n> allows 'git_munmap()' to return -1 on error, as required, with\n> errno set to EINVAL.\n>\n> Using a list in the log of mmap-ed blocks, along with the resulting\n> linear search, means that the performance of the code is directly\n> proportional to the number of concurrently active memory mapped\n> file regions. The number of such regions is not expected to be\n> excessive.\n>\n> Signed-off-by: Ramsay Jones \n> ---\n> compat/mmap.c | 57 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++-\n> 1 file changed, 56 insertions(+), 1 deletion(-)\n>\n> diff --git a/compat/mmap.c b/compat/mmap.c\n> index c9d46d1..400e034 100644\n> --- a/compat/mmap.c\n> +++ b/compat/mmap.c\n> @@ -1,14 +1,61 @@\n> #include \"../git-compat-util.h\"\n>\n> +struct mmbd { /* memory mapped block descriptor */\n> +\tstruct mmbd *next; /* next in list */\n> +\tvoid *start; /* pointer to memory mapped block */\n> +\tsize_t length; /* length of memory mapped block */\n> +};\n> +\n> +static struct mmbd *head; /* head of mmb descriptor list */\n> +\n> +\n> +static void add_desc(struct mmbd *desc, void *start, size_t length)\n> +{\n> +\tdesc->start = start;\n> +\tdesc->length = length;\n> +\tdesc->next = head;\n> +\thead = desc;\n> +}\n> +\n> +static void free_desc(struct mmbd *desc)\n> +{\n> +\tif (head == desc)\n> +\t\thead = head->next;\n> +\telse {\n> +\t\tstruct mmbd *d = head;\n> +\t\tfor (; d; d = d->next) {\n> +\t\t\tif (d->next == desc) {\n> +\t\t\t\td->next = desc->next;\n> +\t\t\t\tbreak;\n> +\t\t\t}\n> +\t\t}\n> +\t}\n> +\tfree(desc);\n> +}\n> +\n> +static struct mmbd *find_desc(void *start)\n> +{\n> +\tstruct mmbd *d = head;\n> +\tfor (; d; d = d->next) {\n> +\t\tif (d->start == start)\n> +\t\t\treturn d;\n> +\t}\n> +\treturn NULL;\n> +}\n> +\n> void *git_mmap(void *start, size_t length, int prot, int flags, int fd, off_t offset)\n> {\n> \tsize_t n = 0;\n> +\tstruct mmbd *desc = NULL;\n>\n> \tif (start != NULL || !(flags & MAP_PRIVATE))\n> \t\tdie(\"Invalid usage of mmap when built with NO_MMAP\");\n>\n> \tstart = xmalloc(length);\n> -\tif (start == NULL) {\n> +\tdesc = xmalloc(sizeof(*desc));\n> +\tif (!start || !desc) {\n> +\t\tfree(start);\n> +\t\tfree(desc);\n> \t\terrno = ENOMEM;\n> \t\treturn MAP_FAILED;\n> \t}\n> @@ -25,18 +72,26 @@ void *git_mmap(void *start, size_t length, int prot, int flags, int fd, off_t of\n> \t\t\tif (errno == EAGAIN || errno == EINTR)\n> \t\t\t\tcontinue;\n> \t\t\tfree(start);\n> +\t\t\tfree(desc);\n> \t\t\terrno = EACCES;\n> \t\t\treturn MAP_FAILED;\n> \t\t}\n>\n> \t\tn += count;\n> \t}\n> +\tadd_desc(desc, start, length);\n>\n> \treturn start;\n> }\n>\n> int git_munmap(void *start, size_t length)\n> {\n> +\tstruct mmbd *d = find_desc(start);\n> +\tif (!d) {\n> +\t\terrno = EINVAL;\n> +\t\treturn -1;\n> +\t}\n> +\tfree_desc(d);\n> \tfree(start);\n> \treturn 0;\n> }\n> --\n> 1.8.3\n"},{"id":"223566","messageId":"87wqopsj84.fsf@gmail.com","threadId":"34419","inReplyTo":"CACsJy8DA=AD8=_kVYLNMS53fb8QwqKG4=8fA1cYAQBTn1gcj0Q@mail.gmail.com","subject":"Re: [PATCH v2 09/19] ls-files.c: use index api","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2013-07-17T08:07:07Z","receivedAt":"2013-07-17T08:07:07Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"Duy Nguyen writes:\n\n> On Sat, Jul 13, 2013 at 12:26 AM, Thomas Gummerer wrote:\n>> + if (!with_tree) {\n>> + memset(opts, 0, sizeof(*opts));\n>> + opts->pathspec = &pathspec_struct;\n>> + opts->read_staged = 1;\n>> + if (show_resolve_undo)\n>> + opts->read_resolve_undo = 1;\n>> + read_cache_filtered(opts);\n>\n> So you load partial index here.\n>\n>> + } else {\n>> + read_cache();\n>> + }\n>> + /* be nice with submodule paths ending in a slash */\n>> + if (pathspec)\n>> + strip_trailing_slash_from_submodules();\n>\n> Then strip_trailing_slash_from_submodules will attempt to convert\n> pathspec \"foo/\" to \"foo\" if \"foo\" exists in the index and is a\n> gitlink. But becaues \"foo/\" is used to load the partial index, \"foo\"\n> is not loaded (is it?) and this could become incorrect no-op. I\n> suggest you go through the pathspec once checking for ones ending with\n> '/'. If so strip_trailing_... may potentially update pathspec, just\n> load full index. If no pathspec ends with '/', strip_trail... is no-op\n> and we can do partial loading safely.\n\nIt was loaded, because the adjusted_pathspec algorithm stripped the\ntrailing slash and then everything until the next slash. That's\noverkill except when the trailing slash had to be stripped.\n\nI'll make the adjusted_pathspec algorithm more restrictive, so the last\ntrailing slash is no longer stripped. If a pathspec contains a trailing\nslash I'll load the whole index, as you suggested.\n"},{"id":"223567","messageId":"87txjtsj4a.fsf@gmail.com","threadId":"34419","inReplyTo":"CACsJy8Ckcuk11Vmhyo-2wk6bMtNo17=Q9KXU9B-JAi0j7Cmi+A@mail.gmail.com","subject":"Re: [PATCH v2 10/19] documentation: add documentation of the index-v5 file format","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2013-07-17T08:09:25Z","receivedAt":"2013-07-17T08:09:25Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"Duy Nguyen writes:\n\n> On Sat, Jul 13, 2013 at 12:26 AM, Thomas Gummerer wrote:\n>> +== Directory offsets (diroffsets)\n>> +\n>> + diroffset (32-bits): offset to the directory relative to the beginning\n>> + of the index file. There are ndir + 1 offsets in the diroffset table,\n>> + the last is pointing to the end of the last direntry. With this last\n>> + entry, we are able to replace the strlen of when reading the directory\n>\n> strlen of what?\n\nOf the dirname. Will fix in the re-roll, thanks for noticing.\n\n>> + name, by calculating it from diroffset[n+1]-diroffset[n]-61. 61 is the\n>> + size of the directory data, which follows each each directory + the\n>> + crc sum + the NUL byte.\n>> +\n>> + This part is needed for making the directory entries bisectable and\n>> + thus allowing a binary search.\n>\n> Just thinking out loud. Maybe this section and fileoffsets should be\n> made optional extensions. So far I see no use for them. It's nice for\n> a program to occasionally look at a single entry, but such programs do\n> not exist (yet). For inotify monitor that may want to update a single\n> file's stat, it could regenerate the index with {dir,file}offsets\n> extensions the first time it attempts to update the index, then it\n> could do bsearch.\n\nThere is a use already, namely saving the strlen for the filename.\nPreviously we had the length of the filename in the lower bits of the\nflags, but decided to remove it to avoid having extended flags. That in\naddition to the use case in the future warrants keeping them in the\nindex, I think.\n"},{"id":"223568","messageId":"87r4exsj14.fsf@gmail.com","threadId":"34419","inReplyTo":"CACsJy8DDRN5aTOzxuiGSa6UjbBiVUYwHjPcnN+8mw4xKmvkx-A@mail.gmail.com","subject":"Re: [PATCH v2 12/19] read-cache: read index-v5","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2013-07-17T08:11:19Z","receivedAt":"2013-07-17T08:11:19Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"Duy Nguyen writes:\n\n[..]\n\n>> +static int read_entries(struct index_state *istate, struct directory_entry **de,\n>> + unsigned int *entry_offset, void **mmap,\n>> + unsigned long mmap_size, unsigned int *nr,\n>> + unsigned int *foffsetblock)\n>> +{\n>> + struct cache_entry *head = NULL, *tail = NULL;\n>> + struct conflict_entry *conflict_queue;\n>> + struct cache_entry *ce;\n>> + int i;\n>> +\n>> + conflict_queue = NULL;\n>> + if (read_conflicts(&conflict_queue, *de, mmap, mmap_size) < 0)\n>> + return -1;\n>> + for (i = 0; i < (*de)->de_nfiles; i++) {\n>> + if (read_entry(&ce,\n>> + *de,\n>> + entry_offset,\n>> + mmap,\n>> + mmap_size,\n>> + foffsetblock) < 0)\n>> + return -1;\n>> + ce_queue_push(&head, &tail, ce);\n>> + *foffsetblock += 4;\n>> +\n>> + /*\n>> + * Add the conflicted entries at the end of the index file\n>> + * to the in memory format\n>> + */\n>> + if (conflict_queue &&\n>> + (conflict_queue->entries->flags & CONFLICT_CONFLICTED) != 0 &&\n>> + !cache_name_compare(conflict_queue->name, conflict_queue->namelen,\n>> + ce->name, ce_namelen(ce))) {\n>> + struct conflict_part *cp;\n>> + cp = conflict_queue->entries;\n>> + cp = cp->next;\n>> + while (cp) {\n>> + ce = convert_conflict_part(cp,\n>> + conflict_queue->name,\n>> + conflict_queue->namelen);\n>> + ce_queue_push(&head, &tail, ce);\n>> + conflict_part_head_remove(&cp);\n>> + }\n>> + conflict_entry_head_remove(&conflict_queue);\n>> + }\n>\n> I start to wonder if separating staged entries is a good idea. It\n> seems to make the code more complicated. The good point about conflict\n> section at the end of the file is you can just truncate() it out.\n> Another way is putting staged entries in fileentries, sorted\n> alphabetically then by stage number, and a flag indicating if the\n> entry is valid. When you remove resolve an entry, just set the flag to\n> invalid (partial write), so that read code will skip it.\n>\n> I think this approach is reasonably cheap (unless there are a lot of\n> conflicts) and it simplifies this piece of code. truncate() may be\n> overrated anyway. In my experience, I \"git add \" as soon as I\n> resolve (so that \"git diff\" shrinks). One entry at a time, one\n> index write at a time. I don't think I ever resolve everything then\n> \"git add -u .\", which is where truncate() shines because staged\n> entries are removed all at once. We should optimize for one file\n> resolution at a time, imo.\n\nThanks for your comments. I'll address the other ones once we decided\nto do with the conflicts.\n\nIt does make the code quite a bit more complicated, but also has one\nadvantage that you overlooked. We wouldn't truncate() when resolving\nthe conflicts. The resolve undo data is stored with the conflicts and\ntherefore we could just flip a bit and set the stage of the cache-entry\nin the main index to 0 (always once we got partial writing). This can\nbe fast both in case we resolve one entry at a time and when we resolve\na lot of entries. The advantage is even bigger when we resolve one\nentry at a time, when we otherwise would have to re-write the index for\neach conflict resolution.\n\n[..]\n"},{"id":"223569","messageId":"87oba1siz6.fsf@gmail.com","threadId":"34419","inReplyTo":"CACsJy8AiRCRvGmj4ZV+sc68d1z=S7YrRgPtPgEK+-zzNg7HcsA@mail.gmail.com","subject":"Re: [PATCH v2 00/19] Index-v5","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2013-07-17T08:12:29Z","receivedAt":"2013-07-17T08:12:29Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"Duy Nguyen writes:\n\n> On Mon, Jul 15, 2013 at 4:30 PM, Thomas Gummerer wrote:\n>> Duy Nguyen writes:\n>>\n>>> On Sat, Jul 13, 2013 at 12:26 AM, Thomas Gummerer wrote:\n>>>> t/perf/p0003-index.sh | 59 +\n>>>> t/t2104-update-index-skip-worktree.sh | 1 +\n>>>\n>>> For such a big code addition, the test part seems modest. Speaking\n>>> from my experience, I rarely run perf tests and \"make test\" does not\n>>> activate v5 code at all. A few more tests would be nice. The good news\n>>> is I changed default index version to 5 and ran \"make test\", all\n>>> passed.\n>>\n>> I was using the test suite with index version 5 as default index version\n>> for testing of the new index file format. I think that's the best way\n>> to test the index, as it covers all aspects.\n>\n> We need other people to run the test suite with v5 to catch bugs that\n> only appear on other platforms. Perhaps an env variable to allow the\n> test suite to override the binary's default index version and a new\n> make target \"test-v5\" maybe.\n\nAh ok, I understand. I think it's best to add a GIT_INDEX_VERSION=x\nconfig option to config.mak, where x is the index version that should be\ntested. This is also the way other test options are set\n(e.g. NO_SVN_TESTS). In addition this allows testing other versions of\nthe index file too.\n\n>> Maybe we should add a test\n>> that covers the basic functionality, just to make sure nothing obvious\n>> is broken when running the test suite with index-v2? Something like\n>> this maybe:\n>\n> Yep. v5 specfic test cases can cover some corner cases that the rest\n> of the test suite does not care. Haven't read your t1600 though.\n\nOk. I can't think of any corner cases right now that would be in v5,\nbut not in other versions, but if I catch some I'll add tests.\n"},{"id":"223580","messageId":"87li55s5t7.fsf@gmail.com","threadId":"34419","inReplyTo":"CACsJy8A61gfWsMecJha-3GoWXFT0bb3zAMJ5YT-a=vs=_3WghQ@mail.gmail.com","subject":"Re: [PATCH v2 07/19] make sure partially read index is not changed","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2013-07-17T12:56:52Z","receivedAt":"2013-07-17T12:56:52Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"Duy Nguyen writes:\n\n> On Sat, Jul 13, 2013 at 12:26 AM, Thomas Gummerer wrote:\n>> A partially read index file currently cannot be written to disk. Make\n>> sure that never happens, by erroring out when a caller tries to change a\n>> partially read index. The caller is responsible for reading the whole\n>> index when it's trying to change it later.\n>>\n>> Forcing the caller to load the right part of the index file instead of\n>> re-reading it when changing it, gives a bit of a performance advantage,\n>> by avoiding to read parts of the index twice.\n>\n> If you want to be strict about this, put similar check in\n> fill_stat_cache_info (used by entry.c), cache_tree_invalidate_path and\n> convert the code base (maybe except unpack-trees.c, just put a check\n> in the beginning of unpack_trees()) to use wrapper function to change\n> ce_flags (some still update ce_flags directly, grep them). Some flags\n> are in memory only and should be allowed to change in partial index,\n> some are to be written on disk and should not be allowed.\n\nI'm not sure anymore we should even be this strict. A partially read\nindex will never make it to the disk, because write_index checks if it's\nfully read. I think the check in write_index and read_index_filtered\nwould be enough.\n\nThe only disadvantage would be that it takes a little longer to catch an\nerror that should never happen in the first place. If it occurs the\nuser has bigger problems than the time that not having caught the\nerror earlier adds to the execution of the command.\n\nI also don't see a clean way to add the check to fill_stat_cache_info or\ncache_tree_invalidate_path, because we would need an additional\nparameter for the index_state, or at least for index_state->filter_opts,\nwhich doesn't do anything but check if the index is only partially loaded.\n"},{"id":"223622","messageId":"7v7ggo68o9.fsf@alter.siamese.dyndns.org","threadId":"34419","inReplyTo":"87oba1siz6.fsf@gmail.com","subject":"Re: [PATCH v2 00/19] Index-v5","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-07-17T23:58:14Z","receivedAt":"2013-07-17T23:58:14Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Thomas Gummerer writes:\n\n> Ah ok, I understand. I think it's best to add a GIT_INDEX_VERSION=x\n> config option to config.mak, where x is the index version that should be\n> tested.\n\nWhatever you do, please do not call it GIT_INDEX_VERSION _if_ it is\nonly to be used while testing. Have string \"TEST\" somewhere in the\nname, and make t/test-lib-functions.sh take notice.\n\nCurrently, the way for the user to show the preference as to which\nindex version to use is to explicitly set the version once, and then\nwe will (try to) propagate it inside the repository.\n\nI would not mind if we add a mechanism to make write_index() notice\nthe environment variable GIT_INDEX_VERSION and write the index in\nthat version. But that is conceptually very different from whatever\nyou give \"make VARIABLE=blah\" from the command line when building\nGit (or set in config.mak which amounts to the same thing).\n"},{"id":"223765","messageId":"87ppueha80.fsf@gmail.com","threadId":"34419","inReplyTo":"CACsJy8A9E5yEQZTfw6sT+2VZfDHLx5iC-mb+otRkez7f-5Mq0Q@mail.gmail.com","subject":"Re: [PATCH v2 03/19] read-cache: move index v2 specific functions to their own file","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2013-07-19T14:53:51Z","receivedAt":"2013-07-19T14:53:51Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"Duy Nguyen writes:\n\n> On Sat, Jul 13, 2013 at 12:26 AM, Thomas Gummerer wrote:\n>> @@ -489,8 +479,8 @@ extern void *read_blob_data_from_index(struct index_state *, const char *, unsig\n>> #define CE_MATCH_RACY_IS_DIRTY 02\n>> /* do stat comparison even if CE_SKIP_WORKTREE is true */\n>> #define CE_MATCH_IGNORE_SKIP_WORKTREE 04\n>> -extern int ie_match_stat(const struct index_state *, const struct cache_entry *, struct stat *, unsigned int);\n>> -extern int ie_modified(const struct index_state *, const struct cache_entry *, struct stat *, unsigned int);\n>> +extern int ie_match_stat(struct index_state *, const struct cache_entry *, struct stat *, unsigned int);\n>> +extern int ie_modified(struct index_state *, const struct cache_entry *, struct stat *, unsigned int);\n>>\n>\n> I would rather we keep \"const struct index_state*\" if we could. I\n> tried putting \"const\" back and found that ce_match_stat_basic() calls\n> set_istate_ops(), which writes to \"struct index_state\". Putting\n> set_istate_ops() in ce_match_stat_basic() may seem convenient, but\n> does not make much sense (why would a match_stat function update\n> index_ops?). I think you could move it out and\n>\n> - read_index calls set_istate_ops\n> - (bonus) discard_index probably should reset \"version\" field to zero\n> and clear internal_ops\n> - the callers that use index without read_index must call\n> initialize_index() or something, which in turn calls set_istate_ops.\n> initialize_index() may take the preferred index version\n> - do not let update-index modifies version field directly when\n> --index-version is given. Wrap it with set_index_version() or\n> something, so we can do internal conversion from one version to\n> another if needed\n> - remove set_istate_ops in write_index(), we may need internal_ops\n> long before writing. When write_index is called, internal_ops should\n> be already initialized\n\nOk, this makes sense. The only thing that I'm a little worried about is\ncatching all code paths that need to initialize the index. I'll\nimplement these suggestions in the re-roll.\n"},{"id":"223787","messageId":"87vc46lacn.fsf@gmail.com","threadId":"34419","inReplyTo":"7v7ggo68o9.fsf@alter.siamese.dyndns.org","subject":"Re: [PATCH v2 00/19] Index-v5","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2013-07-19T17:37:28Z","receivedAt":"2013-07-19T17:37:28Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"Junio C Hamano writes:\n\n> Thomas Gummerer writes:\n>\n>> Ah ok, I understand. I think it's best to add a GIT_INDEX_VERSION=x\n>> config option to config.mak, where x is the index version that should be\n>> tested.\n>\n> Whatever you do, please do not call it GIT_INDEX_VERSION _if_ it is\n> only to be used while testing. Have string \"TEST\" somewhere in the\n> name, and make t/test-lib-functions.sh take notice.\n>\n> Currently, the way for the user to show the preference as to which\n> index version to use is to explicitly set the version once, and then\n> we will (try to) propagate it inside the repository.\n>\n> I would not mind if we add a mechanism to make write_index() notice\n> the environment variable GIT_INDEX_VERSION and write the index in\n> that version. But that is conceptually very different from whatever\n> you give \"make VARIABLE=blah\" from the command line when building\n> Git (or set in config.mak which amounts to the same thing).\n\nWhat I currently did is add a environment variable GIT_INDEX_VERSION\nthat is used only if there is no index yet, to make sure existing\nrepositories aren't affected and still have to be converted explicitly\nby using git update-index.\n\nFor the tests I simply did export GIT_INDEX_VERSION in test-lib.sh to\nallow the addition of GIT_INDEX_VERSION in config.mak. Should I rename\nthat to GIT_INDEX_VERSION_TEST and do something like\n\nset_index_version() {\n export GIT_INDEX_VERSION=$GIT_INDEX_VERSION_TEST\n}\n\nin test-lib-functions.sh instead, does that make sense?\n"},{"id":"223792","messageId":"7vehauwgnp.fsf@alter.siamese.dyndns.org","threadId":"34419","inReplyTo":"87vc46lacn.fsf@gmail.com","subject":"Re: [PATCH v2 00/19] Index-v5","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-07-19T18:25:46Z","receivedAt":"2013-07-19T18:25:46Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Thomas Gummerer writes:\n\n> What I currently did is add a environment variable GIT_INDEX_VERSION\n> that is used only if there is no index yet, to make sure existing\n> repositories aren't affected and still have to be converted explicitly\n> by using git update-index.\n>\n> For the tests I simply did export GIT_INDEX_VERSION in test-lib.sh to\n> allow the addition of GIT_INDEX_VERSION in config.mak. Should I rename\n> that to GIT_INDEX_VERSION_TEST and do something like\n>\n> set_index_version() {\n> export GIT_INDEX_VERSION=$GIT_INDEX_VERSION_TEST\n> }\n>\n> in test-lib-functions.sh instead, does that make sense?\n\nMostly Yes ;-). You have to write it in a valid POSIX shell, i.e.\n\n\tset_index_version () {\n\t\tGIT_INDEX_VERSION=$TEST_GIT_INDEX_VERSION\n export GIT_INDEX_VERSION\n\t}\n"},{"id":"224549","messageId":"CACsJy8D9Q5=TF7MD1PUid-HEsCb7+JKo6=Rv6zXtjRYO3fPVYQ@mail.gmail.com","threadId":"34419","inReplyTo":"1373650024-3001-11-git-send-email-t.gummerer@gmail.com","subject":"Re: [PATCH v2 10/19] documentation: add documentation of the index-v5 file format","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2013-08-04T11:26:14Z","receivedAt":"2013-08-04T11:26:14Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Sat, Jul 13, 2013 at 12:26 AM, Thomas Gummerer wrote:\n> +== Header\n> + sig (32-bits): Signature:\n> + The signature is { 'D', 'I', 'R', 'C' } (stands for \"dircache\")\n> +\n> + vnr (32-bits): Version number:\n> + The current supported versions are 2, 3, 4 and 5.\n> +\n> + ndir (32-bits): number of directories in the index.\n> +\n> + nfile (32-bits): number of file entries in the index.\n\nI just noticed that \"file\" command can show the number of entries,\nsomething like this\n\n.git/index: Git index, version 4, 2625 entries\n\nMaybe we can swap ndir and nfile to immitate pre-v5 format, so\nunmodified \"file\" still shows correct number of entries/files with v5?\n-- \nDuy\n"},{"id":"224561","messageId":"878v0hqqw5.fsf@gmail.com","threadId":"34419","inReplyTo":"CACsJy8D9Q5=TF7MD1PUid-HEsCb7+JKo6=Rv6zXtjRYO3fPVYQ@mail.gmail.com","subject":"Re: [PATCH v2 10/19] documentation: add documentation of the index-v5 file format","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2013-08-04T17:58:50Z","receivedAt":"2013-08-04T17:58:50Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"Duy Nguyen writes:\n\n> On Sat, Jul 13, 2013 at 12:26 AM, Thomas Gummerer wrote:\n>> +== Header\n>> + sig (32-bits): Signature:\n>> + The signature is { 'D', 'I', 'R', 'C' } (stands for \"dircache\")\n>> +\n>> + vnr (32-bits): Version number:\n>> + The current supported versions are 2, 3, 4 and 5.\n>> +\n>> + ndir (32-bits): number of directories in the index.\n>> +\n>> + nfile (32-bits): number of file entries in the index.\n>\n> I just noticed that \"file\" command can show the number of entries,\n> something like this\n>\n> .git/index: Git index, version 4, 2625 entries\n>\n> Maybe we can swap ndir and nfile to immitate pre-v5 format, so\n> unmodified \"file\" still shows correct number of entries/files with v5?\n\nCool, I didn't know that worked. In that case it makes sense to\nimmitate pre-v5 format. Will change this in the re-roll.\n\nJust a heads up about my progress, I'm currently trying out your\nsuggestions about storing the directories as tree-format when reading\nand storing the dummy entries in the index. I'll make some experiments\nto see if the change is worth it, and will then send a re-roll, probably\nin a couple of weeks.\n"},{"id":"224734","messageId":"87mwotrk95.fsf@gmail.com","threadId":"34419","inReplyTo":"CACsJy8C6HRCYMR3Q=j-D=2kgzvA7=0tauSnwrjpXzSPZWe+VZw@mail.gmail.com","subject":"Re: [PATCH v2 12/19] read-cache: read index-v5","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2013-08-07T08:13:42Z","receivedAt":"2013-08-07T08:13:42Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"Duy Nguyen writes:\n\n> On Sat, Jul 13, 2013 at 12:26 AM, Thomas Gummerer wrote:\n>> +struct directory_entry {\n>> + struct directory_entry *next;\n>> + struct directory_entry *next_hash;\n>> + struct cache_entry *ce;\n>> + struct cache_entry *ce_last;\n>> + struct conflict_entry *conflict;\n>> + struct conflict_entry *conflict_last;\n>> + unsigned int conflict_size;\n>> + unsigned int de_foffset;\n>> + unsigned int de_cr;\n>> + unsigned int de_ncr;\n>> + unsigned int de_nsubtrees;\n>> + unsigned int de_nfiles;\n>> + unsigned int de_nentries;\n>> + unsigned char sha1[20];\n>> + unsigned short de_flags;\n>> + unsigned int de_pathlen;\n>> + char pathname[FLEX_ARRAY];\n>> +};\n>> +\n>> +struct conflict_part {\n>> + struct conflict_part *next;\n>> + unsigned short flags;\n>> + unsigned short entry_mode;\n>> + unsigned char sha1[20];\n>> +};\n>> +\n>> +struct conflict_entry {\n>> + struct conflict_entry *next;\n>> + unsigned int nfileconflicts;\n>> + struct conflict_part *entries;\n>> + unsigned int namelen;\n>> + unsigned int pathlen;\n>> + char name[FLEX_ARRAY];\n>> +};\n>> +\n>> +struct ondisk_conflict_part {\n>> + unsigned short flags;\n>> + unsigned short entry_mode;\n>> + unsigned char sha1[20];\n>> +};\n>\n> These new structs should probably be in read-cache-v5.c, or read-cache.h\n\nMakes sense, thanks.\n\n>> #define cache_entry_size(len) (offsetof(struct cache_entry,name) + (len) + 1)\n>> +#define directory_entry_size(len) (offsetof(struct directory_entry,pathname) + (len) + 1)\n>> +#define conflict_entry_size(len) (offsetof(struct conflict_entry,name) + (len) + 1)\n>\n> These are used by read-cache-v5.c only so far. I'd say move them to\n> read-cache.h or read-cache-v5.c together with the new structs.\n\nThanks.\n\n>> +struct ondisk_cache_entry {\n>> + unsigned short flags;\n>> + unsigned short mode;\n>> + struct cache_time mtime;\n>> + unsigned int size;\n>> + int stat_crc;\n>> + unsigned char sha1[20];\n>> +};\n>> +\n>> +struct ondisk_directory_entry {\n>> + unsigned int foffset;\n>> + unsigned int cr;\n>> + unsigned int ncr;\n>> + unsigned int nsubtrees;\n>> + unsigned int nfiles;\n>> + unsigned int nentries;\n>> + unsigned char sha1[20];\n>> + unsigned short flags;\n>> +};\n>\n> Perhaps use uint32_t, uint16_t and friends for all on-disk structures?\n\nWe got this in the makefile, so I think we should be fine without it.\nIt still makes sense for clarity though I think.\n\n ifdef NO_UINTMAX_T\n\t BASIC_CFLAGS += -Duintmax_t=uint32_t\n endif\n\nWhile at it I'll make the code for v[234] use them too.\n\n>> +static struct directory_entry *read_directories(unsigned int *dir_offset,\n>> + unsigned int *dir_table_offset,\n>> + void *mmap,\n>> + int mmap_size)\n>> +{\n>> + int i, ondisk_directory_size;\n>> + uint32_t *filecrc, *beginning, *end;\n>> + struct directory_entry *current = NULL;\n>> + struct ondisk_directory_entry *disk_de;\n>> + struct directory_entry *de;\n>> + unsigned int data_len, len;\n>> + char *name;\n>> +\n>> + /*\n>> + * Length of pathname + nul byte for termination + size of\n>> + * members of ondisk_directory_entry. (Just using the size\n>> + * of the struct doesn't work, because there may be padding\n>> + * bytes for the struct)\n>> + */\n>> + ondisk_directory_size = sizeof(disk_de->flags)\n>> + + sizeof(disk_de->foffset)\n>> + + sizeof(disk_de->cr)\n>> + + sizeof(disk_de->ncr)\n>> + + sizeof(disk_de->nsubtrees)\n>> + + sizeof(disk_de->nfiles)\n>> + + sizeof(disk_de->nentries)\n>> + + sizeof(disk_de->sha1);\n>> + name = ptr_add(mmap, *dir_offset);\n>> + beginning = ptr_add(mmap, *dir_table_offset);\n>> + end = ptr_add(mmap, *dir_table_offset + 4);\n>> + len = ntoh_l(*end) - ntoh_l(*beginning) - ondisk_directory_size - 5;\n>> + disk_de = ptr_add(mmap, *dir_offset + len + 1);\n>> + de = directory_entry_from_ondisk(disk_de, name, len);\n>> + de->next = NULL;\n>> +\n>> + data_len = len + 1 + ondisk_directory_size;\n>> + filecrc = ptr_add(mmap, *dir_offset + data_len);\n>> + if (!check_crc32(0, ptr_add(mmap, *dir_offset), data_len, ntoh_l(*filecrc)))\n>> + goto unmap;\n>> +\n>> + *dir_table_offset += 4;\n>> + *dir_offset += data_len + 4; /* crc code */\n>> +\n>> + current = de;\n>> + for (i = 0; i < de->de_nsubtrees; i++) {\n>> + current->next = read_directories(dir_offset, dir_table_offset,\n>> + mmap, mmap_size);\n>> + while (current->next)\n>> + current = current->next;\n>> + }\n>> +\n>> + return de;\n>> +unmap:\n>> + munmap(mmap, mmap_size);\n>> + die(\"directory crc doesn't match for '%s'\", de->pathname);\n>> +}\n>\n> You don't have to munmap when you die() anway.\n\nWill change that in the re-roll.\n\n> I'm not sure if flatten\n> the directory hierarchy into a list (linked by next pointer) is a good\n> idea, or we should maintain the tree structure in memory. Still a lot\n> of reading to figure that out..\n>\n> I skipped from here..\n>\n>> +static void ce_queue_push(struct cache_entry **head,\n>> + struct cache_entry **tail,\n>> + struct cache_entry *ce)\n>> +{\n>\n> ...\n>\n>> +static int read_conflicts(struct conflict_entry **head,\n>> + struct directory_entry *de,\n>> + void **mmap, unsigned long mmap_size)\n>> +{\n>\n> till the end of this function. Not interested in conflict stuff yet.\n>\n>\n>> +static struct directory_entry *read_head_directories(struct index_state *istate,\n>> + unsigned int *entry_offset,\n>> + unsigned int *foffsetblock,\n>> + unsigned int *ndirs,\n>> + void *mmap, unsigned long mmap_size)\n>> +{\n>\n> Maybe read_all_directories is a better nam.\n\nMakes sense, thanks.\n\n>> +static int read_index_filtered_v5(struct index_state *istate, void *mmap,\n>> + unsigned long mmap_size, struct filter_opts *opts)\n>> +{\n>> + unsigned int entry_offset, ndirs, foffsetblock, nr = 0;\n>> + struct directory_entry *root_directory, *de;\n>> + const char **adjusted_pathspec = NULL;\n>> + int need_root = 0, i, n;\n>> + char *oldpath, *seen;\n>> +\n>> + ...\n>> +\n>> + de = root_directory;\n>> + while (de) {\n>> + if (need_root ||\n>> + match_pathspec(adjusted_pathspec, de->pathname, de->de_pathlen, 0, NULL)) {\n>> + unsigned int subdir_foffsetblock = de->de_foffset + foffsetblock;\n>> + unsigned int *off = mmap + subdir_foffsetblock;\n>> + unsigned int subdir_entry_offset = entry_offset + ntoh_l(*off);\n>> + oldpath = de->pathname;\n>> + do {\n>> + if (read_entries(istate, &de, &subdir_entry_offset,\n>> + &mmap, mmap_size, &nr,\n>> + &subdir_foffsetblock) < 0)\n>> + return -1;\n>> + } while (de && !prefixcmp(de->pathname, oldpath));\n>> + } else\n>> + de = de->next;\n>> + }\n>\n> Hm.. if we maintain a tree structure here (one link to the first\n> subdir, one link to the next sibling), I think the \"do\" loop could be\n> done without prefixcmp. Just check if \"de\" returned by read_entries is\n> the next sibling \"de\" (iow the end of current directory recursively).\n\nYes, the tree-structure makes sense. I've implemented it a bit\ndifferently though, instead of using two pointers, I'm using one pointer\nto an array of directory entries, which can be iterated over.\n\n>> + istate->cache_nr = nr;\n>> + return 0;\n>> +}\n>> +\n>> +static int read_index_v5(struct index_state *istate, void *mmap,\n>> + unsigned long mmap_size, struct filter_opts *opts)\n>> +{\n>> + unsigned int entry_offset, ndirs, foffsetblock, nr = 0;\n>> + struct directory_entry *root_directory, *de;\n>> +\n>> + if (opts != NULL)\n>> + return read_index_filtered_v5(istate, mmap, mmap_size, opts);\n>> +\n>> + root_directory = read_head_directories(istate, &entry_offset,\n>> + &foffsetblock, &ndirs,\n>> + mmap, mmap_size);\n>> + de = root_directory;\n>> + while (de)\n>> + if (read_entries(istate, &de, &entry_offset, &mmap,\n>> + mmap_size, &nr, &foffsetblock) < 0)\n>> + return -1;\n>> + istate->cache_nr = nr;\n>> + return 0;\n>> +}\n>\n> Make it call read_index_filtered_v5 with an empty pathspec instead.\n> match_pathspec* returns true immediately if pathspec is empty. Without\n> the removal of prefixcmp() in the \"do\" loop mentioned above,\n> read_index_filtered_v5 can't be more expensive than this version.\n\nYes right, will change in the re-roll.\n\n> That was it! Lunch time! Maybe I'll read the rest in the afternoon, or\n> someday next week.\n\nThanks a lot for taking the time to review my code.\n"},{"id":"224735","messageId":"87k3jxrjs9.fsf@gmail.com","threadId":"34419","inReplyTo":"CACsJy8DDRN5aTOzxuiGSa6UjbBiVUYwHjPcnN+8mw4xKmvkx-A@mail.gmail.com","subject":"Re: [PATCH v2 12/19] read-cache: read index-v5","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2013-08-07T08:23:50Z","receivedAt":"2013-08-07T08:23:50Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"Duy Nguyen writes:\n\n> A little bit more..\n>\n> On Sat, Jul 13, 2013 at 12:26 AM, Thomas Gummerer wrote:\n>> +static void ce_queue_push(struct cache_entry **head,\n>> + struct cache_entry **tail,\n>> + struct cache_entry *ce)\n>> +{\n>> + if (!*head) {\n>> + *head = *tail = ce;\n>> + (*tail)->next = NULL;\n>> + return;\n>> + }\n>> +\n>> + (*tail)->next = ce;\n>> + ce->next = NULL;\n>> + *tail = (*tail)->next;\n>\n> No no no. \"next\" is for name-hash.c don't \"reuse\" it here. And I don't\n> think you really need to maintain a linked list of cache_entries by\n> the way. More on read_entries comments below..\n\nYou're right, I don't need it for the reading code. I do need to keep a\nlist of cache-entries for writing though, where a linked list is best\nsuited for the task. I'll use a next_ce pointer for that.\n\n>> +}\n>> +\n>> +static struct cache_entry *ce_queue_pop(struct cache_entry **head)\n>> +{\n>> + struct cache_entry *ce;\n>> +\n>> + ce = *head;\n>> + *head = (*head)->next;\n>> + return ce;\n>> +}\n>\n> Same as ce_queue_push.\n>\n>> +static int read_entries(struct index_state *istate, struct directory_entry **de,\n>> + unsigned int *entry_offset, void **mmap,\n>> + unsigned long mmap_size, unsigned int *nr,\n>> + unsigned int *foffsetblock)\n>> +{\n>> + struct cache_entry *head = NULL, *tail = NULL;\n>> + struct conflict_entry *conflict_queue;\n>> + struct cache_entry *ce;\n>> + int i;\n>> +\n>> + conflict_queue = NULL;\n>> + if (read_conflicts(&conflict_queue, *de, mmap, mmap_size) < 0)\n>> + return -1;\n>> + for (i = 0; i < (*de)->de_nfiles; i++) {\n>> + if (read_entry(&ce,\n>> + *de,\n>> + entry_offset,\n>> + mmap,\n>> + mmap_size,\n>> + foffsetblock) < 0)\n>> + return -1;\n>> + ce_queue_push(&head, &tail, ce);\n>> + *foffsetblock += 4;\n>> +\n>> + /*\n>> + * Add the conflicted entries at the end of the index file\n>> + * to the in memory format\n>> + */\n>> + if (conflict_queue &&\n>> + (conflict_queue->entries->flags & CONFLICT_CONFLICTED) != 0 &&\n>> + !cache_name_compare(conflict_queue->name, conflict_queue->namelen,\n>> + ce->name, ce_namelen(ce))) {\n>> + struct conflict_part *cp;\n>> + cp = conflict_queue->entries;\n>> + cp = cp->next;\n>> + while (cp) {\n>> + ce = convert_conflict_part(cp,\n>> + conflict_queue->name,\n>> + conflict_queue->namelen);\n>> + ce_queue_push(&head, &tail, ce);\n>> + conflict_part_head_remove(&cp);\n>> + }\n>> + conflict_entry_head_remove(&conflict_queue);\n>> + }\n>\n> I start to wonder if separating staged entries is a good idea. It\n> seems to make the code more complicated. The good point about conflict\n> section at the end of the file is you can just truncate() it out.\n> Another way is putting staged entries in fileentries, sorted\n> alphabetically then by stage number, and a flag indicating if the\n> entry is valid. When you remove resolve an entry, just set the flag to\n> invalid (partial write), so that read code will skip it.\n>\n> I think this approach is reasonably cheap (unless there are a lot of\n> conflicts) and it simplifies this piece of code. truncate() may be\n> overrated anyway. In my experience, I \"git add \" as soon as I\n> resolve (so that \"git diff\" shrinks). One entry at a time, one\n> index write at a time. I don't think I ever resolve everything then\n> \"git add -u .\", which is where truncate() shines because staged\n> entries are removed all at once. We should optimize for one file\n> resolution at a time, imo.\n>\n>> + }\n>> +\n>> + *de = (*de)->next;\n>> +\n>> + while (head) {\n>> + if (*de != NULL\n>> + && strcmp(head->name, (*de)->pathname) > 0) {\n>> + read_entries(istate,\n>> + de,\n>> + entry_offset,\n>> + mmap,\n>> + mmap_size,\n>> + nr,\n>> + foffsetblock);\n>> + } else {\n>> + ce = ce_queue_pop(&head);\n>> + set_index_entry(istate, *nr, ce);\n>> + (*nr)++;\n>> + ce->next = NULL;\n>> + }\n>\n> In this loop, you do some sort of merge sort combining a list of\n> sorted files and a list of sorted directories (which will be expanded\n> to bunches of files by the recursive read_entries). strcmp() does two\n> things. Assume we're in directory 'a', it makes sure that subdirectory\n> 'a/b' is only inserted after file 'a/a' is inserted to maintain index\n> sort order. It also makes sure that 'de' of an outside directory 'b'\n> will not be expanded here. strcmp will be called for every file,\n> checking _full_ path. Sounds expensive.\n>\n> If you maintain two pointers first_subdir and next_sibling in \"de\" as\n> in my previous mail, you know \"de\" 'b' is out without string\n> comparison (de->next_sibling would be NULL). With that, the purpose of\n> strcmp is simply making sure that 'a/b' is inserted after 'a/a'.\n> Because we know all possible \"de\" is a subdirectory of 'a', we don't\n> need to compare the prefix 'a/'. We still need to strcmp on every\n> file, but we only need to compare the file name, not the leading\n> directory anymore. Cheaper.\n>\n> Going further, I wonder if we could eliminate strcmp completely. We\n> could store dummy entries in fileentries to mark the positions of the\n> subdirectories. When we encounter a dummy entry, we call\n> read_entries() instead of set_index_entry.\n\nI've tried implementing both versions with the internal tree structure,\nsee below for timings.\n\nsimpleapi is the code that was posted to the list, HEAD uses a\ntree-structure for the directories internally, and replaces strcmp with\ncache_name_compare and leaves out the prefix. This tree uses dummy\nentries for the sub-directories.\n\nDummy entries are single NUL-bytes mixed with the cache-entries, which\nshould give optimal performance. I haven't thought much about\ncorruption of the index, but I don't think we would need any additional\ncrc checksums or similar.\n\nThe performance advantage seems very slim to none when using the dummy\nentries to avoid the comparison though, so I don't think it makes sense\nto make the index more complicated for a very small speed-up.\n\nTest simpleapi HEAD this tree \n-------------------------------------------------------------------------------------------------------------\n0003.2: v[23]: update-index 3.22(2.61+0.58) 3.20(2.56+0.61) -0.6% 3.22(2.67+0.53) +0.0% \n0003.3: v[23]: grep nonexistent -- subdir 1.65(1.36+0.27) 1.65(1.34+0.30) +0.0% 1.67(1.34+0.31) +1.2% \n0003.4: v[23]: ls-files -- subdir 1.50(1.20+0.29) 1.54(1.18+0.34) +2.7% 1.53(1.22+0.30) +2.0% \n0003.6: v4: update-index 2.69(2.28+0.39) 2.70(2.21+0.47) +0.4% 2.70(2.27+0.41) +0.4% \n0003.7: v4: grep nonexistent -- subdir 1.33(0.98+0.34) 1.36(1.01+0.33) +2.3% 1.34(1.03+0.30) +0.8% \n0003.8: v4: ls-files -- subdir 1.21(0.86+0.33) 1.23(0.91+0.30) +1.7% 1.22(0.90+0.30) +0.8% \n0003.10: v5: update-index 2.12(1.58+0.52) 2.20(1.67+0.52) +3.8% 2.19(1.66+0.51) +3.3% \n0003.11: v5: ls-files 1.57(1.21+0.34) 1.55(1.20+0.33) -1.3% 1.52(1.21+0.29) -3.2% \n0003.12: v5: grep nonexistent -- subdir 0.08(0.06+0.01) 0.07(0.04+0.02) -12.5% 0.07(0.04+0.02) -12.5%\n0003.13: v5: ls-files -- subdir 0.07(0.04+0.01) 0.06(0.05+0.00) -14.3% 0.07(0.05+0.01) +0.0% \n\n\n> If you merge the \"read_entry\" for loop in this while loop, you don't\n> need to maintain a linked list of ce to pop out one by one here.\n\nMakes sense, I'll do that.\n\n>> + }\n>> + return 0;\n>> +}\n>> +\n> --\n> Duy\n"},{"id":"224780","messageId":"CACsJy8C4r=5K3ZdzNnNJ95xS3_xnsTSH2bUyLJ+rwRv4Jxo2zw@mail.gmail.com","threadId":"34419","inReplyTo":"87r4exsj14.fsf@gmail.com","subject":"Re: [PATCH v2 12/19] read-cache: read index-v5","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2013-08-08T02:00:51Z","receivedAt":"2013-08-08T02:00:51Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Wed, Jul 17, 2013 at 3:11 PM, Thomas Gummerer wrote:\n> Duy Nguyen writes:\n>\n> [..]\n>\n>>> +static int read_entries(struct index_state *istate, struct directory_entry **de,\n>>> + unsigned int *entry_offset, void **mmap,\n>>> + unsigned long mmap_size, unsigned int *nr,\n>>> + unsigned int *foffsetblock)\n>>> +{\n>>> + struct cache_entry *head = NULL, *tail = NULL;\n>>> + struct conflict_entry *conflict_queue;\n>>> + struct cache_entry *ce;\n>>> + int i;\n>>> +\n>>> + conflict_queue = NULL;\n>>> + if (read_conflicts(&conflict_queue, *de, mmap, mmap_size) < 0)\n>>> + return -1;\n>>> + for (i = 0; i < (*de)->de_nfiles; i++) {\n>>> + if (read_entry(&ce,\n>>> + *de,\n>>> + entry_offset,\n>>> + mmap,\n>>> + mmap_size,\n>>> + foffsetblock) < 0)\n>>> + return -1;\n>>> + ce_queue_push(&head, &tail, ce);\n>>> + *foffsetblock += 4;\n>>> +\n>>> + /*\n>>> + * Add the conflicted entries at the end of the index file\n>>> + * to the in memory format\n>>> + */\n>>> + if (conflict_queue &&\n>>> + (conflict_queue->entries->flags & CONFLICT_CONFLICTED) != 0 &&\n>>> + !cache_name_compare(conflict_queue->name, conflict_queue->namelen,\n>>> + ce->name, ce_namelen(ce))) {\n>>> + struct conflict_part *cp;\n>>> + cp = conflict_queue->entries;\n>>> + cp = cp->next;\n>>> + while (cp) {\n>>> + ce = convert_conflict_part(cp,\n>>> + conflict_queue->name,\n>>> + conflict_queue->namelen);\n>>> + ce_queue_push(&head, &tail, ce);\n>>> + conflict_part_head_remove(&cp);\n>>> + }\n>>> + conflict_entry_head_remove(&conflict_queue);\n>>> + }\n>>\n>> I start to wonder if separating staged entries is a good idea. It\n>> seems to make the code more complicated. The good point about conflict\n>> section at the end of the file is you can just truncate() it out.\n>> Another way is putting staged entries in fileentries, sorted\n>> alphabetically then by stage number, and a flag indicating if the\n>> entry is valid. When you remove resolve an entry, just set the flag to\n>> invalid (partial write), so that read code will skip it.\n>>\n>> I think this approach is reasonably cheap (unless there are a lot of\n>> conflicts) and it simplifies this piece of code. truncate() may be\n>> overrated anyway. In my experience, I \"git add \" as soon as I\n>> resolve (so that \"git diff\" shrinks). One entry at a time, one\n>> index write at a time. I don't think I ever resolve everything then\n>> \"git add -u .\", which is where truncate() shines because staged\n>> entries are removed all at once. We should optimize for one file\n>> resolution at a time, imo.\n>\n> Thanks for your comments. I'll address the other ones once we decided\n> to do with the conflicts.\n>\n> It does make the code quite a bit more complicated, but also has one\n> advantage that you overlooked.\n\nI did overlook, although my goal is to keep the code simpler, not more\ncomlicated. The thinking is if we can find everything in fileentries\ntable, the code here is simplified, so..\n\n> We wouldn't truncate() when resolving\n> the conflicts. The resolve undo data is stored with the conflicts and\n> therefore we could just flip a bit and set the stage of the cache-entry\n> in the main index to 0 (always once we got partial writing). This can\n> be fast both in case we resolve one entry at a time and when we resolve\n> a lot of entries. The advantage is even bigger when we resolve one\n> entry at a time, when we otherwise would have to re-write the index for\n> each conflict resolution.\n\nIf I understand it correctly, filentries can only contain stage-0 or\nstage-1 entries, \"stage > 0\" entries are stored in conflict data. Once\na conflict is solved, you update the stage-1 entry in fileentries,\nturning it to stage-0 and recalculate the entry checksum. Conflict\ndata remains there to function as the old REUC extension. Correct?\n\nFirst of all, if that's true, we only need 1 bit for stage in fileentries table.\n\nSecondly, you may get away with looking up to conflict data in this\nfunction by storing all stages in fileentries (now we need 2-bit\nstage), replicated in conflict data for reuc function. When you\nresolve conflict, you flip stage-1 to stage-0, and flip (a new bit) to\nmark stage-2 entry invalid so the code knows to skip it. Next time the\nindex is rewritten, invalid entries are removed, but we still have old\nstage entries in conflict data. The flipping business is pretty much\nwhat you plan anyway, but the reading code does not need to look at\nboth fileentries and conflict data at the same time.\n\nWhat do you think?\n-- \nDuy\n"},{"id":"224781","messageId":"CACsJy8Bf=SW-Lt6Xr+1LU_dQ_Rqj-3BLSCmqrOjP1RJTkBunZw@mail.gmail.com","threadId":"34419","inReplyTo":"87k3jxrjs9.fsf@gmail.com","subject":"Re: [PATCH v2 12/19] read-cache: read index-v5","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2013-08-08T02:09:40Z","receivedAt":"2013-08-08T02:09:40Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Wed, Aug 7, 2013 at 3:23 PM, Thomas Gummerer wrote:\n>> On Sat, Jul 13, 2013 at 12:26 AM, Thomas Gummerer wrote:\n>>> +static void ce_queue_push(struct cache_entry **head,\n>>> + struct cache_entry **tail,\n>>> + struct cache_entry *ce)\n>>> +{\n>>> + if (!*head) {\n>>> + *head = *tail = ce;\n>>> + (*tail)->next = NULL;\n>>> + return;\n>>> + }\n>>> +\n>>> + (*tail)->next = ce;\n>>> + ce->next = NULL;\n>>> + *tail = (*tail)->next;\n>>\n>> No no no. \"next\" is for name-hash.c don't \"reuse\" it here. And I don't\n>> think you really need to maintain a linked list of cache_entries by\n>> the way. More on read_entries comments below..\n>\n> You're right, I don't need it for the reading code. I do need to keep a\n> list of cache-entries for writing though, where a linked list is best\n> suited for the task. I'll use a next_ce pointer for that.\n\nHmm.. yeah you need to maintain temporary structures before writing\nout direntries, then fileentries table. We can't just write as we\ntraverse through the cache in one go. Maybe a new field in cache_entry\nfor the linked list, or maintain the linked list off cache_entry..\nwhichever is easier for you.\n\n> I've tried implementing both versions with the internal tree structure,\n> see below for timings.\n>\n> simpleapi is the code that was posted to the list, HEAD uses a\n> tree-structure for the directories internally, and replaces strcmp with\n> cache_name_compare and leaves out the prefix. This tree uses dummy\n> entries for the sub-directories.\n>\n> Dummy entries are single NUL-bytes mixed with the cache-entries, which\n> should give optimal performance. I haven't thought much about\n> corruption of the index, but I don't think we would need any additional\n> crc checksums or similar.\n>\n> The performance advantage seems very slim to none when using the dummy\n> entries to avoid the comparison though, so I don't think it makes sense\n> to make the index more complicated for a very small speed-up.\n\nAgreed.\n\n> Test simpleapi HEAD this tree\n> -------------------------------------------------------------------------------------------------------------\n> 0003.2: v[23]: update-index 3.22(2.61+0.58) 3.20(2.56+0.61) -0.6% 3.22(2.67+0.53) +0.0%\n> 0003.3: v[23]: grep nonexistent -- subdir 1.65(1.36+0.27) 1.65(1.34+0.30) +0.0% 1.67(1.34+0.31) +1.2%\n> 0003.4: v[23]: ls-files -- subdir 1.50(1.20+0.29) 1.54(1.18+0.34) +2.7% 1.53(1.22+0.30) +2.0%\n> 0003.6: v4: update-index 2.69(2.28+0.39) 2.70(2.21+0.47) +0.4% 2.70(2.27+0.41) +0.4%\n> 0003.7: v4: grep nonexistent -- subdir 1.33(0.98+0.34) 1.36(1.01+0.33) +2.3% 1.34(1.03+0.30) +0.8%\n> 0003.8: v4: ls-files -- subdir 1.21(0.86+0.33) 1.23(0.91+0.30) +1.7% 1.22(0.90+0.30) +0.8%\n> 0003.10: v5: update-index 2.12(1.58+0.52) 2.20(1.67+0.52) +3.8% 2.19(1.66+0.51) +3.3%\n> 0003.11: v5: ls-files 1.57(1.21+0.34) 1.55(1.20+0.33) -1.3% 1.52(1.21+0.29) -3.2%\n> 0003.12: v5: grep nonexistent -- subdir 0.08(0.06+0.01) 0.07(0.04+0.02) -12.5% 0.07(0.04+0.02) -12.5%\n> 0003.13: v5: ls-files -- subdir 0.07(0.04+0.01) 0.06(0.05+0.00) -14.3% 0.07(0.05+0.01) +0.0%\n-- \nDuy\n"},{"id":"224810","messageId":"87haf0qpld.fsf@gmail.com","threadId":"34419","inReplyTo":"CACsJy8C4r=5K3ZdzNnNJ95xS3_xnsTSH2bUyLJ+rwRv4Jxo2zw@mail.gmail.com","subject":"Re: [PATCH v2 12/19] read-cache: read index-v5","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2013-08-08T13:28:14Z","receivedAt":"2013-08-08T13:28:14Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"Duy Nguyen writes:\n\n> On Wed, Jul 17, 2013 at 3:11 PM, Thomas Gummerer wrote:\n>> Duy Nguyen writes:\n>>\n>> [..]\n>>\n>>>> +static int read_entries(struct index_state *istate, struct directory_entry **de,\n>>>> + unsigned int *entry_offset, void **mmap,\n>>>> + unsigned long mmap_size, unsigned int *nr,\n>>>> + unsigned int *foffsetblock)\n>>>> +{\n>>>> + struct cache_entry *head = NULL, *tail = NULL;\n>>>> + struct conflict_entry *conflict_queue;\n>>>> + struct cache_entry *ce;\n>>>> + int i;\n>>>> +\n>>>> + conflict_queue = NULL;\n>>>> + if (read_conflicts(&conflict_queue, *de, mmap, mmap_size) < 0)\n>>>> + return -1;\n>>>> + for (i = 0; i < (*de)->de_nfiles; i++) {\n>>>> + if (read_entry(&ce,\n>>>> + *de,\n>>>> + entry_offset,\n>>>> + mmap,\n>>>> + mmap_size,\n>>>> + foffsetblock) < 0)\n>>>> + return -1;\n>>>> + ce_queue_push(&head, &tail, ce);\n>>>> + *foffsetblock += 4;\n>>>> +\n>>>> + /*\n>>>> + * Add the conflicted entries at the end of the index file\n>>>> + * to the in memory format\n>>>> + */\n>>>> + if (conflict_queue &&\n>>>> + (conflict_queue->entries->flags & CONFLICT_CONFLICTED) != 0 &&\n>>>> + !cache_name_compare(conflict_queue->name, conflict_queue->namelen,\n>>>> + ce->name, ce_namelen(ce))) {\n>>>> + struct conflict_part *cp;\n>>>> + cp = conflict_queue->entries;\n>>>> + cp = cp->next;\n>>>> + while (cp) {\n>>>> + ce = convert_conflict_part(cp,\n>>>> + conflict_queue->name,\n>>>> + conflict_queue->namelen);\n>>>> + ce_queue_push(&head, &tail, ce);\n>>>> + conflict_part_head_remove(&cp);\n>>>> + }\n>>>> + conflict_entry_head_remove(&conflict_queue);\n>>>> + }\n>>>\n>>> I start to wonder if separating staged entries is a good idea. It\n>>> seems to make the code more complicated. The good point about conflict\n>>> section at the end of the file is you can just truncate() it out.\n>>> Another way is putting staged entries in fileentries, sorted\n>>> alphabetically then by stage number, and a flag indicating if the\n>>> entry is valid. When you remove resolve an entry, just set the flag to\n>>> invalid (partial write), so that read code will skip it.\n>>>\n>>> I think this approach is reasonably cheap (unless there are a lot of\n>>> conflicts) and it simplifies this piece of code. truncate() may be\n>>> overrated anyway. In my experience, I \"git add \" as soon as I\n>>> resolve (so that \"git diff\" shrinks). One entry at a time, one\n>>> index write at a time. I don't think I ever resolve everything then\n>>> \"git add -u .\", which is where truncate() shines because staged\n>>> entries are removed all at once. We should optimize for one file\n>>> resolution at a time, imo.\n>>\n>> Thanks for your comments. I'll address the other ones once we decided\n>> to do with the conflicts.\n>>\n>> It does make the code quite a bit more complicated, but also has one\n>> advantage that you overlooked.\n>\n> I did overlook, although my goal is to keep the code simpler, not more\n> comlicated. The thinking is if we can find everything in fileentries\n> table, the code here is simplified, so..\n>\n>> We wouldn't truncate() when resolving\n>> the conflicts. The resolve undo data is stored with the conflicts and\n>> therefore we could just flip a bit and set the stage of the cache-entry\n>> in the main index to 0 (always once we got partial writing). This can\n>> be fast both in case we resolve one entry at a time and when we resolve\n>> a lot of entries. The advantage is even bigger when we resolve one\n>> entry at a time, when we otherwise would have to re-write the index for\n>> each conflict resolution.\n>\n> If I understand it correctly, filentries can only contain stage-0 or\n> stage-1 entries, \"stage > 0\" entries are stored in conflict data. Once\n> a conflict is solved, you update the stage-1 entry in fileentries,\n> turning it to stage-0 and recalculate the entry checksum. Conflict\n> data remains there to function as the old REUC extension. Correct?\n\nCorrect.\n\n> First of all, if that's true, we only need 1 bit for stage in fileentries table.\n>\n> Secondly, you may get away with looking up to conflict data in this\n> function by storing all stages in fileentries (now we need 2-bit\n> stage), replicated in conflict data for reuc function. When you\n> resolve conflict, you flip stage-1 to stage-0, and flip (a new bit) to\n> mark stage-2 entry invalid so the code knows to skip it. Next time the\n> index is rewritten, invalid entries are removed, but we still have old\n> stage entries in conflict data. The flipping business is pretty much\n> what you plan anyway, but the reading code does not need to look at\n> both fileentries and conflict data at the same time.\n>\n> What do you think?\n\nI think that's a good idea, thanks. It may slow the index down a bit\nwhen there are a lot of conflicts, but I think once we get to the point\nthat it slows the index down noticeably, the user has much bigger\nproblems anyway and will probably not try to resolve them all.\n\nThe invalid entry flag could also be used when entries are only removed\nfrom the index, so we can just set that bit instead of re-writing the\nwhole index file.\n\nI'll try to implement this idea to see if the experiments match the\nexpectations and then send a re-roll.\n"},{"id":"224901","messageId":"87eha3qabc.fsf@gmail.com","threadId":"34419","inReplyTo":"CACsJy8C4r=5K3ZdzNnNJ95xS3_xnsTSH2bUyLJ+rwRv4Jxo2zw@mail.gmail.com","subject":"Re: [PATCH v2 12/19] read-cache: read index-v5","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2013-08-09T13:10:31Z","receivedAt":"2013-08-09T13:10:31Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"Duy Nguyen writes:\n\n> On Wed, Jul 17, 2013 at 3:11 PM, Thomas Gummerer wrote:\n>> Duy Nguyen writes:\n>>\n>> [..]\n>>\n>>>> +static int read_entries(struct index_state *istate, struct directory_entry **de,\n>>>> + unsigned int *entry_offset, void **mmap,\n>>>> + unsigned long mmap_size, unsigned int *nr,\n>>>> + unsigned int *foffsetblock)\n>>>> +{\n>>>> + struct cache_entry *head = NULL, *tail = NULL;\n>>>> + struct conflict_entry *conflict_queue;\n>>>> + struct cache_entry *ce;\n>>>> + int i;\n>>>> +\n>>>> + conflict_queue = NULL;\n>>>> + if (read_conflicts(&conflict_queue, *de, mmap, mmap_size) < 0)\n>>>> + return -1;\n>>>> + for (i = 0; i < (*de)->de_nfiles; i++) {\n>>>> + if (read_entry(&ce,\n>>>> + *de,\n>>>> + entry_offset,\n>>>> + mmap,\n>>>> + mmap_size,\n>>>> + foffsetblock) < 0)\n>>>> + return -1;\n>>>> + ce_queue_push(&head, &tail, ce);\n>>>> + *foffsetblock += 4;\n>>>> +\n>>>> + /*\n>>>> + * Add the conflicted entries at the end of the index file\n>>>> + * to the in memory format\n>>>> + */\n>>>> + if (conflict_queue &&\n>>>> + (conflict_queue->entries->flags & CONFLICT_CONFLICTED) != 0 &&\n>>>> + !cache_name_compare(conflict_queue->name, conflict_queue->namelen,\n>>>> + ce->name, ce_namelen(ce))) {\n>>>> + struct conflict_part *cp;\n>>>> + cp = conflict_queue->entries;\n>>>> + cp = cp->next;\n>>>> + while (cp) {\n>>>> + ce = convert_conflict_part(cp,\n>>>> + conflict_queue->name,\n>>>> + conflict_queue->namelen);\n>>>> + ce_queue_push(&head, &tail, ce);\n>>>> + conflict_part_head_remove(&cp);\n>>>> + }\n>>>> + conflict_entry_head_remove(&conflict_queue);\n>>>> + }\n>>>\n>>> I start to wonder if separating staged entries is a good idea. It\n>>> seems to make the code more complicated. The good point about conflict\n>>> section at the end of the file is you can just truncate() it out.\n>>> Another way is putting staged entries in fileentries, sorted\n>>> alphabetically then by stage number, and a flag indicating if the\n>>> entry is valid. When you remove resolve an entry, just set the flag to\n>>> invalid (partial write), so that read code will skip it.\n>>>\n>>> I think this approach is reasonably cheap (unless there are a lot of\n>>> conflicts) and it simplifies this piece of code. truncate() may be\n>>> overrated anyway. In my experience, I \"git add \" as soon as I\n>>> resolve (so that \"git diff\" shrinks). One entry at a time, one\n>>> index write at a time. I don't think I ever resolve everything then\n>>> \"git add -u .\", which is where truncate() shines because staged\n>>> entries are removed all at once. We should optimize for one file\n>>> resolution at a time, imo.\n>>\n>> Thanks for your comments. I'll address the other ones once we decided\n>> to do with the conflicts.\n>>\n>> It does make the code quite a bit more complicated, but also has one\n>> advantage that you overlooked.\n>\n> I did overlook, although my goal is to keep the code simpler, not more\n> comlicated. The thinking is if we can find everything in fileentries\n> table, the code here is simplified, so..\n>\n>> We wouldn't truncate() when resolving\n>> the conflicts. The resolve undo data is stored with the conflicts and\n>> therefore we could just flip a bit and set the stage of the cache-entry\n>> in the main index to 0 (always once we got partial writing). This can\n>> be fast both in case we resolve one entry at a time and when we resolve\n>> a lot of entries. The advantage is even bigger when we resolve one\n>> entry at a time, when we otherwise would have to re-write the index for\n>> each conflict resolution.\n>\n> If I understand it correctly, filentries can only contain stage-0 or\n> stage-1 entries, \"stage > 0\" entries are stored in conflict data. Once\n> a conflict is solved, you update the stage-1 entry in fileentries,\n> turning it to stage-0 and recalculate the entry checksum. Conflict\n> data remains there to function as the old REUC extension. Correct?\n>\n> First of all, if that's true, we only need 1 bit for stage in fileentries table.\n>\n> Secondly, you may get away with looking up to conflict data in this\n> function by storing all stages in fileentries (now we need 2-bit\n> stage), replicated in conflict data for reuc function. When you\n> resolve conflict, you flip stage-1 to stage-0, and flip (a new bit) to\n> mark stage-2 entry invalid so the code knows to skip it. Next time the\n> index is rewritten, invalid entries are removed, but we still have old\n> stage entries in conflict data. The flipping business is pretty much\n> what you plan anyway, but the reading code does not need to look at\n> both fileentries and conflict data at the same time.\n>\n> What do you think?\n\nI've now tried it out for my synthetic repository, and created ~115,000\nconflicts in it. It's a only a little slower even for that large number\nof conflicts (which I don't think will ever happen in practice) but the\ncode is definitely simpler, so I will go with this. The times for the\nold and new version are below.\n\nTest HEAD~1 this tree\n-----------------------------------------------------------------------------------\n0003.2: v[23]: update-index 3.50(2.87+0.61) 3.52(2.89+0.60) +0.6%\n0003.3: v[23]: grep nonexistent -- subdir 1.80(1.43+0.35) 1.86(1.50+0.34) +3.3%\n0003.4: v[23]: ls-files -- subdir 1.67(1.29+0.36) 1.70(1.31+0.37) +1.8%\n0003.6: v4: update-index 2.97(2.44+0.51) 3.00(2.50+0.48) +1.0%\n0003.7: v4: grep nonexistent -- subdir 1.45(1.12+0.31) 1.48(1.06+0.40) +2.1%\n0003.8: v4: ls-files -- subdir 1.33(0.99+0.33) 1.35(1.02+0.32) +1.5%\n0003.10: v5: update-index 2.42(1.87+0.54) 2.50(1.84+0.63) +3.3%\n0003.11: v5: ls-files 1.75(1.38+0.35) 1.80(1.37+0.41) +2.9%\n0003.12: v5: grep nonexistent -- subdir 0.07(0.05+0.01) 0.07(0.05+0.01) +0.0%\n0003.13: v5: ls-files -- subdir 0.07(0.05+0.01) 0.07(0.06+0.00) +0.0%\n"}]}