{"thread":{"id":"31208","subject":"[PATCH/RFC v3 05/13] Make in-memory format aware of stat_crc","startedAt":"2012-08-08T11:17:48Z","lastAt":"2012-08-10T15:40:15Z","messageCount":34,"participants":["Thomas Gummerer","Nguyen Thai Ngoc Duy","Johannes Sixt","Junio C Hamano","Thomas Rast"],"isPatch":true,"patchVersion":3,"patchTotal":13},"messages":[{"id":"196650","messageId":"1344424681-31469-1-git-send-email-t.gummerer@gmail.com","threadId":"31208","inReplyTo":null,"subject":"[PATCH/RFC v3 0/13] Introduce index file format version 5","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2012-08-08T11:17:48Z","receivedAt":"2012-08-08T11:17:48Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"Previous rounds of this series were on $gmane/202752 and $gmane/202923.\n\nThis round includes a major change, as it is splitting up read-cache.c\ninto read-cache.c, read-cache-v2.c and read-cache-v5.c. It uses\nindex->ops to call the specific functions as suggested by Duy.\n\nIt also changes the index format slightly, removing the size from the\nstat_crc and adding it as separate field, as suggested by Robin. This\nmakes the format incompatible with the one in the previous two series.\n\n[PATCH/RFC 01/13] Move index v2 specific functions to their own file\nThis previously was patch 1/2/3/4, but since the code is now moved\nto a different file, it seems to make more sense to do it in one patch.\nAlso moved the size check to the verify_hdr functions for the respective\nindex formats.\n\n[PATCH/RFC 02/13] t2104: Don't fail for index versions other than\nDo the update index at the very beginning.\n\n[PATCH/RFC 03/13] t3700: Avoid interfering with the racy code\nInstead of sleeping for a second, test-chmtime now uses an absolute\ntime, to avoid the racy code.\n\n[PATCH/RFC 04/13] Add documentation of the index-v5 file format\nAdded the size to the cache-entries.\n\n[PATCH/RFC 05/13] Make in-memory format aware of stat_crc\nRemoved size from the stat_crc.\n\n[PATCH/RFC 06/13] Read index-v5\nChanged the pointer arithmetic to not use void * directly, by\ncasting it to a char * first.\n\n[PATCH/RFC 07/13] Read resolve-undo data\n[PATCH/RFC 08/13] Read cache-tree in index-v5\n[PATCH/RFC 09/13] Write index-v5\n[PATCH/RFC 10/13] Write index-v5 cache-tree data\n[PATCH/RFC 11/13] Write resolve-undo data for index-v5\n[PATCH/RFC 12/13] update-index.c: always rewrite the index when\nInstead of using a force-rewrite option, always rewrite the index\nif a index-version is given.\n\n[PATCH/RFC 13/13] p0002-index.sh: add perf test for the index\nInstead of using the force-rewrite option, which was removed from\nupdate-index, use git update-index --index-version=n for doing the\npreformance testing.\n"},{"id":"196657","messageId":"1344424681-31469-2-git-send-email-t.gummerer@gmail.com","threadId":"31208","inReplyTo":"1344424681-31469-1-git-send-email-t.gummerer@gmail.com","subject":"[PATCH/RFC v3 01/13] Move index v2 specific functions to their own file","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2012-08-08T11:17:49Z","receivedAt":"2012-08-08T11:17:49Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"Move index version 2 specific functions to their own file,\nto prepare for the addition of a new index file format. With\nthe split into two files we have the non-index specific\nfunctions in read-cache.c and the index-v2 specific functions\nin read-cache-v2.c\n\nHelped-by: Nguyen Thai Ngoc Duy \nSigned-off-by: Thomas Gummerer \n---\n Makefile | 2 +\n cache.h | 13 +-\n read-cache-v2.c | 581 +++++++++++++++++++++++++++++++++++++++++++++++\n read-cache.c | 613 +++-----------------------------------------------\n read-cache.h | 57 +++++\n test-index-version.c | 7 +-\n 6 files changed, 683 insertions(+), 590 deletions(-)\n create mode 100644 read-cache-v2.c\n create mode 100644 read-cache.h\n\ndiff --git a/Makefile b/Makefile\nindex 4b58b91..b4a7c73 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -645,6 +645,7 @@ LIB_H += progress.h\n LIB_H += prompt.h\n LIB_H += quote.h\n LIB_H += reachable.h\n+LIB_H += read-cache.h\n LIB_H += reflog-walk.h\n LIB_H += refs.h\n LIB_H += remote.h\n@@ -768,6 +769,7 @@ LIB_OBJS += prompt.o\n LIB_OBJS += quote.o\n LIB_OBJS += reachable.o\n LIB_OBJS += read-cache.o\n+LIB_OBJS += read-cache-v2.o\n LIB_OBJS += reflog-walk.o\n LIB_OBJS += refs.o\n LIB_OBJS += remote.o\ndiff --git a/cache.h b/cache.h\nindex 67f28b4..c77cdbe 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -94,16 +94,8 @@ unsigned long git_deflate_bound(git_zstream *, unsigned long);\n */\n #define DEFAULT_GIT_PORT 9418\n \n-/*\n- * Basic data structures for the directory cache\n- */\n \n #define CACHE_SIGNATURE 0x44495243\t/* \"DIRC\" */\n-struct cache_header {\n-\tunsigned int hdr_signature;\n-\tunsigned int hdr_version;\n-\tunsigned int hdr_entries;\n-};\n \n #define INDEX_FORMAT_LB 2\n #define INDEX_FORMAT_UB 4\n@@ -267,6 +259,7 @@ struct index_state {\n \tunsigned name_hash_initialized : 1,\n \t\t initialized : 1;\n \tstruct hash_table name_hash;\n+\tstruct index_ops *ops;\n };\n \n extern struct index_state the_index;\n@@ -471,8 +464,8 @@ extern int index_name_is_other(const struct index_state *, const char *, int);\n #define CE_MATCH_RACY_IS_DIRTY\t\t02\n /* do stat comparison even if CE_SKIP_WORKTREE is true */\n #define CE_MATCH_IGNORE_SKIP_WORKTREE\t04\n-extern int ie_match_stat(const struct index_state *, struct cache_entry *, struct stat *, unsigned int);\n-extern int ie_modified(const struct index_state *, struct cache_entry *, struct stat *, unsigned int);\n+extern int ie_match_stat(struct index_state *, struct cache_entry *, struct stat *, unsigned int);\n+extern int ie_modified(struct index_state *, struct cache_entry *, struct stat *, unsigned int);\n \n struct pathspec {\n \tconst char **raw; /* get_pathspec() result, not freed by free_pathspec() */\ndiff --git a/read-cache-v2.c b/read-cache-v2.c\nnew file mode 100644\nindex 0000000..38f1791\n--- /dev/null\n+++ b/read-cache-v2.c\n@@ -0,0 +1,581 @@\n+#include \"cache.h\"\n+#include \"read-cache.h\"\n+#include \"resolve-undo.h\"\n+#include \"cache-tree.h\"\n+#include \"varint.h\"\n+\n+/* Mask for the name length in ce_flags in the on-disk index */\n+#define CE_NAMEMASK (0x0fff)\n+\n+struct cache_header {\n+\tunsigned int hdr_entries;\n+};\n+\n+/*****************************************************************\n+ * Index File I/O\n+ *****************************************************************/\n+\n+/*\n+ * dev/ino/uid/gid/size are also just tracked to the low 32 bits\n+ * Again - this is just a (very strong in practice) heuristic that\n+ * the inode hasn't changed.\n+ *\n+ * We save the fields in big-endian order to allow using the\n+ * index file over NFS transparently.\n+ */\n+struct ondisk_cache_entry {\n+\tstruct cache_time ctime;\n+\tstruct cache_time mtime;\n+\tunsigned int dev;\n+\tunsigned int ino;\n+\tunsigned int mode;\n+\tunsigned int uid;\n+\tunsigned int gid;\n+\tunsigned int size;\n+\tunsigned char sha1[20];\n+\tunsigned short flags;\n+\tchar name[FLEX_ARRAY]; /* more */\n+};\n+\n+/*\n+ * This struct is used when CE_EXTENDED bit is 1\n+ * The struct must match ondisk_cache_entry exactly from\n+ * ctime till flags\n+ */\n+struct ondisk_cache_entry_extended {\n+\tstruct cache_time ctime;\n+\tstruct cache_time mtime;\n+\tunsigned int dev;\n+\tunsigned int ino;\n+\tunsigned int mode;\n+\tunsigned int uid;\n+\tunsigned int gid;\n+\tunsigned int size;\n+\tunsigned char sha1[20];\n+\tunsigned short flags;\n+\tunsigned short flags2;\n+\tchar name[FLEX_ARRAY]; /* more */\n+};\n+\n+/* These are only used for v3 or lower */\n+#define align_flex_name(STRUCT,len) ((offsetof(struct STRUCT,name) + (len) + 8) & ~7)\n+#define ondisk_cache_entry_size(len) align_flex_name(ondisk_cache_entry,len)\n+#define ondisk_cache_entry_extended_size(len) align_flex_name(ondisk_cache_entry_extended,len)\n+#define ondisk_ce_size(ce) (((ce)->ce_flags & CE_EXTENDED) ? \\\n+\t\t\t ondisk_cache_entry_extended_size(ce_namelen(ce)) : \\\n+\t\t\t ondisk_cache_entry_size(ce_namelen(ce)))\n+\n+static int verify_hdr(void *mmap, unsigned long size)\n+{\n+\tgit_SHA_CTX c;\n+\tunsigned char sha1[20];\n+\n+\tif (size < sizeof(struct cache_version_header)\n+\t\t\t+ sizeof(struct cache_header) + 20)\n+\t\tdie(\"index file smaller than expected\");\n+\n+\tgit_SHA1_Init(&c);\n+\tgit_SHA1_Update(&c, mmap, size - 20);\n+\tgit_SHA1_Final(sha1, &c);\n+\tif (hashcmp(sha1, (unsigned char *)mmap + size - 20))\n+\t\treturn error(\"bad index file sha1 signature\");\n+\treturn 0;\n+}\n+\n+static int match_stat_basic(struct cache_entry *ce,\n+\t\t\t\tstruct stat *st,\n+\t\t\t\tint changed)\n+{\n+\tif (ce->ce_mtime.sec != (unsigned int)st->st_mtime)\n+\t\tchanged |= MTIME_CHANGED;\n+\tif (trust_ctime && ce->ce_ctime.sec != (unsigned int)st->st_ctime)\n+\t\tchanged |= CTIME_CHANGED;\n+\n+#ifdef USE_NSEC\n+\tif (ce->ce_mtime.nsec != ST_MTIME_NSEC(*st))\n+\t\tchanged |= MTIME_CHANGED;\n+\tif (trust_ctime && ce->ce_ctime.nsec != ST_CTIME_NSEC(*st))\n+\t\tchanged |= CTIME_CHANGED;\n+#endif\n+\n+\tif (ce->ce_uid != (unsigned int) st->st_uid ||\n+\t ce->ce_gid != (unsigned int) st->st_gid)\n+\t\tchanged |= OWNER_CHANGED;\n+\tif (ce->ce_ino != (unsigned int) st->st_ino)\n+\t\tchanged |= INODE_CHANGED;\n+\n+#ifdef USE_STDEV\n+\t/*\n+\t * st_dev breaks on network filesystems where different\n+\t * clients will have different views of what \"device\"\n+\t * the filesystem is on\n+\t */\n+\tif (ce->ce_dev != (unsigned int) st->st_dev)\n+\t\tchanged |= INODE_CHANGED;\n+#endif\n+\n+\tif (ce->ce_size != (unsigned int) st->st_size)\n+\t\tchanged |= DATA_CHANGED;\n+\n+\t/* Racily smudged entry? */\n+\tif (!ce->ce_size) {\n+\t\tif (!is_empty_blob_sha1(ce->sha1))\n+\t\t\tchanged |= DATA_CHANGED;\n+\t}\n+\n+\treturn changed;\n+}\n+\n+static struct cache_entry *cache_entry_from_ondisk(struct ondisk_cache_entry *ondisk,\n+\t\t\t\t\t\t unsigned int flags,\n+\t\t\t\t\t\t const char *name,\n+\t\t\t\t\t\t size_t len)\n+{\n+\tstruct cache_entry *ce = xmalloc(cache_entry_size(len));\n+\n+\tce->ce_ctime.sec = ntoh_l(ondisk->ctime.sec);\n+\tce->ce_mtime.sec = ntoh_l(ondisk->mtime.sec);\n+\tce->ce_ctime.nsec = ntoh_l(ondisk->ctime.nsec);\n+\tce->ce_mtime.nsec = ntoh_l(ondisk->mtime.nsec);\n+\tce->ce_dev = ntoh_l(ondisk->dev);\n+\tce->ce_ino = ntoh_l(ondisk->ino);\n+\tce->ce_mode = ntoh_l(ondisk->mode);\n+\tce->ce_uid = ntoh_l(ondisk->uid);\n+\tce->ce_gid = ntoh_l(ondisk->gid);\n+\tce->ce_size = ntoh_l(ondisk->size);\n+\tce->ce_flags = flags & ~CE_NAMEMASK;\n+\tce->ce_namelen = len;\n+\thashcpy(ce->sha1, ondisk->sha1);\n+\tmemcpy(ce->name, name, len);\n+\tce->name[len] = '\\0';\n+\treturn ce;\n+}\n+\n+/*\n+ * Adjacent cache entries tend to share the leading paths, so it makes\n+ * sense to only store the differences in later entries. In the v4\n+ * on-disk format of the index, each on-disk cache entry stores the\n+ * number of bytes to be stripped from the end of the previous name,\n+ * and the bytes to append to the result, to come up with its name.\n+ */\n+static unsigned long expand_name_field(struct strbuf *name, const char *cp_)\n+{\n+\tconst unsigned char *ep, *cp = (const unsigned char *)cp_;\n+\tsize_t len = decode_varint(&cp);\n+\n+\tif (name->len < len)\n+\t\tdie(\"malformed name field in the index\");\n+\tstrbuf_remove(name, name->len - len, len);\n+\tfor (ep = cp; *ep; ep++)\n+\t\t; /* find the end */\n+\tstrbuf_add(name, cp, ep - cp);\n+\treturn (const char *)ep + 1 - cp_;\n+}\n+\n+static struct cache_entry *create_from_disk(struct ondisk_cache_entry *ondisk,\n+\t\t\t\t\t unsigned long *ent_size,\n+\t\t\t\t\t struct strbuf *previous_name)\n+{\n+\tstruct cache_entry *ce;\n+\tsize_t len;\n+\tconst char *name;\n+\tunsigned int flags;\n+\n+\t/* On-disk flags are just 16 bits */\n+\tflags = ntoh_s(ondisk->flags);\n+\tlen = flags & CE_NAMEMASK;\n+\n+\tif (flags & CE_EXTENDED) {\n+\t\tstruct ondisk_cache_entry_extended *ondisk2;\n+\t\tint extended_flags;\n+\t\tondisk2 = (struct ondisk_cache_entry_extended *)ondisk;\n+\t\textended_flags = ntoh_s(ondisk2->flags2) << 16;\n+\t\t/* We do not yet understand any bit out of CE_EXTENDED_FLAGS */\n+\t\tif (extended_flags & ~CE_EXTENDED_FLAGS)\n+\t\t\tdie(\"Unknown index entry format %08x\", extended_flags);\n+\t\tflags |= extended_flags;\n+\t\tname = ondisk2->name;\n+\t}\n+\telse\n+\t\tname = ondisk->name;\n+\n+\tif (!previous_name) {\n+\t\t/* v3 and earlier */\n+\t\tif (len == CE_NAMEMASK)\n+\t\t\tlen = strlen(name);\n+\t\tce = cache_entry_from_ondisk(ondisk, flags, name, len);\n+\n+\t\t*ent_size = ondisk_ce_size(ce);\n+\t} else {\n+\t\tunsigned long consumed;\n+\t\tconsumed = expand_name_field(previous_name, name);\n+\t\tce = cache_entry_from_ondisk(ondisk, flags,\n+\t\t\t\t\t previous_name->buf,\n+\t\t\t\t\t previous_name->len);\n+\n+\t\t*ent_size = (name - ((char *)ondisk)) + consumed;\n+\t}\n+\treturn ce;\n+}\n+\n+static int read_index_extension(struct index_state *istate,\n+\t\t\t\tconst char *ext, void *data, unsigned long sz)\n+{\n+\tswitch (CACHE_EXT(ext)) {\n+\tcase CACHE_EXT_TREE:\n+\t\tistate->cache_tree = cache_tree_read(data, sz);\n+\t\tbreak;\n+\tcase CACHE_EXT_RESOLVE_UNDO:\n+\t\tistate->resolve_undo = resolve_undo_read(data, sz);\n+\t\tbreak;\n+\tdefault:\n+\t\tif (*ext < 'A' || 'Z' < *ext)\n+\t\t\treturn error(\"index uses %.4s extension, which we do not understand\",\n+\t\t\t\t ext);\n+\t\tfprintf(stderr, \"ignoring %.4s extension\\n\", ext);\n+\t\tbreak;\n+\t}\n+\treturn 0;\n+}\n+\n+static void read_index_v2(struct index_state *istate, void *mmap, int mmap_size, int fd)\n+{\n+\tint i;\n+\tunsigned long src_offset;\n+\tstruct cache_version_header *hdr;\n+\tstruct cache_header *hdr_v2;\n+\tstruct strbuf previous_name_buf = STRBUF_INIT, *previous_name;\n+\n+\thdr = mmap;\n+\thdr_v2 = (struct cache_header *)((char *)mmap + sizeof(*hdr));\n+\tistate->version = ntohl(hdr->hdr_version);\n+\tistate->cache_nr = ntohl(hdr_v2->hdr_entries);\n+\tistate->cache_alloc = alloc_nr(istate->cache_nr);\n+\tistate->cache = xcalloc(istate->cache_alloc, sizeof(struct cache_entry *));\n+\tistate->initialized = 1;\n+\n+\tif (istate->version == 4)\n+\t\tprevious_name = &previous_name_buf;\n+\telse\n+\t\tprevious_name = NULL;\n+\n+\tsrc_offset = sizeof(*hdr) + sizeof(*hdr_v2);\n+\tfor (i = 0; i < istate->cache_nr; i++) {\n+\t\tstruct ondisk_cache_entry *disk_ce;\n+\t\tstruct cache_entry *ce;\n+\t\tunsigned long consumed;\n+\n+\t\tdisk_ce = (struct ondisk_cache_entry *)((char *)mmap + src_offset);\n+\t\tce = create_from_disk(disk_ce, &consumed, previous_name);\n+\t\tset_index_entry(istate, i, ce);\n+\n+\t\tsrc_offset += consumed;\n+\t}\n+\tstrbuf_release(&previous_name_buf);\n+\n+\twhile (src_offset <= mmap_size - 20 - 8) {\n+\t\t/* After an array of active_nr index entries,\n+\t\t * there can be arbitrary number of extended\n+\t\t * sections, each of which is prefixed with\n+\t\t * extension name (4-byte) and section length\n+\t\t * in 4-byte network byte order.\n+\t\t */\n+\t\tuint32_t extsize;\n+\t\tmemcpy(&extsize, (char *)mmap + src_offset + 4, 4);\n+\t\textsize = ntohl(extsize);\n+\t\tif (read_index_extension(istate,\n+\t\t\t\t\t(const char *) mmap + src_offset,\n+\t\t\t\t\t(char *) mmap + src_offset + 8,\n+\t\t\t\t\textsize) < 0)\n+\t\t\tgoto unmap;\n+\t\tsrc_offset += 8;\n+\t\tsrc_offset += extsize;\n+\t}\n+\treturn;\n+unmap:\n+\tmunmap(mmap, mmap_size);\n+\tdie(\"index file corrupt\");\n+}\n+\n+#define WRITE_BUFFER_SIZE 8192\n+static unsigned char write_buffer[WRITE_BUFFER_SIZE];\n+static unsigned long write_buffer_len;\n+\n+static int ce_write_flush(git_SHA_CTX *context, int fd)\n+{\n+\tunsigned int buffered = write_buffer_len;\n+\tif (buffered) {\n+\t\tgit_SHA1_Update(context, write_buffer, buffered);\n+\t\tif (write_in_full(fd, write_buffer, buffered) != buffered)\n+\t\t\treturn -1;\n+\t\twrite_buffer_len = 0;\n+\t}\n+\treturn 0;\n+}\n+\n+static int ce_write(git_SHA_CTX *context, int fd, void *data, unsigned int len)\n+{\n+\twhile (len) {\n+\t\tunsigned int buffered = write_buffer_len;\n+\t\tunsigned int partial = WRITE_BUFFER_SIZE - buffered;\n+\t\tif (partial > len)\n+\t\t\tpartial = len;\n+\t\tmemcpy(write_buffer + buffered, data, partial);\n+\t\tbuffered += partial;\n+\t\tif (buffered == WRITE_BUFFER_SIZE) {\n+\t\t\twrite_buffer_len = buffered;\n+\t\t\tif (ce_write_flush(context, fd))\n+\t\t\t\treturn -1;\n+\t\t\tbuffered = 0;\n+\t\t}\n+\t\twrite_buffer_len = buffered;\n+\t\tlen -= partial;\n+\t\tdata = (char *) data + partial;\n+\t}\n+\treturn 0;\n+}\n+\n+static int write_index_ext_header(git_SHA_CTX *context, int fd,\n+\t\t\t\t unsigned int ext, unsigned int sz)\n+{\n+\text = htonl(ext);\n+\tsz = htonl(sz);\n+\treturn ((ce_write(context, fd, &ext, 4) < 0) ||\n+\t\t(ce_write(context, fd, &sz, 4) < 0)) ? -1 : 0;\n+}\n+\n+static int ce_flush(git_SHA_CTX *context, int fd)\n+{\n+\tunsigned int left = write_buffer_len;\n+\n+\tif (left) {\n+\t\twrite_buffer_len = 0;\n+\t\tgit_SHA1_Update(context, write_buffer, left);\n+\t}\n+\n+\t/* Flush first if not enough space for SHA1 signature */\n+\tif (left + 20 > WRITE_BUFFER_SIZE) {\n+\t\tif (write_in_full(fd, write_buffer, left) != left)\n+\t\t\treturn -1;\n+\t\tleft = 0;\n+\t}\n+\n+\t/* Append the SHA1 signature at the end */\n+\tgit_SHA1_Final(write_buffer + left, context);\n+\tleft += 20;\n+\treturn (write_in_full(fd, write_buffer, left) != left) ? -1 : 0;\n+}\n+\n+static void ce_smudge_racily_clean_entry(struct index_state *istate, struct cache_entry *ce)\n+{\n+\t/*\n+\t * The only thing we care about in this function is to smudge the\n+\t * falsely clean entry due to touch-update-touch race, so we leave\n+\t * everything else as they are. We are called for entries whose\n+\t * ce_mtime match the index file mtime.\n+\t *\n+\t * Note that this actually does not do much for gitlinks, for\n+\t * which ce_match_stat_basic() always goes to the actual\n+\t * contents. The caller checks with is_racy_timestamp() which\n+\t * always says \"no\" for gitlinks, so we are not called for them ;-)\n+\t */\n+\tstruct stat st;\n+\n+\tif (lstat(ce->name, &st) < 0)\n+\t\treturn;\n+\tif (ce_match_stat_basic(istate, ce, &st))\n+\t\treturn;\n+\tif (ce_modified_check_fs(ce, &st)) {\n+\t\t/* This is \"racily clean\"; smudge it. Note that this\n+\t\t * is a tricky code. At first glance, it may appear\n+\t\t * that it can break with this sequence:\n+\t\t *\n+\t\t * $ echo xyzzy >frotz\n+\t\t * $ git-update-index --add frotz\n+\t\t * $ : >frotz\n+\t\t * $ sleep 3\n+\t\t * $ echo filfre >nitfol\n+\t\t * $ git-update-index --add nitfol\n+\t\t *\n+\t\t * but it does not. When the second update-index runs,\n+\t\t * it notices that the entry \"frotz\" has the same timestamp\n+\t\t * as index, and if we were to smudge it by resetting its\n+\t\t * size to zero here, then the object name recorded\n+\t\t * in index is the 6-byte file but the cached stat information\n+\t\t * becomes zero --- which would then match what we would\n+\t\t * obtain from the filesystem next time we stat(\"frotz\").\n+\t\t *\n+\t\t * However, the second update-index, before calling\n+\t\t * this function, notices that the cached size is 6\n+\t\t * bytes and what is on the filesystem is an empty\n+\t\t * file, and never calls us, so the cached size information\n+\t\t * for \"frotz\" stays 6 which does not match the filesystem.\n+\t\t */\n+\t\tce->ce_size = 0;\n+\t}\n+}\n+\n+/* Copy miscellaneous fields but not the name */\n+static char *copy_cache_entry_to_ondisk(struct ondisk_cache_entry *ondisk,\n+\t\t\t\t struct cache_entry *ce)\n+{\n+\tshort flags;\n+\n+\tondisk->ctime.sec = htonl(ce->ce_ctime.sec);\n+\tondisk->mtime.sec = htonl(ce->ce_mtime.sec);\n+\tondisk->ctime.nsec = htonl(ce->ce_ctime.nsec);\n+\tondisk->mtime.nsec = htonl(ce->ce_mtime.nsec);\n+\tondisk->dev = htonl(ce->ce_dev);\n+\tondisk->ino = htonl(ce->ce_ino);\n+\tondisk->mode = htonl(ce->ce_mode);\n+\tondisk->uid = htonl(ce->ce_uid);\n+\tondisk->gid = htonl(ce->ce_gid);\n+\tondisk->size = htonl(ce->ce_size);\n+\thashcpy(ondisk->sha1, ce->sha1);\n+\n+\tflags = ce->ce_flags;\n+\tflags |= (ce_namelen(ce) >= CE_NAMEMASK ? CE_NAMEMASK : ce_namelen(ce));\n+\tondisk->flags = htons(flags);\n+\tif (ce->ce_flags & CE_EXTENDED) {\n+\t\tstruct ondisk_cache_entry_extended *ondisk2;\n+\t\tondisk2 = (struct ondisk_cache_entry_extended *)ondisk;\n+\t\tondisk2->flags2 = htons((ce->ce_flags & CE_EXTENDED_FLAGS) >> 16);\n+\t\treturn ondisk2->name;\n+\t}\n+\telse {\n+\t\treturn ondisk->name;\n+\t}\n+}\n+\n+static int ce_write_entry(git_SHA_CTX *c, int fd, struct cache_entry *ce,\n+\t\t\t struct strbuf *previous_name)\n+{\n+\tint size;\n+\tstruct ondisk_cache_entry *ondisk;\n+\tchar *name;\n+\tint result;\n+\n+\tif (!previous_name) {\n+\t\tsize = ondisk_ce_size(ce);\n+\t\tondisk = xcalloc(1, size);\n+\t\tname = copy_cache_entry_to_ondisk(ondisk, ce);\n+\t\tmemcpy(name, ce->name, ce_namelen(ce));\n+\t} else {\n+\t\tint common, to_remove, prefix_size;\n+\t\tunsigned char to_remove_vi[16];\n+\t\tfor (common = 0;\n+\t\t (ce->name[common] &&\n+\t\t common < previous_name->len &&\n+\t\t ce->name[common] == previous_name->buf[common]);\n+\t\t common++)\n+\t\t\t; /* still matching */\n+\t\tto_remove = previous_name->len - common;\n+\t\tprefix_size = encode_varint(to_remove, to_remove_vi);\n+\n+\t\tif (ce->ce_flags & CE_EXTENDED)\n+\t\t\tsize = offsetof(struct ondisk_cache_entry_extended, name);\n+\t\telse\n+\t\t\tsize = offsetof(struct ondisk_cache_entry, name);\n+\t\tsize += prefix_size + (ce_namelen(ce) - common + 1);\n+\n+\t\tondisk = xcalloc(1, size);\n+\t\tname = copy_cache_entry_to_ondisk(ondisk, ce);\n+\t\tmemcpy(name, to_remove_vi, prefix_size);\n+\t\tmemcpy(name + prefix_size, ce->name + common, ce_namelen(ce) - common);\n+\n+\t\tstrbuf_splice(previous_name, common, to_remove,\n+\t\t\t ce->name + common, ce_namelen(ce) - common);\n+\t}\n+\n+\tresult = ce_write(c, fd, ondisk, size);\n+\tfree(ondisk);\n+\treturn result;\n+}\n+\n+static int write_index_v2(struct index_state *istate, int newfd)\n+{\n+\tgit_SHA_CTX c;\n+\tstruct cache_version_header hdr;\n+\tstruct cache_header hdr_v2;\n+\tint i, err, removed, extended, hdr_version;\n+\tstruct cache_entry **cache = istate->cache;\n+\tint entries = istate->cache_nr;\n+\tstruct stat st;\n+\tstruct strbuf previous_name_buf = STRBUF_INIT, *previous_name;\n+\n+\tfor (i = removed = extended = 0; i < entries; i++) {\n+\t\tif (cache[i]->ce_flags & CE_REMOVE)\n+\t\t\tremoved++;\n+\n+\t\t/* reduce extended entries if possible */\n+\t\tcache[i]->ce_flags &= ~CE_EXTENDED;\n+\t\tif (cache[i]->ce_flags & CE_EXTENDED_FLAGS) {\n+\t\t\textended++;\n+\t\t\tcache[i]->ce_flags |= CE_EXTENDED;\n+\t\t}\n+\t}\n+\n+\t/* demote version 3 to version 2 when the latter suffices */\n+\tif (istate->version == 3 || istate->version == 2)\n+\t\tistate->version = extended ? 3 : 2;\n+\n+\thdr_version = istate->version;\n+\n+\thdr.hdr_signature = htonl(CACHE_SIGNATURE);\n+\thdr.hdr_version = htonl(hdr_version);\n+\thdr_v2.hdr_entries = htonl(entries - removed);\n+\n+\tgit_SHA1_Init(&c);\n+\tif (ce_write(&c, newfd, &hdr, sizeof(hdr)) < 0)\n+\t\treturn -1;\n+\tif (ce_write(&c, newfd, &hdr_v2, sizeof(hdr_v2)) < 0)\n+\t\treturn -1;\n+\n+\tprevious_name = (hdr_version == 4) ? &previous_name_buf : NULL;\n+\tfor (i = 0; i < entries; i++) {\n+\t\tstruct cache_entry *ce = cache[i];\n+\t\tif (ce->ce_flags & CE_REMOVE)\n+\t\t\tcontinue;\n+\t\tif (!ce_uptodate(ce) && is_racy_timestamp(istate, ce))\n+\t\t\tce_smudge_racily_clean_entry(istate, ce);\n+\t\tif (ce_write_entry(&c, newfd, ce, previous_name) < 0)\n+\t\t\treturn -1;\n+\t}\n+\tstrbuf_release(&previous_name_buf);\n+\n+\t/* Write extension data here */\n+\tif (istate->cache_tree) {\n+\t\tstruct strbuf sb = STRBUF_INIT;\n+\n+\t\tcache_tree_write(&sb, istate->cache_tree);\n+\t\terr = write_index_ext_header(&c, newfd, CACHE_EXT_TREE, sb.len) < 0\n+\t\t\t|| ce_write(&c, newfd, sb.buf, sb.len) < 0;\n+\t\tstrbuf_release(&sb);\n+\t\tif (err)\n+\t\t\treturn -1;\n+\t}\n+\tif (istate->resolve_undo) {\n+\t\tstruct strbuf sb = STRBUF_INIT;\n+\n+\t\tresolve_undo_write(&sb, istate->resolve_undo);\n+\t\terr = write_index_ext_header(&c, newfd, CACHE_EXT_RESOLVE_UNDO,\n+\t\t\t\t\t sb.len) < 0\n+\t\t\t|| ce_write(&c, newfd, sb.buf, sb.len) < 0;\n+\t\tstrbuf_release(&sb);\n+\t\tif (err)\n+\t\t\treturn -1;\n+\t}\n+\n+\tif (ce_flush(&c, newfd) || fstat(newfd, &st))\n+\t\treturn -1;\n+\tistate->timestamp.sec = (unsigned int)st.st_mtime;\n+\tistate->timestamp.nsec = ST_MTIME_NSEC(st);\n+\treturn 0;\n+}\n+\n+struct index_ops v2_ops = {\n+\tmatch_stat_basic,\n+\tverify_hdr,\n+\tread_index_v2,\n+\twrite_index_v2\n+};\ndiff --git a/read-cache.c b/read-cache.c\nindex 2f8159f..125e6a0 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -5,6 +5,7 @@\n */\n #define NO_THE_INDEX_COMPATIBILITY_MACROS\n #include \"cache.h\"\n+#include \"read-cache.h\"\n #include \"cache-tree.h\"\n #include \"refs.h\"\n #include \"dir.h\"\n@@ -17,26 +18,9 @@\n \n static struct cache_entry *refresh_cache_entry(struct cache_entry *ce, int really);\n \n-/* Mask for the name length in ce_flags in the on-disk index */\n-\n-#define CE_NAMEMASK (0x0fff)\n-\n-/* Index extensions.\n- *\n- * The first letter should be 'A'..'Z' for extensions that are not\n- * necessary for a correct operation (i.e. optimization data).\n- * When new extensions are added that _needs_ to be understood in\n- * order to correctly interpret the index file, pick character that\n- * is outside the range, to cause the reader to abort.\n- */\n-\n-#define CACHE_EXT(s) ( (s[0]<<24)|(s[1]<<16)|(s[2]<<8)|(s[3]) )\n-#define CACHE_EXT_TREE 0x54524545\t/* \"TREE\" */\n-#define CACHE_EXT_RESOLVE_UNDO 0x52455543 /* \"REUC\" */\n-\n struct index_state the_index;\n \n-static void set_index_entry(struct index_state *istate, int nr, struct cache_entry *ce)\n+void set_index_entry(struct index_state *istate, int nr, struct cache_entry *ce)\n {\n \tistate->cache[nr] = ce;\n \tadd_name_hash(istate, ce);\n@@ -143,7 +127,7 @@ static int ce_compare_gitlink(struct cache_entry *ce)\n \treturn hashcmp(sha1, ce->sha1);\n }\n \n-static int ce_modified_check_fs(struct cache_entry *ce, struct stat *st)\n+int ce_modified_check_fs(struct cache_entry *ce, struct stat *st)\n {\n \tswitch (st->st_mode & S_IFMT) {\n \tcase S_IFREG:\n@@ -163,7 +147,17 @@ static int ce_modified_check_fs(struct cache_entry *ce, struct stat *st)\n \treturn 0;\n }\n \n-static int ce_match_stat_basic(struct cache_entry *ce, struct stat *st)\n+static void check_set_istate_ops(struct index_state *istate)\n+{\n+\tif (!istate->version)\n+\t\tistate->version = INDEX_FORMAT_DEFAULT;\n+\tif (!istate->ops)\n+\t\tif (istate->version >= 2 && istate->version <=4)\n+\t\t\tistate->ops = &v2_ops;\n+}\n+\n+int ce_match_stat_basic(struct index_state *istate,\n+\t\tstruct cache_entry *ce, struct stat *st)\n {\n \tunsigned int changed = 0;\n \n@@ -195,47 +189,13 @@ static int ce_match_stat_basic(struct cache_entry *ce, struct stat *st)\n \tdefault:\n \t\tdie(\"internal error: ce_mode is %o\", ce->ce_mode);\n \t}\n-\tif (ce->ce_mtime.sec != (unsigned int)st->st_mtime)\n-\t\tchanged |= MTIME_CHANGED;\n-\tif (trust_ctime && ce->ce_ctime.sec != (unsigned int)st->st_ctime)\n-\t\tchanged |= CTIME_CHANGED;\n-\n-#ifdef USE_NSEC\n-\tif (ce->ce_mtime.nsec != ST_MTIME_NSEC(*st))\n-\t\tchanged |= MTIME_CHANGED;\n-\tif (trust_ctime && ce->ce_ctime.nsec != ST_CTIME_NSEC(*st))\n-\t\tchanged |= CTIME_CHANGED;\n-#endif\n-\n-\tif (ce->ce_uid != (unsigned int) st->st_uid ||\n-\t ce->ce_gid != (unsigned int) st->st_gid)\n-\t\tchanged |= OWNER_CHANGED;\n-\tif (ce->ce_ino != (unsigned int) st->st_ino)\n-\t\tchanged |= INODE_CHANGED;\n-\n-#ifdef USE_STDEV\n-\t/*\n-\t * st_dev breaks on network filesystems where different\n-\t * clients will have different views of what \"device\"\n-\t * the filesystem is on\n-\t */\n-\tif (ce->ce_dev != (unsigned int) st->st_dev)\n-\t\tchanged |= INODE_CHANGED;\n-#endif\n-\n-\tif (ce->ce_size != (unsigned int) st->st_size)\n-\t\tchanged |= DATA_CHANGED;\n-\n-\t/* Racily smudged entry? */\n-\tif (!ce->ce_size) {\n-\t\tif (!is_empty_blob_sha1(ce->sha1))\n-\t\t\tchanged |= DATA_CHANGED;\n-\t}\n \n+\tcheck_set_istate_ops(istate);\n+\tchanged = istate->ops->match_stat_basic(ce, st, changed);\n \treturn changed;\n }\n \n-static int is_racy_timestamp(const struct index_state *istate, struct cache_entry *ce)\n+int is_racy_timestamp(const struct index_state *istate, struct cache_entry *ce)\n {\n \treturn (!S_ISGITLINK(ce->ce_mode) &&\n \t\tistate->timestamp.sec &&\n@@ -250,7 +210,7 @@ static int is_racy_timestamp(const struct index_state *istate, struct cache_entr\n \t\t );\n }\n \n-int ie_match_stat(const struct index_state *istate,\n+int ie_match_stat(struct index_state *istate,\n \t\t struct cache_entry *ce, struct stat *st,\n \t\t unsigned int options)\n {\n@@ -278,7 +238,7 @@ int ie_match_stat(const struct index_state *istate,\n \tif (ce->ce_flags & CE_INTENT_TO_ADD)\n \t\treturn DATA_CHANGED | TYPE_CHANGED | MODE_CHANGED;\n \n-\tchanged = ce_match_stat_basic(ce, st);\n+\tchanged = ce_match_stat_basic(istate, ce, st);\n \n \t/*\n \t * Within 1 second of this sequence:\n@@ -306,7 +266,7 @@ int ie_match_stat(const struct index_state *istate,\n \treturn changed;\n }\n \n-int ie_modified(const struct index_state *istate,\n+int ie_modified(struct index_state *istate,\n \t\tstruct cache_entry *ce, struct stat *st, unsigned int options)\n {\n \tint changed, changed_fs;\n@@ -1191,98 +1151,18 @@ static struct cache_entry *refresh_cache_entry(struct cache_entry *ce, int reall\n }\n \n \n-/*****************************************************************\n- * Index File I/O\n- *****************************************************************/\n-\n-#define INDEX_FORMAT_DEFAULT 3\n-\n-/*\n- * dev/ino/uid/gid/size are also just tracked to the low 32 bits\n- * Again - this is just a (very strong in practice) heuristic that\n- * the inode hasn't changed.\n- *\n- * We save the fields in big-endian order to allow using the\n- * index file over NFS transparently.\n- */\n-struct ondisk_cache_entry {\n-\tstruct cache_time ctime;\n-\tstruct cache_time mtime;\n-\tunsigned int dev;\n-\tunsigned int ino;\n-\tunsigned int mode;\n-\tunsigned int uid;\n-\tunsigned int gid;\n-\tunsigned int size;\n-\tunsigned char sha1[20];\n-\tunsigned short flags;\n-\tchar name[FLEX_ARRAY]; /* more */\n-};\n-\n-/*\n- * This struct is used when CE_EXTENDED bit is 1\n- * The struct must match ondisk_cache_entry exactly from\n- * ctime till flags\n- */\n-struct ondisk_cache_entry_extended {\n-\tstruct cache_time ctime;\n-\tstruct cache_time mtime;\n-\tunsigned int dev;\n-\tunsigned int ino;\n-\tunsigned int mode;\n-\tunsigned int uid;\n-\tunsigned int gid;\n-\tunsigned int size;\n-\tunsigned char sha1[20];\n-\tunsigned short flags;\n-\tunsigned short flags2;\n-\tchar name[FLEX_ARRAY]; /* more */\n-};\n-\n-/* These are only used for v3 or lower */\n-#define align_flex_name(STRUCT,len) ((offsetof(struct STRUCT,name) + (len) + 8) & ~7)\n-#define ondisk_cache_entry_size(len) align_flex_name(ondisk_cache_entry,len)\n-#define ondisk_cache_entry_extended_size(len) align_flex_name(ondisk_cache_entry_extended,len)\n-#define ondisk_ce_size(ce) (((ce)->ce_flags & CE_EXTENDED) ? \\\n-\t\t\t ondisk_cache_entry_extended_size(ce_namelen(ce)) : \\\n-\t\t\t ondisk_cache_entry_size(ce_namelen(ce)))\n-\n-static int verify_hdr(struct cache_header *hdr, unsigned long size)\n+static int verify_hdr_version(struct index_state *istate,\n+\t\tstruct cache_version_header *hdr, unsigned long size)\n {\n-\tgit_SHA_CTX c;\n-\tunsigned char sha1[20];\n \tint hdr_version;\n \n \tif (hdr->hdr_signature != htonl(CACHE_SIGNATURE))\n \t\treturn error(\"bad signature\");\n \thdr_version = ntohl(hdr->hdr_version);\n-\tif (hdr_version < 2 || 4 < hdr_version)\n+\tif (hdr_version >= 2 && hdr_version <= 4)\n+\t\tistate->ops = &v2_ops;\n+\telse\n \t\treturn error(\"bad index version %d\", hdr_version);\n-\tgit_SHA1_Init(&c);\n-\tgit_SHA1_Update(&c, hdr, size - 20);\n-\tgit_SHA1_Final(sha1, &c);\n-\tif (hashcmp(sha1, (unsigned char *)hdr + size - 20))\n-\t\treturn error(\"bad index file sha1 signature\");\n-\treturn 0;\n-}\n-\n-static int read_index_extension(struct index_state *istate,\n-\t\t\t\tconst char *ext, void *data, unsigned long sz)\n-{\n-\tswitch (CACHE_EXT(ext)) {\n-\tcase CACHE_EXT_TREE:\n-\t\tistate->cache_tree = cache_tree_read(data, sz);\n-\t\tbreak;\n-\tcase CACHE_EXT_RESOLVE_UNDO:\n-\t\tistate->resolve_undo = resolve_undo_read(data, sz);\n-\t\tbreak;\n-\tdefault:\n-\t\tif (*ext < 'A' || 'Z' < *ext)\n-\t\t\treturn error(\"index uses %.4s extension, which we do not understand\",\n-\t\t\t\t ext);\n-\t\tfprintf(stderr, \"ignoring %.4s extension\\n\", ext);\n-\t\tbreak;\n-\t}\n \treturn 0;\n }\n \n@@ -1291,128 +1171,14 @@ int read_index(struct index_state *istate)\n \treturn read_index_from(istate, get_index_file());\n }\n \n-#ifndef NEEDS_ALIGNED_ACCESS\n-#define ntoh_s(var) ntohs(var)\n-#define ntoh_l(var) ntohl(var)\n-#else\n-static inline uint16_t ntoh_s_force_align(void *p)\n-{\n-\tuint16_t x;\n-\tmemcpy(&x, p, sizeof(x));\n-\treturn ntohs(x);\n-}\n-static inline uint32_t ntoh_l_force_align(void *p)\n-{\n-\tuint32_t x;\n-\tmemcpy(&x, p, sizeof(x));\n-\treturn ntohl(x);\n-}\n-#define ntoh_s(var) ntoh_s_force_align(&(var))\n-#define ntoh_l(var) ntoh_l_force_align(&(var))\n-#endif\n-\n-static struct cache_entry *cache_entry_from_ondisk(struct ondisk_cache_entry *ondisk,\n-\t\t\t\t\t\t unsigned int flags,\n-\t\t\t\t\t\t const char *name,\n-\t\t\t\t\t\t size_t len)\n-{\n-\tstruct cache_entry *ce = xmalloc(cache_entry_size(len));\n-\n-\tce->ce_ctime.sec = ntoh_l(ondisk->ctime.sec);\n-\tce->ce_mtime.sec = ntoh_l(ondisk->mtime.sec);\n-\tce->ce_ctime.nsec = ntoh_l(ondisk->ctime.nsec);\n-\tce->ce_mtime.nsec = ntoh_l(ondisk->mtime.nsec);\n-\tce->ce_dev = ntoh_l(ondisk->dev);\n-\tce->ce_ino = ntoh_l(ondisk->ino);\n-\tce->ce_mode = ntoh_l(ondisk->mode);\n-\tce->ce_uid = ntoh_l(ondisk->uid);\n-\tce->ce_gid = ntoh_l(ondisk->gid);\n-\tce->ce_size = ntoh_l(ondisk->size);\n-\tce->ce_flags = flags & ~CE_NAMEMASK;\n-\tce->ce_namelen = len;\n-\thashcpy(ce->sha1, ondisk->sha1);\n-\tmemcpy(ce->name, name, len);\n-\tce->name[len] = '\\0';\n-\treturn ce;\n-}\n-\n-/*\n- * Adjacent cache entries tend to share the leading paths, so it makes\n- * sense to only store the differences in later entries. In the v4\n- * on-disk format of the index, each on-disk cache entry stores the\n- * number of bytes to be stripped from the end of the previous name,\n- * and the bytes to append to the result, to come up with its name.\n- */\n-static unsigned long expand_name_field(struct strbuf *name, const char *cp_)\n-{\n-\tconst unsigned char *ep, *cp = (const unsigned char *)cp_;\n-\tsize_t len = decode_varint(&cp);\n-\n-\tif (name->len < len)\n-\t\tdie(\"malformed name field in the index\");\n-\tstrbuf_remove(name, name->len - len, len);\n-\tfor (ep = cp; *ep; ep++)\n-\t\t; /* find the end */\n-\tstrbuf_add(name, cp, ep - cp);\n-\treturn (const char *)ep + 1 - cp_;\n-}\n-\n-static struct cache_entry *create_from_disk(struct ondisk_cache_entry *ondisk,\n-\t\t\t\t\t unsigned long *ent_size,\n-\t\t\t\t\t struct strbuf *previous_name)\n-{\n-\tstruct cache_entry *ce;\n-\tsize_t len;\n-\tconst char *name;\n-\tunsigned int flags;\n-\n-\t/* On-disk flags are just 16 bits */\n-\tflags = ntoh_s(ondisk->flags);\n-\tlen = flags & CE_NAMEMASK;\n-\n-\tif (flags & CE_EXTENDED) {\n-\t\tstruct ondisk_cache_entry_extended *ondisk2;\n-\t\tint extended_flags;\n-\t\tondisk2 = (struct ondisk_cache_entry_extended *)ondisk;\n-\t\textended_flags = ntoh_s(ondisk2->flags2) << 16;\n-\t\t/* We do not yet understand any bit out of CE_EXTENDED_FLAGS */\n-\t\tif (extended_flags & ~CE_EXTENDED_FLAGS)\n-\t\t\tdie(\"Unknown index entry format %08x\", extended_flags);\n-\t\tflags |= extended_flags;\n-\t\tname = ondisk2->name;\n-\t}\n-\telse\n-\t\tname = ondisk->name;\n-\n-\tif (!previous_name) {\n-\t\t/* v3 and earlier */\n-\t\tif (len == CE_NAMEMASK)\n-\t\t\tlen = strlen(name);\n-\t\tce = cache_entry_from_ondisk(ondisk, flags, name, len);\n-\n-\t\t*ent_size = ondisk_ce_size(ce);\n-\t} else {\n-\t\tunsigned long consumed;\n-\t\tconsumed = expand_name_field(previous_name, name);\n-\t\tce = cache_entry_from_ondisk(ondisk, flags,\n-\t\t\t\t\t previous_name->buf,\n-\t\t\t\t\t previous_name->len);\n-\n-\t\t*ent_size = (name - ((char *)ondisk)) + consumed;\n-\t}\n-\treturn ce;\n-}\n-\n /* remember to discard_cache() before reading a different cache! */\n int read_index_from(struct index_state *istate, const char *path)\n {\n-\tint fd, i;\n+\tint fd;\n \tstruct stat st;\n-\tunsigned long src_offset;\n-\tstruct cache_header *hdr;\n+\tstruct cache_version_header *hdr;\n \tvoid *mmap;\n \tsize_t mmap_size;\n-\tstruct strbuf previous_name_buf = STRBUF_INIT, *previous_name;\n \n \terrno = EBUSY;\n \tif (istate->initialized)\n@@ -1433,69 +1199,27 @@ int read_index_from(struct index_state *istate, const char *path)\n \n \terrno = EINVAL;\n \tmmap_size = xsize_t(st.st_size);\n-\tif (mmap_size < sizeof(struct cache_header) + 20)\n-\t\tdie(\"index file smaller than expected\");\n-\n \tmmap = xmmap(NULL, mmap_size, PROT_READ | PROT_WRITE, MAP_PRIVATE, fd, 0);\n-\tclose(fd);\n \tif (mmap == MAP_FAILED)\n \t\tdie_errno(\"unable to map index file\");\n \n \thdr = mmap;\n-\tif (verify_hdr(hdr, mmap_size) < 0)\n+\tif (verify_hdr_version(istate, hdr, mmap_size) < 0)\n \t\tgoto unmap;\n \n-\tistate->version = ntohl(hdr->hdr_version);\n-\tistate->cache_nr = ntohl(hdr->hdr_entries);\n-\tistate->cache_alloc = alloc_nr(istate->cache_nr);\n-\tistate->cache = xcalloc(istate->cache_alloc, sizeof(struct cache_entry *));\n-\tistate->initialized = 1;\n-\n-\tif (istate->version == 4)\n-\t\tprevious_name = &previous_name_buf;\n-\telse\n-\t\tprevious_name = NULL;\n-\n-\tsrc_offset = sizeof(*hdr);\n-\tfor (i = 0; i < istate->cache_nr; i++) {\n-\t\tstruct ondisk_cache_entry *disk_ce;\n-\t\tstruct cache_entry *ce;\n-\t\tunsigned long consumed;\n-\n-\t\tdisk_ce = (struct ondisk_cache_entry *)((char *)mmap + src_offset);\n-\t\tce = create_from_disk(disk_ce, &consumed, previous_name);\n-\t\tset_index_entry(istate, i, ce);\n+\tif (istate->ops->verify_hdr(mmap, mmap_size) < 0)\n+\t\tgoto unmap;\n \n-\t\tsrc_offset += consumed;\n-\t}\n-\tstrbuf_release(&previous_name_buf);\n+\tistate->ops->read_index(istate, mmap, mmap_size, fd);\n \tistate->timestamp.sec = st.st_mtime;\n \tistate->timestamp.nsec = ST_MTIME_NSEC(st);\n \n-\twhile (src_offset <= mmap_size - 20 - 8) {\n-\t\t/* After an array of active_nr index entries,\n-\t\t * there can be arbitrary number of extended\n-\t\t * sections, each of which is prefixed with\n-\t\t * extension name (4-byte) and section length\n-\t\t * in 4-byte network byte order.\n-\t\t */\n-\t\tuint32_t extsize;\n-\t\tmemcpy(&extsize, (char *)mmap + src_offset + 4, 4);\n-\t\textsize = ntohl(extsize);\n-\t\tif (read_index_extension(istate,\n-\t\t\t\t\t (const char *) mmap + src_offset,\n-\t\t\t\t\t (char *) mmap + src_offset + 8,\n-\t\t\t\t\t extsize) < 0)\n-\t\t\tgoto unmap;\n-\t\tsrc_offset += 8;\n-\t\tsrc_offset += extsize;\n-\t}\n+\tclose(fd);\n \tmunmap(mmap, mmap_size);\n \treturn istate->cache_nr;\n \n unmap:\n \tmunmap(mmap, mmap_size);\n-\terrno = EINVAL;\n \tdie(\"index file corrupt\");\n }\n \n@@ -1534,201 +1258,6 @@ int unmerged_index(const struct index_state *istate)\n \treturn 0;\n }\n \n-#define WRITE_BUFFER_SIZE 8192\n-static unsigned char write_buffer[WRITE_BUFFER_SIZE];\n-static unsigned long write_buffer_len;\n-\n-static int ce_write_flush(git_SHA_CTX *context, int fd)\n-{\n-\tunsigned int buffered = write_buffer_len;\n-\tif (buffered) {\n-\t\tgit_SHA1_Update(context, write_buffer, buffered);\n-\t\tif (write_in_full(fd, write_buffer, buffered) != buffered)\n-\t\t\treturn -1;\n-\t\twrite_buffer_len = 0;\n-\t}\n-\treturn 0;\n-}\n-\n-static int ce_write(git_SHA_CTX *context, int fd, void *data, unsigned int len)\n-{\n-\twhile (len) {\n-\t\tunsigned int buffered = write_buffer_len;\n-\t\tunsigned int partial = WRITE_BUFFER_SIZE - buffered;\n-\t\tif (partial > len)\n-\t\t\tpartial = len;\n-\t\tmemcpy(write_buffer + buffered, data, partial);\n-\t\tbuffered += partial;\n-\t\tif (buffered == WRITE_BUFFER_SIZE) {\n-\t\t\twrite_buffer_len = buffered;\n-\t\t\tif (ce_write_flush(context, fd))\n-\t\t\t\treturn -1;\n-\t\t\tbuffered = 0;\n-\t\t}\n-\t\twrite_buffer_len = buffered;\n-\t\tlen -= partial;\n-\t\tdata = (char *) data + partial;\n-\t}\n-\treturn 0;\n-}\n-\n-static int write_index_ext_header(git_SHA_CTX *context, int fd,\n-\t\t\t\t unsigned int ext, unsigned int sz)\n-{\n-\text = htonl(ext);\n-\tsz = htonl(sz);\n-\treturn ((ce_write(context, fd, &ext, 4) < 0) ||\n-\t\t(ce_write(context, fd, &sz, 4) < 0)) ? -1 : 0;\n-}\n-\n-static int ce_flush(git_SHA_CTX *context, int fd)\n-{\n-\tunsigned int left = write_buffer_len;\n-\n-\tif (left) {\n-\t\twrite_buffer_len = 0;\n-\t\tgit_SHA1_Update(context, write_buffer, left);\n-\t}\n-\n-\t/* Flush first if not enough space for SHA1 signature */\n-\tif (left + 20 > WRITE_BUFFER_SIZE) {\n-\t\tif (write_in_full(fd, write_buffer, left) != left)\n-\t\t\treturn -1;\n-\t\tleft = 0;\n-\t}\n-\n-\t/* Append the SHA1 signature at the end */\n-\tgit_SHA1_Final(write_buffer + left, context);\n-\tleft += 20;\n-\treturn (write_in_full(fd, write_buffer, left) != left) ? -1 : 0;\n-}\n-\n-static void ce_smudge_racily_clean_entry(struct cache_entry *ce)\n-{\n-\t/*\n-\t * The only thing we care about in this function is to smudge the\n-\t * falsely clean entry due to touch-update-touch race, so we leave\n-\t * everything else as they are. We are called for entries whose\n-\t * ce_mtime match the index file mtime.\n-\t *\n-\t * Note that this actually does not do much for gitlinks, for\n-\t * which ce_match_stat_basic() always goes to the actual\n-\t * contents. The caller checks with is_racy_timestamp() which\n-\t * always says \"no\" for gitlinks, so we are not called for them ;-)\n-\t */\n-\tstruct stat st;\n-\n-\tif (lstat(ce->name, &st) < 0)\n-\t\treturn;\n-\tif (ce_match_stat_basic(ce, &st))\n-\t\treturn;\n-\tif (ce_modified_check_fs(ce, &st)) {\n-\t\t/* This is \"racily clean\"; smudge it. Note that this\n-\t\t * is a tricky code. At first glance, it may appear\n-\t\t * that it can break with this sequence:\n-\t\t *\n-\t\t * $ echo xyzzy >frotz\n-\t\t * $ git-update-index --add frotz\n-\t\t * $ : >frotz\n-\t\t * $ sleep 3\n-\t\t * $ echo filfre >nitfol\n-\t\t * $ git-update-index --add nitfol\n-\t\t *\n-\t\t * but it does not. When the second update-index runs,\n-\t\t * it notices that the entry \"frotz\" has the same timestamp\n-\t\t * as index, and if we were to smudge it by resetting its\n-\t\t * size to zero here, then the object name recorded\n-\t\t * in index is the 6-byte file but the cached stat information\n-\t\t * becomes zero --- which would then match what we would\n-\t\t * obtain from the filesystem next time we stat(\"frotz\").\n-\t\t *\n-\t\t * However, the second update-index, before calling\n-\t\t * this function, notices that the cached size is 6\n-\t\t * bytes and what is on the filesystem is an empty\n-\t\t * file, and never calls us, so the cached size information\n-\t\t * for \"frotz\" stays 6 which does not match the filesystem.\n-\t\t */\n-\t\tce->ce_size = 0;\n-\t}\n-}\n-\n-/* Copy miscellaneous fields but not the name */\n-static char *copy_cache_entry_to_ondisk(struct ondisk_cache_entry *ondisk,\n-\t\t\t\t struct cache_entry *ce)\n-{\n-\tshort flags;\n-\n-\tondisk->ctime.sec = htonl(ce->ce_ctime.sec);\n-\tondisk->mtime.sec = htonl(ce->ce_mtime.sec);\n-\tondisk->ctime.nsec = htonl(ce->ce_ctime.nsec);\n-\tondisk->mtime.nsec = htonl(ce->ce_mtime.nsec);\n-\tondisk->dev = htonl(ce->ce_dev);\n-\tondisk->ino = htonl(ce->ce_ino);\n-\tondisk->mode = htonl(ce->ce_mode);\n-\tondisk->uid = htonl(ce->ce_uid);\n-\tondisk->gid = htonl(ce->ce_gid);\n-\tondisk->size = htonl(ce->ce_size);\n-\thashcpy(ondisk->sha1, ce->sha1);\n-\n-\tflags = ce->ce_flags;\n-\tflags |= (ce_namelen(ce) >= CE_NAMEMASK ? CE_NAMEMASK : ce_namelen(ce));\n-\tondisk->flags = htons(flags);\n-\tif (ce->ce_flags & CE_EXTENDED) {\n-\t\tstruct ondisk_cache_entry_extended *ondisk2;\n-\t\tondisk2 = (struct ondisk_cache_entry_extended *)ondisk;\n-\t\tondisk2->flags2 = htons((ce->ce_flags & CE_EXTENDED_FLAGS) >> 16);\n-\t\treturn ondisk2->name;\n-\t}\n-\telse {\n-\t\treturn ondisk->name;\n-\t}\n-}\n-\n-static int ce_write_entry(git_SHA_CTX *c, int fd, struct cache_entry *ce,\n-\t\t\t struct strbuf *previous_name)\n-{\n-\tint size;\n-\tstruct ondisk_cache_entry *ondisk;\n-\tchar *name;\n-\tint result;\n-\n-\tif (!previous_name) {\n-\t\tsize = ondisk_ce_size(ce);\n-\t\tondisk = xcalloc(1, size);\n-\t\tname = copy_cache_entry_to_ondisk(ondisk, ce);\n-\t\tmemcpy(name, ce->name, ce_namelen(ce));\n-\t} else {\n-\t\tint common, to_remove, prefix_size;\n-\t\tunsigned char to_remove_vi[16];\n-\t\tfor (common = 0;\n-\t\t (ce->name[common] &&\n-\t\t common < previous_name->len &&\n-\t\t ce->name[common] == previous_name->buf[common]);\n-\t\t common++)\n-\t\t\t; /* still matching */\n-\t\tto_remove = previous_name->len - common;\n-\t\tprefix_size = encode_varint(to_remove, to_remove_vi);\n-\n-\t\tif (ce->ce_flags & CE_EXTENDED)\n-\t\t\tsize = offsetof(struct ondisk_cache_entry_extended, name);\n-\t\telse\n-\t\t\tsize = offsetof(struct ondisk_cache_entry, name);\n-\t\tsize += prefix_size + (ce_namelen(ce) - common + 1);\n-\n-\t\tondisk = xcalloc(1, size);\n-\t\tname = copy_cache_entry_to_ondisk(ondisk, ce);\n-\t\tmemcpy(name, to_remove_vi, prefix_size);\n-\t\tmemcpy(name + prefix_size, ce->name + common, ce_namelen(ce) - common);\n-\n-\t\tstrbuf_splice(previous_name, common, to_remove,\n-\t\t\t ce->name + common, ce_namelen(ce) - common);\n-\t}\n-\n-\tresult = ce_write(c, fd, ondisk, size);\n-\tfree(ondisk);\n-\treturn result;\n-}\n-\n static int has_racy_timestamp(struct index_state *istate)\n {\n \tint entries = istate->cache_nr;\n@@ -1756,83 +1285,9 @@ void update_index_if_able(struct index_state *istate, struct lock_file *lockfile\n \n int write_index(struct index_state *istate, int newfd)\n {\n-\tgit_SHA_CTX c;\n-\tstruct cache_header hdr;\n-\tint i, err, removed, extended, hdr_version;\n-\tstruct cache_entry **cache = istate->cache;\n-\tint entries = istate->cache_nr;\n-\tstruct stat st;\n-\tstruct strbuf previous_name_buf = STRBUF_INIT, *previous_name;\n-\n-\tfor (i = removed = extended = 0; i < entries; i++) {\n-\t\tif (cache[i]->ce_flags & CE_REMOVE)\n-\t\t\tremoved++;\n+\tcheck_set_istate_ops(istate);\n \n-\t\t/* reduce extended entries if possible */\n-\t\tcache[i]->ce_flags &= ~CE_EXTENDED;\n-\t\tif (cache[i]->ce_flags & CE_EXTENDED_FLAGS) {\n-\t\t\textended++;\n-\t\t\tcache[i]->ce_flags |= CE_EXTENDED;\n-\t\t}\n-\t}\n-\n-\tif (!istate->version)\n-\t\tistate->version = INDEX_FORMAT_DEFAULT;\n-\n-\t/* demote version 3 to version 2 when the latter suffices */\n-\tif (istate->version == 3 || istate->version == 2)\n-\t\tistate->version = extended ? 3 : 2;\n-\n-\thdr_version = istate->version;\n-\n-\thdr.hdr_signature = htonl(CACHE_SIGNATURE);\n-\thdr.hdr_version = htonl(hdr_version);\n-\thdr.hdr_entries = htonl(entries - removed);\n-\n-\tgit_SHA1_Init(&c);\n-\tif (ce_write(&c, newfd, &hdr, sizeof(hdr)) < 0)\n-\t\treturn -1;\n-\n-\tprevious_name = (hdr_version == 4) ? &previous_name_buf : NULL;\n-\tfor (i = 0; i < entries; i++) {\n-\t\tstruct cache_entry *ce = cache[i];\n-\t\tif (ce->ce_flags & CE_REMOVE)\n-\t\t\tcontinue;\n-\t\tif (!ce_uptodate(ce) && is_racy_timestamp(istate, ce))\n-\t\t\tce_smudge_racily_clean_entry(ce);\n-\t\tif (ce_write_entry(&c, newfd, ce, previous_name) < 0)\n-\t\t\treturn -1;\n-\t}\n-\tstrbuf_release(&previous_name_buf);\n-\n-\t/* Write extension data here */\n-\tif (istate->cache_tree) {\n-\t\tstruct strbuf sb = STRBUF_INIT;\n-\n-\t\tcache_tree_write(&sb, istate->cache_tree);\n-\t\terr = write_index_ext_header(&c, newfd, CACHE_EXT_TREE, sb.len) < 0\n-\t\t\t|| ce_write(&c, newfd, sb.buf, sb.len) < 0;\n-\t\tstrbuf_release(&sb);\n-\t\tif (err)\n-\t\t\treturn -1;\n-\t}\n-\tif (istate->resolve_undo) {\n-\t\tstruct strbuf sb = STRBUF_INIT;\n-\n-\t\tresolve_undo_write(&sb, istate->resolve_undo);\n-\t\terr = write_index_ext_header(&c, newfd, CACHE_EXT_RESOLVE_UNDO,\n-\t\t\t\t\t sb.len) < 0\n-\t\t\t|| ce_write(&c, newfd, sb.buf, sb.len) < 0;\n-\t\tstrbuf_release(&sb);\n-\t\tif (err)\n-\t\t\treturn -1;\n-\t}\n-\n-\tif (ce_flush(&c, newfd) || fstat(newfd, &st))\n-\t\treturn -1;\n-\tistate->timestamp.sec = (unsigned int)st.st_mtime;\n-\tistate->timestamp.nsec = ST_MTIME_NSEC(st);\n-\treturn 0;\n+\treturn istate->ops->write_index(istate, newfd);\n }\n \n /*\ndiff --git a/read-cache.h b/read-cache.h\nnew file mode 100644\nindex 0000000..78de6a2\n--- /dev/null\n+++ b/read-cache.h\n@@ -0,0 +1,57 @@\n+/* Index extensions.\n+ *\n+ * The first letter should be 'A'..'Z' for extensions that are not\n+ * necessary for a correct operation (i.e. optimization data).\n+ * When new extensions are added that _needs_ to be understood in\n+ * order to correctly interpret the index file, pick character that\n+ * is outside the range, to cause the reader to abort.\n+ */\n+\n+#define CACHE_EXT(s) ( (s[0]<<24)|(s[1]<<16)|(s[2]<<8)|(s[3]) )\n+#define CACHE_EXT_TREE 0x54524545\t/* \"TREE\" */\n+#define CACHE_EXT_RESOLVE_UNDO 0x52455543 /* \"REUC\" */\n+\n+#define INDEX_FORMAT_DEFAULT 3\n+\n+/*\n+ * Basic data structures for the directory cache\n+ */\n+struct cache_version_header {\n+\tunsigned int hdr_signature;\n+\tunsigned int hdr_version;\n+};\n+\n+struct index_ops {\n+\tint (*match_stat_basic)(struct cache_entry *ce, struct stat *st, int changed);\n+\tint (*verify_hdr)(void *mmap, unsigned long size);\n+\tvoid (*read_index)(struct index_state *istate, void *mmap, int mmap_size, int fd);\n+\tint (*write_index)(struct index_state *istate, int newfd);\n+};\n+\n+extern struct index_ops v2_ops;\n+\n+#ifndef NEEDS_ALIGNED_ACCESS\n+#define ntoh_s(var) ntohs(var)\n+#define ntoh_l(var) ntohl(var)\n+#else\n+static inline uint16_t ntoh_s_force_align(void *p)\n+{\n+\tuint16_t x;\n+\tmemcpy(&x, p, sizeof(x));\n+\treturn ntohs(x);\n+}\n+static inline uint32_t ntoh_l_force_align(void *p)\n+{\n+\tuint32_t x;\n+\tmemcpy(&x, p, sizeof(x));\n+\treturn ntohl(x);\n+}\n+#define ntoh_s(var) ntoh_s_force_align(&(var))\n+#define ntoh_l(var) ntoh_l_force_align(&(var))\n+#endif\n+\n+extern int ce_modified_check_fs(struct cache_entry *ce, struct stat *st);\n+extern int ce_match_stat_basic(struct index_state *istate,\n+\t\tstruct cache_entry *ce, struct stat *st);\n+extern int is_racy_timestamp(const struct index_state *istate, struct cache_entry *ce);\n+extern void set_index_entry(struct index_state *istate, int nr, struct cache_entry *ce);\ndiff --git a/test-index-version.c b/test-index-version.c\nindex bfaad9e..3899a2f 100644\n--- a/test-index-version.c\n+++ b/test-index-version.c\n@@ -1,8 +1,13 @@\n #include \"cache.h\"\n \n+struct cache_version_header {\n+\tunsigned int hdr_signature;\n+\tunsigned int hdr_version;\n+};\n+\n int main(int argc, const char **argv)\n {\n-\tstruct cache_header hdr;\n+\tstruct cache_version_header hdr;\n \tint version;\n \n \tmemset(&hdr,0,sizeof(hdr));\n-- \n1.7.10.GIT\n"},{"id":"196651","messageId":"1344424681-31469-3-git-send-email-t.gummerer@gmail.com","threadId":"31208","inReplyTo":"1344424681-31469-1-git-send-email-t.gummerer@gmail.com","subject":"[PATCH/RFC v3 02/13] t2104: Don't fail for index versions other than [23]","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2012-08-08T11:17:50Z","receivedAt":"2012-08-08T11:17:50Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"t2104 currently checks for the exact index version 2 or 3,\ndepending if there is a skip-worktree flag or not. Other\nindex versions do not use extended flags and thus cannot\nbe tested for version changes.\n\nMake this test update the index to version 2 at the beginning\nof the test. Testing the skip-worktree flags for the default\nindex format is still covered by t7011 and t7012.\n\nSigned-off-by: Thomas Gummerer \n---\n t/t2104-update-index-skip-worktree.sh | 1 +\n 1 file changed, 1 insertion(+)\n\ndiff --git a/t/t2104-update-index-skip-worktree.sh b/t/t2104-update-index-skip-worktree.sh\nindex 1d0879b..bd9644f 100755\n--- a/t/t2104-update-index-skip-worktree.sh\n+++ b/t/t2104-update-index-skip-worktree.sh\n@@ -22,6 +22,7 @@ H sub/2\n EOF\n \n test_expect_success 'setup' '\n+\tgit update-index --index-version=2 &&\n \tmkdir sub &&\n \ttouch ./1 ./2 sub/1 sub/2 &&\n \tgit add 1 2 sub/1 sub/2 &&\n-- \n1.7.10.GIT\n"},{"id":"196654","messageId":"1344424681-31469-4-git-send-email-t.gummerer@gmail.com","threadId":"31208","inReplyTo":"1344424681-31469-1-git-send-email-t.gummerer@gmail.com","subject":"[PATCH/RFC v3 03/13] t3700: Avoid interfering with the racy code","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2012-08-08T11:17:51Z","receivedAt":"2012-08-08T11:17:51Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"The new git racy code uses the mtime of cache-entries as smudge\nmarker for racily clean entries. The work of checking the file-system\nif the entry really changed is offloaded to the reader. This interferes\nwith this test, because the entry is racily smudged and thus has\nmtime 0.\n\nTo avoid interfering with the racy code, we use a time relative\nto the time returned by time(3), instead of a time relative to\nthe mtime of the cache entries.\n\nSigned-off-by: Thomas Gummerer \n---\n t/t3700-add.sh | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/t/t3700-add.sh b/t/t3700-add.sh\nindex 874b3a6..829d36d 100755\n--- a/t/t3700-add.sh\n+++ b/t/t3700-add.sh\n@@ -184,7 +184,7 @@ test_expect_success 'git add --refresh with pathspec' '\n \techo >foo && echo >bar && echo >baz &&\n \tgit add foo bar baz && H=$(git rev-parse :foo) && git rm -f foo &&\n \techo \"100644 $H 3\tfoo\" | git update-index --index-info &&\n-\ttest-chmtime -60 bar baz &&\n+\ttest-chmtime =-60 bar baz &&\n \t>expect &&\n \tgit add --refresh bar >actual &&\n \ttest_cmp expect actual &&\n-- \n1.7.10.GIT\n"},{"id":"196652","messageId":"1344424681-31469-5-git-send-email-t.gummerer@gmail.com","threadId":"31208","inReplyTo":"1344424681-31469-1-git-send-email-t.gummerer@gmail.com","subject":"[PATCH/RFC v3 04/13] Add documentation of the index-v5 file format","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2012-08-08T11:17:52Z","receivedAt":"2012-08-08T11:17:52Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"Add a documentation of the index file format version 5 to\nDocumentation/technical.\n\nHelped-by: Michael Haggerty \nHelped-by: Junio C Hamano \nHelped-by: Thomas Rast \nHelped-by: Nguyen Thai Ngoc Duy \nHelped-by: Robin Rosenberg \nSigned-off-by: Thomas Gummerer \n---\n Documentation/technical/index-file-format-v5.txt | 285 ++++++++++++++++++++++\n 1 file changed, 285 insertions(+)\n create mode 100644 Documentation/technical/index-file-format-v5.txt\n\ndiff --git a/Documentation/technical/index-file-format-v5.txt b/Documentation/technical/index-file-format-v5.txt\nnew file mode 100644\nindex 0000000..6707f06\n--- /dev/null\n+++ b/Documentation/technical/index-file-format-v5.txt\n@@ -0,0 +1,285 @@\n+GIT index format\n+================\n+\n+== The git index file format\n+\n+ The git index file (.git/index) documents the status of the files\n+ in the git staging area.\n+\n+ The staging area is used for preparing commits, merging, etc.\n+\n+ All binary numbers are in network byte order. Version 5 is described\n+ here.\n+\n+ - A 20-byte header consisting of\n+\n+ sig (32-bits): Signature:\n+ The signature is { 'D', 'I', 'R', 'C' } (stands for \"dircache\")\n+\n+ vnr (32-bits): Version number:\n+ The current supported versions are 2, 3, 4 and 5.\n+\n+ ndir (32-bits): number of directories in the index.\n+\n+ nfile (32-bits): number of file entries in the index.\n+\n+ fblockoffset (32-bits): offset to the file block, relative to the\n+ beginning of the file.\n+\n+ - Offset to the extensions.\n+\n+ nextensions (32-bits): number of extensions.\n+\n+ extoffset (32-bits): offset to the extension. (Possibly none, as\n+ many as indicated in the 4-byte number of extensions)\n+\n+ headercrc (32-bits): crc checksum for the header and extension\n+ offsets\n+\n+ - diroffsets (ndir * directory offsets): A directory offset for each\n+ of the ndir directories in the index, sorted by pathname (of the\n+ directory it's pointing to) (see below). The diroffsets are relative\n+ to the beginning of the direntries block. [1]\n+\n+ - direntries (ndir * directory entries): A directory entry for each\n+ of the ndir directories in the index, sorted by pathname (see\n+ below). [2]\n+\n+ - fileoffsets (nfile * file offsets): A file offset for each of the\n+ nfile files in the index (see below). The file offsets are relative\n+ to the beginning of the fileentries block. [1]\n+\n+ - fileentries (nfile * file entries): A file entry for each of the\n+ nfile files in the index (see below).\n+\n+ - crdata: A number of entries for conflicted data/resolved conflicts\n+ (see below).\n+\n+ - Extensions (Currently none, see below in the future)\n+\n+ Extensions are identified by signature. Optional extensions can\n+ be ignored if GIT does not understand them.\n+\n+ GIT supports an arbitrary number of extension, but currently none\n+ is implemented. [3]\n+\n+ extsig (32-bits): extension signature. If the first byte is 'A'..'Z'\n+ the extension is optional and can be ignored.\n+\n+ extsize (32-bits): size of the extension, excluding the header\n+ (extsig, extsize, extchecksum).\n+\n+ extchecksum (32-bits): crc32 checksum of the extension signature\n+ and size.\n+\n+ - Extension data.\n+\n+\n+== Directory offsets (diroffsets)\n+\n+ diroffset (32-bits): offset to the directory relative to the beginning\n+ of the index file. There are ndir + 1 offsets in the diroffset table,\n+ the last is pointing to the end of the last direntry. With this last\n+ entry, we can replace the strlen when reading each filename, by\n+ calculating its length with the offsets.\n+\n+ This part is needed for making the directory entries bisectable and\n+ thus allowing a binary search.\n+\n+== Directory entry (direntries)\n+ \n+ Directory entries are sorted in lexicographic order by the name \n+ of their path starting with the root.\n+ \n+ pathname (variable length, nul terminated): relative to top level\n+ directory (without the leading slash). '/' is used as path\n+ separator. A string of length 0 ('') indicates the root directory.\n+ The special path components \".\", and \"..\" (without quotes) are\n+ disallowed. The path also includes a trailing slash. [9]\n+\n+ foffset (32-bits): offset to the lexicographically first file in \n+ the file offsets (fileoffsets), relative to the beginning of\n+ the fileoffset block.\n+\n+ cr (32-bits): offset to conflicted/resolved data at the end of the\n+ index. 0 if there is no such data. [4]\n+\n+ ncr (32-bits): number of conflicted/resolved data entries at the\n+ end of the index if the offset is non 0. If cr is 0, ncr is\n+ also 0.\n+\n+ nsubtrees (32-bits): number of subtrees this tree has in the index.\n+\n+ nfiles (32-bits): number of files in the directory, that are in\n+ the index.\n+\n+ nentries (32-bits): number of entries in the index that is covered\n+ by the tree this entry represents. (-1 if the entry is invalid).\n+ This number includes all the files in this tree, recursively.\n+\n+ objname (160-bits): object name for the object that would result\n+ from writing this span of index as a tree. This is only valid\n+ if nentries is valid, meaning the cache-tree is valid.\n+\n+ flags (16-bits): 'flags' field split into (high to low bits) (For\n+ D/F conflicts)\n+ \n+ stage (2-bits): stage of the directory during merge\n+\n+ 14-bit unused\n+\n+ dircrc (32-bits): crc32 checksum for each directory entry.\n+\n+ The last 24 bytes (4-byte number of entries + 160-bit object name) are\n+ for the cache tree. An entry can be in an invalidated state which is\n+ represented by having -1 in the entry_count field.\n+\n+ The entries are written out in the top-down, depth-first order. The\n+ first entry represents the root level of the repository, followed by\n+ the first subtree - let's call it A - of the root level, followed by\n+ the first subtree of A, ... There is no prefix compression for\n+ directories.\n+\n+== File offsets (fileoffsets)\n+\n+ fileoffset (32-bits): offset to the file.\n+\n+ This part is needed for making the file entries bisectable and\n+ thus allowing a binary search. There are nfile + 1 offsets in the\n+ fileoffset table, the last is pointing to the end of the last\n+ fileentry. With this last entry, we can replace the strlen when\n+ reading each filename, by calculating its length with the offsets.\n+\n+== File entry (fileentries)\n+ \n+ File entries are sorted in ascending order on the name field, after the\n+ respective offset given by the directory entries. All file names are\n+ prefix compressed, meaning the file name is relative to the directory.\n+\n+ filename (variable length, nul terminated). The exact encoding is \n+ undefined, but the filename cannot contain a NUL byte (iow, the same\n+ encoding as a UNIX pathname).\n+\n+ flags (16-bits): 'flags' field split into (high to low bits)\n+\n+ assumevalid (1-bit): assume-valid flag\n+\n+ intenttoadd (1-bit): intent-to-add flag, used by \"git add -N\".\n+ Extended flag in index v3.\n+\n+ stage (2-bit): stage of the file during merge\n+\n+ skipworktree (1-bit): skip-worktree flag, used by sparse checkout.\n+ Extended flag in index v3.\n+\n+ 11-bit unused, must be zero [6]\n+\n+ mode (16-bits): file mode, split into (high to low bits)\n+\n+ objtype (4-bits): object type\n+ valid values in binary are 1000 (regular file), 1010 (symbolic\n+ link) and 1110 (gitlink)\n+\n+ 3-bit unused\n+\n+ permission (9-bits): unix permission. Only 0755 and 0644 are valid\n+ for regular files. Symbolic links and gitlinks have value 0 in \n+ this field.\n+\n+ mtimes (32-bits): mtime seconds, the last time a file's data changed\n+ this is stat(2) data\n+\n+ mtimens (32-bits): mtime nanosecond fractions\n+ this is stat(2) data\n+\n+ file size (32-bits): The on-disk size, trucated to 32-bit.\n+ this is stat(2) data\n+\n+ statcrc (32-bits): crc32 checksum over ctime seconds, ctime\n+ nanoseconds, ino, dev, uid, gid (All stat(2) data\n+ except mtime and file size). If the statcrc is 0 it will\n+ be ignored. [7]\n+\n+ objhash (160-bits): SHA-1 for the represented object\n+\n+ entrycrc (32-bits): crc32 checksum for the file entry. The crc code\n+ includes the offset to the offset to the file, relative to the\n+ beginning of the file.\n+\n+== Conflict data\n+\n+ A conflict is represented in the index as a set of higher stage entries.\n+ These entries are stored at the end of the index. When a conflict is \n+ resolved (e.g. with \"git add path\"). A bit is flipped, to indicate that\n+ the conflict is resolved, but the entries will be kept, so that\n+ conflicts can be recreated (e.g. with \"git checkout -m\", in case users\n+ want to redo a conflict resolution from scratch.\n+\n+ The first part of a conflict (usually stage 1) will be stored both in\n+ the entries part of the index and in the conflict part. All other parts\n+ will only be stored in the conflict part.\n+\n+ filename (variable length, nul terminated): filename of the entry,\n+ relative to its containing directory).\n+\n+ nfileconflicts (32-bits): number of conflicts for the file [8]\n+\n+ flags (nfileconflicts * flags) (16-bits): 'flags' field split into:\n+ \n+ conflicted (1-bit): conflicted state (conflicted/resolved) (1 if\n+ conflicted)\n+\n+ stage (2-bits): stage during merge.\n+ \n+ 13-bit unused\n+\n+ entry_mode (nfileconflicts * entry mode) (16-bits): octal numbers, entry\n+ mode of eache entry in the different stages. (How many is defined by\n+ the 4-byte number before)\n+\n+ objectnames (nfileconflicts * object names) (160-bits): object names \n+ of the different stages.\n+\n+ conflictcrc (32-bits): crc32 checksum over conflict data.\n+\n+== Design explanations\n+\n+[1] The directory and file offsets are included in the index format\n+ to enable bisectability of the index, for binary searches.Updating\n+ a single entry and partial reading will benefit from this.\n+\n+[2] The directories are saved in their own block, to be able to\n+ quickly search for a directory in the index. They include a\n+ offset to the (lexically) first file in the directory.\n+\n+[3] The data of the cache-tree extension and the resolve undo\n+ extension is now part of the index itself, but if other extensions\n+ come up in the future, there is no need to change the index, they\n+ can simply be added at the end.\n+\n+[4] To avoid rewrites of the whole index when there are conflicts or\n+ conflicts are being resolved, conflicted data will be stored at\n+ the end of the index. To mark the conflict resolved, just a bit\n+ has to be flipped. The data will still be there, if a user wants\n+ to redo the conflict resolution.\n+\n+[5] Since only 4 modes are effectively allowed in git but 32-bit are\n+ used to store them, having a two bit flag for the mode is enough\n+ and saves 4 byte per entry.\n+\n+[6] The length of the file name was dropped, since each file name is\n+ nul terminated anyway.\n+\n+[7] Since all stat data (except mtime and ctime) is just used for\n+ checking if a file has changed a checksum of the data is enough.\n+ In addition to that Thomas Rast suggested ctime could be ditched\n+ completely (core.trustctime=false) and thus included in the\n+ checksum. This would save 24 bytes per index entry, which would\n+ be about 4 MB on the Webkit index.\n+ (Thanks for the suggestion to Michael Haggerty)\n+\n+[8] Since there can be more stage #1 entries, it is necessary to know\n+ the number of conflict data entries there are.\n+\n+[9] As Michael Haggerty pointed out on the mailing list, storing the\n+ trailing slash will simplify a few operations.\n-- \n1.7.10.GIT\n"},{"id":"196648","messageId":"1344424681-31469-6-git-send-email-t.gummerer@gmail.com","threadId":"31208","inReplyTo":"1344424681-31469-1-git-send-email-t.gummerer@gmail.com","subject":"[PATCH/RFC v3 05/13] Make in-memory format aware of stat_crc","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2012-08-08T11:17:53Z","receivedAt":"2012-08-08T11:17:53Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"Make the in-memory format aware of the stat_crc used by index-v5.\nIt is simply ignored by index version prior to v5.\n\nSigned-off-by: Thomas Gummerer \n---\n cache.h | 1 +\n read-cache.c | 25 +++++++++++++++++++++++++\n 2 files changed, 26 insertions(+)\n\ndiff --git a/cache.h b/cache.h\nindex c77cdbe..bfe3099 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -122,6 +122,7 @@ struct cache_entry {\n \tunsigned int ce_flags;\n \tunsigned int ce_namelen;\n \tunsigned char sha1[20];\n+\tuint32_t ce_stat_crc;\n \tstruct cache_entry *next;\n \tstruct cache_entry *dir_next;\n \tchar name[FLEX_ARRAY]; /* more */\ndiff --git a/read-cache.c b/read-cache.c\nindex 125e6a0..d8f8b74 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -51,6 +51,29 @@ void rename_index_entry_at(struct index_state *istate, int nr, const char *new_n\n \tadd_index_entry(istate, new, ADD_CACHE_OK_TO_ADD|ADD_CACHE_OK_TO_REPLACE);\n }\n \n+static uint32_t calculate_stat_crc(struct cache_entry *ce)\n+{\n+\tunsigned int ctimens = 0;\n+\tuint32_t stat, stat_crc;\n+\n+\tstat = htonl(ce->ce_ctime.sec);\n+\tstat_crc = crc32(0, (Bytef*)&stat, 4);\n+#ifdef USE_NSEC\n+\tctimens = ce->ce_ctime.nsec;\n+#endif\n+\tstat = htonl(ctimens);\n+\tstat_crc = crc32(stat_crc, (Bytef*)&stat, 4);\n+\tstat = htonl(ce->ce_ino);\n+\tstat_crc = crc32(stat_crc, (Bytef*)&stat, 4);\n+\tstat = htonl(ce->ce_dev);\n+\tstat_crc = crc32(stat_crc, (Bytef*)&stat, 4);\n+\tstat = htonl(ce->ce_uid);\n+\tstat_crc = crc32(stat_crc, (Bytef*)&stat, 4);\n+\tstat = htonl(ce->ce_gid);\n+\tstat_crc = crc32(stat_crc, (Bytef*)&stat, 4);\n+\treturn stat_crc;\n+}\n+\n /*\n * This only updates the \"non-critical\" parts of the directory\n * cache, ie the parts that aren't tracked by GIT, and only used\n@@ -73,6 +96,8 @@ void fill_stat_cache_info(struct cache_entry *ce, struct stat *st)\n \n \tif (S_ISREG(st->st_mode))\n \t\tce_mark_uptodate(ce);\n+\n+\tce->ce_stat_crc = calculate_stat_crc(ce);\n }\n \n static int ce_compare_data(struct cache_entry *ce, struct stat *st)\n-- \n1.7.10.GIT\n"},{"id":"196656","messageId":"1344424681-31469-7-git-send-email-t.gummerer@gmail.com","threadId":"31208","inReplyTo":"1344424681-31469-1-git-send-email-t.gummerer@gmail.com","subject":"[PATCH/RFC v3 06/13] Read index-v5","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2012-08-08T11:17:54Z","receivedAt":"2012-08-08T11:17:54Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"Make git read the index file version 5 without complaining.\n\nThis version of the reader doesn't read neither the cache-tree\nnor the resolve undo data, but doesn't choke on an index that\nincludes such data.\n\nHelped-by: Nguyen Thai Ngoc Duy \nHelped-by: Thomas Rast \nSigned-off-by: Thomas Gummerer \n---\n Makefile | 1 +\n cache.h | 72 +++++++\n read-cache-v5.c | 589 +++++++++++++++++++++++++++++++++++++++++++++++++++++++\n read-cache.c | 1 -\n 4 files changed, 662 insertions(+), 1 deletion(-)\n create mode 100644 read-cache-v5.c\n\ndiff --git a/Makefile b/Makefile\nindex b4a7c73..77be175 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -770,6 +770,7 @@ LIB_OBJS += quote.o\n LIB_OBJS += reachable.o\n LIB_OBJS += read-cache.o\n LIB_OBJS += read-cache-v2.o\n+LIB_OBJS += read-cache-v5.o\n LIB_OBJS += reflog-walk.o\n LIB_OBJS += refs.o\n LIB_OBJS += remote.o\ndiff --git a/cache.h b/cache.h\nindex bfe3099..a0a1781 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -110,6 +110,15 @@ struct cache_time {\n \tunsigned int nsec;\n };\n \n+/*\n+ * The *next pointer is used in read_entries_v5 for holding\n+ * all the elements of a directory, and points to the next\n+ * cache_entry in a directory.\n+ *\n+ * It is reset by the add_name_hash call in set_index_entry\n+ * to set it to point to the next cache_entry in the\n+ * correct in-memory format ordering.\n+ */\n struct cache_entry {\n \tstruct cache_time ce_ctime;\n \tstruct cache_time ce_mtime;\n@@ -128,11 +137,58 @@ struct cache_entry {\n \tchar name[FLEX_ARRAY]; /* more */\n };\n \n+struct directory_entry {\n+\tstruct directory_entry *next;\n+\tstruct directory_entry *next_hash;\n+\tstruct cache_entry *ce;\n+\tstruct cache_entry *ce_last;\n+\tstruct conflict_entry *conflict;\n+\tstruct conflict_entry *conflict_last;\n+\tunsigned int conflict_size;\n+\tunsigned int de_foffset;\n+\tunsigned int de_cr;\n+\tunsigned int de_ncr;\n+\tunsigned int de_nsubtrees;\n+\tunsigned int de_nfiles;\n+\tunsigned int de_nentries;\n+\tunsigned char sha1[20];\n+\tunsigned short de_flags;\n+\tunsigned int de_pathlen;\n+\tchar pathname[FLEX_ARRAY];\n+};\n+\n+struct conflict_part {\n+\tstruct conflict_part *next;\n+\tunsigned short flags;\n+\tunsigned short entry_mode;\n+\tunsigned char sha1[20];\n+};\n+\n+struct conflict_entry {\n+\tstruct conflict_entry *next;\n+\tunsigned int nfileconflicts;\n+\tstruct conflict_part *entries;\n+\tunsigned int namelen;\n+\tunsigned int pathlen;\n+\tchar name[FLEX_ARRAY];\n+};\n+\n+struct ondisk_conflict_part {\n+\tunsigned short flags;\n+\tunsigned short entry_mode;\n+\tunsigned char sha1[20];\n+};\n+\n+#define CE_NAMEMASK (0x0fff)\n #define CE_STAGEMASK (0x3000)\n #define CE_EXTENDED (0x4000)\n #define CE_VALID (0x8000)\n #define CE_STAGESHIFT 12\n \n+#define CONFLICT_CONFLICTED (0x8000)\n+#define CONFLICT_STAGESHIFT 13\n+#define CONFLICT_STAGEMASK (0x6000)\n+\n /*\n * Range 0xFFFF0000 in ce_flags is divided into\n * two parts: in-memory flags and on-disk ones.\n@@ -166,6 +222,18 @@ struct cache_entry {\n #define CE_EXTENDED_FLAGS (CE_INTENT_TO_ADD | CE_SKIP_WORKTREE)\n \n /*\n+ * Representation of the extended on-disk flags in the v5 format.\n+ * They must not collide with the ordinary on-disk flags, and need to\n+ * fit in 16 bits. Note however that v5 does not save the name\n+ * length.\n+ */\n+#define CE_INTENT_TO_ADD_V5 (0x4000)\n+#define CE_SKIP_WORKTREE_V5 (0x0800)\n+#if (CE_VALID|CE_STAGEMASK) & (CE_INTENTTOADD_V5|CE_SKIPWORKTREE_V5)\n+#error \"v5 on-disk flags collide with ordinary on-disk flags\"\n+#endif\n+\n+/*\n * Safeguard to avoid saving wrong flags:\n * - CE_EXTENDED2 won't get saved until its semantic is known\n * - Bits in 0x0000FFFF have been saved in ce_flags already\n@@ -203,6 +271,8 @@ static inline unsigned create_ce_flags(unsigned stage)\n #define ce_skip_worktree(ce) ((ce)->ce_flags & CE_SKIP_WORKTREE)\n #define ce_mark_uptodate(ce) ((ce)->ce_flags |= CE_UPTODATE)\n \n+#define conflict_stage(c) ((CONFLICT_STAGEMASK & (c)->flags) >> CONFLICT_STAGESHIFT)\n+\n #define ce_permissions(mode) (((mode) & 0100) ? 0755 : 0644)\n static inline unsigned int create_ce_mode(unsigned int mode)\n {\n@@ -249,6 +319,8 @@ static inline unsigned int canon_mode(unsigned int mode)\n }\n \n #define cache_entry_size(len) (offsetof(struct cache_entry,name) + (len) + 1)\n+#define directory_entry_size(len) (offsetof(struct directory_entry,pathname) + (len) + 1)\n+#define conflict_entry_size(len) (offsetof(struct conflict_entry,name) + (len) + 1)\n \n struct index_state {\n \tstruct cache_entry **cache;\ndiff --git a/read-cache-v5.c b/read-cache-v5.c\nnew file mode 100644\nindex 0000000..ec1201d\n--- /dev/null\n+++ b/read-cache-v5.c\n@@ -0,0 +1,589 @@\n+#include \"cache.h\"\n+#include \"read-cache.h\"\n+#include \"resolve-undo.h\"\n+#include \"cache-tree.h\"\n+\n+struct cache_header {\n+\tunsigned int hdr_ndir;\n+\tunsigned int hdr_nfile;\n+\tunsigned int hdr_fblockoffset;\n+\tunsigned int hdr_nextension;\n+};\n+\n+/*****************************************************************\n+ * Index File I/O\n+ *****************************************************************/\n+\n+struct ondisk_cache_entry {\n+\tunsigned short flags;\n+\tunsigned short mode;\n+\tstruct cache_time mtime;\n+\tunsigned int size;\n+\tint stat_crc;\n+\tunsigned char sha1[20];\n+};\n+\n+struct ondisk_directory_entry {\n+\tunsigned int foffset;\n+\tunsigned int cr;\n+\tunsigned int ncr;\n+\tunsigned int nsubtrees;\n+\tunsigned int nfiles;\n+\tunsigned int nentries;\n+\tunsigned char sha1[20];\n+\tunsigned short flags;\n+};\n+\n+static int check_crc32(int initialcrc,\n+\t\t\tvoid *data,\n+\t\t\tsize_t len,\n+\t\t\tunsigned int expected_crc)\n+{\n+\tint crc;\n+\n+\tcrc = crc32(initialcrc, (Bytef*)data, len);\n+\treturn crc == expected_crc;\n+}\n+\n+static int match_stat_crc(struct stat *st, uint32_t expected_crc)\n+{\n+\tuint32_t data, stat_crc = 0;\n+\tunsigned int ctimens = 0;\n+\n+\tdata = htonl(st->st_ctime);\n+\tstat_crc = crc32(0, (Bytef*)&data, 4);\n+#ifdef USE_NSEC\n+\tctimens = ST_MTIME_NSEC(*st);\n+#endif\n+\tdata = htonl(ctimens);\n+\tstat_crc = crc32(stat_crc, (Bytef*)&data, 4);\n+\tdata = htonl(st->st_ino);\n+\tstat_crc = crc32(stat_crc, (Bytef*)&data, 4);\n+\tdata = htonl(st->st_dev);\n+\tstat_crc = crc32(stat_crc, (Bytef*)&data, 4);\n+\tdata = htonl(st->st_uid);\n+\tstat_crc = crc32(stat_crc, (Bytef*)&data, 4);\n+\tdata = htonl(st->st_gid);\n+\tstat_crc = crc32(stat_crc, (Bytef*)&data, 4);\n+\n+\treturn stat_crc == expected_crc;\n+}\n+\n+static int match_stat_basic(struct cache_entry *ce,\n+\t\t\t\tstruct stat *st,\n+\t\t\t\tint changed)\n+{\n+\n+\tif (ce->ce_mtime.sec != 0 && ce->ce_mtime.sec != (unsigned int)st->st_mtime)\n+\t\tchanged |= MTIME_CHANGED;\n+#ifdef USE_NSEC\n+\tif (ce->ce_mtime.nsec != 0 && ce->ce_mtime.nsec != ST_MTIME_NSEC(*st))\n+\t\tchanged |= MTIME_CHANGED;\n+#endif\n+\tif (ce->ce_size != (unsigned int)st->st_size)\n+\t\tchanged |= DATA_CHANGED;\n+\n+\tif (ce->ce_stat_crc != 0 && !match_stat_crc(st, ce->ce_stat_crc)) {\n+\t\tchanged |= OWNER_CHANGED;\n+\t\tchanged |= INODE_CHANGED;\n+\t}\n+\t/* Racily smudged entry? */\n+\tif (!ce->ce_mtime.sec && !ce->ce_mtime.nsec) {\n+\t\tif (!changed && !is_empty_blob_sha1(ce->sha1) && ce_modified_check_fs(ce, st))\n+\t\t\tchanged |= DATA_CHANGED;\n+\t}\n+\treturn changed;\n+}\n+\n+static int verify_hdr(void *mmap, unsigned long size)\n+{\n+\tuint32_t *filecrc;\n+\tunsigned int header_size;\n+\tstruct cache_version_header *hdr;\n+\tstruct cache_header *hdr_v5;\n+\n+\tif (size < sizeof(struct cache_version_header)\n+\t\t\t+ sizeof (struct cache_header) + 4)\n+\t\tdie(\"index file smaller than expected\");\n+\n+\thdr = mmap;\n+\thdr_v5 = (struct cache_header *)((char *)mmap + sizeof(*hdr));\n+\t/* Size of the header + the size of the extensionoffsets */\n+\theader_size = sizeof(*hdr) + sizeof(*hdr_v5) + hdr_v5->hdr_nextension * 4;\n+\t/* Initialize crc */\n+\tfilecrc = (uint32_t *)((char *)mmap + header_size);\n+\tif (!check_crc32(0, hdr, header_size, ntohl(*filecrc)))\n+\t\treturn error(\"bad index file header crc signature\");\n+\treturn 0;\n+}\n+\n+static struct cache_entry *cache_entry_from_ondisk(struct ondisk_cache_entry *ondisk,\n+\t\t\t\t\t\t struct directory_entry *de,\n+\t\t\t\t\t\t char *name,\n+\t\t\t\t\t\t size_t len,\n+\t\t\t\t\t\t size_t prefix_len)\n+{\n+\tstruct cache_entry *ce = xmalloc(cache_entry_size(len + de->de_pathlen));\n+\tint flags;\n+\n+\tflags = ntoh_s(ondisk->flags);\n+\tce->ce_ctime.sec = 0;\n+\tce->ce_mtime.sec = ntoh_l(ondisk->mtime.sec);\n+\tce->ce_ctime.nsec = 0;\n+\tce->ce_mtime.nsec = ntoh_l(ondisk->mtime.nsec);\n+\tce->ce_dev = 0;\n+\tce->ce_ino = 0;\n+\tce->ce_mode = ntoh_s(ondisk->mode);\n+\tce->ce_uid = 0;\n+\tce->ce_gid = 0;\n+\tce->ce_size = ntoh_l(ondisk->size);\n+\tce->ce_flags = flags & CE_STAGEMASK;\n+\tce->ce_flags |= flags & CE_VALID;\n+\tif (flags & CE_INTENT_TO_ADD_V5)\n+\t\tce->ce_flags |= CE_INTENT_TO_ADD;\n+\tif (flags & CE_SKIP_WORKTREE_V5)\n+\t\tce->ce_flags |= CE_SKIP_WORKTREE;\n+\tce->ce_stat_crc = ntoh_l(ondisk->stat_crc);\n+\tce->ce_namelen = len + de->de_pathlen;\n+\thashcpy(ce->sha1, ondisk->sha1);\n+\tmemcpy(ce->name, de->pathname, de->de_pathlen);\n+\tmemcpy(ce->name + de->de_pathlen, name, len);\n+\tce->name[len + de->de_pathlen] = '\\0';\n+\treturn ce;\n+}\n+\n+static struct directory_entry *directory_entry_from_ondisk(struct ondisk_directory_entry *ondisk,\n+\t\t\t\t\t\t const char *name,\n+\t\t\t\t\t\t size_t len)\n+{\n+\tstruct directory_entry *de = xmalloc(directory_entry_size(len));\n+\n+\n+\tmemcpy(de->pathname, name, len);\n+\tde->pathname[len] = '\\0';\n+\tde->de_flags = ntoh_s(ondisk->flags);\n+\tde->de_foffset = ntoh_l(ondisk->foffset);\n+\tde->de_cr = ntoh_l(ondisk->cr);\n+\tde->de_ncr = ntoh_l(ondisk->ncr);\n+\tde->de_nsubtrees = ntoh_l(ondisk->nsubtrees);\n+\tde->de_nfiles = ntoh_l(ondisk->nfiles);\n+\tde->de_nentries = ntoh_l(ondisk->nentries);\n+\tde->de_pathlen = len;\n+\thashcpy(de->sha1, ondisk->sha1);\n+\treturn de;\n+}\n+\n+static struct conflict_part *conflict_part_from_ondisk(struct ondisk_conflict_part *ondisk)\n+{\n+\tstruct conflict_part *cp = xmalloc(sizeof(struct conflict_part));\n+\n+\tcp->flags = ntoh_s(ondisk->flags);\n+\tcp->entry_mode = ntoh_s(ondisk->entry_mode);\n+\thashcpy(cp->sha1, ondisk->sha1);\n+\treturn cp;\n+}\n+\n+static struct cache_entry *convert_conflict_part(struct conflict_part *cp,\n+\t\t\t\t\t\tchar * name,\n+\t\t\t\t\t\tunsigned int len)\n+{\n+\n+\tstruct cache_entry *ce = xmalloc(cache_entry_size(len));\n+\n+\tce->ce_ctime.sec = 0;\n+\tce->ce_mtime.sec = 0;\n+\tce->ce_ctime.nsec = 0;\n+\tce->ce_mtime.nsec = 0;\n+\tce->ce_dev = 0;\n+\tce->ce_ino = 0;\n+\tce->ce_mode = cp->entry_mode;\n+\tce->ce_uid = 0;\n+\tce->ce_gid = 0;\n+\tce->ce_size = 0;\n+\tce->ce_flags = conflict_stage(cp) << CE_STAGESHIFT;\n+\tce->ce_stat_crc = 0;\n+\tce->ce_namelen = len;\n+\thashcpy(ce->sha1, cp->sha1);\n+\tmemcpy(ce->name, name, len);\n+\tce->name[len] = '\\0';\n+\treturn ce;\n+}\n+\n+static struct directory_entry *read_directories(unsigned int *dir_offset,\n+\t\t\t\tunsigned int *dir_table_offset,\n+\t\t\t\tvoid *mmap,\n+\t\t\t\tint mmap_size)\n+{\n+\tint i, ondisk_directory_size;\n+\tuint32_t *filecrc, *beginning, *end;\n+\tstruct directory_entry *current = NULL;\n+\tstruct ondisk_directory_entry *disk_de;\n+\tstruct directory_entry *de;\n+\tunsigned int data_len, len;\n+\tchar *name;\n+\n+\tondisk_directory_size = sizeof(disk_de->flags)\n+\t\t+ sizeof(disk_de->foffset)\n+\t\t+ sizeof(disk_de->cr)\n+\t\t+ sizeof(disk_de->ncr)\n+\t\t+ sizeof(disk_de->nsubtrees)\n+\t\t+ sizeof(disk_de->nfiles)\n+\t\t+ sizeof(disk_de->nentries)\n+\t\t+ sizeof(disk_de->sha1);\n+\tname = (char *)mmap + *dir_offset;\n+\tbeginning = (uint32_t *)((char *)mmap + *dir_table_offset);\n+\tend = (uint32_t *)((char *)mmap + *dir_table_offset + 4);\n+\tlen = ntoh_l(*end) - ntoh_l(*beginning) - ondisk_directory_size - 5;\n+\tdisk_de = (struct ondisk_directory_entry *)\n+\t\t\t((char *)mmap + *dir_offset + len + 1);\n+\tde = directory_entry_from_ondisk(disk_de, name, len);\n+\tde->next = NULL;\n+\n+\t/* Length of pathname + nul byte for termination + size of\n+\t * members of ondisk_directory_entry. (Just using the size\n+\t * of the stuct doesn't work, because there may be padding\n+\t * bytes for the struct)\n+\t */\n+\tdata_len = len + 1 + ondisk_directory_size;\n+\n+\tfilecrc = (uint32_t *)((char *)mmap + *dir_offset + data_len);\n+\tif (!check_crc32(0, mmap + *dir_offset, data_len, ntoh_l(*filecrc)))\n+\t\tgoto unmap;\n+\n+\t*dir_table_offset += 4;\n+\t*dir_offset += data_len + 4; /* crc code */\n+\n+\tcurrent = de;\n+\tfor (i = 0; i < de->de_nsubtrees; i++) {\n+\t\tcurrent->next = read_directories(dir_offset, dir_table_offset,\n+\t\t\t\t\t\tmmap, mmap_size);\n+\t\twhile (current->next)\n+\t\t\tcurrent = current->next;\n+\t}\n+\n+\treturn de;\n+unmap:\n+\tmunmap(mmap, mmap_size);\n+\tdie(\"directory crc doesn't match for '%s'\", de->pathname);\n+}\n+\n+static struct cache_entry *read_entry(struct directory_entry *de,\n+\t\t\tunsigned long *entry_offset,\n+\t\t\tvoid **mmap,\n+\t\t\tunsigned long mmap_size,\n+\t\t\tunsigned int *foffsetblock,\n+\t\t\tint fd)\n+{\n+\tint len, crc_wrong, i = 0, offset_to_offset;\n+\tchar *name;\n+\tuint32_t foffsetblockcrc;\n+\tuint32_t *filecrc, *beginning, *end;\n+\tstruct cache_entry *ce;\n+\tstruct ondisk_cache_entry *disk_ce;\n+\n+\tdo {\n+\t\tname = (char *)*mmap + *entry_offset;\n+\t\tbeginning = (uint32_t *)((char *)*mmap + *foffsetblock);\n+\t\tend = (uint32_t *)((char *)*mmap + *foffsetblock + 4);\n+\t\tlen = ntoh_l(*end) - ntoh_l(*beginning) - sizeof(struct ondisk_cache_entry) - 5;\n+\t\tdisk_ce = (struct ondisk_cache_entry *)\n+\t\t\t\t((char *)*mmap + *entry_offset + len + 1);\n+\t\tce = cache_entry_from_ondisk(disk_ce, de, name, len, de->de_pathlen);\n+\t\tfilecrc = (uint32_t *)((char *)*mmap + *entry_offset + len + 1 + sizeof(*disk_ce));\n+\t\toffset_to_offset = htonl(*foffsetblock);\n+\t\tfoffsetblockcrc = crc32(0, (Bytef*)&offset_to_offset, 4);\n+\t\tcrc_wrong = !check_crc32(foffsetblockcrc,\n+\t\t\t(char *)*mmap + *entry_offset, len + 1 + sizeof(*disk_ce),\n+\t\t\tntoh_l(*filecrc));\n+\t\tif (crc_wrong) {\n+\t\t\t/* wait for 10 milliseconds */\n+\t\t\tusleep(10*1000);\n+\t\t\tmunmap(*mmap, mmap_size);\n+\t\t\t*mmap = xmmap(NULL, mmap_size, PROT_READ | PROT_WRITE, MAP_PRIVATE, fd, 0);\n+\t\t}\n+\t\ti++;\n+\t\t/*\n+\t\t * Retry for 500 ms maximum, before giving up and saying the\n+\t\t * checksum is wrong.\n+\t\t */\n+\t} while (crc_wrong && i < 50);\n+\tif (crc_wrong)\n+\t\tgoto unmap;\n+\t*entry_offset += len + 1 + sizeof(*disk_ce) + 4;\n+\treturn ce;\n+unmap:\n+\tmunmap(*mmap, mmap_size);\n+\tdie(\"file crc doesn't match for '%s'\", ce->name);\n+}\n+\n+static void ce_queue_push(struct cache_entry **head,\n+\t\t\t struct cache_entry **tail,\n+\t\t\t struct cache_entry *ce)\n+{\n+\tif (!*head) {\n+\t\t*head = *tail = ce;\n+\t\t(*tail)->next = NULL;\n+\t\treturn;\n+\t}\n+\n+\t(*tail)->next = ce;\n+\tce->next = NULL;\n+\t*tail = (*tail)->next;\n+}\n+\n+static void conflict_entry_push(struct conflict_entry **head,\n+\t\t\t\tstruct conflict_entry **tail,\n+\t\t\t\tstruct conflict_entry *conflict_entry)\n+{\n+\tif (!*head) {\n+\t\t*head = *tail = conflict_entry;\n+\t\t(*tail)->next = NULL;\n+\t\treturn;\n+\t}\n+\n+\t(*tail)->next = conflict_entry;\n+\tconflict_entry->next = NULL;\n+\t*tail = (*tail)->next;\n+}\n+\n+static struct cache_entry *ce_queue_pop(struct cache_entry **head)\n+{\n+\tstruct cache_entry *ce;\n+\n+\tce = *head;\n+\t*head = (*head)->next;\n+\treturn ce;\n+}\n+\n+static void conflict_part_head_remove(struct conflict_part **head)\n+{\n+\tstruct conflict_part *to_free;\n+\n+\tto_free = *head;\n+\t*head = (*head)->next;\n+\tfree(to_free);\n+}\n+\n+static void conflict_entry_head_remove(struct conflict_entry **head)\n+{\n+\tstruct conflict_entry *to_free;\n+\n+\tto_free = *head;\n+\t*head = (*head)->next;\n+\tfree(to_free);\n+}\n+\n+struct conflict_entry *create_new_conflict(char *name, int len, int pathlen)\n+{\n+\tstruct conflict_entry *conflict_entry;\n+\n+\tif (pathlen)\n+\t\tpathlen++;\n+\tconflict_entry = xmalloc(conflict_entry_size(len));\n+\tconflict_entry->entries = NULL;\n+\tconflict_entry->nfileconflicts = 0;\n+\tconflict_entry->namelen = len;\n+\tmemcpy(conflict_entry->name, name, len);\n+\tconflict_entry->name[len] = '\\0';\n+\tconflict_entry->pathlen = pathlen;\n+\tconflict_entry->next = NULL;\n+\n+\treturn conflict_entry;\n+}\n+\n+void add_part_to_conflict_entry(struct directory_entry *de,\n+\t\t\t\t\tstruct conflict_entry *entry,\n+\t\t\t\t\tstruct conflict_part *conflict_part)\n+{\n+\n+\tstruct conflict_part *conflict_search;\n+\n+\tentry->nfileconflicts++;\n+\tde->conflict_size += sizeof(struct ondisk_conflict_part);\n+\tif (!entry->entries)\n+\t\tentry->entries = conflict_part;\n+\telse {\n+\t\tconflict_search = entry->entries;\n+\t\twhile (conflict_search->next)\n+\t\t\tconflict_search = conflict_search->next;\n+\t\tconflict_search->next = conflict_part;\n+\t}\n+}\n+\n+static struct conflict_entry *read_conflicts(struct directory_entry *de,\n+\t\t\t\t\t\tvoid **mmap,\n+\t\t\t\t\t\tunsigned long mmap_size,\n+\t\t\t\t\t\tint fd)\n+{\n+\tstruct conflict_entry *head, *tail;\n+\tunsigned int croffset, i, j = 0;\n+\tchar *full_name;\n+\n+\tcroffset = de->de_cr;\n+\ttail = NULL;\n+\thead = NULL;\n+\tfor (i = 0; i < de->de_ncr; i++) {\n+\t\tstruct conflict_entry *conflict_new;\n+\t\tunsigned int len, *nfileconflicts;\n+\t\tchar *name;\n+\t\tvoid *crc_start;\n+\t\tint k, offset, crc_wrong;\n+\t\tuint32_t *filecrc;\n+\n+\t\tdo {\n+\t\t\toffset = croffset;\n+\t\t\tcrc_start = (void *)((char *)*mmap + offset);\n+\t\t\tname = (char *)*mmap + offset;\n+\t\t\tlen = strlen(name);\n+\t\t\toffset += len + 1;\n+\t\t\tnfileconflicts = (unsigned int *)((char *)*mmap + offset);\n+\t\t\toffset += 4;\n+\n+\t\t\tfull_name = xmalloc(sizeof(char) * (len + de->de_pathlen));\n+\t\t\tmemcpy(full_name, de->pathname, de->de_pathlen);\n+\t\t\tmemcpy(full_name + de->de_pathlen, name, len);\n+\t\t\tconflict_new = create_new_conflict(full_name,\n+\t\t\t\t\tlen + de->de_pathlen, de->de_pathlen);\n+\t\t\tfor (k = 0; k < ntoh_l(*nfileconflicts); k++) {\n+\t\t\t\tstruct ondisk_conflict_part *ondisk;\n+\t\t\t\tstruct conflict_part *cp;\n+\n+\t\t\t\tondisk = (struct ondisk_conflict_part *)((char *)*mmap + offset);\n+\t\t\t\tcp = conflict_part_from_ondisk(ondisk);\n+\t\t\t\tcp->next = NULL;\n+\t\t\t\tadd_part_to_conflict_entry(de, conflict_new, cp);\n+\t\t\t\toffset += sizeof(struct ondisk_conflict_part);\n+\t\t\t}\n+\t\t\tfilecrc = (uint32_t *)((char *)*mmap + offset);\n+\t\t\tcrc_wrong = !check_crc32(0, crc_start,\n+\t\t\t\tlen + 1 + 4 + conflict_new->nfileconflicts\n+\t\t\t\t* sizeof(struct ondisk_conflict_part),\n+\t\t\t\tntoh_l(*filecrc));\n+\t\t\tif (crc_wrong) {\n+\t\t\t\t/* wait for 10 milliseconds */\n+\t\t\t\tusleep(10*1000);\n+\t\t\t\tmunmap(*mmap, mmap_size);\n+\t\t\t\t*mmap = xmmap(NULL, mmap_size, PROT_READ | PROT_WRITE, MAP_PRIVATE, fd, 0);\n+\t\t\t}\n+\t\t\tfree(full_name);\n+\t\t\tj++;\n+\t\t} while (crc_wrong && j < 50);\n+\t\tif (crc_wrong)\n+\t\t\tgoto unmap;\n+\t\tcroffset = offset + 4;\n+\t\tconflict_entry_push(&head, &tail, conflict_new);\n+\t}\n+\treturn head;\n+unmap:\n+\tmunmap(*mmap, mmap_size);\n+\tdie(\"wrong crc for conflict: %s\", full_name);\n+}\n+\n+static struct directory_entry *read_entries(struct index_state *istate,\n+\t\t\t\t\tstruct directory_entry *de,\n+\t\t\t\t\tunsigned long *entry_offset,\n+\t\t\t\t\tvoid **mmap,\n+\t\t\t\t\tunsigned long mmap_size,\n+\t\t\t\t\tint *nr,\n+\t\t\t\t\tunsigned int *foffsetblock,\n+\t\t\t\t\tint fd)\n+{\n+\tstruct cache_entry *head = NULL, *tail = NULL;\n+\tstruct conflict_entry *conflict_queue;\n+\tstruct cache_entry *ce;\n+\tint i;\n+\n+\tconflict_queue = read_conflicts(de, mmap, mmap_size, fd);\n+\tfor (i = 0; i < de->de_nfiles; i++) {\n+\t\tce = read_entry(de,\n+\t\t\t\tentry_offset,\n+\t\t\t\tmmap,\n+\t\t\t\tmmap_size,\n+\t\t\t\tfoffsetblock,\n+\t\t\t\tfd);\n+\t\tce_queue_push(&head, &tail, ce);\n+\t\t*foffsetblock += 4;\n+\n+\t\t/* Add the conflicted entries at the end of the index file\n+\t\t * to the in memory format\n+\t\t */\n+\t\tif (conflict_queue &&\n+\t\t (conflict_queue->entries->flags & CONFLICT_CONFLICTED) != 0 &&\n+\t\t !cache_name_compare(conflict_queue->name, conflict_queue->namelen,\n+\t\t\t\t\tce->name, ce_namelen(ce))) {\n+\t\t\tstruct conflict_part *cp;\n+\t\t\tcp = conflict_queue->entries;\n+\t\t\tcp = cp->next;\n+\t\t\twhile (cp) {\n+\t\t\t\tce = convert_conflict_part(cp,\n+\t\t\t\t\t\tconflict_queue->name,\n+\t\t\t\t\t\tconflict_queue->namelen);\n+\t\t\t\tce_queue_push(&head, &tail, ce);\n+\t\t\t\tconflict_part_head_remove(&cp);\n+\t\t\t}\n+\t\t\tconflict_entry_head_remove(&conflict_queue);\n+\t\t}\n+\t}\n+\n+\tde = de->next;\n+\n+\twhile (head) {\n+\t\tif (de != NULL\n+\t\t && strcmp(head->name, de->pathname) > 0) {\n+\t\t\tde = read_entries(istate,\n+\t\t\t\t\tde,\n+\t\t\t\t\tentry_offset,\n+\t\t\t\t\tmmap,\n+\t\t\t\t\tmmap_size,\n+\t\t\t\t\tnr,\n+\t\t\t\t\tfoffsetblock,\n+\t\t\t\t\tfd);\n+\t\t} else {\n+\t\t\tce = ce_queue_pop(&head);\n+\t\t\tset_index_entry(istate, *nr, ce);\n+\t\t\t(*nr)++;\n+\t\t}\n+\t}\n+\n+\treturn de;\n+}\n+\n+static void read_index_v5(struct index_state *istate, void *mmap, int mmap_size, int fd)\n+{\n+\tunsigned long entry_offset;\n+\tunsigned int dir_offset, dir_table_offset;\n+\tstruct cache_version_header *hdr;\n+\tstruct cache_header *hdr_v5;\n+\tstruct directory_entry *root_directory, *de;\n+\tint nr;\n+\tunsigned int foffsetblock;\n+\n+\thdr = mmap;\n+\thdr_v5 = (struct cache_header *)((char *)mmap + sizeof(*hdr));\n+\tistate->version = ntohl(hdr->hdr_version);\n+\tistate->cache_nr = ntohl(hdr_v5->hdr_nfile);\n+\tistate->cache_alloc = alloc_nr(istate->cache_nr);\n+\tistate->cache = xcalloc(istate->cache_alloc, sizeof(struct cache_entry *));\n+\tistate->initialized = 1;\n+\n+\t/* Skip size of the header + crc sum + size of offsets */\n+\tdir_offset = sizeof(*hdr) + sizeof(*hdr_v5) + 4 + (ntohl(hdr_v5->hdr_ndir) + 1) * 4;\n+\tdir_table_offset = sizeof(*hdr) + sizeof(*hdr_v5) + 4;\n+\troot_directory = read_directories(&dir_offset, &dir_table_offset, mmap, mmap_size);\n+\n+\tentry_offset = ntohl(hdr_v5->hdr_fblockoffset);\n+\n+\tnr = 0;\n+\tfoffsetblock = dir_offset;\n+\tde = root_directory;\n+\twhile (de)\n+\t\tde = read_entries(istate, de, &entry_offset,\n+\t\t\t\t&mmap, mmap_size, &nr, &foffsetblock, fd);\n+}\n+\n+struct index_ops v5_ops = {\n+\tmatch_stat_basic,\n+\tverify_hdr,\n+\tread_index_v5,\n+\tNULL\n+};\ndiff --git a/read-cache.c b/read-cache.c\nindex d8f8b74..5fb4d2e 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -1175,7 +1175,6 @@ static struct cache_entry *refresh_cache_entry(struct cache_entry *ce, int reall\n \treturn refresh_cache_ent(&the_index, ce, really, NULL, NULL);\n }\n \n-\n static int verify_hdr_version(struct index_state *istate,\n \t\tstruct cache_version_header *hdr, unsigned long size)\n {\n-- \n1.7.10.GIT\n"},{"id":"196658","messageId":"1344424681-31469-8-git-send-email-t.gummerer@gmail.com","threadId":"31208","inReplyTo":"1344424681-31469-1-git-send-email-t.gummerer@gmail.com","subject":"[PATCH/RFC v3 07/13] Read resolve-undo data","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2012-08-08T11:17:55Z","receivedAt":"2012-08-08T11:17:55Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"Make git read the resolve-undo data from the index.\n\nSince the resolve-undo data is joined with the conflicts in\nthe ondisk format of the index file version 5, conflicts and\nresolved data is read at the same time, and the resolve-undo\ndata is then converted to the in-memory format.\n\nHelped-by: Thomas Rast \nSigned-off-by: Thomas Gummerer \n---\n read-cache-v5.c | 1 +\n resolve-undo.c | 36 ++++++++++++++++++++++++++++++++++++\n resolve-undo.h | 2 ++\n 3 files changed, 39 insertions(+)\n\ndiff --git a/read-cache-v5.c b/read-cache-v5.c\nindex ec1201d..b47398d 100644\n--- a/read-cache-v5.c\n+++ b/read-cache-v5.c\n@@ -494,6 +494,7 @@ static struct directory_entry *read_entries(struct index_state *istate,\n \tint i;\n \n \tconflict_queue = read_conflicts(de, mmap, mmap_size, fd);\n+\tresolve_undo_convert_v5(istate, conflict_queue);\n \tfor (i = 0; i < de->de_nfiles; i++) {\n \t\tce = read_entry(de,\n \t\t\t\tentry_offset,\ndiff --git a/resolve-undo.c b/resolve-undo.c\nindex 72b4612..f96c6ba 100644\n--- a/resolve-undo.c\n+++ b/resolve-undo.c\n@@ -170,3 +170,39 @@ void unmerge_index(struct index_state *istate, const char **pathspec)\n \t\ti = unmerge_index_entry_at(istate, i);\n \t}\n }\n+\n+void resolve_undo_convert_v5(struct index_state *istate,\n+\t\t\t\t\tstruct conflict_entry *ce)\n+{\n+\tint i;\n+\n+\twhile (ce) {\n+\t\tstruct string_list_item *lost;\n+\t\tstruct resolve_undo_info *ui;\n+\t\tstruct conflict_part *cp;\n+\n+\t\tif (ce->entries && (ce->entries->flags & CONFLICT_CONFLICTED) != 0) {\n+\t\t\tce = ce->next;\n+\t\t\tcontinue;\n+\t\t}\n+\t\tif (!istate->resolve_undo) {\n+\t\t\tistate->resolve_undo = xcalloc(1, sizeof(struct string_list));\n+\t\t\tistate->resolve_undo->strdup_strings = 1;\n+\t\t}\n+\n+\t\tlost = string_list_insert(istate->resolve_undo, ce->name);\n+\t\tif (!lost->util)\n+\t\t\tlost->util = xcalloc(1, sizeof(*ui));\n+\t\tui = lost->util;\n+\n+\t\tcp = ce->entries;\n+\t\tfor (i = 0; i < 3; i++)\n+\t\t\tui->mode[i] = 0;\n+\t\twhile (cp) {\n+\t\t\tui->mode[conflict_stage(cp) - 1] = cp->entry_mode;\n+\t\t\thashcpy(ui->sha1[conflict_stage(cp) - 1], cp->sha1);\n+\t\t\tcp = cp->next;\n+\t\t}\n+\t\tce = ce->next;\n+\t}\n+}\ndiff --git a/resolve-undo.h b/resolve-undo.h\nindex 8458769..ab660a6 100644\n--- a/resolve-undo.h\n+++ b/resolve-undo.h\n@@ -13,4 +13,6 @@ extern void resolve_undo_clear_index(struct index_state *);\n extern int unmerge_index_entry_at(struct index_state *, int);\n extern void unmerge_index(struct index_state *, const char **);\n \n+extern void resolve_undo_convert_v5(struct index_state *, struct conflict_entry *);\n+\n #endif\n-- \n1.7.10.GIT\n"},{"id":"196649","messageId":"1344424681-31469-9-git-send-email-t.gummerer@gmail.com","threadId":"31208","inReplyTo":"1344424681-31469-1-git-send-email-t.gummerer@gmail.com","subject":"[PATCH/RFC v3 08/13] Read cache-tree in index-v5","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2012-08-08T11:17:56Z","receivedAt":"2012-08-08T11:17:56Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"Since the cache-tree data is saved as part of the directory data,\nwe already read it at the beginning of the index. The cache-tree\nis only converted from this directory data.\n\nThe cache-tree data is arranged in a tree, with the children sorted by\npathlen at each node, while the ondisk format is sorted lexically.\nSo we have to rebuild this format from the on-disk directory list.\n\nSigned-off-by: Thomas Gummerer \n---\n cache-tree.c | 93 +++++++++++++++++++++++++++++++++++++++++++++++++++++++\n cache-tree.h | 10 ++++++\n read-cache-v5.c | 1 +\n 3 files changed, 104 insertions(+)\n\ndiff --git a/cache-tree.c b/cache-tree.c\nindex 28ed657..440cd04 100644\n--- a/cache-tree.c\n+++ b/cache-tree.c\n@@ -519,6 +519,99 @@ struct cache_tree *cache_tree_read(const char *buffer, unsigned long size)\n \treturn read_one(&buffer, &size);\n }\n \n+static struct cache_tree *convert_one(struct directory_queue *queue, int dirnr)\n+{\n+\tint i, subtree_nr;\n+\tstruct cache_tree *it;\n+\tstruct directory_queue *down;\n+\n+\tit = cache_tree();\n+\tit->entry_count = queue[dirnr].de->de_nentries;\n+\tsubtree_nr = queue[dirnr].de->de_nsubtrees;\n+\tif (0 <= it->entry_count)\n+\t\thashcpy(it->sha1, queue[dirnr].de->sha1);\n+\n+\t/*\n+\t* Just a heuristic -- we do not add directories that often but\n+\t* we do not want to have to extend it immediately when we do,\n+\t* hence +2.\n+\t*/\n+\tit->subtree_alloc = subtree_nr + 2;\n+\tit->down = xcalloc(it->subtree_alloc, sizeof(struct cache_tree_sub *));\n+\tdown = queue[dirnr].down;\n+\tfor (i = 0; i < subtree_nr; i++) {\n+\t\tstruct cache_tree *sub;\n+\t\tstruct cache_tree_sub *subtree;\n+\t\tchar *buf, *name;\n+\n+\t\tname = \"\";\n+\t\tbuf = strtok(down[i].de->pathname, \"/\");\n+\t\twhile (buf) {\n+\t\t\tname = buf;\n+\t\t\tbuf = strtok(NULL, \"/\");\n+\t\t}\n+\t\tsub = convert_one(down, i);\n+\t\tif(!sub)\n+\t\t\tgoto free_return;\n+\t\tsubtree = cache_tree_sub(it, name);\n+\t\tsubtree->cache_tree = sub;\n+\t}\n+\tif (subtree_nr != it->subtree_nr)\n+\t\tdie(\"cache-tree: internal error\");\n+\treturn it;\n+ free_return:\n+\tcache_tree_free(&it);\n+\treturn NULL;\n+}\n+\n+static int compare_cache_tree_elements(const void *a, const void *b)\n+{\n+\tconst struct directory_entry *de1, *de2;\n+\n+\tde1 = ((const struct directory_queue *)a)->de;\n+\tde2 = ((const struct directory_queue *)b)->de;\n+\treturn subtree_name_cmp(de1->pathname, de1->de_pathlen,\n+\t\t\t\tde2->pathname, de2->de_pathlen);\n+}\n+\n+static struct directory_entry *sort_directories(struct directory_entry *de,\n+\t\t\t\t\t\tstruct directory_queue *queue)\n+{\n+\tint i, nsubtrees;\n+\n+\tnsubtrees = de->de_nsubtrees;\n+\tfor (i = 0; i < nsubtrees; i++) {\n+\t\tstruct directory_entry *new_de;\n+\t\tde = de->next;\n+\t\tnew_de = xmalloc(directory_entry_size(de->de_pathlen));\n+\t\tmemcpy(new_de, de, directory_entry_size(de->de_pathlen));\n+\t\tqueue[i].de = new_de;\n+\t\tif (de->de_nsubtrees) {\n+\t\t\tqueue[i].down = xcalloc(de->de_nsubtrees,\n+\t\t\t\t\tsizeof(struct directory_queue));\n+\t\t\tde = sort_directories(de,\n+\t\t\t\t\tqueue[i].down);\n+\t\t}\n+\t}\n+\tqsort(queue, nsubtrees, sizeof(struct directory_queue),\n+\t\t\tcompare_cache_tree_elements);\n+\treturn de;\n+}\n+\n+struct cache_tree *cache_tree_convert_v5(struct directory_entry *de)\n+{\n+\tstruct directory_queue *queue;\n+\n+\tif (!de->de_nentries)\n+\t\treturn NULL;\n+\tqueue = xcalloc(1, sizeof(struct directory_queue));\n+\tqueue[0].de = de;\n+\tqueue[0].down = xcalloc(de->de_nsubtrees, sizeof(struct directory_queue));\n+\n+\tsort_directories(de, queue[0].down);\n+\treturn convert_one(queue, 0);\n+}\n+\n static struct cache_tree *cache_tree_find(struct cache_tree *it, const char *path)\n {\n \tif (!it)\ndiff --git a/cache-tree.h b/cache-tree.h\nindex d8cb2e9..7f29d26 100644\n--- a/cache-tree.h\n+++ b/cache-tree.h\n@@ -20,6 +20,11 @@ struct cache_tree {\n \tstruct cache_tree_sub **down;\n };\n \n+struct directory_queue {\n+\tstruct directory_queue *down;\n+\tstruct directory_entry *de;\n+};\n+\n struct cache_tree *cache_tree(void);\n void cache_tree_free(struct cache_tree **);\n void cache_tree_invalidate_path(struct cache_tree *, const char *);\n@@ -27,6 +32,11 @@ struct cache_tree_sub *cache_tree_sub(struct cache_tree *, const char *);\n \n void cache_tree_write(struct strbuf *, struct cache_tree *root);\n struct cache_tree *cache_tree_read(const char *buffer, unsigned long size);\n+/*\n+ * This function modifys the directory argument that is given to it.\n+ * Don't use it if the directory entries are still needed after.\n+ */\n+struct cache_tree *cache_tree_convert_v5(struct directory_entry *de);\n \n int cache_tree_fully_valid(struct cache_tree *);\n int cache_tree_update(struct cache_tree *, struct cache_entry **, int, int);\ndiff --git a/read-cache-v5.c b/read-cache-v5.c\nindex b47398d..57d0fb5 100644\n--- a/read-cache-v5.c\n+++ b/read-cache-v5.c\n@@ -580,6 +580,7 @@ static void read_index_v5(struct index_state *istate, void *mmap, int mmap_size,\n \twhile (de)\n \t\tde = read_entries(istate, de, &entry_offset,\n \t\t\t\t&mmap, mmap_size, &nr, &foffsetblock, fd);\n+\tistate->cache_tree = cache_tree_convert_v5(root_directory);\n }\n \n struct index_ops v5_ops = {\n-- \n1.7.10.GIT\n"},{"id":"196659","messageId":"1344424681-31469-10-git-send-email-t.gummerer@gmail.com","threadId":"31208","inReplyTo":"1344424681-31469-1-git-send-email-t.gummerer@gmail.com","subject":"[PATCH/RFC v3 09/13] Write index-v5","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2012-08-08T11:17:57Z","receivedAt":"2012-08-08T11:17:57Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"Write the index version 5 file format to disk. This version doesn't\nwrite the cache-tree data and resolve-undo data to the file.\n\nThe main work is done when filtering out the directories from the\ncurrent in-memory format, where in the same turn also the conflicts\nand the file data is calculated.\n\nHelped-by: Nguyen Thai Ngoc Duy \nHelped-by: Thomas Rast \nSigned-off-by: Thomas Gummerer \n---\n cache.h | 10 +-\n read-cache-v5.c | 589 ++++++++++++++++++++++++++++++++++++++++++++++++++++++-\n read-cache.c | 19 +-\n read-cache.h | 3 +\n 4 files changed, 611 insertions(+), 10 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex a0a1781..3fa348d 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -98,7 +98,7 @@ unsigned long git_deflate_bound(git_zstream *, unsigned long);\n #define CACHE_SIGNATURE 0x44495243\t/* \"DIRC\" */\n \n #define INDEX_FORMAT_LB 2\n-#define INDEX_FORMAT_UB 4\n+#define INDEX_FORMAT_UB 5\n \n /*\n * The \"cache_time\" is just the low 32 bits of the\n@@ -510,6 +510,7 @@ extern int verify_path(const char *path);\n extern struct cache_entry *index_name_exists(struct index_state *istate, const char *name, int namelen, int igncase);\n extern int index_name_stage_pos(const struct index_state *, const char *name, int namelen, int stage);\n extern int index_name_pos(const struct index_state *, const char *name, int namelen);\n+extern struct directory_entry *init_directory_entry(char *pathname, int len);\n #define ADD_CACHE_OK_TO_ADD 1\t\t/* Ok to add */\n #define ADD_CACHE_OK_TO_REPLACE 2\t/* Ok to replace file/directory */\n #define ADD_CACHE_SKIP_DFCHECK 4\t/* Ok to skip DF conflict checks */\n@@ -1244,6 +1245,13 @@ static inline ssize_t write_str_in_full(int fd, const char *str)\n \treturn write_in_full(fd, str, strlen(str));\n }\n \n+/* index-v5 helper functions */\n+extern char *super_directory(const char *filename);\n+extern void insert_directory_entry(struct directory_entry *, struct hash_table *, int *, unsigned int *, uint32_t);\n+extern void add_conflict_to_directory_entry(struct directory_entry *, struct conflict_entry *);\n+extern void add_part_to_conflict_entry(struct directory_entry *, struct conflict_entry *, struct conflict_part *);\n+extern struct conflict_entry *create_new_conflict(char *, int, int);\n+\n /* pager.c */\n extern void setup_pager(void);\n extern const char *pager_program;\ndiff --git a/read-cache-v5.c b/read-cache-v5.c\nindex 57d0fb5..45f7acd 100644\n--- a/read-cache-v5.c\n+++ b/read-cache-v5.c\n@@ -583,9 +583,596 @@ static void read_index_v5(struct index_state *istate, void *mmap, int mmap_size,\n \tistate->cache_tree = cache_tree_convert_v5(root_directory);\n }\n \n+#define WRITE_BUFFER_SIZE 8192\n+static unsigned char write_buffer[WRITE_BUFFER_SIZE];\n+static unsigned long write_buffer_len;\n+\n+static int ce_write_flush(int fd)\n+{\n+\tunsigned int buffered = write_buffer_len;\n+\tif (buffered) {\n+\t\tif (write_in_full(fd, write_buffer, buffered) != buffered)\n+\t\t\treturn -1;\n+\t\twrite_buffer_len = 0;\n+\t}\n+\treturn 0;\n+}\n+\n+static int ce_write(uint32_t *crc, int fd, void *data, unsigned int len)\n+{\n+\tif (crc)\n+\t\t*crc = crc32(*crc, (Bytef*)data, len);\n+\twhile (len) {\n+\t\tunsigned int buffered = write_buffer_len;\n+\t\tunsigned int partial = WRITE_BUFFER_SIZE - buffered;\n+\t\tif (partial > len)\n+\t\t\tpartial = len;\n+\t\tmemcpy(write_buffer + buffered, data, partial);\n+\t\tbuffered += partial;\n+\t\tif (buffered == WRITE_BUFFER_SIZE) {\n+\t\t\twrite_buffer_len = buffered;\n+\t\t\tif (ce_write_flush(fd))\n+\t\t\t\treturn -1;\n+\t\t\tbuffered = 0;\n+\t\t}\n+\t\twrite_buffer_len = buffered;\n+\t\tlen -= partial;\n+\t\tdata = (char *) data + partial;\n+\t}\n+\treturn 0;\n+}\n+\n+static int ce_flush(int fd)\n+{\n+\tunsigned int left = write_buffer_len;\n+\n+\tif (left)\n+\t\twrite_buffer_len = 0;\n+\n+\tif (write_in_full(fd, write_buffer, left) != left)\n+\t\treturn -1;\n+\n+\treturn 0;\n+}\n+\n+static void ce_smudge_racily_clean_entry(struct cache_entry *ce)\n+{\n+\t/*\n+\t * This method shall only be called if the timestamp of ce\n+\t * is racy (check with is_racy_timestamp). If the timestamp\n+\t * is racy, the writer will just set the time to 0.\n+\t *\n+\t * The reader (match_stat_basic) will then take care\n+\t * of checking if the entry is really changed or not, by\n+\t * taking into account the stat_crc and if that hasn't changed\n+\t * checking the sha1.\n+\t */\n+\tce->ce_mtime.sec = 0;\n+\tce->ce_mtime.nsec = 0;\n+}\n+\n+char *super_directory(const char *filename)\n+{\n+\tchar *slash;\n+\n+\tslash = strrchr(filename, '/');\n+\tif (slash)\n+\t\treturn xmemdupz(filename, slash-filename);\n+\treturn NULL;\n+}\n+\n+struct directory_entry *init_directory_entry(char *pathname, int len)\n+{\n+\tstruct directory_entry *de = xmalloc(directory_entry_size(len));\n+\n+\tmemcpy(de->pathname, pathname, len);\n+\tde->pathname[len] = '\\0';\n+\tde->de_flags = 0;\n+\tde->de_foffset = 0;\n+\tde->de_cr = 0;\n+\tde->de_ncr = 0;\n+\tde->de_nsubtrees = 0;\n+\tde->de_nfiles = 0;\n+\tde->de_nentries = 0;\n+\tmemset(de->sha1, 0, 20);\n+\tde->de_pathlen = len;\n+\tde->next = NULL;\n+\tde->next_hash = NULL;\n+\tde->ce = NULL;\n+\tde->ce_last = NULL;\n+\tde->conflict = NULL;\n+\tde->conflict_last = NULL;\n+\tde->conflict_size = 0;\n+\treturn de;\n+}\n+\n+static void ondisk_from_directory_entry(struct directory_entry *de,\n+\t\t\t\t\tstruct ondisk_directory_entry *ondisk)\n+{\n+\tondisk->foffset = htonl(de->de_foffset);\n+\tondisk->cr = htonl(de->de_cr);\n+\tondisk->ncr = htonl(de->de_ncr);\n+\tondisk->nsubtrees = htonl(de->de_nsubtrees);\n+\tondisk->nfiles = htonl(de->de_nfiles);\n+\tondisk->nentries = htonl(de->de_nentries);\n+\thashcpy(ondisk->sha1, de->sha1);\n+\tondisk->flags = htons(de->de_flags);\n+}\n+\n+static struct conflict_part *conflict_part_from_inmemory(struct cache_entry *ce)\n+{\n+\tstruct conflict_part *conflict;\n+\tint flags;\n+\n+\tconflict = xmalloc(sizeof(struct conflict_part));\n+\tflags = CONFLICT_CONFLICTED;\n+\tflags |= ce_stage(ce) << CONFLICT_STAGESHIFT;\n+\tconflict->flags = flags;\n+\tconflict->entry_mode = ce->ce_mode;\n+\tconflict->next = NULL;\n+\thashcpy(conflict->sha1, ce->sha1);\n+\treturn conflict;\n+}\n+\n+static void conflict_to_ondisk(struct conflict_part *cp,\n+\t\t\t\tstruct ondisk_conflict_part *ondisk)\n+{\n+\tondisk->flags = htons(cp->flags);\n+\tondisk->entry_mode = htons(cp->entry_mode);\n+\thashcpy(ondisk->sha1, cp->sha1);\n+}\n+\n+void add_conflict_to_directory_entry(struct directory_entry *de,\n+\t\t\t\t\tstruct conflict_entry *conflict_entry)\n+{\n+\tde->de_ncr++;\n+\tde->conflict_size += conflict_entry->namelen + 1 + 8 - conflict_entry->pathlen;\n+\tconflict_entry_push(&de->conflict, &de->conflict_last, conflict_entry);\n+}\n+\n+void insert_directory_entry(struct directory_entry *de,\n+\t\t\tstruct hash_table *table,\n+\t\t\tint *total_dir_len,\n+\t\t\tunsigned int *ndir,\n+\t\t\tuint32_t crc)\n+{\n+\tstruct directory_entry *insert;\n+\n+\tinsert = (struct directory_entry *)insert_hash(crc, de, table);\n+\tif (insert) {\n+\t\tde->next_hash = insert->next_hash;\n+\t\tinsert->next_hash = de;\n+\t}\n+\t(*ndir)++;\n+\tif (de->de_pathlen == 0)\n+\t\t(*total_dir_len)++;\n+\telse\n+\t\t*total_dir_len += de->de_pathlen + 2;\n+}\n+\n+static struct conflict_entry *create_conflict_entry_from_ce(struct cache_entry *ce,\n+\t\t\t\t\t\t\t\tint pathlen)\n+{\n+\treturn create_new_conflict(ce->name, ce_namelen(ce), pathlen);\n+}\n+\n+static struct directory_entry *compile_directory_data(struct index_state *istate,\n+\t\t\t\t\t\tint nfile,\n+\t\t\t\t\t\tunsigned int *ndir,\n+\t\t\t\t\t\tint *non_conflicted,\n+\t\t\t\t\t\tint *total_dir_len,\n+\t\t\t\t\t\tint *total_file_len)\n+{\n+\tint i, dir_len = -1;\n+\tchar *dir;\n+\tstruct directory_entry *de, *current, *search, *found, *new, *previous_entry;\n+\tstruct cache_entry **cache = istate->cache;\n+\tstruct conflict_entry *conflict_entry;\n+\tstruct hash_table table;\n+\tuint32_t crc;\n+\n+\tinit_hash(&table);\n+\tde = init_directory_entry(\"\", 0);\n+\tcurrent = de;\n+\t*ndir = 1;\n+\t*total_dir_len = 1;\n+\tcrc = crc32(0, (Bytef*)de->pathname, de->de_pathlen);\n+\tinsert_hash(crc, de, &table);\n+\tconflict_entry = NULL;\n+\tfor (i = 0; i < nfile; i++) {\n+\t\tint new_entry;\n+\t\tif (cache[i]->ce_flags & CE_REMOVE)\n+\t\t\tcontinue;\n+\n+\t\tnew_entry = !ce_stage(cache[i]) || !conflict_entry\n+\t\t || cache_name_compare(conflict_entry->name, conflict_entry->namelen,\n+\t\t\t\t\tcache[i]->name, ce_namelen(cache[i]));\n+\t\tif (new_entry)\n+\t\t\t(*non_conflicted)++;\n+\t\tif (dir_len < 0 || strncmp(cache[i]->name, dir, dir_len)\n+\t\t || cache[i]->name[dir_len] != '/'\n+\t\t || strchr(cache[i]->name + dir_len + 1, '/')) {\n+\t\t\tdir = super_directory(cache[i]->name);\n+\t\t\tif (!dir)\n+\t\t\t\tdir_len = 0;\n+\t\t\telse\n+\t\t\t\tdir_len = strlen(dir);\n+\t\t\tcrc = crc32(0, (Bytef*)dir, dir_len);\n+\t\t\tfound = lookup_hash(crc, &table);\n+\t\t\tsearch = found;\n+\t\t\twhile (search && dir_len != 0 && strcmp(dir, search->pathname) != 0)\n+\t\t\t\tsearch = search->next_hash;\n+\t\t}\n+\t\tprevious_entry = current;\n+\t\tif (!search || !found) {\n+\t\t\tnew = init_directory_entry(dir, dir_len);\n+\t\t\tcurrent->next = new;\n+\t\t\tcurrent = current->next;\n+\t\t\tinsert_directory_entry(new, &table, total_dir_len, ndir, crc);\n+\t\t\tsearch = current;\n+\t\t}\n+\t\tif (new_entry) {\n+\t\t\tsearch->de_nfiles++;\n+\t\t\t*total_file_len += ce_namelen(cache[i]) + 1;\n+\t\t\tif (search->de_pathlen)\n+\t\t\t\t*total_file_len -= search->de_pathlen + 1;\n+\t\t\tce_queue_push(&(search->ce), &(search->ce_last), cache[i]);\n+\t\t}\n+\t\tif (ce_stage(cache[i]) > 0) {\n+\t\t\tstruct conflict_part *conflict_part;\n+\t\t\tif (new_entry) {\n+\t\t\t\tconflict_entry = create_conflict_entry_from_ce(cache[i], search->de_pathlen);\n+\t\t\t\tadd_conflict_to_directory_entry(search, conflict_entry);\n+\t\t\t}\n+\t\t\tconflict_part = conflict_part_from_inmemory(cache[i]);\n+\t\t\tadd_part_to_conflict_entry(search, conflict_entry, conflict_part);\n+\t\t}\n+\t\tif (dir && !found) {\n+\t\t\tstruct directory_entry *no_subtrees;\n+\n+\t\t\tno_subtrees = current;\n+\t\t\tdir = super_directory(dir);\n+\t\t\tif (dir)\n+\t\t\t\tdir_len = strlen(dir);\n+\t\t\telse\n+\t\t\t\tdir_len = 0;\n+\t\t\tcrc = crc32(0, (Bytef*)dir, dir_len);\n+\t\t\tfound = lookup_hash(crc, &table);\n+\t\t\twhile (!found) {\n+\t\t\t\tnew = init_directory_entry(dir, dir_len);\n+\t\t\t\tnew->de_nsubtrees = 1;\n+\t\t\t\tnew->next = no_subtrees;\n+\t\t\t\tno_subtrees = new;\n+\t\t\t\tinsert_directory_entry(new, &table, total_dir_len, ndir, crc);\n+\t\t\t\tdir = super_directory(dir);\n+\t\t\t\tif (!dir)\n+\t\t\t\t\tdir_len = 0;\n+\t\t\t\telse\n+\t\t\t\t\tdir_len = strlen(dir);\n+\t\t\t\tcrc = crc32(0, (Bytef*)dir, dir_len);\n+\t\t\t\tfound = lookup_hash(crc, &table);\n+\t\t\t}\n+\t\t\tsearch = found;\n+\t\t\twhile (search->next_hash && strcmp(dir, search->pathname) != 0)\n+\t\t\t\tsearch = search->next_hash;\n+\t\t\tif (search)\n+\t\t\t\tfound = search;\n+\t\t\tfound->de_nsubtrees++;\n+\t\t\tprevious_entry->next = no_subtrees;\n+\t\t}\n+\t}\n+\treturn de;\n+}\n+\n+static void ondisk_from_cache_entry(struct cache_entry *ce,\n+\t\t\t\t struct ondisk_cache_entry *ondisk)\n+{\n+\tunsigned int flags;\n+\n+\tflags = ce->ce_flags & CE_STAGEMASK;\n+\tflags |= ce->ce_flags & CE_VALID;\n+\tif (ce->ce_flags & CE_INTENT_TO_ADD)\n+\t\tflags |= CE_INTENT_TO_ADD_V5;\n+\tif (ce->ce_flags & CE_SKIP_WORKTREE)\n+\t\tflags |= CE_SKIP_WORKTREE_V5;\n+\tondisk->flags = htons(flags);\n+\tondisk->mode = htons(ce->ce_mode);\n+\tondisk->mtime.sec = htonl(ce->ce_mtime.sec);\n+#ifdef USE_NSEC\n+\tondisk->mtime.nsec = htonl(ce->ce_mtime.nsec);\n+#else\n+\tondisk->mtime.nsec = 0;\n+#endif\n+\tondisk->size = htonl(ce->ce_size);\n+\tif (!ce->ce_stat_crc)\n+\t\tce->ce_stat_crc = calculate_stat_crc(ce);\n+\tondisk->stat_crc = htonl(ce->ce_stat_crc);\n+\thashcpy(ondisk->sha1, ce->sha1);\n+}\n+\n+static int write_directories(struct directory_entry *de, int fd, int conflict_offset)\n+{\n+\tstruct directory_entry *current;\n+\tstruct ondisk_directory_entry ondisk;\n+\tint current_offset, offset_write, ondisk_size, foffset;\n+\tuint32_t crc;\n+\n+\t/*\n+\t * This is needed because the compiler aligns structs to sizes multipe\n+\t * of 4\n+\t */\n+\tondisk_size = sizeof(ondisk.flags)\n+\t\t+ sizeof(ondisk.foffset)\n+\t\t+ sizeof(ondisk.cr)\n+\t\t+ sizeof(ondisk.ncr)\n+\t\t+ sizeof(ondisk.nsubtrees)\n+\t\t+ sizeof(ondisk.nfiles)\n+\t\t+ sizeof(ondisk.nentries)\n+\t\t+ sizeof(ondisk.sha1);\n+\tcurrent = de;\n+\tcurrent_offset = 0;\n+\tfoffset = 0;\n+\twhile (current) {\n+\t\tint pathlen;\n+\n+\t\toffset_write = htonl(current_offset);\n+\t\tif (ce_write(NULL, fd, &offset_write, 4) < 0)\n+\t\t\treturn -1;\n+\t\tif (current->de_pathlen == 0)\n+\t\t\tpathlen = 0;\n+\t\telse\n+\t\t\tpathlen = current->de_pathlen + 1;\n+\t\tcurrent_offset += pathlen + 1 + ondisk_size + 4;\n+\t\tcurrent = current->next;\n+\t}\n+\t/*\n+\t * Write one more offset, which points to the end of the entries,\n+\t * because we use it for calculating the dir length, instead of\n+\t * using strlen.\n+\t */\n+\toffset_write = htonl(current_offset);\n+\tif (ce_write(NULL, fd, &offset_write, 4) < 0)\n+\t\treturn -1;\n+\tcurrent = de;\n+\twhile (current) {\n+\t\tcrc = 0;\n+\t\tif (current->de_pathlen == 0) {\n+\t\t\tif (ce_write(&crc, fd, current->pathname, 1) < 0)\n+\t\t\t\treturn -1;\n+\t\t} else {\n+\t\t\tchar *path;\n+\t\t\tpath = xmalloc(sizeof(char) * (current->de_pathlen + 2));\n+\t\t\tmemcpy(path, current->pathname, current->de_pathlen);\n+\t\t\tmemcpy(path + current->de_pathlen, \"/\\0\", 2);\n+\t\t\tif (ce_write(&crc, fd, path, current->de_pathlen + 2) < 0)\n+\t\t\t\treturn -1;\n+\t\t}\n+\t\tcurrent->de_foffset = foffset;\n+\t\tcurrent->de_cr = conflict_offset;\n+\t\tondisk_from_directory_entry(current, &ondisk);\n+\t\tif (ce_write(&crc, fd, &ondisk, ondisk_size) < 0)\n+\t\t\treturn -1;\n+\t\tcrc = htonl(crc);\n+\t\tif (ce_write(NULL, fd, &crc, 4) < 0)\n+\t\t\treturn -1;\n+\t\tconflict_offset += current->conflict_size;\n+\t\tfoffset += current->de_nfiles * 4;\n+\t\tcurrent = current->next;\n+\t}\n+\treturn 0;\n+}\n+\n+static int write_entries(struct index_state *istate,\n+\t\t\t struct directory_entry *de,\n+\t\t\t int entries,\n+\t\t\t int fd,\n+\t\t\t int offset_to_offset)\n+{\n+\tint offset, offset_write, ondisk_size;\n+\tstruct directory_entry *current;\n+\n+\toffset = 0;\n+\tondisk_size = sizeof(struct ondisk_cache_entry);\n+\tcurrent = de;\n+\twhile (current) {\n+\t\tint pathlen;\n+\t\tstruct cache_entry *ce = current->ce;\n+\n+\t\tif (current->de_pathlen == 0)\n+\t\t\tpathlen = 0;\n+\t\telse\n+\t\t\tpathlen = current->de_pathlen + 1;\n+\t\twhile (ce) {\n+\t\t\tif (ce->ce_flags & CE_REMOVE)\n+\t\t\t\tcontinue;\n+\t\t\tif (!ce_uptodate(ce) && is_racy_timestamp(istate, ce))\n+\t\t\t\tce_smudge_racily_clean_entry(ce);\n+\n+\t\t\toffset_write = htonl(offset);\n+\t\t\tif (ce_write(NULL, fd, &offset_write, 4) < 0)\n+\t\t\t\treturn -1;\n+\t\t\toffset += ce_namelen(ce) - pathlen + 1 + ondisk_size + 4;\n+\t\t\tce = ce->next;\n+\t\t}\n+\t\tcurrent = current->next;\n+\t}\n+\t/*\n+\t * Write one more offset, which points to the end of the entries,\n+\t * because we use it for calculating the file length, instead of\n+\t * using strlen.\n+\t */\n+\toffset_write = htonl(offset);\n+\tif (ce_write(NULL, fd, &offset_write, 4) < 0)\n+\t\treturn -1;\n+\n+\toffset = offset_to_offset;\n+\tcurrent = de;\n+\twhile (current) {\n+\t\tint pathlen;\n+\t\tstruct cache_entry *ce = current->ce;\n+\n+\t\tif (current->de_pathlen == 0)\n+\t\t\tpathlen = 0;\n+\t\telse\n+\t\t\tpathlen = current->de_pathlen + 1;\n+\t\twhile (ce) {\n+\t\t\tstruct ondisk_cache_entry ondisk;\n+\t\t\tuint32_t crc, calc_crc;\n+\n+\t\t\tif (ce->ce_flags & CE_REMOVE)\n+\t\t\t\tcontinue;\n+\t\t\tcalc_crc = htonl(offset);\n+\t\t\tcrc = crc32(0, (Bytef*)&calc_crc, 4);\n+\t\t\tif (ce_write(&crc, fd, ce->name + pathlen,\n+\t\t\t\t\tce_namelen(ce) - pathlen + 1) < 0)\n+\t\t\t\treturn -1;\n+\t\t\tondisk_from_cache_entry(ce, &ondisk);\n+\t\t\tif (ce_write(&crc, fd, &ondisk, ondisk_size) < 0)\n+\t\t\t\treturn -1;\n+\t\t\tcrc = htonl(crc);\n+\t\t\tif (ce_write(NULL, fd, &crc, 4) < 0)\n+\t\t\t\treturn -1;\n+\t\t\toffset += 4;\n+\t\t\tce = ce->next;\n+\t\t}\n+\t\tcurrent = current->next;\n+\t}\n+\treturn 0;\n+}\n+\n+static int write_conflict(struct conflict_entry *conflict, int fd)\n+{\n+\tstruct conflict_entry *current;\n+\tstruct conflict_part *current_part;\n+\tuint32_t crc;\n+\n+\tcurrent = conflict;\n+\twhile (current) {\n+\t\tunsigned int to_write;\n+\n+\t\tcrc = 0;\n+\t\tif (ce_write(&crc, fd,\n+\t\t (Bytef*)(current->name + current->pathlen),\n+\t\t current->namelen - current->pathlen) < 0)\n+\t\t\treturn -1;\n+\t\tif (ce_write(&crc, fd, (Bytef*)\"\\0\", 1) < 0)\n+\t\t\treturn -1;\n+\t\tto_write = htonl(current->nfileconflicts);\n+\t\tif (ce_write(&crc, fd, (Bytef*)&to_write, 4) < 0)\n+\t\t\treturn -1;\n+\t\tcurrent_part = current->entries;\n+\t\twhile (current_part) {\n+\t\t\tstruct ondisk_conflict_part ondisk;\n+\n+\t\t\tconflict_to_ondisk(current_part, &ondisk);\n+\t\t\tif (ce_write(&crc, fd, (Bytef*)&ondisk, sizeof(struct ondisk_conflict_part)) < 0)\n+\t\t\t\treturn 0;\n+\t\t\tcurrent_part = current_part->next;\n+\t\t}\n+\t\tto_write = htonl(crc);\n+\t\tif (ce_write(NULL, fd, (Bytef*)&to_write, 4) < 0)\n+\t\t\treturn -1;\n+\t\tcurrent = current->next;\n+\t}\n+\treturn 0;\n+}\n+\n+static int write_conflicts(struct index_state *istate,\n+\t\t\t struct directory_entry *de,\n+\t\t\t int fd)\n+{\n+\tstruct directory_entry *current;\n+\n+\tcurrent = de;\n+\twhile (current) {\n+\t\tif (current->de_ncr != 0) {\n+\t\t\tif (write_conflict(current->conflict, fd) < 0)\n+\t\t\t\treturn -1;\n+\t\t}\n+\t\tcurrent = current->next;\n+\t}\n+\treturn 0;\n+}\n+\n+static int write_index_v5(struct index_state *istate, int newfd)\n+{\n+\tstruct cache_version_header hdr;\n+\tstruct cache_header hdr_v5;\n+\tstruct cache_entry **cache = istate->cache;\n+\tstruct directory_entry *de;\n+\tstruct ondisk_directory_entry *ondisk;\n+\tint entries = istate->cache_nr;\n+\tint i, removed, non_conflicted, total_dir_len, ondisk_directory_size;\n+\tint total_file_len, conflict_offset, offset_to_offset;\n+\tunsigned int ndir;\n+\tuint32_t crc;\n+\n+\tfor (i = removed = 0; i < entries; i++) {\n+\t\tif (cache[i]->ce_flags & CE_REMOVE)\n+\t\t\tremoved++;\n+\t}\n+\thdr.hdr_signature = htonl(CACHE_SIGNATURE);\n+\thdr.hdr_version = htonl(istate->version);\n+\thdr_v5.hdr_nfile = htonl(entries - removed);\n+\thdr_v5.hdr_nextension = htonl(0); /* Currently no extensions are supported */\n+\n+\tnon_conflicted = 0;\n+\ttotal_dir_len = 0;\n+\ttotal_file_len = 0;\n+\tde = compile_directory_data(istate, entries, &ndir, &non_conflicted,\n+\t\t\t&total_dir_len, &total_file_len);\n+\thdr_v5.hdr_ndir = htonl(ndir);\n+\n+\t/*\n+\t * This is needed because the compiler aligns structs to sizes multipe\n+\t * of 4\n+\t */\n+\tondisk_directory_size = sizeof(ondisk->flags)\n+\t\t+ sizeof(ondisk->foffset)\n+\t\t+ sizeof(ondisk->cr)\n+\t\t+ sizeof(ondisk->ncr)\n+\t\t+ sizeof(ondisk->nsubtrees)\n+\t\t+ sizeof(ondisk->nfiles)\n+\t\t+ sizeof(ondisk->nentries)\n+\t\t+ sizeof(ondisk->sha1);\n+\thdr_v5.hdr_fblockoffset = htonl(sizeof(hdr) + sizeof(hdr_v5) + 4\n+\t\t+ (ndir + 1) * 4\n+\t\t+ total_dir_len\n+\t\t+ ndir * (ondisk_directory_size + 4)\n+\t\t+ (non_conflicted + 1) * 4);\n+\n+\tcrc = 0;\n+\tif (ce_write(&crc, newfd, &hdr, sizeof(hdr)) < 0)\n+\t\treturn -1;\n+\tif (ce_write(&crc, newfd, &hdr_v5, sizeof(hdr_v5)) < 0)\n+\t\treturn -1;\n+\tcrc = htonl(crc);\n+\tif (ce_write(NULL, newfd, &crc, 4) < 0)\n+\t\treturn -1;\n+\n+\tconflict_offset = sizeof(hdr) + sizeof(hdr_v5) + 4\n+\t\t+ (ndir + 1) * 4\n+\t\t+ total_dir_len\n+\t\t+ ndir * (ondisk_directory_size + 4)\n+\t\t+ (non_conflicted + 1) * 4\n+\t\t+ total_file_len\n+\t\t+ non_conflicted * (sizeof(struct ondisk_cache_entry) + 4);\n+\tif (write_directories(de, newfd, conflict_offset) < 0)\n+\t\treturn -1;\n+\toffset_to_offset = sizeof(hdr) + sizeof(hdr_v5) + 4\n+\t\t+ (ndir + 1) * 4\n+\t\t+ total_dir_len\n+\t\t+ ndir * (ondisk_directory_size + 4);\n+\tif (write_entries(istate, de, entries, newfd, offset_to_offset) < 0)\n+\t\treturn -1;\n+\tif (write_conflicts(istate, de, newfd) < 0)\n+\t\treturn -1;\n+\treturn ce_flush(newfd);\n+}\n+\n struct index_ops v5_ops = {\n \tmatch_stat_basic,\n \tverify_hdr,\n \tread_index_v5,\n-\tNULL\n+\twrite_index_v5\n };\ndiff --git a/read-cache.c b/read-cache.c\nindex 5fb4d2e..199ba75 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -51,7 +51,7 @@ void rename_index_entry_at(struct index_state *istate, int nr, const char *new_n\n \tadd_index_entry(istate, new, ADD_CACHE_OK_TO_ADD|ADD_CACHE_OK_TO_REPLACE);\n }\n \n-static uint32_t calculate_stat_crc(struct cache_entry *ce)\n+uint32_t calculate_stat_crc(struct cache_entry *ce)\n {\n \tunsigned int ctimens = 0;\n \tuint32_t stat, stat_crc;\n@@ -172,13 +172,15 @@ int ce_modified_check_fs(struct cache_entry *ce, struct stat *st)\n \treturn 0;\n }\n \n-static void check_set_istate_ops(struct index_state *istate)\n+void set_istate_ops(struct index_state *istate)\n {\n \tif (!istate->version)\n \t\tistate->version = INDEX_FORMAT_DEFAULT;\n-\tif (!istate->ops)\n-\t\tif (istate->version >= 2 && istate->version <=4)\n-\t\t\tistate->ops = &v2_ops;\n+\n+\tif (istate->version >= 2 && istate->version <=4)\n+\t\tistate->ops = &v2_ops;\n+\tif (istate->version == 5)\n+\t\tistate->ops = &v5_ops;\n }\n \n int ce_match_stat_basic(struct index_state *istate,\n@@ -215,7 +217,7 @@ int ce_match_stat_basic(struct index_state *istate,\n \t\tdie(\"internal error: ce_mode is %o\", ce->ce_mode);\n \t}\n \n-\tcheck_set_istate_ops(istate);\n+\tset_istate_ops(istate);\n \tchanged = istate->ops->match_stat_basic(ce, st, changed);\n \treturn changed;\n }\n@@ -1185,6 +1187,8 @@ static int verify_hdr_version(struct index_state *istate,\n \thdr_version = ntohl(hdr->hdr_version);\n \tif (hdr_version >= 2 && hdr_version <= 4)\n \t\tistate->ops = &v2_ops;\n+\telse if (hdr_version == 5)\n+\t\tistate->ops = &v5_ops;\n \telse\n \t\treturn error(\"bad index version %d\", hdr_version);\n \treturn 0;\n@@ -1306,10 +1310,9 @@ void update_index_if_able(struct index_state *istate, struct lock_file *lockfile\n \telse\n \t\trollback_lock_file(lockfile);\n }\n-\n int write_index(struct index_state *istate, int newfd)\n {\n-\tcheck_set_istate_ops(istate);\n+\tset_istate_ops(istate);\n \n \treturn istate->ops->write_index(istate, newfd);\n }\ndiff --git a/read-cache.h b/read-cache.h\nindex 78de6a2..aa920ef 100644\n--- a/read-cache.h\n+++ b/read-cache.h\n@@ -29,6 +29,7 @@ struct index_ops {\n };\n \n extern struct index_ops v2_ops;\n+extern struct index_ops v5_ops;\n \n #ifndef NEEDS_ALIGNED_ACCESS\n #define ntoh_s(var) ntohs(var)\n@@ -55,3 +56,5 @@ extern int ce_match_stat_basic(struct index_state *istate,\n \t\tstruct cache_entry *ce, struct stat *st);\n extern int is_racy_timestamp(const struct index_state *istate, struct cache_entry *ce);\n extern void set_index_entry(struct index_state *istate, int nr, struct cache_entry *ce);\n+extern uint32_t calculate_stat_crc(struct cache_entry *ce);\n+extern void set_istate_ops(struct index_state *istate);\n-- \n1.7.10.GIT\n"},{"id":"196653","messageId":"1344424681-31469-11-git-send-email-t.gummerer@gmail.com","threadId":"31208","inReplyTo":"1344424681-31469-1-git-send-email-t.gummerer@gmail.com","subject":"[PATCH/RFC v3 10/13] Write index-v5 cache-tree data","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2012-08-08T11:17:58Z","receivedAt":"2012-08-08T11:17:58Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"Write the cache-tree data for the index version 5 file format. The\nin-memory cache-tree data is converted to the ondisk format, by adding\nit to the directory entries, that were compiled from the cache-entries\nin the step before.\n\nSigned-off-by: Thomas Gummerer \n---\n cache-tree.c | 52 ++++++++++++++++++++++++++++++++++++++++++++++++++++\n cache-tree.h | 1 +\n read-cache.c | 1 +\n 3 files changed, 54 insertions(+)\n\ndiff --git a/cache-tree.c b/cache-tree.c\nindex 440cd04..e167b61 100644\n--- a/cache-tree.c\n+++ b/cache-tree.c\n@@ -612,6 +612,58 @@ struct cache_tree *cache_tree_convert_v5(struct directory_entry *de)\n \treturn convert_one(queue, 0);\n }\n \n+\n+static void convert_one_to_ondisk_v5(struct hash_table *table, struct cache_tree *it,\n+\t\t\t\tconst char *path, int pathlen, uint32_t crc)\n+{\n+\tint i;\n+\tstruct directory_entry *found, *search;\n+\n+\tcrc = crc32(crc, (Bytef*)path, pathlen);\n+\tfound = lookup_hash(crc, table);\n+\tsearch = found;\n+\twhile (search && strcmp(path, search->pathname + search->de_pathlen - strlen(path)) != 0)\n+\t\tsearch = search->next_hash;\n+\tif (!search)\n+\t\treturn;\n+\t/*\n+\t * The number of subtrees is already calculated by\n+\t * compile_directory_data, therefore we only need to\n+\t * add the entry_count\n+\t */\n+\tsearch->de_nentries = it->entry_count;\n+\tif (0 <= it->entry_count)\n+\t\thashcpy(search->sha1, it->sha1);\n+\tif (strcmp(path, \"\") != 0)\n+\t\tcrc = crc32(crc, (Bytef*)\"/\", 1);\n+\n+#if DEBUG\n+\tif (0 <= it->entry_count)\n+\t\tfprintf(stderr, \"cache-tree <%.*s> (%d ent, %d subtree) %s\\n\",\n+\t\t\tpathlen, path, it->entry_count, it->subtree_nr,\n+\t\t\tsha1_to_hex(it->sha1));\n+\telse\n+\t\tfprintf(stderr, \"cache-tree <%.*s> (%d subtree) invalid\\n\",\n+\t\t\tpathlen, path, it->subtree_nr);\n+#endif\n+\n+\tfor (i = 0; i < it->subtree_nr; i++) {\n+\t\tstruct cache_tree_sub *down = it->down[i];\n+\t\tif (i) {\n+\t\t\tstruct cache_tree_sub *prev = it->down[i-1];\n+\t\t\tif (subtree_name_cmp(down->name, down->namelen,\n+\t\t\t\t\t prev->name, prev->namelen) <= 0)\n+\t\t\t\tdie(\"fatal - unsorted cache subtree\");\n+\t\t}\n+\t\tconvert_one_to_ondisk_v5(table, down->cache_tree, down->name, down->namelen, crc);\n+\t}\n+}\n+\n+void cache_tree_to_ondisk_v5(struct hash_table *table, struct cache_tree *root)\n+{\n+\tconvert_one_to_ondisk_v5(table, root, \"\", 0, 0);\n+}\n+\n static struct cache_tree *cache_tree_find(struct cache_tree *it, const char *path)\n {\n \tif (!it)\ndiff --git a/cache-tree.h b/cache-tree.h\nindex 7f29d26..e08bc31 100644\n--- a/cache-tree.h\n+++ b/cache-tree.h\n@@ -37,6 +37,7 @@ struct cache_tree *cache_tree_read(const char *buffer, unsigned long size);\n * Don't use it if the directory entries are still needed after.\n */\n struct cache_tree *cache_tree_convert_v5(struct directory_entry *de);\n+void cache_tree_to_ondisk_v5(struct hash_table *table, struct cache_tree *root);\n \n int cache_tree_fully_valid(struct cache_tree *);\n int cache_tree_update(struct cache_tree *, struct cache_entry **, int, int);\ndiff --git a/read-cache.c b/read-cache.c\nindex 199ba75..962d6a2 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -1310,6 +1310,7 @@ void update_index_if_able(struct index_state *istate, struct lock_file *lockfile\n \telse\n \t\trollback_lock_file(lockfile);\n }\n+\n int write_index(struct index_state *istate, int newfd)\n {\n \tset_istate_ops(istate);\n-- \n1.7.10.GIT\n"},{"id":"196661","messageId":"1344424681-31469-12-git-send-email-t.gummerer@gmail.com","threadId":"31208","inReplyTo":"1344424681-31469-1-git-send-email-t.gummerer@gmail.com","subject":"[PATCH/RFC v3 11/13] Write resolve-undo data for index-v5","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2012-08-08T11:17:59Z","receivedAt":"2012-08-08T11:17:59Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"Write the resolve undo data to the ondisk format, by joining the data\nin the resolve-undo string-list with the already existing conflicts\nthat were compiled before, when searching the directories and add\nthem to the corresponding directory entries.\n\nHelped-by: Thomas Rast \nSigned-off-by: Thomas Gummerer \n---\n read-cache-v5.c | 3 ++\n resolve-undo.c | 93 +++++++++++++++++++++++++++++++++++++++++++++++++++++++\n resolve-undo.h | 1 +\n 3 files changed, 97 insertions(+)\n\ndiff --git a/read-cache-v5.c b/read-cache-v5.c\nindex 45f7acd..3d03111 100644\n--- a/read-cache-v5.c\n+++ b/read-cache-v5.c\n@@ -861,6 +861,9 @@ static struct directory_entry *compile_directory_data(struct index_state *istate\n \t\t\tprevious_entry->next = no_subtrees;\n \t\t}\n \t}\n+\tif (istate->cache_tree)\n+\t\tcache_tree_to_ondisk_v5(&table, istate->cache_tree);\n+\tresolve_undo_to_ondisk_v5(&table, istate->resolve_undo, ndir, total_dir_len, de);\n \treturn de;\n }\n \ndiff --git a/resolve-undo.c b/resolve-undo.c\nindex f96c6ba..4568dcc 100644\n--- a/resolve-undo.c\n+++ b/resolve-undo.c\n@@ -206,3 +206,96 @@ void resolve_undo_convert_v5(struct index_state *istate,\n \t\tce = ce->next;\n \t}\n }\n+\n+void resolve_undo_to_ondisk_v5(struct hash_table *table,\n+\t\t\t\tstruct string_list *resolve_undo,\n+\t\t\t\tunsigned int *ndir, int *total_dir_len,\n+\t\t\t\tstruct directory_entry *de)\n+{\n+\tstruct string_list_item *item;\n+\tstruct directory_entry *search;\n+\n+\tif (!resolve_undo)\n+\t\treturn;\n+\tfor_each_string_list_item(item, resolve_undo) {\n+\t\tstruct conflict_entry *conflict_entry;\n+\t\tstruct resolve_undo_info *ui = item->util;\n+\t\tchar *super;\n+\t\tint i, dir_len, len;\n+\t\tuint32_t crc;\n+\t\tstruct directory_entry *found, *current, *new_tree;\n+\n+\t\tif (!ui)\n+\t\t\tcontinue;\n+\n+\t\tsuper = super_directory(item->string);\n+\t\tif (!super)\n+\t\t\tdir_len = 0;\n+\t\telse\n+\t\t\tdir_len = strlen(super);\n+\t\tcrc = crc32(0, (Bytef*)super, dir_len);\n+\t\tfound = lookup_hash(crc, table);\n+\t\tcurrent = NULL;\n+\t\tnew_tree = NULL;\n+\t\t\n+\t\twhile (!found) {\n+\t\t\tstruct directory_entry *new;\n+\n+\t\t\tnew = init_directory_entry(super, dir_len);\n+\t\t\tif (!current)\n+\t\t\t\tcurrent = new;\n+\t\t\tinsert_directory_entry(new, table, total_dir_len, ndir, crc);\n+\t\t\tif (new_tree != NULL)\n+\t\t\t\tnew->de_nsubtrees = 1;\n+\t\t\tnew->next = new_tree;\n+\t\t\tnew_tree = new;\n+\t\t\tsuper = super_directory(super);\n+\t\t\tif (!super)\n+\t\t\t\tdir_len = 0;\n+\t\t\telse\n+\t\t\t\tdir_len = strlen(super);\n+\t\t\tcrc = crc32(0, (Bytef*)super, dir_len);\n+\t\t\tfound = lookup_hash(crc, table);\n+\t\t}\n+\t\tsearch = found;\n+\t\twhile (search->next_hash && strcmp(super, search->pathname) != 0)\n+\t\t\tsearch = search->next_hash;\n+\t\tif (search && !current)\n+\t\t\tcurrent = search;\n+\t\tif (!search && !current)\n+\t\t\tcurrent = new_tree;\n+\t\tif (!super && new_tree) {\n+\t\t\tnew_tree->next = de->next;\n+\t\t\tde->next = new_tree;\n+\t\t\tde->de_nsubtrees++;\n+\t\t} else if (new_tree) {\n+\t\t\tstruct directory_entry *temp;\n+\n+\t\t\tsearch = de->next;\n+\t\t\twhile (strcmp(super, search->pathname))\n+\t\t\t\tsearch = search->next;\n+\t\t\ttemp = new_tree;\n+\t\t\twhile (temp->next)\n+\t\t\t\ttemp = temp->next;\n+\t\t\tsearch->de_nsubtrees++;\n+\t\t\ttemp->next = search->next;\n+\t\t\tsearch->next = new_tree;\n+\t\t}\n+\n+\t\tlen = strlen(item->string);\n+\t\tconflict_entry = create_new_conflict(item->string, len, current->de_pathlen);\n+\t\tadd_conflict_to_directory_entry(current, conflict_entry);\n+\t\tfor (i = 0; i < 3; i++) {\n+\t\t\tif (ui->mode[i]) {\n+\t\t\t\tstruct conflict_part *cp;\n+\n+\t\t\t\tcp = xmalloc(sizeof(struct conflict_part));\n+\t\t\t\tcp->flags = (i + 1) << CONFLICT_STAGESHIFT;\n+\t\t\t\tcp->entry_mode = ui->mode[i];\n+\t\t\t\tcp->next = NULL;\n+\t\t\t\thashcpy(cp->sha1, ui->sha1[i]);\n+\t\t\t\tadd_part_to_conflict_entry(current, conflict_entry, cp);\n+\t\t\t}\n+\t\t}\n+\t}\n+}\ndiff --git a/resolve-undo.h b/resolve-undo.h\nindex ab660a6..ff80d84 100644\n--- a/resolve-undo.h\n+++ b/resolve-undo.h\n@@ -14,5 +14,6 @@ extern int unmerge_index_entry_at(struct index_state *, int);\n extern void unmerge_index(struct index_state *, const char **);\n \n extern void resolve_undo_convert_v5(struct index_state *, struct conflict_entry *);\n+extern void resolve_undo_to_ondisk_v5(struct hash_table *, struct string_list *, unsigned int *, int *, struct directory_entry *);\n \n #endif\n-- \n1.7.10.GIT\n"},{"id":"196660","messageId":"1344424681-31469-13-git-send-email-t.gummerer@gmail.com","threadId":"31208","inReplyTo":"1344424681-31469-1-git-send-email-t.gummerer@gmail.com","subject":"[PATCH/RFC v3 12/13] update-index.c: always rewrite the index when index-version is given","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2012-08-08T11:18:00Z","receivedAt":"2012-08-08T11:18:00Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"Make git update-index always rewrite the index, if a index-version\nis given. This is used for performance testing, to have a reader\nand writer for the whole index.\n\nSigned-off-by: Thomas Gummerer \n---\n builtin/update-index.c | 4 +++-\n 1 file changed, 3 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/update-index.c b/builtin/update-index.c\nindex 4ce341c..c31d176 100644\n--- a/builtin/update-index.c\n+++ b/builtin/update-index.c\n@@ -6,6 +6,7 @@\n #include \"cache.h\"\n #include \"quote.h\"\n #include \"cache-tree.h\"\n+#include \"read-cache.h\"\n #include \"tree-walk.h\"\n #include \"builtin.h\"\n #include \"refs.h\"\n@@ -861,6 +862,7 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)\n \t\tif (the_index.version != preferred_index_format)\n \t\t\tactive_cache_changed = 1;\n \t\tthe_index.version = preferred_index_format;\n+\t\tset_istate_ops(&the_index);\n \t}\n \n \tif (read_from_stdin) {\n@@ -886,7 +888,7 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)\n \t\tstrbuf_release(&buf);\n \t}\n \n-\tif (active_cache_changed) {\n+\tif (active_cache_changed || preferred_index_format) {\n \t\tif (newfd < 0) {\n \t\t\tif (refresh_args.flags & REFRESH_QUIET)\n \t\t\t\texit(128);\n-- \n1.7.10.GIT\n"},{"id":"196655","messageId":"1344424681-31469-14-git-send-email-t.gummerer@gmail.com","threadId":"31208","inReplyTo":"1344424681-31469-1-git-send-email-t.gummerer@gmail.com","subject":"[PATCH/RFC v3 13/13] p0002-index.sh: add perf test for the index formats","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2012-08-08T11:18:01Z","receivedAt":"2012-08-08T11:18:01Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"From: Thomas Rast \n\nAdd a performance test for index version [23]/4/5 by using\ngit update-index --update-index=[345], thus testing both the reader\nand the writer speed of all index formats.\n\nSigned-off-by: Thomas Rast \nSigned-off-by: Thomas Gummerer \n---\n t/perf/p0002-index.sh | 33 +++++++++++++++++++++++++++++++++\n 1 file changed, 33 insertions(+)\n create mode 100755 t/perf/p0002-index.sh\n\ndiff --git a/t/perf/p0002-index.sh b/t/perf/p0002-index.sh\nnew file mode 100755\nindex 0000000..140c7a0\n--- /dev/null\n+++ b/t/perf/p0002-index.sh\n@@ -0,0 +1,33 @@\n+#!/bin/sh\n+\n+test_description=\"Tests index versions [23]/4/5\"\n+\n+. ./perf-lib.sh\n+\n+test_perf_large_repo\n+\n+test_expect_success 'convert to v3' '\n+\tgit update-index --index-version=3\n+'\n+\n+test_perf 'v[23]: update-index' '\n+\tgit update-index --index-version=3 >/dev/null\n+'\n+\n+test_expect_success 'convert to v4' '\n+\tgit update-index --index-version=4\n+'\n+\n+test_perf 'v4: update-index' '\n+\tgit update-index --index-version=4 >/dev/null\n+'\n+\n+test_expect_success 'convert to v5' '\n+\tgit update-index --index-version=5\n+'\n+\n+test_perf 'v5: update-index' '\n+\tgit update-index --index-version=5 >/dev/null\n+'\n+\n+test_done\n-- \n1.7.10.GIT\n"},{"id":"196662","messageId":"CACsJy8CY--Z_BfscRCG8+Om1Z8JsFgj0BkjNyMJnwsbmyGRLXQ@mail.gmail.com","threadId":"31208","inReplyTo":"1344424681-31469-2-git-send-email-t.gummerer@gmail.com","subject":"Re: [PATCH/RFC v3 01/13] Move index v2 specific functions to their own file","fromName":"Nguyen Thai Ngoc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2012-08-08T12:04:48Z","receivedAt":"2012-08-08T12:04:48Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Wed, Aug 8, 2012 at 6:17 PM, Thomas Gummerer wrote:\n> Move index version 2 specific functions to their own file,\n> to prepare for the addition of a new index file format. With\n> the split into two files we have the non-index specific\n> functions in read-cache.c and the index-v2 specific functions\n> in read-cache-v2.c\n\nYou still mix code changes and code move in one patch, but we can skip\nit for now.\n\n> --- a/cache.h\n> +++ b/cache.h\n> @@ -267,6 +259,7 @@ struct index_state {\n> unsigned name_hash_initialized : 1,\n> initialized : 1;\n> struct hash_table name_hash;\n> + struct index_ops *ops;\n> };\n\nDo we really need to modify \"ops\" content? If not make it \"const\nstruct index_ops *ops;\" which makes..\n\n> @@ -471,8 +464,8 @@ extern int index_name_is_other(const struct index_state *, const char *, int);\n> #define CE_MATCH_RACY_IS_DIRTY 02\n> /* do stat comparison even if CE_SKIP_WORKTREE is true */\n> #define CE_MATCH_IGNORE_SKIP_WORKTREE 04\n> -extern int ie_match_stat(const struct index_state *, struct cache_entry *, struct stat *, unsigned int);\n> -extern int ie_modified(const struct index_state *, struct cache_entry *, struct stat *, unsigned int);\n> +extern int ie_match_stat(struct index_state *, struct cache_entry *, struct stat *, unsigned int);\n> +extern int ie_modified(struct index_state *, struct cache_entry *, struct stat *, unsigned int);\n\n..this hunk go away\n-- \nDuy\n"},{"id":"196663","messageId":"CACsJy8DZ-+0XD2HxkTFuXKXJFH66MCLLrfHtcMF-1GqBb2AmVg@mail.gmail.com","threadId":"31208","inReplyTo":"1344424681-31469-7-git-send-email-t.gummerer@gmail.com","subject":"Re: [PATCH/RFC v3 06/13] Read index-v5","fromName":"Nguyen Thai Ngoc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2012-08-08T12:05:33Z","receivedAt":"2012-08-08T12:05:33Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"uOn Wed, Aug 8, 2012 at 6:17 PM, Thomas Gummerer wrote:\n> +static struct cache_entry *read_entry(struct directory_entry *de,\n> + unsigned long *entry_offset,\n> + void **mmap,\n> + unsigned long mmap_size,\n> + unsigned int *foffsetblock,\n> + int fd)\n> +{\n> ....\n> + if (crc_wrong) {\n> + /* wait for 10 milliseconds */\n> + usleep(10*1000);\n> + munmap(*mmap, mmap_size);\n> + *mmap = xmmap(NULL, mmap_size, PROT_READ | PROT_WRITE, MAP_PRIVATE, fd, 0);\n> + }\n\nDo we really need to munmap and mmap again? I don't see mmap man page\nmention anything about \"refreshing\" the mmap'd memory with file\nchanges, not sure how it works. msync() seems for writing only.\n\nIf remapping is necessary, how about mremap? What I want to see is\nwhether we could avoid passing \"fd\" down to here.\n\n> +struct index_ops v5_ops = {\n> + match_stat_basic,\n> + verify_hdr,\n> + read_index_v5,\n> + NULL\n> +};\n\nIf you do it right, putting write_index_v2 here should work because\nin-core structure is not changed (except that write_index_v2 is static\nfunction, well..). Maybe putting write_index to this struct is a wrong\ndecision. We should be able to read_index_v5+write_index_v2 and\nread_index_v2+write_index_v5.\n-- \nDuy\n"},{"id":"196664","messageId":"50225901.5040405@viscovery.net","threadId":"31208","inReplyTo":"CACsJy8DZ-+0XD2HxkTFuXKXJFH66MCLLrfHtcMF-1GqBb2AmVg@mail.gmail.com","subject":"Re: [PATCH/RFC v3 06/13] Read index-v5","fromName":"Johannes Sixt","fromEmail":"j.sixt@viscovery.net","sentAt":"2012-08-08T12:18:09Z","receivedAt":"2012-08-08T12:18:09Z","isPatch":true,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"Am 8/8/2012 14:05, schrieb Nguyen Thai Ngoc Duy:\n> uOn Wed, Aug 8, 2012 at 6:17 PM, Thomas Gummerer wrote:\n>> +static struct cache_entry *read_entry(struct directory_entry *de,\n>> + unsigned long *entry_offset,\n>> + void **mmap,\n>> + unsigned long mmap_size,\n>> + unsigned int *foffsetblock,\n>> + int fd)\n>> +{\n>> ....\n>> + if (crc_wrong) {\n>> + /* wait for 10 milliseconds */\n>> + usleep(10*1000);\n>> + munmap(*mmap, mmap_size);\n>> + *mmap = xmmap(NULL, mmap_size, PROT_READ | PROT_WRITE, MAP_PRIVATE, fd, 0);\n>> + }\n> \n> Do we really need to munmap and mmap again?\n\nYes. mmap may be the pread()-based implementation from compat/mmap.c.\n\n-- Hannes\n"},{"id":"196676","messageId":"7vipcttiuf.fsf@alter.siamese.dyndns.org","threadId":"31208","inReplyTo":"CACsJy8DZ-+0XD2HxkTFuXKXJFH66MCLLrfHtcMF-1GqBb2AmVg@mail.gmail.com","subject":"Re: [PATCH/RFC v3 06/13] Read index-v5","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2012-08-08T17:05:28Z","receivedAt":"2012-08-08T17:05:28Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Nguyen Thai Ngoc Duy writes:\n\n>> +struct index_ops v5_ops = {\n>> + match_stat_basic,\n>> + verify_hdr,\n>> + read_index_v5,\n>> + NULL\n>> +};\n>\n> If you do it right, putting write_index_v2 here should work because\n> in-core structure is not changed (except that write_index_v2 is static\n> function, well..). Maybe putting write_index to this struct is a wrong\n> decision. We should be able to read_index_v5+write_index_v2 and\n> read_index_v2+write_index_v5.\n\nThe \"right way\" is to have a global API function\n\n write_index_in_format(int version, int fd);\n\nwhich calls \"the_index->index_ops.write_index[version](the_index, fd)\",\nno?\n"},{"id":"196691","messageId":"20120808192041.GA25671@tommy-fedora.surfnet.iacbox","threadId":"31208","inReplyTo":"CACsJy8CY--Z_BfscRCG8+Om1Z8JsFgj0BkjNyMJnwsbmyGRLXQ@mail.gmail.com","subject":"Re: [PATCH/RFC v3 01/13] Move index v2 specific functions to their own file","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2012-08-08T19:21:16Z","receivedAt":"2012-08-08T19:21:16Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"\n\nOn 08/08, Nguyen Thai Ngoc Duy wrote:\n> On Wed, Aug 8, 2012 at 6:17 PM, Thomas Gummerer wrote:\n> > Move index version 2 specific functions to their own file,\n> > to prepare for the addition of a new index file format. With\n> > the split into two files we have the non-index specific\n> > functions in read-cache.c and the index-v2 specific functions\n> > in read-cache-v2.c\n> \n> You still mix code changes and code move in one patch, but we can skip\n> it for now.\n> \n> > --- a/cache.h\n> > +++ b/cache.h\n> > @@ -267,6 +259,7 @@ struct index_state {\n> > unsigned name_hash_initialized : 1,\n> > initialized : 1;\n> > struct hash_table name_hash;\n> > + struct index_ops *ops;\n> > };\n> \n> Do we really need to modify \"ops\" content? If not make it \"const\n> struct index_ops *ops;\" which makes..\n> \n> > @@ -471,8 +464,8 @@ extern int index_name_is_other(const struct index_state *, const char *, int);\n> > #define CE_MATCH_RACY_IS_DIRTY 02\n> > /* do stat comparison even if CE_SKIP_WORKTREE is true */\n> > #define CE_MATCH_IGNORE_SKIP_WORKTREE 04\n> > -extern int ie_match_stat(const struct index_state *, struct cache_entry *, struct stat *, unsigned int);\n> > -extern int ie_modified(const struct index_state *, struct cache_entry *, struct stat *, unsigned int);\n> > +extern int ie_match_stat(struct index_state *, struct cache_entry *, struct stat *, unsigned int);\n> > +extern int ie_modified(struct index_state *, struct cache_entry *, struct stat *, unsigned int);\n> \n> ..this hunk go away\n\nUnfortunately I think we need to modify it, because in some cases\nwhen ie_modified/ie_match_stat is called, index->ops is not set yet\n(happens when the index is empty and thus not read first). Therefore\nwe have to set it in ce_match_stat_basic to have access to the\nformat specific match_stat_basic.\n\nWe could also call match_stat_basic directly from the v[25]_ops\nstruct, but I do not think that would be the cleaner solution, or\nis there any other way to do it, which I can't see?\n"},{"id":"196693","messageId":"20120808192915.GB25671@tommy-fedora.surfnet.iacbox","threadId":"31208","inReplyTo":"CACsJy8DZ-+0XD2HxkTFuXKXJFH66MCLLrfHtcMF-1GqBb2AmVg@mail.gmail.com","subject":"Re: [PATCH/RFC v3 06/13] Read index-v5","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2012-08-08T19:29:15Z","receivedAt":"2012-08-08T19:29:15Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"On 08/08, Nguyen Thai Ngoc Duy wrote:\n> uOn Wed, Aug 8, 2012 at 6:17 PM, Thomas Gummerer wrote:\n> > +static struct cache_entry *read_entry(struct directory_entry *de,\n> > + unsigned long *entry_offset,\n> > + void **mmap,\n> > + unsigned long mmap_size,\n> > + unsigned int *foffsetblock,\n> > + int fd)\n> > +{\n> > ....\n> > + if (crc_wrong) {\n> > + /* wait for 10 milliseconds */\n> > + usleep(10*1000);\n> > + munmap(*mmap, mmap_size);\n> > + *mmap = xmmap(NULL, mmap_size, PROT_READ | PROT_WRITE, MAP_PRIVATE, fd, 0);\n> > + }\n> \n> Do we really need to munmap and mmap again? I don't see mmap man page\n> mention anything about \"refreshing\" the mmap'd memory with file\n> changes, not sure how it works. msync() seems for writing only.\n> \n> If remapping is necessary, how about mremap? What I want to see is\n> whether we could avoid passing \"fd\" down to here.\n> \n> > +struct index_ops v5_ops = {\n> > + match_stat_basic,\n> > + verify_hdr,\n> > + read_index_v5,\n> > + NULL\n> > +};\n> \n> If you do it right, putting write_index_v2 here should work because\n> in-core structure is not changed (except that write_index_v2 is static\n> function, well..). Maybe putting write_index to this struct is a wrong\n> decision. We should be able to read_index_v5+write_index_v2 and\n> read_index_v2+write_index_v5.\n\nHrm I don't think it should be a problem putting write_index_v2 here,\nbut the way I currently handle it is by replacing the whole index_ops.\nI think it's a good thing that the index_ops version always matches\nthe index version, to avoid any problems there. The only place where\nthis can happen in the current code is in update-index.c when we change\nthe index version.\n"},{"id":"196757","messageId":"7vtxwbn2qe.fsf@alter.siamese.dyndns.org","threadId":"31208","inReplyTo":"1344424681-31469-2-git-send-email-t.gummerer@gmail.com","subject":"Re: [PATCH/RFC v3 01/13] Move index v2 specific functions to their own file","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2012-08-09T22:02:17Z","receivedAt":"2012-08-09T22:02:17Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Thomas Gummerer writes:\n\n> /* remember to discard_cache() before reading a different cache! */\n> int read_index_from(struct index_state *istate, const char *path)\n> {\n> ...\n> \tmmap = xmmap(NULL, mmap_size, PROT_READ | PROT_WRITE, MAP_PRIVATE, fd, 0);\n> -\tclose(fd);\n> \tif (mmap == MAP_FAILED)\n> \t\tdie_errno(\"unable to map index file\");\n> \n> \thdr = mmap;\n> -\tif (verify_hdr(hdr, mmap_size) < 0)\n> +\tif (verify_hdr_version(istate, hdr, mmap_size) < 0)\n> \t\tgoto unmap;\n> ...\n> +\tif (istate->ops->verify_hdr(mmap, mmap_size) < 0)\n> +\t\tgoto unmap;\n> \n> +\tistate->ops->read_index(istate, mmap, mmap_size, fd);\n> ...\n> +\tclose(fd);\n\nThis looks utterly wrong.\n\nYou already have mapped the whole thing, so there is nothing to be\nread from fd. You have everything in-core. Leaving fd open and\npass it around looks like it is asking for trouble and confusion.\n\nIf you found that an entry you read halfway has an inconsistent crc,\nand if you suspect that is because somebody else was writing to the\nsame index, it is a _sure_ sign that you are not alone, and all the\nentries you read so far to the core, even if they weren't touched by\nthat sombody else when you read them, may be stale, and worse yet,\nwhat you are going to read may be inconsistent with what you've read\nand have in-core (e.g. you may have read \"f\" before somebody else\nthat is racing with you have turned it into a directory, and your\nnext read may find \"f/d\" in the index without crc error).\n\nOne sane way to avoid reading such an inconsistent state may be to\nredo this whole function, starting from the part that calls mmap().\nIOW,\n\n\tdo {\n\t\tfd = open()\n\t\tmmap = xmmap(fd);\n\t\tclose(fd);\n verify_various_fields(mmap);\n status = istate->ops->read_index(istate, mmap, mmap_size));\n\t} while (status == READ_AGAIN);\n\nI do not think the \"pass fd around so that we can redo the mapping\ndeep inside the callchain\" is either a good idea or necessary.\n"},{"id":"196758","messageId":"7vobmjn0wv.fsf@alter.siamese.dyndns.org","threadId":"31208","inReplyTo":"1344424681-31469-5-git-send-email-t.gummerer@gmail.com","subject":"Re: [PATCH/RFC v3 04/13] Add documentation of the index-v5 file format","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2012-08-09T22:41:36Z","receivedAt":"2012-08-09T22:41:36Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Thomas Gummerer writes:\n\n> +GIT index format\n> +================\n> +\n> +== The git index file format\n> +\n> + The git index file (.git/index) documents the status of the files\n> + in the git staging area.\n> +\n> + The staging area is used for preparing commits, merging, etc.\n\nThe above two are not about \"index file format\". It is an\nexplanation of what the index is.\n\n> + All binary numbers are in network byte order. Version 5 is described\n> + here.\n\nI had to read between these two lines something like \n\n \"\"The index file consists of various sections; the sections\n appear in the following order in the file.\"\"\"\n\nto make sense of the document.\n\n> + - A 20-byte header consisting of\n> +\n> + sig (32-bits): Signature:\n> + The signature is { 'D', 'I', 'R', 'C' } (stands for \"dircache\")\n> +\n> + vnr (32-bits): Version number:\n> + The current supported versions are 2, 3, 4 and 5.\n> +\n> + ndir (32-bits): number of directories in the index.\n> +\n> + nfile (32-bits): number of file entries in the index.\n> +\n> + fblockoffset (32-bits): offset to the file block, relative to the\n> + beginning of the file.\n\nOk.\n\n> + - Offset to the extensions.\n>\n> + nextensions (32-bits): number of extensions.\n> +\n> + extoffset (32-bits): offset to the extension. (Possibly none, as\n> + many as indicated in the 4-byte number of extensions)\n\nOK.\n\n> + headercrc (32-bits): crc checksum for the header and extension\n> + offsets\n\nThis may have to have the same \" -
\" at the same\nlevel as \"A 20-byte header\" and \"Offset to the ext\"; as it stands,\nit looks as if it is part of \"Offset to the ext\" which consists of\n12 bytes.\n\n> + - diroffsets (ndir * directory offsets): A directory offset for each\n> + of the ndir directories in the index, sorted by pathname (of the\n> + directory it's pointing to) (see below). The diroffsets are relative\n> + to the beginning of the direntries block. [1]\n\n\"ndir * diroffsets\" confused me. I think you meant to say that this\n\"diroffsets\" section consists of ndir entries of something and that\neach of that something is a directory offset. It is unclear how \"a\ndirectory offset\" is represented, except that it is \"relative to the\nbeginning of direntry block\" (and it is unclear what and where the\ndirentry block is from the information given up to this point) and\nthe reader can guess it is in \"network byte order\" (assuming it is a\nbinary number). Perhaps\n\n\tdiroffsets (ndir entries of \"directory offset\"): A 4-byte\n\toffset relative to the beginning of the \"direntries block\"\n\t(see below) for each of the ...\n\nand drop the last sentence?\n\nOther tables may want to be adjusted in a similar fashion.\n\n> +== Directory offsets (diroffsets)\n> +\n> + diroffset (32-bits): offset to the directory relative to the beginning\n> + of the index file. There are ndir + 1 offsets in the diroffset table,\n> + the last is pointing to the end of the last direntry. With this last\n> + entry, we can replace the strlen when reading each filename, by\n> + calculating its length with the offsets.\n\nThe mention of \"strlen\" looks very out of place. The reader may be\nable to guess that you want to say that the nth \"string\" is between\ndiroffset[n] and diroffset[n+1], and these \"string\"s are densely\npacked so strlen(diroffset[n]) and diroffset[n+1]-diroffset[n] are\neither the same thing (or with a fixed difference, if each \"string\"\nis accompanied by some fixed-length data), but it is unclear what\nthese \"strings\" represent, especially because the name of the table\nimplies that you are talking about directories but strlen talks\nabout filename.\n\n> +== Design explanations\n> + ...\n> +[3] The data of the cache-tree extension and the resolve undo\n> + extension is now part of the index itself, but if other extensions\n> + come up in the future, there is no need to change the index, they\n> + can simply be added at the end.\n\nInteresting. When we added extensions, we said that there is no\nneed to change the index to add new features, they can simply be\nadded at the end. Perhaps the file offset table can be added as an\nextension to v2 to give us the same bisectability, allowing us a\nsingle entry in-place replacementability, without defining an\nentirely different format?\n"},{"id":"196760","messageId":"7vk3x7n0fl.fsf@alter.siamese.dyndns.org","threadId":"31208","inReplyTo":"1344424681-31469-8-git-send-email-t.gummerer@gmail.com","subject":"Re: [PATCH/RFC v3 07/13] Read resolve-undo data","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2012-08-09T22:51:58Z","receivedAt":"2012-08-09T22:51:58Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Thomas Gummerer writes:\n\n> Make git read the resolve-undo data from the index.\n>\n> Since the resolve-undo data is joined with the conflicts in\n> the ondisk format of the index file version 5, conflicts and\n> resolved data is read at the same time, and the resolve-undo\n> data is then converted to the in-memory format.\n\nThis, and the next one, are both about reading extension data from\nthe v2 formatted index, no?\n\nAgain, mild NAK.\n\nI think it is a lot more logical for the v5 code to read data stored\nin the resolve-undo and cache-tree extensions using the public API\njust like other users of these data do, and write out whatever in a\nway that is specific to the v5 index format.\n\nIf the v5 codepath needs some information that is not exposed to\nother users of istate->resolve_undo and istate->cache_tree, then the\nstory is different, but I do not think that is the case.\n\n>\n> Helped-by: Thomas Rast \n> Signed-off-by: Thomas Gummerer \n> ---\n> read-cache-v5.c | 1 +\n> resolve-undo.c | 36 ++++++++++++++++++++++++++++++++++++\n> resolve-undo.h | 2 ++\n> 3 files changed, 39 insertions(+)\n>\n> diff --git a/read-cache-v5.c b/read-cache-v5.c\n> index ec1201d..b47398d 100644\n> --- a/read-cache-v5.c\n> +++ b/read-cache-v5.c\n> @@ -494,6 +494,7 @@ static struct directory_entry *read_entries(struct index_state *istate,\n> \tint i;\n> \n> \tconflict_queue = read_conflicts(de, mmap, mmap_size, fd);\n> +\tresolve_undo_convert_v5(istate, conflict_queue);\n> \tfor (i = 0; i < de->de_nfiles; i++) {\n> \t\tce = read_entry(de,\n> \t\t\t\tentry_offset,\n> diff --git a/resolve-undo.c b/resolve-undo.c\n> index 72b4612..f96c6ba 100644\n> --- a/resolve-undo.c\n> +++ b/resolve-undo.c\n> @@ -170,3 +170,39 @@ void unmerge_index(struct index_state *istate, const char **pathspec)\n> \t\ti = unmerge_index_entry_at(istate, i);\n> \t}\n> }\n> +\n> +void resolve_undo_convert_v5(struct index_state *istate,\n> +\t\t\t\t\tstruct conflict_entry *ce)\n> +{\n> +\tint i;\n> +\n> +\twhile (ce) {\n> +\t\tstruct string_list_item *lost;\n> +\t\tstruct resolve_undo_info *ui;\n> +\t\tstruct conflict_part *cp;\n> +\n> +\t\tif (ce->entries && (ce->entries->flags & CONFLICT_CONFLICTED) != 0) {\n> +\t\t\tce = ce->next;\n> +\t\t\tcontinue;\n> +\t\t}\n> +\t\tif (!istate->resolve_undo) {\n> +\t\t\tistate->resolve_undo = xcalloc(1, sizeof(struct string_list));\n> +\t\t\tistate->resolve_undo->strdup_strings = 1;\n> +\t\t}\n> +\n> +\t\tlost = string_list_insert(istate->resolve_undo, ce->name);\n> +\t\tif (!lost->util)\n> +\t\t\tlost->util = xcalloc(1, sizeof(*ui));\n> +\t\tui = lost->util;\n> +\n> +\t\tcp = ce->entries;\n> +\t\tfor (i = 0; i < 3; i++)\n> +\t\t\tui->mode[i] = 0;\n> +\t\twhile (cp) {\n> +\t\t\tui->mode[conflict_stage(cp) - 1] = cp->entry_mode;\n> +\t\t\thashcpy(ui->sha1[conflict_stage(cp) - 1], cp->sha1);\n> +\t\t\tcp = cp->next;\n> +\t\t}\n> +\t\tce = ce->next;\n> +\t}\n> +}\n> diff --git a/resolve-undo.h b/resolve-undo.h\n> index 8458769..ab660a6 100644\n> --- a/resolve-undo.h\n> +++ b/resolve-undo.h\n> @@ -13,4 +13,6 @@ extern void resolve_undo_clear_index(struct index_state *);\n> extern int unmerge_index_entry_at(struct index_state *, int);\n> extern void unmerge_index(struct index_state *, const char **);\n> \n> +extern void resolve_undo_convert_v5(struct index_state *, struct conflict_entry *);\n> +\n> #endif\n"},{"id":"196761","messageId":"20120809225407.GB5127@tommy-fedora.scientificnet.net","threadId":"31208","inReplyTo":"7vtxwbn2qe.fsf@alter.siamese.dyndns.org","subject":"Re: [PATCH/RFC v3 01/13] Move index v2 specific functions to their own file","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2012-08-09T22:54:07Z","receivedAt":"2012-08-09T22:54:07Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"On 08/09, Junio C Hamano wrote:\n> Thomas Gummerer writes:\n> \n> > /* remember to discard_cache() before reading a different cache! */\n> > int read_index_from(struct index_state *istate, const char *path)\n> > {\n> > ...\n> > \tmmap = xmmap(NULL, mmap_size, PROT_READ | PROT_WRITE, MAP_PRIVATE, fd, 0);\n> > -\tclose(fd);\n> > \tif (mmap == MAP_FAILED)\n> > \t\tdie_errno(\"unable to map index file\");\n> > \n> > \thdr = mmap;\n> > -\tif (verify_hdr(hdr, mmap_size) < 0)\n> > +\tif (verify_hdr_version(istate, hdr, mmap_size) < 0)\n> > \t\tgoto unmap;\n> > ...\n> > +\tif (istate->ops->verify_hdr(mmap, mmap_size) < 0)\n> > +\t\tgoto unmap;\n> > \n> > +\tistate->ops->read_index(istate, mmap, mmap_size, fd);\n> > ...\n> > +\tclose(fd);\n> \n> This looks utterly wrong.\n> \n> You already have mapped the whole thing, so there is nothing to be\n> read from fd. You have everything in-core. Leaving fd open and\n> pass it around looks like it is asking for trouble and confusion.\n> \n> If you found that an entry you read halfway has an inconsistent crc,\n> and if you suspect that is because somebody else was writing to the\n> same index, it is a _sure_ sign that you are not alone, and all the\n> entries you read so far to the core, even if they weren't touched by\n> that sombody else when you read them, may be stale, and worse yet,\n> what you are going to read may be inconsistent with what you've read\n> and have in-core (e.g. you may have read \"f\" before somebody else\n> that is racing with you have turned it into a directory, and your\n> next read may find \"f/d\" in the index without crc error).\n> \n> One sane way to avoid reading such an inconsistent state may be to\n> redo this whole function, starting from the part that calls mmap().\n> IOW,\n> \n> \tdo {\n> \t\tfd = open()\n> \t\tmmap = xmmap(fd);\n> \t\tclose(fd);\n> verify_various_fields(mmap);\n> status = istate->ops->read_index(istate, mmap, mmap_size));\n> \t} while (status == READ_AGAIN);\n> \n> I do not think the \"pass fd around so that we can redo the mapping\n> deep inside the callchain\" is either a good idea or necessary.\n\nThanks, that looks better. I'll change it for the re-roll.\n"},{"id":"196762","messageId":"20120809231039.GC5127@tommy-fedora.scientificnet.net","threadId":"31208","inReplyTo":"7vobmjn0wv.fsf@alter.siamese.dyndns.org","subject":"Re: [PATCH/RFC v3 04/13] Add documentation of the index-v5 file format","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2012-08-09T23:10:39Z","receivedAt":"2012-08-09T23:10:39Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"On 08/09, Junio C Hamano wrote:\n> Thomas Gummerer writes:\n> \n> > +GIT index format\n> > +================\n> > +\n> > +== The git index file format\n> > +\n> > + The git index file (.git/index) documents the status of the files\n> > + in the git staging area.\n> > +\n> > + The staging area is used for preparing commits, merging, etc.\n> \n> The above two are not about \"index file format\". It is an\n> explanation of what the index is.\n> \n> > + All binary numbers are in network byte order. Version 5 is described\n> > + here.\n> \n> I had to read between these two lines something like \n> \n> \"\"The index file consists of various sections; the sections\n> appear in the following order in the file.\"\"\"\n> \n> to make sense of the document.\n\nThanks, I'll add that.\n\n> > + - A 20-byte header consisting of\n> > +\n> > + sig (32-bits): Signature:\n> > + The signature is { 'D', 'I', 'R', 'C' } (stands for \"dircache\")\n> > +\n> > + vnr (32-bits): Version number:\n> > + The current supported versions are 2, 3, 4 and 5.\n> > +\n> > + ndir (32-bits): number of directories in the index.\n> > +\n> > + nfile (32-bits): number of file entries in the index.\n> > +\n> > + fblockoffset (32-bits): offset to the file block, relative to the\n> > + beginning of the file.\n> \n> Ok.\n> \n> > + - Offset to the extensions.\n> >\n> > + nextensions (32-bits): number of extensions.\n> > +\n> > + extoffset (32-bits): offset to the extension. (Possibly none, as\n> > + many as indicated in the 4-byte number of extensions)\n> \n> OK.\n> \n> > + headercrc (32-bits): crc checksum for the header and extension\n> > + offsets\n> \n> This may have to have the same \" -
\" at the same\n> level as \"A 20-byte header\" and \"Offset to the ext\"; as it stands,\n> it looks as if it is part of \"Offset to the ext\" which consists of\n> 12 bytes.\n\nThanks, I'll try to write it down more clearly.\n\n> > + - diroffsets (ndir * directory offsets): A directory offset for each\n> > + of the ndir directories in the index, sorted by pathname (of the\n> > + directory it's pointing to) (see below). The diroffsets are relative\n> > + to the beginning of the direntries block. [1]\n> \n> \"ndir * diroffsets\" confused me. I think you meant to say that this\n> \"diroffsets\" section consists of ndir entries of something and that\n> each of that something is a directory offset. It is unclear how \"a\n> directory offset\" is represented, except that it is \"relative to the\n> beginning of direntry block\" (and it is unclear what and where the\n> direntry block is from the information given up to this point) and\n> the reader can guess it is in \"network byte order\" (assuming it is a\n> binary number). Perhaps\n> \n> \tdiroffsets (ndir entries of \"directory offset\"): A 4-byte\n> \toffset relative to the beginning of the \"direntries block\"\n> \t(see below) for each of the ...\n> \n> and drop the last sentence?\n> \n> Other tables may want to be adjusted in a similar fashion.\n\nYes, that's what I menat to say. Thanks.\n\n> > +== Directory offsets (diroffsets)\n> > +\n> > + diroffset (32-bits): offset to the directory relative to the beginning\n> > + of the index file. There are ndir + 1 offsets in the diroffset table,\n> > + the last is pointing to the end of the last direntry. With this last\n> > + entry, we can replace the strlen when reading each filename, by\n> > + calculating its length with the offsets.\n> \n> The mention of \"strlen\" looks very out of place. The reader may be\n> able to guess that you want to say that the nth \"string\" is between\n> diroffset[n] and diroffset[n+1], and these \"string\"s are densely\n> packed so strlen(diroffset[n]) and diroffset[n+1]-diroffset[n] are\n> either the same thing (or with a fixed difference, if each \"string\"\n> is accompanied by some fixed-length data), but it is unclear what\n> these \"strings\" represent, especially because the name of the table\n> implies that you are talking about directories but strlen talks\n> about filename.\n\nHrm maybe better like this:\n\n+ diroffset (32-bits): offset to the directory relative to the beginning\n+ of the index file. There are ndir + 1 offsets in the diroffset table,\n+ the last is pointing to the end of the last direntry. With this last\n+ entry, we are able to replace the strlen of when reading the directory\n+ name, by calculating it from diroffset[n+1]-diroffset[n]-61. 61 is the\n+ size of the directory data, which follows each each directory + the\n+ crc sum + the NUL byte.\n\n> > +== Design explanations\n> > + ...\n> > +[3] The data of the cache-tree extension and the resolve undo\n> > + extension is now part of the index itself, but if other extensions\n> > + come up in the future, there is no need to change the index, they\n> > + can simply be added at the end.\n> \n> Interesting. When we added extensions, we said that there is no\n> need to change the index to add new features, they can simply be\n> added at the end. Perhaps the file offset table can be added as an\n> extension to v2 to give us the same bisectability, allowing us a\n> single entry in-place replacementability, without defining an\n> entirely different format?\n\nOnly part of this is true. v2 would allow us to add the file offset\ntable as extension, but the problem is the design of the sha-1 over\nthe whole file at the end. That would only allow single entry \nreplacements, if we then re-read the file and calculate the sha-1 at\nthe end. Partial reading also could only be implemented when reading\nthe whole file first to check the sha-1, which defeats it's purpose.\n"},{"id":"196763","messageId":"7vfw7vmzfb.fsf@alter.siamese.dyndns.org","threadId":"31208","inReplyTo":"7vobmjn0wv.fsf@alter.siamese.dyndns.org","subject":"Re: [PATCH/RFC v3 04/13] Add documentation of the index-v5 file format","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2012-08-09T23:13:44Z","receivedAt":"2012-08-09T23:13:44Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n>> +== Design explanations\n>> + ...\n>> +[3] The data of the cache-tree extension and the resolve undo\n>> + extension is now part of the index itself, but if other extensions\n>> + come up in the future, there is no need to change the index, they\n>> + can simply be added at the end.\n>\n> Interesting. When we added extensions, we said that there is no\n> need to change the index to add new features, they can simply be\n> added at the end. Perhaps the file offset table can be added as an\n> extension to v2 to give us the same bisectability, allowing us a\n> single entry in-place replacementability, without defining an\n> entirely different format?\n\nJust to avoid wasting people's time, in case they try to respond to\nthis part which was tongue-in-cheek.\n\nThere is a valid technical reason why the above cannot be done with\nthe original index format and why a new extension does not make\nsense. There is no quick way to locate the extensions section in the\nfile without reading all the entries first, so by the time you know\nwhere the extensions are, you already know all the paths, and have\nenough information to build a table necessary to do the bisection\nwithout a separate data in a new extension.\n"},{"id":"196764","messageId":"20120809232320.GD5127@tommy-fedora.scientificnet.net","threadId":"31208","inReplyTo":"7vk3x7n0fl.fsf@alter.siamese.dyndns.org","subject":"Re: [PATCH/RFC v3 07/13] Read resolve-undo data","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2012-08-09T23:23:20Z","receivedAt":"2012-08-09T23:23:20Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"On 08/09, Junio C Hamano wrote:\n> Thomas Gummerer writes:\n> \n> > Make git read the resolve-undo data from the index.\n> >\n> > Since the resolve-undo data is joined with the conflicts in\n> > the ondisk format of the index file version 5, conflicts and\n> > resolved data is read at the same time, and the resolve-undo\n> > data is then converted to the in-memory format.\n> \n> This, and the next one, are both about reading extension data from\n> the v2 formatted index, no?\n\nYes, exactly.\n\n> Again, mild NAK.\n> \n> I think it is a lot more logical for the v5 code to read data stored\n> in the resolve-undo and cache-tree extensions using the public API\n> just like other users of these data do, and write out whatever in a\n> way that is specific to the v5 index format.\n> \n> If the v5 codepath needs some information that is not exposed to\n> other users of istate->resolve_undo and istate->cache_tree, then the\n> story is different, but I do not think that is the case.\n\nSorry it's not clear to me what you mean with using the public API here.\nDo you mean using resolve_undo_write() and resolve_undo_read()? I\nwouldn't think those two methods would be really useful, as they expect\nthe data mangled in to a char* or return it as struct strbuf*. And I\ndon't see the other methods doing something more useful. Or I could\nuse the resolve-undo string_list directly, and just move the function\nto read-cache-v5.c, or am I missing something here?\n\n> >\n> > Helped-by: Thomas Rast \n> > Signed-off-by: Thomas Gummerer \n> > ---\n> > read-cache-v5.c | 1 +\n> > resolve-undo.c | 36 ++++++++++++++++++++++++++++++++++++\n> > resolve-undo.h | 2 ++\n> > 3 files changed, 39 insertions(+)\n> >\n> > diff --git a/read-cache-v5.c b/read-cache-v5.c\n> > index ec1201d..b47398d 100644\n> > --- a/read-cache-v5.c\n> > +++ b/read-cache-v5.c\n> > @@ -494,6 +494,7 @@ static struct directory_entry *read_entries(struct index_state *istate,\n> > \tint i;\n> > \n> > \tconflict_queue = read_conflicts(de, mmap, mmap_size, fd);\n> > +\tresolve_undo_convert_v5(istate, conflict_queue);\n> > \tfor (i = 0; i < de->de_nfiles; i++) {\n> > \t\tce = read_entry(de,\n> > \t\t\t\tentry_offset,\n> > diff --git a/resolve-undo.c b/resolve-undo.c\n> > index 72b4612..f96c6ba 100644\n> > --- a/resolve-undo.c\n> > +++ b/resolve-undo.c\n> > @@ -170,3 +170,39 @@ void unmerge_index(struct index_state *istate, const char **pathspec)\n> > \t\ti = unmerge_index_entry_at(istate, i);\n> > \t}\n> > }\n> > +\n> > +void resolve_undo_convert_v5(struct index_state *istate,\n> > +\t\t\t\t\tstruct conflict_entry *ce)\n> > +{\n> > +\tint i;\n> > +\n> > +\twhile (ce) {\n> > +\t\tstruct string_list_item *lost;\n> > +\t\tstruct resolve_undo_info *ui;\n> > +\t\tstruct conflict_part *cp;\n> > +\n> > +\t\tif (ce->entries && (ce->entries->flags & CONFLICT_CONFLICTED) != 0) {\n> > +\t\t\tce = ce->next;\n> > +\t\t\tcontinue;\n> > +\t\t}\n> > +\t\tif (!istate->resolve_undo) {\n> > +\t\t\tistate->resolve_undo = xcalloc(1, sizeof(struct string_list));\n> > +\t\t\tistate->resolve_undo->strdup_strings = 1;\n> > +\t\t}\n> > +\n> > +\t\tlost = string_list_insert(istate->resolve_undo, ce->name);\n> > +\t\tif (!lost->util)\n> > +\t\t\tlost->util = xcalloc(1, sizeof(*ui));\n> > +\t\tui = lost->util;\n> > +\n> > +\t\tcp = ce->entries;\n> > +\t\tfor (i = 0; i < 3; i++)\n> > +\t\t\tui->mode[i] = 0;\n> > +\t\twhile (cp) {\n> > +\t\t\tui->mode[conflict_stage(cp) - 1] = cp->entry_mode;\n> > +\t\t\thashcpy(ui->sha1[conflict_stage(cp) - 1], cp->sha1);\n> > +\t\t\tcp = cp->next;\n> > +\t\t}\n> > +\t\tce = ce->next;\n> > +\t}\n> > +}\n> > diff --git a/resolve-undo.h b/resolve-undo.h\n> > index 8458769..ab660a6 100644\n> > --- a/resolve-undo.h\n> > +++ b/resolve-undo.h\n> > @@ -13,4 +13,6 @@ extern void resolve_undo_clear_index(struct index_state *);\n> > extern int unmerge_index_entry_at(struct index_state *, int);\n> > extern void unmerge_index(struct index_state *, const char **);\n> > \n> > +extern void resolve_undo_convert_v5(struct index_state *, struct conflict_entry *);\n> > +\n> > #endif\n"},{"id":"196765","messageId":"7vboijmx5i.fsf@alter.siamese.dyndns.org","threadId":"31208","inReplyTo":"20120809232320.GD5127@tommy-fedora.scientificnet.net","subject":"Re: [PATCH/RFC v3 07/13] Read resolve-undo data","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2012-08-10T00:02:49Z","receivedAt":"2012-08-10T00:02:49Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Thomas Gummerer writes:\n\n> On 08/09, Junio C Hamano wrote:\n>> Thomas Gummerer writes:\n>> \n>> > Make git read the resolve-undo data from the index.\n>> >\n>> > Since the resolve-undo data is joined with the conflicts in\n>> > the ondisk format of the index file version 5, conflicts and\n>> > resolved data is read at the same time, and the resolve-undo\n>> > data is then converted to the in-memory format.\n>> \n>> This, and the next one, are both about reading extension data from\n>> the v2 formatted index, no?\n>\n> Yes, exactly.\n>\n>> Again, mild NAK.\n>> \n>> I think it is a lot more logical for the v5 code to read data stored\n>> in the resolve-undo and cache-tree extensions using the public API\n>> just like other users of these data do, and write out whatever in a\n>> way that is specific to the v5 index format.\n>> \n>> If the v5 codepath needs some information that is not exposed to\n>> other users of istate->resolve_undo and istate->cache_tree, then the\n>> story is different, but I do not think that is the case.\n>\n> Sorry it's not clear to me what you mean with using the public API here.\n> Do you mean using resolve_undo_write() and resolve_undo_read()?\n\nThe code that reads from istate->resolve_undo is fine to do the v5\nspecific conversion, but it does not belong to resolve-undo.c file\nwhich is about the resolve-undo extension. Moving it to v5 specific\nfile you added for this topic, read-cache-v5.c, and everything looks\nmore logical. When we taught ls-files to show the paths with\nresolve-undo data, we didn't add any function to resolve-undo.c that\ndoes ls-files's work for it. Instead, ls-files just uses the public\nAPI (the data structure you find at the_index.resolve_undo is part\nof the API) to find what it needs to learn, and I think v5 code can\ndo the same.\n\n\"then the story is different\" comment refers to a possibilty that\nv5 code might need something more than callers outside resolve-undo.c\ncan find from its public interface, but I do not think it is the\ncase.\n"},{"id":"196766","messageId":"7v7gt7mwo3.fsf@alter.siamese.dyndns.org","threadId":"31208","inReplyTo":"7vtxwbn2qe.fsf@alter.siamese.dyndns.org","subject":"Re: [PATCH/RFC v3 01/13] Move index v2 specific functions to their own file","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2012-08-10T00:13:16Z","receivedAt":"2012-08-10T00:13:16Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n> If you found that an entry you read halfway has an inconsistent crc,\n> and if you suspect that is because somebody else was writing to the\n> same index, it is a _sure_ sign that you are not alone, and all the\n> entries you read so far to the core, even if they weren't touched by\n> that sombody else when you read them, may be stale, and worse yet,\n> what you are going to read may be inconsistent with what you've read\n> and have in-core (e.g. you may have read \"f\" before somebody else\n> that is racing with you have turned it into a directory, and your\n> next read may find \"f/d\" in the index without crc error).\n>\n> One sane way to avoid reading such an inconsistent state may be to\n> redo this whole function, starting from the part that calls mmap().\n> IOW,\n>\n> \tdo {\n> \t\tfd = open()\n> \t\tmmap = xmmap(fd);\n> \t\tclose(fd);\n> verify_various_fields(mmap);\n> status = istate->ops->read_index(istate, mmap, mmap_size));\n> \t} while (status == READ_AGAIN);\n>\n> I do not think the \"pass fd around so that we can redo the mapping\n> deep inside the callchain\" is either a good idea or necessary.\n\nBy the way, you can only detect such inconsistency when you are\nlucky enough that you catch the other person in the middle of\nwriting.\n\nIf the index you are looking at holds a large tree with very many\npaths, it is possible that there are two large directories, and\nafter you read all entries from one, the other process starts\nmodifying the paths in that directory, without you ever finding it\nout. If the goal of the topic is to make the index work better in\nprojects with large trees, it may be wise to think about locking the\nwhole thing, so that you do not have to rely on the per-entry crc\nand you being lucky to detect such a race. The per-entry crc, as\nfar as I understand, may have been introduced primarily to detect\non-disk data corruption; it is not a suitable mechanism to detect\nconflicting accesses.\n"},{"id":"196773","messageId":"CACsJy8ChnS4cE3YnQQA8YQ5ebCA06QBXxEB7Oqhv4z_1y9Wc8w@mail.gmail.com","threadId":"31208","inReplyTo":"7v7gt7mwo3.fsf@alter.siamese.dyndns.org","subject":"Re: [PATCH/RFC v3 01/13] Move index v2 specific functions to their own file","fromName":"Nguyen Thai Ngoc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2012-08-10T02:23:07Z","receivedAt":"2012-08-10T02:23:07Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Fri, Aug 10, 2012 at 7:13 AM, Junio C Hamano wrote:\n> By the way, you can only detect such inconsistency when you are\n> lucky enough that you catch the other person in the middle of\n> writing.\n>\n> If the index you are looking at holds a large tree with very many\n> paths, it is possible that there are two large directories, and\n> after you read all entries from one, the other process starts\n> modifying the paths in that directory, without you ever finding it\n> out. If the goal of the topic is to make the index work better in\n> projects with large trees, it may be wise to think about locking the\n> whole thing, so that you do not have to rely on the per-entry crc\n> and you being lucky to detect such a race. The per-entry crc, as\n> far as I understand, may have been introduced primarily to detect\n> on-disk data corruption; it is not a suitable mechanism to detect\n> conflicting accesses.\n\nSo we acquire the lock just before we need to write, or at the time we\nopen for reading?\n-- \nDuy\n"},{"id":"196787","messageId":"20120810092733.GE5127@tommy-fedora.scientificnet.net","threadId":"31208","inReplyTo":"7vboijmx5i.fsf@alter.siamese.dyndns.org","subject":"Re: [PATCH/RFC v3 07/13] Read resolve-undo data","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2012-08-10T09:27:34Z","receivedAt":"2012-08-10T09:27:34Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"On 08/09, Junio C Hamano wrote:\n> Thomas Gummerer writes:\n> \n> > On 08/09, Junio C Hamano wrote:\n> >> Thomas Gummerer writes:\n> >> \n> >> > Make git read the resolve-undo data from the index.\n> >> >\n> >> > Since the resolve-undo data is joined with the conflicts in\n> >> > the ondisk format of the index file version 5, conflicts and\n> >> > resolved data is read at the same time, and the resolve-undo\n> >> > data is then converted to the in-memory format.\n> >> \n> >> This, and the next one, are both about reading extension data from\n> >> the v2 formatted index, no?\n> >\n> > Yes, exactly.\n> >\n> >> Again, mild NAK.\n> >> \n> >> I think it is a lot more logical for the v5 code to read data stored\n> >> in the resolve-undo and cache-tree extensions using the public API\n> >> just like other users of these data do, and write out whatever in a\n> >> way that is specific to the v5 index format.\n> >> \n> >> If the v5 codepath needs some information that is not exposed to\n> >> other users of istate->resolve_undo and istate->cache_tree, then the\n> >> story is different, but I do not think that is the case.\n> >\n> > Sorry it's not clear to me what you mean with using the public API here.\n> > Do you mean using resolve_undo_write() and resolve_undo_read()?\n> \n> The code that reads from istate->resolve_undo is fine to do the v5\n> specific conversion, but it does not belong to resolve-undo.c file\n> which is about the resolve-undo extension. Moving it to v5 specific\n> file you added for this topic, read-cache-v5.c, and everything looks\n> more logical. When we taught ls-files to show the paths with\n> resolve-undo data, we didn't add any function to resolve-undo.c that\n> does ls-files's work for it. Instead, ls-files just uses the public\n> API (the data structure you find at the_index.resolve_undo is part\n> of the API) to find what it needs to learn, and I think v5 code can\n> do the same.\n> \n> \"then the story is different\" comment refers to a possibilty that\n> v5 code might need something more than callers outside resolve-undo.c\n> can find from its public interface, but I do not think it is the\n> case.\n\nOk, thanks for the clarification, will change it for the re-roll.\n"},{"id":"196791","messageId":"87hasan7ty.fsf@thomas.inf.ethz.ch","threadId":"31208","inReplyTo":"7vtxwbn2qe.fsf@alter.siamese.dyndns.org","subject":"Re: [PATCH/RFC v3 01/13] Move index v2 specific functions to their own file","fromName":"Thomas Rast","fromEmail":"trast@student.ethz.ch","sentAt":"2012-08-10T14:24:25Z","receivedAt":"2012-08-10T14:24:25Z","isPatch":true,"sender":{"key":"tr@thomasrast.ch","avatar":"https://avatars.githubusercontent.com/u/153510?v=4"},"body":"Junio C Hamano writes:\n\n> If you found that an entry you read halfway has an inconsistent crc,\n> and if you suspect that is because somebody else was writing to the\n> same index, it is a _sure_ sign that you are not alone, and all the\n> entries you read so far to the core, even if they weren't touched by\n> that sombody else when you read them, may be stale, and worse yet,\n> what you are going to read may be inconsistent with what you've read\n> and have in-core (e.g. you may have read \"f\" before somebody else\n> that is racing with you have turned it into a directory, and your\n> next read may find \"f/d\" in the index without crc error).\n\nThe intention for v5 (admittedly this probably requires a lot more\ndocumentation) was to only allow an in-place update in two cases:\n\n* updating the data fields (*not* the name) of a file entry,\n\n* adding or removing conflict entries at the end.\n\nThe latter probably requires a bit more thought to make it safe, too.\nBut I think the idea always was that any write that changes the basic\nlayout of the file (so that you would read something wrong) will need a\nfull rewrite. Otherwise we're too far in DB land. Most updates will be\nof the \"update the stat and/or sha1 of a file\" kind, anyway.\n\n-- \nThomas Rast\ntrast@{inf,student}.ethz.ch\n"},{"id":"196795","messageId":"7vobmilro1.fsf@alter.siamese.dyndns.org","threadId":"31208","inReplyTo":"87hasan7ty.fsf@thomas.inf.ethz.ch","subject":"Re: [PATCH/RFC v3 01/13] Move index v2 specific functions to their own file","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2012-08-10T14:58:54Z","receivedAt":"2012-08-10T14:58:54Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Thomas Rast writes:\n\n> But I think the idea always was that any write that changes the basic\n> layout of the file (so that you would read something wrong) will need a\n> full rewrite. Otherwise we're too far in DB land.\n>\n> Most updates will be\n> of the \"update the stat and/or sha1 of a file\" kind, anyway.\n\nYes, I agree the v5 format documented in the series does not let you\ndo anything other than the kind of updates without rewriting [*1*]\n\nBut that does not fundamentally change the story that a new format\nand a new way to access the index to cope with larger projects would\nwant to come up with a solution to address the competing read/write\nissue, or at least help to make it easier to solve the issue in the\nfuture.\n\n\"That problem is not new\" is not an answer when the question is \"We\nstill have the problem\".\n\n\n[Footnote]\n\n*1* While my gut feeling matches your guess that the kind of updates\nwould be the majority, I do not think anybody did numbers to\nsubstanticate it, which we may want to see happen.\n"},{"id":"196802","messageId":"20120810154015.GF5127@tommy-fedora.scientificnet.net","threadId":"31208","inReplyTo":"7vobmilro1.fsf@alter.siamese.dyndns.org","subject":"Re: [PATCH/RFC v3 01/13] Move index v2 specific functions to their own file","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2012-08-10T15:40:15Z","receivedAt":"2012-08-10T15:40:15Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"On 08/10, Junio C Hamano wrote:\n> Thomas Rast writes:\n> \n> > But I think the idea always was that any write that changes the basic\n> > layout of the file (so that you would read something wrong) will need a\n> > full rewrite. Otherwise we're too far in DB land.\n> >\n> > Most updates will be\n> > of the \"update the stat and/or sha1 of a file\" kind, anyway.\n> \n> Yes, I agree the v5 format documented in the series does not let you\n> do anything other than the kind of updates without rewriting [*1*]\n> \n> But that does not fundamentally change the story that a new format\n> and a new way to access the index to cope with larger projects would\n> want to come up with a solution to address the competing read/write\n> issue, or at least help to make it easier to solve the issue in the\n> future.\n> \n> \"That problem is not new\" is not an answer when the question is \"We\n> still have the problem\".\n> \n> \n> [Footnote]\n> \n> *1* While my gut feeling matches your guess that the kind of updates\n> would be the majority, I do not think anybody did numbers to\n> substanticate it, which we may want to see happen.\n\nHrm anther way to solve this may be the following. The idea would be\nto just check if the index_file changed basically using the same\nheuristic we already use to detect file changes. (use the stat data,\nmtime, size, etc.)\n\nWith this code we do not rely on the crc sums to check if the index\nneeds to be re-read anymore and don't have a problem if part of the\nindex changes, after we read it (we know the index changed from its\nmtime and can just re-read it). Another thing that would have to\nchange is that we can't use die if a crc is wrong, but some return\ncode, but that shouldn't be a problem. I'm not sure I'm not missing\nsomething here though.\n\ndo {\n\tfd = open()\n\tfstat(fd, &st_old);\n\tmmap = xmmap(fd);\n\tverify_various_fields(mmap);\n\tistate->ops->read_index(istate,\n\t\t\t\tmmap,\n\t\t\t\tmmap_size));\n\tfstat(fd, &st_new);\n\tclose(fd);\n} while (stat_data_doesnt_match(st_old, st_new));\n"}]}