{"thread":{"id":"31170","subject":"[PATCH 02/16] Modify read functions to prepare for other index formats","startedAt":"2012-08-02T11:01:50Z","lastAt":"2012-08-03T16:12:26Z","messageCount":38,"participants":["Thomas Gummerer","Nguyen Thai Ngoc Duy","Thomas Rast","Junio C Hamano"],"isPatch":true,"patchVersion":1,"patchTotal":16},"messages":[{"id":"196331","messageId":"1343905326-23790-1-git-send-email-t.gummerer@gmail.com","threadId":"31170","inReplyTo":null,"subject":"[RFC 0/16] Introduce index file format version 5","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2012-08-02T11:01:50Z","receivedAt":"2012-08-02T11:01:50Z","isPatch":false,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"Series of patches to introduce the index version 5 file format. This\nseries does not include any fancy stuff like partial loading or partial\nwriting yet, though it's possible to do that with the new format.\n\nThere was already a POC for partial loading, which gave pretty good\nresults, which was however broken in all but the general case, so it's\nnot included yet. (for timings see: http://thread.gmane.org/gmane.comp.version-control.git/201964/focus=202019)\n\nThe first 4 patches are refactoring the old code, splitting it up into\ndifferent functions, as a preparation for index-v5.\n\nPatches 5 and 6 fix testcases for index v5.\n\nPatch 9..11 introduce the reader for index-v5. I've split those\npatches up to read the main index first, then the resolve-undo\ndata and then the cache-tree, to make it easier to review them.\n\nThe same goes for patches 12..14, which introduce the writer, again\nsplit up in writing the main index, resolve-undo data and cache-tree\ndata.\n\nPatch 15 adds a option to update index to force-rewrite the index,\nso rewriting it even if nothing has changed. This is later used\nfor performance testing, to test the performance for both the reader and\nthe writer.\n\nPatch 16 adds the performance test, which compares the time for\nforce-rewrites for index-v[23], index-v4 and index-v5.\n\nThe default index format is still set to 3, it can be changed in\nread-cache.c (INDEX_FORMAT_DEFAULT)\n\n[PATCH 01/16] Modify cache_header to prepare for other index formats\n[PATCH 02/16] Modify read functions to prepare for other index\n[PATCH 03/16] Modify match_stat_basic to prepare for other index\n[PATCH 04/16] Modify write functions to prepare for other index\n[PATCH 05/16] t2104: Don't fail when index version is 5\n[PATCH 06/16] t3700: sleep for 1 second, to avoid interfering with\n[PATCH 07/16] Add documentation of the index-v5 file format\n[PATCH 08/16] Make in-memory format aware of stat_crc\n[PATCH 09/16] Read index-v5\n[PATCH 10/16] Read resolve-undo data\n[PATCH 11/16] Read cache-tree in index-v5\n[PATCH 12/16] Write index-v5\n[PATCH 13/16] Write index-v5 cache-tree data\n[PATCH 14/16] Write resolve-undo data for index-v5\n[PATCH 15/16] update-index.c: add a force-rewrite option\n[PATCH 16/16] p0002-index.sh: add perf test for the index formats\n\nDocumentation/technical/index-file-format-v5.txt |  281 ++++++++++++++++++++++++++++++++++\nbuiltin/update-index.c                           |    5 +-\ncache-tree.c                                     |  145 ++++++++++++++++++\ncache-tree.h                                     |    7 +\ncache.h                                          |   96 +++++++++++-\nread-cache.c                                     | 1519 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-----------------\nresolve-undo.c                                   |  129 ++++++++++++++++\nresolve-undo.h                                   |    3 +\nt/perf/p0002-index.sh                            |   33 ++++\nt/t2104-update-index-skip-worktree.sh            |   15 +-\nt/t3700-add.sh                                   |    1 +\ntest-index-version.c                             |    2 +-\n12 files changed, 2082 insertions(+), 154 deletions(-)\n"},{"id":"196343","messageId":"1343905326-23790-2-git-send-email-t.gummerer@gmail.com","threadId":"31170","inReplyTo":"1343905326-23790-1-git-send-email-t.gummerer@gmail.com","subject":"[PATCH 01/16] Modify cache_header to prepare for other index formats","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2012-08-02T11:01:51Z","receivedAt":"2012-08-02T11:01:51Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"Modify the cache_header such that other index file formats\ncan be added and reusing the common part of each index format.\n\nThe signature and version have to be present in every\nversion of the index file format, to check if it can be read\nby a specific version of git, while other entries (eg. number\nof entries for index v2/3/4) can be different from one file\nformat to another. Therefore it is split to its own struct.\n\nSigned-off-by: Thomas Gummerer <t.gummerer@gmail.com>\n---\n cache.h              |  5 ++++-\n read-cache.c         | 20 +++++++++++++-------\n test-index-version.c |  2 +-\n 3 files changed, 18 insertions(+), 9 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 6e9a243..d4028ef 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -99,9 +99,12 @@ unsigned long git_deflate_bound(git_zstream *, unsigned long);\n  */\n \n #define CACHE_SIGNATURE 0x44495243\t/* \"DIRC\" */\n-struct cache_header {\n+struct cache_version_header {\n \tunsigned int hdr_signature;\n \tunsigned int hdr_version;\n+};\n+\n+struct cache_header_v2 {\n \tunsigned int hdr_entries;\n };\n \ndiff --git a/read-cache.c b/read-cache.c\nindex ab00d02..c44b5f7 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -1247,7 +1247,7 @@ struct ondisk_cache_entry_extended {\n \t\t\t    ondisk_cache_entry_extended_size(ce_namelen(ce)) : \\\n \t\t\t    ondisk_cache_entry_size(ce_namelen(ce)))\n \n-static int verify_hdr(struct cache_header *hdr, unsigned long size)\n+static int verify_hdr(struct cache_version_header *hdr, unsigned long size)\n {\n \tgit_SHA_CTX c;\n \tunsigned char sha1[20];\n@@ -1409,7 +1409,8 @@ int read_index_from(struct index_state *istate, const char *path)\n \tint fd, i;\n \tstruct stat st;\n \tunsigned long src_offset;\n-\tstruct cache_header *hdr;\n+\tstruct cache_version_header *hdr;\n+\tstruct cache_header_v2 *hdr_v2;\n \tvoid *mmap;\n \tsize_t mmap_size;\n \tstruct strbuf previous_name_buf = STRBUF_INIT, *previous_name;\n@@ -1433,7 +1434,7 @@ int read_index_from(struct index_state *istate, const char *path)\n \n \terrno = EINVAL;\n \tmmap_size = xsize_t(st.st_size);\n-\tif (mmap_size < sizeof(struct cache_header) + 20)\n+\tif (mmap_size < sizeof(struct cache_version_header) + 20)\n \t\tdie(\"index file smaller than expected\");\n \n \tmmap = xmmap(NULL, mmap_size, PROT_READ | PROT_WRITE, MAP_PRIVATE, fd, 0);\n@@ -1442,11 +1443,13 @@ int read_index_from(struct index_state *istate, const char *path)\n \t\tdie_errno(\"unable to map index file\");\n \n \thdr = mmap;\n+\thdr_v2 =  mmap + sizeof(*hdr);\n \tif (verify_hdr(hdr, mmap_size) < 0)\n \t\tgoto unmap;\n \n+\thdr_v2 = mmap + sizeof(*hdr);\n \tistate->version = ntohl(hdr->hdr_version);\n-\tistate->cache_nr = ntohl(hdr->hdr_entries);\n+\tistate->cache_nr = ntohl(hdr_v2->hdr_entries);\n \tistate->cache_alloc = alloc_nr(istate->cache_nr);\n \tistate->cache = xcalloc(istate->cache_alloc, sizeof(struct cache_entry *));\n \tistate->initialized = 1;\n@@ -1456,7 +1459,7 @@ int read_index_from(struct index_state *istate, const char *path)\n \telse\n \t\tprevious_name = NULL;\n \n-\tsrc_offset = sizeof(*hdr);\n+\tsrc_offset = sizeof(*hdr) + sizeof(*hdr_v2);\n \tfor (i = 0; i < istate->cache_nr; i++) {\n \t\tstruct ondisk_cache_entry *disk_ce;\n \t\tstruct cache_entry *ce;\n@@ -1757,7 +1760,8 @@ void update_index_if_able(struct index_state *istate, struct lock_file *lockfile\n int write_index(struct index_state *istate, int newfd)\n {\n \tgit_SHA_CTX c;\n-\tstruct cache_header hdr;\n+\tstruct cache_version_header hdr;\n+\tstruct cache_header_v2 hdr_v2;\n \tint i, err, removed, extended, hdr_version;\n \tstruct cache_entry **cache = istate->cache;\n \tint entries = istate->cache_nr;\n@@ -1787,11 +1791,13 @@ int write_index(struct index_state *istate, int newfd)\n \n \thdr.hdr_signature = htonl(CACHE_SIGNATURE);\n \thdr.hdr_version = htonl(hdr_version);\n-\thdr.hdr_entries = htonl(entries - removed);\n+\thdr_v2.hdr_entries = htonl(entries - removed);\n \n \tgit_SHA1_Init(&c);\n \tif (ce_write(&c, newfd, &hdr, sizeof(hdr)) < 0)\n \t\treturn -1;\n+\tif (ce_write(&c, newfd, &hdr_v2, sizeof(hdr_v2)) < 0)\n+\t\treturn -1;\n \n \tprevious_name = (hdr_version == 4) ? &previous_name_buf : NULL;\n \tfor (i = 0; i < entries; i++) {\ndiff --git a/test-index-version.c b/test-index-version.c\nindex bfaad9e..f21372a 100644\n--- a/test-index-version.c\n+++ b/test-index-version.c\n@@ -2,7 +2,7 @@\n \n int main(int argc, const char **argv)\n {\n-\tstruct cache_header hdr;\n+\tstruct cache_version_header hdr;\n \tint version;\n \n \tmemset(&hdr,0,sizeof(hdr));\n-- \n1.7.10.886.gdf6792c.dirty\n"},{"id":"196330","messageId":"1343905326-23790-3-git-send-email-t.gummerer@gmail.com","threadId":"31170","inReplyTo":"1343905326-23790-1-git-send-email-t.gummerer@gmail.com","subject":"[PATCH 02/16] Modify read functions to prepare for other index formats","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2012-08-02T11:01:52Z","receivedAt":"2012-08-02T11:01:52Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"Modify the read_index_from function, splitting it up into\none function that stays the same for every index format,\ndoing the basic operations such as verifying the header,\nand a function which is specific for each index version,\nwhich does the real reading of the index.\n\nSigned-off-by: Thomas Gummerer <t.gummerer@gmail.com>\n---\n cache.h      |   1 +\n read-cache.c | 108 +++++++++++++++++++++++++++++++++++------------------------\n 2 files changed, 66 insertions(+), 43 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex d4028ef..3aa70d8 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -435,6 +435,7 @@ extern int init_db(const char *template_dir, unsigned int flags);\n /* Initialize and use the cache information */\n extern int read_index(struct index_state *);\n extern int read_index_preload(struct index_state *, const char **pathspec);\n+extern void read_index_v2(struct index_state *, void *mmap, int);\n extern int read_index_from(struct index_state *, const char *path);\n extern int is_index_unborn(struct index_state *);\n extern int read_index_unmerged(struct index_state *);\ndiff --git a/read-cache.c b/read-cache.c\nindex c44b5f7..3d83f05 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -1247,10 +1247,8 @@ struct ondisk_cache_entry_extended {\n \t\t\t    ondisk_cache_entry_extended_size(ce_namelen(ce)) : \\\n \t\t\t    ondisk_cache_entry_size(ce_namelen(ce)))\n \n-static int verify_hdr(struct cache_version_header *hdr, unsigned long size)\n+static int verify_hdr_version(struct cache_version_header *hdr, unsigned long size)\n {\n-\tgit_SHA_CTX c;\n-\tunsigned char sha1[20];\n \tint hdr_version;\n \n \tif (hdr->hdr_signature != htonl(CACHE_SIGNATURE))\n@@ -1258,6 +1256,14 @@ static int verify_hdr(struct cache_version_header *hdr, unsigned long size)\n \thdr_version = ntohl(hdr->hdr_version);\n \tif (hdr_version < 2 || 4 < hdr_version)\n \t\treturn error(\"bad index version %d\", hdr_version);\n+\treturn 0;\n+}\n+\n+static int verify_hdr_v2(struct cache_version_header *hdr, unsigned long size)\n+{\n+\tgit_SHA_CTX c;\n+\tunsigned char sha1[20];\n+\n \tgit_SHA1_Init(&c);\n \tgit_SHA1_Update(&c, hdr, size - 20);\n \tgit_SHA1_Final(sha1, &c);\n@@ -1403,50 +1409,15 @@ static struct cache_entry *create_from_disk(struct ondisk_cache_entry *ondisk,\n \treturn ce;\n }\n \n-/* remember to discard_cache() before reading a different cache! */\n-int read_index_from(struct index_state *istate, const char *path)\n+void read_index_v2(struct index_state *istate, void *mmap, int mmap_size)\n {\n-\tint fd, i;\n-\tstruct stat st;\n+\tint i;\n \tunsigned long src_offset;\n \tstruct cache_version_header *hdr;\n \tstruct cache_header_v2 *hdr_v2;\n-\tvoid *mmap;\n-\tsize_t mmap_size;\n \tstruct strbuf previous_name_buf = STRBUF_INIT, *previous_name;\n \n-\terrno = EBUSY;\n-\tif (istate->initialized)\n-\t\treturn istate->cache_nr;\n-\n-\terrno = ENOENT;\n-\tistate->timestamp.sec = 0;\n-\tistate->timestamp.nsec = 0;\n-\tfd = open(path, O_RDONLY);\n-\tif (fd < 0) {\n-\t\tif (errno == ENOENT)\n-\t\t\treturn 0;\n-\t\tdie_errno(\"index file open failed\");\n-\t}\n-\n-\tif (fstat(fd, &st))\n-\t\tdie_errno(\"cannot stat the open index\");\n-\n-\terrno = EINVAL;\n-\tmmap_size = xsize_t(st.st_size);\n-\tif (mmap_size < sizeof(struct cache_version_header) + 20)\n-\t\tdie(\"index file smaller than expected\");\n-\n-\tmmap = xmmap(NULL, mmap_size, PROT_READ | PROT_WRITE, MAP_PRIVATE, fd, 0);\n-\tclose(fd);\n-\tif (mmap == MAP_FAILED)\n-\t\tdie_errno(\"unable to map index file\");\n-\n \thdr = mmap;\n-\thdr_v2 =  mmap + sizeof(*hdr);\n-\tif (verify_hdr(hdr, mmap_size) < 0)\n-\t\tgoto unmap;\n-\n \thdr_v2 = mmap + sizeof(*hdr);\n \tistate->version = ntohl(hdr->hdr_version);\n \tistate->cache_nr = ntohl(hdr_v2->hdr_entries);\n@@ -1472,8 +1443,6 @@ int read_index_from(struct index_state *istate, const char *path)\n \t\tsrc_offset += consumed;\n \t}\n \tstrbuf_release(&previous_name_buf);\n-\tistate->timestamp.sec = st.st_mtime;\n-\tistate->timestamp.nsec = ST_MTIME_NSEC(st);\n \n \twhile (src_offset <= mmap_size - 20 - 8) {\n \t\t/* After an array of active_nr index entries,\n@@ -1493,12 +1462,65 @@ int read_index_from(struct index_state *istate, const char *path)\n \t\tsrc_offset += 8;\n \t\tsrc_offset += extsize;\n \t}\n+\treturn;\n+unmap:\n+\tmunmap(mmap, mmap_size);\n+\tdie(\"index file corrupt\");\n+}\n+\n+/* remember to discard_cache() before reading a different cache! */\n+int read_index_from(struct index_state *istate, const char *path)\n+{\n+\tint fd;\n+\tstruct stat st;\n+\tstruct cache_version_header *hdr;\n+\tvoid *mmap;\n+\tsize_t mmap_size;\n+\n+\terrno = EBUSY;\n+\tif (istate->initialized)\n+\t\treturn istate->cache_nr;\n+\n+\terrno = ENOENT;\n+\tistate->timestamp.sec = 0;\n+\tistate->timestamp.nsec = 0;\n+\tfd = open(path, O_RDONLY);\n+\tif (fd < 0) {\n+\t\tif (errno == ENOENT)\n+\t\t\treturn 0;\n+\t\tdie_errno(\"index file open failed\");\n+\t}\n+\n+\tif (fstat(fd, &st))\n+\t\tdie_errno(\"cannot stat the open index\");\n+\n+\terrno = EINVAL;\n+\tmmap_size = xsize_t(st.st_size);\n+\tif (mmap_size < sizeof(struct cache_version_header) + 20)\n+\t\tdie(\"index file smaller than expected\");\n+\n+\tmmap = xmmap(NULL, mmap_size, PROT_READ | PROT_WRITE, MAP_PRIVATE, fd, 0);\n+\tclose(fd);\n+\tif (mmap == MAP_FAILED)\n+\t\tdie_errno(\"unable to map index file\");\n+\n+\thdr = mmap;\n+\tif (verify_hdr_version(hdr, mmap_size) < 0)\n+\t\tgoto unmap;\n+\n+\tif (verify_hdr_v2(hdr, mmap_size) < 0)\n+\t\tgoto unmap;\n+\n+\tread_index_v2(istate, mmap, mmap_size);\n+\n+\tistate->timestamp.sec = st.st_mtime;\n+\tistate->timestamp.nsec = ST_MTIME_NSEC(st);\n+\n \tmunmap(mmap, mmap_size);\n \treturn istate->cache_nr;\n \n unmap:\n \tmunmap(mmap, mmap_size);\n-\terrno = EINVAL;\n \tdie(\"index file corrupt\");\n }\n \n-- \n1.7.10.886.gdf6792c.dirty\n"},{"id":"196332","messageId":"1343905326-23790-4-git-send-email-t.gummerer@gmail.com","threadId":"31170","inReplyTo":"1343905326-23790-1-git-send-email-t.gummerer@gmail.com","subject":"[PATCH 03/16] Modify match_stat_basic to prepare for other index formats","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2012-08-02T11:01:53Z","receivedAt":"2012-08-02T11:01:53Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"Modify match_stat_basic, into one function that handles the\ngeneral case, which is the same for all index formats, and\na function that handles the specific parts for each index\nfile version.\n\nSigned-off-by: Thomas Gummerer <t.gummerer@gmail.com>\n---\n read-cache.c | 77 +++++++++++++++++++++++++++++++-----------------------------\n 1 file changed, 40 insertions(+), 37 deletions(-)\n\ndiff --git a/read-cache.c b/read-cache.c\nindex 3d83f05..6a0af35 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -163,38 +163,10 @@ static int ce_modified_check_fs(struct cache_entry *ce, struct stat *st)\n \treturn 0;\n }\n \n-static int ce_match_stat_basic(struct cache_entry *ce, struct stat *st)\n+static int ce_match_stat_basic_v2(struct cache_entry *ce,\n+\t\t\t\tstruct stat *st,\n+\t\t\t\tint changed)\n {\n-\tunsigned int changed = 0;\n-\n-\tif (ce->ce_flags & CE_REMOVE)\n-\t\treturn MODE_CHANGED | DATA_CHANGED | TYPE_CHANGED;\n-\n-\tswitch (ce->ce_mode & S_IFMT) {\n-\tcase S_IFREG:\n-\t\tchanged |= !S_ISREG(st->st_mode) ? TYPE_CHANGED : 0;\n-\t\t/* We consider only the owner x bit to be relevant for\n-\t\t * \"mode changes\"\n-\t\t */\n-\t\tif (trust_executable_bit &&\n-\t\t    (0100 & (ce->ce_mode ^ st->st_mode)))\n-\t\t\tchanged |= MODE_CHANGED;\n-\t\tbreak;\n-\tcase S_IFLNK:\n-\t\tif (!S_ISLNK(st->st_mode) &&\n-\t\t    (has_symlinks || !S_ISREG(st->st_mode)))\n-\t\t\tchanged |= TYPE_CHANGED;\n-\t\tbreak;\n-\tcase S_IFGITLINK:\n-\t\t/* We ignore most of the st_xxx fields for gitlinks */\n-\t\tif (!S_ISDIR(st->st_mode))\n-\t\t\tchanged |= TYPE_CHANGED;\n-\t\telse if (ce_compare_gitlink(ce))\n-\t\t\tchanged |= DATA_CHANGED;\n-\t\treturn changed;\n-\tdefault:\n-\t\tdie(\"internal error: ce_mode is %o\", ce->ce_mode);\n-\t}\n \tif (ce->ce_mtime.sec != (unsigned int)st->st_mtime)\n \t\tchanged |= MTIME_CHANGED;\n \tif (trust_ctime && ce->ce_ctime.sec != (unsigned int)st->st_ctime)\n@@ -235,6 +207,43 @@ static int ce_match_stat_basic(struct cache_entry *ce, struct stat *st)\n \treturn changed;\n }\n \n+static int ce_match_stat_basic(struct cache_entry *ce, struct stat *st)\n+{\n+\tunsigned int changed = 0;\n+\n+\tif (ce->ce_flags & CE_REMOVE)\n+\t\treturn MODE_CHANGED | DATA_CHANGED | TYPE_CHANGED;\n+\n+\tswitch (ce->ce_mode & S_IFMT) {\n+\tcase S_IFREG:\n+\t\tchanged |= !S_ISREG(st->st_mode) ? TYPE_CHANGED : 0;\n+\t\t/* We consider only the owner x bit to be relevant for\n+\t\t * \"mode changes\"\n+\t\t */\n+\t\tif (trust_executable_bit &&\n+\t\t    (0100 & (ce->ce_mode ^ st->st_mode)))\n+\t\t\tchanged |= MODE_CHANGED;\n+\t\tbreak;\n+\tcase S_IFLNK:\n+\t\tif (!S_ISLNK(st->st_mode) &&\n+\t\t    (has_symlinks || !S_ISREG(st->st_mode)))\n+\t\t\tchanged |= TYPE_CHANGED;\n+\t\tbreak;\n+\tcase S_IFGITLINK:\n+\t\t/* We ignore most of the st_xxx fields for gitlinks */\n+\t\tif (!S_ISDIR(st->st_mode))\n+\t\t\tchanged |= TYPE_CHANGED;\n+\t\telse if (ce_compare_gitlink(ce))\n+\t\t\tchanged |= DATA_CHANGED;\n+\t\treturn changed;\n+\tdefault:\n+\t\tdie(\"internal error: ce_mode is %o\", ce->ce_mode);\n+\t}\n+\n+\tchanged = ce_match_stat_basic_v2(ce, st, changed);\n+\treturn changed;\n+}\n+\n static int is_racy_timestamp(const struct index_state *istate, struct cache_entry *ce)\n {\n \treturn (!S_ISGITLINK(ce->ce_mode) &&\n@@ -1443,7 +1452,6 @@ void read_index_v2(struct index_state *istate, void *mmap, int mmap_size)\n \t\tsrc_offset += consumed;\n \t}\n \tstrbuf_release(&previous_name_buf);\n-\n \twhile (src_offset <= mmap_size - 20 - 8) {\n \t\t/* After an array of active_nr index entries,\n \t\t * there can be arbitrary number of extended\n@@ -1500,7 +1508,6 @@ int read_index_from(struct index_state *istate, const char *path)\n \t\tdie(\"index file smaller than expected\");\n \n \tmmap = xmmap(NULL, mmap_size, PROT_READ | PROT_WRITE, MAP_PRIVATE, fd, 0);\n-\tclose(fd);\n \tif (mmap == MAP_FAILED)\n \t\tdie_errno(\"unable to map index file\");\n \n@@ -1512,7 +1519,6 @@ int read_index_from(struct index_state *istate, const char *path)\n \t\tgoto unmap;\n \n \tread_index_v2(istate, mmap, mmap_size);\n-\n \tistate->timestamp.sec = st.st_mtime;\n \tistate->timestamp.nsec = ST_MTIME_NSEC(st);\n \n@@ -1802,9 +1808,6 @@ int write_index(struct index_state *istate, int newfd)\n \t\t}\n \t}\n \n-\tif (!istate->version)\n-\t\tistate->version = INDEX_FORMAT_DEFAULT;\n-\n \t/* demote version 3 to version 2 when the latter suffices */\n \tif (istate->version == 3 || istate->version == 2)\n \t\tistate->version = extended ? 3 : 2;\n-- \n1.7.10.886.gdf6792c.dirty\n"},{"id":"196335","messageId":"1343905326-23790-5-git-send-email-t.gummerer@gmail.com","threadId":"31170","inReplyTo":"1343905326-23790-1-git-send-email-t.gummerer@gmail.com","subject":"[PATCH 04/16] Modify write functions to prepare for other index formats","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2012-08-02T11:01:54Z","receivedAt":"2012-08-02T11:01:54Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"Modify the write_index function to add the possibility to add\nother index formats, that are written in a different way. Also\nmark all functions, which shall only be used with v2-v4 as v2\nfunctions.\n\nSigned-off-by: Thomas Gummerer <t.gummerer@gmail.com>\n---\n read-cache.c | 40 ++++++++++++++++++++++++----------------\n 1 file changed, 24 insertions(+), 16 deletions(-)\n\ndiff --git a/read-cache.c b/read-cache.c\nindex 6a0af35..1c804e1 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -1581,7 +1581,7 @@ static int ce_write_flush(git_SHA_CTX *context, int fd)\n \treturn 0;\n }\n \n-static int ce_write(git_SHA_CTX *context, int fd, void *data, unsigned int len)\n+static int ce_write_v2(git_SHA_CTX *context, int fd, void *data, unsigned int len)\n {\n \twhile (len) {\n \t\tunsigned int buffered = write_buffer_len;\n@@ -1603,13 +1603,13 @@ static int ce_write(git_SHA_CTX *context, int fd, void *data, unsigned int len)\n \treturn 0;\n }\n \n-static int write_index_ext_header(git_SHA_CTX *context, int fd,\n+static int write_index_ext_header_v2(git_SHA_CTX *context, int fd,\n \t\t\t\t  unsigned int ext, unsigned int sz)\n {\n \text = htonl(ext);\n \tsz = htonl(sz);\n-\treturn ((ce_write(context, fd, &ext, 4) < 0) ||\n-\t\t(ce_write(context, fd, &sz, 4) < 0)) ? -1 : 0;\n+\treturn ((ce_write_v2(context, fd, &ext, 4) < 0) ||\n+\t\t(ce_write_v2(context, fd, &sz, 4) < 0)) ? -1 : 0;\n }\n \n static int ce_flush(git_SHA_CTX *context, int fd)\n@@ -1634,7 +1634,7 @@ static int ce_flush(git_SHA_CTX *context, int fd)\n \treturn (write_in_full(fd, write_buffer, left) != left) ? -1 : 0;\n }\n \n-static void ce_smudge_racily_clean_entry(struct cache_entry *ce)\n+static void ce_smudge_racily_clean_entry_v2(struct cache_entry *ce)\n {\n \t/*\n \t * The only thing we care about in this function is to smudge the\n@@ -1715,7 +1715,7 @@ static char *copy_cache_entry_to_ondisk(struct ondisk_cache_entry *ondisk,\n \t}\n }\n \n-static int ce_write_entry(git_SHA_CTX *c, int fd, struct cache_entry *ce,\n+static int ce_write_entry_v2(git_SHA_CTX *c, int fd, struct cache_entry *ce,\n \t\t\t  struct strbuf *previous_name)\n {\n \tint size;\n@@ -1755,7 +1755,7 @@ static int ce_write_entry(git_SHA_CTX *c, int fd, struct cache_entry *ce,\n \t\t\t      ce->name + common, ce_namelen(ce) - common);\n \t}\n \n-\tresult = ce_write(c, fd, ondisk, size);\n+\tresult = ce_write_v2(c, fd, ondisk, size);\n \tfree(ondisk);\n \treturn result;\n }\n@@ -1785,7 +1785,7 @@ void update_index_if_able(struct index_state *istate, struct lock_file *lockfile\n \t\trollback_lock_file(lockfile);\n }\n \n-int write_index(struct index_state *istate, int newfd)\n+int write_index_v2(struct index_state *istate, int newfd)\n {\n \tgit_SHA_CTX c;\n \tstruct cache_version_header hdr;\n@@ -1819,9 +1819,9 @@ int write_index(struct index_state *istate, int newfd)\n \thdr_v2.hdr_entries = htonl(entries - removed);\n \n \tgit_SHA1_Init(&c);\n-\tif (ce_write(&c, newfd, &hdr, sizeof(hdr)) < 0)\n+\tif (ce_write_v2(&c, newfd, &hdr, sizeof(hdr)) < 0)\n \t\treturn -1;\n-\tif (ce_write(&c, newfd, &hdr_v2, sizeof(hdr_v2)) < 0)\n+\tif (ce_write_v2(&c, newfd, &hdr_v2, sizeof(hdr_v2)) < 0)\n \t\treturn -1;\n \n \tprevious_name = (hdr_version == 4) ? &previous_name_buf : NULL;\n@@ -1830,8 +1830,8 @@ int write_index(struct index_state *istate, int newfd)\n \t\tif (ce->ce_flags & CE_REMOVE)\n \t\t\tcontinue;\n \t\tif (!ce_uptodate(ce) && is_racy_timestamp(istate, ce))\n-\t\t\tce_smudge_racily_clean_entry(ce);\n-\t\tif (ce_write_entry(&c, newfd, ce, previous_name) < 0)\n+\t\t\tce_smudge_racily_clean_entry_v2(ce);\n+\t\tif (ce_write_entry_v2(&c, newfd, ce, previous_name) < 0)\n \t\t\treturn -1;\n \t}\n \tstrbuf_release(&previous_name_buf);\n@@ -1841,8 +1841,8 @@ int write_index(struct index_state *istate, int newfd)\n \t\tstruct strbuf sb = STRBUF_INIT;\n \n \t\tcache_tree_write(&sb, istate->cache_tree);\n-\t\terr = write_index_ext_header(&c, newfd, CACHE_EXT_TREE, sb.len) < 0\n-\t\t\t|| ce_write(&c, newfd, sb.buf, sb.len) < 0;\n+\t\terr = write_index_ext_header_v2(&c, newfd, CACHE_EXT_TREE, sb.len) < 0\n+\t\t\t|| ce_write_v2(&c, newfd, sb.buf, sb.len) < 0;\n \t\tstrbuf_release(&sb);\n \t\tif (err)\n \t\t\treturn -1;\n@@ -1851,9 +1851,9 @@ int write_index(struct index_state *istate, int newfd)\n \t\tstruct strbuf sb = STRBUF_INIT;\n \n \t\tresolve_undo_write(&sb, istate->resolve_undo);\n-\t\terr = write_index_ext_header(&c, newfd, CACHE_EXT_RESOLVE_UNDO,\n+\t\terr = write_index_ext_header_v2(&c, newfd, CACHE_EXT_RESOLVE_UNDO,\n \t\t\t\t\t     sb.len) < 0\n-\t\t\t|| ce_write(&c, newfd, sb.buf, sb.len) < 0;\n+\t\t\t|| ce_write_v2(&c, newfd, sb.buf, sb.len) < 0;\n \t\tstrbuf_release(&sb);\n \t\tif (err)\n \t\t\treturn -1;\n@@ -1866,6 +1866,14 @@ int write_index(struct index_state *istate, int newfd)\n \treturn 0;\n }\n \n+int write_index(struct index_state *istate, int newfd)\n+{\n+\tif (!istate->version)\n+\t\tistate->version = INDEX_FORMAT_DEFAULT;\n+\n+\treturn write_index_v2(istate, newfd);\n+}\n+\n /*\n  * Read the index file that is potentially unmerged into given\n  * index_state, dropping any unmerged entries.  Returns true if\n-- \n1.7.10.886.gdf6792c.dirty\n"},{"id":"196333","messageId":"1343905326-23790-6-git-send-email-t.gummerer@gmail.com","threadId":"31170","inReplyTo":"1343905326-23790-1-git-send-email-t.gummerer@gmail.com","subject":"[PATCH 05/16] t2104: Don't fail when index version is 5","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2012-08-02T11:01:55Z","receivedAt":"2012-08-02T11:01:55Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"The test t2104 currently checks if the index version is correctly\nreduced to 2/increased to 3, when an entry need extended flags,\nor doesn't use them anymore. Since index-v5 doesn't have extended\nflags (the extended flags are part of the normal flags), we simply\nadd a check if the index version is 2/3 (whichever is correct for\nthat test) or 5.\n\nSigned-off-by: Thomas Gummerer <t.gummerer@gmail.com>\n---\n t/t2104-update-index-skip-worktree.sh | 15 +++++++++------\n 1 file changed, 9 insertions(+), 6 deletions(-)\n\ndiff --git a/t/t2104-update-index-skip-worktree.sh b/t/t2104-update-index-skip-worktree.sh\nindex 1d0879b..e66f23e 100755\n--- a/t/t2104-update-index-skip-worktree.sh\n+++ b/t/t2104-update-index-skip-worktree.sh\n@@ -28,8 +28,9 @@ test_expect_success 'setup' '\n \tgit ls-files -t | test_cmp expect.full -\n '\n \n-test_expect_success 'index is at version 2' '\n-\ttest \"$(test-index-version < .git/index)\" = 2\n+test_expect_success 'index is at version 2 or version 5' '\n+\ttest \"$(test-index-version < .git/index)\" = 2 ||\n+\ttest \"$(test-index-version < .git/index)\" = 5\n '\n \n test_expect_success 'update-index --skip-worktree' '\n@@ -37,8 +38,9 @@ test_expect_success 'update-index --skip-worktree' '\n \tgit ls-files -t | test_cmp expect.skip -\n '\n \n-test_expect_success 'index is at version 3 after having some skip-worktree entries' '\n-\ttest \"$(test-index-version < .git/index)\" = 3\n+test_expect_success 'index is at version 3 or version 5 after having some skip-worktree entries' '\n+\ttest \"$(test-index-version < .git/index)\" = 3 ||\n+\ttest \"$(test-index-version < .git/index)\" = 5\n '\n \n test_expect_success 'ls-files -t' '\n@@ -50,8 +52,9 @@ test_expect_success 'update-index --no-skip-worktree' '\n \tgit ls-files -t | test_cmp expect.full -\n '\n \n-test_expect_success 'index version is back to 2 when there is no skip-worktree entry' '\n-\ttest \"$(test-index-version < .git/index)\" = 2\n+test_expect_success 'index version is back to 2 or 5 when there is no skip-worktree entry' '\n+\ttest \"$(test-index-version < .git/index)\" = 2 ||\n+\ttest \"$(test-index-version < .git/index)\" = 5\n '\n \n test_done\n-- \n1.7.10.886.gdf6792c.dirty\n"},{"id":"196334","messageId":"1343905326-23790-7-git-send-email-t.gummerer@gmail.com","threadId":"31170","inReplyTo":"1343905326-23790-1-git-send-email-t.gummerer@gmail.com","subject":"[PATCH 06/16] t3700: sleep for 1 second, to avoid interfering with the racy code","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2012-08-02T11:01:56Z","receivedAt":"2012-08-02T11:01:56Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"The new git racy code uses the mtime of cache-entries to smudge\na racy clean entry, and loads the work, of checking the file-system\nif the entry has really changed, off to the reader. This interferes\nwith this test, because the entry is racily smudged and thus has\nmtime 0. We wait 1 second to avoid smudging the entry and getting\ncorrect test results.\n\nSigned-off-by: Thomas Gummerer <t.gummerer@gmail.com>\n---\n t/t3700-add.sh | 1 +\n 1 file changed, 1 insertion(+)\n\ndiff --git a/t/t3700-add.sh b/t/t3700-add.sh\nindex 874b3a6..4d70805 100755\n--- a/t/t3700-add.sh\n+++ b/t/t3700-add.sh\n@@ -184,6 +184,7 @@ test_expect_success 'git add --refresh with pathspec' '\n \techo >foo && echo >bar && echo >baz &&\n \tgit add foo bar baz && H=$(git rev-parse :foo) && git rm -f foo &&\n \techo \"100644 $H 3\tfoo\" | git update-index --index-info &&\n+\tsleep 1 &&\n \ttest-chmtime -60 bar baz &&\n \t>expect &&\n \tgit add --refresh bar >actual &&\n-- \n1.7.10.886.gdf6792c.dirty\n"},{"id":"196346","messageId":"1343905326-23790-8-git-send-email-t.gummerer@gmail.com","threadId":"31170","inReplyTo":"1343905326-23790-1-git-send-email-t.gummerer@gmail.com","subject":"[PATCH 07/16] Add documentation of the index-v5 file format","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2012-08-02T11:01:57Z","receivedAt":"2012-08-02T11:01:57Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"Add a documentation of the index file format version 5 to\nDocumentation/technical.\n\nSigned-off-by: Thomas Gummerer <t.gummerer@gmail.com>\n---\n Documentation/technical/index-file-format-v5.txt | 281 +++++++++++++++++++++++\n 1 file changed, 281 insertions(+)\n create mode 100644 Documentation/technical/index-file-format-v5.txt\n\ndiff --git a/Documentation/technical/index-file-format-v5.txt b/Documentation/technical/index-file-format-v5.txt\nnew file mode 100644\nindex 0000000..6253e34\n--- /dev/null\n+++ b/Documentation/technical/index-file-format-v5.txt\n@@ -0,0 +1,281 @@\n+GIT index format\n+================\n+\n+== The git index file format\n+\n+   The git index file (.git/index) documents the status of the files\n+     in the git staging area.\n+\n+   The staging area is used for preparing commits, merging, etc.\n+\n+   All binary numbers are in network byte order. Version 5 is described\n+     here.\n+\n+   - A 20-byte header consisting of\n+\n+     sig (32-bits): Signature:\n+       The signature is { 'D', 'I', 'R', 'C' } (stands for \"dircache\")\n+\n+     vnr (32-bits): Version number:\n+       The current supported versions are 2, 3, 4 and 5.\n+\n+     ndir (32-bits): number of directories in the index.\n+\n+     nfile (32-bits): number of file entries in the index.\n+\n+     fblockoffset (32-bits): offset to the file block, relative to the\n+       beginning of the file.\n+\n+   - Offset to the extensions.\n+\n+     nextensions (32-bits): number of extensions.\n+\n+     extoffset (32-bits): offset to the extension. (Possibly none, as\n+       many as indicated in the 4-byte number of extensions)\n+\n+     headercrc (32-bits): crc checksum for the header and extension\n+       offsets\n+\n+   - diroffsets (ndir * directory offsets): A directory offset for each\n+       of the ndir directories in the index, sorted by pathname (of the\n+       directory it's pointing to) (see below). The diroffsets are relative\n+       to the beginning of the direntries block. [1]\n+\n+   - direntries (ndir * directory entries): A directory entry for each\n+       of the ndir directories in the index, sorted by pathname (see\n+       below). [2]\n+\n+   - fileoffsets (nfile * file offsets): A file offset for each of the\n+       nfile files in the index (see below). The file offsets are relative\n+       to the beginning of the fileentries block. [1]\n+\n+   - fileentries (nfile * file entries): A file entry for each of the\n+       nfile files in the index (see below).\n+\n+   - crdata: A number of entries for conflicted data/resolved conflicts\n+       (see below).\n+\n+   - Extensions (Currently none, see below in the future)\n+\n+     Extensions are identified by signature. Optional extensions can\n+     be ignored if GIT does not understand them.\n+\n+     GIT supports an arbitrary number of extension, but currently none\n+     is implemented. [3]\n+\n+     extsig (32-bits): extension signature. If the first byte is 'A'..'Z'\n+     the extension is optional and can be ignored.\n+\n+     extsize (32-bits): size of the extension, excluding the header\n+       (extsig, extsize, extchecksum).\n+\n+     extchecksum (32-bits): crc32 checksum of the extension signature\n+       and size.\n+\n+    - Extension data.\n+\n+\n+== Directory offsets (diroffsets)\n+\n+  diroffset (32-bits): offset to the directory relative to the beginning\n+    of the index file. There are ndir + 1 offsets in the diroffset table,\n+    the last is pointing to the end of the last direntry. With this last\n+    entry, we can replace the strlen when reading each filename, by\n+    calculating its length with the offsets.\n+\n+  This part is needed for making the directory entries bisectable and\n+    thus allowing a binary search.\n+\n+== Directory entry (direntries)\n+  \n+  Directory entries are sorted in lexicographic order by the name \n+    of their path starting with the root.\n+  \n+  pathname (variable length, nul terminated): relative to top level\n+    directory (without the leading slash). '/' is used as path\n+    separator. A string of length 0 ('') indicates the root directory.\n+    The special path components \".\", and \"..\" (without quotes) are\n+    disallowed. The path also includes a trailing slash. [9]\n+\n+  foffset (32-bits): offset to the lexicographically first file in \n+    the file offsets (fileoffsets), relative to the beginning of\n+    the fileoffset block.\n+\n+  cr (32-bits): offset to conflicted/resolved data at the end of the\n+    index. 0 if there is no such data. [4]\n+\n+  ncr (32-bits): number of conflicted/resolved data entries at the\n+    end of the index if the offset is non 0. If cr is 0, ncr is\n+    also 0.\n+\n+  nsubtrees (32-bits): number of subtrees this tree has in the index.\n+\n+  nfiles (32-bits): number of files in the directory, that are in\n+    the index.\n+\n+  nentries (32-bits): number of entries in the index that is covered\n+    by the tree this entry represents. (-1 if the entry is invalid).\n+    This number includes all the files in this tree, recursively.\n+\n+  objname (160-bits): object name for the object that would result\n+    from writing this span of index as a tree. This is only valid\n+    if nentries is valid, meaning the cache-tree is valid.\n+\n+  flags (16-bits): 'flags' field split into (high to low bits) (For\n+    D/F conflicts)\n+    \n+    stage (2-bits): stage of the directory during merge\n+\n+    14-bit unused\n+\n+  dircrc (32-bits): crc32 checksum for each directory entry.\n+\n+  The last 24 bytes (4-byte number of entries + 160-bit object name) are\n+    for the cache tree. An entry can be in an invalidated state which is\n+    represented by having -1 in the entry_count field.\n+\n+  The entries are written out in the top-down, depth-first order. The\n+    first entry represents the root level of the repository, followed by\n+    the first subtree - let's call it A - of the root level, followed by\n+    the first subtree of A, ... There is no prefix compression for\n+    directories.\n+\n+== File offsets (fileoffsets)\n+\n+  fileoffset (32-bits): offset to the file.\n+\n+  This part is needed for making the file entries bisectable and\n+    thus allowing a binary search. There are nfile + 1 offsets in the\n+    fileoffset table, the last is pointing to the end of the last\n+    fileentry. With this last entry, we can replace the strlen when\n+    reading each filename, by calculating its length with the offsets.\n+\n+== File entry (fileentries)\n+  \n+  File entries are sorted in ascending order on the name field, after the\n+  respective offset given by the directory entries. All file names are\n+  prefix compressed, meaning the file name is relative to the directory.\n+\n+  filename (variable length, nul terminated). The exact encoding is \n+    undefined, but the filename cannot contain a NUL byte (iow, the same\n+    encoding as a UNIX pathname).\n+\n+  flags (16-bits): 'flags' field split into (high to low bits)\n+\n+    assumevalid (1-bit): assume-valid flag\n+\n+    intenttoadd (1-bit): intent-to-add flag, used by \"git add -N\".\n+      Extended flag in index v3.\n+\n+    stage (2-bit): stage of the file during merge\n+\n+    skipworktree (1-bit): skip-worktree flag, used by sparse checkout.\n+      Extended flag in index v3.\n+\n+    11-bit unused, must be zero [6]\n+\n+  mode (16-bits): file mode, split into (high to low bits)\n+\n+    objtype (4-bits): object type\n+      valid values in binary are 1000 (regular file), 1010 (symbolic\n+      link) and 1110 (gitlink)\n+\n+    3-bit unused\n+\n+    permission (9-bits): unix permission. Only 0755 and 0644 are valid\n+      for regular files. Symbolic links and gitlinks have value 0 in \n+      this field.\n+\n+  mtimes (32-bits): mtime seconds, the last time a file's data changed\n+    this is stat(2) data\n+\n+  mtimens (32-bits): mtime nanosecond fractions\n+    this is stat(2) data\n+\n+  statcrc (32-bits): crc32 checksum over ctime seconds, ctime\n+    nanoseconds, ino, file size, dev, uid, gid (All stat(2) data\n+    except mtime) [7]\n+\n+  objhash (160-bits): SHA-1 for the represented object\n+\n+# This will probably be changed in future versions as discussed here: http://colabti.org/irclogger/irclogger_log/git-devel?date=2012-06-21\n+  entrycrc (32-bits): crc32 checksum for the file entry. The crc code\n+    includes the offset to the file.\n+\n+== Conflict data\n+\n+  A conflict is represented in the index as a set of higher stage entries.\n+  These entries are stored at the end of the index. When a conflict is \n+  resolved (e.g. with \"git add path\"). A bit is flipped, to indicate that\n+  the conflict is resolved, but the entries will be kept, so that\n+  conflicts can be recreated (e.g. with \"git checkout -m\", in case users\n+  want to redo a conflict resolution from scratch.\n+\n+  The first part of a conflict (usually stage 1) will be stored both in\n+  the entries part of the index and in the conflict part. All other parts\n+  will only be stored in the conflict part.\n+\n+  filename (variable length, nul terminated): filename of the entry,\n+    relative to its containing directory).\n+\n+  nfileconflicts (32-bits): number of conflicts for the file [8]\n+\n+  flags (nfileconflicts * flags) (16-bits): 'flags' field split into:\n+    \n+    conflicted (1-bit): conflicted state (conflicted/resolved) (1 if\n+      conflicted)\n+\n+    stage (2-bits): stage during merge.\n+   \n+    13-bit unused\n+\n+  entry_mode (nfileconflicts * entry mode) (16-bits): octal numbers, entry\n+    mode of eache entry in the different stages. (How many is defined by\n+    the 4-byte number before)\n+\n+  objectnames (nfileconflicts * object names) (160-bits): object names \n+    of the different stages.\n+\n+  conflictcrc (32-bits): crc32 checksum over conflict data.\n+\n+== Design explanations\n+\n+[1] The directory and file offsets are included in the index format\n+    to enable bisectability of the index, for binary searches.Updating\n+    a single entry and partial reading will benefit from this.\n+\n+[2] The directories are saved in their own block, to be able to\n+    quickly search for a directory in the index. They include a\n+    offset to the (lexically) first file in the directory.\n+\n+[3] The data of the cache-tree extension and the resolve undo\n+    extension is now part of the index itself, but if other extensions\n+    come up in the future, there is no need to change the index, they\n+    can simply be added at the end.\n+\n+[4] To avoid rewrites of the whole index when there are conflicts or\n+    conflicts are being resolved, conflicted data will be stored at\n+    the end of the index. To mark the conflict resolved, just a bit\n+    has to be flipped. The data will still be there, if a user wants\n+    to redo the conflict resolution.\n+\n+[5] Since only 4 modes are effectively allowed in git but 32-bit are\n+    used to store them, having a two bit flag for the mode is enough\n+    and saves 4 byte per entry.\n+\n+[6] The length of the file name was dropped, since each file name is\n+    nul terminated anyway.\n+\n+[7] Since all stat data (except mtime and ctime) is just used for\n+    checking if a file has changed a checksum of the data is enough.\n+    In addition to that Thomas Rast suggested ctime could be ditched\n+    completely (core.trustctime=false) and thus included in the\n+    checksum. This would save 24 bytes per index entry, which would\n+    be about 4 MB on the Webkit index.\n+    (Thanks for the suggestion to Michael Haggerty)\n+\n+[8] Since there can be more stage #1 entries, it is necessary to know\n+    the number of conflict data entries there are.\n+\n+[9] As Michael Haggerty pointed out on the mailing list, storing the\n+    trailing slash will simplify a few operations.\n-- \n1.7.10.886.gdf6792c.dirty\n"},{"id":"196340","messageId":"1343905326-23790-9-git-send-email-t.gummerer@gmail.com","threadId":"31170","inReplyTo":"1343905326-23790-1-git-send-email-t.gummerer@gmail.com","subject":"[PATCH 08/16] Make in-memory format aware of stat_crc","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2012-08-02T11:01:58Z","receivedAt":"2012-08-02T11:01:58Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"Make the in-memory format aware of the stat_crc used by index-v5.\nIt is simply ignored by index version prior to v5.\n\nSigned-off-by: Thomas Gummerer <t.gummerer@gmail.com>\n---\n cache.h      |  1 +\n read-cache.c | 27 +++++++++++++++++++++++++++\n 2 files changed, 28 insertions(+)\n\ndiff --git a/cache.h b/cache.h\nindex 3aa70d8..7247944 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -133,6 +133,7 @@ struct cache_entry {\n \tunsigned int ce_flags;\n \tunsigned int ce_namelen;\n \tunsigned char sha1[20];\n+\tuint32_t ce_stat_crc;\n \tstruct cache_entry *next;\n \tstruct cache_entry *dir_next;\n \tchar name[FLEX_ARRAY]; /* more */\ndiff --git a/read-cache.c b/read-cache.c\nindex 1c804e1..dfe5ee2 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -67,6 +67,31 @@ void rename_index_entry_at(struct index_state *istate, int nr, const char *new_n\n \tadd_index_entry(istate, new, ADD_CACHE_OK_TO_ADD|ADD_CACHE_OK_TO_REPLACE);\n }\n \n+static uint32_t calculate_stat_crc(struct cache_entry *ce)\n+{\n+\tunsigned int ctimens = 0;\n+\tuint32_t stat, stat_crc;\n+\n+\tstat = htonl(ce->ce_ctime.sec);\n+\tstat_crc = crc32(0, (Bytef*)&stat, 4);\n+#ifdef USE_NSEC\n+\tctimens = ce->ce_ctime.nsec;\n+#endif\n+\tstat = htonl(ctimens);\n+\tstat_crc = crc32(stat_crc, (Bytef*)&stat, 4);\n+\tstat = htonl(ce->ce_ino);\n+\tstat_crc = crc32(stat_crc, (Bytef*)&stat, 4);\n+\tstat = htonl(ce->ce_size);\n+\tstat_crc = crc32(stat_crc, (Bytef*)&stat, 4);\n+\tstat = htonl(ce->ce_dev);\n+\tstat_crc = crc32(stat_crc, (Bytef*)&stat, 4);\n+\tstat = htonl(ce->ce_uid);\n+\tstat_crc = crc32(stat_crc, (Bytef*)&stat, 4);\n+\tstat = htonl(ce->ce_gid);\n+\tstat_crc = crc32(stat_crc, (Bytef*)&stat, 4);\n+\treturn stat_crc;\n+}\n+\n /*\n  * This only updates the \"non-critical\" parts of the directory\n  * cache, ie the parts that aren't tracked by GIT, and only used\n@@ -89,6 +114,8 @@ void fill_stat_cache_info(struct cache_entry *ce, struct stat *st)\n \n \tif (S_ISREG(st->st_mode))\n \t\tce_mark_uptodate(ce);\n+\n+\tce->ce_stat_crc = calculate_stat_crc(ce);\n }\n \n static int ce_compare_data(struct cache_entry *ce, struct stat *st)\n-- \n1.7.10.886.gdf6792c.dirty\n"},{"id":"196345","messageId":"1343905326-23790-10-git-send-email-t.gummerer@gmail.com","threadId":"31170","inReplyTo":"1343905326-23790-1-git-send-email-t.gummerer@gmail.com","subject":"[PATCH 09/16] Read index-v5","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2012-08-02T11:01:59Z","receivedAt":"2012-08-02T11:01:59Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"Make git read the index file version 5 without complaining.\n\nThis version of the reader doesn't read neither the cache-tree\nnor the resolve undo data, but doesn't choke on an index that\nincludes such data.\n\nSigned-off-by: Thomas Gummerer <t.gummerer@gmail.com>\n---\n cache.h      |  79 +++++++++\n read-cache.c | 570 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-\n 2 files changed, 644 insertions(+), 5 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 7247944..91d9b45 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -108,6 +108,13 @@ struct cache_header_v2 {\n \tunsigned int hdr_entries;\n };\n \n+struct cache_header_v5 {\n+\tunsigned int hdr_ndir;\n+\tunsigned int hdr_nfile;\n+\tunsigned int hdr_fblockoffset;\n+\tunsigned int hdr_nextension;\n+};\n+\n #define INDEX_FORMAT_LB 2\n #define INDEX_FORMAT_UB 4\n \n@@ -121,6 +128,15 @@ struct cache_time {\n \tunsigned int nsec;\n };\n \n+/*\n+ * The *next pointer is used in read_entries_v5 for holding\n+ * all the elements of a directory, and points to the next\n+ * cache_entry in a directory.\n+ *\n+ * It is reset by the add_name_hash call in set_index_entry\n+ * to set it to point to the next cache_entry in the\n+ * correct in-memory format ordering.\n+ */\n struct cache_entry {\n \tstruct cache_time ce_ctime;\n \tstruct cache_time ce_mtime;\n@@ -139,12 +155,58 @@ struct cache_entry {\n \tchar name[FLEX_ARRAY]; /* more */\n };\n \n+struct directory_entry {\n+\tstruct directory_entry *next;\n+\tstruct directory_entry *next_hash;\n+\tstruct cache_entry *ce;\n+\tstruct cache_entry *ce_last;\n+\tstruct conflict_entry *conflict;\n+\tstruct conflict_entry *conflict_last;\n+\tunsigned int conflict_size;\n+\tunsigned int de_foffset;\n+\tunsigned int de_cr;\n+\tunsigned int de_ncr;\n+\tunsigned int de_nsubtrees;\n+\tunsigned int de_nfiles;\n+\tunsigned int de_nentries;\n+\tunsigned char sha1[20];\n+\tunsigned short de_flags;\n+\tunsigned int de_pathlen;\n+\tchar pathname[FLEX_ARRAY];\n+};\n+\n+struct conflict_part {\n+\tstruct conflict_part *next;\n+\tunsigned short flags;\n+\tunsigned short entry_mode;\n+\tunsigned char sha1[20];\n+};\n+\n+struct conflict_entry {\n+\tstruct conflict_entry *next;\n+\tunsigned int nfileconflicts;\n+\tstruct conflict_part *entries;\n+\tunsigned int namelen;\n+\tunsigned int pathlen;\n+\tchar name[FLEX_ARRAY];\n+};\n+\n+struct ondisk_conflict_part {\n+\tunsigned short flags;\n+\tunsigned short entry_mode;\n+\tunsigned char sha1[20];\n+};\n+\n #define CE_NAMEMASK  (0x0fff)\n #define CE_STAGEMASK (0x3000)\n #define CE_EXTENDED  (0x4000)\n #define CE_VALID     (0x8000)\n #define CE_STAGESHIFT 12\n \n+#define CONFLICT_CONFLICTED (0x8000)\n+#define CONFLICT_STAGESHIFT 13\n+#define CONFLICT_STAGEMASK (0x6000)\n+\n /*\n  * Range 0xFFFF0000 in ce_flags is divided into\n  * two parts: in-memory flags and on-disk ones.\n@@ -178,6 +240,18 @@ struct cache_entry {\n #define CE_EXTENDED_FLAGS (CE_INTENT_TO_ADD | CE_SKIP_WORKTREE)\n \n /*\n+ * Representation of the extended on-disk flags in the v5 format.\n+ * They must not collide with the ordinary on-disk flags, and need to\n+ * fit in 16 bits.  Note however that v5 does not save the name\n+ * length.\n+ */\n+#define CE_INTENT_TO_ADD_V5  (0x4000)\n+#define CE_SKIP_WORKTREE_V5  (0x0800)\n+#if (CE_VALID|CE_STAGEMASK) & (CE_INTENTTOADD_V5|CE_SKIPWORKTREE_V5)\n+#error \"v5 on-disk flags collide with ordinary on-disk flags\"\n+#endif\n+\n+/*\n  * Safeguard to avoid saving wrong flags:\n  *  - CE_EXTENDED2 won't get saved until its semantic is known\n  *  - Bits in 0x0000FFFF have been saved in ce_flags already\n@@ -215,6 +289,8 @@ static inline unsigned create_ce_flags(unsigned stage)\n #define ce_skip_worktree(ce) ((ce)->ce_flags & CE_SKIP_WORKTREE)\n #define ce_mark_uptodate(ce) ((ce)->ce_flags |= CE_UPTODATE)\n \n+#define conflict_stage(c) ((CONFLICT_STAGEMASK & (c)->flags) >> CONFLICT_STAGESHIFT)\n+\n #define ce_permissions(mode) (((mode) & 0100) ? 0755 : 0644)\n static inline unsigned int create_ce_mode(unsigned int mode)\n {\n@@ -261,6 +337,8 @@ static inline unsigned int canon_mode(unsigned int mode)\n }\n \n #define cache_entry_size(len) (offsetof(struct cache_entry,name) + (len) + 1)\n+#define directory_entry_size(len) (offsetof(struct directory_entry,pathname) + (len) + 1)\n+#define conflict_entry_size(len) (offsetof(struct conflict_entry,name) + (len) + 1)\n \n struct index_state {\n \tstruct cache_entry **cache;\n@@ -437,6 +515,7 @@ extern int init_db(const char *template_dir, unsigned int flags);\n extern int read_index(struct index_state *);\n extern int read_index_preload(struct index_state *, const char **pathspec);\n extern void read_index_v2(struct index_state *, void *mmap, int);\n+extern void read_index_v5(struct index_state *, void *mmap, int, int);\n extern int read_index_from(struct index_state *, const char *path);\n extern int is_index_unborn(struct index_state *);\n extern int read_index_unmerged(struct index_state *);\ndiff --git a/read-cache.c b/read-cache.c\nindex dfe5ee2..884c2a7 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -234,6 +234,55 @@ static int ce_match_stat_basic_v2(struct cache_entry *ce,\n \treturn changed;\n }\n \n+static int match_stat_crc(struct stat *st, uint32_t expected_crc)\n+{\n+\tuint32_t data, stat_crc = 0;\n+\tunsigned int ctimens = 0;\n+\n+\tdata = htonl(st->st_ctime);\n+\tstat_crc = crc32(0, (Bytef*)&data, 4);\n+#ifdef USE_NSEC\n+\tctimens = ST_MTIME_NSEC(*st);\n+#endif\n+\tdata = htonl(ctimens);\n+\tstat_crc = crc32(stat_crc, (Bytef*)&data, 4);\n+\tdata = htonl(st->st_ino);\n+\tstat_crc = crc32(stat_crc, (Bytef*)&data, 4);\n+\tdata = htonl(st->st_size);\n+\tstat_crc = crc32(stat_crc, (Bytef*)&data, 4);\n+\tdata = htonl(st->st_dev);\n+\tstat_crc = crc32(stat_crc, (Bytef*)&data, 4);\n+\tdata = htonl(st->st_uid);\n+\tstat_crc = crc32(stat_crc, (Bytef*)&data, 4);\n+\tdata = htonl(st->st_gid);\n+\tstat_crc = crc32(stat_crc, (Bytef*)&data, 4);\n+\n+\treturn stat_crc == expected_crc;\n+}\n+\n+static int ce_match_stat_basic_v5(struct cache_entry *ce,\n+\t\t\t\tstruct stat *st,\n+\t\t\t\tint changed)\n+{\n+\n+\tif (ce->ce_mtime.sec != 0 && ce->ce_mtime.sec != (unsigned int)st->st_mtime)\n+\t\tchanged |= MTIME_CHANGED;\n+#ifdef USE_NSEC\n+\tif (ce->ce_mtime.nsec != 0 && ce->ce_mtime.nsec != ST_MTIME_NSEC(*st))\n+\t\tchanged |= MTIME_CHANGED;\n+#endif\n+\tif (!match_stat_crc(st, ce->ce_stat_crc)) {\n+\t\tchanged |= OWNER_CHANGED;\n+\t\tchanged |= INODE_CHANGED;\n+\t}\n+\t/* Racily smudged entry? */\n+\tif (!ce->ce_mtime.sec && !ce->ce_mtime.nsec) {\n+\t\tif (!changed && !is_empty_blob_sha1(ce->sha1) && ce_modified_check_fs(ce, st))\n+\t\t\tchanged |= DATA_CHANGED;\n+\t}\n+\treturn changed;\n+}\n+\n static int ce_match_stat_basic(struct cache_entry *ce, struct stat *st)\n {\n \tunsigned int changed = 0;\n@@ -267,7 +316,10 @@ static int ce_match_stat_basic(struct cache_entry *ce, struct stat *st)\n \t\tdie(\"internal error: ce_mode is %o\", ce->ce_mode);\n \t}\n \n-\tchanged = ce_match_stat_basic_v2(ce, st, changed);\n+\tif (the_index.version != 5)\n+\t\tchanged = ce_match_stat_basic_v2(ce, st, changed);\n+\telse\n+\t\tchanged = ce_match_stat_basic_v5(ce, st, changed);\n \treturn changed;\n }\n \n@@ -1275,6 +1327,25 @@ struct ondisk_cache_entry_extended {\n \tchar name[FLEX_ARRAY]; /* more */\n };\n \n+struct ondisk_cache_entry_v5 {\n+\tunsigned short flags;\n+\tunsigned short mode;\n+\tstruct cache_time mtime;\n+\tint stat_crc;\n+\tunsigned char sha1[20];\n+};\n+\n+struct ondisk_directory_entry {\n+\tunsigned int foffset;\n+\tunsigned int cr;\n+\tunsigned int ncr;\n+\tunsigned int nsubtrees;\n+\tunsigned int nfiles;\n+\tunsigned int nentries;\n+\tunsigned char sha1[20];\n+\tunsigned short flags;\n+};\n+\n /* These are only used for v3 or lower */\n #define align_flex_name(STRUCT,len) ((offsetof(struct STRUCT,name) + (len) + 8) & ~7)\n #define ondisk_cache_entry_size(len) align_flex_name(ondisk_cache_entry,len)\n@@ -1283,6 +1354,17 @@ struct ondisk_cache_entry_extended {\n \t\t\t    ondisk_cache_entry_extended_size(ce_namelen(ce)) : \\\n \t\t\t    ondisk_cache_entry_size(ce_namelen(ce)))\n \n+static int check_crc32(int initialcrc,\n+\t\t\tvoid *data,\n+\t\t\tsize_t len,\n+\t\t\tunsigned int expected_crc)\n+{\n+\tint crc;\n+\n+\tcrc = crc32(initialcrc, (Bytef*)data, len);\n+\treturn crc == expected_crc;\n+}\n+\n static int verify_hdr_version(struct cache_version_header *hdr, unsigned long size)\n {\n \tint hdr_version;\n@@ -1290,7 +1372,7 @@ static int verify_hdr_version(struct cache_version_header *hdr, unsigned long si\n \tif (hdr->hdr_signature != htonl(CACHE_SIGNATURE))\n \t\treturn error(\"bad signature\");\n \thdr_version = ntohl(hdr->hdr_version);\n-\tif (hdr_version < 2 || 4 < hdr_version)\n+\tif (hdr_version < 2 || 5 < hdr_version)\n \t\treturn error(\"bad index version %d\", hdr_version);\n \treturn 0;\n }\n@@ -1308,6 +1390,24 @@ static int verify_hdr_v2(struct cache_version_header *hdr, unsigned long size)\n \treturn 0;\n }\n \n+static int verify_hdr_v5(void *mmap)\n+{\n+\tuint32_t *filecrc;\n+\tunsigned int header_size_v5;\n+\tstruct cache_version_header *hdr;\n+\tstruct cache_header_v5 *hdr_v5;\n+\n+\thdr = mmap;\n+\thdr_v5 = mmap + sizeof(*hdr);\n+\t/* Size of the header + the size of the extensionoffsets */\n+\theader_size_v5 = sizeof(*hdr_v5) + hdr_v5->hdr_nextension * 4;\n+\t/* Initialize crc */\n+\tfilecrc = mmap + sizeof(*hdr) + header_size_v5;\n+\tif (!check_crc32(0, hdr, sizeof(*hdr) + header_size_v5, ntohl(*filecrc)))\n+\t\treturn error(\"bad index file header crc signature\");\n+\treturn 0;\n+}\n+\n static int read_index_extension(struct index_state *istate,\n \t\t\t\tconst char *ext, void *data, unsigned long sz)\n {\n@@ -1372,12 +1472,105 @@ static struct cache_entry *cache_entry_from_ondisk(struct ondisk_cache_entry *on\n \tce->ce_size       = ntoh_l(ondisk->size);\n \tce->ce_flags      = flags & ~CE_NAMEMASK;\n \tce->ce_namelen    = len;\n+\tce->ce_stat_crc   = 0;\n \thashcpy(ce->sha1, ondisk->sha1);\n \tmemcpy(ce->name, name, len);\n \tce->name[len] = '\\0';\n \treturn ce;\n }\n \n+static struct cache_entry *cache_entry_from_ondisk_v5(struct ondisk_cache_entry_v5 *ondisk,\n+\t\t\t\t\t\t   struct directory_entry *de,\n+\t\t\t\t\t\t   char *name,\n+\t\t\t\t\t\t   size_t len,\n+\t\t\t\t\t\t   size_t prefix_len)\n+{\n+\tstruct cache_entry *ce = xmalloc(cache_entry_size(len + de->de_pathlen));\n+\tint flags;\n+\n+\tflags = ntoh_s(ondisk->flags);\n+\tce->ce_ctime.sec  = 0;\n+\tce->ce_mtime.sec  = ntoh_l(ondisk->mtime.sec);\n+\tce->ce_ctime.nsec = 0;\n+\tce->ce_mtime.nsec = ntoh_l(ondisk->mtime.nsec);\n+\tce->ce_dev        = 0;\n+\tce->ce_ino        = 0;\n+\tce->ce_mode       = ntoh_s(ondisk->mode);\n+\tce->ce_uid        = 0;\n+\tce->ce_gid        = 0;\n+\tce->ce_size       = 0;\n+\tce->ce_flags      = flags & CE_STAGEMASK;\n+\tce->ce_flags     |= flags & CE_VALID;\n+\tif (flags & CE_INTENT_TO_ADD_V5)\n+\t\tce->ce_flags |= CE_INTENT_TO_ADD;\n+\tif (flags & CE_SKIP_WORKTREE_V5)\n+\t\tce->ce_flags |= CE_SKIP_WORKTREE;\n+\tce->ce_stat_crc   = ntoh_l(ondisk->stat_crc);\n+\tce->ce_namelen    = len + de->de_pathlen;\n+\thashcpy(ce->sha1, ondisk->sha1);\n+\tmemcpy(ce->name, de->pathname, de->de_pathlen);\n+\tmemcpy(ce->name + de->de_pathlen, name, len);\n+\tce->name[len + de->de_pathlen] = '\\0';\n+\treturn ce;\n+}\n+\n+static struct directory_entry *directory_entry_from_ondisk(struct ondisk_directory_entry *ondisk,\n+\t\t\t\t\t\t   const char *name,\n+\t\t\t\t\t\t   size_t len)\n+{\n+\tstruct directory_entry *de = xmalloc(directory_entry_size(len));\n+\n+\n+\tmemcpy(de->pathname, name, len);\n+\tde->pathname[len] = '\\0';\n+\tde->de_flags      = ntoh_s(ondisk->flags);\n+\tde->de_foffset    = ntoh_l(ondisk->foffset);\n+\tde->de_cr         = ntoh_l(ondisk->cr);\n+\tde->de_ncr        = ntoh_l(ondisk->ncr);\n+\tde->de_nsubtrees  = ntoh_l(ondisk->nsubtrees);\n+\tde->de_nfiles     = ntoh_l(ondisk->nfiles);\n+\tde->de_nentries   = ntoh_l(ondisk->nentries);\n+\tde->de_pathlen    = len;\n+\thashcpy(de->sha1, ondisk->sha1);\n+\treturn de;\n+}\n+\n+static struct conflict_part *conflict_part_from_ondisk(struct ondisk_conflict_part *ondisk)\n+{\n+\tstruct conflict_part *cp = xmalloc(sizeof(struct conflict_part));\n+\n+\tcp->flags      = ntoh_s(ondisk->flags);\n+\tcp->entry_mode = ntoh_s(ondisk->entry_mode);\n+\thashcpy(cp->sha1, ondisk->sha1);\n+\treturn cp;\n+}\n+\n+static struct cache_entry *convert_conflict_part(struct conflict_part *cp,\n+\t\t\t\t\t\tchar * name,\n+\t\t\t\t\t\tunsigned int len)\n+{\n+\n+\tstruct cache_entry *ce = xmalloc(cache_entry_size(len));\n+\n+\tce->ce_ctime.sec  = 0;\n+\tce->ce_mtime.sec  = 0;\n+\tce->ce_ctime.nsec = 0;\n+\tce->ce_mtime.nsec = 0;\n+\tce->ce_dev        = 0;\n+\tce->ce_ino        = 0;\n+\tce->ce_mode       = cp->entry_mode;\n+\tce->ce_uid        = 0;\n+\tce->ce_gid        = 0;\n+\tce->ce_size       = 0;\n+\tce->ce_flags      = conflict_stage(cp) << CE_STAGESHIFT;\n+\tce->ce_stat_crc   = 0;\n+\tce->ce_namelen    = len;\n+\thashcpy(ce->sha1, cp->sha1);\n+\tmemcpy(ce->name, name, len);\n+\tce->name[len] = '\\0';\n+\treturn ce;\n+}\n+\n /*\n  * Adjacent cache entries tend to share the leading paths, so it makes\n  * sense to only store the differences in later entries.  In the v4\n@@ -1445,6 +1638,332 @@ static struct cache_entry *create_from_disk(struct ondisk_cache_entry *ondisk,\n \treturn ce;\n }\n \n+static struct directory_entry *read_directories_v5(unsigned int *dir_offset,\n+\t\t\t\tunsigned int *dir_table_offset,\n+\t\t\t\tvoid *mmap,\n+\t\t\t\tint mmap_size)\n+{\n+\tint i, ondisk_directory_size;\n+\tuint32_t *filecrc, *beginning, *end;\n+\tstruct directory_entry *current = NULL;\n+\tstruct ondisk_directory_entry *disk_de;\n+\tstruct directory_entry *de;\n+\tunsigned int data_len, len;\n+\tchar *name;\n+\n+\tondisk_directory_size = sizeof(disk_de->flags)\n+\t\t+ sizeof(disk_de->foffset)\n+\t\t+ sizeof(disk_de->cr)\n+\t\t+ sizeof(disk_de->ncr)\n+\t\t+ sizeof(disk_de->nsubtrees)\n+\t\t+ sizeof(disk_de->nfiles)\n+\t\t+ sizeof(disk_de->nentries)\n+\t\t+ sizeof(disk_de->sha1);\n+\tname = (char *)mmap + *dir_offset;\n+\tbeginning = mmap + *dir_table_offset;\n+\tend = mmap + *dir_table_offset + 4;\n+\tlen = ntoh_l(*end) - ntoh_l(*beginning) - ondisk_directory_size - 5;\n+\tdisk_de = (struct ondisk_directory_entry *)\n+\t\t\t((char *)mmap + *dir_offset + len + 1);\n+\tde = directory_entry_from_ondisk(disk_de, name, len);\n+\tde->next = NULL;\n+\n+\t/* Length of pathname + nul byte for termination + size of\n+\t * members of ondisk_directory_entry. (Just using the size\n+\t * of the stuct doesn't work, because there may be padding\n+\t * bytes for the struct)\n+\t */\n+\tdata_len = len + 1 + ondisk_directory_size;\n+\n+\tfilecrc = mmap + *dir_offset + data_len;\n+\tif (!check_crc32(0, mmap + *dir_offset, data_len, ntoh_l(*filecrc)))\n+\t\tgoto unmap;\n+\n+\t*dir_table_offset += 4;\n+\t*dir_offset += data_len + 4; /* crc code */\n+\n+\tcurrent = de;\n+\tfor (i = 0; i < de->de_nsubtrees; i++) {\n+\t\tcurrent->next = read_directories_v5(dir_offset, dir_table_offset,\n+\t\t\t\t\t\tmmap, mmap_size);\n+\t\twhile (current->next)\n+\t\t\tcurrent = current->next;\n+\t}\n+\n+\treturn de;\n+unmap:\n+\tmunmap(mmap, mmap_size);\n+\tdie(\"directory crc doesn't match for '%s'\", de->pathname);\n+}\n+\n+static struct cache_entry *read_entry_v5(struct directory_entry *de,\n+\t\t\tunsigned long *entry_offset,\n+\t\t\tvoid **mmap,\n+\t\t\tunsigned long mmap_size,\n+\t\t\tunsigned int *foffsetblock,\n+\t\t\tint fd)\n+{\n+\tint len, crc_wrong, i = 0, offset_to_offset;\n+\tchar *name;\n+\tuint32_t foffsetblockcrc;\n+\tuint32_t *filecrc, *beginning, *end;\n+\tstruct cache_entry *ce;\n+\tstruct ondisk_cache_entry_v5 *disk_ce;\n+\n+\tdo {\n+\t\tname = (char *)*mmap + *entry_offset;\n+\t\tbeginning = *mmap + *foffsetblock;\n+\t\tend = *mmap + *foffsetblock + 4;\n+\t\tlen = ntoh_l(*end) - ntoh_l(*beginning) - sizeof(struct ondisk_cache_entry_v5) - 5;\n+\t\tdisk_ce = (struct ondisk_cache_entry_v5 *)\n+\t\t\t\t((char *)*mmap + *entry_offset + len + 1);\n+\t\tce = cache_entry_from_ondisk_v5(disk_ce, de, name, len, de->de_pathlen);\n+\t\tfilecrc = *mmap + *entry_offset + len + 1 + sizeof(*disk_ce);\n+\t\toffset_to_offset = htonl(*foffsetblock);\n+\t\tfoffsetblockcrc = crc32(0, (Bytef*)&offset_to_offset, 4);\n+\t\tcrc_wrong = !check_crc32(foffsetblockcrc,\n+\t\t\t*mmap + *entry_offset, len + 1 + sizeof(*disk_ce),\n+\t\t\tntoh_l(*filecrc));\n+\t\tif (crc_wrong) {\n+\t\t\t/* wait for 10 milliseconds */\n+\t\t\tusleep(10*1000);\n+\t\t\tmunmap(*mmap, mmap_size);\n+\t\t\t*mmap = xmmap(NULL, mmap_size, PROT_READ | PROT_WRITE, MAP_PRIVATE, fd, 0);\n+\t\t}\n+\t\ti++;\n+\t\t/*\n+\t\t * Retry for 500 ms maximum, before giving up and saying the\n+\t\t * checksum is wrong.\n+\t\t */\n+\t} while (crc_wrong && i < 50);\n+\tif (crc_wrong)\n+\t\tgoto unmap;\n+\t*entry_offset += len + 1 + sizeof(*disk_ce) + 4;\n+\treturn ce;\n+unmap:\n+\tmunmap(*mmap, mmap_size);\n+\tdie(\"file crc doesn't match for '%s'\", ce->name);\n+}\n+\n+static void ce_queue_push(struct cache_entry **head,\n+\t\t\t     struct cache_entry **tail,\n+\t\t\t     struct cache_entry *ce)\n+{\n+\tif (!*head) {\n+\t\t*head = *tail = ce;\n+\t\t(*tail)->next = NULL;\n+\t\treturn;\n+\t}\n+\n+\t(*tail)->next = ce;\n+\tce->next = NULL;\n+\t*tail = (*tail)->next;\n+}\n+\n+static void conflict_entry_push(struct conflict_entry **head,\n+\t\t\t\tstruct conflict_entry **tail,\n+\t\t\t\tstruct conflict_entry *conflict_entry)\n+{\n+\tif (!*head) {\n+\t\t*head = *tail = conflict_entry;\n+\t\t(*tail)->next = NULL;\n+\t\treturn;\n+\t}\n+\n+\t(*tail)->next = conflict_entry;\n+\tconflict_entry->next = NULL;\n+\t*tail = (*tail)->next;\n+}\n+\n+static struct cache_entry *ce_queue_pop(struct cache_entry **head)\n+{\n+\tstruct cache_entry *ce;\n+\n+\tce = *head;\n+\t*head = (*head)->next;\n+\treturn ce;\n+}\n+\n+static void conflict_part_head_remove(struct conflict_part **head)\n+{\n+\tstruct conflict_part *to_free;\n+\n+\tto_free = *head;\n+\t*head = (*head)->next;\n+\tfree(to_free);\n+}\n+\n+static void conflict_entry_head_remove(struct conflict_entry **head)\n+{\n+\tstruct conflict_entry *to_free;\n+\n+\tto_free = *head;\n+\t*head = (*head)->next;\n+\tfree(to_free);\n+}\n+\n+struct conflict_entry *create_new_conflict(char *name, int len, int pathlen)\n+{\n+\tstruct conflict_entry *conflict_entry;\n+\n+\tif (pathlen)\n+\t\tpathlen++;\n+\tconflict_entry = xmalloc(conflict_entry_size(len));\n+\tconflict_entry->entries = NULL;\n+\tconflict_entry->nfileconflicts = 0;\n+\tconflict_entry->namelen = len;\n+\tmemcpy(conflict_entry->name, name, len);\n+\tconflict_entry->name[len] = '\\0';\n+\tconflict_entry->pathlen = pathlen;\n+\tconflict_entry->next = NULL;\n+\n+\treturn conflict_entry;\n+}\n+\n+static struct conflict_entry *create_conflict_entry_from_ce(struct cache_entry *ce,\n+\t\t\t\t\t\t\t\tint pathlen)\n+{\n+\treturn create_new_conflict(ce->name, ce_namelen(ce), pathlen);\n+}\n+\n+static struct conflict_entry *read_conflicts_v5(struct directory_entry *de,\n+\t\t\t\t\t\tvoid **mmap,\n+\t\t\t\t\t\tunsigned long mmap_size,\n+\t\t\t\t\t\tint fd)\n+{\n+\tstruct conflict_entry *head, *tail;\n+\tunsigned int croffset, i, j = 0;\n+\tchar *full_name;\n+\n+\tcroffset = de->de_cr;\n+\ttail = NULL;\n+\thead = NULL;\n+\tfor (i = 0; i < de->de_ncr; i++) {\n+\t\tstruct conflict_entry *conflict_new;\n+\t\tunsigned int len, *nfileconflicts;\n+\t\tchar *name;\n+\t\tvoid *crc_start;\n+\t\tint k, offset, crc_wrong;\n+\t\tuint32_t *filecrc;\n+\n+\t\tdo {\n+\t\t\toffset = croffset;\n+\t\t\tcrc_start = *mmap + offset;\n+\t\t\tname = (char *)*mmap + offset;\n+\t\t\tlen = strlen(name);\n+\t\t\toffset += len + 1;\n+\t\t\tnfileconflicts = *mmap + offset;\n+\t\t\toffset += 4;\n+\n+\t\t\tfull_name = xmalloc(sizeof(char) * (len + de->de_pathlen));\n+\t\t\tmemcpy(full_name, de->pathname, de->de_pathlen);\n+\t\t\tmemcpy(full_name + de->de_pathlen, name, len);\n+\t\t\tconflict_new = create_new_conflict(full_name,\n+\t\t\t\t\tlen + de->de_pathlen, de->de_pathlen);\n+\t\t\tfor (k = 0; k < ntoh_l(*nfileconflicts); k++) {\n+\t\t\t\tstruct ondisk_conflict_part *ondisk;\n+\t\t\t\tstruct conflict_part *cp;\n+\n+\t\t\t\tondisk = *mmap + offset;\n+\t\t\t\tcp = conflict_part_from_ondisk(ondisk);\n+\t\t\t\tcp->next = NULL;\n+\t\t\t\tadd_part_to_conflict_entry(de, conflict_new, cp);\n+\t\t\t\toffset += sizeof(struct ondisk_conflict_part);\n+\t\t\t}\n+\t\t\tfilecrc = *mmap + offset;\n+\t\t\tcrc_wrong = !check_crc32(0, crc_start,\n+\t\t\t\tlen + 1 + 4 + conflict_new->nfileconflicts\n+\t\t\t\t* sizeof(struct ondisk_conflict_part),\n+\t\t\t\tntoh_l(*filecrc));\n+\t\t\tif (crc_wrong) {\n+\t\t\t\t/* wait for 10 milliseconds */\n+\t\t\t\tusleep(10*1000);\n+\t\t\t\tmunmap(*mmap, mmap_size);\n+\t\t\t\t*mmap = xmmap(NULL, mmap_size, PROT_READ | PROT_WRITE, MAP_PRIVATE, fd, 0);\n+\t\t\t}\n+\t\t\tfree(full_name);\n+\t\t\tj++;\n+\t\t} while (crc_wrong && j < 50);\n+\t\tif (crc_wrong)\n+\t\t\tgoto unmap;\n+\t\tcroffset = offset + 4;\n+\t\tconflict_entry_push(&head, &tail, conflict_new);\n+\t}\n+\treturn head;\n+unmap:\n+\tmunmap(*mmap, mmap_size);\n+\tdie(\"wrong crc for conflict: %s\", full_name);\n+}\n+\n+static struct directory_entry *read_entries_v5(struct index_state *istate,\n+\t\t\t\t\tstruct directory_entry *de,\n+\t\t\t\t\tunsigned long *entry_offset,\n+\t\t\t\t\tvoid **mmap,\n+\t\t\t\t\tunsigned long mmap_size,\n+\t\t\t\t\tint *nr,\n+\t\t\t\t\tunsigned int *foffsetblock,\n+\t\t\t\t\tint fd)\n+{\n+\tstruct cache_entry *head = NULL, *tail = NULL;\n+\tstruct conflict_entry *conflict_queue;\n+\tstruct cache_entry *ce;\n+\tint i;\n+\n+\tconflict_queue = read_conflicts_v5(de, mmap, mmap_size, fd);\n+\tfor (i = 0; i < de->de_nfiles; i++) {\n+\t\tce = read_entry_v5(de,\n+\t\t\t\tentry_offset,\n+\t\t\t\tmmap,\n+\t\t\t\tmmap_size,\n+\t\t\t\tfoffsetblock,\n+\t\t\t\tfd);\n+\t\tce_queue_push(&head, &tail, ce);\n+\t\t*foffsetblock += 4;\n+\n+\t\t/* Add the conflicted entries at the end of the index file\n+\t\t * to the in memory format\n+\t\t */\n+\t\tif (conflict_queue &&\n+\t\t    (conflict_queue->entries->flags & CONFLICT_CONFLICTED) != 0 &&\n+\t\t    !cache_name_compare(conflict_queue->name, conflict_queue->namelen,\n+\t\t\t\t\tce->name, ce_namelen(ce))) {\n+\t\t\tstruct conflict_part *cp;\n+\t\t\tcp = conflict_queue->entries;\n+\t\t\tcp = cp->next;\n+\t\t\twhile (cp) {\n+\t\t\t\tce = convert_conflict_part(cp,\n+\t\t\t\t\t\tconflict_queue->name,\n+\t\t\t\t\t\tconflict_queue->namelen);\n+\t\t\t\tce_queue_push(&head, &tail, ce);\n+\t\t\t\tconflict_part_head_remove(&cp);\n+\t\t\t}\n+\t\t\tconflict_entry_head_remove(&conflict_queue);\n+\t\t}\n+\t}\n+\n+\tde = de->next;\n+\n+\twhile (head) {\n+\t\tif (de != NULL\n+\t\t    && strcmp(head->name, de->pathname) > 0) {\n+\t\t\tde = read_entries_v5(istate,\n+\t\t\t\t\tde,\n+\t\t\t\t\tentry_offset,\n+\t\t\t\t\tmmap,\n+\t\t\t\t\tmmap_size,\n+\t\t\t\t\tnr,\n+\t\t\t\t\tfoffsetblock,\n+\t\t\t\t\tfd);\n+\t\t} else {\n+\t\t\tce = ce_queue_pop(&head);\n+\t\t\tset_index_entry(istate, *nr, ce);\n+\t\t\t(*nr)++;\n+\t\t}\n+\t}\n+\n+\treturn de;\n+}\n+\n void read_index_v2(struct index_state *istate, void *mmap, int mmap_size)\n {\n \tint i;\n@@ -1503,6 +2022,39 @@ unmap:\n \tdie(\"index file corrupt\");\n }\n \n+void read_index_v5(struct index_state *istate, void *mmap, int mmap_size, int fd)\n+{\n+\tunsigned long entry_offset;\n+\tunsigned int dir_offset, dir_table_offset;\n+\tstruct cache_version_header *hdr;\n+\tstruct cache_header_v5 *hdr_v5;\n+\tstruct directory_entry *root_directory, *de;\n+\tint nr;\n+\tunsigned int foffsetblock;\n+\n+\thdr = mmap;\n+\thdr_v5 = mmap + sizeof(*hdr);\n+\tistate->version = ntohl(hdr->hdr_version);\n+\tistate->cache_nr = ntohl(hdr_v5->hdr_nfile);\n+\tistate->cache_alloc = alloc_nr(istate->cache_nr);\n+\tistate->cache = xcalloc(istate->cache_alloc, sizeof(struct cache_entry *));\n+\tistate->initialized = 1;\n+\n+\t/* Skip size of the header + crc sum + size of offsets */\n+\tdir_offset = sizeof(*hdr) + sizeof(*hdr_v5) + 4 + (ntohl(hdr_v5->hdr_ndir) + 1) * 4;\n+\tdir_table_offset = sizeof(*hdr) + sizeof(*hdr_v5) + 4;\n+\troot_directory = read_directories_v5(&dir_offset, &dir_table_offset, mmap, mmap_size);\n+\n+\tentry_offset = ntohl(hdr_v5->hdr_fblockoffset);\n+\n+\tnr = 0;\n+\tfoffsetblock = dir_offset;\n+\tde = root_directory;\n+\twhile (de)\n+\t\tde = read_entries_v5(istate, de, &entry_offset,\n+\t\t\t\t&mmap, mmap_size, &nr, &foffsetblock, fd);\n+}\n+\n /* remember to discard_cache() before reading a different cache! */\n int read_index_from(struct index_state *istate, const char *path)\n {\n@@ -1542,10 +2094,18 @@ int read_index_from(struct index_state *istate, const char *path)\n \tif (verify_hdr_version(hdr, mmap_size) < 0)\n \t\tgoto unmap;\n \n-\tif (verify_hdr_v2(hdr, mmap_size) < 0)\n-\t\tgoto unmap;\n+\tif (htonl(hdr->hdr_version) != 5) {\n+\t\tif (verify_hdr_v2(hdr, mmap_size) < 0)\n+\t\t\tgoto unmap;\n \n-\tread_index_v2(istate, mmap, mmap_size);\n+\t\tread_index_v2(istate, mmap, mmap_size);\n+\t} else {\n+\t\tif (verify_hdr_v5(hdr) < 0)\n+\t\t\tgoto unmap;\n+\n+\t\tread_index_v5(istate, mmap, mmap_size, fd);\n+\t}\n+\tclose(fd);\n \tistate->timestamp.sec = st.st_mtime;\n \tistate->timestamp.nsec = ST_MTIME_NSEC(st);\n \n-- \n1.7.10.886.gdf6792c.dirty\n"},{"id":"196344","messageId":"1343905326-23790-11-git-send-email-t.gummerer@gmail.com","threadId":"31170","inReplyTo":"1343905326-23790-1-git-send-email-t.gummerer@gmail.com","subject":"[PATCH 10/16] Read resolve-undo data","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2012-08-02T11:02:00Z","receivedAt":"2012-08-02T11:02:00Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"Make git read the resolve-undo data from the index.\n\nSince the resolve-undo data is joined with the conflicts in\nthe ondisk format of the index file version 5, conflicts and\nresolved data is read at the same time, and the resolve-undo\ndata is then converted to the in-memory format.\n\nSigned-off-by: Thomas Gummerer <t.gummerer@gmail.com>\n---\n read-cache.c   |  1 +\n resolve-undo.c | 36 ++++++++++++++++++++++++++++++++++++\n resolve-undo.h |  2 ++\n 3 files changed, 39 insertions(+)\n\ndiff --git a/read-cache.c b/read-cache.c\nindex 884c2a7..cef9a4e 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -1910,6 +1910,7 @@ static struct directory_entry *read_entries_v5(struct index_state *istate,\n \tint i;\n \n \tconflict_queue = read_conflicts_v5(de, mmap, mmap_size, fd);\n+\tresolve_undo_convert_v5(istate, conflict_queue);\n \tfor (i = 0; i < de->de_nfiles; i++) {\n \t\tce = read_entry_v5(de,\n \t\t\t\tentry_offset,\ndiff --git a/resolve-undo.c b/resolve-undo.c\nindex 72b4612..f96c6ba 100644\n--- a/resolve-undo.c\n+++ b/resolve-undo.c\n@@ -170,3 +170,39 @@ void unmerge_index(struct index_state *istate, const char **pathspec)\n \t\ti = unmerge_index_entry_at(istate, i);\n \t}\n }\n+\n+void resolve_undo_convert_v5(struct index_state *istate,\n+\t\t\t\t\tstruct conflict_entry *ce)\n+{\n+\tint i;\n+\n+\twhile (ce) {\n+\t\tstruct string_list_item *lost;\n+\t\tstruct resolve_undo_info *ui;\n+\t\tstruct conflict_part *cp;\n+\n+\t\tif (ce->entries && (ce->entries->flags & CONFLICT_CONFLICTED) != 0) {\n+\t\t\tce = ce->next;\n+\t\t\tcontinue;\n+\t\t}\n+\t\tif (!istate->resolve_undo) {\n+\t\t\tistate->resolve_undo = xcalloc(1, sizeof(struct string_list));\n+\t\t\tistate->resolve_undo->strdup_strings = 1;\n+\t\t}\n+\n+\t\tlost = string_list_insert(istate->resolve_undo, ce->name);\n+\t\tif (!lost->util)\n+\t\t\tlost->util = xcalloc(1, sizeof(*ui));\n+\t\tui = lost->util;\n+\n+\t\tcp = ce->entries;\n+\t\tfor (i = 0; i < 3; i++)\n+\t\t\tui->mode[i] = 0;\n+\t\twhile (cp) {\n+\t\t\tui->mode[conflict_stage(cp) - 1] = cp->entry_mode;\n+\t\t\thashcpy(ui->sha1[conflict_stage(cp) - 1], cp->sha1);\n+\t\t\tcp = cp->next;\n+\t\t}\n+\t\tce = ce->next;\n+\t}\n+}\ndiff --git a/resolve-undo.h b/resolve-undo.h\nindex 8458769..ab660a6 100644\n--- a/resolve-undo.h\n+++ b/resolve-undo.h\n@@ -13,4 +13,6 @@ extern void resolve_undo_clear_index(struct index_state *);\n extern int unmerge_index_entry_at(struct index_state *, int);\n extern void unmerge_index(struct index_state *, const char **);\n \n+extern void resolve_undo_convert_v5(struct index_state *, struct conflict_entry *);\n+\n #endif\n-- \n1.7.10.886.gdf6792c.dirty\n"},{"id":"196337","messageId":"1343905326-23790-12-git-send-email-t.gummerer@gmail.com","threadId":"31170","inReplyTo":"1343905326-23790-1-git-send-email-t.gummerer@gmail.com","subject":"[PATCH 11/16] Read cache-tree in index-v5","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2012-08-02T11:02:01Z","receivedAt":"2012-08-02T11:02:01Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"Since the cache-tree data is saved as part of the directory data,\nwe have already read it, when we want to read the cache-tree. The\ncache-tree then only has to be converted from the directory data.\n\nThe cache-tree isn't lexically sorted, but after the pathlen at\neach level, therefore the directories have to be reordered with\nrespect to the ondisk layout.\n\nSigned-off-by: Thomas Gummerer <t.gummerer@gmail.com>\n---\n cache-tree.c | 93 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n cache-tree.h |  6 ++++\n read-cache.c |  1 +\n 3 files changed, 100 insertions(+)\n\ndiff --git a/cache-tree.c b/cache-tree.c\nindex 28ed657..6a314aa 100644\n--- a/cache-tree.c\n+++ b/cache-tree.c\n@@ -519,6 +519,99 @@ struct cache_tree *cache_tree_read(const char *buffer, unsigned long size)\n \treturn read_one(&buffer, &size);\n }\n \n+struct cache_tree *convert_one(struct directory_queue *queue, int dirnr)\n+{\n+\tint i, subtree_nr;\n+\tstruct cache_tree *it;\n+\tstruct directory_queue *down;\n+\n+\tit = cache_tree();\n+\tit->entry_count = queue[dirnr].de->de_nentries;\n+\tsubtree_nr = queue[dirnr].de->de_nsubtrees;\n+\tif (0 <= it->entry_count)\n+\t\thashcpy(it->sha1, queue[dirnr].de->sha1);\n+\n+\t/*\n+\t* Just a heuristic -- we do not add directories that often but\n+\t* we do not want to have to extend it immediately when we do,\n+\t* hence +2.\n+\t*/\n+\tit->subtree_alloc = subtree_nr + 2;\n+\tit->down = xcalloc(it->subtree_alloc, sizeof(struct cache_tree_sub *));\n+\tdown = queue[dirnr].down;\n+\tfor (i = 0; i < subtree_nr; i++) {\n+\t\tstruct cache_tree *sub;\n+\t\tstruct cache_tree_sub *subtree;\n+\t\tchar *buf, *name;\n+\n+\t\tname = \"\";\n+\t\tbuf = strtok(down[i].de->pathname, \"/\");\n+\t\twhile (buf) {\n+\t\t\tname = buf;\n+\t\t\tbuf = strtok(NULL, \"/\");\n+\t\t}\n+\t\tsub = convert_one(down, i);\n+\t\tif(!sub)\n+\t\t\tgoto free_return;\n+\t\tsubtree = cache_tree_sub(it, name);\n+\t\tsubtree->cache_tree = sub;\n+\t}\n+\tif (subtree_nr != it->subtree_nr)\n+\t\tdie(\"cache-tree: internal error\");\n+\treturn it;\n+ free_return:\n+\tcache_tree_free(&it);\n+\treturn NULL;\n+}\n+\n+static int compare_cache_tree_elements(const void *a, const void *b)\n+{\n+\tconst struct directory_entry *de1, *de2;\n+\n+\tde1 = ((const struct directory_queue *)a)->de;\n+\tde2 = ((const struct directory_queue *)b)->de;\n+\treturn subtree_name_cmp(de1->pathname, de1->de_pathlen,\n+\t\t\t\tde2->pathname, de2->de_pathlen);\n+}\n+\n+static struct directory_entry *sort_directories(struct directory_entry *de,\n+\t\t\t\t\t\tstruct directory_queue *queue)\n+{\n+\tint i, nsubtrees;\n+\n+\tnsubtrees = de->de_nsubtrees;\n+\tfor (i = 0; i < nsubtrees; i++) {\n+\t\tstruct directory_entry *new_de;\n+\t\tde = de->next;\n+\t\tnew_de = xmalloc(directory_entry_size(de->de_pathlen));\n+\t\tmemcpy(new_de, de, directory_entry_size(de->de_pathlen));\n+\t\tqueue[i].de = new_de;\n+\t\tif (de->de_nsubtrees) {\n+\t\t\tqueue[i].down = xcalloc(de->de_nsubtrees,\n+\t\t\t\t\tsizeof(struct directory_queue));\n+\t\t\tde = sort_directories(de,\n+\t\t\t\t\tqueue[i].down);\n+\t\t}\n+\t}\n+\tqsort(queue, nsubtrees, sizeof(struct directory_queue),\n+\t\t\tcompare_cache_tree_elements);\n+\treturn de;\n+}\n+\n+struct cache_tree *cache_tree_convert_v5(struct directory_entry *de)\n+{\n+\tstruct directory_queue *queue;\n+\n+\tif (!de->de_nentries)\n+\t\treturn NULL;\n+\tqueue = xcalloc(1, sizeof(struct directory_queue));\n+\tqueue[0].de = de;\n+\tqueue[0].down = xcalloc(de->de_nsubtrees, sizeof(struct directory_queue));\n+\n+\tsort_directories(de, queue[0].down);\n+\treturn convert_one(queue, 0);\n+}\n+\n static struct cache_tree *cache_tree_find(struct cache_tree *it, const char *path)\n {\n \tif (!it)\ndiff --git a/cache-tree.h b/cache-tree.h\nindex d8cb2e9..f4131a6 100644\n--- a/cache-tree.h\n+++ b/cache-tree.h\n@@ -20,6 +20,11 @@ struct cache_tree {\n \tstruct cache_tree_sub **down;\n };\n \n+struct directory_queue {\n+\tstruct directory_queue *down;\n+\tstruct directory_entry *de;\n+};\n+\n struct cache_tree *cache_tree(void);\n void cache_tree_free(struct cache_tree **);\n void cache_tree_invalidate_path(struct cache_tree *, const char *);\n@@ -27,6 +32,7 @@ struct cache_tree_sub *cache_tree_sub(struct cache_tree *, const char *);\n \n void cache_tree_write(struct strbuf *, struct cache_tree *root);\n struct cache_tree *cache_tree_read(const char *buffer, unsigned long size);\n+struct cache_tree *cache_tree_convert_v5(struct directory_entry *de);\n \n int cache_tree_fully_valid(struct cache_tree *);\n int cache_tree_update(struct cache_tree *, struct cache_entry **, int, int);\ndiff --git a/read-cache.c b/read-cache.c\nindex cef9a4e..fd095ec 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -2054,6 +2054,7 @@ void read_index_v5(struct index_state *istate, void *mmap, int mmap_size, int fd\n \twhile (de)\n \t\tde = read_entries_v5(istate, de, &entry_offset,\n \t\t\t\t&mmap, mmap_size, &nr, &foffsetblock, fd);\n+\tistate->cache_tree = cache_tree_convert_v5(root_directory);\n }\n \n /* remember to discard_cache() before reading a different cache! */\n-- \n1.7.10.886.gdf6792c.dirty\n"},{"id":"196342","messageId":"1343905326-23790-13-git-send-email-t.gummerer@gmail.com","threadId":"31170","inReplyTo":"1343905326-23790-1-git-send-email-t.gummerer@gmail.com","subject":"[PATCH 12/16] Write index-v5","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2012-08-02T11:02:02Z","receivedAt":"2012-08-02T11:02:02Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"Write the index version 5 file format to disk. This version doesn't\nwrite the cache-tree data and resolve-undo data to the file.\n\nThe main work is done when filtering out the directories from the\ncurrent in-memory format, where in the same turn also the conflicts\nand the file data is calculated.\n\nSigned-off-by: Thomas Gummerer <t.gummerer@gmail.com>\n---\n cache.h      |  10 +-\n read-cache.c | 602 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-\n 2 files changed, 609 insertions(+), 3 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 91d9b45..fe3b446 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -116,7 +116,7 @@ struct cache_header_v5 {\n };\n \n #define INDEX_FORMAT_LB 2\n-#define INDEX_FORMAT_UB 4\n+#define INDEX_FORMAT_UB 5\n \n /*\n  * The \"cache_time\" is just the low 32 bits of the\n@@ -526,6 +526,7 @@ extern int verify_path(const char *path);\n extern struct cache_entry *index_name_exists(struct index_state *istate, const char *name, int namelen, int igncase);\n extern int index_name_stage_pos(const struct index_state *, const char *name, int namelen, int stage);\n extern int index_name_pos(const struct index_state *, const char *name, int namelen);\n+extern struct directory_entry *init_directory_entry(char *pathname, int len);\n #define ADD_CACHE_OK_TO_ADD 1\t\t/* Ok to add */\n #define ADD_CACHE_OK_TO_REPLACE 2\t/* Ok to replace file/directory */\n #define ADD_CACHE_SKIP_DFCHECK 4\t/* Ok to skip DF conflict checks */\n@@ -1247,6 +1248,13 @@ static inline ssize_t write_str_in_full(int fd, const char *str)\n \treturn write_in_full(fd, str, strlen(str));\n }\n \n+/* index-v5 helper functions */\n+extern char *super_directory(const char *filename);\n+extern void insert_directory_entry(struct directory_entry *, struct hash_table *, int *, unsigned int *, uint32_t);\n+extern void add_conflict_to_directory_entry(struct directory_entry *, struct conflict_entry *);\n+extern void add_part_to_conflict_entry(struct directory_entry *, struct conflict_entry *, struct conflict_part *);\n+extern struct conflict_entry *create_new_conflict(char *, int, int);\n+\n /* pager.c */\n extern void setup_pager(void);\n extern const char *pager_program;\ndiff --git a/read-cache.c b/read-cache.c\nindex fd095ec..0f1a218 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -1283,7 +1283,7 @@ static struct cache_entry *refresh_cache_entry(struct cache_entry *ce, int reall\n  * Index File I/O\n  *****************************************************************/\n \n-#define INDEX_FORMAT_DEFAULT 3\n+#define INDEX_FORMAT_DEFAULT 5\n \n /*\n  * dev/ino/uid/gid/size are also just tracked to the low 32 bits\n@@ -2170,6 +2170,17 @@ static int ce_write_flush(git_SHA_CTX *context, int fd)\n \treturn 0;\n }\n \n+static int ce_write_flush_v5(int fd)\n+{\n+\tunsigned int buffered = write_buffer_len;\n+\tif (buffered) {\n+\t\tif (write_in_full(fd, write_buffer, buffered) != buffered)\n+\t\t\treturn -1;\n+\t\twrite_buffer_len = 0;\n+\t}\n+\treturn 0;\n+}\n+\n static int ce_write_v2(git_SHA_CTX *context, int fd, void *data, unsigned int len)\n {\n \twhile (len) {\n@@ -2192,6 +2203,30 @@ static int ce_write_v2(git_SHA_CTX *context, int fd, void *data, unsigned int le\n \treturn 0;\n }\n \n+static int ce_write_v5(uint32_t *crc, int fd, void *data, unsigned int len)\n+{\n+\tif (crc)\n+\t\t*crc = crc32(*crc, (Bytef*)data, len);\n+\twhile (len) {\n+\t\tunsigned int buffered = write_buffer_len;\n+\t\tunsigned int partial = WRITE_BUFFER_SIZE - buffered;\n+\t\tif (partial > len)\n+\t\t\tpartial = len;\n+\t\tmemcpy(write_buffer + buffered, data, partial);\n+\t\tbuffered += partial;\n+\t\tif (buffered == WRITE_BUFFER_SIZE) {\n+\t\t\twrite_buffer_len = buffered;\n+\t\t\tif (ce_write_flush_v5(fd))\n+\t\t\t\treturn -1;\n+\t\t\tbuffered = 0;\n+\t\t}\n+\t\twrite_buffer_len = buffered;\n+\t\tlen -= partial;\n+\t\tdata = (char *) data + partial;\n+\t}\n+\treturn 0;\n+}\n+\n static int write_index_ext_header_v2(git_SHA_CTX *context, int fd,\n \t\t\t\t  unsigned int ext, unsigned int sz)\n {\n@@ -2223,6 +2258,19 @@ static int ce_flush(git_SHA_CTX *context, int fd)\n \treturn (write_in_full(fd, write_buffer, left) != left) ? -1 : 0;\n }\n \n+static int ce_flush_v5(int fd)\n+{\n+\tunsigned int left = write_buffer_len;\n+\n+\tif (left)\n+\t\twrite_buffer_len = 0;\n+\n+\tif (write_in_full(fd, write_buffer, left) != left)\n+\t\treturn -1;\n+\n+\treturn 0;\n+}\n+\n static void ce_smudge_racily_clean_entry_v2(struct cache_entry *ce)\n {\n \t/*\n@@ -2272,6 +2320,22 @@ static void ce_smudge_racily_clean_entry_v2(struct cache_entry *ce)\n \t}\n }\n \n+static void ce_smudge_racily_clean_entry_v5(struct cache_entry *ce)\n+{\n+\t/*\n+\t * This method shall only be called if the timestamp of ce\n+\t * is racy (check with is_racy_timestamp). If the timestamp\n+\t * is racy, the writer will just set the time to 0.\n+\t *\n+\t * The reader (ce_match_stat_basic_v5) will then take care\n+\t * of checking if the entry is really changed or not, by\n+\t * taking into account the stat_crc and if that hasn't changed\n+\t * checking the sha1.\n+\t */\n+\tce->ce_mtime.sec = 0;\n+\tce->ce_mtime.nsec = 0;\n+}\n+\n /* Copy miscellaneous fields but not the name */\n static char *copy_cache_entry_to_ondisk(struct ondisk_cache_entry *ondisk,\n \t\t\t\t       struct cache_entry *ce)\n@@ -2455,12 +2519,546 @@ int write_index_v2(struct index_state *istate, int newfd)\n \treturn 0;\n }\n \n+char *super_directory(const char *filename)\n+{\n+\tchar *slash;\n+\n+\tslash = strrchr(filename, '/');\n+\tif (slash)\n+\t\treturn xmemdupz(filename, slash-filename);\n+\treturn NULL;\n+}\n+\n+struct directory_entry *init_directory_entry(char *pathname, int len)\n+{\n+\tstruct directory_entry *de = xmalloc(directory_entry_size(len));\n+\n+\tmemcpy(de->pathname, pathname, len);\n+\tde->pathname[len] = '\\0';\n+\tde->de_flags      = 0;\n+\tde->de_foffset    = 0;\n+\tde->de_cr         = 0;\n+\tde->de_ncr        = 0;\n+\tde->de_nsubtrees  = 0;\n+\tde->de_nfiles     = 0;\n+\tde->de_nentries   = 0;\n+\tmemset(de->sha1, 0, 20);\n+\tde->de_pathlen    = len;\n+\tde->next          = NULL;\n+\tde->next_hash     = NULL;\n+\tde->ce            = NULL;\n+\tde->ce_last       = NULL;\n+\tde->conflict      = NULL;\n+\tde->conflict_last = NULL;\n+\tde->conflict_size = 0;\n+\treturn de;\n+}\n+\n+static void ondisk_from_directory_entry(struct directory_entry *de,\n+\t\t\t\t\tstruct ondisk_directory_entry *ondisk)\n+{\n+\tondisk->foffset   = htonl(de->de_foffset);\n+\tondisk->cr        = htonl(de->de_cr);\n+\tondisk->ncr       = htonl(de->de_ncr);\n+\tondisk->nsubtrees = htonl(de->de_nsubtrees);\n+\tondisk->nfiles    = htonl(de->de_nfiles);\n+\tondisk->nentries  = htonl(de->de_nentries);\n+\thashcpy(ondisk->sha1, de->sha1);\n+\tondisk->flags     = htons(de->de_flags);\n+}\n+\n+static struct conflict_part *conflict_part_from_inmemory(struct cache_entry *ce)\n+{\n+\tstruct conflict_part *conflict;\n+\tshort flags;\n+\n+\tconflict = xmalloc(sizeof(struct conflict_part));\n+\tflags                = CONFLICT_CONFLICTED;\n+\tflags               |= ce_stage(ce) << CONFLICT_STAGESHIFT;\n+\tconflict->flags      = flags;\n+\tconflict->entry_mode = ce->ce_mode;\n+\tconflict->next       = NULL;\n+\thashcpy(conflict->sha1, ce->sha1);\n+\treturn conflict;\n+}\n+\n+static void conflict_to_ondisk(struct conflict_part *cp,\n+\t\t\t\tstruct ondisk_conflict_part *ondisk)\n+{\n+\tondisk->flags      = htons(cp->flags);\n+\tondisk->entry_mode = htons(cp->entry_mode);\n+\thashcpy(ondisk->sha1, cp->sha1);\n+}\n+\n+void add_conflict_to_directory_entry(struct directory_entry *de,\n+\t\t\t\t\tstruct conflict_entry *conflict_entry)\n+{\n+\tde->de_ncr++;\n+\tde->conflict_size += conflict_entry->namelen + 1 + 8 - conflict_entry->pathlen;\n+\tconflict_entry_push(&de->conflict, &de->conflict_last, conflict_entry);\n+}\n+\n+void add_part_to_conflict_entry(struct directory_entry *de,\n+\t\t\t\t\tstruct conflict_entry *entry,\n+\t\t\t\t\tstruct conflict_part *conflict_part)\n+{\n+\n+\tstruct conflict_part *conflict_search;\n+\n+\tentry->nfileconflicts++;\n+\tde->conflict_size += sizeof(struct ondisk_conflict_part);\n+\tif (!entry->entries)\n+\t\tentry->entries = conflict_part;\n+\telse {\n+\t\tconflict_search = entry->entries;\n+\t\twhile (conflict_search->next)\n+\t\t\tconflict_search = conflict_search->next;\n+\t\tconflict_search->next = conflict_part;\n+\t}\n+}\n+\n+void insert_directory_entry(struct directory_entry *de,\n+\t\t\tstruct hash_table *table,\n+\t\t\tint *total_dir_len,\n+\t\t\tunsigned int *ndir,\n+\t\t\tuint32_t crc)\n+{\n+\tstruct directory_entry *insert;\n+\n+\tinsert = (struct directory_entry *)insert_hash(crc, de, table);\n+\tif (insert) {\n+\t\tde->next_hash = insert->next_hash;\n+\t\tinsert->next_hash = de;\n+\t}\n+\t(*ndir)++;\n+\tif (de->de_pathlen == 0)\n+\t\t(*total_dir_len)++;\n+\telse\n+\t\t*total_dir_len += de->de_pathlen + 2;\n+}\n+\n+static struct directory_entry *compile_directory_data(struct index_state *istate,\n+\t\t\t\t\t\tint nfile,\n+\t\t\t\t\t\tunsigned int *ndir,\n+\t\t\t\t\t\tint *non_conflicted,\n+\t\t\t\t\t\tint *total_dir_len,\n+\t\t\t\t\t\tint *total_file_len)\n+{\n+\tint i, dir_len = -1;\n+\tchar *dir;\n+\tstruct directory_entry *de, *current, *search, *found, *new, *previous_entry;\n+\tstruct cache_entry **cache = istate->cache;\n+\tstruct conflict_entry *conflict_entry;\n+\tstruct hash_table table;\n+\tuint32_t crc;\n+\n+\tinit_hash(&table);\n+\tde = init_directory_entry(\"\", 0);\n+\tcurrent = de;\n+\t*ndir = 1;\n+\t*total_dir_len = 1;\n+\tcrc = crc32(0, (Bytef*)de->pathname, de->de_pathlen);\n+\tinsert_hash(crc, de, &table);\n+\tconflict_entry = NULL;\n+\tfor (i = 0; i < nfile; i++) {\n+\t\tint new_entry;\n+\t\tif (cache[i]->ce_flags & CE_REMOVE)\n+\t\t\tcontinue;\n+\n+\t\tnew_entry = !ce_stage(cache[i]) || !conflict_entry\n+\t\t    || cache_name_compare(conflict_entry->name, conflict_entry->namelen,\n+\t\t\t\t\tcache[i]->name, ce_namelen(cache[i]));\n+\t\tif (new_entry)\n+\t\t\t(*non_conflicted)++;\n+\t\tif (dir_len < 0 || strncmp(cache[i]->name, dir, dir_len)\n+\t\t    || cache[i]->name[dir_len] != '/'\n+\t\t    || strchr(cache[i]->name + dir_len + 1, '/')) {\n+\t\t\tdir = super_directory(cache[i]->name);\n+\t\t\tif (!dir)\n+\t\t\t\tdir_len = 0;\n+\t\t\telse\n+\t\t\t\tdir_len = strlen(dir);\n+\t\t\tcrc = crc32(0, (Bytef*)dir, dir_len);\n+\t\t\tfound = lookup_hash(crc, &table);\n+\t\t\tsearch = found;\n+\t\t\twhile (search && dir_len != 0 && strcmp(dir, search->pathname) != 0)\n+\t\t\t\tsearch = search->next_hash;\n+\t\t}\n+\t\tprevious_entry = current;\n+\t\tif (!search || !found) {\n+\t\t\tnew = init_directory_entry(dir, dir_len);\n+\t\t\tcurrent->next = new;\n+\t\t\tcurrent = current->next;\n+\t\t\tinsert_directory_entry(new, &table, total_dir_len, ndir, crc);\n+\t\t\tsearch = current;\n+\t\t}\n+\t\tif (new_entry) {\n+\t\t\tsearch->de_nfiles++;\n+\t\t\t*total_file_len += ce_namelen(cache[i]) + 1;\n+\t\t\tif (search->de_pathlen)\n+\t\t\t\t*total_file_len -= search->de_pathlen + 1;\n+\t\t\tce_queue_push(&(search->ce), &(search->ce_last), cache[i]);\n+\t\t}\n+\t\tif (ce_stage(cache[i]) > 0) {\n+\t\t\tstruct conflict_part *conflict_part;\n+\t\t\tif (new_entry) {\n+\t\t\t\tconflict_entry = create_conflict_entry_from_ce(cache[i], search->de_pathlen);\n+\t\t\t\tadd_conflict_to_directory_entry(search, conflict_entry);\n+\t\t\t}\n+\t\t\tconflict_part = conflict_part_from_inmemory(cache[i]);\n+\t\t\tadd_part_to_conflict_entry(search, conflict_entry, conflict_part);\n+\t\t}\n+\t\tif (dir && !found) {\n+\t\t\tstruct directory_entry *no_subtrees;\n+\n+\t\t\tno_subtrees = current;\n+\t\t\tdir = super_directory(dir);\n+\t\t\tif (dir)\n+\t\t\t\tdir_len = strlen(dir);\n+\t\t\telse\n+\t\t\t\tdir_len = 0;\n+\t\t\tcrc = crc32(0, (Bytef*)dir, dir_len);\n+\t\t\tfound = lookup_hash(crc, &table);\n+\t\t\twhile (!found) {\n+\t\t\t\tnew = init_directory_entry(dir, dir_len);\n+\t\t\t\tnew->de_nsubtrees = 1;\n+\t\t\t\tnew->next = no_subtrees;\n+\t\t\t\tno_subtrees = new;\n+\t\t\t\tinsert_directory_entry(new, &table, total_dir_len, ndir, crc);\n+\t\t\t\tdir = super_directory(dir);\n+\t\t\t\tif (!dir)\n+\t\t\t\t\tdir_len = 0;\n+\t\t\t\telse\n+\t\t\t\t\tdir_len = strlen(dir);\n+\t\t\t\tcrc = crc32(0, (Bytef*)dir, dir_len);\n+\t\t\t\tfound = lookup_hash(crc, &table);\n+\t\t\t}\n+\t\t\tsearch = found;\n+\t\t\twhile (search->next_hash && strcmp(dir, search->pathname) != 0)\n+\t\t\t\tsearch = search->next_hash;\n+\t\t\tif (search)\n+\t\t\t\tfound = search;\n+\t\t\tfound->de_nsubtrees++;\n+\t\t\tprevious_entry->next = no_subtrees;\n+\t\t}\n+\t}\n+\treturn de;\n+}\n+\n+static void ondisk_from_cache_entry(struct cache_entry *ce,\n+\t\t\t\t    struct ondisk_cache_entry_v5 *ondisk)\n+{\n+\tunsigned int flags;\n+\n+\tflags  = ce->ce_flags & CE_STAGEMASK;\n+\tflags |= ce->ce_flags & CE_VALID;\n+\tif (ce->ce_flags & CE_INTENT_TO_ADD)\n+\t\tflags |= CE_INTENT_TO_ADD_V5;\n+\tif (ce->ce_flags & CE_SKIP_WORKTREE)\n+\t\tflags |= CE_SKIP_WORKTREE_V5;\n+\tondisk->flags      = htons(flags);\n+\tondisk->mode       = htons(ce->ce_mode);\n+\tondisk->mtime.sec  = htonl(ce->ce_mtime.sec);\n+#ifdef USE_NSEC\n+\tondisk->mtime.nsec = htonl(ce->ce_mtime.nsec);\n+#else\n+\tondisk->mtime.nsec = 0;\n+#endif\n+\tif (!ce->ce_stat_crc)\n+\t\tce->ce_stat_crc = calculate_stat_crc(ce);\n+\tondisk->stat_crc   = htonl(ce->ce_stat_crc);\n+\thashcpy(ondisk->sha1, ce->sha1);\n+}\n+\n+static int write_directories_v5(struct directory_entry *de, int fd, int conflict_offset)\n+{\n+\tstruct directory_entry *current;\n+\tstruct ondisk_directory_entry ondisk;\n+\tint current_offset, offset_write, ondisk_size, foffset;\n+\tuint32_t crc;\n+\n+\t/*\n+\t * This is needed because the compiler aligns structs to sizes multipe\n+\t * of 4\n+\t */\n+\tondisk_size = sizeof(ondisk.flags)\n+\t\t+ sizeof(ondisk.foffset)\n+\t\t+ sizeof(ondisk.cr)\n+\t\t+ sizeof(ondisk.ncr)\n+\t\t+ sizeof(ondisk.nsubtrees)\n+\t\t+ sizeof(ondisk.nfiles)\n+\t\t+ sizeof(ondisk.nentries)\n+\t\t+ sizeof(ondisk.sha1);\n+\tcurrent = de;\n+\tcurrent_offset = 0;\n+\tfoffset = 0;\n+\twhile (current) {\n+\t\tint pathlen;\n+\n+\t\toffset_write = htonl(current_offset);\n+\t\tif (ce_write_v5(NULL, fd, &offset_write, 4) < 0)\n+\t\t\treturn -1;\n+\t\tif (current->de_pathlen == 0)\n+\t\t\tpathlen = 0;\n+\t\telse\n+\t\t\tpathlen = current->de_pathlen + 1;\n+\t\tcurrent_offset += pathlen + 1 + ondisk_size + 4;\n+\t\tcurrent = current->next;\n+\t}\n+\t/*\n+\t * Write one more offset, which points to the end of the entries,\n+\t * because we use it for calculating the dir length, instead of\n+\t * using strlen.\n+\t */\n+\toffset_write = htonl(current_offset);\n+\tif (ce_write_v5(NULL, fd, &offset_write, 4) < 0)\n+\t\treturn -1;\n+\tcurrent = de;\n+\twhile (current) {\n+\t\tcrc = 0;\n+\t\tif (current->de_pathlen == 0) {\n+\t\t\tif (ce_write_v5(&crc, fd, current->pathname, 1) < 0)\n+\t\t\t\treturn -1;\n+\t\t} else {\n+\t\t\tchar *path;\n+\t\t\tpath = xmalloc(sizeof(char) * (current->de_pathlen + 2));\n+\t\t\tmemcpy(path, current->pathname, current->de_pathlen);\n+\t\t\tmemcpy(path + current->de_pathlen, \"/\\0\", 2);\n+\t\t\tif (ce_write_v5(&crc, fd, path, current->de_pathlen + 2) < 0)\n+\t\t\t\treturn -1;\n+\t\t}\n+\t\tcurrent->de_foffset = foffset;\n+\t\tcurrent->de_cr = conflict_offset;\n+\t\tondisk_from_directory_entry(current, &ondisk);\n+\t\tif (ce_write_v5(&crc, fd, &ondisk, ondisk_size) < 0)\n+\t\t\treturn -1;\n+\t\tcrc = htonl(crc);\n+\t\tif (ce_write_v5(NULL, fd, &crc, 4) < 0)\n+\t\t\treturn -1;\n+\t\tconflict_offset += current->conflict_size;\n+\t\tfoffset += current->de_nfiles * 4;\n+\t\tcurrent = current->next;\n+\t}\n+\treturn 0;\n+}\n+\n+static int write_entries_v5(struct index_state *istate,\n+\t\t\t    struct directory_entry *de,\n+\t\t\t    int entries,\n+\t\t\t    int fd,\n+\t\t\t    int offset_to_offset)\n+{\n+\tint offset, offset_write, ondisk_size;\n+\tstruct directory_entry *current;\n+\n+\toffset = 0;\n+\tondisk_size = sizeof(struct ondisk_cache_entry_v5);\n+\tcurrent = de;\n+\twhile (current) {\n+\t\tint pathlen;\n+\t\tstruct cache_entry *ce = current->ce;\n+\n+\t\tif (current->de_pathlen == 0)\n+\t\t\tpathlen = 0;\n+\t\telse\n+\t\t\tpathlen = current->de_pathlen + 1;\n+\t\twhile (ce) {\n+\t\t\tif (ce->ce_flags & CE_REMOVE)\n+\t\t\t\tcontinue;\n+\t\t\tif (!ce_uptodate(ce) && is_racy_timestamp(istate, ce))\n+\t\t\t\tce_smudge_racily_clean_entry_v5(ce);\n+\n+\t\t\toffset_write = htonl(offset);\n+\t\t\tif (ce_write_v5(NULL, fd, &offset_write, 4) < 0)\n+\t\t\t\treturn -1;\n+\t\t\toffset += ce_namelen(ce) - pathlen + 1 + ondisk_size + 4;\n+\t\t\tce = ce->next;\n+\t\t}\n+\t\tcurrent = current->next;\n+\t}\n+\t/*\n+\t * Write one more offset, which points to the end of the entries,\n+\t * because we use it for calculating the file length, instead of\n+\t * using strlen.\n+\t */\n+\toffset_write = htonl(offset);\n+\tif (ce_write_v5(NULL, fd, &offset_write, 4) < 0)\n+\t\treturn -1;\n+\n+\toffset = offset_to_offset;\n+\tcurrent = de;\n+\twhile (current) {\n+\t\tint pathlen;\n+\t\tstruct cache_entry *ce = current->ce;\n+\n+\t\tif (current->de_pathlen == 0)\n+\t\t\tpathlen = 0;\n+\t\telse\n+\t\t\tpathlen = current->de_pathlen + 1;\n+\t\twhile (ce) {\n+\t\t\tstruct ondisk_cache_entry_v5 ondisk;\n+\t\t\tuint32_t crc, calc_crc;\n+\n+\t\t\tif (ce->ce_flags & CE_REMOVE)\n+\t\t\t\tcontinue;\n+\t\t\tcalc_crc = htonl(offset);\n+\t\t\tcrc = crc32(0, (Bytef*)&calc_crc, 4);\n+\t\t\tif (ce_write_v5(&crc, fd, ce->name + pathlen,\n+\t\t\t\t\tce_namelen(ce) - pathlen + 1) < 0)\n+\t\t\t\treturn -1;\n+\t\t\tondisk_from_cache_entry(ce, &ondisk);\n+\t\t\tif (ce_write_v5(&crc, fd, &ondisk, ondisk_size) < 0)\n+\t\t\t\treturn -1;\n+\t\t\tcrc = htonl(crc);\n+\t\t\tif (ce_write_v5(NULL, fd, &crc, 4) < 0)\n+\t\t\t\treturn -1;\n+\t\t\toffset += 4;\n+\t\t\tce = ce->next;\n+\t\t}\n+\t\tcurrent = current->next;\n+\t}\n+\treturn 0;\n+}\n+\n+static int write_conflict_v5(struct conflict_entry *conflict, int fd)\n+{\n+\tstruct conflict_entry *current;\n+\tstruct conflict_part *current_part;\n+\tuint32_t crc;\n+\n+\tcurrent = conflict;\n+\twhile (current) {\n+\t\tunsigned int to_write;\n+\n+\t\tcrc = 0;\n+\t\tif (ce_write_v5(&crc, fd,\n+\t\t     (Bytef*)(current->name + current->pathlen),\n+\t\t     current->namelen - current->pathlen) < 0)\n+\t\t\treturn -1;\n+\t\tif (ce_write_v5(&crc, fd, (Bytef*)\"\\0\", 1) < 0)\n+\t\t\treturn -1;\n+\t\tto_write = htonl(current->nfileconflicts);\n+\t\tif (ce_write_v5(&crc, fd, (Bytef*)&to_write, 4) < 0)\n+\t\t\treturn -1;\n+\t\tcurrent_part = current->entries;\n+\t\twhile (current_part) {\n+\t\t\tstruct ondisk_conflict_part ondisk;\n+\n+\t\t\tconflict_to_ondisk(current_part, &ondisk);\n+\t\t\tif (ce_write_v5(&crc, fd, (Bytef*)&ondisk, sizeof(struct ondisk_conflict_part)) < 0)\n+\t\t\t\treturn 0;\n+\t\t\tcurrent_part = current_part->next;\n+\t\t}\n+\t\tto_write = htonl(crc);\n+\t\tif (ce_write_v5(NULL, fd, (Bytef*)&to_write, 4) < 0)\n+\t\t\treturn -1;\n+\t\tcurrent = current->next;\n+\t}\n+\treturn 0;\n+}\n+\n+static int write_conflicts_v5(struct index_state *istate,\n+\t\t\t      struct directory_entry *de,\n+\t\t\t      int fd)\n+{\n+\tstruct directory_entry *current;\n+\n+\tcurrent = de;\n+\twhile (current) {\n+\t\tif (current->de_ncr != 0) {\n+\t\t\tif (write_conflict_v5(current->conflict, fd) < 0)\n+\t\t\t\treturn -1;\n+\t\t}\n+\t\tcurrent = current->next;\n+\t}\n+\treturn 0;\n+}\n+\n+static int write_index_v5(struct index_state *istate, int newfd)\n+{\n+\tstruct cache_version_header hdr;\n+\tstruct cache_header_v5 hdr_v5;\n+\tstruct cache_entry **cache = istate->cache;\n+\tstruct directory_entry *de;\n+\tstruct ondisk_directory_entry *ondisk;\n+\tint entries = istate->cache_nr;\n+\tint i, removed, non_conflicted, total_dir_len, ondisk_directory_size;\n+\tint total_file_len, conflict_offset, offset_to_offset;\n+\tunsigned int ndir;\n+\tuint32_t crc;\n+\n+\tfor (i = removed = 0; i < entries; i++) {\n+\t\tif (cache[i]->ce_flags & CE_REMOVE)\n+\t\t\tremoved++;\n+\t}\n+\thdr.hdr_signature = htonl(CACHE_SIGNATURE);\n+\thdr.hdr_version = htonl(istate->version);\n+\thdr_v5.hdr_nfile = htonl(entries - removed);\n+\thdr_v5.hdr_nextension = htonl(0); /* Currently no extensions are supported */\n+\n+\tnon_conflicted = 0;\n+\ttotal_dir_len = 0;\n+\ttotal_file_len = 0;\n+\tde = compile_directory_data(istate, entries, &ndir, &non_conflicted,\n+\t\t\t&total_dir_len, &total_file_len);\n+\thdr_v5.hdr_ndir = htonl(ndir);\n+\n+\t/*\n+\t * This is needed because the compiler aligns structs to sizes multipe\n+\t * of 4\n+\t */\n+\tondisk_directory_size = sizeof(ondisk->flags)\n+\t\t+ sizeof(ondisk->foffset)\n+\t\t+ sizeof(ondisk->cr)\n+\t\t+ sizeof(ondisk->ncr)\n+\t\t+ sizeof(ondisk->nsubtrees)\n+\t\t+ sizeof(ondisk->nfiles)\n+\t\t+ sizeof(ondisk->nentries)\n+\t\t+ sizeof(ondisk->sha1);\n+\thdr_v5.hdr_fblockoffset = htonl(sizeof(hdr) + sizeof(hdr_v5) + 4\n+\t\t+ (ndir + 1) * 4\n+\t\t+ total_dir_len\n+\t\t+ ndir * (ondisk_directory_size + 4)\n+\t\t+ (non_conflicted + 1) * 4);\n+\n+\tcrc = 0;\n+\tif (ce_write_v5(&crc, newfd, &hdr, sizeof(hdr)) < 0)\n+\t\treturn -1;\n+\tif (ce_write_v5(&crc, newfd, &hdr_v5, sizeof(hdr_v5)) < 0)\n+\t\treturn -1;\n+\tcrc = htonl(crc);\n+\tif (ce_write_v5(NULL, newfd, &crc, 4) < 0)\n+\t\treturn -1;\n+\n+\tconflict_offset = sizeof(hdr) + sizeof(hdr_v5) + 4\n+\t\t+ (ndir + 1) * 4\n+\t\t+ total_dir_len\n+\t\t+ ndir * (ondisk_directory_size + 4)\n+\t\t+ (non_conflicted + 1) * 4\n+\t\t+ total_file_len\n+\t\t+ non_conflicted * (sizeof(struct ondisk_cache_entry_v5) + 4);\n+\tif (write_directories_v5(de, newfd, conflict_offset) < 0)\n+\t\treturn -1;\n+\toffset_to_offset = sizeof(hdr) + sizeof(hdr_v5) + 4\n+\t\t+ (ndir + 1) * 4\n+\t\t+ total_dir_len\n+\t\t+ ndir * (ondisk_directory_size + 4);\n+\tif (write_entries_v5(istate, de, entries, newfd, offset_to_offset) < 0)\n+\t\treturn -1;\n+\tif (write_conflicts_v5(istate, de, newfd) < 0)\n+\t\treturn -1;\n+\treturn ce_flush_v5(newfd);\n+}\n+\n int write_index(struct index_state *istate, int newfd)\n {\n \tif (!istate->version)\n \t\tistate->version = INDEX_FORMAT_DEFAULT;\n \n-\treturn write_index_v2(istate, newfd);\n+\tif (istate->version != 5)\n+\t\treturn write_index_v2(istate, newfd);\n+\telse\n+\t\treturn write_index_v5(istate, newfd);\n }\n \n /*\n-- \n1.7.10.886.gdf6792c.dirty\n"},{"id":"196339","messageId":"1343905326-23790-14-git-send-email-t.gummerer@gmail.com","threadId":"31170","inReplyTo":"1343905326-23790-1-git-send-email-t.gummerer@gmail.com","subject":"[PATCH 13/16] Write index-v5 cache-tree data","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2012-08-02T11:02:03Z","receivedAt":"2012-08-02T11:02:03Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"Write the cache-tree data for the index version 5 file format. The\nin-memory cache-tree data is converted to the ondisk format, by adding\nit to the directory entries, that were compiled from the cache-entries\nin the step before.\n\nSigned-off-by: Thomas Gummerer <t.gummerer@gmail.com>\n---\n cache-tree.c | 52 ++++++++++++++++++++++++++++++++++++++++++++++++++++\n cache-tree.h |  1 +\n read-cache.c |  2 ++\n 3 files changed, 55 insertions(+)\n\ndiff --git a/cache-tree.c b/cache-tree.c\nindex 6a314aa..bf945ef 100644\n--- a/cache-tree.c\n+++ b/cache-tree.c\n@@ -612,6 +612,58 @@ struct cache_tree *cache_tree_convert_v5(struct directory_entry *de)\n \treturn convert_one(queue, 0);\n }\n \n+\n+static void convert_one_to_ondisk_v5(struct hash_table *table, struct cache_tree *it,\n+\t\t\t\tconst char *path, int pathlen, uint32_t crc)\n+{\n+\tint i;\n+\tstruct directory_entry *found, *search;\n+\n+\tcrc = crc32(crc, (Bytef*)path, pathlen);\n+\tfound = lookup_hash(crc, table);\n+\tsearch = found;\n+\twhile (search && strcmp(path, search->pathname + search->de_pathlen - strlen(path)) != 0)\n+\t\tsearch = search->next_hash;\n+\tif (!search)\n+\t\treturn;\n+\t/*\n+\t * The number of subtrees is already calculated by\n+\t * compile_directory_data, therefore we only need to\n+\t * add the entry_count\n+\t */\n+\tsearch->de_nentries = it->entry_count;\n+\tif (0 <= it->entry_count)\n+\t\thashcpy(search->sha1, it->sha1);\n+\tif (strcmp(path, \"\") != 0)\n+\t\tcrc = crc32(crc, (Bytef*)\"/\", 1);\n+\n+#if DEBUG\n+\tif (0 <= it->entry_count)\n+\t\tfprintf(stderr, \"cache-tree <%.*s> (%d ent, %d subtree) %s\\n\",\n+\t\t\tpathlen, path, it->entry_count, it->subtree_nr,\n+\t\t\tsha1_to_hex(it->sha1));\n+\telse\n+\t\tfprintf(stderr, \"cache-tree <%.*s> (%d subtree) invalid\\n\",\n+\t\t\tpathlen, path, it->subtree_nr);\n+#endif\n+\n+\tfor (i = 0; i < it->subtree_nr; i++) {\n+\t\tstruct cache_tree_sub *down = it->down[i];\n+\t\tif (i) {\n+\t\t\tstruct cache_tree_sub *prev = it->down[i-1];\n+\t\t\tif (subtree_name_cmp(down->name, down->namelen,\n+\t\t\t\t\t     prev->name, prev->namelen) <= 0)\n+\t\t\t\tdie(\"fatal - unsorted cache subtree\");\n+\t\t}\n+\t\tconvert_one_to_ondisk_v5(table, down->cache_tree, down->name, down->namelen, crc);\n+\t}\n+}\n+\n+void cache_tree_to_ondisk_v5(struct hash_table *table, struct cache_tree *root)\n+{\n+\tconvert_one_to_ondisk_v5(table, root, \"\", 0, 0);\n+}\n+\n static struct cache_tree *cache_tree_find(struct cache_tree *it, const char *path)\n {\n \tif (!it)\ndiff --git a/cache-tree.h b/cache-tree.h\nindex f4131a6..a41b684 100644\n--- a/cache-tree.h\n+++ b/cache-tree.h\n@@ -33,6 +33,7 @@ struct cache_tree_sub *cache_tree_sub(struct cache_tree *, const char *);\n void cache_tree_write(struct strbuf *, struct cache_tree *root);\n struct cache_tree *cache_tree_read(const char *buffer, unsigned long size);\n struct cache_tree *cache_tree_convert_v5(struct directory_entry *de);\n+void cache_tree_to_ondisk_v5(struct hash_table *table, struct cache_tree *root);\n \n int cache_tree_fully_valid(struct cache_tree *);\n int cache_tree_update(struct cache_tree *, struct cache_entry **, int, int);\ndiff --git a/read-cache.c b/read-cache.c\nindex 0f1a218..6538ddf 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -2742,6 +2742,8 @@ static struct directory_entry *compile_directory_data(struct index_state *istate\n \t\t\tprevious_entry->next = no_subtrees;\n \t\t}\n \t}\n+\tif (istate->cache_tree)\n+\t\tcache_tree_to_ondisk_v5(&table, istate->cache_tree);\n \treturn de;\n }\n \n-- \n1.7.10.886.gdf6792c.dirty\n"},{"id":"196336","messageId":"1343905326-23790-15-git-send-email-t.gummerer@gmail.com","threadId":"31170","inReplyTo":"1343905326-23790-1-git-send-email-t.gummerer@gmail.com","subject":"[PATCH 14/16] Write resolve-undo data for index-v5","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2012-08-02T11:02:04Z","receivedAt":"2012-08-02T11:02:04Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"Write the resolve undo data to the ondisk format, by joining the data\nin the resolve-undo string-list with the already existing conflicts\nthat were compiled before, when searching the directories and add\nthem to the corresponding directory entries.\n\nSigned-off-by: Thomas Gummerer <t.gummerer@gmail.com>\n---\n read-cache.c   |  1 +\n resolve-undo.c | 93 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n resolve-undo.h |  1 +\n 3 files changed, 95 insertions(+)\n\ndiff --git a/read-cache.c b/read-cache.c\nindex 6538ddf..a4ca40a 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -2744,6 +2744,7 @@ static struct directory_entry *compile_directory_data(struct index_state *istate\n \t}\n \tif (istate->cache_tree)\n \t\tcache_tree_to_ondisk_v5(&table, istate->cache_tree);\n+\tresolve_undo_to_ondisk_v5(&table, istate->resolve_undo, ndir, total_dir_len, de);\n \treturn de;\n }\n \ndiff --git a/resolve-undo.c b/resolve-undo.c\nindex f96c6ba..4568dcc 100644\n--- a/resolve-undo.c\n+++ b/resolve-undo.c\n@@ -206,3 +206,96 @@ void resolve_undo_convert_v5(struct index_state *istate,\n \t\tce = ce->next;\n \t}\n }\n+\n+void resolve_undo_to_ondisk_v5(struct hash_table *table,\n+\t\t\t\tstruct string_list *resolve_undo,\n+\t\t\t\tunsigned int *ndir, int *total_dir_len,\n+\t\t\t\tstruct directory_entry *de)\n+{\n+\tstruct string_list_item *item;\n+\tstruct directory_entry *search;\n+\n+\tif (!resolve_undo)\n+\t\treturn;\n+\tfor_each_string_list_item(item, resolve_undo) {\n+\t\tstruct conflict_entry *conflict_entry;\n+\t\tstruct resolve_undo_info *ui = item->util;\n+\t\tchar *super;\n+\t\tint i, dir_len, len;\n+\t\tuint32_t crc;\n+\t\tstruct directory_entry *found, *current, *new_tree;\n+\n+\t\tif (!ui)\n+\t\t\tcontinue;\n+\n+\t\tsuper = super_directory(item->string);\n+\t\tif (!super)\n+\t\t\tdir_len = 0;\n+\t\telse\n+\t\t\tdir_len = strlen(super);\n+\t\tcrc = crc32(0, (Bytef*)super, dir_len);\n+\t\tfound = lookup_hash(crc, table);\n+\t\tcurrent = NULL;\n+\t\tnew_tree = NULL;\n+\t\t\n+\t\twhile (!found) {\n+\t\t\tstruct directory_entry *new;\n+\n+\t\t\tnew = init_directory_entry(super, dir_len);\n+\t\t\tif (!current)\n+\t\t\t\tcurrent = new;\n+\t\t\tinsert_directory_entry(new, table, total_dir_len, ndir, crc);\n+\t\t\tif (new_tree != NULL)\n+\t\t\t\tnew->de_nsubtrees = 1;\n+\t\t\tnew->next = new_tree;\n+\t\t\tnew_tree = new;\n+\t\t\tsuper = super_directory(super);\n+\t\t\tif (!super)\n+\t\t\t\tdir_len = 0;\n+\t\t\telse\n+\t\t\t\tdir_len = strlen(super);\n+\t\t\tcrc = crc32(0, (Bytef*)super, dir_len);\n+\t\t\tfound = lookup_hash(crc, table);\n+\t\t}\n+\t\tsearch = found;\n+\t\twhile (search->next_hash && strcmp(super, search->pathname) != 0)\n+\t\t\tsearch = search->next_hash;\n+\t\tif (search && !current)\n+\t\t\tcurrent = search;\n+\t\tif (!search && !current)\n+\t\t\tcurrent = new_tree;\n+\t\tif (!super && new_tree) {\n+\t\t\tnew_tree->next = de->next;\n+\t\t\tde->next = new_tree;\n+\t\t\tde->de_nsubtrees++;\n+\t\t} else if (new_tree) {\n+\t\t\tstruct directory_entry *temp;\n+\n+\t\t\tsearch = de->next;\n+\t\t\twhile (strcmp(super, search->pathname))\n+\t\t\t\tsearch = search->next;\n+\t\t\ttemp = new_tree;\n+\t\t\twhile (temp->next)\n+\t\t\t\ttemp = temp->next;\n+\t\t\tsearch->de_nsubtrees++;\n+\t\t\ttemp->next = search->next;\n+\t\t\tsearch->next = new_tree;\n+\t\t}\n+\n+\t\tlen = strlen(item->string);\n+\t\tconflict_entry = create_new_conflict(item->string, len, current->de_pathlen);\n+\t\tadd_conflict_to_directory_entry(current, conflict_entry);\n+\t\tfor (i = 0; i < 3; i++) {\n+\t\t\tif (ui->mode[i]) {\n+\t\t\t\tstruct conflict_part *cp;\n+\n+\t\t\t\tcp = xmalloc(sizeof(struct conflict_part));\n+\t\t\t\tcp->flags = (i + 1) << CONFLICT_STAGESHIFT;\n+\t\t\t\tcp->entry_mode = ui->mode[i];\n+\t\t\t\tcp->next = NULL;\n+\t\t\t\thashcpy(cp->sha1, ui->sha1[i]);\n+\t\t\t\tadd_part_to_conflict_entry(current, conflict_entry, cp);\n+\t\t\t}\n+\t\t}\n+\t}\n+}\ndiff --git a/resolve-undo.h b/resolve-undo.h\nindex ab660a6..ff80d84 100644\n--- a/resolve-undo.h\n+++ b/resolve-undo.h\n@@ -14,5 +14,6 @@ extern int unmerge_index_entry_at(struct index_state *, int);\n extern void unmerge_index(struct index_state *, const char **);\n \n extern void resolve_undo_convert_v5(struct index_state *, struct conflict_entry *);\n+extern void resolve_undo_to_ondisk_v5(struct hash_table *, struct string_list *, unsigned int *, int *, struct directory_entry *);\n \n #endif\n-- \n1.7.10.886.gdf6792c.dirty\n"},{"id":"196341","messageId":"1343905326-23790-16-git-send-email-t.gummerer@gmail.com","threadId":"31170","inReplyTo":"1343905326-23790-1-git-send-email-t.gummerer@gmail.com","subject":"[PATCH 15/16] update-index.c: add a force-rewrite option","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2012-08-02T11:02:05Z","receivedAt":"2012-08-02T11:02:05Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"Add a force-rewrite option to update-index, which allows the user\nto rewrite the index, even if there are no changes. This can be used\nto do performance tests of both the reader and the writer.\n\nSigned-off-by: Thomas Gummerer <t.gummerer@gmail.com>\n---\n builtin/update-index.c | 5 ++++-\n 1 file changed, 4 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/update-index.c b/builtin/update-index.c\nindex e747def..063bf54 100644\n--- a/builtin/update-index.c\n+++ b/builtin/update-index.c\n@@ -24,6 +24,7 @@ static int allow_remove;\n static int allow_replace;\n static int info_only;\n static int force_remove;\n+static int force_rewrite;\n static int verbose;\n static int mark_valid_only;\n static int mark_skip_worktree_only;\n@@ -734,6 +735,8 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)\n \t\tOPT_BIT(0, \"unmerged\", &refresh_args.flags,\n \t\t\t\"refresh even if index contains unmerged entries\",\n \t\t\tREFRESH_UNMERGED),\n+\t\tOPT_SET_INT(0, \"force-rewrite\", &force_rewrite,\n+\t\t\t\"force a index rewrite even if there is no change\", 1),\n \t\t{OPTION_CALLBACK, 0, \"refresh\", &refresh_args, NULL,\n \t\t\t\"refresh stat information\",\n \t\t\tPARSE_OPT_NOARG | PARSE_OPT_NONEG,\n@@ -892,7 +895,7 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)\n \t\tstrbuf_release(&buf);\n \t}\n \n-\tif (active_cache_changed) {\n+\tif (active_cache_changed || force_rewrite) {\n \t\tif (newfd < 0) {\n \t\t\tif (refresh_args.flags & REFRESH_QUIET)\n \t\t\t\texit(128);\n-- \n1.7.10.886.gdf6792c.dirty\n"},{"id":"196338","messageId":"1343905326-23790-17-git-send-email-t.gummerer@gmail.com","threadId":"31170","inReplyTo":"1343905326-23790-1-git-send-email-t.gummerer@gmail.com","subject":"[PATCH 16/16] p0002-index.sh: add perf test for the index formats","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2012-08-02T11:02:06Z","receivedAt":"2012-08-02T11:02:06Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"Add a performance test for index version [23]/4/5 by using\ngit update-index --force-rewrite, thus testing both the reader\nand the writer speed of all index formats.\n\nSigned-off-by: Thomas Gummerer <t.gummerer@gmail.com>\n---\n t/perf/p0002-index.sh | 33 +++++++++++++++++++++++++++++++++\n 1 file changed, 33 insertions(+)\n create mode 100755 t/perf/p0002-index.sh\n\ndiff --git a/t/perf/p0002-index.sh b/t/perf/p0002-index.sh\nnew file mode 100755\nindex 0000000..2996357\n--- /dev/null\n+++ b/t/perf/p0002-index.sh\n@@ -0,0 +1,33 @@\n+#!/bin/sh\n+\n+test_description=\"Tests index versions [23]/4/5\"\n+\n+. ./perf-lib.sh\n+\n+test_perf_large_repo\n+\n+test_expect_success 'convert to v3' '\n+\tgit update-index --index-version=3\n+'\n+\n+test_perf 'v[23]: update-index' '\n+\tgit update-index --force-rewrite >/dev/null\n+'\n+\n+test_expect_success 'convert to v4' '\n+\tgit update-index --index-version=4\n+'\n+\n+test_perf 'v4: update-index' '\n+\tgit update-index --force-rewrite >/dev/null\n+'\n+\n+test_expect_success 'convert to v5' '\n+\tgit update-index --index-version=5\n+'\n+\n+test_perf 'v5: update-index' '\n+\tgit update-index --force-rewrite >/dev/null\n+'\n+\n+test_done\n-- \n1.7.10.886.gdf6792c.dirty\n"},{"id":"196348","messageId":"CACsJy8De=1RR9yAQ79zzNM_iALF1OCB2L9A8pMK_v+ixZqgJug@mail.gmail.com","threadId":"31170","inReplyTo":"1343905326-23790-1-git-send-email-t.gummerer@gmail.com","subject":"Re: [RFC 0/16] Introduce index file format version 5","fromName":"Nguyen Thai Ngoc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2012-08-02T12:10:57Z","receivedAt":"2012-08-02T12:10:57Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Thu, Aug 2, 2012 at 6:01 PM, Thomas Gummerer <t.gummerer@gmail.com> wrote:\n> Documentation/technical/index-file-format-v5.txt |  281 ++++++++++++++++++++++++++++++++++\n> builtin/update-index.c                           |    5 +-\n> cache-tree.c                                     |  145 ++++++++++++++++++\n> cache-tree.h                                     |    7 +\n> cache.h                                          |   96 +++++++++++-\n> read-cache.c                                     | 1519 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-----------------\n> resolve-undo.c                                   |  129 ++++++++++++++++\n> resolve-undo.h                                   |    3 +\n> t/perf/p0002-index.sh                            |   33 ++++\n> t/t2104-update-index-skip-worktree.sh            |   15 +-\n> t/t3700-add.sh                                   |    1 +\n> test-index-version.c                             |    2 +-\n> 12 files changed, 2082 insertions(+), 154 deletions(-)\n\nThis is not good (too many pluses in read-cache.c) if it comes from\nformat-patch. Does it?\n-- \nDuy\n"},{"id":"196349","messageId":"CACsJy8C4D=nmcLmX_mahrx9TMCjzzqWdEsGU9W=gmhJVAqdk4A@mail.gmail.com","threadId":"31170","inReplyTo":"1343905326-23790-2-git-send-email-t.gummerer@gmail.com","subject":"Re: [PATCH 01/16] Modify cache_header to prepare for other index formats","fromName":"Nguyen Thai Ngoc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2012-08-02T12:15:08Z","receivedAt":"2012-08-02T12:15:08Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Thu, Aug 2, 2012 at 6:01 PM, Thomas Gummerer <t.gummerer@gmail.com> wrote:\n> diff --git a/cache.h b/cache.h\n> index 6e9a243..d4028ef 100644\n> --- a/cache.h\n> +++ b/cache.h\n> @@ -99,9 +99,12 @@ unsigned long git_deflate_bound(git_zstream *, unsigned long);\n>   */\n>\n>  #define CACHE_SIGNATURE 0x44495243     /* \"DIRC\" */\n> -struct cache_header {\n> +struct cache_version_header {\n>         unsigned int hdr_signature;\n>         unsigned int hdr_version;\n> +};\n> +\n> +struct cache_header_v2 {\n>         unsigned int hdr_entries;\n>  };\n>\n\nThese structs should probably be moved to read-cache.c. We can\nredeclare cache_version_header again in test-index-version.c (it does\nnot look too bad to me).\n-- \nDuy\n"},{"id":"196350","messageId":"CACsJy8Cr4DoO5aw4-CJPN8j1CiWw+pU+5=_cB+mmViUvMU6BOw@mail.gmail.com","threadId":"31170","inReplyTo":"1343905326-23790-4-git-send-email-t.gummerer@gmail.com","subject":"Re: [PATCH 03/16] Modify match_stat_basic to prepare for other index formats","fromName":"Nguyen Thai Ngoc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2012-08-02T12:20:40Z","receivedAt":"2012-08-02T12:20:40Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Thu, Aug 2, 2012 at 6:01 PM, Thomas Gummerer <t.gummerer@gmail.com> wrote:\n> @@ -1443,7 +1452,6 @@ void read_index_v2(struct index_state *istate, void *mmap, int mmap_size)\n>                 src_offset += consumed;\n>         }\n>         strbuf_release(&previous_name_buf);\n> -\n>         while (src_offset <= mmap_size - 20 - 8) {\n>                 /* After an array of active_nr index entries,\n>                  * there can be arbitrary number of extended\n> @@ -1500,7 +1508,6 @@ int read_index_from(struct index_state *istate, const char *path)\n>                 die(\"index file smaller than expected\");\n>\n>         mmap = xmmap(NULL, mmap_size, PROT_READ | PROT_WRITE, MAP_PRIVATE, fd, 0);\n> -       close(fd);\n>         if (mmap == MAP_FAILED)\n>                 die_errno(\"unable to map index file\");\n>\n> @@ -1512,7 +1519,6 @@ int read_index_from(struct index_state *istate, const char *path)\n>                 goto unmap;\n>\n>         read_index_v2(istate, mmap, mmap_size);\n> -\n>         istate->timestamp.sec = st.st_mtime;\n>         istate->timestamp.nsec = ST_MTIME_NSEC(st);\n>\n\nyou could have done this in 02/16 when you introduced this block.\n\n> @@ -1802,9 +1808,6 @@ int write_index(struct index_state *istate, int newfd)\n>                 }\n>         }\n>\n> -       if (!istate->version)\n> -               istate->version = INDEX_FORMAT_DEFAULT;\n> -\n>         /* demote version 3 to version 2 when the latter suffices */\n>         if (istate->version == 3 || istate->version == 2)\n>                 istate->version = extended ? 3 : 2;\n\nwhy? it does not seem to be related to the commit message.\n-- \nDuy\n"},{"id":"196351","messageId":"CACsJy8Av_ODO--DAsznP8R6DnOP_GQXKxVQDKeVYg=E3mZUnEQ@mail.gmail.com","threadId":"31170","inReplyTo":"1343905326-23790-5-git-send-email-t.gummerer@gmail.com","subject":"Re: [PATCH 04/16] Modify write functions to prepare for other index formats","fromName":"Nguyen Thai Ngoc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2012-08-02T12:22:58Z","receivedAt":"2012-08-02T12:22:58Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Thu, Aug 2, 2012 at 6:01 PM, Thomas Gummerer <t.gummerer@gmail.com> wrote:\n> @@ -1785,7 +1785,7 @@ void update_index_if_able(struct index_state *istate, struct lock_file *lockfile\n>                 rollback_lock_file(lockfile);\n>  }\n>\n> -int write_index(struct index_state *istate, int newfd)\n> +int write_index_v2(struct index_state *istate, int newfd)\n>  {\n>         git_SHA_CTX c;\n>         struct cache_version_header hdr;\n\nmake it static function too (and read_index_v2 too, I think)\n-- \nDuy\n"},{"id":"196352","messageId":"CACsJy8Co8warnQg+fKmXnO_wt=dTPBjRRe+Apic6-GBrTr_3rg@mail.gmail.com","threadId":"31170","inReplyTo":"1343905326-23790-10-git-send-email-t.gummerer@gmail.com","subject":"Re: [PATCH 09/16] Read index-v5","fromName":"Nguyen Thai Ngoc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2012-08-02T12:45:58Z","receivedAt":"2012-08-02T12:45:58Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"General note. I wonder if we should create a separate source file for\nv5 (at least the low level handling part). Partial reading/writing\nwill come (hopefully soon) and read-cache.c on master is already close\nto 2000 lines.\n\nOn Thu, Aug 2, 2012 at 6:01 PM, Thomas Gummerer <t.gummerer@gmail.com> wrote:\n> +static struct cache_entry *cache_entry_from_ondisk_v5(struct ondisk_cache_entry_v5 *ondisk,\n> +                                                  struct directory_entry *de,\n> +                                                  char *name,\n> +                                                  size_t len,\n> +                                                  size_t prefix_len)\n> +{\n> +       struct cache_entry *ce = xmalloc(cache_entry_size(len + de->de_pathlen));\n> +       int flags;\n> +\n> +       flags = ntoh_s(ondisk->flags);\n\nhuh? ntoh_s (and ntoh_l below)? search/replace problem?\n\n> +       ce->ce_ctime.sec  = 0;\n> +       ce->ce_mtime.sec  = ntoh_l(ondisk->mtime.sec);\n-- \nDuy\n"},{"id":"196353","messageId":"CACsJy8AG_dbqentgM+Y9pE=y0bGs2u57sBTF0oG5rNP4HSqY_A@mail.gmail.com","threadId":"31170","inReplyTo":"1343905326-23790-17-git-send-email-t.gummerer@gmail.com","subject":"Re: [PATCH 16/16] p0002-index.sh: add perf test for the index formats","fromName":"Nguyen Thai Ngoc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2012-08-02T12:50:31Z","receivedAt":"2012-08-02T12:50:31Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Thu, Aug 2, 2012 at 6:02 PM, Thomas Gummerer <t.gummerer@gmail.com> wrote:\n> Add a performance test for index version [23]/4/5 by using\n> git update-index --force-rewrite, thus testing both the reader\n> and the writer speed of all index formats.\n\nOn the testing side, it may be an interesting idea to force the whole\ntest suite to use v5 by default. There are a few test cases that\nrequire a specific index version. We can identify and rule them out.\nThat'll give v5 a lot more exercises.\n-- \nDuy\n"},{"id":"196354","messageId":"20120802134718.GA1000@tgummerer.unibz.it","threadId":"31170","inReplyTo":"CACsJy8De=1RR9yAQ79zzNM_iALF1OCB2L9A8pMK_v+ixZqgJug@mail.gmail.com","subject":"Re: [RFC 0/16] Introduce index file format version 5","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2012-08-02T13:47:19Z","receivedAt":"2012-08-02T13:47:19Z","isPatch":false,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"On 08/02, Nguyen Thai Ngoc Duy wrote:\n> On Thu, Aug 2, 2012 at 6:01 PM, Thomas Gummerer <t.gummerer@gmail.com> wrote:\n> > Documentation/technical/index-file-format-v5.txt |  281 ++++++++++++++++++++++++++++++++++\n> > builtin/update-index.c                           |    5 +-\n> > cache-tree.c                                     |  145 ++++++++++++++++++\n> > cache-tree.h                                     |    7 +\n> > cache.h                                          |   96 +++++++++++-\n> > read-cache.c                                     | 1519 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-----------------\n> > resolve-undo.c                                   |  129 ++++++++++++++++\n> > resolve-undo.h                                   |    3 +\n> > t/perf/p0002-index.sh                            |   33 ++++\n> > t/t2104-update-index-skip-worktree.sh            |   15 +-\n> > t/t3700-add.sh                                   |    1 +\n> > test-index-version.c                             |    2 +-\n> > 12 files changed, 2082 insertions(+), 154 deletions(-)\n> \n> This is not good (too many pluses in read-cache.c) if it comes from\n> format-patch. Does it?\n\nNo this comes from git diff --stat. It's probably just my terminal\nwidth then?\n"},{"id":"196355","messageId":"CACsJy8ArnHCXsaW8K47OdcWgYe3=jEPukt-c61kOx1raTNvUEA@mail.gmail.com","threadId":"31170","inReplyTo":"20120802134718.GA1000@tgummerer.unibz.it","subject":"Re: [RFC 0/16] Introduce index file format version 5","fromName":"Nguyen Thai Ngoc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2012-08-02T13:53:08Z","receivedAt":"2012-08-02T13:53:08Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Thu, Aug 2, 2012 at 8:47 PM, Thomas Gummerer <t.gummerer@gmail.com> wrote:\n> On 08/02, Nguyen Thai Ngoc Duy wrote:\n>> On Thu, Aug 2, 2012 at 6:01 PM, Thomas Gummerer <t.gummerer@gmail.com> wrote:\n>> > Documentation/technical/index-file-format-v5.txt |  281 ++++++++++++++++++++++++++++++++++\n>> > builtin/update-index.c                           |    5 +-\n>> > cache-tree.c                                     |  145 ++++++++++++++++++\n>> > cache-tree.h                                     |    7 +\n>> > cache.h                                          |   96 +++++++++++-\n>> > read-cache.c                                     | 1519 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++-----------------\n>> > resolve-undo.c                                   |  129 ++++++++++++++++\n>> > resolve-undo.h                                   |    3 +\n>> > t/perf/p0002-index.sh                            |   33 ++++\n>> > t/t2104-update-index-skip-worktree.sh            |   15 +-\n>> > t/t3700-add.sh                                   |    1 +\n>> > test-index-version.c                             |    2 +-\n>> > 12 files changed, 2082 insertions(+), 154 deletions(-)\n>>\n>> This is not good (too many pluses in read-cache.c) if it comes from\n>> format-patch. Does it?\n>\n> No this comes from git diff --stat. It's probably just my terminal\n> width then?\n>\n\nAh ok. Recent git learns to use much screen estate as possible but\nformat-patch keep it fit wihtin 75 or so columns.\n-- \nDuy\n"},{"id":"196356","messageId":"20120802135647.GB1000@tgummerer.unibz.it","threadId":"31170","inReplyTo":"CACsJy8AG_dbqentgM+Y9pE=y0bGs2u57sBTF0oG5rNP4HSqY_A@mail.gmail.com","subject":"Re: [PATCH 16/16] p0002-index.sh: add perf test for the index formats","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2012-08-02T13:56:47Z","receivedAt":"2012-08-02T13:56:47Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"On 08/02, Nguyen Thai Ngoc Duy wrote:\n> On Thu, Aug 2, 2012 at 6:02 PM, Thomas Gummerer <t.gummerer@gmail.com> wrote:\n> > Add a performance test for index version [23]/4/5 by using\n> > git update-index --force-rewrite, thus testing both the reader\n> > and the writer speed of all index formats.\n> \n> On the testing side, it may be an interesting idea to force the whole\n> test suite to use v5 by default. There are a few test cases that\n> require a specific index version. We can identify and rule them out.\n> That'll give v5 a lot more exercises.\n\nThe test suite already works with index v5. Patch 5 and 6 fix the only\ncases where index-v5 would fail. To make it use index-v5 you can just\nset the INDEX_VERSION_DEFAULT to 5 in read-cache.c\n\nI've never used it for performance testing yet, I'll try to run it with\nboth versions and post the results. I don't think the results will be\na lot different thought, since the test-suite usually uses small\nrepositories.\n"},{"id":"196357","messageId":"20120802140453.GC1000@tgummerer.unibz.it","threadId":"31170","inReplyTo":"CACsJy8Co8warnQg+fKmXnO_wt=dTPBjRRe+Apic6-GBrTr_3rg@mail.gmail.com","subject":"Re: [PATCH 09/16] Read index-v5","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2012-08-02T14:04:53Z","receivedAt":"2012-08-02T14:04:53Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"On 08/02, Nguyen Thai Ngoc Duy wrote:\n> General note. I wonder if we should create a separate source file for\n> v5 (at least the low level handling part). Partial reading/writing\n> will come (hopefully soon) and read-cache.c on master is already close\n> to 2000 lines.\n\nTo me it would make sense, but we'd probably have to split it to\nat least 3 files, one for index-v2, one for index-v5 and one for\nthe general functions/api.\n\n> On Thu, Aug 2, 2012 at 6:01 PM, Thomas Gummerer <t.gummerer@gmail.com> wrote:\n> > +static struct cache_entry *cache_entry_from_ondisk_v5(struct ondisk_cache_entry_v5 *ondisk,\n> > +                                                  struct directory_entry *de,\n> > +                                                  char *name,\n> > +                                                  size_t len,\n> > +                                                  size_t prefix_len)\n> > +{\n> > +       struct cache_entry *ce = xmalloc(cache_entry_size(len + de->de_pathlen));\n> > +       int flags;\n> > +\n> > +       flags = ntoh_s(ondisk->flags);\n> \n> huh? ntoh_s (and ntoh_l below)? search/replace problem?\n\nNo, they are correct, Junio introduced this functions with index-v4 for\nsystems which need aligned access. They are defined as written below.\n\n> #ifndef NEEDS_ALIGNED_ACCESS\n> #define ntoh_s(var) ntohs(var)\n> #define ntoh_l(var) ntohl(var)\n> #else\n> static inline uint16_t ntoh_s_force_align(void *p)\n> {\n> \tuint16_t x;\n> \tmemcpy(&x, p, sizeof(x));\n> \treturn ntohs(x);\n> }\n> static inline uint32_t ntoh_l_force_align(void *p)\n> {\n> \tuint32_t x;\n> \tmemcpy(&x, p, sizeof(x));\n> \treturn ntohl(x);\n> }\n> #define ntoh_s(var)\n> ntoh_s_force_align(&(var))\n> #define ntoh_l(var)\n> ntoh_l_force_align(&(var))\n> #endif\n\n> > +       ce->ce_ctime.sec  = 0;\n> > +       ce->ce_mtime.sec  = ntoh_l(ondisk->mtime.sec);\n> -- \n> Duy\n"},{"id":"196358","messageId":"20120802141143.GD1000@tgummerer.unibz.it","threadId":"31170","inReplyTo":"CACsJy8Av_ODO--DAsznP8R6DnOP_GQXKxVQDKeVYg=E3mZUnEQ@mail.gmail.com","subject":"Re: [PATCH 04/16] Modify write functions to prepare for other index formats","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2012-08-02T14:11:43Z","receivedAt":"2012-08-02T14:11:43Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"\nOn 08/02, Nguyen Thai Ngoc Duy wrote:\n> On Thu, Aug 2, 2012 at 6:01 PM, Thomas Gummerer <t.gummerer@gmail.com> wrote:\n> > @@ -1785,7 +1785,7 @@ void update_index_if_able(struct index_state *istate, struct lock_file *lockfile\n> >                 rollback_lock_file(lockfile);\n> >  }\n> >\n> > -int write_index(struct index_state *istate, int newfd)\n> > +int write_index_v2(struct index_state *istate, int newfd)\n> >  {\n> >         git_SHA_CTX c;\n> >         struct cache_version_header hdr;\n> \n> make it static function too (and read_index_v2 too, I think)\n> -- \n> Duy\n\nMakes sense, thanks!\n"},{"id":"196359","messageId":"20120802141626.GE1000@tgummerer.unibz.it","threadId":"31170","inReplyTo":"CACsJy8Cr4DoO5aw4-CJPN8j1CiWw+pU+5=_cB+mmViUvMU6BOw@mail.gmail.com","subject":"Re: [PATCH 03/16] Modify match_stat_basic to prepare for other index formats","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2012-08-02T14:16:26Z","receivedAt":"2012-08-02T14:16:26Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"On 08/02, Nguyen Thai Ngoc Duy wrote:\n> On Thu, Aug 2, 2012 at 6:01 PM, Thomas Gummerer <t.gummerer@gmail.com> wrote:\n> > @@ -1443,7 +1452,6 @@ void read_index_v2(struct index_state *istate, void *mmap, int mmap_size)\n> >                 src_offset += consumed;\n> >         }\n> >         strbuf_release(&previous_name_buf);\n> > -\n> >         while (src_offset <= mmap_size - 20 - 8) {\n> >                 /* After an array of active_nr index entries,\n> >                  * there can be arbitrary number of extended\n> > @@ -1500,7 +1508,6 @@ int read_index_from(struct index_state *istate, const char *path)\n> >                 die(\"index file smaller than expected\");\n> >\n> >         mmap = xmmap(NULL, mmap_size, PROT_READ | PROT_WRITE, MAP_PRIVATE, fd, 0);\n> > -       close(fd);\n> >         if (mmap == MAP_FAILED)\n> >                 die_errno(\"unable to map index file\");\n> >\n> > @@ -1512,7 +1519,6 @@ int read_index_from(struct index_state *istate, const char *path)\n> >                 goto unmap;\n> >\n> >         read_index_v2(istate, mmap, mmap_size);\n> > -\n> >         istate->timestamp.sec = st.st_mtime;\n> >         istate->timestamp.nsec = ST_MTIME_NSEC(st);\n> >\n> \n> you could have done this in 02/16 when you introduced this block.\n\nThanks.\n\n> > @@ -1802,9 +1808,6 @@ int write_index(struct index_state *istate, int newfd)\n> >                 }\n> >         }\n> >\n> > -       if (!istate->version)\n> > -               istate->version = INDEX_FORMAT_DEFAULT;\n> > -\n> >         /* demote version 3 to version 2 when the latter suffices */\n> >         if (istate->version == 3 || istate->version == 2)\n> >                 istate->version = extended ? 3 : 2;\n> \n> why? it does not seem to be related to the commit message.\n\nSorry this is wrong, it should belong to patch 4. Thanks for noticing.\n"},{"id":"196395","messageId":"CACsJy8AQUwb8noutwdcN-6Piyo1rrj7_EUMjrXB-83J9AtM6kQ@mail.gmail.com","threadId":"31170","inReplyTo":"1343905326-23790-1-git-send-email-t.gummerer@gmail.com","subject":"Re: [RFC 0/16] Introduce index file format version 5","fromName":"Nguyen Thai Ngoc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2012-08-03T03:16:20Z","receivedAt":"2012-08-03T03:16:20Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Thu, Aug 2, 2012 at 6:01 PM, Thomas Gummerer <t.gummerer@gmail.com> wrote:\n> Series of patches to introduce the index version 5 file format. This\n> series does not include any fancy stuff like partial loading or partial\n> writing yet, though it's possible to do that with the new format.\n\nI applied the series on top of master. I had to manually resolve\n09/16. You may want to rebase the series on master for the reroll\n(less work for Junio) and remove trailing whitespaces in the patches.\n\nAll tests passed (with v5 by default (*), I notice it now), which is\nwonderful. I'll have a closer look on the following days. Thank you\nfor working on this.\n\n(*) while it's good to run tests with v5 by default. I'm not sure if\nwe should make it by default in the next git release that comes with\nv5. For one thing it'll stop older gits from using shared repos. And\nwe're not sure whether v5 introduces significant overhead in common\nuse cases.\n-- \nDuy\n"},{"id":"196398","messageId":"878vdwcrmd.fsf@thomas.inf.ethz.ch","threadId":"31170","inReplyTo":"1343905326-23790-6-git-send-email-t.gummerer@gmail.com","subject":"Re: [PATCH 05/16] t2104: Don't fail when index version is 5","fromName":"Thomas Rast","fromEmail":"trast@inf.ethz.ch","sentAt":"2012-08-03T08:22:02Z","receivedAt":"2012-08-03T08:22:02Z","isPatch":true,"sender":{"key":"tr@thomasrast.ch","avatar":"https://avatars.githubusercontent.com/u/153510?v=4"},"body":"Thomas Gummerer <t.gummerer@gmail.com> writes:\n\n> The test t2104 currently checks if the index version is correctly\n> reduced to 2/increased to 3, when an entry need extended flags,\n> or doesn't use them anymore. Since index-v5 doesn't have extended\n> flags (the extended flags are part of the normal flags), we simply\n> add a check if the index version is 2/3 (whichever is correct for\n> that test) or 5.\n\nNext time we set a new index format as default (which might be when we\nmake v4 the default!), we'll have to patch this again.  Wouldn't it make\nmore sense to let them depend on a \"default index format is v2\"\nprerequisite?\n\n> -test_expect_success 'index is at version 2' '\n> -\ttest \"$(test-index-version < .git/index)\" = 2\n> +test_expect_success 'index is at version 2 or version 5' '\n> +\ttest \"$(test-index-version < .git/index)\" = 2 ||\n> +\ttest \"$(test-index-version < .git/index)\" = 5\n>  '\n\n-- \nThomas Rast\ntrast@{inf,student}.ethz.ch\n"},{"id":"196399","messageId":"87haskbcmi.fsf@thomas.inf.ethz.ch","threadId":"31170","inReplyTo":"1343905326-23790-12-git-send-email-t.gummerer@gmail.com","subject":"Re: [PATCH 11/16] Read cache-tree in index-v5","fromName":"Thomas Rast","fromEmail":"trast@inf.ethz.ch","sentAt":"2012-08-03T08:31:17Z","receivedAt":"2012-08-03T08:31:17Z","isPatch":true,"sender":{"key":"tr@thomasrast.ch","avatar":"https://avatars.githubusercontent.com/u/153510?v=4"},"body":"Thomas Gummerer <t.gummerer@gmail.com> writes:\n\n> Since the cache-tree data is saved as part of the directory data,\n> we have already read it, when we want to read the cache-tree. The\n> cache-tree then only has to be converted from the directory data.\n\nI think the first sentence is wrong.  You have already read it at the\nvery beginning of reading the index format, when you parsed the\ndirectory records, haven't you?\n\n> The cache-tree isn't lexically sorted, but after the pathlen at\n> each level, therefore the directories have to be reordered with\n> respect to the ondisk layout.\n\nI'm not a native speaker either, but I think this does't parse well.\nMaybe\n\n  The cache-tree data is arranged in a tree, with the children sorted by\n  pathlen at each node.  So we have to rebuild this format from the\n  on-disk directory list.\n\n> +\tfor (i = 0; i < subtree_nr; i++) {\n> +\t\tstruct cache_tree *sub;\n> +\t\tstruct cache_tree_sub *subtree;\n> +\t\tchar *buf, *name;\n> +\n> +\t\tname = \"\";\n> +\t\tbuf = strtok(down[i].de->pathname, \"/\");\n\nman 3 strtok says\n\n   Be cautious when using these functions.  If you do use them, note\n   that:\n\n   * These functions modify their first argument.\n\n   * These functions cannot be used on constant strings.\n\n   * The identity of the delimiting character is lost.\n\n   * The strtok() function uses a static buffer while parsing, so it's\n     not thread safe.  Use strtok_r() if this matters to you.\n\nI don't think the last point will be a problem, but what about modifying\nthe argument?\n\n-- \nThomas Rast\ntrast@{inf,student}.ethz.ch\n"},{"id":"196403","messageId":"87pq785odw.fsf@thomas.inf.ethz.ch","threadId":"31170","inReplyTo":"1343905326-23790-1-git-send-email-t.gummerer@gmail.com","subject":"Re: [RFC 0/16] Introduce index file format version 5","fromName":"Thomas Rast","fromEmail":"trast@student.ethz.ch","sentAt":"2012-08-03T09:13:47Z","receivedAt":"2012-08-03T09:13:47Z","isPatch":false,"sender":{"key":"tr@thomasrast.ch","avatar":"https://avatars.githubusercontent.com/u/153510?v=4"},"body":"Thomas Gummerer <t.gummerer@gmail.com> writes:\n\n> [PATCH 01/16] Modify cache_header to prepare for other index formats\n> [PATCH 02/16] Modify read functions to prepare for other index\n> [PATCH 03/16] Modify match_stat_basic to prepare for other index\n> [PATCH 04/16] Modify write functions to prepare for other index\n> [PATCH 05/16] t2104: Don't fail when index version is 5\n> [PATCH 06/16] t3700: sleep for 1 second, to avoid interfering with\n> [PATCH 07/16] Add documentation of the index-v5 file format\n> [PATCH 08/16] Make in-memory format aware of stat_crc\n> [PATCH 09/16] Read index-v5\n> [PATCH 10/16] Read resolve-undo data\n> [PATCH 11/16] Read cache-tree in index-v5\n> [PATCH 12/16] Write index-v5\n> [PATCH 13/16] Write index-v5 cache-tree data\n> [PATCH 14/16] Write resolve-undo data for index-v5\n> [PATCH 15/16] update-index.c: add a force-rewrite option\n> [PATCH 16/16] p0002-index.sh: add perf test for the index formats\n\nI haven't had time for more than a cursory look yet, but good job on the\nsplits.  This is a large improvement over what you had in Zurich!\n\nOne thing that you need to be more careful about is attribution of the\nsource code.  Credit is very important because it's the only thing\npeople get for their OSS work.  For some patches you received lots of\ninput and help by many people.  For example, the documentation patch\nthat casts the format in stone (or will, when it's finished), should\nhave \"Helped-by:\" for *at least* Michael, Junio, and Duy.  You should\ndig in the ML archives for other people who may have contributed ideas.\n\nAlso, anything that contains nontrivial code from me needs my S-o-b; off\nthe top of my head that's just 16/16, which AFAICS is even completely\nunchanged (!) and needs to come with a From (and my S-o-b).  (I'm not\ngoing to be anal about any of the work we did in Zurich, let's just\nclassify that as \"help\" like above.)\n\n-- \nThomas Rast\ntrast@{inf,student}.ethz.ch\n"},{"id":"196412","messageId":"20120803123431.GA3226@tgummerer.unibz.it","threadId":"31170","inReplyTo":"87pq785odw.fsf@thomas.inf.ethz.ch","subject":"Re: [RFC 0/16] Introduce index file format version 5","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2012-08-03T12:34:31Z","receivedAt":"2012-08-03T12:34:31Z","isPatch":false,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"On 08/03, Thomas Rast wrote:\n> Thomas Gummerer <t.gummerer@gmail.com> writes:\n> \n> > [PATCH 01/16] Modify cache_header to prepare for other index formats\n> > [PATCH 02/16] Modify read functions to prepare for other index\n> > [PATCH 03/16] Modify match_stat_basic to prepare for other index\n> > [PATCH 04/16] Modify write functions to prepare for other index\n> > [PATCH 05/16] t2104: Don't fail when index version is 5\n> > [PATCH 06/16] t3700: sleep for 1 second, to avoid interfering with\n> > [PATCH 07/16] Add documentation of the index-v5 file format\n> > [PATCH 08/16] Make in-memory format aware of stat_crc\n> > [PATCH 09/16] Read index-v5\n> > [PATCH 10/16] Read resolve-undo data\n> > [PATCH 11/16] Read cache-tree in index-v5\n> > [PATCH 12/16] Write index-v5\n> > [PATCH 13/16] Write index-v5 cache-tree data\n> > [PATCH 14/16] Write resolve-undo data for index-v5\n> > [PATCH 15/16] update-index.c: add a force-rewrite option\n> > [PATCH 16/16] p0002-index.sh: add perf test for the index formats\n> \n> I haven't had time for more than a cursory look yet, but good job on the\n> splits.  This is a large improvement over what you had in Zurich!\n\nThanks, it was easier when seeing the code, instead of just thinking\nabout them of the top of my head.\n\n> One thing that you need to be more careful about is attribution of the\n> source code.  Credit is very important because it's the only thing\n> people get for their OSS work.  For some patches you received lots of\n> input and help by many people.  For example, the documentation patch\n> that casts the format in stone (or will, when it's finished), should\n> have \"Helped-by:\" for *at least* Michael, Junio, and Duy.  You should\n> dig in the ML archives for other people who may have contributed ideas.\n> \n> Also, anything that contains nontrivial code from me needs my S-o-b; off\n> the top of my head that's just 16/16, which AFAICS is even completely\n> unchanged (!) and needs to come with a From (and my S-o-b).  (I'm not\n> going to be anal about any of the work we did in Zurich, let's just\n> classify that as \"help\" like above.)\n\nMy apologies, i forgot to add them. I'll make sure to include all\ncredits in the re-roll.\n"},{"id":"196413","messageId":"20120803124113.GB3226@tgummerer.unibz.it","threadId":"31170","inReplyTo":"87haskbcmi.fsf@thomas.inf.ethz.ch","subject":"Re: [PATCH 11/16] Read cache-tree in index-v5","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2012-08-03T12:41:13Z","receivedAt":"2012-08-03T12:41:13Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"On 08/03, Thomas Rast wrote:\n> Thomas Gummerer <t.gummerer@gmail.com> writes:\n> \n> > Since the cache-tree data is saved as part of the directory data,\n> > we have already read it, when we want to read the cache-tree. The\n> > cache-tree then only has to be converted from the directory data.\n> \n> I think the first sentence is wrong.  You have already read it at the\n> very beginning of reading the index format, when you parsed the\n> directory records, haven't you?\n\nYes, that's what I wanted to say, I'll rephrase this.\n\n> > The cache-tree isn't lexically sorted, but after the pathlen at\n> > each level, therefore the directories have to be reordered with\n> > respect to the ondisk layout.\n> \n> I'm not a native speaker either, but I think this does't parse well.\n> Maybe\n> \n>   The cache-tree data is arranged in a tree, with the children sorted by\n>   pathlen at each node.  So we have to rebuild this format from the\n>   on-disk directory list.\n\nThanks, that sounds better.\n\n> > +\tfor (i = 0; i < subtree_nr; i++) {\n> > +\t\tstruct cache_tree *sub;\n> > +\t\tstruct cache_tree_sub *subtree;\n> > +\t\tchar *buf, *name;\n> > +\n> > +\t\tname = \"\";\n> > +\t\tbuf = strtok(down[i].de->pathname, \"/\");\n> \n> man 3 strtok says\n> \n>    Be cautious when using these functions.  If you do use them, note\n>    that:\n> \n>    * These functions modify their first argument.\n> \n>    * These functions cannot be used on constant strings.\n> \n>    * The identity of the delimiting character is lost.\n> \n>    * The strtok() function uses a static buffer while parsing, so it's\n>      not thread safe.  Use strtok_r() if this matters to you.\n> \n> I don't think the last point will be a problem, but what about modifying\n> the argument?\n\nHrm the function is only called at the end of read_index_v5, after\nwhich we don't need the directory entries anymore. If we want to use\nthe directory entries for a new in-memory format later however, we\nshould probably change it. Maybe a comment for the cache_tree_convert_v5\nfunction would be enough?\n"},{"id":"196414","messageId":"20120803124239.GC3226@tgummerer.unibz.it","threadId":"31170","inReplyTo":"878vdwcrmd.fsf@thomas.inf.ethz.ch","subject":"Re: [PATCH 05/16] t2104: Don't fail when index version is 5","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2012-08-03T12:42:39Z","receivedAt":"2012-08-03T12:42:39Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"On 08/03, Thomas Rast wrote:\n> Thomas Gummerer <t.gummerer@gmail.com> writes:\n> \n> > The test t2104 currently checks if the index version is correctly\n> > reduced to 2/increased to 3, when an entry need extended flags,\n> > or doesn't use them anymore. Since index-v5 doesn't have extended\n> > flags (the extended flags are part of the normal flags), we simply\n> > add a check if the index version is 2/3 (whichever is correct for\n> > that test) or 5.\n> \n> Next time we set a new index format as default (which might be when we\n> make v4 the default!), we'll have to patch this again.  Wouldn't it make\n> more sense to let them depend on a \"default index format is v2\"\n> prerequisite?\n\nSounds good to me, since formats other than v[23] don't do anything\nin this test anyway.\n\n> > -test_expect_success 'index is at version 2' '\n> > -\ttest \"$(test-index-version < .git/index)\" = 2\n> > +test_expect_success 'index is at version 2 or version 5' '\n> > +\ttest \"$(test-index-version < .git/index)\" = 2 ||\n> > +\ttest \"$(test-index-version < .git/index)\" = 5\n> >  '\n> \n> -- \n> Thomas Rast\n> trast@{inf,student}.ethz.ch\n"},{"id":"196415","messageId":"20120803124632.GD3226@tgummerer.unibz.it","threadId":"31170","inReplyTo":"CACsJy8AQUwb8noutwdcN-6Piyo1rrj7_EUMjrXB-83J9AtM6kQ@mail.gmail.com","subject":"Re: [RFC 0/16] Introduce index file format version 5","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2012-08-03T12:46:32Z","receivedAt":"2012-08-03T12:46:32Z","isPatch":false,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"On 08/03, Nguyen Thai Ngoc Duy wrote:\n> On Thu, Aug 2, 2012 at 6:01 PM, Thomas Gummerer <t.gummerer@gmail.com> wrote:\n> > Series of patches to introduce the index version 5 file format. This\n> > series does not include any fancy stuff like partial loading or partial\n> > writing yet, though it's possible to do that with the new format.\n> \n> I applied the series on top of master. I had to manually resolve\n> 09/16. You may want to rebase the series on master for the reroll\n> (less work for Junio) and remove trailing whitespaces in the patches.\n\nThanks, I'll do that for the reroll.\n\n> All tests passed (with v5 by default (*), I notice it now), which is\n> wonderful. I'll have a closer look on the following days. Thank you\n> for working on this.\n> \n> (*) while it's good to run tests with v5 by default. I'm not sure if\n> we should make it by default in the next git release that comes with\n> v5. For one thing it'll stop older gits from using shared repos. And\n> we're not sure whether v5 introduces significant overhead in common\n> use cases.\n\nI didn't intent to make index v5 the format, I just oversaw the hunk\nthat was changing it. I completely agree that v5 should not be default\nyet to avoid any surprises, also for users of jgit, libgit2 etc.\n"},{"id":"196418","messageId":"7vwr1gar9x.fsf@alter.siamese.dyndns.org","threadId":"31170","inReplyTo":"878vdwcrmd.fsf@thomas.inf.ethz.ch","subject":"Re: [PATCH 05/16] t2104: Don't fail when index version is 5","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2012-08-03T16:12:26Z","receivedAt":"2012-08-03T16:12:26Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Thomas Rast <trast@inf.ethz.ch> writes:\n\n> Thomas Gummerer <t.gummerer@gmail.com> writes:\n>\n>> The test t2104 currently checks if the index version is correctly\n>> reduced to 2/increased to 3, when an entry need extended flags,\n>> or doesn't use them anymore. Since index-v5 doesn't have extended\n>> flags (the extended flags are part of the normal flags), we simply\n>> add a check if the index version is 2/3 (whichever is correct for\n>> that test) or 5.\n>\n> Next time we set a new index format as default (which might be when we\n> make v4 the default!), we'll have to patch this again.  Wouldn't it make\n> more sense to let them depend on a \"default index format is v2\"\n> prerequisite?\n\nMy preference is not to change the default index version for now,\nand for a test that specifically checks features of a particular\nindex version, force the index version near the beginning of test\nusing \"update-index --index-version $num\".\n\nFor t2104, I think forcing the index version to 2 at the beginning\nand not worry about v4 or later at all would be the right thing to\ndo.  That way, we will make sure older versions are still supported\nwith the new code that is capable of reading and writing newer ones.\n\n>> -test_expect_success 'index is at version 2' '\n>> -\ttest \"$(test-index-version < .git/index)\" = 2\n>> +test_expect_success 'index is at version 2 or version 5' '\n>> +\ttest \"$(test-index-version < .git/index)\" = 2 ||\n>> +\ttest \"$(test-index-version < .git/index)\" = 5\n>>  '\n"}]}