{"thread":{"id":"20432","subject":"[PATCH 5/5] full integration of rev-cache into git's revision walker, completed test suite","startedAt":"2009-08-06T09:55:50Z","lastAt":"2009-08-07T13:47:26Z","messageCount":3,"participants":["Nick Edelen","Sam Vilain"],"isPatch":true,"patchVersion":1,"patchTotal":5},"messages":[{"id":"119770","messageId":"op.ux8i7ceotdk399@sirnot.private","threadId":"20432","inReplyTo":null,"subject":"[PATCH 5/5] full integration of rev-cache into git's revision walker, completed test suite","fromName":"Nick Edelen","fromEmail":"sirnot@gmail.com","sentAt":"2009-08-06T09:55:50Z","receivedAt":"2009-08-06T09:55:50Z","isPatch":true,"sender":{"key":"sirnot@gmail.com","avatar":null},"body":"This last patch provides a working integration of rev-cache into the revision \nwalker, along with some touch-ups:\n - integration into revision walker and list-objects\n - addition of 'unique' field to commit objects, optionally initialized in \n   rev-cache with the objects introduced in that commit\n - tweak of object generation to take advantage of the 'unique' field\n - more fluid handling of damaged cache slices\n - numerous tests for both features from the previous patch, and the \nintegration's integrity\n\n'Integration' is rather broad -- a more detailed description follows for each \naspect:\n - rev-cache\nthe traversal mechanism is updated to handle many of the non-prune options \nrev-list does (date limiting, slop-handling, etc.), and is adjusted to allow \nfor non-fatal cache-traversal failures.\n\n - revision walker\nboth limited and unlimited traversal attempt to use the cache when possible, \nsmoothly falling back if it's not.\n\n - list-objects\nobject listing does not recurse into cached trees, and has been adjusted to \nguarantee commit-tag-tree-blob ordering.\n\nSigned-off-by: Nick Edelen <sirnot@gmail.com>\n\n---\n builtin-rev-cache.c       |   46 +++++++-\n commit.c                  |    2 +-\n commit.h                  |    1 +\n list-objects.c            |   49 ++++++-\n rev-cache.c               |  303 ++++++++++++++++++++++++++++++++++++++------\n revision.c                |   87 +++++++++++---\n t/t6015-rev-cache-list.sh |  120 ++++++++++++++++--\n 7 files changed, 529 insertions(+), 79 deletions(-)\n\ndiff --git a/builtin-rev-cache.c b/builtin-rev-cache.c\nindex 44916f5..05ee9dd 100755\n--- a/builtin-rev-cache.c\n+++ b/builtin-rev-cache.c\n@@ -3,6 +3,7 @@\n #include \"commit.h\"\n #include \"diff.h\"\n #include \"revision.h\"\n+#include \"list-objects.h\"\n \n #define DEFAULT_IGNORE_SLICE_SIZE\t\t5000000 /* in bytes */\n \n@@ -69,6 +70,43 @@ static int handle_add(int argc, const char *argv[]) /* args beyond this command\n \treturn 0;\n }\n \n+static void show_commit(struct commit *commit, void *data)\n+{\n+\tprintf(\"%s\\n\", sha1_to_hex(commit->object.sha1));\n+}\n+\n+static void show_object(struct object *obj, const struct name_path *path, const char *last)\n+{\n+\tprintf(\"%s\\n\", sha1_to_hex(obj->sha1));\n+}\n+\n+static int test_rev_list(int argc, const char *argv[])\n+{\n+\tstruct rev_info revs;\n+\tunsigned int flags = 0;\n+\tint i;\n+\t\n+\tinit_revisions(&revs, 0);\n+\t\n+\tfor (i = 0; i < argc; i++) {\n+\t\tif (!strcmp(argv[i], \"--not\"))\n+\t\t\tflags ^= UNINTERESTING;\n+\t\telse if (!strcmp(argv[i], \"--objects\"))\n+\t\t\trevs.tree_objects = revs.blob_objects = 1;\n+\t\telse\n+\t\t\thandle_revision_arg(argv[i], &revs, flags, 1);\n+\t}\n+\t\n+\tsetup_revisions(0, 0, &revs, 0);\n+\trevs.topo_order = 1;\n+\trevs.lifo = 1;\n+\tprepare_revision_walk(&revs);\n+\t\n+\ttraverse_commit_list(&revs, show_commit, show_object, 0);\n+\t\n+\treturn 0;\n+}\n+\n static int handle_walk(int argc, const char *argv[])\n {\n \tstruct commit *commit;\n@@ -117,17 +155,17 @@ static int handle_walk(int argc, const char *argv[])\n \tif (retval < 0)\n \t\treturn retval;\n \t\n-\tprintf(\"queue:\\n\");\n+\tfprintf(stderr, \"queue:\\n\");\n \twhile ((commit = pop_commit(&queue)) != 0) {\n \t\tprintf(\"%s\\n\", sha1_to_hex(commit->object.sha1));\n \t}\n \t\n-\tprintf(\"work:\\n\");\n+\tfprintf(stderr, \"work:\\n\");\n \twhile ((commit = pop_commit(&work)) != 0) {\n \t\tprintf(\"%s\\n\", sha1_to_hex(commit->object.sha1));\n \t}\n \t\n-\tprintf(\"pending:\\n\");\n+\tfprintf(stderr, \"pending:\\n\");\n \tfor (i = 0; i < revs.pending.nr; i++) {\n \t\tstruct object *obj = revs.pending.objects[i].item;\n \t\t\n@@ -235,6 +273,8 @@ int cmd_rev_cache(int argc, const char *argv[], const char *prefix)\n \t\tr = handle_walk(argc, argv);\n \telse if (!strcmp(arg, \"index\"))\n \t\tr = handle_index(argc, argv);\n+\telse if (!strcmp(arg, \"test\"))\n+\t\tr = test_rev_list(argc, argv);\n \telse\n \t\treturn handle_help();\n \t\ndiff --git a/commit.c b/commit.c\nindex afe504f..1b710c2 100644\n--- a/commit.c\n+++ b/commit.c\n@@ -252,7 +252,7 @@ int parse_commit_buffer(struct commit *item, void *buffer, unsigned long size)\n \t\t\t     sha1_to_hex(item->object.sha1));\n \titem->tree = lookup_tree(parent);\n \tbufptr += 46; /* \"tree \" + \"hex sha1\" + \"\\n\" */\n- \tpptr = &item->parents;\n+\tpptr = &item->parents;\n \twhile (pop_commit(pptr))\n \t\t; /* clear anything from cache */\n \ndiff --git a/commit.h b/commit.h\nindex 0529fcf..7c50a7f 100644\n--- a/commit.h\n+++ b/commit.h\n@@ -20,6 +20,7 @@ struct commit {\n \tstruct tree *tree;\n \tchar *buffer;\n \tunsigned long size;\n+\tstruct object_list *unique;\n };\n \n extern int save_commit_buffer;\ndiff --git a/list-objects.c b/list-objects.c\nindex 8953548..958c0f8 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -74,22 +74,33 @@ static void process_tree(struct rev_info *revs,\n \t\tdie(\"bad tree object\");\n \tif (obj->flags & (UNINTERESTING | SEEN))\n \t\treturn;\n-\tif (parse_tree(tree) < 0)\n-\t\tdie(\"bad tree object %s\", sha1_to_hex(obj->sha1));\n+\t\n \tobj->flags |= SEEN;\n \tshow(obj, path, name);\n+\tif (obj->flags & FACE_VALUE)\n+\t\treturn;\n+\t\n+\t/* traverse_commit_list is only used for enumeration purposes, \n+\t * ie. nothing relies on trees being parsed in this routine */\n+\tif (parse_tree(tree) < 0)\n+\t\tdie(\"bad tree object %s\", sha1_to_hex(obj->sha1));\n+\n \tme.up = path;\n \tme.elem = name;\n \tme.elem_len = strlen(name);\n-\n \tinit_tree_desc(&desc, tree->buffer, tree->size);\n \n \twhile (tree_entry(&desc, &entry)) {\n-\t\tif (S_ISDIR(entry.mode))\n+\t\tif (S_ISDIR(entry.mode)) {\n+\t\t\tstruct tree *subtree = lookup_tree(entry.sha1);\n+\t\t\tif (!subtree)\n+\t\t\t\tcontinue;\n+\t\t\t\n+\t\t\tsubtree->object.flags &= ~FACE_VALUE;\n \t\t\tprocess_tree(revs,\n-\t\t\t\t     lookup_tree(entry.sha1),\n+\t\t\t\t     subtree,\n \t\t\t\t     show, &me, entry.path);\n-\t\telse if (S_ISGITLINK(entry.mode))\n+\t\t} else if (S_ISGITLINK(entry.mode))\n \t\t\tprocess_gitlink(revs, entry.sha1,\n \t\t\t\t\tshow, &me, entry.path);\n \t\telse\n@@ -136,6 +147,7 @@ void mark_edges_uninteresting(struct commit_list *list,\n \n static void add_pending_tree(struct rev_info *revs, struct tree *tree)\n {\n+\ttree->object.flags &= ~FACE_VALUE;\n \tadd_pending_object(revs, &tree->object, \"\");\n }\n \n@@ -146,17 +158,27 @@ void traverse_commit_list(struct rev_info *revs,\n {\n \tint i;\n \tstruct commit *commit;\n+\tenum object_type what = OBJ_TAG;\n+\tchar face_value = 0;\n \n \twhile ((commit = get_revision(revs)) != NULL) {\n-\t\tadd_pending_tree(revs, commit->tree);\n+\t\tif (!(commit->object.flags & FACE_VALUE))\n+\t\t\tadd_pending_tree(revs, commit->tree);\n+\t\telse \n+\t\t\tface_value = 1;\n \t\tshow_commit(commit, data);\n \t}\n+\t\n+loop_objects:\n \tfor (i = 0; i < revs->pending.nr; i++) {\n \t\tstruct object_array_entry *pending = revs->pending.objects + i;\n \t\tstruct object *obj = pending->item;\n \t\tconst char *name = pending->name;\n \t\tif (obj->flags & (UNINTERESTING | SEEN))\n \t\t\tcontinue;\n+\t\tif (obj->type != what && face_value)\n+\t\t\tcontinue;\n+\t\t\n \t\tif (obj->type == OBJ_TAG) {\n \t\t\tobj->flags |= SEEN;\n \t\t\tshow_object(obj, NULL, name);\n@@ -175,6 +197,19 @@ void traverse_commit_list(struct rev_info *revs,\n \t\tdie(\"unknown pending object %s (%s)\",\n \t\t    sha1_to_hex(obj->sha1), name);\n \t}\n+\tif (face_value) {\n+\t\tswitch (what) {\n+\t\tcase OBJ_TAG : \n+\t\t\twhat = OBJ_TREE;\n+\t\t\tgoto loop_objects;\n+\t\tcase OBJ_TREE : \n+\t\t\twhat = OBJ_BLOB;\n+\t\t\tgoto loop_objects;\n+\t\tdefault : \n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\t\n \tif (revs->pending.nr) {\n \t\tfree(revs->pending.objects);\n \t\trevs->pending.nr = 0;\ndiff --git a/rev-cache.c b/rev-cache.c\nindex 4cda858..e6808e7 100755\n--- a/rev-cache.c\n+++ b/rev-cache.c\n@@ -65,14 +65,20 @@ struct object_entry {\n \t/* size */\n };\n \n+struct bad_slice {\n+\tunsigned char sha1[20];\n+\tstruct bad_slice *next;\n+};\n+\n /* list resembles pack index format */\n static uint32_t fanout[0xff + 2];\n \n static unsigned char *idx_map;\n static int idx_size;\n static struct index_header idx_head;\n+static char no_idx, save_unique, add_to_pending;\n+static struct bad_slice *bad_slices;\n static unsigned char *idx_caches;\n-static char no_idx;\n \n static struct strbuf *g_buffer;\n \n@@ -91,10 +97,36 @@ static struct strbuf *g_buffer;\n #define ACTUAL_OBJECT_ENTRY_SIZE(e)\t\t(OE_SIZE + PATH_SIZE((e)->merge_nr + (e)->split_nr) + (e)->size_size)\n #define ENTRY_SIZE_OFFSET(e)\t\t\t(ACTUAL_OBJECT_ENTRY_SIZE(e) - (e)->size_size)\n \n-#define SLOP\t\t5\n+#define SLOP\t\t\t5\n+\n+#define HAS_UNIQUES\t\tFACE_VALUE\n \n /* initialization */\n \n+static void mark_bad_slice(unsigned char *sha1)\n+{\n+\tstruct bad_slice *bad;\n+\t\n+\tbad = xcalloc(sizeof(struct bad_slice), 1);\n+\thashcpy(bad->sha1, sha1);\n+\t\n+\tbad->next = bad_slices;\n+\tbad_slices = bad;\n+}\n+\n+static int is_bad_slice(unsigned char *sha1)\n+{\n+\tstruct bad_slice *bad = bad_slices;\n+\t\n+\twhile (bad) {\n+\t\tif (!hashcmp(bad->sha1, sha1))\n+\t\t\treturn 1;\n+\t\tbad = bad->next;\n+\t}\n+\t\n+\treturn 0;\n+}\n+\n static int get_index_head(unsigned char *map, int len, struct index_header *head, uint32_t *fanout, unsigned char **caches)\n {\n \tstruct index_header whead;\n@@ -210,6 +242,7 @@ static struct index_entry *search_index(unsigned char *sha1)\n unsigned char *get_cache_slice(struct commit *commit)\n {\n \tstruct index_entry *ie;\n+\tunsigned char *sha1;\n \t\n \tif (!idx_map) {\n \t\tif (no_idx)\n@@ -221,8 +254,13 @@ unsigned char *get_cache_slice(struct commit *commit)\n \t\treturn 0;\n \t\n \tie = search_index(commit->object.sha1);\n-\tif (ie && ie->cache_index < idx_head.cache_nr)\n-\t\treturn idx_caches + ie->cache_index * 20;\n+\tif (ie && ie->cache_index < idx_head.cache_nr) {\n+\t\tsha1 = idx_caches + ie->cache_index * 20;\n+\t\t\n+\t\tif (is_bad_slice(sha1))\n+\t\t\treturn 0;\n+\t\treturn sha1;\n+\t}\n \t\n \treturn 0;\n }\n@@ -232,8 +270,29 @@ unsigned char *get_cache_slice(struct commit *commit)\n \n static unsigned long decode_size(unsigned char *str, int len);\n \n+/* on failure */\n+static void restore_commit(struct commit *commit)\n+{\n+\tif (commit->unique) {\n+\t\tfree(commit->unique);\n+\t\tcommit->unique = 0;\n+\t}\n+\t\n+\tcommit->object.flags &= ~(ADDED | SEEN | FACE_VALUE);\n+\t\n+\tif (!commit->object.parsed) {\n+\t\twhile (pop_commit(&commit->parents)) \n+\t\t\t;\n+\t\t\n+\t\tparse_commit(commit);\n+\t}\n+\t\n+}\n+\n static void handle_noncommit(struct rev_info *revs, struct commit *commit, struct object_entry *entry)\n {\n+\tstatic struct commit *last_commit = 0;\n+\tstatic struct object_list **last_unique = 0;\n \tstruct blob *blob;\n \tstruct tree *tree;\n \tstruct object *obj;\n@@ -271,11 +330,27 @@ static void handle_noncommit(struct rev_info *revs, struct commit *commit, struc\n \t\treturn;\n \t}\n \t\n+\t/* add to unique list if we're not an end */\n+\tif (save_unique && (commit->object.flags & FACE_VALUE)) {\n+\t\tif (last_commit != commit) {\n+\t\t\tlast_commit = commit;\n+\t\t\tlast_unique = 0;\n+\t\t}\n+\t\t\n+\t\tif (!last_unique)\n+\t\t\tlast_unique = &commit->unique;\n+\t\t\n+\t\tobject_list_append(obj, last_unique);\n+\t\tlast_unique = &(*last_unique)->next;\n+\t}\n+\t\n \tobj->flags |= FACE_VALUE;\n-\tadd_pending_object(revs, obj, \"\");\n+\tif (add_to_pending)\n+\t\tadd_pending_object(revs, obj, \"\");\n }\n \n-static int setup_traversal(struct cache_slice_header *head, unsigned char *map, struct commit *commit, struct commit_list **work)\n+static int setup_traversal(struct cache_slice_header *head, unsigned char *map, struct commit *commit, struct commit_list **work, \n+\tstruct commit_list **unwork, int *ipath_nr, int *upath_nr, char *ioutside)\n {\n \tstruct index_entry *iep;\n \tstruct object_entry *oep;\n@@ -284,6 +359,11 @@ static int setup_traversal(struct cache_slice_header *head, unsigned char *map,\n \t\n \tiep = search_index(commit->object.sha1);\n \toep = OE_CAST(map + ntohl(iep->pos));\n+\tif (commit->object.flags & UNINTERESTING) {\n+\t\t++*upath_nr;\n+\t\toep->uninteresting = 1;\n+\t} else\n+\t\t++*ipath_nr;\n \toep->include = 1;\n \tretval = ntohl(iep->pos);\n \t\n@@ -299,10 +379,14 @@ static int setup_traversal(struct cache_slice_header *head, unsigned char *map,\n \t\t/* is this in our cache slice? */\n \t\tiep = search_index(obj->sha1);\n \t\tif (!iep || hashcmp(idx_caches + iep->cache_index * 20, head->sha1)) {\n+\t\t\t/* there are interesing objects outside the slice */\n+\t\t\tif (!(obj->flags & UNINTERESTING))\n+\t\t\t\t*ioutside = 1;\n+\t\t\t\n \t\t\tprev = wp;\n \t\t\twp = wp->next;\n \t\t\twpp = &wp;\n- \t\t\tcontinue;\n+\t\t\tcontinue;\n \t\t}\n \t\t\n \t\tt = ntohl(iep->pos);\n@@ -313,11 +397,20 @@ static int setup_traversal(struct cache_slice_header *head, unsigned char *map,\n \t\tif (t < retval)\n \t\t\tretval = t;\n \t\t\n+\t\t/* count even if not in slice so we can stop enumerating if possible */\n+\t\tif (obj->flags & UNINTERESTING)\n+\t\t\t++*upath_nr;\n+\t\telse\n+\t\t\t++*ipath_nr;\n+\t\t\n \t\t/* remove from work list */\n \t\tco = pop_commit(wpp);\n \t\twp = *wpp;\n \t\tif (prev)\n \t\t\tprev->next = wp;\n+\t\t\n+\t\t/* ...and store in temp list so we can restore work on failure */\n+\t\tcommit_list_insert(co, unwork);\n \t}\n \t\n \treturn retval;\n@@ -334,16 +427,21 @@ static int traverse_cache_slice_1(struct cache_slice_header *head, unsigned char\n \tunsigned long *date_so_far, int *slop_so_far, \n \tstruct commit_list ***queue, struct commit_list **work)\n {\n-\tstruct commit_list *insert_cache = 0;\n+\tstruct commit_list *insert_cache = 0, *myq = 0, **myqp = &myq, *mywork = 0, **myworkp = &mywork, *unwork = 0;\n \tstruct commit **last_objects, *co;\n-\tint i, total_path_nr = head->path_nr, retval = -1;\n-\tchar consume_children = 0;\n+\tunsigned long date = date_so_far ? *date_so_far : ~0ul;\n+\tint i, ipath_nr = 0, upath_nr = 0, orig_obj_nr = 0, \n+\t\ttotal_path_nr = head->path_nr, retval = -1, slop = slop_so_far ? *slop_so_far : SLOP;\n+\tchar consume_children = 0, ioutside = 0;\n \tunsigned char *paths;\n \t\n+\t/* take note in case we need to regress */\n+\torig_obj_nr = revs->pending.nr;\n+\t\n \tpaths = xcalloc(total_path_nr, PATH_WIDTH);\n \tlast_objects = xcalloc(total_path_nr, sizeof(struct commit *));\n \t\n-\ti = setup_traversal(head, map, commit, work);\n+\ti = setup_traversal(head, map, commit, work, &unwork, &ipath_nr, &upath_nr, &ioutside);\n \t\n \t/* i already set */\n \twhile (i < head->size) {\n@@ -386,6 +484,7 @@ static int traverse_cache_slice_1(struct cache_slice_header *head, unsigned char\n \t\t\n \t\tif ((paths[path] & IPATH) && (paths[path] & UPATH)) {\n \t\t\tpaths[path] = UPATH;\n+\t\t\tipath_nr--;\n \t\t\t\n \t\t\t/* mark edge */\n \t\t\tif (last_objects[path]) {\n@@ -396,6 +495,7 @@ static int traverse_cache_slice_1(struct cache_slice_header *head, unsigned char\n \t\t\t\tlast_objects[path]->object.flags &= ~FACE_VALUE;\n \t\t\t\tlast_objects[path] = 0;\n \t\t\t}\n+\t\t\tobj->flags |= BOUNDARY;\n \t\t}\n \t\t\n \t\t/* now we gotta re-assess the whole interesting thing... */\n@@ -419,8 +519,10 @@ static int traverse_cache_slice_1(struct cache_slice_header *head, unsigned char\n \t\t\t\t\t\tlast_objects[p]->object.flags &= ~FACE_VALUE;\n \t\t\t\t\t\tlast_objects[p] = 0;\n \t\t\t\t\t}\n-\t\t\t\t} else if (last_objects[p] && !last_objects[p]->object.parsed)\n+\t\t\t\t\tobj->flags |= BOUNDARY;\n+\t\t\t\t} else if (last_objects[p] && !last_objects[p]->object.parsed) {\n \t\t\t\t\tcommit_list_insert(co, &last_objects[p]->parents);\n+\t\t\t\t}\n \t\t\t\t\n \t\t\t\t/* can't close a merge path until all are parents have been encountered */\n \t\t\t\tif (GET_COUNT(paths[p])) {\n@@ -430,44 +532,90 @@ static int traverse_cache_slice_1(struct cache_slice_header *head, unsigned char\n \t\t\t\t\t\tcontinue;\n \t\t\t\t}\n \t\t\t\t\n+\t\t\t\tif (paths[p] & IPATH)\n+\t\t\t\t\tipath_nr--;\n+\t\t\t\telse\n+\t\t\t\t\tupath_nr--;\n+\t\t\t\t\n \t\t\t\tpaths[p] = 0;\n \t\t\t\tlast_objects[p] = 0;\n \t\t\t}\n \t\t}\n \t\t\n \t\t/* make topo relations */\n-\t\tif (last_objects[path] && !last_objects[path]->object.parsed)\n+\t\tif (last_objects[path] && !last_objects[path]->object.parsed) {\n \t\t\tcommit_list_insert(co, &last_objects[path]->parents);\n+\t\t}\n+\t\t\n+\t\t/* we've been here already */\n+\t\tif (obj->flags & ADDED) {\n+\t\t\tif (entry->uninteresting && !(obj->flags & UNINTERESTING)) {\n+\t\t\t\tobj->flags |= UNINTERESTING;\n+\t\t\t\tmark_parents_uninteresting(co);\n+\t\t\t\tupath_nr--;\n+\t\t\t} else if (!entry->uninteresting)\n+\t\t\t\tipath_nr--;\n+\t\t\t\n+\t\t\tpaths[path] = 0;\n+\t\t\tcontinue;\n+\t\t}\n \t\t\n \t\t/* initialize commit */\n \t\tif (!entry->is_end) {\n \t\t\tco->date = ntohl(entry->date);\n- \t\t\tobj->flags |= ADDED | FACE_VALUE;\n+\t\t\tobj->flags |= ADDED | FACE_VALUE;\n \t\t} else\n \t\t\tparse_commit(co);\n \t\t\n \t\tobj->flags |= SEEN;\n- \t\t\n- \t\tif (entry->uninteresting)\n- \t\t\tobj->flags |= UNINTERESTING;\n+\t\t\n+\t\tif (entry->uninteresting)\n+\t\t\tobj->flags |= UNINTERESTING;\n+\t\telse if (co->date < date)\n+\t\t\tdate = co->date;\n \t\t\n \t\t/* we need to know what the edges are */\n \t\tlast_objects[path] = co;\n \t\t\n \t\t/* add to list */\n-\t\tif (!(obj->flags & UNINTERESTING) || revs->show_all) {\n-\t\t\tif (entry->is_end)\n-\t\t\t\tinsert_by_date_cached(co, work, insert_cache, &insert_cache);\n-\t\t\telse\n-\t\t\t\t*queue = &commit_list_insert(co, *queue)->next;\n+\t\tif (slop && !(revs->min_age != -1 && co->date > revs->min_age)) {\n+\t\t\t\n+\t\t\tif (!(obj->flags & UNINTERESTING) || revs->show_all) {\n+\t\t\t\tif (entry->is_end)\n+\t\t\t\t\tmyworkp = &commit_list_insert(co, myworkp)->next;\n+\t\t\t\telse\n+\t\t\t\t\tmyqp = &commit_list_insert(co, myqp)->next;\n+\t\t\t\t\n+\t\t\t\t/* add children to list as well */\n+\t\t\t\tif (obj->flags & UNINTERESTING)\n+\t\t\t\t\tconsume_children = 0;\n+\t\t\t\telse\n+\t\t\t\t\tconsume_children = 1;\n+\t\t\t}\n \t\t\t\n-\t\t\t/* add children to list as well */\n-\t\t\tif (obj->flags & UNINTERESTING)\n-\t\t\t\tconsume_children = 0;\n-\t\t\telse \n-\t\t\t\tconsume_children = 1;\n \t\t}\n \t\t\n+\t\t/* should we continue? */\n+\t\tif (!slop) {\n+\t\t\tif (!upath_nr) {\n+\t\t\t\tbreak;\n+\t\t\t} else if (ioutside || revs->show_all) {\n+\t\t\t\t/* pass it back to rev-list\n+\t\t\t\t * we purposely ignore everything outside this cache, so we don't needlessly traverse the whole \n+\t\t\t\t * thing on uninteresting, but that does mean that we may need to bounce back \n+\t\t\t\t * and forth a few times with rev-list */\n+\t\t\t\tmyworkp = &commit_list_insert(co, myworkp)->next;\n+\t\t\t\t\n+\t\t\t\tpaths[path] = 0;\n+\t\t\t\tupath_nr--;\n+\t\t\t} else {\n+\t\t\t\tbreak;\n+\t\t\t}\n+\t\t} else if (!ipath_nr && co->date <= date)\n+\t\t\tslop--;\n+\t\telse\n+\t\t\tslop = SLOP;\n+\t\t\n \t\t/* open parents */\n \t\tif (entry->merge_nr) {\n \t\t\tint j, off = index + OE_SIZE;\n@@ -482,6 +630,11 @@ static int traverse_cache_slice_1(struct cache_slice_header *head, unsigned char\n \t\t\t\tif (paths[p] & flag)\n \t\t\t\t\tcontinue;\n \t\t\t\t\n+\t\t\t\tif (flag == IPATH)\n+\t\t\t\t\tipath_nr++;\n+\t\t\t\telse\n+\t\t\t\t\tupath_nr++;\n+\t\t\t\t\n \t\t\t\tpaths[p] |= flag;\n \t\t\t}\n \t\t\t\n@@ -491,12 +644,55 @@ static int traverse_cache_slice_1(struct cache_slice_header *head, unsigned char\n \t\t\n \t}\n \t\n+\tif (date_so_far)\n+\t\t*date_so_far = date;\n+\tif (slop_so_far)\n+\t\t*slop_so_far = slop;\n \tretval = 0;\n \t\n+\t/* success: attach to given lists */\n+\tif (myqp != &myq) {\n+\t\t**queue = myq;\n+\t\t*queue = myqp;\n+\t}\n+\t\n+\twhile ((co = pop_commit(&mywork)) != 0) {\n+\t\tinsert_by_date_cached(co, work, insert_cache, &insert_cache);\n+\t}\n+\t\n+\t/* free backup */\n+\twhile (pop_commit(&unwork)) \n+\t\t;\n+\t\n end:\n \tfree(paths);\n \tfree(last_objects);\n \t\n+\t/* failure: restore work to previous condition\n+\t * (cache corruption should *not* be fatal) */\n+\tif (retval) {\n+\t\twhile ((co = pop_commit(&unwork)) != 0) {\n+\t\t\trestore_commit(co);\n+\t\t\tco->object.flags |= SEEN;\n+\t\t\tinsert_by_date(co, work);\n+\t\t}\n+\t\t\n+\t\t/* free lists */\n+\t\twhile ((co = pop_commit(&myq)) != 0)\n+\t\t\trestore_commit(co);\n+\t\t\n+\t\twhile ((co = pop_commit(&mywork)) != 0)\n+\t\t\trestore_commit(co);\n+\t\t\n+\t\t/* truncate object array */\n+\t\tfor (i = orig_obj_nr; i < revs->pending.nr; i++) {\n+\t\t\tstruct object *obj = revs->pending.objects[i].item;\n+\t\t\t\n+\t\t\tobj->flags &= ~FACE_VALUE;\n+\t\t}\n+\t\trevs->pending.nr = orig_obj_nr;\n+\t}\n+\t\n \treturn retval;\n }\n \n@@ -545,6 +741,8 @@ int traverse_cache_slice(struct rev_info *revs,\n \t\n \t/* load options */\n \trci = &revs->rev_cache_info;\n+\tsave_unique = rci->save_unique;\n+\tadd_to_pending = rci->add_to_pending;\n \t\n \tmemset(&head, 0, sizeof(struct cache_slice_header));\n \t\n@@ -568,6 +766,10 @@ end:\n \tif (fd != -1)\n \t\tclose(fd);\n \t\n+\t/* remember this! */\n+\tif (retval)\n+\t\tmark_bad_slice(cache_sha1);\n+\t\n \treturn retval;\n }\n \n@@ -762,6 +964,7 @@ static void handle_paths(struct commit *commit, struct object_entry *object, str\n \t\tif (pt->commit == commit) {\n \t\t\tif (paths[pt->path] != PATH_IN_USE)\n \t\t\t\tpaths[pt->path]--;\n+\t\t\t\n \t\t\tremove_path_track(ppt, 0);\n \t\t\tpt = *ppt;\n \t\t} else {\n@@ -1021,6 +1224,21 @@ static int add_unique_objects(struct commit *commit)\n \tint i, j, next;\n \tchar is_first = 1;\n \t\n+\t/* but wait!  is this itself from a slice? */\n+\tif (commit->unique) {\n+\t\tstruct object_list *olist;\n+\t\t\n+\t\tolist = commit->unique;\n+\t\ti = 0;\n+\t\twhile (olist) {\n+\t\t\tadd_object_entry(olist->item->sha1, 0, 0, 0);\n+\t\t\ti++;\n+\t\t\tolist = olist->next;\n+\t\t}\n+\t\t\n+\t\treturn i;\n+\t}\n+\t\n \t/* ...no, calculate unique objects */\n \tstrbuf_init(&os, 0);\n \tstrbuf_init(&ost, 0);\n@@ -1074,10 +1292,13 @@ static int add_unique_objects(struct commit *commit)\n \tfor (i = 0; i < os.len; i += 20)\n \t\tadd_object_entry((unsigned char *)(os.buf + i), 0, 0, 0);\n \t\n+\t/* last but not least, the main tree */\n+\tadd_object_entry(commit->tree->object.sha1, 0, 0, 0);\n+\t\n \tstrbuf_release(&ost);\n \tstrbuf_release(&os);\n \t\n-\treturn i / 20;\n+\treturn i / 20 + 1;\n }\n \n static void init_revcache_directory(void)\n@@ -1196,14 +1417,8 @@ int make_cache_slice(struct rev_cache_info *rci,\n \t\tadd_object_entry(0, &object, &merge_paths, &split_paths);\n \t\tobject_nr++;\n \t\t\n-\t\tif (rci->objects && !(commit->object.flags & TREESAME)) {\n-\t\t\t/* add all unique children for this commit */\n-\t\t\tadd_object_entry(commit->tree->object.sha1, 0, 0, 0);\n-\t\t\tobject_nr++;\n-\t\t\t\n-\t\t\tif (!object.is_end)\n-\t\t\t\tobject_nr += add_unique_objects(commit);\n-\t\t}\n+\t\tif (rci->objects && !(commit->object.flags & TREESAME) && !object.is_end)\n+\t\t\tobject_nr += add_unique_objects(commit);\n \t\t\n \t\t/* print every ~1MB or so */\n \t\tif (buffer.len > 1000000) {\n@@ -1481,7 +1696,6 @@ void starts_from_slices(struct rev_info *revs, unsigned int flags, unsigned char\n \t\n }\n \n-\n /* the most work-intensive attributes in the cache are the unique objects and size, both \n  * of which can be re-used.  although path structures will be isomorphic, path generation is \n  * not particularly expensive, and at any rate we need to re-sort the commits */\n@@ -1527,7 +1741,7 @@ int coagulate_cache_slices(struct rev_cache_info *rci, struct rev_info *revs)\n \t\t\t\t}\n \t\t\t}\n \t\t\t\n-\t\t\tstring_list_insert(base, &files);\n+\t\t\tstring_list_insert(de->d_name, &files);\n \t\t}\n \t\t\n \t\tclosedir(dirh);\n@@ -1548,15 +1762,20 @@ int coagulate_cache_slices(struct rev_cache_info *rci, struct rev_info *revs)\n \tfor (i = 0; i < files.nr; i++) {\n \t\tchar *name = files.items[i].string;\n \t\t\n-\t\tfprintf(stderr, \"removing %s\\n\", name);\n-\t\tunlink_or_warn(name);\n+\t\t/* in the odd case of only having one cache slice we effectively just remaking the index... */\n+\t\tif (strlen(name) >= 40 && !strncmp(name, sha1_to_hex(cache_sha1), 40))\n+\t\t\tcontinue;\n+\t\t\n+\t\tstrncpy(base + baselen + 1, name, sizeof(base) - baselen - 1);\n+\t\tfprintf(stderr, \"removing %s\\n\", base);\n+\t\tunlink_or_warn(base);\n \t}\n \t\n \tstring_list_clear(&files, 0);\n \t\n \tfd = open(git_path(\"rev-cache/%s\", sha1_to_hex(cache_sha1)), O_RDWR);\n \tif (fd < 0 || fstat(fd, &fi))\n-\t\tdie(\"what?  I can't open/query the cache I just generated\");\n+\t\tdie(\"what?  I can't open/query the cache I just generated\\n (sha1: %s)\", sha1_to_hex(cache_sha1));\n \t\n \tif (make_cache_index(rci, cache_sha1, fd, fi.st_size) < 0)\n \t\tdie(\"can't make new index\");\n@@ -1610,4 +1829,4 @@ int regenerate_cache_index(struct rev_cache_info *rci)\n \t}\n \t\n \treturn 0;\n-}\n\\ No newline at end of file\n+}\ndiff --git a/revision.c b/revision.c\nindex 485bf72..e76fd1d 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -638,6 +638,8 @@ static int limit_list(struct rev_info *revs)\n \tstruct commit_list *list = revs->commits;\n \tstruct commit_list *newlist = NULL;\n \tstruct commit_list **p = &newlist;\n+\tunsigned char *cache_sha1;\n+\tchar used_cache;\n \n \twhile (list) {\n \t\tstruct commit_list *entry = list;\n@@ -650,24 +652,39 @@ static int limit_list(struct rev_info *revs)\n \n \t\tif (revs->max_age != -1 && (commit->date < revs->max_age))\n \t\t\tobj->flags |= UNINTERESTING;\n-\t\tif (add_parents_to_list(revs, commit, &list, NULL) < 0)\n-\t\t\treturn -1;\n-\t\tif (obj->flags & UNINTERESTING) {\n-\t\t\tmark_parents_uninteresting(commit);\n-\t\t\tif (revs->show_all)\n-\t\t\t\tp = &commit_list_insert(commit, p)->next;\n-\t\t\tslop = still_interesting(list, date, slop);\n-\t\t\tif (slop)\n+\t\t\n+\t\t/* rev-cache to the rescue!!! */\n+\t\tused_cache = 0;\n+\t\tif (!revs->dont_cache_me && !(obj->flags & ADDED)) {\n+\t\t\tcache_sha1 = get_cache_slice(commit);\n+\t\t\tif (cache_sha1) {\n+\t\t\t\tif (traverse_cache_slice(revs, cache_sha1, commit, &date, &slop, &p, &list) < 0)\n+\t\t\t\t\tused_cache = 0;\n+\t\t\t\telse\n+\t\t\t\t\tused_cache = 1;\n+\t\t\t}\n+\t\t}\n+\t\t\n+\t\tif (!used_cache) {\n+\t\t\tif (add_parents_to_list(revs, commit, &list, NULL) < 0)\n+\t\t\t\treturn -1;\n+\t\t\tif (obj->flags & UNINTERESTING) {\n+\t\t\t\tmark_parents_uninteresting(commit); /* ME: why? */\n+\t\t\t\tif (revs->show_all)\n+\t\t\t\t\tp = &commit_list_insert(commit, p)->next;\n+\t\t\t\tslop = still_interesting(list, date, slop);\n+\t\t\t\tif (slop > 0)\n+\t\t\t\t\tcontinue;\n+\t\t\t\t/* If showing all, add the whole pending list to the end */\n+\t\t\t\tif (revs->show_all)\n+\t\t\t\t\t*p = list;\n+\t\t\t\tbreak;\n+\t\t\t}\n+\t\t\tif (revs->min_age != -1 && (commit->date > revs->min_age))\n \t\t\t\tcontinue;\n-\t\t\t/* If showing all, add the whole pending list to the end */\n-\t\t\tif (revs->show_all)\n-\t\t\t\t*p = list;\n-\t\t\tbreak;\n+\t\t\tdate = commit->date;\n+\t\t\tp = &commit_list_insert(commit, p)->next;\n \t\t}\n-\t\tif (revs->min_age != -1 && (commit->date > revs->min_age))\n-\t\t\tcontinue;\n-\t\tdate = commit->date;\n-\t\tp = &commit_list_insert(commit, p)->next;\n \n \t\tshow = show_early_output;\n \t\tif (!show)\n@@ -813,6 +830,8 @@ void init_revisions(struct rev_info *revs, const char *prefix)\n \t\trevs->diffopt.prefix = prefix;\n \t\trevs->diffopt.prefix_length = strlen(prefix);\n \t}\n+\t\n+\tinit_rci(&revs->rev_cache_info);\n }\n \n static void add_pending_commit_list(struct rev_info *revs,\n@@ -1372,6 +1391,11 @@ int setup_revisions(int argc, const char **argv, struct rev_info *revs, const ch\n \tif (revs->reflog_info && revs->graph)\n \t\tdie(\"cannot combine --walk-reflogs with --graph\");\n \n+\t/* limits on caching\n+\t * todo: implement this functionality */\n+\tif (revs->prune || revs->diff)\n+\t\trevs->dont_cache_me = 1;\n+\n \treturn left;\n }\n \n@@ -1654,6 +1678,8 @@ static int commit_match(struct commit *commit, struct rev_info *opt)\n {\n \tif (!opt->grep_filter.pattern_list)\n \t\treturn 1;\n+\tif (!commit->object.parsed)\n+\t\tparse_commit(commit);\n \treturn grep_buffer(&opt->grep_filter,\n \t\t\t   NULL, /* we say nothing, not even filename */\n \t\t\t   commit->buffer, strlen(commit->buffer));\n@@ -1706,6 +1732,7 @@ static struct commit *get_revision_1(struct rev_info *revs)\n \tdo {\n \t\tstruct commit_list *entry = revs->commits;\n \t\tstruct commit *commit = entry->item;\n+\t\tstruct object *obj = &commit->object;\n \n \t\trevs->commits = entry->next;\n \t\tfree(entry);\n@@ -1722,11 +1749,39 @@ static struct commit *get_revision_1(struct rev_info *revs)\n \t\t\tif (revs->max_age != -1 &&\n \t\t\t    (commit->date < revs->max_age))\n \t\t\t\tcontinue;\n+\t\t\t\n+\t\t\tif (!revs->dont_cache_me) {\n+\t\t\t\tstruct commit_list *queue = 0, **queuep = &queue;;\n+\t\t\t\tunsigned char *cache_sha1;\n+\t\t\t\t\n+\t\t\t\tif (obj->flags & ADDED)\n+\t\t\t\t\tgoto skip_parenting;\n+\t\t\t\t\n+\t\t\t\tcache_sha1 = get_cache_slice(commit);\n+\t\t\t\tif (cache_sha1) {\n+\t\t\t\t\tif (!traverse_cache_slice(revs, cache_sha1, commit, 0, 0, &queuep, &revs->commits)) {\n+\t\t\t\t\t\tstruct commit_list *work = revs->commits;\n+\t\t\t\t\t\t\n+\t\t\t\t\t\t/* attach queue to end of ->commits */\n+\t\t\t\t\t\twhile (work && work->next)\n+\t\t\t\t\t\t\twork = work->next;\n+\t\t\t\t\t\t\n+\t\t\t\t\t\tif (work)\n+\t\t\t\t\t\t\twork->next = queue;\n+\t\t\t\t\t\telse\n+\t\t\t\t\t\t\trevs->commits = queue;\n+\t\t\t\t\t\t\n+\t\t\t\t\t\tgoto skip_parenting;\n+\t\t\t\t\t}\n+\t\t\t\t}\n+\t\t\t}\n+\t\t\t\n \t\t\tif (add_parents_to_list(revs, commit, &revs->commits, NULL) < 0)\n \t\t\t\tdie(\"Failed to traverse parents of commit %s\",\n \t\t\t\t    sha1_to_hex(commit->object.sha1));\n \t\t}\n \n+skip_parenting:\n \t\tswitch (simplify_commit(revs, commit)) {\n \t\tcase commit_ignore:\n \t\t\tcontinue;\ndiff --git a/t/t6015-rev-cache-list.sh b/t/t6015-rev-cache-list.sh\nindex 9cd722a..bfd9525 100755\n--- a/t/t6015-rev-cache-list.sh\n+++ b/t/t6015-rev-cache-list.sh\n@@ -33,7 +33,8 @@ test_expect_success 'init repo' '\n \techo omg >smoke/bong && \n \tgit add . && \n \tgit commit -m \"omg\" && \n-\t\n+\n+\tsleep 2 && \n \tgit branch b4 && \n \tgit checkout b4 && \n \techo shazam >file8 && \n@@ -42,9 +43,9 @@ test_expect_success 'init repo' '\n \tgit merge -m \"merge b2\" b2 && \n \t\n \techo bam >smoke/pipe && \n-\tgit add .\n+\tgit add . && \n \tgit commit -m \"bam\" && \n-\t\n+\n \tgit checkout master && \n \techo pow >file7 && \n \tgit add . && \n@@ -67,18 +68,26 @@ test_expect_success 'init repo' '\n \tgit add . && \n \tgit commit -m \"lol\" && \n \n+\tsleep 2 && \n \tgit checkout master && \n \tgit merge -m \"triple merge\" b1 b11 && \n \tgit rm -r d1 &&  \n+\tsleep 2 && \n \tgit commit -a -m \"oh noes\"\n '\n \n-git-rev-list HEAD --not HEAD~3 >proper_commit_list_limited\n-git-rev-list HEAD >proper_commit_list\n-git-rev-list HEAD --objects >proper_object_list\n+max_date=`git-rev-list --timestamp HEAD~1 --max-count=1 | grep -e \"^[0-9]*\" -o`\n+min_date=`git-rev-list --timestamp b4 --max-count=1 | grep -e \"^[0-9]*\" -o`\n+\n+git-rev-list --topo-order HEAD --not HEAD~3 >proper_commit_list_limited\n+git-rev-list --topo-order HEAD --not HEAD~2 >proper_commit_list_limited2\n+git-rev-list --topo-order HEAD >proper_commit_list\n+git-rev-list --objects HEAD >proper_object_list\n+git-rev-list HEAD --max-age=$min_date --min-age=$max_date >proper_list_date_limited\n+\n+cache_sha1=`git-rev-cache add HEAD 2>output.err`\n \n test_expect_success 'make cache slice' '\n-\tgit-rev-cache add HEAD 2>output.err && \n \tgrep \"final return value: 0\" output.err\n '\n \n@@ -98,7 +107,7 @@ test_expect_success 'test rev-caches walker directly (unlimited)' '\n \ttest -z `$sha1diff list proper_commit_list`\n '\n \n-test_expect_success 'test rev-list rev-list traversal (limited)' '\n+test_expect_success 'test rev-list traversal (limited)' '\n \tgit-rev-list HEAD --not HEAD~3 >list && \n \ttest -z `$sha1diff list proper_commit_list_limited`\n '\n@@ -114,15 +123,106 @@ test_expect_success 'test rev-caches walker with objects' '\n \ttest -z `$sha1diff list proper_object_list`\n '\n \n-test_expect_success 'test rev-list with objects (limited)' '\n+test_expect_success 'test rev-list with objects (topo order)' '\n \tgit-rev-list --topo-order --objects HEAD >list && \n \ttest -z `$sha1diff list proper_object_list`\n '\n \n-test_expect_success 'test rev-list with objects (unlimited)' '\n+test_expect_success 'test rev-list with objects (no order)' '\n \tgit-rev-list --objects HEAD >list && \n \ttest -z `$sha1diff list proper_object_list`\n '\n \n+#verify age limiting\n+test_expect_success 'test rev-list date limiting (topo order)' '\n+\tgit-rev-list --topo-order --max-age=$min_date --min-age=$max_date HEAD >list && \n+\ttest -z `$sha1diff list proper_list_date_limited`\n+'\n+\n+test_expect_success 'test rev-list date limiting (no order)' '\n+\tgit-rev-list --max-age=$min_date --min-age=$max_date HEAD >list && \n+\ttest -z `sha1diff list proper_list_date_limited`\n+'\n+\n+#check partial cache slice\n+test_expect_success 'saving old cache and generating partial slice' '\n+\tcp \".git/rev-cache/$cache_sha1\" .git/rev-cache/.old && \n+\trm \".git/rev-cache/$cache_sha1\" .git/rev-cache/index && \n+\n+\tgit-rev-cache add HEAD~2 2>output.err && \n+\tgrep \"final return value: 0\" output.err\n+'\n+\n+test_expect_success 'rev-list with wholly interesting partial slice' '\n+\tgit-rev-list --topo-order HEAD >list &&\n+\ttest_cmp list proper_commit_list\n+'\n+\n+test_expect_success 'rev-list with partly uninteresting partial slice' '\n+\tgit-rev-list --topo-order HEAD --not HEAD~3 >list && \n+\ttest_cmp list proper_commit_list_limited\n+'\n+\n+test_expect_success 'rev-list with wholly uninteresting partial slice' '\n+\tgit-rev-list --topo-order HEAD --not HEAD~2 >list && \n+\ttest_cmp list proper_commit_list_limited2\n+'\n+\n+#try out index generation and fuse (note that --all == HEAD in this case)\n+#probably should make a test for that too...\n+test_expect_success 'make fresh slice' '\n+\tgit-rev-cache add --all --fresh 2>output.err && \n+\tgrep \"final return value: 0\" output.err\n+'\n+\n+test_expect_success 'check dual slices' '\n+\tgit-rev-list --topo-order HEAD~2 HEAD >list && \n+\ttest_cmp list proper_commit_list\n+'\n+\n+test_expect_success 'regenerate index' '\n+\trm .git/rev-cache/index && \n+\tgit-rev-cache index 2>output.err && \n+\tgrep \"final return value: 0\" output.err\n+'\n+\n+test_expect_success 'fuse slices' '\n+\ttest -e .git/rev-cache/.old && \n+\tgit-rev-cache fuse 2>output.err && \n+\tgrep \"final return value: 0\" output.err && \n+\ttest_cmp .git/rev-cache/$cache_sha1 .git/rev-cache/.old\n+'\n+\n+#make sure we can smoothly handle corrupted caches\n+test_expect_success 'corrupt slice' '\n+\techo bla >.git/rev-cache/$cache_sha1\n+'\n+\n+test_expect_success 'test rev-list traversal (limited) (corrupt slice)' '\n+\tgit-rev-list HEAD --not HEAD~3 >list && \n+\ttest -z `$sha1diff list proper_commit_list_limited`\n+'\n+\n+test_expect_success 'test rev-list traversal (unlimited) (corrupt slice)' '\n+\tgit-rev-list HEAD >list && \n+\ttest -z `$sha1diff list proper_commit_list`\n+'\n+\n+test_expect_success 'corrupt index' '\n+\techo blu >.git/rev-cache/index\n+'\n+\n+test_expect_success 'test rev-list traversal (limited) (corrupt index)' '\n+\tgit-rev-list HEAD --not HEAD~3 >list && \n+\ttest -z `$sha1diff list proper_commit_list_limited`\n+'\n+\n+test_expect_success 'test rev-list traversal (unlimited) (corrupt index)' '\n+\tgit-rev-list HEAD >list && \n+\ttest -z `$sha1diff list proper_commit_list`\n+'\n+\n+#test --ignore-size in fuse?\n+\n test_done\n \n-- \n"},{"id":"119876","messageId":"4A7B9DEB.1060002@vilain.net","threadId":"20432","inReplyTo":"op.ux8i7ceotdk399@sirnot.private","subject":"Re: [PATCH 5/5] full integration of rev-cache into git's revision walker, completed test suite","fromName":"Sam Vilain","fromEmail":"sam@vilain.net","sentAt":"2009-08-07T03:22:19Z","receivedAt":"2009-08-07T03:22:19Z","isPatch":true,"sender":{"key":"sam@vilain.net","avatar":"https://gravatar.com/avatar/8fc840ca854dbf6f7065b4335e3b934951c1dca3b11db688e95e471901f8f4a8?d=mp&s=160"},"body":"Nick Edelen wrote:\n> This last patch provides a working integration of rev-cache into the revision \n> walker, along with some touch-ups:\n>   \n\n\"This last patch\" is redundant.  You can just write \"Integrate rev-cache\ninto the revision walker;\"\n\n>  - integration into revision walker and list-objects\n>  - addition of 'unique' field to commit objects, optionally initialized in \n>    rev-cache with the objects introduced in that commit\n>  - tweak of object generation to take advantage of the 'unique' field\n>  - more fluid handling of damaged cache slices\n>  - numerous tests for both features from the previous patch, and the \n> integration's integrity\n>   \n\nDoes it make sense to split this integration up or stage it somehow?\n\n> 'Integration' is rather broad -- a more detailed description follows for each \n> aspect:\n>  - rev-cache\n> the traversal mechanism is updated to handle many of the non-prune options \n> rev-list does (date limiting, slop-handling, etc.), and is adjusted to allow \n> for non-fatal cache-traversal failures.\n>\n>  - revision walker\n> both limited and unlimited traversal attempt to use the cache when possible, \n> smoothly falling back if it's not.\n>\n>  - list-objects\n> object listing does not recurse into cached trees, and has been adjusted to \n> guarantee commit-tag-tree-blob ordering.\n>   \n\nEspecially given the length of this part of the commit message.\n\nSam\n"},{"id":"119903","messageId":"op.uyaolcvttdk399@sirnot.private","threadId":"20432","inReplyTo":"op.ux8i7ceotdk399@sirnot.private","subject":"Re: [PATCH 5/5] full integration of rev-cache into git's revision walker, completed test suite","fromName":"Nick Edelen","fromEmail":"sirnot@gmail.com","sentAt":"2009-08-07T13:47:26Z","receivedAt":"2009-08-07T13:47:26Z","isPatch":true,"sender":{"key":"sirnot@gmail.com","avatar":null},"body":"A small patch to eliminate the python dependency in rev-cache's test-suite.  \nThis is a holdover for the 'integration' patch until a revised patchset is sent.\n\nSigned-off-by: Nick Edelen <sirnot@gmail.com>\n\n---\nI've also implemented name caching, but I'd like to cleanup that (and the docs \n;-) up a bit before posting it.  I'll have everything ready for public display \nby tonight (I've got an appointment for the next several hours).\n\n t/t6015-rev-cache-list.sh |   39 +++++++++++++++++++++------------------\n t/t6015-sha1-dump-diff.py |   36 ------------------------------------\n 2 files changed, 21 insertions(+), 54 deletions(-)\n\ndiff --git a/t/t6015-rev-cache-list.sh b/t/t6015-rev-cache-list.sh\nindex bfd9525..7c3719d 100755\n--- a/t/t6015-rev-cache-list.sh\n+++ b/t/t6015-rev-cache-list.sh\n@@ -3,7 +3,11 @@\n test_description='git rev-cache tests'\n . ./test-lib.sh\n \n-sha1diff=\"python $TEST_DIRECTORY/t6015-sha1-dump-diff.py\"\n+test_cmp_sorted() {\n+\tgrep -io \"[a-f0-9]*\" $1 | sort >.tmpfile1 && \n+\tgrep -io \"[a-f0-9]*\" $2 | sort >.tmpfile2 && \n+\ttest_cmp .tmpfile1 .tmpfile2\n+}\n \n # we want a totally wacked out branch structure...\n # we need branching and merging of sizes up through 3, tree \n@@ -99,49 +103,49 @@ test_expect_success 'remake cache slice' '\n #check core mechanics and rev-list hook for commits\n test_expect_success 'test rev-caches walker directly (limited)' '\n \tgit-rev-cache walk HEAD --not HEAD~3 >list && \n-\ttest -z `$sha1diff list proper_commit_list_limited`\n+\ttest_cmp_sorted list proper_commit_list_limited\n '\n \n test_expect_success 'test rev-caches walker directly (unlimited)' '\n \tgit-rev-cache walk HEAD >list && \n-\ttest -z `$sha1diff list proper_commit_list`\n+\ttest_cmp_sorted list proper_commit_list\n '\n \n test_expect_success 'test rev-list traversal (limited)' '\n \tgit-rev-list HEAD --not HEAD~3 >list && \n-\ttest -z `$sha1diff list proper_commit_list_limited`\n+\ttest_cmp list proper_commit_list_limited\n '\n \n test_expect_success 'test rev-list traversal (unlimited)' '\n \tgit-rev-list HEAD >list && \n-\ttest -z `$sha1diff list proper_commit_list`\n+\ttest_cmp list proper_commit_list\n '\n \n #do the same for objects\n test_expect_success 'test rev-caches walker with objects' '\n \tgit-rev-cache walk --objects HEAD >list && \n-\ttest -z `$sha1diff list proper_object_list`\n+\ttest_cmp_sorted list proper_object_list\n '\n \n test_expect_success 'test rev-list with objects (topo order)' '\n \tgit-rev-list --topo-order --objects HEAD >list && \n-\ttest -z `$sha1diff list proper_object_list`\n+\ttest_cmp_sorted list proper_object_list\n '\n \n test_expect_success 'test rev-list with objects (no order)' '\n \tgit-rev-list --objects HEAD >list && \n-\ttest -z `$sha1diff list proper_object_list`\n+\ttest_cmp_sorted list proper_object_list\n '\n \n #verify age limiting\n test_expect_success 'test rev-list date limiting (topo order)' '\n \tgit-rev-list --topo-order --max-age=$min_date --min-age=$max_date HEAD >list && \n-\ttest -z `$sha1diff list proper_list_date_limited`\n+\ttest_cmp_sorted list proper_list_date_limited\n '\n \n test_expect_success 'test rev-list date limiting (no order)' '\n \tgit-rev-list --max-age=$min_date --min-age=$max_date HEAD >list && \n-\ttest -z `sha1diff list proper_list_date_limited`\n+\ttest_cmp_sorted list proper_list_date_limited\n '\n \n #check partial cache slice\n@@ -154,7 +158,7 @@ test_expect_success 'saving old cache and generating partial slice' '\n '\n \n test_expect_success 'rev-list with wholly interesting partial slice' '\n-\tgit-rev-list --topo-order HEAD >list &&\n+\tgit-rev-list --topo-order HEAD >list && \n \ttest_cmp list proper_commit_list\n '\n \n@@ -199,13 +203,13 @@ test_expect_success 'corrupt slice' '\n '\n \n test_expect_success 'test rev-list traversal (limited) (corrupt slice)' '\n-\tgit-rev-list HEAD --not HEAD~3 >list && \n-\ttest -z `$sha1diff list proper_commit_list_limited`\n+\tgit-rev-list --topo-order HEAD --not HEAD~3 >list && \n+\ttest_cmp list proper_commit_list_limited\n '\n \n test_expect_success 'test rev-list traversal (unlimited) (corrupt slice)' '\n \tgit-rev-list HEAD >list && \n-\ttest -z `$sha1diff list proper_commit_list`\n+\ttest_cmp_sorted list proper_commit_list\n '\n \n test_expect_success 'corrupt index' '\n@@ -213,16 +217,15 @@ test_expect_success 'corrupt index' '\n '\n \n test_expect_success 'test rev-list traversal (limited) (corrupt index)' '\n-\tgit-rev-list HEAD --not HEAD~3 >list && \n-\ttest -z `$sha1diff list proper_commit_list_limited`\n+\tgit-rev-list --topo-order HEAD --not HEAD~3 >list && \n+\ttest_cmp list proper_commit_list_limited\n '\n \n test_expect_success 'test rev-list traversal (unlimited) (corrupt index)' '\n \tgit-rev-list HEAD >list && \n-\ttest -z `$sha1diff list proper_commit_list`\n+\ttest_cmp_sorted list proper_commit_list\n '\n \n #test --ignore-size in fuse?\n \n test_done\n-\ndiff --git a/t/t6015-sha1-dump-diff.py b/t/t6015-sha1-dump-diff.py\ndeleted file mode 100755\nindex 7fc30e7..0000000\n--- a/t/t6015-sha1-dump-diff.py\n+++ /dev/null\n@@ -1,36 +0,0 @@\n-\n-import sys, re\n-\n-if len(sys.argv) < 3 : \n-\tsys.exit(0)\n-\n-f = open(sys.argv[1], 'r')\n-dict = {}\n-for line in f : \n-\tif len(line) >= 40 and re.match(r'^[0-9a-fA-F]{40}', line[:40]) != None : \n-\t\tdict[line[:40]] = 1\n-\n-f.close()\n-\n-f = open(sys.argv[2], 'r')\n-for line in f :\n-\tif len(line) < 40 : \n-\t\tcontinue\n-\t\n-\thash = line[:40]\n-\tif re.match(r'^[0-9a-fA-F]{40}', hash) == None : \n-\t\tcontinue\n-\t\n-\tif hash in dict : \n-\t\tdict[hash] -= 1\n-\telse : \n-\t\tdict[hash] = -1\n-\n-f.close()\n-\n-for k in dict.keys() : \n-\tif dict[k] == 0 : \n-\t\tdel dict[k]\n-\n-if len(dict) : \n-\tprint dict\n---\n"}]}