{"thread":{"id":"12316","subject":"[PATCH 01/10] add generic, type aware object chain walker","startedAt":"2008-02-25T21:54:51Z","lastAt":"2008-02-27T07:48:59Z","messageCount":13,"participants":["Martin Koegler","Junio C Hamano"],"isPatch":true,"patchVersion":1,"patchTotal":10},"messages":[{"id":"69928","messageId":"12039765002329-git-send-email-mkoegler@auto.tuwien.ac.at","threadId":"12316","inReplyTo":null,"subject":"[PATCH 01/10] add generic, type aware object chain walker","fromName":"Martin Koegler","fromEmail":"mkoegler@auto.tuwien.ac.at","sentAt":"2008-02-25T21:54:51Z","receivedAt":"2008-02-25T21:54:51Z","isPatch":true,"sender":{"key":"mkoegler@auto.tuwien.ac.at","avatar":null},"body":"The requirements are:\n* it may not crash on NULL pointers\n* a callback function is needed, as index-pack/unpack-objects\n  need to do different things\n* the type information is needed to check the expected <-> real type\n  and print better error messages\n\nSigned-off-by: Martin Koegler <mkoegler@auto.tuwien.ac.at>\n---\n Makefile |    4 +-\n cache.h  |    1 +\n fsck.c   |   91 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n fsck.h   |   23 +++++++++++++++\n 4 files changed, 117 insertions(+), 2 deletions(-)\n create mode 100644 fsck.c\n create mode 100644 fsck.h\n\ndiff --git a/Makefile b/Makefile\nindex 021520f..3bb2034 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -304,7 +304,7 @@ LIB_H = \\\n \trun-command.h strbuf.h tag.h tree.h git-compat-util.h revision.h \\\n \ttree-walk.h log-tree.h dir.h path-list.h unpack-trees.h builtin.h \\\n \tutf8.h reflog-walk.h patch-ids.h attr.h decorate.h progress.h \\\n-\tmailmap.h remote.h parse-options.h transport.h diffcore.h hash.h\n+\tmailmap.h remote.h parse-options.h transport.h diffcore.h hash.h fsck.h\n \n DIFF_OBJS = \\\n \tdiff.o diff-lib.o diffcore-break.o diffcore-order.o \\\n@@ -327,7 +327,7 @@ LIB_OBJS = \\\n \talloc.o merge-file.o path-list.o help.o unpack-trees.o $(DIFF_OBJS) \\\n \tcolor.o wt-status.o archive-zip.o archive-tar.o shallow.o utf8.o \\\n \tconvert.o attr.o decorate.o progress.o mailmap.o symlinks.o remote.o \\\n-\ttransport.o bundle.o walker.o parse-options.o ws.o archive.o\n+\ttransport.o bundle.o walker.o parse-options.o ws.o archive.o fsck.o\n \n BUILTIN_OBJS = \\\n \tbuiltin-add.o \\\ndiff --git a/cache.h b/cache.h\nindex 4fa69f0..3001bc3 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -273,6 +273,7 @@ enum object_type {\n \t/* 5 for future expansion */\n \tOBJ_OFS_DELTA = 6,\n \tOBJ_REF_DELTA = 7,\n+\tOBJ_ANY,\n \tOBJ_MAX,\n };\n \ndiff --git a/fsck.c b/fsck.c\nnew file mode 100644\nindex 0000000..b7f9354\n--- /dev/null\n+++ b/fsck.c\n@@ -0,0 +1,91 @@\n+#include \"cache.h\"\n+#include \"object.h\"\n+#include \"blob.h\"\n+#include \"tree.h\"\n+#include \"tree-walk.h\"\n+#include \"commit.h\"\n+#include \"tag.h\"\n+#include \"fsck.h\"\n+\n+static int fsck_walk_tree(struct tree *tree, fsck_walk_func walk, void *data)\n+{\n+\tstruct tree_desc desc;\n+\tstruct name_entry entry;\n+\tint res = 0;\n+\n+\tif (parse_tree(tree))\n+\t\treturn -1;\n+\n+\tinit_tree_desc(&desc, tree->buffer, tree->size);\n+\twhile (tree_entry(&desc, &entry)) {\n+\t\tint result;\n+\t\t\n+\t\tif (S_ISGITLINK(entry.mode))\n+\t\t\tcontinue;\n+\t\tif (S_ISDIR(entry.mode))\n+\t\t\tresult = walk(&lookup_tree(entry.sha1)->object, OBJ_TREE, data);\n+\t\telse if (S_ISREG(entry.mode) || S_ISLNK(entry.mode))\n+\t\t\tresult = walk(&lookup_blob(entry.sha1)->object, OBJ_BLOB, data);\n+\t\telse {\n+\t\t\tresult = error(\"in tree %s: entry %s has bad mode %.6o\\n\",\n+\t\t\t               sha1_to_hex(tree->object.sha1), entry.path, entry.mode);\n+\t\t}\n+\t\tif (result < 0)\n+\t\t\treturn result;\n+\t\tif (!res)\n+\t\t\tres = result;\n+\t}\n+\treturn res;\n+}\n+\n+static int fsck_walk_commit(struct commit *commit, fsck_walk_func walk, void *data)\n+{\n+\tstruct commit_list *parents;\n+\tint res;\n+\tint result;\n+\n+\tif (parse_commit(commit))\n+\t\treturn -1;\n+\n+\tresult = walk((struct object*)commit->tree, OBJ_TREE, data);\n+\tif (result < 0)\n+\t\treturn result;\n+\tres = result;\n+\n+\tparents = commit->parents;\n+\twhile (parents) {\n+\t\tresult = walk((struct object*)parents->item, OBJ_COMMIT, data);\n+\t\tif (result < 0)\n+\t\t\treturn result;\n+\t\tif (!res)\n+\t\t\tres = result;\n+\t\tparents = parents->next;\n+\t}\n+\treturn res;\n+}\n+\n+static int fsck_walk_tag(struct tag *tag, fsck_walk_func walk, void *data)\n+{\n+\tif (parse_tag(tag))\n+\t\treturn -1;\n+\treturn walk(tag->tagged, OBJ_ANY, data);\n+}\n+\n+int fsck_walk(struct object *obj, fsck_walk_func walk, void *data)\n+{\n+\tif (!obj)\n+\t\treturn -1;\n+\tswitch(obj->type) {\n+\tcase OBJ_BLOB:\n+\t\treturn 0;\n+\tcase OBJ_TREE:\n+\t\treturn fsck_walk_tree((struct tree*)obj, walk, data);\n+\tcase OBJ_COMMIT:\n+\t\treturn fsck_walk_commit((struct commit*)obj, walk, data);\n+\tcase OBJ_TAG:\n+\t\treturn fsck_walk_tag((struct tag*)obj, walk, data);\n+\tdefault:\n+\t\terror(\"Unknown object type for %s\", sha1_to_hex(obj->sha1));\n+\t\treturn -1;\n+\t}\n+}\ndiff --git a/fsck.h b/fsck.h\nnew file mode 100644\nindex 0000000..4a1f02a\n--- /dev/null\n+++ b/fsck.h\n@@ -0,0 +1,23 @@\n+#ifndef GIT_FSCK_H\n+#define GIT_FSCK_H\n+\n+/* \n+ * callback function for fsck_walk\n+ * type is the expected type of the object or OBJ_ANY\n+ * the return value is:\n+ *     0\teverything OK\n+ *     <0\terror signaled and abort\n+ *     >0\terror signaled and do not abort\n+ */\n+typedef int (*fsck_walk_func)(struct object *obj, int type, void *data);\n+\n+/* descend in all linked child objects\n+ * the return value is:\n+ *    -1\terror in processing the object\n+ *    <0\treturn value of the callback, which lead to an abort\n+ *    >0\treturn value of the first sigaled error >0 (in the case of no other errors)\n+ *    0\t\teverything OK\n+ */\n+int fsck_walk(struct object *obj, fsck_walk_func walk, void *data);\n+\n+#endif\n-- \n1.5.4.3.g3c5f\n"},{"id":"69929","messageId":"12039765004039-git-send-email-mkoegler@auto.tuwien.ac.at","threadId":"12316","inReplyTo":"12039765002329-git-send-email-mkoegler@auto.tuwien.ac.at","subject":"[PATCH 02/10] builtin-fsck: move away from object-refs to fsck_walk","fromName":"Martin Koegler","fromEmail":"mkoegler@auto.tuwien.ac.at","sentAt":"2008-02-25T21:54:52Z","receivedAt":"2008-02-25T21:54:52Z","isPatch":true,"sender":{"key":"mkoegler@auto.tuwien.ac.at","avatar":null},"body":"Signed-off-by: Martin Koegler <mkoegler@auto.tuwien.ac.at>\n---\n builtin-fsck.c |  100 ++++++++++++++++++++++++++++++++++++++++----------------\n 1 files changed, 72 insertions(+), 28 deletions(-)\n\ndiff --git a/builtin-fsck.c b/builtin-fsck.c\nindex cc7524b..a2e6f53 100644\n--- a/builtin-fsck.c\n+++ b/builtin-fsck.c\n@@ -8,6 +8,7 @@\n #include \"pack.h\"\n #include \"cache-tree.h\"\n #include \"tree-walk.h\"\n+#include \"fsck.h\"\n #include \"parse-options.h\"\n \n #define REACHABLE 0x0001\n@@ -63,13 +64,74 @@ static int objwarning(struct object *obj, const char *err, ...)\n \treturn -1;\n }\n \n+static int mark_object(struct object *obj, int type, void *data)\n+{\n+\tstruct tree *tree = NULL;\n+\tstruct object *parent = data;\n+\tint result;\n+\n+\tif (!obj) {\n+\t\tprintf(\"broken link from %7s %s\\n\",\n+\t\t\t   typename(parent->type), sha1_to_hex(parent->sha1));\n+\t\tprintf(\"broken link from %7s %s\\n\",\n+\t\t\t   (type==OBJ_ANY?\"unknown\":typename(type)), \"unknown\");\n+\t\terrors_found |= ERROR_REACHABLE;\n+\t\treturn 1;\n+\t}\n+\n+\tif (type != OBJ_ANY && obj->type != type) {\n+\t\tobjerror(parent, \"wrong object type in link\");\n+\t}\n+\n+\tif (obj->flags & REACHABLE)\n+\t\treturn 0;\n+\tobj->flags |= REACHABLE;\n+\tif (!obj->parsed) {\n+\t\tif (parent && !has_sha1_file(obj->sha1)) {\n+\t\t\tprintf(\"broken link from %7s %s\\n\",\n+\t\t\t\t typename(parent->type), sha1_to_hex(parent->sha1));\n+\t\t\tprintf(\"              to %7s %s\\n\",\n+\t\t\t\t typename(obj->type), sha1_to_hex(obj->sha1));\n+\t\t\terrors_found |= ERROR_REACHABLE;\n+\t\t}\n+\t\treturn 1;\n+\t}\n+\n+\tif (obj->type == OBJ_TREE) {\n+\t\tobj->parsed = 0;\n+\t\ttree = (struct tree*)obj;\n+\t\tif (parse_tree(tree) < 0)\n+\t\t\treturn 1; /* error already displayed */\n+\t}\n+\tresult = fsck_walk(obj, mark_object, obj);\n+\tif (tree) {\n+\t\tfree(tree->buffer);\n+\t\ttree->buffer=NULL;\n+\t}\n+\tif (result < 0) \n+\t\tresult = 1;\n+\t\n+\treturn result;\n+}\n+\n+static void mark_object_reachable(struct object *obj)\n+{\n+\tmark_object(obj, OBJ_ANY, 0);\n+}\n+\n+static int mark_used(struct object *obj, int type, void *data)\n+{\n+\tif (!obj)\n+\t\treturn 1;\n+\tobj->used = 1;\n+\treturn 0;\n+}\n+\n /*\n  * Check a single reachable object\n  */\n static void check_reachable_object(struct object *obj)\n {\n-\tconst struct object_refs *refs;\n-\n \t/*\n \t * We obviously want the object to be parsed,\n \t * except if it was in a pack-file and we didn't\n@@ -82,25 +144,6 @@ static void check_reachable_object(struct object *obj)\n \t\terrors_found |= ERROR_REACHABLE;\n \t\treturn;\n \t}\n-\n-\t/*\n-\t * Check that everything that we try to reference is also good.\n-\t */\n-\trefs = lookup_object_refs(obj);\n-\tif (refs) {\n-\t\tunsigned j;\n-\t\tfor (j = 0; j < refs->count; j++) {\n-\t\t\tstruct object *ref = refs->ref[j];\n-\t\t\tif (ref->parsed ||\n-\t\t\t    (has_sha1_file(ref->sha1)))\n-\t\t\t\tcontinue;\n-\t\t\tprintf(\"broken link from %7s %s\\n\",\n-\t\t\t       typename(obj->type), sha1_to_hex(obj->sha1));\n-\t\t\tprintf(\"              to %7s %s\\n\",\n-\t\t\t       typename(ref->type), sha1_to_hex(ref->sha1));\n-\t\t\terrors_found |= ERROR_REACHABLE;\n-\t\t}\n-\t}\n }\n \n /*\n@@ -414,6 +457,8 @@ static int fsck_sha1(const unsigned char *sha1)\n \tif (obj->flags & SEEN)\n \t\treturn 0;\n \tobj->flags |= SEEN;\n+\tif (fsck_walk(obj, mark_used, 0))\n+\t\tobjerror(obj, \"broken links\");\n \tif (obj->type == OBJ_BLOB)\n \t\treturn 0;\n \tif (obj->type == OBJ_TREE)\n@@ -538,13 +583,13 @@ static int fsck_handle_reflog_ent(unsigned char *osha1, unsigned char *nsha1,\n \t\tobj = lookup_object(osha1);\n \t\tif (obj) {\n \t\t\tobj->used = 1;\n-\t\t\tmark_reachable(obj, REACHABLE);\n+\t\t\tmark_object_reachable(obj);\n \t\t}\n \t}\n \tobj = lookup_object(nsha1);\n \tif (obj) {\n \t\tobj->used = 1;\n-\t\tmark_reachable(obj, REACHABLE);\n+\t\tmark_object_reachable(obj);\n \t}\n \treturn 0;\n }\n@@ -574,7 +619,7 @@ static int fsck_handle_ref(const char *refname, const unsigned char *sha1, int f\n \t\terror(\"%s: not a commit\", refname);\n \tdefault_refs++;\n \tobj->used = 1;\n-\tmark_reachable(obj, REACHABLE);\n+\tmark_object_reachable(obj);\n \n \treturn 0;\n }\n@@ -660,7 +705,7 @@ static int fsck_cache_tree(struct cache_tree *it)\n \t\t\t      sha1_to_hex(it->sha1));\n \t\t\treturn 1;\n \t\t}\n-\t\tmark_reachable(obj, REACHABLE);\n+\t\tmark_object_reachable(obj);\n \t\tobj->used = 1;\n \t\tif (obj->type != OBJ_TREE)\n \t\t\terr |= objerror(obj, \"non-tree in cache-tree\");\n@@ -693,7 +738,6 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)\n {\n \tint i, heads;\n \n-\ttrack_object_refs = 1;\n \terrors_found = 0;\n \n \targc = parse_options(argc, argv, fsck_opts, fsck_usage, 0);\n@@ -741,7 +785,7 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)\n \t\t\t\tcontinue;\n \n \t\t\tobj->used = 1;\n-\t\t\tmark_reachable(obj, REACHABLE);\n+\t\t\tmark_object_reachable(obj);\n \t\t\theads++;\n \t\t\tcontinue;\n \t\t}\n@@ -773,7 +817,7 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)\n \t\t\t\tcontinue;\n \t\t\tobj = &blob->object;\n \t\t\tobj->used = 1;\n-\t\t\tmark_reachable(obj, REACHABLE);\n+\t\t\tmark_object_reachable(obj);\n \t\t}\n \t\tif (active_cache_tree)\n \t\t\tfsck_cache_tree(active_cache_tree);\n-- \n1.5.4.3.g3c5f\n"},{"id":"69931","messageId":"12039765003484-git-send-email-mkoegler@auto.tuwien.ac.at","threadId":"12316","inReplyTo":"12039765004039-git-send-email-mkoegler@auto.tuwien.ac.at","subject":"[PATCH 03/10] Remove unused object-ref code","fromName":"Martin Koegler","fromEmail":"mkoegler@auto.tuwien.ac.at","sentAt":"2008-02-25T21:54:53Z","receivedAt":"2008-02-25T21:54:53Z","isPatch":true,"sender":{"key":"mkoegler@auto.tuwien.ac.at","avatar":null},"body":"Signed-off-by: Martin Koegler <mkoegler@auto.tuwien.ac.at>\n---\n Makefile               |    2 +-\n builtin-fetch-pack.c   |    1 -\n builtin-pack-objects.c |    1 -\n builtin-rev-list.c     |    1 -\n commit.c               |   11 ------\n object-refs.c          |   87 ------------------------------------------------\n object.h               |    8 ----\n tag.c                  |    6 ---\n tree.c                 |   48 --------------------------\n upload-pack.c          |    1 -\n walker.c               |    1 -\n 11 files changed, 1 insertions(+), 166 deletions(-)\n delete mode 100644 object-refs.c\n\ndiff --git a/Makefile b/Makefile\nindex 3bb2034..2d7d6a8 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -319,7 +319,7 @@ LIB_OBJS = \\\n \tpatch-ids.o \\\n \tobject.o pack-check.o pack-write.o patch-delta.o path.o pkt-line.o \\\n \tsideband.o reachable.o reflog-walk.o \\\n-\tquote.o read-cache.o refs.o run-command.o dir.o object-refs.o \\\n+\tquote.o read-cache.o refs.o run-command.o dir.o \\\n \tserver-info.o setup.o sha1_file.o sha1_name.o strbuf.o \\\n \ttag.o tree.o usage.o config.o environment.o ctype.o copy.o \\\n \trevision.o pager.o tree-walk.o xdiff-interface.o \\\ndiff --git a/builtin-fetch-pack.c b/builtin-fetch-pack.c\nindex f401352..25f1915 100644\n--- a/builtin-fetch-pack.c\n+++ b/builtin-fetch-pack.c\n@@ -386,7 +386,6 @@ static int everything_local(struct ref **refs, int nr_match, char **match)\n \tint retval;\n \tunsigned long cutoff = 0;\n \n-\ttrack_object_refs = 0;\n \tsave_commit_buffer = 0;\n \n \tfor (ref = *refs; ref; ref = ref->next) {\ndiff --git a/builtin-pack-objects.c b/builtin-pack-objects.c\nindex d2bb12e..8c148e1 100644\n--- a/builtin-pack-objects.c\n+++ b/builtin-pack-objects.c\n@@ -2013,7 +2013,6 @@ static void get_object_list(int ac, const char **av)\n \n \tinit_revisions(&revs, NULL);\n \tsave_commit_buffer = 0;\n-\ttrack_object_refs = 0;\n \tsetup_revisions(ac, av, &revs, NULL);\n \n \twhile (fgets(line, sizeof(line), stdin) != NULL) {\ndiff --git a/builtin-rev-list.c b/builtin-rev-list.c\nindex 6f7d5f8..921113f 100644\n--- a/builtin-rev-list.c\n+++ b/builtin-rev-list.c\n@@ -607,7 +607,6 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix)\n \t\tusage(rev_list_usage);\n \n \tsave_commit_buffer = revs.verbose_header || revs.grep_filter;\n-\ttrack_object_refs = 0;\n \tif (bisect_list)\n \t\trevs.limited = 1;\n \ndiff --git a/commit.c b/commit.c\nindex 22ce776..6684c4e 100644\n--- a/commit.c\n+++ b/commit.c\n@@ -290,17 +290,6 @@ int parse_commit_buffer(struct commit *item, void *buffer, unsigned long size)\n \t}\n \titem->date = parse_commit_date(bufptr, tail);\n \n-\tif (track_object_refs) {\n-\t\tunsigned i = 0;\n-\t\tstruct commit_list *p;\n-\t\tstruct object_refs *refs = alloc_object_refs(n_refs);\n-\t\tif (item->tree)\n-\t\t\trefs->ref[i++] = &item->tree->object;\n-\t\tfor (p = item->parents; p; p = p->next)\n-\t\t\trefs->ref[i++] = &p->item->object;\n-\t\tset_object_refs(&item->object, refs);\n-\t}\n-\n \treturn 0;\n }\n \ndiff --git a/object-refs.c b/object-refs.c\ndeleted file mode 100644\nindex 5345671..0000000\n--- a/object-refs.c\n+++ /dev/null\n@@ -1,87 +0,0 @@\n-#include \"cache.h\"\n-#include \"object.h\"\n-#include \"decorate.h\"\n-\n-int track_object_refs = 0;\n-\n-static struct decoration ref_decorate;\n-\n-struct object_refs *lookup_object_refs(struct object *base)\n-{\n-\treturn lookup_decoration(&ref_decorate, base);\n-}\n-\n-static void add_object_refs(struct object *obj, struct object_refs *refs)\n-{\n-\tif (add_decoration(&ref_decorate, obj, refs))\n-\t\tdie(\"object %s tried to add refs twice!\", sha1_to_hex(obj->sha1));\n-}\n-\n-struct object_refs *alloc_object_refs(unsigned count)\n-{\n-\tstruct object_refs *refs;\n-\tsize_t size = sizeof(*refs) + count*sizeof(struct object *);\n-\n-\trefs = xcalloc(1, size);\n-\trefs->count = count;\n-\treturn refs;\n-}\n-\n-static int compare_object_pointers(const void *a, const void *b)\n-{\n-\tconst struct object * const *pa = a;\n-\tconst struct object * const *pb = b;\n-\tif (*pa == *pb)\n-\t\treturn 0;\n-\telse if (*pa < *pb)\n-\t\treturn -1;\n-\telse\n-\t\treturn 1;\n-}\n-\n-void set_object_refs(struct object *obj, struct object_refs *refs)\n-{\n-\tunsigned int i, j;\n-\n-\t/* Do not install empty list of references */\n-\tif (refs->count < 1) {\n-\t\tfree(refs);\n-\t\treturn;\n-\t}\n-\n-\t/* Sort the list and filter out duplicates */\n-\tqsort(refs->ref, refs->count, sizeof(refs->ref[0]),\n-\t      compare_object_pointers);\n-\tfor (i = j = 1; i < refs->count; i++) {\n-\t\tif (refs->ref[i] != refs->ref[i - 1])\n-\t\t\trefs->ref[j++] = refs->ref[i];\n-\t}\n-\tif (j < refs->count) {\n-\t\t/* Duplicates were found - reallocate list */\n-\t\tsize_t size = sizeof(*refs) + j*sizeof(struct object *);\n-\t\trefs->count = j;\n-\t\trefs = xrealloc(refs, size);\n-\t}\n-\n-\tfor (i = 0; i < refs->count; i++)\n-\t\trefs->ref[i]->used = 1;\n-\tadd_object_refs(obj, refs);\n-}\n-\n-void mark_reachable(struct object *obj, unsigned int mask)\n-{\n-\tconst struct object_refs *refs;\n-\n-\tif (!track_object_refs)\n-\t\tdie(\"cannot do reachability with object refs turned off\");\n-\t/* If we've been here already, don't bother */\n-\tif (obj->flags & mask)\n-\t\treturn;\n-\tobj->flags |= mask;\n-\trefs = lookup_object_refs(obj);\n-\tif (refs) {\n-\t\tunsigned i;\n-\t\tfor (i = 0; i < refs->count; i++)\n-\t\t\tmark_reachable(refs->ref[i], mask);\n-\t}\n-}\ndiff --git a/object.h b/object.h\nindex 397bbfa..036bd66 100644\n--- a/object.h\n+++ b/object.h\n@@ -35,14 +35,11 @@ struct object {\n \tunsigned char sha1[20];\n };\n \n-extern int track_object_refs;\n-\n extern const char *typename(unsigned int type);\n extern int type_from_string(const char *str);\n \n extern unsigned int get_max_object_index(void);\n extern struct object *get_indexed_object(unsigned int);\n-extern struct object_refs *lookup_object_refs(struct object *);\n \n /** Internal only **/\n struct object *lookup_object(const unsigned char *sha1);\n@@ -61,11 +58,6 @@ struct object *parse_object_buffer(const unsigned char *sha1, enum object_type t\n /** Returns the object, with potentially excess memory allocated. **/\n struct object *lookup_unknown_object(const unsigned  char *sha1);\n \n-struct object_refs *alloc_object_refs(unsigned count);\n-void set_object_refs(struct object *obj, struct object_refs *refs);\n-\n-void mark_reachable(struct object *obj, unsigned int mask);\n-\n struct object_list *object_list_insert(struct object *item,\n \t\t\t\t       struct object_list **list_p);\n \ndiff --git a/tag.c b/tag.c\nindex 990134f..4470d2b 100644\n--- a/tag.c\n+++ b/tag.c\n@@ -87,12 +87,6 @@ int parse_tag_buffer(struct tag *item, void *data, unsigned long size)\n \t\titem->tagged = NULL;\n \t}\n \n-\tif (item->tagged && track_object_refs) {\n-\t\tstruct object_refs *refs = alloc_object_refs(1);\n-\t\trefs->ref[0] = item->tagged;\n-\t\tset_object_refs(&item->object, refs);\n-\t}\n-\n \treturn 0;\n }\n \ndiff --git a/tree.c b/tree.c\nindex 87708ef..4b1825c 100644\n--- a/tree.c\n+++ b/tree.c\n@@ -202,52 +202,6 @@ struct tree *lookup_tree(const unsigned char *sha1)\n \treturn (struct tree *) obj;\n }\n \n-/*\n- * NOTE! Tree refs to external git repositories\n- * (ie gitlinks) do not count as real references.\n- *\n- * You don't have to have those repositories\n- * available at all, much less have the objects\n- * accessible from the current repository.\n- */\n-static void track_tree_refs(struct tree *item)\n-{\n-\tint n_refs = 0, i;\n-\tstruct object_refs *refs;\n-\tstruct tree_desc desc;\n-\tstruct name_entry entry;\n-\n-\t/* Count how many entries there are.. */\n-\tinit_tree_desc(&desc, item->buffer, item->size);\n-\twhile (tree_entry(&desc, &entry)) {\n-\t\tif (S_ISGITLINK(entry.mode))\n-\t\t\tcontinue;\n-\t\tn_refs++;\n-\t}\n-\n-\t/* Allocate object refs and walk it again.. */\n-\ti = 0;\n-\trefs = alloc_object_refs(n_refs);\n-\tinit_tree_desc(&desc, item->buffer, item->size);\n-\twhile (tree_entry(&desc, &entry)) {\n-\t\tstruct object *obj;\n-\n-\t\tif (S_ISGITLINK(entry.mode))\n-\t\t\tcontinue;\n-\t\tif (S_ISDIR(entry.mode))\n-\t\t\tobj = &lookup_tree(entry.sha1)->object;\n-\t\telse if (S_ISREG(entry.mode) || S_ISLNK(entry.mode))\n-\t\t\tobj = &lookup_blob(entry.sha1)->object;\n-\t\telse {\n-\t\t\twarning(\"in tree %s: entry %s has bad mode %.6o\\n\",\n-\t\t\t     sha1_to_hex(item->object.sha1), entry.path, entry.mode);\n-\t\t\tobj = lookup_unknown_object(entry.sha1);\n-\t\t}\n-\t\trefs->ref[i++] = obj;\n-\t}\n-\tset_object_refs(&item->object, refs);\n-}\n-\n int parse_tree_buffer(struct tree *item, void *buffer, unsigned long size)\n {\n \tif (item->object.parsed)\n@@ -256,8 +210,6 @@ int parse_tree_buffer(struct tree *item, void *buffer, unsigned long size)\n \titem->buffer = buffer;\n \titem->size = size;\n \n-\tif (track_object_refs)\n-\t\ttrack_tree_refs(item);\n \treturn 0;\n }\n \ndiff --git a/upload-pack.c b/upload-pack.c\nindex b26d053..e5421db 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -393,7 +393,6 @@ static int get_common_commits(void)\n \tchar hex[41], last_hex[41];\n \tint len;\n \n-\ttrack_object_refs = 0;\n \tsave_commit_buffer = 0;\n \n \tfor(;;) {\ndiff --git a/walker.c b/walker.c\nindex adc3e80..c10eca8 100644\n--- a/walker.c\n+++ b/walker.c\n@@ -256,7 +256,6 @@ int walker_fetch(struct walker *walker, int targets, char **target,\n \tint i;\n \n \tsave_commit_buffer = 0;\n-\ttrack_object_refs = 0;\n \n \tfor (i = 0; i < targets; i++) {\n \t\tif (!write_ref || !write_ref[i])\n-- \n1.5.4.3.g3c5f\n"},{"id":"69934","messageId":"12039765002219-git-send-email-mkoegler@auto.tuwien.ac.at","threadId":"12316","inReplyTo":"12039765003484-git-send-email-mkoegler@auto.tuwien.ac.at","subject":"[PATCH 04/10] builtin-fsck: reports missing parent commits","fromName":"Martin Koegler","fromEmail":"mkoegler@auto.tuwien.ac.at","sentAt":"2008-02-25T21:54:54Z","receivedAt":"2008-02-25T21:54:54Z","isPatch":true,"sender":{"key":"mkoegler@auto.tuwien.ac.at","avatar":null},"body":"parse_commit ignores parent commits with certain errors\n(eg. a non commit object is already loaded under the sha1 of\nthe parent). To make fsck reports such errors, it has to compare\nthe nummer of parent commits returned by parse commit with the\nnumber of parent commits in the object or in the graft/shallow file.\n\nSigned-off-by: Martin Koegler <mkoegler@auto.tuwien.ac.at>\n---\n builtin-fsck.c |   24 ++++++++++++++++++++++++\n commit.c       |    2 +-\n commit.h       |    1 +\n 3 files changed, 26 insertions(+), 1 deletions(-)\n\ndiff --git a/builtin-fsck.c b/builtin-fsck.c\nindex a2e6f53..198466c 100644\n--- a/builtin-fsck.c\n+++ b/builtin-fsck.c\n@@ -398,6 +398,8 @@ static int fsck_commit(struct commit *commit)\n {\n \tchar *buffer = commit->buffer;\n \tunsigned char tree_sha1[20], sha1[20];\n+\tstruct commit_graft *graft;\n+\tint parents = 0;\n \n \tif (verbose)\n \t\tfprintf(stderr, \"Checking commit %s\\n\",\n@@ -415,6 +417,28 @@ static int fsck_commit(struct commit *commit)\n \t\tif (get_sha1_hex(buffer+7, sha1) || buffer[47] != '\\n')\n \t\t\treturn objerror(&commit->object, \"invalid 'parent' line format - bad sha1\");\n \t\tbuffer += 48;\n+\t\tparents++;\n+\t}\n+\tgraft = lookup_commit_graft(commit->object.sha1);\n+\tif (graft) {\n+\t\tstruct commit_list *p = commit->parents;\n+\t\tparents = 0;\n+\t\twhile (p) {\n+\t\t\tp = p->next;\n+\t\t\tparents++;\n+\t\t}\n+\t\tif (graft->nr_parent == -1 && !parents)\n+\t\t\t; /* shallow commit */\n+\t\telse if (graft->nr_parent != parents)\n+\t\t\treturn objerror(&commit->object, \"graft objects missing\");\n+\t} else {\n+\t\tstruct commit_list *p = commit->parents;\n+\t\twhile (p && parents) {\n+\t\t\tp = p->next;\n+\t\t\tparents--;\n+\t\t}\n+\t\tif (p || parents)\n+\t\t\treturn objerror(&commit->object, \"parent objects missing\");\n \t}\n \tif (memcmp(buffer, \"author \", 7))\n \t\treturn objerror(&commit->object, \"invalid format - expected 'author' line\");\ndiff --git a/commit.c b/commit.c\nindex 6684c4e..94d5b3d 100644\n--- a/commit.c\n+++ b/commit.c\n@@ -193,7 +193,7 @@ static void prepare_commit_graft(void)\n \tcommit_graft_prepared = 1;\n }\n \n-static struct commit_graft *lookup_commit_graft(const unsigned char *sha1)\n+struct commit_graft *lookup_commit_graft(const unsigned char *sha1)\n {\n \tint pos;\n \tprepare_commit_graft();\ndiff --git a/commit.h b/commit.h\nindex 10e2b5d..3ad3dd9 100644\n--- a/commit.h\n+++ b/commit.h\n@@ -101,6 +101,7 @@ struct commit_graft {\n struct commit_graft *read_graft_line(char *buf, int len);\n int register_commit_graft(struct commit_graft *, int);\n int read_graft_file(const char *graft_file);\n+struct commit_graft *lookup_commit_graft(const unsigned char *sha1);\n \n extern struct commit_list *get_merge_bases(struct commit *rev1, struct commit *rev2, int cleanup);\n \n-- \n1.5.4.3.g3c5f\n"},{"id":"69930","messageId":"12039765002397-git-send-email-mkoegler@auto.tuwien.ac.at","threadId":"12316","inReplyTo":"12039765002219-git-send-email-mkoegler@auto.tuwien.ac.at","subject":"[PATCH 05/10] builtin-fsck: move common object checking code to fsck.c","fromName":"Martin Koegler","fromEmail":"mkoegler@auto.tuwien.ac.at","sentAt":"2008-02-25T21:54:55Z","receivedAt":"2008-02-25T21:54:55Z","isPatch":true,"sender":{"key":"mkoegler@auto.tuwien.ac.at","avatar":null},"body":"Signed-off-by: Martin Koegler <mkoegler@auto.tuwien.ac.at>\n---\n builtin-fsck.c |  270 +++++++------------------------------------------------\n fsck.c         |  222 ++++++++++++++++++++++++++++++++++++++++++++++\n fsck.h         |    7 ++\n 3 files changed, 264 insertions(+), 235 deletions(-)\n\ndiff --git a/builtin-fsck.c b/builtin-fsck.c\nindex 198466c..3b21745 100644\n--- a/builtin-fsck.c\n+++ b/builtin-fsck.c\n@@ -55,13 +55,13 @@ static int objerror(struct object *obj, const char *err, ...)\n \treturn -1;\n }\n \n-static int objwarning(struct object *obj, const char *err, ...)\n+static int fsck_error_func(struct object *obj, int type, const char *err, ...)\n {\n \tva_list params;\n \tva_start(params, err);\n-\tobjreport(obj, \"warning\", err, params);\n+\tobjreport(obj, (type == FSCK_WARN)?\"warning\":\"error\", err, params);\n \tva_end(params);\n-\treturn -1;\n+\treturn (type == FSCK_WARN)?0:1;\n }\n \n static int mark_object(struct object *obj, int type, void *data)\n@@ -247,256 +247,56 @@ static void check_connectivity(void)\n \t}\n }\n \n-/*\n- * The entries in a tree are ordered in the _path_ order,\n- * which means that a directory entry is ordered by adding\n- * a slash to the end of it.\n- *\n- * So a directory called \"a\" is ordered _after_ a file\n- * called \"a.c\", because \"a/\" sorts after \"a.c\".\n- */\n-#define TREE_UNORDERED (-1)\n-#define TREE_HAS_DUPS  (-2)\n-\n-static int verify_ordered(unsigned mode1, const char *name1, unsigned mode2, const char *name2)\n+static int fsck_sha1(const unsigned char *sha1)\n {\n-\tint len1 = strlen(name1);\n-\tint len2 = strlen(name2);\n-\tint len = len1 < len2 ? len1 : len2;\n-\tunsigned char c1, c2;\n-\tint cmp;\n-\n-\tcmp = memcmp(name1, name2, len);\n-\tif (cmp < 0)\n+\tstruct object *obj = parse_object(sha1);\n+\tif (!obj) {\n+\t\terrors_found |= ERROR_OBJECT;\n+\t\treturn error(\"%s: object corrupt or missing\",\n+\t\t\t     sha1_to_hex(sha1));\n+\t}\n+\tif (obj->flags & SEEN)\n \t\treturn 0;\n-\tif (cmp > 0)\n-\t\treturn TREE_UNORDERED;\n-\n-\t/*\n-\t * Ok, the first <len> characters are the same.\n-\t * Now we need to order the next one, but turn\n-\t * a '\\0' into a '/' for a directory entry.\n-\t */\n-\tc1 = name1[len];\n-\tc2 = name2[len];\n-\tif (!c1 && !c2)\n-\t\t/*\n-\t\t * git-write-tree used to write out a nonsense tree that has\n-\t\t * entries with the same name, one blob and one tree.  Make\n-\t\t * sure we do not have duplicate entries.\n-\t\t */\n-\t\treturn TREE_HAS_DUPS;\n-\tif (!c1 && S_ISDIR(mode1))\n-\t\tc1 = '/';\n-\tif (!c2 && S_ISDIR(mode2))\n-\t\tc2 = '/';\n-\treturn c1 < c2 ? 0 : TREE_UNORDERED;\n-}\n-\n-static int fsck_tree(struct tree *item)\n-{\n-\tint retval;\n-\tint has_full_path = 0;\n-\tint has_empty_name = 0;\n-\tint has_zero_pad = 0;\n-\tint has_bad_modes = 0;\n-\tint has_dup_entries = 0;\n-\tint not_properly_sorted = 0;\n-\tstruct tree_desc desc;\n-\tunsigned o_mode;\n-\tconst char *o_name;\n-\tconst unsigned char *o_sha1;\n+\tobj->flags |= SEEN;\n \n \tif (verbose)\n-\t\tfprintf(stderr, \"Checking tree %s\\n\",\n-\t\t\t\tsha1_to_hex(item->object.sha1));\n-\n-\tinit_tree_desc(&desc, item->buffer, item->size);\n-\n-\to_mode = 0;\n-\to_name = NULL;\n-\to_sha1 = NULL;\n-\twhile (desc.size) {\n-\t\tunsigned mode;\n-\t\tconst char *name;\n-\t\tconst unsigned char *sha1;\n-\n-\t\tsha1 = tree_entry_extract(&desc, &name, &mode);\n-\n-\t\tif (strchr(name, '/'))\n-\t\t\thas_full_path = 1;\n-\t\tif (!*name)\n-\t\t\thas_empty_name = 1;\n-\t\thas_zero_pad |= *(char *)desc.buffer == '0';\n-\t\tupdate_tree_entry(&desc);\n-\n-\t\tswitch (mode) {\n-\t\t/*\n-\t\t * Standard modes..\n-\t\t */\n-\t\tcase S_IFREG | 0755:\n-\t\tcase S_IFREG | 0644:\n-\t\tcase S_IFLNK:\n-\t\tcase S_IFDIR:\n-\t\tcase S_IFGITLINK:\n-\t\t\tbreak;\n-\t\t/*\n-\t\t * This is nonstandard, but we had a few of these\n-\t\t * early on when we honored the full set of mode\n-\t\t * bits..\n-\t\t */\n-\t\tcase S_IFREG | 0664:\n-\t\t\tif (!check_strict)\n-\t\t\t\tbreak;\n-\t\tdefault:\n-\t\t\thas_bad_modes = 1;\n-\t\t}\n+\t\tfprintf(stderr, \"Checking %s %s\\n\",\n+\t\t\ttypename(obj->type), sha1_to_hex(obj->sha1));\n \n-\t\tif (o_name) {\n-\t\t\tswitch (verify_ordered(o_mode, o_name, mode, name)) {\n-\t\t\tcase TREE_UNORDERED:\n-\t\t\t\tnot_properly_sorted = 1;\n-\t\t\t\tbreak;\n-\t\t\tcase TREE_HAS_DUPS:\n-\t\t\t\thas_dup_entries = 1;\n-\t\t\t\tbreak;\n-\t\t\tdefault:\n-\t\t\t\tbreak;\n-\t\t\t}\n-\t\t}\n+\tif(fsck_walk (obj, mark_used, 0))\n+\t\tobjerror(obj, \"broken links\");\n+\tif (fsck_object (obj, check_strict, fsck_error_func))\n+\t\treturn -1;\n \n-\t\to_mode = mode;\n-\t\to_name = name;\n-\t\to_sha1 = sha1;\n-\t}\n-\tfree(item->buffer);\n-\titem->buffer = NULL;\n+\tif (obj->type == OBJ_TREE) {\n+\t\tstruct tree *item = (struct tree*) obj;\n \n-\tretval = 0;\n-\tif (has_full_path) {\n-\t\tobjwarning(&item->object, \"contains full pathnames\");\n-\t}\n-\tif (has_empty_name) {\n-\t\tobjwarning(&item->object, \"contains empty pathname\");\n-\t}\n-\tif (has_zero_pad) {\n-\t\tobjwarning(&item->object, \"contains zero-padded file modes\");\n-\t}\n-\tif (has_bad_modes) {\n-\t\tobjwarning(&item->object, \"contains bad file modes\");\n+\t\tfree(item->buffer);\n+\t\titem->buffer = NULL;\n \t}\n-\tif (has_dup_entries) {\n-\t\tretval = objerror(&item->object, \"contains duplicate file entries\");\n-\t}\n-\tif (not_properly_sorted) {\n-\t\tretval = objerror(&item->object, \"not properly sorted\");\n-\t}\n-\treturn retval;\n-}\n \n-static int fsck_commit(struct commit *commit)\n-{\n-\tchar *buffer = commit->buffer;\n-\tunsigned char tree_sha1[20], sha1[20];\n-\tstruct commit_graft *graft;\n-\tint parents = 0;\n+\tif (obj->type == OBJ_COMMIT) {\n+\t\tstruct commit *commit = (struct commit*) obj;\n \n-\tif (verbose)\n-\t\tfprintf(stderr, \"Checking commit %s\\n\",\n-\t\t\tsha1_to_hex(commit->object.sha1));\n-\n-\tif (!commit->date)\n-\t\treturn objerror(&commit->object, \"invalid author/committer line\");\n-\n-\tif (memcmp(buffer, \"tree \", 5))\n-\t\treturn objerror(&commit->object, \"invalid format - expected 'tree' line\");\n-\tif (get_sha1_hex(buffer+5, tree_sha1) || buffer[45] != '\\n')\n-\t\treturn objerror(&commit->object, \"invalid 'tree' line format - bad sha1\");\n-\tbuffer += 46;\n-\twhile (!memcmp(buffer, \"parent \", 7)) {\n-\t\tif (get_sha1_hex(buffer+7, sha1) || buffer[47] != '\\n')\n-\t\t\treturn objerror(&commit->object, \"invalid 'parent' line format - bad sha1\");\n-\t\tbuffer += 48;\n-\t\tparents++;\n-\t}\n-\tgraft = lookup_commit_graft(commit->object.sha1);\n-\tif (graft) {\n-\t\tstruct commit_list *p = commit->parents;\n-\t\tparents = 0;\n-\t\twhile (p) {\n-\t\t\tp = p->next;\n-\t\t\tparents++;\n-\t\t}\n-\t\tif (graft->nr_parent == -1 && !parents)\n-\t\t\t; /* shallow commit */\n-\t\telse if (graft->nr_parent != parents)\n-\t\t\treturn objerror(&commit->object, \"graft objects missing\");\n-\t} else {\n-\t\tstruct commit_list *p = commit->parents;\n-\t\twhile (p && parents) {\n-\t\t\tp = p->next;\n-\t\t\tparents--;\n-\t\t}\n-\t\tif (p || parents)\n-\t\t\treturn objerror(&commit->object, \"parent objects missing\");\n-\t}\n-\tif (memcmp(buffer, \"author \", 7))\n-\t\treturn objerror(&commit->object, \"invalid format - expected 'author' line\");\n-\tfree(commit->buffer);\n-\tcommit->buffer = NULL;\n-\tif (!commit->tree)\n-\t\treturn objerror(&commit->object, \"could not load commit's tree %s\", tree_sha1);\n-\tif (!commit->parents && show_root)\n-\t\tprintf(\"root %s\\n\", sha1_to_hex(commit->object.sha1));\n-\treturn 0;\n-}\n+\t\tfree(commit->buffer);\n+\t\tcommit->buffer = NULL;\n \n-static int fsck_tag(struct tag *tag)\n-{\n-\tstruct object *tagged = tag->tagged;\n+\t\tif (!commit->parents && show_root)\n+\t\t\tprintf(\"root %s\\n\", sha1_to_hex(commit->object.sha1));\n+\t}\n \n-\tif (verbose)\n-\t\tfprintf(stderr, \"Checking tag %s\\n\",\n-\t\t\tsha1_to_hex(tag->object.sha1));\n+\tif (obj->type == OBJ_TAG) {\n+\t\tstruct tag *tag = (struct tag*) obj;\n \n-\tif (!tagged) {\n-\t\treturn objerror(&tag->object, \"could not load tagged object\");\n+\t\tif (show_tags && tag->tagged) {\n+\t\t\tprintf(\"tagged %s %s\", typename(tag->tagged->type), sha1_to_hex(tag->tagged->sha1));\n+\t\t\tprintf(\" (%s) in %s\\n\", tag->tag, sha1_to_hex(tag->object.sha1));\n+\t\t}\n \t}\n-\tif (!show_tags)\n-\t\treturn 0;\n \n-\tprintf(\"tagged %s %s\", typename(tagged->type), sha1_to_hex(tagged->sha1));\n-\tprintf(\" (%s) in %s\\n\", tag->tag, sha1_to_hex(tag->object.sha1));\n \treturn 0;\n }\n \n-static int fsck_sha1(const unsigned char *sha1)\n-{\n-\tstruct object *obj = parse_object(sha1);\n-\tif (!obj) {\n-\t\terrors_found |= ERROR_OBJECT;\n-\t\treturn error(\"%s: object corrupt or missing\",\n-\t\t\t     sha1_to_hex(sha1));\n-\t}\n-\tif (obj->flags & SEEN)\n-\t\treturn 0;\n-\tobj->flags |= SEEN;\n-\tif (fsck_walk(obj, mark_used, 0))\n-\t\tobjerror(obj, \"broken links\");\n-\tif (obj->type == OBJ_BLOB)\n-\t\treturn 0;\n-\tif (obj->type == OBJ_TREE)\n-\t\treturn fsck_tree((struct tree *) obj);\n-\tif (obj->type == OBJ_COMMIT)\n-\t\treturn fsck_commit((struct commit *) obj);\n-\tif (obj->type == OBJ_TAG)\n-\t\treturn fsck_tag((struct tag *) obj);\n-\n-\t/* By now, parse_object() would've returned NULL instead. */\n-\treturn objerror(obj, \"unknown type '%d' (internal fsck error)\",\n-\t\t\tobj->type);\n-}\n-\n /*\n  * This is the sorting chunk size: make it reasonably\n  * big so that we can sort well..\ndiff --git a/fsck.c b/fsck.c\nindex b7f9354..3ce33ff 100644\n--- a/fsck.c\n+++ b/fsck.c\n@@ -89,3 +89,225 @@ int fsck_walk(struct object *obj, fsck_walk_func walk, void *data)\n \t\treturn -1;\n \t}\n }\n+\n+/*\n+ * The entries in a tree are ordered in the _path_ order,\n+ * which means that a directory entry is ordered by adding\n+ * a slash to the end of it.\n+ *\n+ * So a directory called \"a\" is ordered _after_ a file\n+ * called \"a.c\", because \"a/\" sorts after \"a.c\".\n+ */\n+#define TREE_UNORDERED (-1)\n+#define TREE_HAS_DUPS  (-2)\n+\n+static int verify_ordered(unsigned mode1, const char *name1, unsigned mode2, const char *name2)\n+{\n+\tint len1 = strlen(name1);\n+\tint len2 = strlen(name2);\n+\tint len = len1 < len2 ? len1 : len2;\n+\tunsigned char c1, c2;\n+\tint cmp;\n+\n+\tcmp = memcmp(name1, name2, len);\n+\tif (cmp < 0)\n+\t\treturn 0;\n+\tif (cmp > 0)\n+\t\treturn TREE_UNORDERED;\n+\n+\t/*\n+\t * Ok, the first <len> characters are the same.\n+\t * Now we need to order the next one, but turn\n+\t * a '\\0' into a '/' for a directory entry.\n+\t */\n+\tc1 = name1[len];\n+\tc2 = name2[len];\n+\tif (!c1 && !c2)\n+\t\t/*\n+\t\t * git-write-tree used to write out a nonsense tree that has\n+\t\t * entries with the same name, one blob and one tree.  Make\n+\t\t * sure we do not have duplicate entries.\n+\t\t */\n+\t\treturn TREE_HAS_DUPS;\n+\tif (!c1 && S_ISDIR(mode1))\n+\t\tc1 = '/';\n+\tif (!c2 && S_ISDIR(mode2))\n+\t\tc2 = '/';\n+\treturn c1 < c2 ? 0 : TREE_UNORDERED;\n+}\n+\n+static int fsck_tree(struct tree *item, int strict, fsck_error error_func)\n+{\n+\tint retval;\n+\tint has_full_path = 0;\n+\tint has_empty_name = 0;\n+\tint has_zero_pad = 0;\n+\tint has_bad_modes = 0;\n+\tint has_dup_entries = 0;\n+\tint not_properly_sorted = 0;\n+\tstruct tree_desc desc;\n+\tunsigned o_mode;\n+\tconst char *o_name;\n+\tconst unsigned char *o_sha1;\n+\n+\tinit_tree_desc(&desc, item->buffer, item->size);\n+\n+\to_mode = 0;\n+\to_name = NULL;\n+\to_sha1 = NULL;\n+\tif (!desc.size)\n+\t\treturn error_func (&item->object, FSCK_ERROR, \"empty tree\");\n+\n+\twhile (desc.size) {\n+\t\tunsigned mode;\n+\t\tconst char *name;\n+\t\tconst unsigned char *sha1;\n+\n+\t\tsha1 = tree_entry_extract(&desc, &name, &mode);\n+\n+\t\tif (strchr(name, '/'))\n+\t\t\thas_full_path = 1;\n+\t\tif (!*name)\n+\t\t\thas_empty_name = 1;\n+\t\thas_zero_pad |= *(char *)desc.buffer == '0';\n+\t\tupdate_tree_entry(&desc);\n+\n+\t\tswitch (mode) {\n+\t\t/*\n+\t\t * Standard modes..\n+\t\t */\n+\t\tcase S_IFREG | 0755:\n+\t\tcase S_IFREG | 0644:\n+\t\tcase S_IFLNK:\n+\t\tcase S_IFDIR:\n+\t\tcase S_IFGITLINK:\n+\t\t\tbreak;\n+\t\t/*\n+\t\t * This is nonstandard, but we had a few of these\n+\t\t * early on when we honored the full set of mode\n+\t\t * bits..\n+\t\t */\n+\t\tcase S_IFREG | 0664:\n+\t\t\tif (!strict)\n+\t\t\t\tbreak;\n+\t\tdefault:\n+\t\t\thas_bad_modes = 1;\n+\t\t}\n+\n+\t\tif (o_name) {\n+\t\t\tswitch (verify_ordered(o_mode, o_name, mode, name)) {\n+\t\t\tcase TREE_UNORDERED:\n+\t\t\t\tnot_properly_sorted = 1;\n+\t\t\t\tbreak;\n+\t\t\tcase TREE_HAS_DUPS:\n+\t\t\t\thas_dup_entries = 1;\n+\t\t\t\tbreak;\n+\t\t\tdefault:\n+\t\t\t\tbreak;\n+\t\t\t}\n+\t\t}\n+\n+\t\to_mode = mode;\n+\t\to_name = name;\n+\t\to_sha1 = sha1;\n+\t}\n+\n+\tretval = 0;\n+\tif (has_full_path) {\n+\t\tretval += error_func(&item->object, FSCK_WARN, \"contains full pathnames\");\n+\t}\n+\tif (has_empty_name) {\n+\t\tretval += error_func(&item->object, FSCK_WARN, \"contains empty pathname\");\n+\t}\n+\tif (has_zero_pad) {\n+\t\tretval += error_func(&item->object, FSCK_WARN, \"contains zero-padded file modes\");\n+\t}\n+\tif (has_bad_modes) {\n+\t\tretval += error_func(&item->object, FSCK_WARN, \"contains bad file modes\");\n+\t}\n+\tif (has_dup_entries) {\n+\t\tretval += error_func(&item->object, FSCK_ERROR, \"contains duplicate file entries\");\n+\t}\n+\tif (not_properly_sorted) {\n+\t\tretval += error_func(&item->object, FSCK_ERROR, \"not properly sorted\");\n+\t}\n+\treturn retval;\n+}\n+\n+static int fsck_commit(struct commit *commit, fsck_error error_func)\n+{\n+\tchar *buffer = commit->buffer;\n+\tunsigned char tree_sha1[20], sha1[20];\n+\tstruct commit_graft *graft;\n+\tint parents = 0;\n+\n+\tif (!commit->date)\n+\t\treturn error_func(&commit->object, FSCK_ERROR, \"invalid author/committer line\");\n+\n+\tif (memcmp(buffer, \"tree \", 5))\n+\t\treturn error_func(&commit->object, FSCK_ERROR, \"invalid format - expected 'tree' line\");\n+\tif (get_sha1_hex(buffer+5, tree_sha1) || buffer[45] != '\\n')\n+\t\treturn error_func(&commit->object, FSCK_ERROR, \"invalid 'tree' line format - bad sha1\");\n+\tbuffer += 46;\n+\twhile (!memcmp(buffer, \"parent \", 7)) {\n+\t\tif (get_sha1_hex(buffer+7, sha1) || buffer[47] != '\\n')\n+\t\t\treturn error_func(&commit->object, FSCK_ERROR, \"invalid 'parent' line format - bad sha1\");\n+\t\tbuffer += 48;\n+\t\tparents++;\n+\t}\n+\tgraft = lookup_commit_graft(commit->object.sha1);\n+\tif (graft) {\n+\t\tstruct commit_list *p = commit->parents;\n+\t\tparents = 0;\n+\t\twhile (p) {\n+\t\t\tp = p->next;\n+\t\t\tparents++;\n+\t\t}\n+\t\tif (graft->nr_parent == -1 && !parents)\n+\t\t\t; /* shallow commit */\n+\t\telse if (graft->nr_parent != parents)\n+\t\t\treturn error_func(&commit->object, FSCK_ERROR, \"graft objects missing\");\n+\t} else {\n+\t\tstruct commit_list *p = commit->parents;\n+\t\twhile (p && parents) {\n+\t\t\tp = p->next;\n+\t\t\tparents--;\n+\t\t}\n+\t\tif (p || parents)\n+\t\t\treturn error_func(&commit->object, FSCK_ERROR, \"parent objects missing\");\n+\t}\n+\tif (memcmp(buffer, \"author \", 7))\n+\t\treturn error_func(&commit->object, FSCK_ERROR, \"invalid format - expected 'author' line\");\n+\tif (!commit->tree)\n+\t\treturn error_func(&commit->object, FSCK_ERROR, \"could not load commit's tree %s\", sha1_to_hex(tree_sha1));\n+\n+\treturn 0;\n+}\n+\n+static int fsck_tag(struct tag *tag, fsck_error error_func)\n+{\n+\tstruct object *tagged = tag->tagged;\n+\n+\tif (!tagged) {\n+\t\treturn error_func(&tag->object, FSCK_ERROR, \"could not load tagged object\");\n+\t}\n+\treturn 0;\n+}\n+\n+int fsck_object(struct object *obj, int strict, fsck_error error_func)\n+{\n+\tif (!obj)\n+\t\treturn error_func(obj, FSCK_ERROR, \"no valid object to fsck\");\n+\n+\tif (obj->type == OBJ_BLOB)\n+\t\treturn 0;\n+\tif (obj->type == OBJ_TREE)\n+\t\treturn fsck_tree((struct tree *) obj, strict, error_func);\n+\tif (obj->type == OBJ_COMMIT)\n+\t\treturn fsck_commit((struct commit *) obj, error_func);\n+\tif (obj->type == OBJ_TAG)\n+\t\treturn fsck_tag((struct tag *) obj, error_func);\n+\n+\treturn error_func(obj, FSCK_ERROR, \"unknown type '%d' (internal fsck error)\",\n+\t\t\t  obj->type);\n+}\ndiff --git a/fsck.h b/fsck.h\nindex 4a1f02a..a5d60d0 100644\n--- a/fsck.h\n+++ b/fsck.h\n@@ -1,6 +1,9 @@\n #ifndef GIT_FSCK_H\n #define GIT_FSCK_H\n \n+#define FSCK_ERROR 1\n+#define FSCK_WARN 2\n+\n /* \n  * callback function for fsck_walk\n  * type is the expected type of the object or OBJ_ANY\n@@ -11,6 +14,9 @@\n  */\n typedef int (*fsck_walk_func)(struct object *obj, int type, void *data);\n \n+/* callback for fsck_object, type is FSCK_ERROR or FSCK_WARN */\n+typedef int (*fsck_error)(struct object *obj, int type, const char *err, ...);\n+\n /* descend in all linked child objects\n  * the return value is:\n  *    -1\terror in processing the object\n@@ -19,5 +25,6 @@ typedef int (*fsck_walk_func)(struct object *obj, int type, void *data);\n  *    0\t\teverything OK\n  */\n int fsck_walk(struct object *obj, fsck_walk_func walk, void *data);\n+int fsck_object(struct object *obj, int strict, fsck_error error_func);\n \n #endif\n-- \n1.5.4.3.g3c5f\n"},{"id":"69933","messageId":"12039765001192-git-send-email-mkoegler@auto.tuwien.ac.at","threadId":"12316","inReplyTo":"12039765002397-git-send-email-mkoegler@auto.tuwien.ac.at","subject":"[PATCH 06/10] add common fsck error printing function","fromName":"Martin Koegler","fromEmail":"mkoegler@auto.tuwien.ac.at","sentAt":"2008-02-25T21:54:56Z","receivedAt":"2008-02-25T21:54:56Z","isPatch":true,"sender":{"key":"mkoegler@auto.tuwien.ac.at","avatar":null},"body":"Signed-off-by: Martin Koegler <mkoegler@auto.tuwien.ac.at>\n---\n fsck.c |   30 ++++++++++++++++++++++++++++++\n fsck.h |    2 ++\n 2 files changed, 32 insertions(+), 0 deletions(-)\n\ndiff --git a/fsck.c b/fsck.c\nindex 3ce33ff..a8c24a9 100644\n--- a/fsck.c\n+++ b/fsck.c\n@@ -311,3 +311,33 @@ int fsck_object(struct object *obj, int strict, fsck_error error_func)\n \treturn error_func(obj, FSCK_ERROR, \"unknown type '%d' (internal fsck error)\",\n \t\t\t  obj->type);\n }\n+\n+int fsck_error_function(struct object *obj, int type, const char* fmt, ...)\n+{\n+\tva_list ap;\n+\tint len;\n+\tstruct strbuf sb;\n+\t\n+\tstrbuf_init(&sb, 0);\n+\tstrbuf_addf(&sb, \"object %s:\", obj->sha1?sha1_to_hex(obj->sha1):\"(null)\");\n+\t\n+\tva_start(ap, fmt);\n+\tlen = vsnprintf(sb.buf + sb.len, strbuf_avail(&sb), fmt, ap);\n+\tva_end(ap);\n+\t\n+\tif (len < 0)\n+\t\tlen = 0;\n+\tif (len >= strbuf_avail(&sb)) {\n+\t\tstrbuf_grow(&sb, len + 2);\n+\t\tva_start(ap, fmt);\n+\t\tlen = vsnprintf(sb.buf + sb.len, strbuf_avail(&sb), fmt, ap);\n+\t\tva_end(ap);\n+\t\tif (len >= strbuf_avail(&sb)) {\n+\t\t\tdie(\"this should not happen, your snprintf is broken\");\n+\t\t}\n+\t}\n+\t\n+\terror(sb.buf);\n+\tstrbuf_release(&sb);\n+\treturn 1;\n+}\ndiff --git a/fsck.h b/fsck.h\nindex a5d60d0..188c84b 100644\n--- a/fsck.h\n+++ b/fsck.h\n@@ -17,6 +17,8 @@ typedef int (*fsck_walk_func)(struct object *obj, int type, void *data);\n /* callback for fsck_object, type is FSCK_ERROR or FSCK_WARN */\n typedef int (*fsck_error)(struct object *obj, int type, const char *err, ...);\n \n+int fsck_error_function(struct object *obj, int type, const char* fmt, ...);\n+\n /* descend in all linked child objects\n  * the return value is:\n  *    -1\terror in processing the object\n-- \n1.5.4.3.g3c5f\n"},{"id":"69932","messageId":"12039765002534-git-send-email-mkoegler@auto.tuwien.ac.at","threadId":"12316","inReplyTo":"12039765001192-git-send-email-mkoegler@auto.tuwien.ac.at","subject":"[PATCH 07/10] unpack-object: cache for non written objects","fromName":"Martin Koegler","fromEmail":"mkoegler@auto.tuwien.ac.at","sentAt":"2008-02-25T21:54:57Z","receivedAt":"2008-02-25T21:54:57Z","isPatch":true,"sender":{"key":"mkoegler@auto.tuwien.ac.at","avatar":null},"body":"Preventing objects with broken links entering the repository\nmeans, that write of some objects must be delayed.\n\nThis patch adds a cache to keep the object data in memory. The delta\nresolving code must also search in the cache.\n\nSigned-off-by: Martin Koegler <mkoegler@auto.tuwien.ac.at>\n---\n builtin-unpack-objects.c |   32 ++++++++++++++++++++++++++++++++\n 1 files changed, 32 insertions(+), 0 deletions(-)\n\ndiff --git a/builtin-unpack-objects.c b/builtin-unpack-objects.c\nindex 1e51865..f18c7e8 100644\n--- a/builtin-unpack-objects.c\n+++ b/builtin-unpack-objects.c\n@@ -8,6 +8,7 @@\n #include \"tag.h\"\n #include \"tree.h\"\n #include \"progress.h\"\n+#include \"decorate.h\"\n \n static int dry_run, quiet, recover, has_errors;\n static const char unpack_usage[] = \"git-unpack-objects [-n] [-q] [-r] < pack-file\";\n@@ -18,6 +19,28 @@ static unsigned int offset, len;\n static off_t consumed_bytes;\n static SHA_CTX ctx;\n \n+struct obj_buffer {\n+\tchar *buffer;\n+\tunsigned long size;\n+};\n+\n+static struct decoration obj_decorate;\n+\n+static struct obj_buffer *lookup_object_buffer(struct object *base)\n+{\n+\treturn lookup_decoration(&obj_decorate, base);\n+}\n+\n+static void add_object_buffer(struct object *object, char *buffer, unsigned long size)\n+{\n+\tstruct obj_buffer *obj;\n+\tobj = xcalloc(1, sizeof(struct obj_buffer));\n+\tobj->buffer = buffer;\n+\tobj->size = size;\n+\tif (add_decoration(&obj_decorate, object, obj))\n+\t\tdie(\"object %s tried to add buffer twice!\", sha1_to_hex(object->sha1));\n+}\n+\n /*\n  * Make sure at least \"min\" bytes are available in the buffer, and\n  * return the pointer to the buffer.\n@@ -252,6 +275,15 @@ static void unpack_delta_entry(enum object_type type, unsigned long delta_size,\n \t\t}\n \t}\n \n+\tstruct object* obj = lookup_object(base_sha1);\n+\tif (obj) {\n+\t\tstruct obj_buffer *obj_buf = lookup_object_buffer (obj);\n+\t\tif (obj_buf) {\n+\t\t\tresolve_delta(nr, obj->type, obj_buf->buffer, obj_buf->size, delta_data, delta_size);\n+\t\t\treturn;\n+\t\t}\n+\t}\n+\n \tbase = read_sha1_file(base_sha1, &type, &base_size);\n \tif (!base) {\n \t\terror(\"failed to read delta-pack base object %s\",\n-- \n1.5.4.3.g3c5f\n"},{"id":"69936","messageId":"12039765003644-git-send-email-mkoegler@auto.tuwien.ac.at","threadId":"12316","inReplyTo":"12039765002534-git-send-email-mkoegler@auto.tuwien.ac.at","subject":"[PATCH 08/10] unpack-objects: prevent writing of inconsistent objects","fromName":"Martin Koegler","fromEmail":"mkoegler@auto.tuwien.ac.at","sentAt":"2008-02-25T21:54:58Z","receivedAt":"2008-02-25T21:54:58Z","isPatch":true,"sender":{"key":"mkoegler@auto.tuwien.ac.at","avatar":null},"body":"This patch introduces a strict mode, which ensures that:\n- no malformed object will be written\n- no object with broken links will be written\n\nThe patch ensures this by delaying the write of all non blob object.\nThese object are written, after all objects they link to are written.\n\nAn error can only result in unreferenced objects.\n\nSigned-off-by: Martin Koegler <mkoegler@auto.tuwien.ac.at>\n---\n Documentation/git-unpack-objects.txt |    3 +\n builtin-unpack-objects.c             |  100 +++++++++++++++++++++++++++++++---\n 2 files changed, 96 insertions(+), 7 deletions(-)\n\ndiff --git a/Documentation/git-unpack-objects.txt b/Documentation/git-unpack-objects.txt\nindex b79be3f..3697896 100644\n--- a/Documentation/git-unpack-objects.txt\n+++ b/Documentation/git-unpack-objects.txt\n@@ -40,6 +40,9 @@ OPTIONS\n \tand make the best effort to recover as many objects as\n \tpossible.\n \n+--strict::\n+\tDon't write objects with broken content or links.\n+\n \n Author\n ------\ndiff --git a/builtin-unpack-objects.c b/builtin-unpack-objects.c\nindex f18c7e8..ec262d5 100644\n--- a/builtin-unpack-objects.c\n+++ b/builtin-unpack-objects.c\n@@ -7,11 +7,13 @@\n #include \"commit.h\"\n #include \"tag.h\"\n #include \"tree.h\"\n+#include \"tree-walk.h\"\n #include \"progress.h\"\n #include \"decorate.h\"\n+#include \"fsck.h\"\n \n-static int dry_run, quiet, recover, has_errors;\n-static const char unpack_usage[] = \"git-unpack-objects [-n] [-q] [-r] < pack-file\";\n+static int dry_run, quiet, recover, has_errors, strict;\n+static const char unpack_usage[] = \"git-unpack-objects [-n] [-q] [-r] [--strict] < pack-file\";\n \n /* We always read in 4kB chunks. */\n static unsigned char buffer[4096];\n@@ -144,9 +146,58 @@ static void add_delta_to_list(unsigned nr, unsigned const char *base_sha1,\n struct obj_info {\n \toff_t offset;\n \tunsigned char sha1[20];\n+\tstruct object * obj;\n };\n \n+#define FLAG_OPEN (1u<<20)\n+#define FLAG_WRITTEN (1u<<21)\n+\n static struct obj_info *obj_list;\n+unsigned nr_objects;\n+\n+static void write_cached_object(struct object* obj)\n+{\n+\tunsigned char sha1[20];\n+\tstruct obj_buffer *obj_buf = lookup_object_buffer(obj);\n+\tif (write_sha1_file(obj_buf->buffer, obj_buf->size, typename(obj->type), sha1) < 0)\n+\t\tdie(\"failed to write object %s\", sha1_to_hex(obj->sha1));\n+\tobj->flags |= FLAG_WRITTEN;\n+}\n+\n+static int check_object(struct object *obj, int type, void *data)\n+{\n+\tif (!obj)\n+\t\treturn 0;\n+\t\n+\tif (obj->flags & FLAG_WRITTEN)\n+\t\treturn 1;\n+\t\n+\tif (type != OBJ_ANY && obj->type != type)\n+\t\tdie(\"object type mismatch\");\n+\t\n+\tif (!(obj->flags & FLAG_OPEN)) {\n+\t\tunsigned long size;\n+\t\tint type = sha1_object_info (obj->sha1, &size);\n+\t\tif (type != obj->type || type <= 0)\n+\t\t\tdie(\"object of unexpected type\");\n+\t\tobj->flags |= FLAG_WRITTEN;\n+\t\treturn 1;\n+\t}\n+\t\n+\tif (fsck_object(obj, 1, fsck_error_function))\n+\t\tdie(\"Error in object\");\n+\tif (!fsck_walk(obj, check_object, 0))\n+\t\tdie(\"Error on reachable objects of %s\", sha1_to_hex(obj->sha1));\n+\twrite_cached_object(obj);\n+\treturn 1;\n+}\n+\n+static void write_rest()\n+{\n+\tunsigned i;\n+\tfor (i = 0; i < nr_objects; i++)\n+\t\tcheck_object(obj_list[i].obj, OBJ_ANY, 0);\n+}\n \n static void added_object(unsigned nr, enum object_type type,\n \t\t\t void *data, unsigned long size);\n@@ -154,9 +205,36 @@ static void added_object(unsigned nr, enum object_type type,\n static void write_object(unsigned nr, enum object_type type,\n \t\t\t void *buf, unsigned long size)\n {\n-\tif (write_sha1_file(buf, size, typename(type), obj_list[nr].sha1) < 0)\n-\t\tdie(\"failed to write object\");\n \tadded_object(nr, type, buf, size);\n+\tif (!strict) {\n+\t\tif (write_sha1_file(buf, size, typename(type), obj_list[nr].sha1) < 0)\n+\t\t\tdie(\"failed to write object\");\n+\t\tfree(buf);\n+\t\tobj_list[nr].obj = 0;\n+\t} else if (type == OBJ_BLOB) {\n+\t\tstruct blob * blob;\n+\t\tif (write_sha1_file(buf, size, typename(type), obj_list[nr].sha1) < 0)\n+\t\t\tdie(\"failed to write object\");\n+\t\tfree(buf);\n+\n+\t\tblob = lookup_blob (obj_list[nr].sha1);\n+\t\tif (blob)\n+\t\t\tblob->object.flags |= FLAG_WRITTEN;\n+\t\telse\n+\t\t\tdie(\"invalid blob object\");\n+\t\tobj_list[nr].obj = 0;\n+\t} else {\n+\t\tstruct object * obj;\n+\t\tint eaten;\n+\t\thash_sha1_file(buf, size, typename(type), obj_list[nr].sha1);\n+\t\tobj = parse_object_buffer(obj_list[nr].sha1, type, size, buf, &eaten);\n+\t\tif (!obj)\n+\t\t\tdie (\"invalid %s\", typename(type));\n+\t\t/* buf is stored via add_object_buffer and in obj, if its a tree or commit */\n+\t\tadd_object_buffer (obj, buf, size);\n+\t\tobj->flags |= FLAG_OPEN;\n+\t\tobj_list[nr].obj = obj;\n+\t}\n }\n \n static void resolve_delta(unsigned nr, enum object_type type,\n@@ -173,7 +251,6 @@ static void resolve_delta(unsigned nr, enum object_type type,\n \t\tdie(\"failed to apply delta\");\n \tfree(delta);\n \twrite_object(nr, type, result, result_size);\n-\tfree(result);\n }\n \n static void added_object(unsigned nr, enum object_type type,\n@@ -203,7 +280,8 @@ static void unpack_non_delta_entry(enum object_type type, unsigned long size,\n \n \tif (!dry_run && buf)\n \t\twrite_object(nr, type, buf, size);\n-\tfree(buf);\n+\telse\n+\t\tfree(buf);\n }\n \n static void unpack_delta_entry(enum object_type type, unsigned long delta_size,\n@@ -345,7 +423,8 @@ static void unpack_all(void)\n \tint i;\n \tstruct progress *progress = NULL;\n \tstruct pack_header *hdr = fill(sizeof(struct pack_header));\n-\tunsigned nr_objects = ntohl(hdr->hdr_entries);\n+\n+\tnr_objects = ntohl(hdr->hdr_entries);\n \n \tif (ntohl(hdr->hdr_signature) != PACK_SIGNATURE)\n \t\tdie(\"bad pack file\");\n@@ -356,6 +435,7 @@ static void unpack_all(void)\n \tif (!quiet)\n \t\tprogress = start_progress(\"Unpacking objects\", nr_objects);\n \tobj_list = xmalloc(nr_objects * sizeof(*obj_list));\n+\tmemset(obj_list, 0, nr_objects * sizeof(*obj_list));\n \tfor (i = 0; i < nr_objects; i++) {\n \t\tunpack_one(i);\n \t\tdisplay_progress(progress, i + 1);\n@@ -391,6 +471,10 @@ int cmd_unpack_objects(int argc, const char **argv, const char *prefix)\n \t\t\t\trecover = 1;\n \t\t\t\tcontinue;\n \t\t\t}\n+\t\t\tif (!strcmp(arg, \"--strict\")) {\n+\t\t\t\tstrict = 1;\n+\t\t\t\tcontinue;\n+\t\t\t}\n \t\t\tif (!prefixcmp(arg, \"--pack_header=\")) {\n \t\t\t\tstruct pack_header *hdr;\n \t\t\t\tchar *c;\n@@ -416,6 +500,8 @@ int cmd_unpack_objects(int argc, const char **argv, const char *prefix)\n \tunpack_all();\n \tSHA1_Update(&ctx, buffer, offset);\n \tSHA1_Final(sha1, &ctx);\n+\tif (strict)\n+\t\twrite_rest();\n \tif (hashcmp(fill(20), sha1))\n \t\tdie(\"final sha1 did not match\");\n \tuse(20);\n-- \n1.5.4.3.g3c5f\n"},{"id":"69937","messageId":"12039765001906-git-send-email-mkoegler@auto.tuwien.ac.at","threadId":"12316","inReplyTo":"12039765003644-git-send-email-mkoegler@auto.tuwien.ac.at","subject":"[PATCH 09/10] index-pack: introduce checking mode","fromName":"Martin Koegler","fromEmail":"mkoegler@auto.tuwien.ac.at","sentAt":"2008-02-25T21:54:59Z","receivedAt":"2008-02-25T21:54:59Z","isPatch":true,"sender":{"key":"mkoegler@auto.tuwien.ac.at","avatar":null},"body":"Adds strict option, which bails out if the pack would\nintroduces broken object or links in the repository.\n\nSigned-off-by: Martin Koegler <mkoegler@auto.tuwien.ac.at>\n---\nThe resource usage for (!strict) is the same.\nFor (strict) we need the struct blob/tree/commit/tag for\neach object (without any data).\n\n Documentation/git-index-pack.txt |    3 +\n index-pack.c                     |   86 +++++++++++++++++++++++++++++++++++++-\n 2 files changed, 88 insertions(+), 1 deletions(-)\n\ndiff --git a/Documentation/git-index-pack.txt b/Documentation/git-index-pack.txt\nindex 72b5d00..a7825b6 100644\n--- a/Documentation/git-index-pack.txt\n+++ b/Documentation/git-index-pack.txt\n@@ -75,6 +75,9 @@ OPTIONS\n \tto force the version for the generated pack index, and to force\n \t64-bit index entries on objects located above the given offset.\n \n+--strict::\n+\tDie, if the pack contains broken objects or links.\n+\n \n Note\n ----\ndiff --git a/index-pack.c b/index-pack.c\nindex 9fd6982..0bbf42e 100644\n--- a/index-pack.c\n+++ b/index-pack.c\n@@ -7,9 +7,10 @@\n #include \"tag.h\"\n #include \"tree.h\"\n #include \"progress.h\"\n+#include \"fsck.h\"\n \n static const char index_pack_usage[] =\n-\"git-index-pack [-v] [-o <index-file>] [{ ---keep | --keep=<msg> }] { <pack-file> | --stdin [--fix-thin] [<pack-file>] }\";\n+\"git-index-pack [-v] [-o <index-file>] [{ ---keep | --keep=<msg> }] [--strict] { <pack-file> | --stdin [--fix-thin] [<pack-file>] }\";\n \n struct object_entry\n {\n@@ -31,6 +32,9 @@ union delta_base {\n  */\n #define UNION_BASE_SZ\t20\n \n+#define FLAG_LINK (1u<<20)\n+#define FLAG_CHECKED (1u<<21)\n+\n struct delta_entry\n {\n \tunion delta_base base;\n@@ -44,6 +48,7 @@ static int nr_deltas;\n static int nr_resolved_deltas;\n \n static int from_stdin;\n+static int strict;\n static int verbose;\n \n static struct progress *progress;\n@@ -56,6 +61,48 @@ static SHA_CTX input_ctx;\n static uint32_t input_crc32;\n static int input_fd, output_fd, pack_fd;\n \n+static int mark_link(struct object* obj, int type, void *data)\n+{\n+\tif (!obj)\n+\t\treturn -1;\n+\n+\tif (type != OBJ_ANY && obj->type != type)\n+\t\tdie(\"object type mismatch at %s\", sha1_to_hex(obj->sha1));\n+\n+\tobj->flags |= FLAG_LINK;\n+\treturn 0;\n+}\n+\n+/* The content of each linked object must have been checked\n+   or it must be already present in the object database */\n+static void check_object(struct object* obj)\n+{\n+\tif (!obj)\n+\t\treturn;\n+\t\n+\tif (!(obj->flags & FLAG_LINK))\n+\t\treturn;\n+\t\n+\tif (!(obj->flags & FLAG_CHECKED)) {\n+\t\tunsigned long size;\n+\t\tint type = sha1_object_info (obj->sha1, &size);\n+\t\tif (type != obj->type || type <= 0)\n+\t\t\tdie(\"object of unexpected type\");\n+\t\tobj->flags |= FLAG_CHECKED;\n+\t\treturn;\n+\t}\n+}\n+\n+static void check_objects()\n+{\n+\tunsigned i, max;\n+\n+\tmax = get_max_object_index();\n+\tfor (i = 0; i < max; i++)\n+\t\tcheck_object(get_indexed_object(i));\n+}\n+\n+\n /* Discard current buffer used content. */\n static void flush(void)\n {\n@@ -341,6 +388,39 @@ static void sha1_object(const void *data, unsigned long size,\n \t\t\tdie(\"SHA1 COLLISION FOUND WITH %s !\", sha1_to_hex(sha1));\n \t\tfree(has_data);\n \t}\n+\tif (strict) {\n+\t\tif (type == OBJ_BLOB) {\n+\t\t\tstruct blob * blob = lookup_blob(sha1);\n+\t\t\tif (blob)\n+\t\t\t\tblob->object.flags |= FLAG_CHECKED;\n+\t\t\telse\n+\t\t\t\tdie(\"invalid blob object %s\", sha1_to_hex(sha1));\n+\t\t} else {\n+\t\t\tstruct object * obj;\n+\t\t\tint eaten;\n+\t\t\tvoid *buf = data;\n+\t\t\t\n+\t\t\t/* we do not need to free the memory here, as the buf is deleted\n+\t\t\t   by the caller */\n+\t\t\tobj = parse_object_buffer(sha1, type, size, buf, &eaten);\n+\t\t\tif (!obj)\n+\t\t\t\tdie(\"invalid %s\", typename(type));\n+\t\t        if (fsck_object(obj, 1, fsck_error_function))\n+\t\t\t\tdie(\"Error in object\");\n+\t\t\tif (fsck_walk(obj, mark_link, 0))\n+\t\t\t\tdie(\"Not all child objects of %s are reachable\", sha1_to_hex(obj->sha1));\n+\n+\t\t\tif (obj->type == OBJ_TREE) {\n+\t\t\t\tstruct tree *item = (struct item*) obj;\n+\t\t\t\titem->buffer = NULL;\n+\t\t\t}\n+\t\t\tif (obj->type == OBJ_COMMIT) {\n+\t\t\t\tstruct commit *commit = (struct commit*) obj;\n+\t\t\t\tcommit->buffer = NULL;\n+\t\t\t}\n+\t\t\tobj->flags |= FLAG_CHECKED;\n+\t\t}\n+\t}\n }\n \n static void resolve_delta(struct object_entry *delta_obj, void *base_data,\n@@ -714,6 +794,8 @@ int main(int argc, char **argv)\n \t\t\t\tfrom_stdin = 1;\n \t\t\t} else if (!strcmp(arg, \"--fix-thin\")) {\n \t\t\t\tfix_thin_pack = 1;\n+\t\t\t} else if (!strcmp(arg, \"--strict\")) {\n+\t\t\t\tstrict = 1;\n \t\t\t} else if (!strcmp(arg, \"--keep\")) {\n \t\t\t\tkeep_msg = \"\";\n \t\t\t} else if (!prefixcmp(arg, \"--keep=\")) {\n@@ -812,6 +894,8 @@ int main(int argc, char **argv)\n \t\t\t    nr_deltas - nr_resolved_deltas);\n \t}\n \tfree(deltas);\n+\tif (strict)\n+\t\tcheck_objects();\n \n \tidx_objects = xmalloc((nr_objects) * sizeof(struct pack_idx_entry *));\n \tfor (i = 0; i < nr_objects; i++)\n-- \n1.5.4.3.g3c5f\n"},{"id":"69935","messageId":"12039765002986-git-send-email-mkoegler@auto.tuwien.ac.at","threadId":"12316","inReplyTo":"12039765001906-git-send-email-mkoegler@auto.tuwien.ac.at","subject":"[PATCH 10/10] receive-pack: use strict mode for unpacking objects","fromName":"Martin Koegler","fromEmail":"mkoegler@auto.tuwien.ac.at","sentAt":"2008-02-25T21:55:00Z","receivedAt":"2008-02-25T21:55:00Z","isPatch":true,"sender":{"key":"mkoegler@auto.tuwien.ac.at","avatar":null},"body":"Signed-off-by: Martin Koegler <mkoegler@auto.tuwien.ac.at>\n---\n Documentation/config.txt |    6 ++++++\n receive-pack.c           |   36 +++++++++++++++++++++++-------------\n 2 files changed, 29 insertions(+), 13 deletions(-)\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex fb6dae0..f319e27 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -922,6 +922,12 @@ imap::\n \tThe configuration variables in the 'imap' section are described\n \tin linkgit:git-imap-send[1].\n \n+receive.fsckObjects::\n+\tIf it is set to true, git-receive-pack will check all received \n+\tobjects. It will abort in the case of a malformed object or a \n+\tbroken link. The result of an abort are only dangling objects.\n+\tThe default value is true.\n+\n receive.unpackLimit::\n \tIf the number of objects received in a push is below this\n \tlimit then the objects will be unpacked into loose object\ndiff --git a/receive-pack.c b/receive-pack.c\nindex 3267495..f5440ff 100644\n--- a/receive-pack.c\n+++ b/receive-pack.c\n@@ -10,6 +10,7 @@\n static const char receive_pack_usage[] = \"git-receive-pack <git-dir>\";\n \n static int deny_non_fast_forwards = 0;\n+static int receive_fsck_objects = 1;\n static int receive_unpack_limit = -1;\n static int transfer_unpack_limit = -1;\n static int unpack_limit = 100;\n@@ -35,6 +36,11 @@ static int receive_pack_config(const char *var, const char *value)\n \t\treturn 0;\n \t}\n \n+\tif (strcmp(var, \"receive.fsckobjects\") == 0) {\n+\t\treceive_fsck_objects = git_config_bool(var, value);\n+\t\treturn 0;\n+\t}\n+\n \treturn git_default_config(var, value);\n }\n \n@@ -367,11 +373,13 @@ static const char *unpack(void)\n \t\t\tntohl(hdr.hdr_version), ntohl(hdr.hdr_entries));\n \n \tif (ntohl(hdr.hdr_entries) < unpack_limit) {\n-\t\tint code;\n-\t\tconst char *unpacker[3];\n-\t\tunpacker[0] = \"unpack-objects\";\n-\t\tunpacker[1] = hdr_arg;\n-\t\tunpacker[2] = NULL;\n+\t\tint code, i = 0;\n+\t\tconst char *unpacker[4];\n+\t\tunpacker[i++] = \"unpack-objects\";\n+\t\tif (receive_fsck_objects)\n+\t\t\tunpacker[i++] = \"--strict\";\n+\t\tunpacker[i++] = hdr_arg;\n+\t\tunpacker[i++] = NULL;\n \t\tcode = run_command_v_opt(unpacker, RUN_GIT_CMD);\n \t\tswitch (code) {\n \t\tcase 0:\n@@ -392,8 +400,8 @@ static const char *unpack(void)\n \t\t\treturn \"unpacker exited with error code\";\n \t\t}\n \t} else {\n-\t\tconst char *keeper[6];\n-\t\tint s, status;\n+\t\tconst char *keeper[7];\n+\t\tint s, status, i = 0;\n \t\tchar keep_arg[256];\n \t\tstruct child_process ip;\n \n@@ -401,12 +409,14 @@ static const char *unpack(void)\n \t\tif (gethostname(keep_arg + s, sizeof(keep_arg) - s))\n \t\t\tstrcpy(keep_arg + s, \"localhost\");\n \n-\t\tkeeper[0] = \"index-pack\";\n-\t\tkeeper[1] = \"--stdin\";\n-\t\tkeeper[2] = \"--fix-thin\";\n-\t\tkeeper[3] = hdr_arg;\n-\t\tkeeper[4] = keep_arg;\n-\t\tkeeper[5] = NULL;\n+\t\tkeeper[i++] = \"index-pack\";\n+\t\tkeeper[i++] = \"--stdin\";\n+\t\tif (receive_fsck_objects)\n+\t\t\tkeeper[i++] = \"--strict\";\n+\t\tkeeper[i++] = \"--fix-thin\";\n+\t\tkeeper[i++] = hdr_arg;\n+\t\tkeeper[i++] = keep_arg;\n+\t\tkeeper[i++] = NULL;\n \t\tmemset(&ip, 0, sizeof(ip));\n \t\tip.argv = keeper;\n \t\tip.out = -1;\n-- \n1.5.4.3.g3c5f\n"},{"id":"69969","messageId":"7vskzg6pmw.fsf@gitster.siamese.dyndns.org","threadId":"12316","inReplyTo":"12039765002397-git-send-email-mkoegler@auto.tuwien.ac.at","subject":"Re: [PATCH 05/10] builtin-fsck: move common object checking code to fsck.c","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2008-02-26T09:19:51Z","receivedAt":"2008-02-26T09:19:51Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Is this series an unadjusted resend or something?  This particular one\nhad funny interaction with your own d4fe07f (git-fsck: report missing\nauthor/commit line in a commit as an error) that is already in\n'master', so I had to munge it by hand.  It was not so pleasant\n(a large chunk of code was moved from builtin-fsck.c to fsck.c),\nbut that is what the maintainer does, so it's Ok.  But I'd like\nyou to eyeball the result to see if it looks sane.\n\nI'll push it out as part of 'pu'.  The tip of your topic is\n154a955 (receive-pack: use strict mode for unpacking objects)\ntonight:\n\n\tSide note: To find a tip of a topic yourself, look for \"Merge\n\tmk/maint-parse-careful\" in \"git log --first-parent\n\torigin/next..origin/pu\" output and find the latest one.\n\nOne thing I noticed was that parse_$type_buffer() family all take\nnon-const void *buf pointers but one new caller you introduced takes\n\"const void *data\" and passes that pointer to them.  I hated to, but\nended up loose-casting it.  You may want to make the function take\nnon-const pointer, but I did not look very carefully.\n\nBy the way, while I was at it, many stylistic issues bugged me too\nmuch, so I ended up fixing them as well:\n\n * Trailing whitespaces; avoid them.\n\n * Indenting with SP not HT; don't.\n\n * Pointer to struct foo type is (struct foo *), not (struct foo*);\n\n * One SP each around comparison operator \"==\";\n\n * One SP around assignment operator \"=\";\n\n * One SP after \"if\", \"while\", \"switch\" and friends before \"(\";\n\n * No SP between function name and \"(\";\n\n * A function without parameter is \"static void foo(void)\", not\n   \"static void foo()\";\n\n * Decl-after-statement; don't.\n\n * Multi-line comment is:\n\n\t/*\n         * This is multi line comment\n         * and this is its second line.\n         */\n\n   not\n\n\t/* This is multi line comment\n           and this is its second */\n\n * If you cast, cast to the right type ;-)\n\n\tstruct tree *item = (struct tree *) obj;\n\n   not\n\n\tstruct tree *item = (struct item*) obj;\n\nPlease do not make me do this again, as I do not have infinite amount\nof time.  This plea is not only about your patch but applies to\neverybody.\n\nI wanted to merge a few new commits to existing topics to 'next' and\nmerge down a few well cooked topics to 'master', but ran out of time\ntonight.  New stuff I received and looked at are all parked in 'pu'\ntonight.\n"},{"id":"70033","messageId":"20080226213523.GA26618@auto.tuwien.ac.at","threadId":"12316","inReplyTo":"7vskzg6pmw.fsf@gitster.siamese.dyndns.org","subject":"Re: [PATCH 05/10] builtin-fsck: move common object checking code to fsck.c","fromName":"Martin Koegler","fromEmail":"mkoegler@auto.tuwien.ac.at","sentAt":"2008-02-26T21:35:23Z","receivedAt":"2008-02-26T21:35:23Z","isPatch":true,"sender":{"key":"mkoegler@auto.tuwien.ac.at","avatar":null},"body":"On Tue, Feb 26, 2008 at 01:19:51AM -0800, Junio C Hamano wrote:\n> Is this series an unadjusted resend or something?  This particular one\n> had funny interaction with your own d4fe07f (git-fsck: report missing\n> author/commit line in a commit as an error) that is already in\n> 'master', so I had to munge it by hand. \n\nI sent the series directly based on master, as Shawn suggested (21c34 is only changing the prefix in Makefile):\n3c5fb6a798a0b686e7818bf1da63791fb94a7b21 receive-pack: use strict mode for unpacking objects\n786bf704ce4067c80055a1fa69be242a59880eb0 index-pack: introduce checking mode\n1f7ae754550fb6e0509c1498ba9de6b5f4bba438 unpack-objects: prevent writing of inconsistent objects\n143aa20e11c70595e4119a3adac0887446524c7f unpack-object: cache for non written objects\n997a515fccb3ef200cb96fbb757366eff8a2ee66 add common fsck error printing function\nb0785b6b99c641b2fec99eb48da340af627e3b0d builtin-fsck: move common object checking code to fsck.c\nfa9c45a16cc194c87c113c9740eb5a6e17b66cc1 builtin-fsck: reports missing parent commits\na93e35027c53f06d2db2adbb14fa916871e23e46 Remove unused object-ref code\n19eae91b8e3d2e72616397edf77a13d4ac79d7ab builtin-fsck: move away from object-refs to fsck_walk\n0ca75709265281548be81cad4f396f4cf936dbfb add generic, type aware object chain walker\n21c34821c02458f45422e747853bde913d43c625 Lokale Anpassung Makefile\n99d8ea2c5ce6fc0b06fe8a43e7c0c108ddad853b git-bundle.txt: Add different strategies to create the bundle\n8e0fbe671f6a63b885702917bf4e7d7a85c59ab4 builtin-for-each-ref.c: fix typo in error message\n8a8bf4690e20a545561249a9b393c1ef3239c03d send-email: test compose functionality\n\nThe patch was sent as usual, so I don't know, why it should not apply.\n\n> It was not so pleasant\n> (a large chunk of code was moved from builtin-fsck.c to fsck.c),\n> but that is what the maintainer does, so it's Ok.  But I'd like\n> you to eyeball the result to see if it looks sane.\n\nI have compared it to 3c5fb6a798a0b686e7818bf1da63791fb94a7b21 and\neverything seems to look OK. I'll do better verification in the next\ndays.\n\n> I'll push it out as part of 'pu'.  The tip of your topic is\n> 154a955 (receive-pack: use strict mode for unpacking objects)\n> tonight:\n> \n> \tSide note: To find a tip of a topic yourself, look for \"Merge\n> \tmk/maint-parse-careful\" in \"git log --first-parent\n> \torigin/next..origin/pu\" output and find the latest one.\n\n> One thing I noticed was that parse_$type_buffer() family all take\n> non-const void *buf pointers but one new caller you introduced takes\n> \"const void *data\" and passes that pointer to them.  I hated to, but\n> ended up loose-casting it.  You may want to make the function take\n> non-const pointer, but I did not look very carefully.\n\nThe easiest thing would be to remove the const from the data parameter\nin sha1_object (index-pack.c).\n\nHow should I handle changes? Send a patch ontop of 154a955 or should I\nsend a amended version of the patches?\n\n> By the way, while I was at it, many stylistic issues bugged me too\n> much, so I ended up fixing them as well:\n>\n[...]\n> \n>  * If you cast, cast to the right type ;-)\n> \n> \tstruct tree *item = (struct tree *) obj;\n> \n>    not\n> \n> \tstruct tree *item = (struct item*) obj;\n>\n> Please do not make me do this again, as I do not have infinite amount\n> of time.  This plea is not only about your patch but applies to\n> everybody.\n\nSorry for struct item. I really should have looked more carefully at the\nmake output.\n\nI did not know all of these styling guidelines. SubmittingPatches only\ntalks about broken mailer. Maybe it would be a good thing to include\nthem somewhere.\n\nAs I'm not very good at catching all these issues, I tried\ncheckpatch.pl (from the linux.git:/scripts) on my patches. After\nturning the 80 characters/line check off, it show me formating errors,\nabout which you complained. So I'll try to run it over my patches in\nthe future.\n\nmfg Martin Kögler\n"},{"id":"70065","messageId":"7vr6eyx2j8.fsf@gitster.siamese.dyndns.org","threadId":"12316","inReplyTo":"20080226213523.GA26618@auto.tuwien.ac.at","subject":"Re: [PATCH 05/10] builtin-fsck: move common object checking code to fsck.c","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2008-02-27T07:48:59Z","receivedAt":"2008-02-27T07:48:59Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"mkoegler@auto.tuwien.ac.at (Martin Koegler) writes:\n\n> I have compared it to 3c5fb6a798a0b686e7818bf1da63791fb94a7b21 and\n> everything seems to look OK. I'll do better verification in the next\n> days.\n\nThanks.\n\n> How should I handle changes? Send a patch ontop of 154a955 or should I\n> send a amended version of the patches?\n\nThe rules under which I operate are (1) 'next', 'master', or\n'maint' will not rewind, hence (2) anything that is merged into\nthese three branches won't be amended, either.\n\nRunning \"git -p show-branch next master maint 154a955\" and\nscrolling to the end would show that up to \"peel_onion: handle\nNULL\" are in 'next' and 'master' (I wanted to make sure these\nsafety-tightening commits are fine and then wanted to merge them\neventually to 'maint', so this topic forked from 'maint' branch\nand is not meant to contain any new stuff in 'master').\n\n ! [next] Merge branch 'db/cover-letter' into next\n  ! [master] git-apply --whitespace=fix: fix off by one thinko\n   ! [maint] Documentation/git-am.txt: Pass -r in the example invoca...\n    ! [154a955] receive-pack: use strict mode for unpacking objects\n ----\n    ...\n    + [154a955] receive-pack: use strict mode for unpacking objects\n    + [154a955^] index-pack: introduce checking mode\n    + [154a955~2] unpack-objects: prevent writing of inconsistent objects\n    + [154a955~3] unpack-object: cache for non written objects\n    + [154a955~4] add common fsck error printing function\n    + [154a955~5] builtin-fsck: move common object checking code to fsck.c\n    + [154a955~6] builtin-fsck: reports missing parent commits\n    + [154a955~7] Remove unused object-ref code\n    + [154a955~8] builtin-fsck: move away from object-refs to fsck_walk\n    + [154a955~9] add generic, type aware object chain walker\n ++ + [154a955~10] peel_onion: handle NULL\n ++ + [154a955~11] check return value from parse_commit() in various functions\n ++ + [154a955~12] parse_commit: don't fail, if object is NULL\n ++ + [154a955~13] revision.c: handle tag->tagged == NULL\n ++ + [154a955~14] reachable.c::process_tree/blob: check for NULL\n ++ + [154a955~15] process_tag: handle tag->tagged == NULL\n ++ + [154a955~16] check results of parse_commit in merge_bases\n ++ + [154a955~17] list-objects.c::process_tree/blob: check for NULL\n ++ + [154a955~18] reachable.c::add_one_tree: handle NULL from lookup_tree\n ++ + [154a955~19] mark_blob/tree_uninteresting: check for NULL\n ++ + [154a955~20] get_sha1_oneline: check return value of parse_object\n ++ + [154a955~21] read_object_with_reference: don't read beyond the buffer\n ++++ [maint~20] GIT 1.5.4.2\n\nSo \"peel_onion\" fix and all commits below it are cast in stone.  But\n\"add generic walker\" and later ones are not, and we can do whatever we\nwant to them.\n\nIf you have additional checks for other commands, that are not included\nin 154a955, they naturally would make separate independent commits to be\napplied on top of 154a955.  But if you find embarrassing typo or a grave\nbug in them, you may want to send a replacement patch instead of\nincremental fix-up, because there is no point to record an earlier\nmistake in the public history only to later amend it, if mistakes are\nalready known.\n\nI am tempted to merge the ones up to \"peel_onion\" to 'maint' soon, by\nthe way.\n\n> I did not know all of these styling guidelines. SubmittingPatches only\n> talks about broken mailer. Maybe it would be a good thing to include\n> them somewhere.\n\nYeah, some words in CodingGuidelines might be a good idea.\n"}]}