{"thread":{"id":"6247","subject":"[PATCH 5/5] reflog expire --fix-stale","startedAt":"2007-01-06T10:16:19Z","lastAt":"2007-01-06T10:16:19Z","messageCount":1,"participants":["Junio C Hamano"],"isPatch":true,"patchVersion":1,"patchTotal":5},"messages":[{"id":"30964","messageId":"7vodpc5uks.fsf@assigned-by-dhcp.cox.net","threadId":"6247","inReplyTo":null,"subject":"[PATCH 5/5] reflog expire --fix-stale","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2007-01-06T10:16:19Z","receivedAt":"2007-01-06T10:16:19Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"The logic in an earlier round to detect reflog entries that\npoint at a broken commit was not sufficient.  Just like we do\nnot trust presense of a commit during pack transfer (we trust\nonly our refs), we should not trust a commit's presense, even if\nthe tree of that commit is complete.\n\nA repository that had reflog enabled on some of the refs that\nwas rewound and then run git-repack or git-prune from older\nversions of git can have reflog entries that point at a commit\nthat still exist but lack commits (or trees and blobs needed for\nthat commit) between it and some commit that is reachable from\none of the refs.\n\nThis revamps the logic -- the definition of \"broken commit\"\nbecomes: a commit that is not reachable from any of the refs and\nthere is a missing object among the commit, tree, or blob\nobjects reachable from it that is not reachable from any of the\nrefs.  Entries in the reflog that refer to such a commit are\nexpired.\n\nSince this computation involves traversing all the reachable\nobjects, i.e. it has the same cost as 'git prune', it is enabled\nonly when a new option --fix-stale.  Fortunately, once this is\nrun, we should not have to ever worry about missing objects,\nbecause the current prune and pack-objects know about reflogs\nand protect objects referred by them.\n\nUnfortunately, this will be absolutely necessary to help people\nmigrate to the newer prune and repack.\n\nSigned-off-by: Junio C Hamano <junkio@cox.net>\n---\n builtin-reflog.c  |  159 +++++++++++++++++++++++++++++++++++++++++++-------\n t/t1410-reflog.sh |  170 +++++++++++++++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 308 insertions(+), 21 deletions(-)\n\ndiff --git a/builtin-reflog.c b/builtin-reflog.c\nindex d3f2f50..1da7da0 100644\n--- a/builtin-reflog.c\n+++ b/builtin-reflog.c\n@@ -4,16 +4,34 @@\n #include \"refs.h\"\n #include \"dir.h\"\n #include \"tree-walk.h\"\n+#include \"diff.h\"\n+#include \"revision.h\"\n+#include \"reachable.h\"\n+\n+/*\n+ * reflog expire\n+ */\n+\n+static const char reflog_expire_usage[] =\n+\"git-reflog expire [--verbose] [--dry-run] [--fix-stale] [--expire=<time>] [--expire-unreachable=<time>] [--all] <refs>...\";\n \n static unsigned long default_reflog_expire;\n static unsigned long default_reflog_expire_unreachable;\n \n+struct cmd_reflog_expire_cb {\n+\tstruct rev_info revs;\n+\tint dry_run;\n+\tint stalefix;\n+\tint verbose;\n+\tunsigned long expire_total;\n+\tunsigned long expire_unreachable;\n+};\n+\n struct expire_reflog_cb {\n \tFILE *newlog;\n \tconst char *ref;\n \tstruct commit *ref_commit;\n-\tunsigned long expire_total;\n-\tunsigned long expire_unreachable;\n+\tstruct cmd_reflog_expire_cb *cmd;\n };\n \n static int tree_is_complete(const unsigned char *sha1)\n@@ -43,6 +61,83 @@ static int tree_is_complete(const unsigned char *sha1)\n \treturn 1;\n }\n \n+#define INCOMPLETE\t(1u<<10)\n+#define STUDYING\t(1u<<11)\n+\n+static int commit_is_complete(struct commit *commit)\n+{\n+\tstruct object_array study;\n+\tstruct object_array found;\n+\tint is_incomplete = 0;\n+\tint i;\n+\n+\t/* early return */\n+\tif (commit->object.flags & SEEN)\n+\t\treturn 1;\n+\tif (commit->object.flags & INCOMPLETE)\n+\t\treturn 0;\n+\t/*\n+\t * Find all commits that are reachable and are not marked as\n+\t * SEEN.  Then make sure the trees and blobs contained are\n+\t * complete.  After that, mark these commits also as SEEN.\n+\t * If some of the objects that are needed to complete this\n+\t * commit are missing, mark this commit as INCOMPLETE.\n+\t */\n+\tmemset(&study, 0, sizeof(study));\n+\tmemset(&found, 0, sizeof(found));\n+\tadd_object_array(&commit->object, NULL, &study);\n+\tadd_object_array(&commit->object, NULL, &found);\n+\tcommit->object.flags |= STUDYING;\n+\twhile (study.nr) {\n+\t\tstruct commit *c;\n+\t\tstruct commit_list *parent;\n+\n+\t\tc = (struct commit *)study.objects[--study.nr].item;\n+\t\tif (!c->object.parsed && !parse_object(c->object.sha1))\n+\t\t\tc->object.flags |= INCOMPLETE;\n+\n+\t\tif (c->object.flags & INCOMPLETE) {\n+\t\t\tis_incomplete = 1;\n+\t\t\tbreak;\n+\t\t}\n+\t\telse if (c->object.flags & SEEN)\n+\t\t\tcontinue;\n+\t\tfor (parent = c->parents; parent; parent = parent->next) {\n+\t\t\tstruct commit *p = parent->item;\n+\t\t\tif (p->object.flags & STUDYING)\n+\t\t\t\tcontinue;\n+\t\t\tp->object.flags |= STUDYING;\n+\t\t\tadd_object_array(&p->object, NULL, &study);\n+\t\t\tadd_object_array(&p->object, NULL, &found);\n+\t\t}\n+\t}\n+\tif (!is_incomplete) {\n+\t\t/* make sure all commits in found have all the\n+\t\t * necessary objects.\n+\t\t */\n+\t\tfor (i = 0; !is_incomplete && i < found.nr; i++) {\n+\t\t\tstruct commit *c =\n+\t\t\t\t(struct commit *)found.objects[i].item;\n+\t\t\tif (!tree_is_complete(c->tree->object.sha1))\n+\t\t\t\tis_incomplete = 1;\n+\t\t}\n+\t\tif (!is_incomplete) {\n+\t\t\t/* mark all found commits as complete, iow SEEN */\n+\t\t\tfor (i = 0; i < found.nr; i++)\n+\t\t\t\tfound.objects[i].item->flags |= SEEN;\n+\t\t}\n+\t}\n+\t/* clear flags from the objects we traversed */\n+\tfor (i = 0; i < found.nr; i++)\n+\t\tfound.objects[i].item->flags &= ~STUDYING;\n+\tif (is_incomplete)\n+\t\tcommit->object.flags |= INCOMPLETE;\n+\t/* free object arrays */\n+\tfree(study.objects);\n+\tfree(found.objects);\n+\treturn !is_incomplete;\n+}\n+\n static int keep_entry(struct commit **it, unsigned char *sha1)\n {\n \tstruct commit *commit;\n@@ -54,9 +149,15 @@ static int keep_entry(struct commit **it, unsigned char *sha1)\n \tif (!commit)\n \t\treturn 0;\n \n-\t/* Make sure everything in this commit exists. */\n-\tparse_object(commit->object.sha1);\n-\tif (!tree_is_complete(commit->tree->object.sha1))\n+\t/*\n+\t * Make sure everything in this commit exists.\n+\t *\n+\t * We have walked all the objects reachable from the refs\n+\t * and cache earlier.  The commits reachable by this commit\n+\t * must meet SEEN commits -- and then we should mark them as\n+\t * SEEN as well.\n+\t */\n+\tif (!commit_is_complete(commit))\n \t\treturn 0;\n \t*it = commit;\n \treturn 1;\n@@ -76,13 +177,14 @@ static int expire_reflog_ent(unsigned char *osha1, unsigned char *nsha1,\n \ttimestamp = strtoul(cp, &ep, 10);\n \tif (*ep != ' ')\n \t\tgoto prune;\n-\tif (timestamp < cb->expire_total)\n+\tif (timestamp < cb->cmd->expire_total)\n \t\tgoto prune;\n \n-\tif (!keep_entry(&old, osha1) || !keep_entry(&new, nsha1))\n+\tif (cb->cmd->stalefix &&\n+\t    (!keep_entry(&old, osha1) || !keep_entry(&new, nsha1)))\n \t\tgoto prune;\n \n-\tif ((timestamp < cb->expire_unreachable) &&\n+\tif ((timestamp < cb->cmd->expire_unreachable) &&\n \t    (!cb->ref_commit ||\n \t     (old && !in_merge_bases(old, cb->ref_commit)) ||\n \t     (new && !in_merge_bases(new, cb->ref_commit))))\n@@ -91,19 +193,15 @@ static int expire_reflog_ent(unsigned char *osha1, unsigned char *nsha1,\n \tif (cb->newlog)\n \t\tfprintf(cb->newlog, \"%s %s %s\",\n \t\t\tsha1_to_hex(osha1), sha1_to_hex(nsha1), data);\n+\tif (cb->cmd->verbose)\n+\t\tprintf(\"keep %s\", data);\n \treturn 0;\n  prune:\n-\tif (!cb->newlog)\n-\t\tfprintf(stderr, \"would prune %s\", data);\n+\tif (!cb->newlog || cb->cmd->verbose)\n+\t\tprintf(\"%sprune %s\", cb->newlog ? \"\" : \"would \", data);\n \treturn 0;\n }\n \n-struct cmd_reflog_expire_cb {\n-\tint dry_run;\n-\tunsigned long expire_total;\n-\tunsigned long expire_unreachable;\n-};\n-\n static int expire_reflog(const char *ref, const unsigned char *sha1, int unused, void *cb_data)\n {\n \tstruct cmd_reflog_expire_cb *cmd = cb_data;\n@@ -134,8 +232,7 @@ static int expire_reflog(const char *ref, const unsigned char *sha1, int unused,\n \t\tfprintf(stderr,\n \t\t\t\"warning: ref '%s' does not point at a commit\\n\", ref);\n \tcb.ref = ref;\n-\tcb.expire_total = cmd->expire_total;\n-\tcb.expire_unreachable = cmd->expire_unreachable;\n+\tcb.cmd = cmd;\n \tfor_each_reflog_ent(ref, expire_reflog_ent, &cb);\n  finish:\n \tif (cb.newlog) {\n@@ -164,9 +261,6 @@ static int reflog_expire_config(const char *var, const char *value)\n \treturn 0;\n }\n \n-static const char reflog_expire_usage[] =\n-\"git-reflog expire [--dry-run] [--expire=<time>] [--expire-unreachable=<time>] [--all] <refs>...\";\n-\n static int cmd_reflog_expire(int argc, const char **argv, const char *prefix)\n {\n \tstruct cmd_reflog_expire_cb cb;\n@@ -186,6 +280,12 @@ static int cmd_reflog_expire(int argc, const char **argv, const char *prefix)\n \tcb.expire_total = default_reflog_expire;\n \tcb.expire_unreachable = default_reflog_expire_unreachable;\n \n+\t/*\n+\t * We can trust the commits and objects reachable from refs\n+\t * even in older repository.  We cannot trust what's reachable\n+\t * from reflog if the repository was pruned with older git.\n+\t */\n+\n \tfor (i = 1; i < argc; i++) {\n \t\tconst char *arg = argv[i];\n \t\tif (!strcmp(arg, \"--dry-run\") || !strcmp(arg, \"-n\"))\n@@ -194,8 +294,12 @@ static int cmd_reflog_expire(int argc, const char **argv, const char *prefix)\n \t\t\tcb.expire_total = approxidate(arg + 9);\n \t\telse if (!strncmp(arg, \"--expire-unreachable=\", 21))\n \t\t\tcb.expire_unreachable = approxidate(arg + 21);\n+\t\telse if (!strcmp(arg, \"--stale-fix\"))\n+\t\t\tcb.stalefix = 1;\n \t\telse if (!strcmp(arg, \"--all\"))\n \t\t\tdo_all = 1;\n+\t\telse if (!strcmp(arg, \"--verbose\"))\n+\t\t\tcb.verbose = 1;\n \t\telse if (!strcmp(arg, \"--\")) {\n \t\t\ti++;\n \t\t\tbreak;\n@@ -205,6 +309,15 @@ static int cmd_reflog_expire(int argc, const char **argv, const char *prefix)\n \t\telse\n \t\t\tbreak;\n \t}\n+\tif (cb.stalefix) {\n+\t\tinit_revisions(&cb.revs, prefix);\n+\t\tif (cb.verbose)\n+\t\t\tprintf(\"Marking reachable objects...\");\n+\t\tmark_reachable_objects(&cb.revs, 0);\n+\t\tif (cb.verbose)\n+\t\t\tputchar('\\n');\n+\t}\n+\n \tif (do_all)\n \t\tstatus |= for_each_ref(expire_reflog, &cb);\n \twhile (i < argc) {\n@@ -219,6 +332,10 @@ static int cmd_reflog_expire(int argc, const char **argv, const char *prefix)\n \treturn status;\n }\n \n+/*\n+ * main \"reflog\"\n+ */\n+\n static const char reflog_usage[] =\n \"git-reflog (expire | ...)\";\n \ndiff --git a/t/t1410-reflog.sh b/t/t1410-reflog.sh\nnew file mode 100755\nindex 0000000..60d2f91\n--- /dev/null\n+++ b/t/t1410-reflog.sh\n@@ -0,0 +1,170 @@\n+#!/bin/sh\n+#\n+# Copyright (c) 2007 Junio C Hamano\n+#\n+\n+test_description='Test prune and reflog expiration'\n+. ./test-lib.sh\n+\n+check_have () {\n+\tgaah= &&\n+\tfor N in \"$@\"\n+\tdo\n+\t\teval \"o=\\$$N\" && git cat-file -t $o || {\n+\t\t\techo Gaah $N\n+\t\t\tgaah=$N\n+\t\t\tbreak\n+\t\t}\n+\tdone &&\n+\ttest -z \"$gaah\"\n+}\n+\n+check_fsck () {\n+\toutput=$(git fsck-objects --full)\n+\tcase \"$1\" in\n+\t'')\n+\t\ttest -z \"$output\" ;;\n+\t*)\n+\t\techo \"$output\" | grep \"$1\" ;;\n+\tesac\n+}\n+\n+corrupt () {\n+\taa=${1%??????????????????????????????????????} zz=${1#??}\n+\tmv .git/objects/$aa/$zz .git/$aa$zz\n+}\n+\n+recover () {\n+\taa=${1%??????????????????????????????????????} zz=${1#??}\n+\tmkdir -p .git/objects/$aa\n+\tmv .git/$aa$zz .git/objects/$aa/$zz\n+}\n+\n+check_dont_have () {\n+\tgaah= &&\n+\tfor N in \"$@\"\n+\tdo\n+\t\teval \"o=\\$$N\"\n+\t\tgit cat-file -t $o && {\n+\t\t\techo Gaah $N\n+\t\t\tgaah=$N\n+\t\t\tbreak\n+\t\t}\n+\tdone\n+\ttest -z \"$gaah\"\n+}\n+\n+test_expect_success setup '\n+\tmkdir -p A/B &&\n+\techo rat >C &&\n+\techo ox >A/D &&\n+\techo tiger >A/B/E &&\n+\tgit add . &&\n+\n+\ttest_tick && git commit -m rabbit &&\n+\tH=`git rev-parse --verify HEAD` &&\n+\tA=`git rev-parse --verify HEAD:A` &&\n+\tB=`git rev-parse --verify HEAD:A/B` &&\n+\tC=`git rev-parse --verify HEAD:C` &&\n+\tD=`git rev-parse --verify HEAD:A/D` &&\n+\tE=`git rev-parse --verify HEAD:A/B/E` &&\n+\tcheck_fsck &&\n+\n+\trm -f C A/B/E &&\n+\techo dragon >F &&\n+\techo snake >A/G &&\n+\tgit add F A/G &&\n+\ttest_tick && git commit -a -m horse &&\n+\tF=`git rev-parse --verify HEAD:F` &&\n+\tG=`git rev-parse --verify HEAD:A/G` &&\n+\tI=`git rev-parse --verify HEAD:A` &&\n+\tJ=`git rev-parse --verify HEAD` &&\n+\tcheck_fsck &&\n+\n+\trm -f A/G &&\n+\ttest_tick && git commit -a -m sheep &&\n+\tK=`git rev-parse --verify HEAD` &&\n+\tcheck_fsck &&\n+\n+\tcheck_have A B C D E F G H I J K &&\n+\n+\tgit prune &&\n+\n+\tcheck_have A B C D E F G H I J K &&\n+\n+\tcheck_fsck &&\n+\n+\tloglen=$(wc -l <.git/logs/refs/heads/master) &&\n+\ttest $loglen = 3\n+'\n+\n+test_expect_success rewind '\n+\ttest_tick && git reset --hard HEAD~2 &&\n+\ttest -f C &&\n+\ttest -f A/B/E &&\n+\t! test -f F &&\n+\t! test -f A/G &&\n+\n+\tcheck_have A B C D E F G H I J K &&\n+\n+\tgit prune &&\n+\n+\tcheck_have A B C D E F G H I J K &&\n+\n+\tloglen=$(wc -l <.git/logs/refs/heads/master) &&\n+\ttest $loglen = 4\n+'\n+\n+test_expect_success 'corrupt and check' '\n+\n+\tcorrupt $F &&\n+\tcheck_fsck \"missing blob $F\"\n+\n+'\n+\n+test_expect_success 'reflog expire --dry-run should not touch reflog' '\n+\n+\tgit reflog expire --dry-run \\\n+\t\t--expire=$(($test_tick - 10000)) \\\n+\t\t--expire-unreachable=$(($test_tick - 10000)) \\\n+\t\t--stale-fix \\\n+\t\t--all &&\n+\n+\tloglen=$(wc -l <.git/logs/refs/heads/master) &&\n+\ttest $loglen = 4 &&\n+\n+\tcheck_fsck \"missing blob $F\"\n+'\n+\n+test_expect_success 'reflog expire' '\n+\n+\tgit reflog expire --verbose \\\n+\t\t--expire=$(($test_tick - 10000)) \\\n+\t\t--expire-unreachable=$(($test_tick - 10000)) \\\n+\t\t--stale-fix \\\n+\t\t--all &&\n+\n+\tloglen=$(wc -l <.git/logs/refs/heads/master) &&\n+\ttest $loglen = 1 &&\n+\n+\tcheck_fsck \"dangling commit $K\"\n+'\n+\n+test_expect_success 'prune and fsck' '\n+\n+\tgit prune &&\n+\tcheck_fsck &&\n+\n+\tcheck_have A B C D E H &&\n+\tcheck_dont_have F G I J K\n+\n+'\n+\n+test_expect_success 'recover and check' '\n+\n+\trecover $F &&\n+\tcheck_fsck \"dangling blob $F\"\n+\n+'\n+\n+test_done\n-- \n1.5.0.rc0.ge0f6\n"}]}