threads / patch / 40418

patch, 4 partsFix gc failure when a remote HEAD goes stale

Subject: [PATCH 0/4] Fix gc failure when a remote HEAD goes stale

## tl;dr

28 messages between Sep 24, 2015 and Oct 8, 2015. Diffs are folded; open one to read it.

replies: 27people: 3as markdown or json

Johannes Schindelin· Sep 24, 2015, 09:13 UTC · lore

There has been a report in the Git for Windows project that gc fails sometimes: https://github.com/git-for-windows/git/issues/423

It turns out that there are cases when a remote HEAD can go stale and it is not the user's fault at all. It can happen, for example, if the active branch in the remote repository gets renamed.

Git's garbage collector should handle this gracefully. The best this developer could come up with, is to simply ignore and delete the now-broken refs.

Johannes Schindelin (4):
  gc: demonstrate failure with stale remote HEAD
  pack-objects: do not get distracted by stale refs
  mark_reachable_objects(): optionally collect broken refs
  gc: remove broken refs
 builtin/pack-objects.c |  1 +
 builtin/prune.c        | 12 +++++++++++-
 builtin/reflog.c       |  2 +-
 reachable.c            | 26 ++++++++++++++++++++------
 reachable.h            |  3 ++-
 t/t6500-gc.sh          | 15 +++++++++++++++
 6 files changed, 50 insertions(+), 9 deletions(-)
-- 
2.5.2.windows.2
Johannes Schindelin· Sep 24, 2015, 09:13 UTC · re: Johannes Schindelin · lore

[PATCH 1/4] gc: demonstrate failure with stale remote HEAD

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
 t/t6500-gc.sh | 15 +++++++++++++++
 1 file changed, 15 insertions(+)
Show changes to t/t6500-gc.sh +15 −0
diff --git a/t/t6500-gc.sh b/t/t6500-gc.sh
index 63194d8..b736774 100755
--- a/t/t6500-gc.sh
+++ b/t/t6500-gc.sh
@@ -30,4 +30,19 @@ test_expect_success 'gc -h with invalid configuration' '
 	test_i18ngrep "[Uu]sage" broken/usage
 '
 
+test_expect_failure 'gc removes broken refs/remotes/<name>/HEAD' '
+	git init remote &&
+	(
+		cd remote &&
+		test_commit initial &&
+		git clone . ../client &&
+		git branch -m develop &&
+		cd ../client &&
+		git fetch --prune &&
+		git gc &&
+		git branch --list -r origin/HEAD >actual &&
+		test_line_count = 0 actual
+	)
+'
+
 test_done
-- 
2.5.2.windows.2
Johannes Schindelin· Sep 24, 2015, 09:13 UTC · re: Johannes Schindelin · lore

[PATCH 2/4] pack-objects: do not get distracted by stale refs

It is quite possible for, say, a remote HEAD to become stale, e.g. when the default branch was renamed.

We should still be able to pack our objects when such a thing happens; simply ignore invalid refs (because they cannot matter for the packing process anyway).

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
 builtin/pack-objects.c | 1 +
 1 file changed, 1 insertion(+)
Show changes to builtin/pack-objects.c +1 −0
diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c
index 1c63f8f..ef2f794 100644
--- a/builtin/pack-objects.c
+++ b/builtin/pack-objects.c
@@ -2499,6 +2499,7 @@ static void get_object_list(int ac, const char **av)
 	int flags = 0;
 
 	init_revisions(&revs, NULL);
+	revs.ignore_missing = 1;
 	save_commit_buffer = 0;
 	setup_revisions(ac, av, &revs, NULL);
 
-- 
2.5.2.windows.2
Jeff King· Sep 24, 2015, 17:03 UTC · re: Johannes Schindelin · lore

Re: [PATCH 2/4] pack-objects: do not get distracted by stale refs

On Thu, Sep 24, 2015 at 11:13:44AM +0200, Johannes Schindelin wrote:
Show 9 quoted lines
> It is quite possible for, say, a remote HEAD to become stale, e.g. when
> the default branch was renamed.
> 
> We should still be able to pack our objects when such a thing happens;
> simply ignore invalid refs (because they cannot matter for the packing
> process anyway).
> [...]
>  	init_revisions(&revs, NULL);
> +	revs.ignore_missing = 1;

I think this is dangerous, because a repack may drop unreferenced history. Imagine you have a corrupted repository where you are missing the commit at the tip of "master", but still have "master^" and the rest of the history, and "git gc --auto" triggers.

Right now, pack-objects will barf and refuse to pack. Your repo is still corrupted, but you can use git-fsck to recover the history minus the single commit. With your patch, the repack will delete the entire history. Of course there are complications like other branches referencing the history, reflogs if it is a non-bare repo, and the 2-week expiration time. But there's definitely a potential to make things worse. I think commands that drop objects (like repack and prune) need to be more careful than other commands about refusing to run when there is something fishy in the repository.

I think we would want to introduce a flag to ignore dangling symbolic refs when running for_each_ref, as those should be harmless. In fact, I think we could omit symrefs completely for reachability analysis; we would hit the ref they refer to anyway during the rest of the traversal.

-Peff
Johannes Schindelin· Sep 24, 2015, 09:13 UTC · re: Johannes Schindelin · lore

[PATCH 3/4] mark_reachable_objects(): optionally collect broken refs

The behavior of `mark_reachable_objects()` without this patch is that it dies if it encounters a broken ref. This is sometimes undesirable, e.g. when garbage collecting in a repository with a stale remote HEAD.

So let's introduce an optional parameter to collect such broken refs. The behavior of the function is unchanged if that parameter is `NULL`.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
 builtin/prune.c  |  2 +-
 builtin/reflog.c |  2 +-
 reachable.c      | 26 ++++++++++++++++++++------
 reachable.h      |  3 ++-
 4 files changed, 24 insertions(+), 9 deletions(-)
Show changes to 4 files +24 −9

builtin/prune.c, builtin/reflog.c, reachable.c, reachable.h

diff --git a/builtin/prune.c b/builtin/prune.c
index 10b03d3..d6f664f 100644
--- a/builtin/prune.c
+++ b/builtin/prune.c
@@ -136,7 +136,7 @@ int cmd_prune(int argc, const char **argv, const char *prefix)
 	if (show_progress)
 		progress = start_progress_delay(_("Checking connectivity"), 0, 0, 2);
 
-	mark_reachable_objects(&revs, 1, expire, progress);
+	mark_reachable_objects(&revs, 1, expire, progress, NULL);
 	stop_progress(&progress);
 	for_each_loose_file_in_objdir(get_object_directory(), prune_object,
 				      prune_cruft, prune_subdir, NULL);
diff --git a/builtin/reflog.c b/builtin/reflog.c
index f96ca2a..cb8758a 100644
--- a/builtin/reflog.c
+++ b/builtin/reflog.c
@@ -583,7 +583,7 @@ static int cmd_reflog_expire(int argc, const char **argv, const char *prefix)
 		init_revisions(&cb.cmd.revs, prefix);
 		if (flags & EXPIRE_REFLOGS_VERBOSE)
 			printf("Marking reachable objects...");
-		mark_reachable_objects(&cb.cmd.revs, 0, 0, NULL);
+		mark_reachable_objects(&cb.cmd.revs, 0, 0, NULL, NULL);
 		if (flags & EXPIRE_REFLOGS_VERBOSE)
 			putchar('\n');
 	}
diff --git a/reachable.c b/reachable.c
index 9cff25b..1fc7ada 100644
--- a/reachable.c
+++ b/reachable.c
@@ -15,6 +15,11 @@ struct connectivity_progress {
 	unsigned long count;
 };
 
+struct add_one_data {
+	struct rev_info *revs;
+	struct string_list *broken_refs;
+};
+
 static void update_progress(struct connectivity_progress *cp)
 {
 	cp->count++;
@@ -25,10 +30,14 @@ static void update_progress(struct connectivity_progress *cp)
 static int add_one_ref(const char *path, const struct object_id *oid,
 		       int flag, void *cb_data)
 {
-	struct object *object = parse_object_or_die(oid->hash, path);
-	struct rev_info *revs = (struct rev_info *)cb_data;
+	struct add_one_data *data = (struct add_one_data *)cb_data;
+	struct object *object = data->broken_refs ? parse_object(oid->hash) :
+		parse_object_or_die(oid->hash, path);
 
-	add_pending_object(revs, object, "");
+	if (!object)
+		string_list_append(data->broken_refs, path);
+	else
+		add_pending_object(data->revs, object, "");
 
 	return 0;
 }
@@ -153,9 +162,11 @@ int add_unseen_recent_objects_to_traversal(struct rev_info *revs,
 
 void mark_reachable_objects(struct rev_info *revs, int mark_reflog,
 			    unsigned long mark_recent,
-			    struct progress *progress)
+			    struct progress *progress,
+			    struct string_list *broken_refs)
 {
 	struct connectivity_progress cp;
+	struct add_one_data data;
 
 	/*
 	 * Set up revision parsing, and mark us as being interested
@@ -168,11 +179,14 @@ void mark_reachable_objects(struct rev_info *revs, int mark_reflog,
 	/* Add all refs from the index file */
 	add_index_objects_to_pending(revs, 0);
 
+	data.revs = revs;
+	data.broken_refs = broken_refs;
+
 	/* Add all external refs */
-	for_each_ref(add_one_ref, revs);
+	for_each_ref(add_one_ref, &data);
 
 	/* detached HEAD is not included in the list above */
-	head_ref(add_one_ref, revs);
+	head_ref(add_one_ref, &data);
 
 	/* Add all reflog info */
 	if (mark_reflog)
diff --git a/reachable.h b/reachable.h
index d23efc3..39de1c7 100644
--- a/reachable.h
+++ b/reachable.h
@@ -5,6 +5,7 @@ struct progress;
 extern int add_unseen_recent_objects_to_traversal(struct rev_info *revs,
 						  unsigned long timestamp);
 extern void mark_reachable_objects(struct rev_info *revs, int mark_reflog,
-				   unsigned long mark_recent, struct progress *);
+				   unsigned long mark_recent, struct progress *,
+				   struct string_list *broken_refs);
 
 #endif
-- 
2.5.2.windows.2
Jeff King· Sep 24, 2015, 17:56 UTC · re: Johannes Schindelin · lore

Re: [PATCH 3/4] mark_reachable_objects(): optionally collect broken refs

On Thu, Sep 24, 2015 at 11:13:52AM +0200, Johannes Schindelin wrote:
Show 6 quoted lines
> The behavior of `mark_reachable_objects()` without this patch is that it
> dies if it encounters a broken ref. This is sometimes undesirable, e.g.
> when garbage collecting in a repository with a stale remote HEAD.
> 
> So let's introduce an optional parameter to collect such broken refs. The
> behavior of the function is unchanged if that parameter is `NULL`.
Similar comment to the last one. :)

I suspect the issues you are seeing are largely new due to the ref-paranoia work I did (merged in 05e816e37). We used to ignore broken refs at the for_each_ref() level, but now we feed them to the calling code (which generally chokes).

So in that sense, a simpler fix than your series would be to simply revert 8d42299 and ff4056bbc. :)

But I think those new checks are valuable, and we really just need to gracefully ignore the dangling-symref case (which is _a_ breakage, but not a dangerous one).

-Peff
Johannes Schindelin· Sep 24, 2015, 09:14 UTC · re: Johannes Schindelin · lore

[PATCH 4/4] gc: remove broken refs

When encountering broken refs, such as a stale remote HEAD (which can happen if the active branch was renamed in the remote), it is more helpful to remove those refs than to exit with an error.

This fixes https://github.com/git-for-windows/git/issues/423
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
 builtin/prune.c | 12 +++++++++++-
 t/t6500-gc.sh   |  2 +-
 2 files changed, 12 insertions(+), 2 deletions(-)
Show changes to 2 files +12 −2

builtin/prune.c, t/t6500-gc.sh

diff --git a/builtin/prune.c b/builtin/prune.c
index d6f664f..1a30f65 100644
--- a/builtin/prune.c
+++ b/builtin/prune.c
@@ -6,6 +6,7 @@
 #include "reachable.h"
 #include "parse-options.h"
 #include "progress.h"
+#include "refs.h"
 
 static const char * const prune_usage[] = {
 	N_("git prune [-n] [-v] [--expire <time>] [--] [<head>...]"),
@@ -100,6 +101,7 @@ static void remove_temporary_files(const char *path)
 int cmd_prune(int argc, const char **argv, const char *prefix)
 {
 	struct rev_info revs;
+	struct string_list broken_refs = STRING_LIST_INIT_DUP;
 	struct progress *progress = NULL;
 	const struct option options[] = {
 		OPT__DRY_RUN(&show_only, N_("do not remove, show only")),
@@ -110,6 +112,7 @@ int cmd_prune(int argc, const char **argv, const char *prefix)
 		OPT_END()
 	};
 	char *s;
+	int i;
 
 	expire = ULONG_MAX;
 	save_commit_buffer = 0;
@@ -136,7 +139,14 @@ int cmd_prune(int argc, const char **argv, const char *prefix)
 	if (show_progress)
 		progress = start_progress_delay(_("Checking connectivity"), 0, 0, 2);
 
-	mark_reachable_objects(&revs, 1, expire, progress, NULL);
+	revs.ignore_missing = 1;
+	mark_reachable_objects(&revs, 1, expire, progress, &broken_refs);
+	for (i = 0; i < broken_refs.nr; i++) {
+		char *path = broken_refs.items[i].string;
+		printf("Removing stale ref %s\n", path);
+		if (!show_only && delete_ref(path, NULL, REF_NODEREF))
+			die("Could not remove stale ref %s", path);
+	}
 	stop_progress(&progress);
 	for_each_loose_file_in_objdir(get_object_directory(), prune_object,
 				      prune_cruft, prune_subdir, NULL);
diff --git a/t/t6500-gc.sh b/t/t6500-gc.sh
index b736774..0ae4271 100755
--- a/t/t6500-gc.sh
+++ b/t/t6500-gc.sh
@@ -30,7 +30,7 @@ test_expect_success 'gc -h with invalid configuration' '
 	test_i18ngrep "[Uu]sage" broken/usage
 '
 
-test_expect_failure 'gc removes broken refs/remotes/<name>/HEAD' '
+test_expect_success 'gc removes broken refs/remotes/<name>/HEAD' '
 	git init remote &&
 	(
 		cd remote &&
-- 
2.5.2.windows.2
Jeff King· Sep 24, 2015, 17:57 UTC · re: Johannes Schindelin · lore

Re: [PATCH 4/4] gc: remove broken refs

On Thu, Sep 24, 2015 at 11:14:02AM +0200, Johannes Schindelin wrote:
> When encountering broken refs, such as a stale remote HEAD (which can
> happen if the active branch was renamed in the remote), it is more
> helpful to remove those refs than to exit with an error.

For the same reasons as in my earlier responses, I think it's dangerous to remove broken refs (it makes a small corruption much worse). It seems reasonably sane to remove a dangling symref, though if we teach for_each_ref to gracefully skip them, then leaving them in place isn't a problem.

-Peff
Junio C Hamano· Sep 25, 2015, 00:08 UTC · re: Jeff King · lore

Re: [PATCH 4/4] gc: remove broken refs

Jeff King <peff@peff.net> writes:
Show 5 quoted lines
> For the same reasons as in my earlier responses, I think it's dangerous
> to remove broken refs (it makes a small corruption much worse). It seems
> reasonably sane to remove a dangling symref, though if we teach
> for_each_ref to gracefully skip them, then leaving them in place isn't a
> problem.

One thing I wondered was if we can reliably tell between a ref that wanted to be a real ref that records a broken object name and a ref that wanted to be a symbolic ref that points a bogus thing, and if we can't, should we worry about it too much. The former is more serious, as the history behind the commit it wanted to but failed to record is at risk of being pruned.

One case that is clearly safe is "ref: refs/heads/gone"; it is not likely to be the result of attempting to write a real object name gone bad by whatever filesystem corruption. On the other hand, an obviously problematic case is an empty file. We cannot tell if the "broken" ref used to anchor the tip of a real history (which is about to be lost with Dscho's patch 1/4) or was merely pointing at another ref (which will not harm the object database if ignored).

So the rule should be
    If resolve_ref_unsafe_1() says it is a symbolic ref, if
    check_ref_format() is OK with the ref it points at, and if that
    pointee is missing, then it is safe to skip.
All other funnies should trigger the safety.

The collection of "broken and can be removed" refs introduced by 3/4 may also have to take that into account, I think.

Jeff King· Sep 25, 2015, 01:35 UTC · re: Junio C Hamano · lore

Re: [PATCH 4/4] gc: remove broken refs

On Thu, Sep 24, 2015 at 05:08:41PM -0700, Junio C Hamano wrote:
Show 22 quoted lines
> One thing I wondered was if we can reliably tell between a ref that
> wanted to be a real ref that records a broken object name and a ref
> that wanted to be a symbolic ref that points a bogus thing, and if
> we can't, should we worry about it too much.  The former is more
> serious, as the history behind the commit it wanted to but failed to
> record is at risk of being pruned.
> 
> One case that is clearly safe is "ref: refs/heads/gone"; it is not
> likely to be the result of attempting to write a real object name
> gone bad by whatever filesystem corruption.  On the other hand, an
> obviously problematic case is an empty file.  We cannot tell if the
> "broken" ref used to anchor the tip of a real history (which is
> about to be lost with Dscho's patch 1/4) or was merely pointing at
> another ref (which will not harm the object database if ignored).
> 
> So the rule should be
> 
>     If resolve_ref_unsafe_1() says it is a symbolic ref, if
>     check_ref_format() is OK with the ref it points at, and if that
>     pointee is missing, then it is safe to skip.
> 
> All other funnies should trigger the safety.

Right, I agree with that rule. If we don't know what it is, we should err on the conservative side (and that _shouldn't_ happen, because does not generate files in .git/refs that it cannot itself understand). But "ref: " is clearly something we understand.

-Peff
Johannes Schindelin· Sep 28, 2015, 14:01 UTC · re: Junio C Hamano · lore

[PATCH v2 0/4] Fix gc failure when a remote HEAD goes stale

There has been a report in the Git for Windows project that gc fails sometimes: https://github.com/git-for-windows/git/issues/423

It turns out that there are cases when a remote HEAD can go stale and it is not the user's fault at all. It can happen, for example, if the active branch in the remote repository gets renamed.

Git's garbage collector should handle this gracefully. The best this developer could come up with, is to simply warn and delete broken symrefs.

Thanks to Junio and Peff for their really valuable sanity check.
Interdiff vs v1 below diffstat.
Johannes Schindelin (4):
  gc: demonstrate failure with stale remote HEAD
  pack-objects: do not get distracted by broken symrefs
  mark_reachable_objects(): optionally collect broken symrefs
  gc: remove broken refs
 builtin/prune.c  | 12 +++++++++++-
 builtin/reflog.c |  2 +-
 reachable.c      | 31 +++++++++++++++++++++++++------
 reachable.h      |  3 ++-
 t/t6500-gc.sh    | 15 +++++++++++++++
 5 files changed, 54 insertions(+), 9 deletions(-)
Show changes to 4 files +19 −15

builtin/pack-objects.c, builtin/prune.c, reachable.c, reachable.h

diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c
index ef2f794..1c63f8f 100644
--- a/builtin/pack-objects.c
+++ b/builtin/pack-objects.c
@@ -2499,7 +2499,6 @@ static void get_object_list(int ac, const char **av)
 	int flags = 0;
 
 	init_revisions(&revs, NULL);
-	revs.ignore_missing = 1;
 	save_commit_buffer = 0;
 	setup_revisions(ac, av, &revs, NULL);
 
diff --git a/builtin/prune.c b/builtin/prune.c
index 1a30f65..337b12a 100644
--- a/builtin/prune.c
+++ b/builtin/prune.c
@@ -101,7 +101,7 @@ static void remove_temporary_files(const char *path)
 int cmd_prune(int argc, const char **argv, const char *prefix)
 {
 	struct rev_info revs;
-	struct string_list broken_refs = STRING_LIST_INIT_DUP;
+	struct string_list broken_symrefs = STRING_LIST_INIT_DUP;
 	struct progress *progress = NULL;
 	const struct option options[] = {
 		OPT__DRY_RUN(&show_only, N_("do not remove, show only")),
@@ -140,9 +140,9 @@ int cmd_prune(int argc, const char **argv, const char *prefix)
 		progress = start_progress_delay(_("Checking connectivity"), 0, 0, 2);
 
 	revs.ignore_missing = 1;
-	mark_reachable_objects(&revs, 1, expire, progress, &broken_refs);
-	for (i = 0; i < broken_refs.nr; i++) {
-		char *path = broken_refs.items[i].string;
+	mark_reachable_objects(&revs, 1, expire, progress, &broken_symrefs);
+	for (i = 0; i < broken_symrefs.nr; i++) {
+		char *path = broken_symrefs.items[i].string;
 		printf("Removing stale ref %s\n", path);
 		if (!show_only && delete_ref(path, NULL, REF_NODEREF))
 			die("Could not remove stale ref %s", path);
diff --git a/reachable.c b/reachable.c
index 1fc7ada..25c4932 100644
--- a/reachable.c
+++ b/reachable.c
@@ -17,7 +17,7 @@ struct connectivity_progress {
 
 struct add_one_data {
 	struct rev_info *revs;
-	struct string_list *broken_refs;
+	struct string_list *broken_symrefs;
 };
 
 static void update_progress(struct connectivity_progress *cp)
@@ -31,13 +31,18 @@ static int add_one_ref(const char *path, const struct object_id *oid,
 		       int flag, void *cb_data)
 {
 	struct add_one_data *data = (struct add_one_data *)cb_data;
-	struct object *object = data->broken_refs ? parse_object(oid->hash) :
-		parse_object_or_die(oid->hash, path);
+	struct object *object;
 
-	if (!object)
-		string_list_append(data->broken_refs, path);
-	else
-		add_pending_object(data->revs, object, "");
+	if ((flag & REF_ISSYMREF) && (flag & REF_ISBROKEN)) {
+		if (data->broken_symrefs)
+			string_list_append(data->broken_symrefs, path);
+		else
+			warning("ref is broken: %s", path);
+		return 0;
+	}
+
+	object = parse_object_or_die(oid->hash, path);
+	add_pending_object(data->revs, object, "");
 
 	return 0;
 }
@@ -163,7 +168,7 @@ int add_unseen_recent_objects_to_traversal(struct rev_info *revs,
 void mark_reachable_objects(struct rev_info *revs, int mark_reflog,
 			    unsigned long mark_recent,
 			    struct progress *progress,
-			    struct string_list *broken_refs)
+			    struct string_list *broken_symrefs)
 {
 	struct connectivity_progress cp;
 	struct add_one_data data;
@@ -180,7 +185,7 @@ void mark_reachable_objects(struct rev_info *revs, int mark_reflog,
 	add_index_objects_to_pending(revs, 0);
 
 	data.revs = revs;
-	data.broken_refs = broken_refs;
+	data.broken_symrefs = broken_symrefs;
 
 	/* Add all external refs */
 	for_each_ref(add_one_ref, &data);
diff --git a/reachable.h b/reachable.h
index 39de1c7..06f1400 100644
--- a/reachable.h
+++ b/reachable.h
@@ -6,6 +6,6 @@ extern int add_unseen_recent_objects_to_traversal(struct rev_info *revs,
 						  unsigned long timestamp);
 extern void mark_reachable_objects(struct rev_info *revs, int mark_reflog,
 				   unsigned long mark_recent, struct progress *,
-				   struct string_list *broken_refs);
+				   struct string_list *broken_symrefs);
 
 #endif
-- 
2.5.3.windows.1.3.gc322723
Johannes Schindelin· Sep 28, 2015, 14:01 UTC · re: Johannes Schindelin · lore

[PATCH v2 1/4] gc: demonstrate failure with stale remote HEAD

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
 t/t6500-gc.sh | 15 +++++++++++++++
 1 file changed, 15 insertions(+)
Show changes to t/t6500-gc.sh +15 −0
diff --git a/t/t6500-gc.sh b/t/t6500-gc.sh
index 63194d8..b736774 100755
--- a/t/t6500-gc.sh
+++ b/t/t6500-gc.sh
@@ -30,4 +30,19 @@ test_expect_success 'gc -h with invalid configuration' '
 	test_i18ngrep "[Uu]sage" broken/usage
 '
 
+test_expect_failure 'gc removes broken refs/remotes/<name>/HEAD' '
+	git init remote &&
+	(
+		cd remote &&
+		test_commit initial &&
+		git clone . ../client &&
+		git branch -m develop &&
+		cd ../client &&
+		git fetch --prune &&
+		git gc &&
+		git branch --list -r origin/HEAD >actual &&
+		test_line_count = 0 actual
+	)
+'
+
 test_done
-- 
2.5.3.windows.1.3.gc322723
Johannes Schindelin· Sep 28, 2015, 14:01 UTC · re: Johannes Schindelin · lore

[PATCH v2 2/4] pack-objects: do not get distracted by broken symrefs

It is quite possible for, say, a remote HEAD to become broken, e.g. when the default branch was renamed.

We should still be able to pack our objects when such a thing happens; simply ignore broken symrefs (because they cannot matter for the packing process anyway).

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
 reachable.c | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)
Show changes to reachable.c +7 −1
diff --git a/reachable.c b/reachable.c
index 9cff25b..64289f3 100644
--- a/reachable.c
+++ b/reachable.c
@@ -25,9 +25,15 @@ static void update_progress(struct connectivity_progress *cp)
 static int add_one_ref(const char *path, const struct object_id *oid,
 		       int flag, void *cb_data)
 {
-	struct object *object = parse_object_or_die(oid->hash, path);
 	struct rev_info *revs = (struct rev_info *)cb_data;
+	struct object *object;
 
+	if ((flag & REF_ISSYMREF) && (flag & REF_ISBROKEN)) {
+		warning("ref is broken: %s", path);
+		return 0;
+	}
+
+	object = parse_object_or_die(oid->hash, path);
 	add_pending_object(revs, object, "");
 
 	return 0;
-- 
2.5.3.windows.1.3.gc322723
Johannes Schindelin· Sep 28, 2015, 14:01 UTC · re: Johannes Schindelin · lore

[PATCH v2 3/4] mark_reachable_objects(): optionally collect broken symrefs

The behavior of `mark_reachable_objects()` without this patch is that it warns if it encounters a broken symref. Sometimes the caller wants to act upon broken symrefs, e.g. when garbage collecting in a repository with a broken remote HEAD.

So let's introduce an optional parameter to collect broken symrefs. The behavior of the function is unchanged if that parameter is `NULL`.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
 builtin/prune.c  |  2 +-
 builtin/reflog.c |  2 +-
 reachable.c      | 25 +++++++++++++++++++------
 reachable.h      |  3 ++-
 4 files changed, 23 insertions(+), 9 deletions(-)
Show changes to 4 files +23 −9

builtin/prune.c, builtin/reflog.c, reachable.c, reachable.h

diff --git a/builtin/prune.c b/builtin/prune.c
index 10b03d3..d6f664f 100644
--- a/builtin/prune.c
+++ b/builtin/prune.c
@@ -136,7 +136,7 @@ int cmd_prune(int argc, const char **argv, const char *prefix)
 	if (show_progress)
 		progress = start_progress_delay(_("Checking connectivity"), 0, 0, 2);
 
-	mark_reachable_objects(&revs, 1, expire, progress);
+	mark_reachable_objects(&revs, 1, expire, progress, NULL);
 	stop_progress(&progress);
 	for_each_loose_file_in_objdir(get_object_directory(), prune_object,
 				      prune_cruft, prune_subdir, NULL);
diff --git a/builtin/reflog.c b/builtin/reflog.c
index f96ca2a..cb8758a 100644
--- a/builtin/reflog.c
+++ b/builtin/reflog.c
@@ -583,7 +583,7 @@ static int cmd_reflog_expire(int argc, const char **argv, const char *prefix)
 		init_revisions(&cb.cmd.revs, prefix);
 		if (flags & EXPIRE_REFLOGS_VERBOSE)
 			printf("Marking reachable objects...");
-		mark_reachable_objects(&cb.cmd.revs, 0, 0, NULL);
+		mark_reachable_objects(&cb.cmd.revs, 0, 0, NULL, NULL);
 		if (flags & EXPIRE_REFLOGS_VERBOSE)
 			putchar('\n');
 	}
diff --git a/reachable.c b/reachable.c
index 64289f3..25c4932 100644
--- a/reachable.c
+++ b/reachable.c
@@ -15,6 +15,11 @@ struct connectivity_progress {
 	unsigned long count;
 };
 
+struct add_one_data {
+	struct rev_info *revs;
+	struct string_list *broken_symrefs;
+};
+
 static void update_progress(struct connectivity_progress *cp)
 {
 	cp->count++;
@@ -25,16 +30,19 @@ static void update_progress(struct connectivity_progress *cp)
 static int add_one_ref(const char *path, const struct object_id *oid,
 		       int flag, void *cb_data)
 {
-	struct rev_info *revs = (struct rev_info *)cb_data;
+	struct add_one_data *data = (struct add_one_data *)cb_data;
 	struct object *object;
 
 	if ((flag & REF_ISSYMREF) && (flag & REF_ISBROKEN)) {
-		warning("ref is broken: %s", path);
+		if (data->broken_symrefs)
+			string_list_append(data->broken_symrefs, path);
+		else
+			warning("ref is broken: %s", path);
 		return 0;
 	}
 
 	object = parse_object_or_die(oid->hash, path);
-	add_pending_object(revs, object, "");
+	add_pending_object(data->revs, object, "");
 
 	return 0;
 }
@@ -159,9 +167,11 @@ int add_unseen_recent_objects_to_traversal(struct rev_info *revs,
 
 void mark_reachable_objects(struct rev_info *revs, int mark_reflog,
 			    unsigned long mark_recent,
-			    struct progress *progress)
+			    struct progress *progress,
+			    struct string_list *broken_symrefs)
 {
 	struct connectivity_progress cp;
+	struct add_one_data data;
 
 	/*
 	 * Set up revision parsing, and mark us as being interested
@@ -174,11 +184,14 @@ void mark_reachable_objects(struct rev_info *revs, int mark_reflog,
 	/* Add all refs from the index file */
 	add_index_objects_to_pending(revs, 0);
 
+	data.revs = revs;
+	data.broken_symrefs = broken_symrefs;
+
 	/* Add all external refs */
-	for_each_ref(add_one_ref, revs);
+	for_each_ref(add_one_ref, &data);
 
 	/* detached HEAD is not included in the list above */
-	head_ref(add_one_ref, revs);
+	head_ref(add_one_ref, &data);
 
 	/* Add all reflog info */
 	if (mark_reflog)
diff --git a/reachable.h b/reachable.h
index d23efc3..06f1400 100644
--- a/reachable.h
+++ b/reachable.h
@@ -5,6 +5,7 @@ struct progress;
 extern int add_unseen_recent_objects_to_traversal(struct rev_info *revs,
 						  unsigned long timestamp);
 extern void mark_reachable_objects(struct rev_info *revs, int mark_reflog,
-				   unsigned long mark_recent, struct progress *);
+				   unsigned long mark_recent, struct progress *,
+				   struct string_list *broken_symrefs);
 
 #endif
-- 
2.5.3.windows.1.3.gc322723
Johannes Schindelin· Sep 28, 2015, 14:02 UTC · re: Johannes Schindelin · lore

[PATCH v2 4/4] gc: remove broken symrefs

When encountering broken symrefs, such as a stale remote HEAD (which can happen if the active branch was renamed in the remote), it is more helpful to remove those symrefs than to exit with an error.

This fixes https://github.com/git-for-windows/git/issues/423
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
 builtin/prune.c | 12 +++++++++++-
 t/t6500-gc.sh   |  2 +-
 2 files changed, 12 insertions(+), 2 deletions(-)
Show changes to 2 files +12 −2

builtin/prune.c, t/t6500-gc.sh

diff --git a/builtin/prune.c b/builtin/prune.c
index d6f664f..337b12a 100644
--- a/builtin/prune.c
+++ b/builtin/prune.c
@@ -6,6 +6,7 @@
 #include "reachable.h"
 #include "parse-options.h"
 #include "progress.h"
+#include "refs.h"
 
 static const char * const prune_usage[] = {
 	N_("git prune [-n] [-v] [--expire <time>] [--] [<head>...]"),
@@ -100,6 +101,7 @@ static void remove_temporary_files(const char *path)
 int cmd_prune(int argc, const char **argv, const char *prefix)
 {
 	struct rev_info revs;
+	struct string_list broken_symrefs = STRING_LIST_INIT_DUP;
 	struct progress *progress = NULL;
 	const struct option options[] = {
 		OPT__DRY_RUN(&show_only, N_("do not remove, show only")),
@@ -110,6 +112,7 @@ int cmd_prune(int argc, const char **argv, const char *prefix)
 		OPT_END()
 	};
 	char *s;
+	int i;
 
 	expire = ULONG_MAX;
 	save_commit_buffer = 0;
@@ -136,7 +139,14 @@ int cmd_prune(int argc, const char **argv, const char *prefix)
 	if (show_progress)
 		progress = start_progress_delay(_("Checking connectivity"), 0, 0, 2);
 
-	mark_reachable_objects(&revs, 1, expire, progress, NULL);
+	revs.ignore_missing = 1;
+	mark_reachable_objects(&revs, 1, expire, progress, &broken_symrefs);
+	for (i = 0; i < broken_symrefs.nr; i++) {
+		char *path = broken_symrefs.items[i].string;
+		printf("Removing stale ref %s\n", path);
+		if (!show_only && delete_ref(path, NULL, REF_NODEREF))
+			die("Could not remove stale ref %s", path);
+	}
 	stop_progress(&progress);
 	for_each_loose_file_in_objdir(get_object_directory(), prune_object,
 				      prune_cruft, prune_subdir, NULL);
diff --git a/t/t6500-gc.sh b/t/t6500-gc.sh
index b736774..0ae4271 100755
--- a/t/t6500-gc.sh
+++ b/t/t6500-gc.sh
@@ -30,7 +30,7 @@ test_expect_success 'gc -h with invalid configuration' '
 	test_i18ngrep "[Uu]sage" broken/usage
 '
 
-test_expect_failure 'gc removes broken refs/remotes/<name>/HEAD' '
+test_expect_success 'gc removes broken refs/remotes/<name>/HEAD' '
 	git init remote &&
 	(
 		cd remote &&
-- 
2.5.3.windows.1.3.gc322723
Junio C Hamano· Sep 28, 2015, 18:41 UTC · re: Johannes Schindelin · lore

Re: [PATCH v2 4/4] gc: remove broken symrefs

Johannes Schindelin <johannes.schindelin@gmx.de> writes:
> When encountering broken symrefs, such as a stale remote HEAD (which can
> happen if the active branch was renamed in the remote), it is more
> helpful to remove those symrefs than to exit with an error.

I think this depends on the perspective. One side of me says that a remote HEAD that points at refs/remotes/origin/topic that no longer exists is still giving me a valuable information and it should take a conscious action by the user to remove it, or evne better to repoint it to a more useful place. And from that point of view, removing is not all that helpful. Keeping them and not allowing them to exit with an error would be a real improvement.

On the other hand, I can certainly understand a view that considers that such a dangling symbolic ref is merely a cruft like any other cruft, and "gc" is all about removing cruft.

It just feels to me that this is a bit more valuable than other kinds of cruft, but maybe it is just me.

Show 66 quoted lines
>
> This fixes https://github.com/git-for-windows/git/issues/423
>
> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
> ---
>  builtin/prune.c | 12 +++++++++++-
>  t/t6500-gc.sh   |  2 +-
>  2 files changed, 12 insertions(+), 2 deletions(-)
>
> diff --git a/builtin/prune.c b/builtin/prune.c
> index d6f664f..337b12a 100644
> --- a/builtin/prune.c
> +++ b/builtin/prune.c
> @@ -6,6 +6,7 @@
>  #include "reachable.h"
>  #include "parse-options.h"
>  #include "progress.h"
> +#include "refs.h"
>  
>  static const char * const prune_usage[] = {
>  	N_("git prune [-n] [-v] [--expire <time>] [--] [<head>...]"),
> @@ -100,6 +101,7 @@ static void remove_temporary_files(const char *path)
>  int cmd_prune(int argc, const char **argv, const char *prefix)
>  {
>  	struct rev_info revs;
> +	struct string_list broken_symrefs = STRING_LIST_INIT_DUP;
>  	struct progress *progress = NULL;
>  	const struct option options[] = {
>  		OPT__DRY_RUN(&show_only, N_("do not remove, show only")),
> @@ -110,6 +112,7 @@ int cmd_prune(int argc, const char **argv, const char *prefix)
>  		OPT_END()
>  	};
>  	char *s;
> +	int i;
>  
>  	expire = ULONG_MAX;
>  	save_commit_buffer = 0;
> @@ -136,7 +139,14 @@ int cmd_prune(int argc, const char **argv, const char *prefix)
>  	if (show_progress)
>  		progress = start_progress_delay(_("Checking connectivity"), 0, 0, 2);
>  
> -	mark_reachable_objects(&revs, 1, expire, progress, NULL);
> +	revs.ignore_missing = 1;
> +	mark_reachable_objects(&revs, 1, expire, progress, &broken_symrefs);
> +	for (i = 0; i < broken_symrefs.nr; i++) {
> +		char *path = broken_symrefs.items[i].string;
> +		printf("Removing stale ref %s\n", path);
> +		if (!show_only && delete_ref(path, NULL, REF_NODEREF))
> +			die("Could not remove stale ref %s", path);
> +	}
>  	stop_progress(&progress);
>  	for_each_loose_file_in_objdir(get_object_directory(), prune_object,
>  				      prune_cruft, prune_subdir, NULL);
> diff --git a/t/t6500-gc.sh b/t/t6500-gc.sh
> index b736774..0ae4271 100755
> --- a/t/t6500-gc.sh
> +++ b/t/t6500-gc.sh
> @@ -30,7 +30,7 @@ test_expect_success 'gc -h with invalid configuration' '
>  	test_i18ngrep "[Uu]sage" broken/usage
>  '
>  
> -test_expect_failure 'gc removes broken refs/remotes/<name>/HEAD' '
> +test_expect_success 'gc removes broken refs/remotes/<name>/HEAD' '
>  	git init remote &&
>  	(
>  		cd remote &&
Junio C Hamano· Sep 28, 2015, 18:49 UTC · re: Junio C Hamano · lore

Re: [PATCH v2 4/4] gc: remove broken symrefs

Junio C Hamano <gitster@pobox.com> writes:
Show 20 quoted lines
> Johannes Schindelin <johannes.schindelin@gmx.de> writes:
>
>> When encountering broken symrefs, such as a stale remote HEAD (which can
>> happen if the active branch was renamed in the remote), it is more
>> helpful to remove those symrefs than to exit with an error.
>
> I think this depends on the perspective.  One side of me says that a
> remote HEAD that points at refs/remotes/origin/topic that no longer
> exists is still giving me a valuable information and it should take
> a conscious action by the user to remove it, or evne better to
> repoint it to a more useful place.  And from that point of view,
> removing is not all that helpful.  Keeping them and not allowing
> them to exit with an error would be a real improvement.
>
> On the other hand, I can certainly understand a view that considers
> that such a dangling symbolic ref is merely a cruft like any other
> cruft, and "gc" is all about removing cruft.
>
> It just feels to me that this is a bit more valuable than other
> kinds of cruft, but maybe it is just me.

Sorry, it is a bad habit of me to send out without concluding remark, leaving the recipient hanging without knowing what the next step should be.

I meant to say that I plan to, and I indeed did, queue these 4 without changes. I am not opposed to the removal so strongly to reject [4/4].

The above comment was that I just do not know if this is the right thing to do, or it will be hurting users.

Thanks.
Show 67 quoted lines
>
>>
>> This fixes https://github.com/git-for-windows/git/issues/423
>>
>> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
>> ---
>>  builtin/prune.c | 12 +++++++++++-
>>  t/t6500-gc.sh   |  2 +-
>>  2 files changed, 12 insertions(+), 2 deletions(-)
>>
>> diff --git a/builtin/prune.c b/builtin/prune.c
>> index d6f664f..337b12a 100644
>> --- a/builtin/prune.c
>> +++ b/builtin/prune.c
>> @@ -6,6 +6,7 @@
>>  #include "reachable.h"
>>  #include "parse-options.h"
>>  #include "progress.h"
>> +#include "refs.h"
>>  
>>  static const char * const prune_usage[] = {
>>  	N_("git prune [-n] [-v] [--expire <time>] [--] [<head>...]"),
>> @@ -100,6 +101,7 @@ static void remove_temporary_files(const char *path)
>>  int cmd_prune(int argc, const char **argv, const char *prefix)
>>  {
>>  	struct rev_info revs;
>> +	struct string_list broken_symrefs = STRING_LIST_INIT_DUP;
>>  	struct progress *progress = NULL;
>>  	const struct option options[] = {
>>  		OPT__DRY_RUN(&show_only, N_("do not remove, show only")),
>> @@ -110,6 +112,7 @@ int cmd_prune(int argc, const char **argv, const char *prefix)
>>  		OPT_END()
>>  	};
>>  	char *s;
>> +	int i;
>>  
>>  	expire = ULONG_MAX;
>>  	save_commit_buffer = 0;
>> @@ -136,7 +139,14 @@ int cmd_prune(int argc, const char **argv, const char *prefix)
>>  	if (show_progress)
>>  		progress = start_progress_delay(_("Checking connectivity"), 0, 0, 2);
>>  
>> -	mark_reachable_objects(&revs, 1, expire, progress, NULL);
>> +	revs.ignore_missing = 1;
>> +	mark_reachable_objects(&revs, 1, expire, progress, &broken_symrefs);
>> +	for (i = 0; i < broken_symrefs.nr; i++) {
>> +		char *path = broken_symrefs.items[i].string;
>> +		printf("Removing stale ref %s\n", path);
>> +		if (!show_only && delete_ref(path, NULL, REF_NODEREF))
>> +			die("Could not remove stale ref %s", path);
>> +	}
>>  	stop_progress(&progress);
>>  	for_each_loose_file_in_objdir(get_object_directory(), prune_object,
>>  				      prune_cruft, prune_subdir, NULL);
>> diff --git a/t/t6500-gc.sh b/t/t6500-gc.sh
>> index b736774..0ae4271 100755
>> --- a/t/t6500-gc.sh
>> +++ b/t/t6500-gc.sh
>> @@ -30,7 +30,7 @@ test_expect_success 'gc -h with invalid configuration' '
>>  	test_i18ngrep "[Uu]sage" broken/usage
>>  '
>>  
>> -test_expect_failure 'gc removes broken refs/remotes/<name>/HEAD' '
>> +test_expect_success 'gc removes broken refs/remotes/<name>/HEAD' '
>>  	git init remote &&
>>  	(
>>  		cd remote &&
Johannes Schindelin· Sep 28, 2015, 19:58 UTC · re: Junio C Hamano · lore

Re: [PATCH v2 4/4] gc: remove broken symrefs

Hi Junio,
On 2015-09-28 20:49, Junio C Hamano wrote:
Show 33 quoted lines
> Junio C Hamano <gitster@pobox.com> writes:
> 
>> Johannes Schindelin <johannes.schindelin@gmx.de> writes:
>>
>>> When encountering broken symrefs, such as a stale remote HEAD (which can
>>> happen if the active branch was renamed in the remote), it is more
>>> helpful to remove those symrefs than to exit with an error.
>>
>> I think this depends on the perspective.  One side of me says that a
>> remote HEAD that points at refs/remotes/origin/topic that no longer
>> exists is still giving me a valuable information and it should take
>> a conscious action by the user to remove it, or evne better to
>> repoint it to a more useful place.  And from that point of view,
>> removing is not all that helpful.  Keeping them and not allowing
>> them to exit with an error would be a real improvement.
>>
>> On the other hand, I can certainly understand a view that considers
>> that such a dangling symbolic ref is merely a cruft like any other
>> cruft, and "gc" is all about removing cruft.
>>
>> It just feels to me that this is a bit more valuable than other
>> kinds of cruft, but maybe it is just me.
> 
> Sorry, it is a bad habit of me to send out without concluding
> remark, leaving the recipient hanging without knowing what the next
> step should be.
> 
> I meant to say that I plan to, and I indeed did, queue these 4
> without changes.  I am not opposed to the removal so strongly to
> reject [4/4].
> 
> The above comment was that I just do not know if this is the right
> thing to do, or it will be hurting users.
Oh, I appreciate your feedback. I am actually not all *that* certain that removing the broken symref is the correct thing. It is this sort of fruitful exchange that allows me to throw out an idea and be relatively certain that something better will come out of v3 or v8 of the patch series than what I had in mind.
To be honest, the most important outcome is probably 2/4 -- which should be enough to fix the issue reported by the Git for Windows user. I could adjust the test so that it no longer insists that `origin/HEAD` be deleted, but still requires that `git gc` succeeds.
I would have no problem to let this sit for a couple of days until the final verdict.

Ciao, Dscho

Junio C Hamano· Oct 5, 2015, 22:06 UTC · re: Johannes Schindelin · lore

Re: [PATCH v2 4/4] gc: remove broken symrefs

Johannes Schindelin <johannes.schindelin@gmx.de> writes:
Show 13 quoted lines
> Oh, I appreciate your feedback. I am actually not all *that* certain
> that removing the broken symref is the correct thing. It is this sort
> of fruitful exchange that allows me to throw out an idea and be
> relatively certain that something better will come out of v3 or v8 of
> the patch series than what I had in mind.
>
> To be honest, the most important outcome is probably 2/4 -- which
> should be enough to fix the issue reported by the Git for Windows
> user. I could adjust the test so that it no longer insists that
> origin/HEAD` be deleted, but still requires that `git gc` succeeds.
>
> I would have no problem to let this sit for a couple of days until
> the final verdict.

... and a few days have passed. I am tempted to do the easy bits first, discarding the parts that both of us feel iffy for now without prejudice, keeping the first two commits with a bit of tweak.

The primary tweak is to t6500 in the first patch, which I retitled below (and the patch shows s/failure/success/ but in the first step that only adds a failing test, it would of course expect a failure).

And with "pack-objects: do not get distracted", the test will start passing.

Show changes to 2 files +3 −5

t/t6500-gc.sh, reachable.c

diff --git a/t/t6500-gc.sh b/t/t6500-gc.sh
index b736774..5d7d414 100755
--- a/t/t6500-gc.sh
+++ b/t/t6500-gc.sh
@@ -30,7 +30,7 @@ test_expect_success 'gc -h with invalid configuration' '
 	test_i18ngrep "[Uu]sage" broken/usage
 '
 
-test_expect_failure 'gc removes broken refs/remotes/<name>/HEAD' '
+test_expect_success 'gc is not aborted due to a stale symref' '
 	git init remote &&
 	(
 		cd remote &&
@@ -39,9 +39,7 @@ test_expect_failure 'gc removes broken refs/remotes/<name>/HEAD' '
 		git branch -m develop &&
 		cd ../client &&
 		git fetch --prune &&
-		git gc &&
-		git branch --list -r origin/HEAD >actual &&
-		test_line_count = 0 actual
+		git gc
 	)
 '
 
diff --git a/reachable.c b/reachable.c
index 6356ae8..4cfd0de 100644
--- a/reachable.c
+++ b/reachable.c
@@ -28,7 +28,7 @@ static int add_one_ref(const char *path, const unsigned char *sha1, int flag, vo
 	struct object *object;
 
 	if ((flag & REF_ISSYMREF) && (flag & REF_ISBROKEN)) {
-		warning("ref is broken: %s", path);
+		warning("symbolic ref is dangling: %s", path);
 		return 0;
 	}
 
Jeff King· Sep 28, 2015, 19:03 UTC · re: Johannes Schindelin · lore

Re: [PATCH v2 4/4] gc: remove broken symrefs

On Mon, Sep 28, 2015 at 04:02:08PM +0200, Johannes Schindelin wrote:
Show 7 quoted lines
> @@ -136,7 +139,14 @@ int cmd_prune(int argc, const char **argv, const char *prefix)
>  	if (show_progress)
>  		progress = start_progress_delay(_("Checking connectivity"), 0, 0, 2);
>  
> -	mark_reachable_objects(&revs, 1, expire, progress, NULL);
> +	revs.ignore_missing = 1;
> +	mark_reachable_objects(&revs, 1, expire, progress, &broken_symrefs);
You should not need this ignore_missing anymore, right?

It is the dangerous thing I mentioned earlier, but I am puzzled why it does not cause t5312 to fail.

-Peff
Johannes Schindelin· Sep 28, 2015, 20:05 UTC · re: Jeff King · lore

Re: [PATCH v2 4/4] gc: remove broken symrefs

Hi Peff,
On 2015-09-28 21:03, Jeff King wrote:
Show 14 quoted lines
> On Mon, Sep 28, 2015 at 04:02:08PM +0200, Johannes Schindelin wrote:
> 
>> @@ -136,7 +139,14 @@ int cmd_prune(int argc, const char **argv, const char *prefix)
>>  	if (show_progress)
>>  		progress = start_progress_delay(_("Checking connectivity"), 0, 0, 2);
>>
>> -	mark_reachable_objects(&revs, 1, expire, progress, NULL);
>> +	revs.ignore_missing = 1;
>> +	mark_reachable_objects(&revs, 1, expire, progress, &broken_symrefs);
> 
> You should not need this ignore_missing anymore, right?
> 
> It is the dangerous thing I mentioned earlier, but I am puzzled why it
> does not cause t5312 to fail.
Gaaaaaah! I edited this out at least twice, yet it creeps back in. Bah!
Junio, would you mind fixing this up locally? If you want me to submit v3, just let me know, I will take care of it tomorrow.

Ciao, Dscho

Johannes Schindelin· Oct 6, 2015, 13:57 UTC · re: Junio C Hamano · lore

[PATCH v3 0/2] Fix gc failure when a remote HEAD goes stale

There has been a report in the Git for Windows project that gc fails sometimes: https://github.com/git-for-windows/git/issues/423

It turns out that there are cases when a remote HEAD can go stale and it is not the user's fault at all. It can happen, for example, if the active branch in the remote repository gets renamed.

Thanks to Junio and Peff for their really valuable sanity check.

Interdiff re: v2 after the diffstat (3/4 and 4/4 were dropped, I am no longer removing the broken symrefs).

Johannes Schindelin (2):
  gc: demonstrate failure with stale remote HEAD
  pack-objects: do not get distracted by broken symrefs
 reachable.c   |  8 +++++++-
 t/t6500-gc.sh | 13 +++++++++++++
 2 files changed, 20 insertions(+), 1 deletion(-)
Show changes to 5 files +11 −37

builtin/prune.c, builtin/reflog.c, reachable.c, reachable.h, t/t6500-gc.sh

diff --git a/builtin/prune.c b/builtin/prune.c
index 337b12a..10b03d3 100644
--- a/builtin/prune.c
+++ b/builtin/prune.c
@@ -6,7 +6,6 @@
 #include "reachable.h"
 #include "parse-options.h"
 #include "progress.h"
-#include "refs.h"
 
 static const char * const prune_usage[] = {
 	N_("git prune [-n] [-v] [--expire <time>] [--] [<head>...]"),
@@ -101,7 +100,6 @@ static void remove_temporary_files(const char *path)
 int cmd_prune(int argc, const char **argv, const char *prefix)
 {
 	struct rev_info revs;
-	struct string_list broken_symrefs = STRING_LIST_INIT_DUP;
 	struct progress *progress = NULL;
 	const struct option options[] = {
 		OPT__DRY_RUN(&show_only, N_("do not remove, show only")),
@@ -112,7 +110,6 @@ int cmd_prune(int argc, const char **argv, const char *prefix)
 		OPT_END()
 	};
 	char *s;
-	int i;
 
 	expire = ULONG_MAX;
 	save_commit_buffer = 0;
@@ -139,14 +136,7 @@ int cmd_prune(int argc, const char **argv, const char *prefix)
 	if (show_progress)
 		progress = start_progress_delay(_("Checking connectivity"), 0, 0, 2);
 
-	revs.ignore_missing = 1;
-	mark_reachable_objects(&revs, 1, expire, progress, &broken_symrefs);
-	for (i = 0; i < broken_symrefs.nr; i++) {
-		char *path = broken_symrefs.items[i].string;
-		printf("Removing stale ref %s\n", path);
-		if (!show_only && delete_ref(path, NULL, REF_NODEREF))
-			die("Could not remove stale ref %s", path);
-	}
+	mark_reachable_objects(&revs, 1, expire, progress);
 	stop_progress(&progress);
 	for_each_loose_file_in_objdir(get_object_directory(), prune_object,
 				      prune_cruft, prune_subdir, NULL);
diff --git a/builtin/reflog.c b/builtin/reflog.c
index cb8758a..f96ca2a 100644
--- a/builtin/reflog.c
+++ b/builtin/reflog.c
@@ -583,7 +583,7 @@ static int cmd_reflog_expire(int argc, const char **argv, const char *prefix)
 		init_revisions(&cb.cmd.revs, prefix);
 		if (flags & EXPIRE_REFLOGS_VERBOSE)
 			printf("Marking reachable objects...");
-		mark_reachable_objects(&cb.cmd.revs, 0, 0, NULL, NULL);
+		mark_reachable_objects(&cb.cmd.revs, 0, 0, NULL);
 		if (flags & EXPIRE_REFLOGS_VERBOSE)
 			putchar('\n');
 	}
diff --git a/reachable.c b/reachable.c
index 25c4932..43616d4 100644
--- a/reachable.c
+++ b/reachable.c
@@ -15,11 +15,6 @@ struct connectivity_progress {
 	unsigned long count;
 };
 
-struct add_one_data {
-	struct rev_info *revs;
-	struct string_list *broken_symrefs;
-};
-
 static void update_progress(struct connectivity_progress *cp)
 {
 	cp->count++;
@@ -30,19 +25,16 @@ static void update_progress(struct connectivity_progress *cp)
 static int add_one_ref(const char *path, const struct object_id *oid,
 		       int flag, void *cb_data)
 {
-	struct add_one_data *data = (struct add_one_data *)cb_data;
+	struct rev_info *revs = (struct rev_info *)cb_data;
 	struct object *object;
 
 	if ((flag & REF_ISSYMREF) && (flag & REF_ISBROKEN)) {
-		if (data->broken_symrefs)
-			string_list_append(data->broken_symrefs, path);
-		else
-			warning("ref is broken: %s", path);
+		warning("symbolic ref is dangling: %s", path);
 		return 0;
 	}
 
 	object = parse_object_or_die(oid->hash, path);
-	add_pending_object(data->revs, object, "");
+	add_pending_object(revs, object, "");
 
 	return 0;
 }
@@ -167,11 +159,9 @@ int add_unseen_recent_objects_to_traversal(struct rev_info *revs,
 
 void mark_reachable_objects(struct rev_info *revs, int mark_reflog,
 			    unsigned long mark_recent,
-			    struct progress *progress,
-			    struct string_list *broken_symrefs)
+			    struct progress *progress)
 {
 	struct connectivity_progress cp;
-	struct add_one_data data;
 
 	/*
 	 * Set up revision parsing, and mark us as being interested
@@ -184,14 +174,11 @@ void mark_reachable_objects(struct rev_info *revs, int mark_reflog,
 	/* Add all refs from the index file */
 	add_index_objects_to_pending(revs, 0);
 
-	data.revs = revs;
-	data.broken_symrefs = broken_symrefs;
-
 	/* Add all external refs */
-	for_each_ref(add_one_ref, &data);
+	for_each_ref(add_one_ref, revs);
 
 	/* detached HEAD is not included in the list above */
-	head_ref(add_one_ref, &data);
+	head_ref(add_one_ref, revs);
 
 	/* Add all reflog info */
 	if (mark_reflog)
diff --git a/reachable.h b/reachable.h
index 06f1400..d23efc3 100644
--- a/reachable.h
+++ b/reachable.h
@@ -5,7 +5,6 @@ struct progress;
 extern int add_unseen_recent_objects_to_traversal(struct rev_info *revs,
 						  unsigned long timestamp);
 extern void mark_reachable_objects(struct rev_info *revs, int mark_reflog,
-				   unsigned long mark_recent, struct progress *,
-				   struct string_list *broken_symrefs);
+				   unsigned long mark_recent, struct progress *);
 
 #endif
diff --git a/t/t6500-gc.sh b/t/t6500-gc.sh
index 0ae4271..5d7d414 100755
--- a/t/t6500-gc.sh
+++ b/t/t6500-gc.sh
@@ -30,7 +30,7 @@ test_expect_success 'gc -h with invalid configuration' '
 	test_i18ngrep "[Uu]sage" broken/usage
 '
 
-test_expect_success 'gc removes broken refs/remotes/<name>/HEAD' '
+test_expect_success 'gc is not aborted due to a stale symref' '
 	git init remote &&
 	(
 		cd remote &&
@@ -39,9 +39,7 @@ test_expect_success 'gc removes broken refs/remotes/<name>/HEAD' '
 		git branch -m develop &&
 		cd ../client &&
 		git fetch --prune &&
-		git gc &&
-		git branch --list -r origin/HEAD >actual &&
-		test_line_count = 0 actual
+		git gc
 	)
 '
 
-- 
2.6.1.windows.1
Johannes Schindelin· Oct 6, 2015, 13:58 UTC · re: Johannes Schindelin · lore

[PATCH v3 1/2] gc: demonstrate failure with stale remote HEAD

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
 t/t6500-gc.sh | 13 +++++++++++++
 1 file changed, 13 insertions(+)
Show changes to t/t6500-gc.sh +13 −0
diff --git a/t/t6500-gc.sh b/t/t6500-gc.sh
index 63194d8..9a3a285 100755
--- a/t/t6500-gc.sh
+++ b/t/t6500-gc.sh
@@ -30,4 +30,17 @@ test_expect_success 'gc -h with invalid configuration' '
 	test_i18ngrep "[Uu]sage" broken/usage
 '
 
+test_expect_failure 'gc removes broken refs/remotes/<name>/HEAD' '
+	git init remote &&
+	(
+		cd remote &&
+		test_commit initial &&
+		git clone . ../client &&
+		git branch -m develop &&
+		cd ../client &&
+		git fetch --prune &&
+		git gc
+	)
+'
+
 test_done
-- 
2.6.1.windows.1
Johannes Schindelin· Oct 6, 2015, 13:59 UTC · re: Johannes Schindelin · lore

[PATCH v3 2/2] pack-objects: do not get distracted by broken symrefs

It is quite possible for, say, a remote HEAD to become broken, e.g. when the default branch was renamed.

We should still be able to pack our objects when such a thing happens; simply ignore broken symrefs (because they cannot matter for the packing process anyway).

This fixes https://github.com/git-for-windows/git/issues/423
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
---
 reachable.c   | 8 +++++++-
 t/t6500-gc.sh | 2 +-
 2 files changed, 8 insertions(+), 2 deletions(-)
Show changes to 2 files +8 −2

reachable.c, t/t6500-gc.sh

diff --git a/reachable.c b/reachable.c
index 9cff25b..43616d4 100644
--- a/reachable.c
+++ b/reachable.c
@@ -25,9 +25,15 @@ static void update_progress(struct connectivity_progress *cp)
 static int add_one_ref(const char *path, const struct object_id *oid,
 		       int flag, void *cb_data)
 {
-	struct object *object = parse_object_or_die(oid->hash, path);
 	struct rev_info *revs = (struct rev_info *)cb_data;
+	struct object *object;
 
+	if ((flag & REF_ISSYMREF) && (flag & REF_ISBROKEN)) {
+		warning("symbolic ref is dangling: %s", path);
+		return 0;
+	}
+
+	object = parse_object_or_die(oid->hash, path);
 	add_pending_object(revs, object, "");
 
 	return 0;
diff --git a/t/t6500-gc.sh b/t/t6500-gc.sh
index 9a3a285..5d7d414 100755
--- a/t/t6500-gc.sh
+++ b/t/t6500-gc.sh
@@ -30,7 +30,7 @@ test_expect_success 'gc -h with invalid configuration' '
 	test_i18ngrep "[Uu]sage" broken/usage
 '
 
-test_expect_failure 'gc removes broken refs/remotes/<name>/HEAD' '
+test_expect_success 'gc is not aborted due to a stale symref' '
 	git init remote &&
 	(
 		cd remote &&
-- 
2.6.1.windows.1
Junio C Hamano· Oct 7, 2015, 17:45 UTC · re: Johannes Schindelin · lore

Re: [PATCH v3 2/2] pack-objects: do not get distracted by broken symrefs

Johannes Schindelin <johannes.schindelin@gmx.de> writes:
Show 11 quoted lines
> It is quite possible for, say, a remote HEAD to become broken, e.g.
> when the default branch was renamed.
>
> We should still be able to pack our objects when such a thing happens;
> simply ignore broken symrefs (because they cannot matter for the packing
> process anyway).
>
> This fixes https://github.com/git-for-windows/git/issues/423
>
> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
> ---

It seems that the result of applying these two patches and log messages of them are the same with what I queued on 'pu', except that the first of these two patches adds a test with a wrong name and then this one does "oops, that was misnamed". So I'll keep what is already queued.

Thanks.
Show 38 quoted lines
>  reachable.c   | 8 +++++++-
>  t/t6500-gc.sh | 2 +-
>  2 files changed, 8 insertions(+), 2 deletions(-)
>
> diff --git a/reachable.c b/reachable.c
> index 9cff25b..43616d4 100644
> --- a/reachable.c
> +++ b/reachable.c
> @@ -25,9 +25,15 @@ static void update_progress(struct connectivity_progress *cp)
>  static int add_one_ref(const char *path, const struct object_id *oid,
>  		       int flag, void *cb_data)
>  {
> -	struct object *object = parse_object_or_die(oid->hash, path);
>  	struct rev_info *revs = (struct rev_info *)cb_data;
> +	struct object *object;
>  
> +	if ((flag & REF_ISSYMREF) && (flag & REF_ISBROKEN)) {
> +		warning("symbolic ref is dangling: %s", path);
> +		return 0;
> +	}
> +
> +	object = parse_object_or_die(oid->hash, path);
>  	add_pending_object(revs, object, "");
>  
>  	return 0;
> diff --git a/t/t6500-gc.sh b/t/t6500-gc.sh
> index 9a3a285..5d7d414 100755
> --- a/t/t6500-gc.sh
> +++ b/t/t6500-gc.sh
> @@ -30,7 +30,7 @@ test_expect_success 'gc -h with invalid configuration' '
>  	test_i18ngrep "[Uu]sage" broken/usage
>  '
>  
> -test_expect_failure 'gc removes broken refs/remotes/<name>/HEAD' '
> +test_expect_success 'gc is not aborted due to a stale symref' '
>  	git init remote &&
>  	(
>  		cd remote &&
Johannes Schindelin· Oct 8, 2015, 19:15 UTC · re: Junio C Hamano · lore

Re: [PATCH v3 2/2] pack-objects: do not get distracted by broken symrefs

Hi Junio,
On 2015-10-07 19:45, Junio C Hamano wrote:
Show 19 quoted lines
> Johannes Schindelin <johannes.schindelin@gmx.de> writes:
> 
>> It is quite possible for, say, a remote HEAD to become broken, e.g.
>> when the default branch was renamed.
>>
>> We should still be able to pack our objects when such a thing happens;
>> simply ignore broken symrefs (because they cannot matter for the packing
>> process anyway).
>>
>> This fixes https://github.com/git-for-windows/git/issues/423
>>
>> Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
>> ---
> 
> It seems that the result of applying these two patches and log
> messages of them are the same with what I queued on 'pu', except
> that the first of these two patches adds a test with a wrong name
> and then this one does "oops, that was misnamed".  So I'll keep what
> is already queued.
Sorry for fixing up the wrong commit. I honestly meant to fix up the first one. And thank you for fixing it up in `pu` already; I should have known better and check first whether you fixed it.
However, there was one more change I made: I wanted to have that link to https://github.com/git-for-windows/git/issues/423 in the commit message to better link the original report with the commit.
Would yo kindly add the line
    This fixes https://github.com/git-for-windows/git/issues/423
before the Signed-off-by lines?

Thanks, Dscho

Junio C Hamano· Oct 8, 2015, 19:42 UTC · re: Johannes Schindelin · lore

Re: [PATCH v3 2/2] pack-objects: do not get distracted by broken symrefs

Johannes Schindelin <johannes.schindelin@gmx.de> writes:
Show 5 quoted lines
> Would yo kindly add the line
>
>     This fixes https://github.com/git-for-windows/git/issues/423
>
> before the Signed-off-by lines?
Oh, sorry, I missed that one.  Fixed.
Thanks.
Johannes Schindelin· Oct 8, 2015, 20:10 UTC · re: Junio C Hamano · lore

Re: [PATCH v3 2/2] pack-objects: do not get distracted by broken symrefs

Hi Junio,
On 2015-10-08 21:42, Junio C Hamano wrote:
Show 11 quoted lines
> Johannes Schindelin <johannes.schindelin@gmx.de> writes:
> 
>> Would yo kindly add the line
>>
>>     This fixes https://github.com/git-for-windows/git/issues/423
>>
>> before the Signed-off-by lines?
> 
> Oh, sorry, I missed that one.  Fixed.
> 
> Thanks.

Thank you! Dscho

← back to recent threads