threads / patch / 57323

patchfetch --prune: exit with error if pruning fails

Subject: [PATCH] fetch --prune: exit with error if pruning fails

## tl;dr

11 messages between Jan 27, 2022 and Jan 31, 2022. Diffs are folded; open one to read it.

replies: 10people: 3as markdown or json

Thomas Gummerer· Jan 27, 2022, 15:37 UTC · lore

When pruning refs fails, we currently print an error to stderr, but still exit 0 from 'git fetch'. Since this is a genuine error fetch should be exiting with some non-zero exit code. Make it so.

The --prune option was introduced in f360d844de ("builtin-fetch: add --prune option", 2009-11-10). Unfortunately it's unclear from that commit whether ignoring the exit code was an oversight or intentional, but it feels like an oversight.

Helped-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Thomas Gummerer <t.gummerer@gmail.com>
---
 builtin/fetch.c  | 12 ++++++++----
 t/t5510-fetch.sh | 10 ++++++++++
 2 files changed, 18 insertions(+), 4 deletions(-)
Show changes to 2 files +18 −4

builtin/fetch.c, t/t5510-fetch.sh

diff --git a/builtin/fetch.c b/builtin/fetch.c
index 5f06b21f8e..54545efedd 100644
--- a/builtin/fetch.c
+++ b/builtin/fetch.c
@@ -1609,11 +1609,15 @@ static int do_fetch(struct transport *transport,
 		 * don't care whether --tags was specified.
 		 */
 		if (rs->nr) {
-			prune_refs(rs, ref_map, transport->url);
+			retcode = prune_refs(rs, ref_map, transport->url);
 		} else {
-			prune_refs(&transport->remote->fetch,
-				   ref_map,
-				   transport->url);
+			retcode = prune_refs(&transport->remote->fetch,
+					     ref_map,
+					     transport->url);
+		}
+		if (retcode) {
+			free_refs(ref_map);
+			goto cleanup;
 		}
 	}
 	if (fetch_and_consume_refs(transport, ref_map, worktrees)) {
diff --git a/t/t5510-fetch.sh b/t/t5510-fetch.sh
index 20f7110ec1..df824cc3d0 100755
--- a/t/t5510-fetch.sh
+++ b/t/t5510-fetch.sh
@@ -164,6 +164,16 @@ test_expect_success 'fetch --prune --tags with refspec prunes based on refspec'
 	git rev-parse sometag
 '
 
+test_expect_success REFFILES 'fetch --prune fails to delete branches' '
+	cd "$D" &&
+	git clone . prune-fail &&
+	cd prune-fail &&
+	git update-ref refs/remotes/origin/extrabranch main &&
+	>.git/packed-refs.new &&
+
+	test_must_fail git fetch --prune origin
+'
+
 test_expect_success 'fetch --atomic works with a single branch' '
 	test_when_finished "rm -rf \"$D\"/atomic" &&
 

-- 
2.31.1
Junio C Hamano· Jan 27, 2022, 19:57 UTC · re: Thomas Gummerer · lore

Re: [PATCH] fetch --prune: exit with error if pruning fails

Thomas Gummerer <t.gummerer@gmail.com> writes:
Show 8 quoted lines
> When pruning refs fails, we currently print an error to stderr, but
> still exit 0 from 'git fetch'.  Since this is a genuine error fetch
> should be exiting with some non-zero exit code.  Make it so.
>
> The --prune option was introduced in f360d844de ("builtin-fetch: add
> --prune option", 2009-11-10).  Unfortunately it's unclear from that
> commit whether ignoring the exit code was an oversight or
> intentional, but it feels like an oversight.

It is a good idea to signal, with a non-zero status, that the user needs to inspect the situation and possibly take a corrective action. I however do not know if it is sufficient to just give the same exit status as returned by prune_refs(), which may or may not happen to crash with other possible non-zero exit status to make it indistinguishable from other kinds of errors.

Show 11 quoted lines
>  		if (rs->nr) {
> -			prune_refs(rs, ref_map, transport->url);
> +			retcode = prune_refs(rs, ref_map, transport->url);
>  		} else {
> -			prune_refs(&transport->remote->fetch,
> -				   ref_map,
> -				   transport->url);
> +			retcode = prune_refs(&transport->remote->fetch,
> +					     ref_map,
> +					     transport->url);
> +		}

It seems that it is possible for do_fetch() to return a negative value, even without this patch. The return value from prune_refs() comes from refs.c::delete_refs(), which can come from error_errno() that is -1, so this patch adds more of the same problem to existing ones, though.

And the value propagates up to cmd_fetch() via fetch_one(). This will be relayed to exit() without changing via this callchain:

    git.c::cmd_main()
      git.c::run_argv()
        git.c::handle_builtin()
          git.c::run_builtin()

This is not a new problem, which does not want to be corrected as part of this patch, but let's leave a mental note as #leftoverbits for now.

> +		if (retcode) {
> +			free_refs(ref_map);
> +			goto cleanup;
>  		}

This part is iffy. We tried to prune refs, we may have removed some of the refs missing from the other side but we may still have some other refs that are missing from the other side due to the failure we noticed.

Is it sensible to abort the fetching? I am undecided, but without further input, my gut reaction is that it is safe and may even be better to treat this as a soft error and try to go closer to where the user wanted to go as much as possible by continuing to fetch from the other side, given that we have already paid for the cost of discovering the refs from the other side.

Show 15 quoted lines
> diff --git a/t/t5510-fetch.sh b/t/t5510-fetch.sh
> index 20f7110ec1..df824cc3d0 100755
> --- a/t/t5510-fetch.sh
> +++ b/t/t5510-fetch.sh
> @@ -164,6 +164,16 @@ test_expect_success 'fetch --prune --tags with refspec prunes based on refspec'
>  	git rev-parse sometag
>  '
>  
> +test_expect_success REFFILES 'fetch --prune fails to delete branches' '
> +	cd "$D" &&
> +	git clone . prune-fail &&
> +	cd prune-fail &&
> +	git update-ref refs/remotes/origin/extrabranch main &&
> +	>.git/packed-refs.new &&
> +	test_must_fail git fetch --prune origin

Is it because the lockfile ".new" on packed-refs prevents deletion of refs but does not block creation and updates to existing refs that it is an effective test for the "--prune" issue? If we somehow "locked" the whole ref updates, then the fetch would fail even without "--prune", so it may be "effective", but smells like knowing too much implementation detail. Yuck, but I do not offhand think of any better way (it is easier to think of worse ways), so without further input, I would say that this is the best (or "least bad") we can do.

Another tangent #leftoverbits is that many tests in this script are so old-style that chdir around without isolating themselves in subshells, while some do. We may want to clean them up when the file is quiescent.

Thanks.
Johannes Schindelin· Jan 28, 2022, 10:13 UTC · re: Junio C Hamano · lore

Re: [PATCH] fetch --prune: exit with error if pruning fails

Hi Junio,
On Thu, 27 Jan 2022, Junio C Hamano wrote:
Show 18 quoted lines
> Thomas Gummerer <t.gummerer@gmail.com> writes:
>
> > +		if (retcode) {
> > +			free_refs(ref_map);
> > +			goto cleanup;
> >  		}
>
> This part is iffy.  We tried to prune refs, we may have removed some
> of the refs missing from the other side but we may still have some
> other refs that are missing from the other side due to the failure
> we noticed.
>
> Is it sensible to abort the fetching?  I am undecided, but without
> further input, my gut reaction is that it is safe and may even be
> better to treat this as a soft error and try to go closer to where
> the user wanted to go as much as possible by continuing to fetch
> from the other side, given that we have already paid for the cost of
> discovering the refs from the other side.

I am not so sure. When pruning failed, there may very well be directories or files in the way of fetching the refs as desired. And it might be even worse if pruning failed _without_ the fetch failing afterwards: the user specifically asked for stale refs to be cleaned up, the command succeeded, but did not do what the user asked for.

Maybe Thomas has an even stronger argument in favor of erroring out. In any case, I don't think that `--prune` should be a "best effort, otherwise just shrug" option. If we wanted that, we could introduce `--prune-best-effort` or some such...

Ciao, Dscho

Thomas Gummerer· Jan 28, 2022, 10:55 UTC · re: Johannes Schindelin · lore

Re: [PATCH] fetch --prune: exit with error if pruning fails

Johannes Schindelin writes:
Show 22 quoted lines
> Hi Junio,
>
> On Thu, 27 Jan 2022, Junio C Hamano wrote:
>
>> Thomas Gummerer <t.gummerer@gmail.com> writes:
>>
>> > +		if (retcode) {
>> > +			free_refs(ref_map);
>> > +			goto cleanup;
>> >  		}
>>
>> This part is iffy.  We tried to prune refs, we may have removed some
>> of the refs missing from the other side but we may still have some
>> other refs that are missing from the other side due to the failure
>> we noticed.
>>
>> Is it sensible to abort the fetching?  I am undecided, but without
>> further input, my gut reaction is that it is safe and may even be
>> better to treat this as a soft error and try to go closer to where
>> the user wanted to go as much as possible by continuing to fetch
>> from the other side, given that we have already paid for the cost of
>> discovering the refs from the other side.
Show 5 quoted lines
> I am not so sure. When pruning failed, there may very well be directories
> or files in the way of fetching the refs as desired. And it might be even
> worse if pruning failed _without_ the fetch failing afterwards: the user
> specifically asked for stale refs to be cleaned up, the command succeeded,
> but did not do what the user asked for.

I was thinking along similar lines here. I was going back and forth between letting the fetch continue, and then exiting with a non-zero exit code, and just erroring out directly. I ended up with the latter because it felt like the right thing to do for the user. The command failed, so I'd think it's more confusing that it does more work after that (even if we already did part of that work).

But I don't care too much one way or another, as long as we end up with a non-zero exit code when the fetch fails.

> Maybe Thomas has an even stronger argument in favor of erroring out. In
> any case, I don't think that `--prune` should be a "best effort, otherwise
> just shrug" option. If we wanted that, we could introduce
> `--prune-best-effort` or some such...

I don't have a strong argument on whether we should error out immediately after failing to prune the refs, or just erroring out after doing the rest of the work, other than exiting early feels right to me.

From the comments upthread it seems to me that the argument is more whether we should continue after failing to prune, and exit with a non-zero exit code, or if we should just fail early.

Though I'm happy to introduce a '--prune-best-effort' option or something like that if we still want to preserve that option for users. I don't really see people wanting to use that though, and it should be very rare that '--prune' fails, but the rest of the fetch succeeds.

Johannes Schindelin· Jan 28, 2022, 12:36 UTC · re: Thomas Gummerer · lore

Re: [PATCH] fetch --prune: exit with error if pruning fails

Hi Thomas & Junio,
On Fri, 28 Jan 2022, Thomas Gummerer wrote:
Show 32 quoted lines
> Johannes Schindelin writes:
>
> > On Thu, 27 Jan 2022, Junio C Hamano wrote:
> >
> >> Thomas Gummerer <t.gummerer@gmail.com> writes:
> >>
> >> > +		if (retcode) {
> >> > +			free_refs(ref_map);
> >> > +			goto cleanup;
> >> >  		}
> >>
> >> This part is iffy.  We tried to prune refs, we may have removed some
> >> of the refs missing from the other side but we may still have some
> >> other refs that are missing from the other side due to the failure
> >> we noticed.
> >>
> >> Is it sensible to abort the fetching?  I am undecided, but without
> >> further input, my gut reaction is that it is safe and may even be
> >> better to treat this as a soft error and try to go closer to where
> >> the user wanted to go as much as possible by continuing to fetch
> >> from the other side, given that we have already paid for the cost of
> >> discovering the refs from the other side.
>
> > I am not so sure. When pruning failed, there may very well be directories
> > or files in the way of fetching the refs as desired. And it might be even
> > worse if pruning failed _without_ the fetch failing afterwards: the user
> > specifically asked for stale refs to be cleaned up, the command succeeded,
> > but did not do what the user asked for.
>
> I was thinking along similar lines here.  I was going back and forth
> between letting the fetch continue, and then exiting with a non-zero
> exit code, and just erroring out directly.

Oh, I think I misunderstood Junio. As long as the failed prune will cause a non-zero exit code, I am fine with continuing to try to fetch.

Ciao, Dscho

Junio C Hamano· Jan 28, 2022, 18:14 UTC · re: Johannes Schindelin · lore

Re: [PATCH] fetch --prune: exit with error if pruning fails

Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:
Show 10 quoted lines
> I am not so sure. When pruning failed, there may very well be directories
> or files in the way of fetching the refs as desired. And it might be even
> worse if pruning failed _without_ the fetch failing afterwards: the user
> specifically asked for stale refs to be cleaned up, the command succeeded,
> but did not do what the user asked for.
>
> Maybe Thomas has an even stronger argument in favor of erroring out. In
> any case, I don't think that `--prune` should be a "best effort, otherwise
> just shrug" option. If we wanted that, we could introduce
> `--prune-best-effort` or some such...

I am not opposed to reporting an error by exiting with non-zero exit code. I never said it should be best effort, and doing the "fetch" part after a failed prune does not make it best effort.

What I am questioning is if it makes sense to stop the fetching part. When we fetch to update multiple refs, we do not stop at the first ref-update failure, but try to do as much as possible and then report an error, no? It is the same thing.

Thomas Gummerer· Jan 28, 2022, 11:04 UTC · re: Junio C Hamano · lore

Re: [PATCH] fetch --prune: exit with error if pruning fails

Junio C Hamano writes:
Show 17 quoted lines
> Thomas Gummerer <t.gummerer@gmail.com> writes:
>
>> When pruning refs fails, we currently print an error to stderr, but
>> still exit 0 from 'git fetch'.  Since this is a genuine error fetch
>> should be exiting with some non-zero exit code.  Make it so.
>>
>> The --prune option was introduced in f360d844de ("builtin-fetch: add
>> --prune option", 2009-11-10).  Unfortunately it's unclear from that
>> commit whether ignoring the exit code was an oversight or
>> intentional, but it feels like an oversight.
>
> It is a good idea to signal, with a non-zero status, that the user
> needs to inspect the situation and possibly take a corrective
> action.  I however do not know if it is sufficient to just give the
> same exit status as returned by prune_refs(), which may or may not
> happen to crash with other possible non-zero exit status to make it
> indistinguishable from other kinds of errors.

Not sure I understand this correctly. Are you saying that we should take the return value from prune refs, and always munge that to the same exit code for 'git fetch --prune' if it fails? E.g. so prune_refs() could return 1 or 2 or 3 or whatever, but we would always want the exit code for 'fetch' to be 1?

Happy to implement that.
Show 25 quoted lines
>> diff --git a/t/t5510-fetch.sh b/t/t5510-fetch.sh
>> index 20f7110ec1..df824cc3d0 100755
>> --- a/t/t5510-fetch.sh
>> +++ b/t/t5510-fetch.sh
>> @@ -164,6 +164,16 @@ test_expect_success 'fetch --prune --tags with refspec prunes based on refspec'
>>  	git rev-parse sometag
>>  '
>>
>> +test_expect_success REFFILES 'fetch --prune fails to delete branches' '
>> +	cd "$D" &&
>> +	git clone . prune-fail &&
>> +	cd prune-fail &&
>> +	git update-ref refs/remotes/origin/extrabranch main &&
>> +	>.git/packed-refs.new &&
>> +	test_must_fail git fetch --prune origin
>
> Is it because the lockfile ".new" on packed-refs prevents deletion
> of refs but does not block creation and updates to existing refs
> that it is an effective test for the "--prune" issue?  If we somehow
> "locked" the whole ref updates, then the fetch would fail even
> without "--prune", so it may be "effective", but smells like knowing
> too much implementation detail.  Yuck, but I do not offhand think of
> any better way (it is easier to think of worse ways), so without
> further input, I would say that this is the best (or "least bad") we
> can do.

Yes, that's correct. New refs will be created as loose refs, so they don't care about packed-refs. However deletions can potentially be happening in packed-refs, and that's why it fails when 'packed-refs.new' exists.

I don't love the test either, but I also can't think of a better way to do this.

Show 6 quoted lines
> Another tangent #leftoverbits is that many tests in this script are
> so old-style that chdir around without isolating themselves in
> subshells, while some do.  We may want to clean them up when the
> file is quiescent.
>
> Thanks.
Johannes Schindelin· Jan 28, 2022, 12:34 UTC · re: Thomas Gummerer · lore

Re: [PATCH] fetch --prune: exit with error if pruning fails

Hi Thomas,
On Fri, 28 Jan 2022, Thomas Gummerer wrote:
Show 29 quoted lines
> Junio C Hamano writes:
>
> > Thomas Gummerer <t.gummerer@gmail.com> writes:
> >
> >> +test_expect_success REFFILES 'fetch --prune fails to delete branches' '
> >> +	cd "$D" &&
> >> +	git clone . prune-fail &&
> >> +	cd prune-fail &&
> >> +	git update-ref refs/remotes/origin/extrabranch main &&
> >> +	>.git/packed-refs.new &&
> >> +	test_must_fail git fetch --prune origin
> >
> > Is it because the lockfile ".new" on packed-refs prevents deletion
> > of refs but does not block creation and updates to existing refs
> > that it is an effective test for the "--prune" issue?  If we somehow
> > "locked" the whole ref updates, then the fetch would fail even
> > without "--prune", so it may be "effective", but smells like knowing
> > too much implementation detail.  Yuck, but I do not offhand think of
> > any better way (it is easier to think of worse ways), so without
> > further input, I would say that this is the best (or "least bad") we
> > can do.
>
> Yes, that's correct.  New refs will be created as loose refs, so they
> don't care about packed-refs.  However deletions can potentially be
> happening in packed-refs, and that's why it fails when 'packed-refs.new'
> exists.
>
> I don't love the test either, but I also can't think of a better way to
> do this.
Maybe add a code comment about it? Something like:
	[...]
	: this will prevent --prune from locking packed-refs &&
	>.git/packed-refs.new &&
	[...]

Ciao, Dscho

Thomas Gummerer· Jan 31, 2022, 13:07 UTC · re: Johannes Schindelin · lore

Re: [PATCH] fetch --prune: exit with error if pruning fails

Johannes Schindelin writes:
Show 40 quoted lines
> Hi Thomas,
>
> On Fri, 28 Jan 2022, Thomas Gummerer wrote:
>
>> Junio C Hamano writes:
>>
>> > Thomas Gummerer <t.gummerer@gmail.com> writes:
>> >
>> >> +test_expect_success REFFILES 'fetch --prune fails to delete branches' '
>> >> +	cd "$D" &&
>> >> +	git clone . prune-fail &&
>> >> +	cd prune-fail &&
>> >> +	git update-ref refs/remotes/origin/extrabranch main &&
>> >> +	>.git/packed-refs.new &&
>> >> +	test_must_fail git fetch --prune origin
>> >
>> > Is it because the lockfile ".new" on packed-refs prevents deletion
>> > of refs but does not block creation and updates to existing refs
>> > that it is an effective test for the "--prune" issue?  If we somehow
>> > "locked" the whole ref updates, then the fetch would fail even
>> > without "--prune", so it may be "effective", but smells like knowing
>> > too much implementation detail.  Yuck, but I do not offhand think of
>> > any better way (it is easier to think of worse ways), so without
>> > further input, I would say that this is the best (or "least bad") we
>> > can do.
>>
>> Yes, that's correct.  New refs will be created as loose refs, so they
>> don't care about packed-refs.  However deletions can potentially be
>> happening in packed-refs, and that's why it fails when 'packed-refs.new'
>> exists.
>>
>> I don't love the test either, but I also can't think of a better way to
>> do this.
>
> Maybe add a code comment about it? Something like:
>
> 	[...]
> 	: this will prevent --prune from locking packed-refs &&
> 	>.git/packed-refs.new &&
> 	[...]
Yeah, I think a comment here is a good idea, thanks!
Thomas Gummerer· Jan 31, 2022, 13:30 UTC · re: Thomas Gummerer · lore

[PATCH v2] fetch --prune: exit with error if pruning fails

When pruning refs fails, we currently print an error to stderr, but still exit 0 from 'git fetch'. Since this is a genuine error fetch should be exiting with some non-zero exit code. Make it so.

The --prune option was introduced in f360d844de ("builtin-fetch: add --prune option", 2009-11-10). Unfortunately it's unclear from that commit whether ignoring the exit code was an oversight or intentional, but it feels like an oversight.

Helped-by: Johannes Schindelin <johannes.schindelin@gmx.de>
Signed-off-by: Thomas Gummerer <t.gummerer@gmail.com>
---
changes in v2:
- retcode will now always be 1 if we get an error from prune_refs,
  no matter what the original error code was.
- we will now continue the fetch, and only exit with a non-zero
  exit code after having finished the rest of the fetch operation
- added a comment in the test explaining why creating a packed-refs.new
  file will make the fetch --prune fail.
 builtin/fetch.c  | 10 ++++++----
 t/t5510-fetch.sh | 11 +++++++++++
 2 files changed, 17 insertions(+), 4 deletions(-)
Show changes to 2 files +17 −4

builtin/fetch.c, t/t5510-fetch.sh

diff --git a/builtin/fetch.c b/builtin/fetch.c
index 5f06b21f8e..648f0694f2 100644
--- a/builtin/fetch.c
+++ b/builtin/fetch.c
@@ -1609,12 +1609,14 @@ static int do_fetch(struct transport *transport,
 		 * don't care whether --tags was specified.
 		 */
 		if (rs->nr) {
-			prune_refs(rs, ref_map, transport->url);
+			retcode = prune_refs(rs, ref_map, transport->url);
 		} else {
-			prune_refs(&transport->remote->fetch,
-				   ref_map,
-				   transport->url);
+			retcode = prune_refs(&transport->remote->fetch,
+					     ref_map,
+					     transport->url);
 		}
+		if (retcode != 0)
+			retcode = 1;
 	}
 	if (fetch_and_consume_refs(transport, ref_map, worktrees)) {
 		free_refs(ref_map);
diff --git a/t/t5510-fetch.sh b/t/t5510-fetch.sh
index 20f7110ec1..ef0da0a63b 100755
--- a/t/t5510-fetch.sh
+++ b/t/t5510-fetch.sh
@@ -164,6 +164,17 @@ test_expect_success 'fetch --prune --tags with refspec prunes based on refspec'
 	git rev-parse sometag
 '
 
+test_expect_success REFFILES 'fetch --prune fails to delete branches' '
+	cd "$D" &&
+	git clone . prune-fail &&
+	cd prune-fail &&
+	git update-ref refs/remotes/origin/extrabranch main &&
+	: this will prevent --prune from locking packed-refs for deleting refs, but adding loose refs still succeeds  &&
+	>.git/packed-refs.new &&
+
+	test_must_fail git fetch --prune origin
+'
+
 test_expect_success 'fetch --atomic works with a single branch' '
 	test_when_finished "rm -rf \"$D\"/atomic" &&
 

-- 
2.31.1
Junio C Hamano· Jan 31, 2022, 19:20 UTC · re: Thomas Gummerer · lore

Re: [PATCH v2] fetch --prune: exit with error if pruning fails

Thomas Gummerer <t.gummerer@gmail.com> writes:
Show 21 quoted lines
> diff --git a/builtin/fetch.c b/builtin/fetch.c
> index 5f06b21f8e..648f0694f2 100644
> --- a/builtin/fetch.c
> +++ b/builtin/fetch.c
> @@ -1609,12 +1609,14 @@ static int do_fetch(struct transport *transport,
>  		 * don't care whether --tags was specified.
>  		 */
>  		if (rs->nr) {
> -			prune_refs(rs, ref_map, transport->url);
> +			retcode = prune_refs(rs, ref_map, transport->url);
>  		} else {
> -			prune_refs(&transport->remote->fetch,
> -				   ref_map,
> -				   transport->url);
> +			retcode = prune_refs(&transport->remote->fetch,
> +					     ref_map,
> +					     transport->url);
>  		}
> +		if (retcode != 0)
> +			retcode = 1;
>  	}

Looks trivially correct, even though a few style things look a bit irritating to my eyes ;-).

Thanks, will queue.

← back to recent threads