threads / patch / 56673

patchtest-lib.sh: try to re-chmod & retry on failed trash removal

Subject: [PATCH] test-lib.sh: try to re-chmod & retry on failed trash removal

## tl;dr

6 messages between Oct 9, 2021 and Oct 12, 2021. Diffs are folded; open one to read it.

replies: 5people: 3as markdown or json

Ævar Arnfjörð Bjarmason· Oct 9, 2021, 13:31 UTC · lore

This fixes a regression in [1] where t0004-unwritable.sh was made to use "test_when_finished" for cleanup, we wouldn't re-chmod the ".git/objects" on failure, leading to a persistent error when running the test.

This can be demonstrated as e.g. (output snipped for less verbosity):
    $ ./t0004-unwritable.sh --run=3 --immediate
    ok 1 # skip setup (--run)
    ok 2 # skip write-tree should notice unwritable repository (--run)
    not ok 3 - commit should notice unwritable repository
    [...]
    $ ./t0004-unwritable.sh --run=3 --immediate
    rm: cannot remove '[...]/trash directory.t0004-unwritable/.git/objects/info': Permission denied
    FATAL: Cannot prepare test area
    [...]

I.e. if any of its tests failed, and the tests were being run under "--debug"[2] or "--immediate"[3] (which was introduced after [1]) we wouldn't re-chmod the object directory. We'd then fail on the next run since the test setup couldn't remove the trash files.

Instead of some version of reverting [1] let's make the test-lib.sh resilient to this edge-case, it will happen due to [1], but also e.g. if the relevant "test-lib.sh" process is kill -9'd during the test run. We should try harder to recover in this case. If we fail to remove the test directory let's retry after (re-)chmod-ing it.

This doesn't need to be guarded by something that's equivalent to "POSIXPERM" since if we don't support "chmod" we were about to fail anyway. Let's also discard any error output from (a possibly nonexisting) "chmod", we'll fail on the subsequent "rm -rf" anyway.

The lack of &&-chaining between the "chmod" and "rm -rf" is intentional, if we fail the first "rm -rf", can't chmod, but then succeed the second time around that's what we were hoping for. We just want to nuke the directory, not carry forward every possible error code or error message.

1. dbda967684d (t0004 (unwritable files): simplify error handling,
   2010-09-06)
2. 5a4a088add3 (test-lib: do not remove trash_directory if called with
   --debug, 2008-08-21)
3. b586744a864 (test: skip clean-up when running under --immediate
   mode, 2011-06-27)
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 t/test-lib.sh | 18 +++++++++++++++---
 1 file changed, 15 insertions(+), 3 deletions(-)
Show changes to t/test-lib.sh +15 −3
diff --git a/t/test-lib.sh b/t/test-lib.sh
index aa1ad8180ed..706ce0cdeb9 100644
--- a/t/test-lib.sh
+++ b/t/test-lib.sh
@@ -1210,10 +1210,10 @@ test_done () {
 			error "Tests passed but trash directory already removed before test cleanup; aborting"
 
 			cd "$TRASH_DIRECTORY/.." &&
-			rm -fr "$TRASH_DIRECTORY" || {
+			remove_trash_directory "$TRASH_DIRECTORY" || {
 				# try again in a bit
 				sleep 5;
-				rm -fr "$TRASH_DIRECTORY"
+				remove_trash_directory "$TRASH_DIRECTORY"
 			} ||
 			error "Tests passed but test cleanup failed; aborting"
 		fi
@@ -1370,6 +1370,18 @@ then
 	exit 1
 fi
 
+# Try really hard to clean up our mess
+remove_trash_directory() {
+	dir="$1"
+	if ! rm -rf "$dir"
+	then
+		say_color info >&3 "Failed to remove trash directory, trying to re-chmod it first..."
+		chmod -R u+w "$dir" 2>/dev/null
+		rm -rf "$dir"
+	fi
+	! test -d "$dir"
+}
+
 # Are we running this test at all?
 remove_trash=
 this_test=${0##*/}
@@ -1388,7 +1400,7 @@ GNUPGHOME="$HOME/gnupg-home-not-used"
 export HOME GNUPGHOME USER_HOME
 
 # Test repository
-rm -fr "$TRASH_DIRECTORY" || {
+remove_trash_directory "$TRASH_DIRECTORY" || {
 	GIT_EXIT_OK=t
 	echo >&5 "FATAL: Cannot prepare test area"
 	exit 1
-- 
2.33.0.1492.gcc3b81fc59a
SZEDER Gábor· Oct 10, 2021, 21:36 UTC · re: Ævar Arnfjörð Bjarmason · lore

Re: [PATCH] test-lib.sh: try to re-chmod & retry on failed trash removal

On Sat, Oct 09, 2021 at 03:31:02PM +0200, Ævar Arnfjörð Bjarmason wrote:
Show 19 quoted lines
> This fixes a regression in [1] where t0004-unwritable.sh was made to
> use "test_when_finished" for cleanup, we wouldn't re-chmod the
> ".git/objects" on failure, leading to a persistent error when running
> the test.
> 
> This can be demonstrated as e.g. (output snipped for less verbosity):
> 
>     $ ./t0004-unwritable.sh --run=3 --immediate
>     ok 1 # skip setup (--run)
>     ok 2 # skip write-tree should notice unwritable repository (--run)
>     not ok 3 - commit should notice unwritable repository
>     [...]
>     $ ./t0004-unwritable.sh --run=3 --immediate
>     rm: cannot remove '[...]/trash directory.t0004-unwritable/.git/objects/info': Permission denied
>     FATAL: Cannot prepare test area
>     [...]
> 
> I.e. if any of its tests failed, and the tests were being run under
> "--debug"[2] or "--immediate"[3] (which was introduced after [1]) we

'--debug' runs the tests as usual, including any 'test_when_finished' commands, so I don't see how it would be affected. And indeed running './t0004-unwritable.sh --run=3 --debug' repeatedly doesn't have any issues with removing the trash directory of the previous run.

Show 60 quoted lines
> wouldn't re-chmod the object directory. We'd then fail on the next run
> since the test setup couldn't remove the trash files.
> 
> Instead of some version of reverting [1] let's make the test-lib.sh
> resilient to this edge-case, it will happen due to [1], but also
> e.g. if the relevant "test-lib.sh" process is kill -9'd during the
> test run. We should try harder to recover in this case. If we fail to
> remove the test directory let's retry after (re-)chmod-ing it.
> 
> This doesn't need to be guarded by something that's equivalent to
> "POSIXPERM" since if we don't support "chmod" we were about to fail
> anyway. Let's also discard any error output from (a possibly
> nonexisting) "chmod", we'll fail on the subsequent "rm -rf"
> anyway.
> 
> The lack of &&-chaining between the "chmod" and "rm -rf" is
> intentional, if we fail the first "rm -rf", can't chmod, but then
> succeed the second time around that's what we were hoping for. We just
> want to nuke the directory, not carry forward every possible error
> code or error message.
> 
> 1. dbda967684d (t0004 (unwritable files): simplify error handling,
>    2010-09-06)
> 2. 5a4a088add3 (test-lib: do not remove trash_directory if called with
>    --debug, 2008-08-21)
> 3. b586744a864 (test: skip clean-up when running under --immediate
>    mode, 2011-06-27)
> 
> Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
> ---
>  t/test-lib.sh | 18 +++++++++++++++---
>  1 file changed, 15 insertions(+), 3 deletions(-)
> 
> diff --git a/t/test-lib.sh b/t/test-lib.sh
> index aa1ad8180ed..706ce0cdeb9 100644
> --- a/t/test-lib.sh
> +++ b/t/test-lib.sh
> @@ -1210,10 +1210,10 @@ test_done () {
>  			error "Tests passed but trash directory already removed before test cleanup; aborting"
>  
>  			cd "$TRASH_DIRECTORY/.." &&
> -			rm -fr "$TRASH_DIRECTORY" || {
> +			remove_trash_directory "$TRASH_DIRECTORY" || {
>  				# try again in a bit
>  				sleep 5;
> -				rm -fr "$TRASH_DIRECTORY"
> +				remove_trash_directory "$TRASH_DIRECTORY"
>  			} ||
>  			error "Tests passed but test cleanup failed; aborting"
>  		fi
> @@ -1370,6 +1370,18 @@ then
>  	exit 1
>  fi
>  
> +# Try really hard to clean up our mess
> +remove_trash_directory() {
> +	dir="$1"
> +	if ! rm -rf "$dir"
> +	then
> +		say_color info >&3 "Failed to remove trash directory, trying to re-chmod it first..."

Is this message really necessary? I don't think it is, and there was no similar message in the previous hunk around that 'sleep 5' either.

Show 21 quoted lines
> +		chmod -R u+w "$dir" 2>/dev/null
> +		rm -rf "$dir"
> +	fi
> +	! test -d "$dir"
> +}
> +
>  # Are we running this test at all?
>  remove_trash=
>  this_test=${0##*/}
> @@ -1388,7 +1400,7 @@ GNUPGHOME="$HOME/gnupg-home-not-used"
>  export HOME GNUPGHOME USER_HOME
>  
>  # Test repository
> -rm -fr "$TRASH_DIRECTORY" || {
> +remove_trash_directory "$TRASH_DIRECTORY" || {
>  	GIT_EXIT_OK=t
>  	echo >&5 "FATAL: Cannot prepare test area"
>  	exit 1
> -- 
> 2.33.0.1492.gcc3b81fc59a
> 
Junio C Hamano· Oct 10, 2021, 22:14 UTC · re: Ævar Arnfjörð Bjarmason · lore

Re: [PATCH] test-lib.sh: try to re-chmod & retry on failed trash removal

Ævar Arnfjörð Bjarmason  <avarab@gmail.com> writes:
Show 16 quoted lines
> This can be demonstrated as e.g. (output snipped for less verbosity):
>
>     $ ./t0004-unwritable.sh --run=3 --immediate
>     ok 1 # skip setup (--run)
>     ok 2 # skip write-tree should notice unwritable repository (--run)
>     not ok 3 - commit should notice unwritable repository
>     [...]
>     $ ./t0004-unwritable.sh --run=3 --immediate
>     rm: cannot remove '[...]/trash directory.t0004-unwritable/.git/objects/info': Permission denied
>     FATAL: Cannot prepare test area
>     [...]
>
> I.e. if any of its tests failed, and the tests were being run under
> "--debug"[2] or "--immediate"[3] (which was introduced after [1]) we
> wouldn't re-chmod the object directory. We'd then fail on the next run
> since the test setup couldn't remove the trash files.

OK, the first thing these scripts do is to create $TRASH_DIRECTORY for their own use, and that is what fails (hence "Cannot prepare").

And making sure we try harder to prepare would be a good thing. Given that understanding of the motivation...

> @@ -1210,10 +1210,10 @@ test_done () {

...it is puzzling why we are touching the code that happens _after_ a test is run. If our philosophy is to leave the mess after a test fails so that the debugging person can take a look without contaminating the forensics data, but blindly and forcibly clean up before starting a new round, reusing the "do more than usual to remove" logic meant to be used for the latter here feels counter productive and also a bit counter intuitive.

Show 12 quoted lines
>  			error "Tests passed but trash directory already removed before test cleanup; aborting"
>  
>  			cd "$TRASH_DIRECTORY/.." &&
> -			rm -fr "$TRASH_DIRECTORY" || {
> +			remove_trash_directory "$TRASH_DIRECTORY" || {
>  				# try again in a bit
>  				sleep 5;
> -				rm -fr "$TRASH_DIRECTORY"
> +				remove_trash_directory "$TRASH_DIRECTORY"
>  			} ||
>  			error "Tests passed but test cleanup failed; aborting"
>  		fi
Show 7 quoted lines
> +# Try really hard to clean up our mess
> +remove_trash_directory() {
> +	dir="$1"
> +	if ! rm -rf "$dir"
> +	then
> +		say_color info >&3 "Failed to remove trash directory, trying to re-chmod it first..."
> +		chmod -R u+w "$dir" 2>/dev/null

If a test lost searchable bit from directories, "u+wx" may be necessary to clean fully, no?

Show 5 quoted lines
> +		rm -rf "$dir"
> +	fi
> +	! test -d "$dir"
> +}
> +
On the other hand...
Show 9 quoted lines
>  # Are we running this test at all?
>  remove_trash=
>  this_test=${0##*/}
> @@ -1388,7 +1400,7 @@ GNUPGHOME="$HOME/gnupg-home-not-used"
>  export HOME GNUPGHOME USER_HOME
>  
>  # Test repository
> -rm -fr "$TRASH_DIRECTORY" || {
> +remove_trash_directory "$TRASH_DIRECTORY" || {

... this one does make sense and matches what the proposed log message sold us the change as.

>  	GIT_EXIT_OK=t
>  	echo >&5 "FATAL: Cannot prepare test area"
>  	exit 1
Thanks.
Junio C Hamano· Oct 11, 2021, 04:34 UTC · re: Junio C Hamano · lore

Re: [PATCH] test-lib.sh: try to re-chmod & retry on failed trash removal

Junio C Hamano <gitster@pobox.com> writes:
>> +		chmod -R u+w "$dir" 2>/dev/null
>
> If a test lost searchable bit from directories, "u+wx" may be
> necessary to clean fully, no?
Sorry, but I was stupid.

If we were to worry about losing the searchable bit, then I should considered the possibility that we may also lose the readable bit, too.

Ævar Arnfjörð Bjarmason· Oct 11, 2021, 01:41 UTC · re: Ævar Arnfjörð Bjarmason · lore

[PATCH v2] test-lib.sh: try to re-chmod & retry on failed trash removal

Try to re-chmod the trash directory on startup if we fail to "rm -rf" it. This fixes problems where the test leaves the trash directory behind in a bad permission state for whatever reason.

This fixes an interaction between [1] where t0004-unwritable.sh was made to use "test_when_finished" for cleanup, and [2] which added the "--immediate" mode. If a test in this file failed when running with "--immediate" we wouldn't run the "test_when_finished" block, which re-chmods the ".git/objects" directory (see [1]).

This can be demonstrated as e.g. (output snipped for less verbosity):
    $ ./t0004-unwritable.sh --run=3 --immediate
    ok 1 # skip setup (--run)
    ok 2 # skip write-tree should notice unwritable repository (--run)
    not ok 3 - commit should notice unwritable repository
    [...]
    $ ./t0004-unwritable.sh --run=3 --immediate
    rm: cannot remove '[...]/trash directory.t0004-unwritable/.git/objects/info': Permission denied
    FATAL: Cannot prepare test area
    [...]

Instead of some version of reverting [1] let's make the test-lib.sh resilient to this edge-case, it will happen due to [1], but also e.g. if the relevant "test-lib.sh" process is kill -9'd during the test run. We should try harder to recover in this case. If we fail to remove the test directory let's retry after (re-)chmod-ing it.

This doesn't need to be guarded by something that's equivalent to "POSIXPERM" since if we don't support "chmod" we were about to fail anyway.

Let's also discard any error output from (a possibly nonexisting) "chmod", we'll fail on the subsequent "rm -rf" anyway, likewise for the first "rm -rf" invocation, we don't want to get the "cannot remove" output if we can get around it with the "chmod", but we do want any error output from the second "rm -rf", in case that doesn't fix the issue.

The lack of &&-chaining between the "chmod" and "rm -rf" is intentional, if we fail the first "rm -rf", can't chmod, but then succeed the second time around that's what we were hoping for. We just want to nuke the directory, not carry forward every possible error code or error message.

1. dbda967684d (t0004 (unwritable files): simplify error handling,
   2010-09-06)
2. b586744a864 (test: skip clean-up when running under --immediate
   mode, 2011-06-27)
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
Range-diff against v1:
1:  d7e88a77fef ! 1:  9cfc0621067 test-lib.sh: try to re-chmod & retry on failed trash removal
    @@ Metadata
      ## Commit message ##
         test-lib.sh: try to re-chmod & retry on failed trash removal
     
    -    This fixes a regression in [1] where t0004-unwritable.sh was made to
    -    use "test_when_finished" for cleanup, we wouldn't re-chmod the
    -    ".git/objects" on failure, leading to a persistent error when running
    -    the test.
    +    Try to re-chmod the trash directory on startup if we fail to "rm -rf"
    +    it. This fixes problems where the test leaves the trash directory
    +    behind in a bad permission state for whatever reason.
    +
    +    This fixes an interaction between [1] where t0004-unwritable.sh was
    +    made to use "test_when_finished" for cleanup, and [2] which added the
    +    "--immediate" mode. If a test in this file failed when running with
    +    "--immediate" we wouldn't run the "test_when_finished" block, which
    +    re-chmods the ".git/objects" directory (see [1]).
     
         This can be demonstrated as e.g. (output snipped for less verbosity):
     
    @@ Commit message
             FATAL: Cannot prepare test area
             [...]
     
    -    I.e. if any of its tests failed, and the tests were being run under
    -    "--debug"[2] or "--immediate"[3] (which was introduced after [1]) we
    -    wouldn't re-chmod the object directory. We'd then fail on the next run
    -    since the test setup couldn't remove the trash files.
    -
         Instead of some version of reverting [1] let's make the test-lib.sh
         resilient to this edge-case, it will happen due to [1], but also
         e.g. if the relevant "test-lib.sh" process is kill -9'd during the
    @@ Commit message
     
         This doesn't need to be guarded by something that's equivalent to
         "POSIXPERM" since if we don't support "chmod" we were about to fail
    -    anyway. Let's also discard any error output from (a possibly
    -    nonexisting) "chmod", we'll fail on the subsequent "rm -rf"
         anyway.
     
    +    Let's also discard any error output from (a possibly nonexisting)
    +    "chmod", we'll fail on the subsequent "rm -rf" anyway, likewise for
    +    the first "rm -rf" invocation, we don't want to get the "cannot
    +    remove" output if we can get around it with the "chmod", but we do
    +    want any error output from the second "rm -rf", in case that doesn't
    +    fix the issue.
    +
         The lack of &&-chaining between the "chmod" and "rm -rf" is
         intentional, if we fail the first "rm -rf", can't chmod, but then
         succeed the second time around that's what we were hoping for. We just
    @@ Commit message
     
         1. dbda967684d (t0004 (unwritable files): simplify error handling,
            2010-09-06)
    -    2. 5a4a088add3 (test-lib: do not remove trash_directory if called with
    -       --debug, 2008-08-21)
    -    3. b586744a864 (test: skip clean-up when running under --immediate
    +    2. b586744a864 (test: skip clean-up when running under --immediate
            mode, 2011-06-27)
     
         Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
     
      ## t/test-lib.sh ##
    -@@ t/test-lib.sh: test_done () {
    - 			error "Tests passed but trash directory already removed before test cleanup; aborting"
    - 
    - 			cd "$TRASH_DIRECTORY/.." &&
    --			rm -fr "$TRASH_DIRECTORY" || {
    -+			remove_trash_directory "$TRASH_DIRECTORY" || {
    - 				# try again in a bit
    - 				sleep 5;
    --				rm -fr "$TRASH_DIRECTORY"
    -+				remove_trash_directory "$TRASH_DIRECTORY"
    - 			} ||
    - 			error "Tests passed but test cleanup failed; aborting"
    - 		fi
    -@@ t/test-lib.sh: then
    - 	exit 1
    - fi
    +@@ t/test-lib.sh: HOME="$TRASH_DIRECTORY"
    + GNUPGHOME="$HOME/gnupg-home-not-used"
    + export HOME GNUPGHOME USER_HOME
      
    -+# Try really hard to clean up our mess
    ++# "rm -rf" existing trash directory, even if a previous run left it
    ++# with bad permissions.
     +remove_trash_directory() {
     +	dir="$1"
    -+	if ! rm -rf "$dir"
    ++	if ! rm -rf "$dir" 2>/dev/null
     +	then
    -+		say_color info >&3 "Failed to remove trash directory, trying to re-chmod it first..."
    -+		chmod -R u+w "$dir" 2>/dev/null
    ++		chmod -R u+wx "$dir"
     +		rm -rf "$dir"
     +	fi
     +	! test -d "$dir"
     +}
     +
    - # Are we running this test at all?
    - remove_trash=
    - this_test=${0##*/}
    -@@ t/test-lib.sh: GNUPGHOME="$HOME/gnupg-home-not-used"
    - export HOME GNUPGHOME USER_HOME
    - 
      # Test repository
     -rm -fr "$TRASH_DIRECTORY" || {
     +remove_trash_directory "$TRASH_DIRECTORY" || {
 t/test-lib.sh | 14 +++++++++++++-
 1 file changed, 13 insertions(+), 1 deletion(-)
Show changes to t/test-lib.sh +13 −1
diff --git a/t/test-lib.sh b/t/test-lib.sh
index aa1ad8180ed..ff6d3c79362 100644
--- a/t/test-lib.sh
+++ b/t/test-lib.sh
@@ -1387,8 +1387,20 @@ HOME="$TRASH_DIRECTORY"
 GNUPGHOME="$HOME/gnupg-home-not-used"
 export HOME GNUPGHOME USER_HOME
 
+# "rm -rf" existing trash directory, even if a previous run left it
+# with bad permissions.
+remove_trash_directory() {
+	dir="$1"
+	if ! rm -rf "$dir" 2>/dev/null
+	then
+		chmod -R u+wx "$dir"
+		rm -rf "$dir"
+	fi
+	! test -d "$dir"
+}
+
 # Test repository
-rm -fr "$TRASH_DIRECTORY" || {
+remove_trash_directory "$TRASH_DIRECTORY" || {
 	GIT_EXIT_OK=t
 	echo >&5 "FATAL: Cannot prepare test area"
 	exit 1
-- 
2.33.0.1505.g38ce83107ad
Junio C Hamano· Oct 12, 2021, 23:03 UTC · re: Ævar Arnfjörð Bjarmason · lore

Re: [PATCH v2] test-lib.sh: try to re-chmod & retry on failed trash removal

Mostly looking good, but I applied the following on top to be squashed in.

Thanks.

-- >8 -- Style fix plus adding more paranoia to cope with the lost 'r' bit.

Show changes to t/test-lib.sh +2 −2
diff --git a/t/test-lib.sh b/t/test-lib.sh
index 138a26c19b..d820910154 100644
--- a/t/test-lib.sh
+++ b/t/test-lib.sh
@@ -1386,11 +1386,11 @@ export HOME GNUPGHOME
 
 # "rm -rf" existing trash directory, even if a previous run left it
 # with bad permissions.
-remove_trash_directory() {
+remove_trash_directory () {
 	dir="$1"
 	if ! rm -rf "$dir" 2>/dev/null
 	then
-		chmod -R u+wx "$dir"
+		chmod -R u+rwx "$dir"
 		rm -rf "$dir"
 	fi
 	! test -d "$dir"

← back to recent threads