threads / patch / 58748

patcht7610: fix flaky timeout issue, don't clone from example.com

Subject: [PATCH] t7610: fix flaky timeout issue, don't clone from example.com

## tl;dr

9 messages between Nov 5, 2022 and Nov 17, 2022. Diffs are folded; open one to read it.

replies: 8people: 2as markdown or json

Ævar Arnfjörð Bjarmason· Nov 5, 2022, 11:54 UTC · lore

When t7610-mergetool.sh runs without failures the git://example.com submodule URLs will never be used. That's because we "git submodule add" it, but then manually populate them so that subsequent "git submodule update -N" won't attempt to clone it, only update it without fetching.

But if we fail in an earlier test it'll have the knock-on effect of having later tests hang on that "git submodule update -N" as we attempt to clone this repository from example.com.

This can be reproduced on "master" by running the test with SANITIZE=leak without "--immediate". With "GIT_TEST_PASSING_SANITIZE_LEAK=true" (which the linux-leaks job uses) we'll skip the test entirely. So we'll only run into this when running it manually, or with the "GIT_TEST_PASSING_SANITIZE_LEAK=check" mode.

That's not because the failure has anything to do with leak detection per-se. It just so happens that we have a leak that'll fail before we've managed to fully set these up, and therefore "git submodule update -N" ends up spawning "git clone".

Let's instead continue lying about the origin of this submodule by providing a URL for it that doesn't work, but now one that *really* doesn't work: /dev/null. If the test is passing we won't ever use this, and if we have knock-on failures we'll fail early, instead of waiting for a timeout.

The behavior of "-N" here might be surprising to some, since it's explained as "[if you use -N we] don’t fetch new objects from the remote site". But (perhaps counter-intuitively) it's only talking about if it needs to do so via "git fetch". In this case we'll end up spawning a "git clone", as we have no submodule set up.

See ff7f089ed10 (mergetool: Teach about submodules, 2011-04-13) for the commit that implemented these "example.com" tests.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 t/t7610-mergetool.sh | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
Show changes to t/t7610-mergetool.sh +2 −2
diff --git a/t/t7610-mergetool.sh b/t/t7610-mergetool.sh
index 8cc64729adb..b1ba0d9a088 100755
--- a/t/t7610-mergetool.sh
+++ b/t/t7610-mergetool.sh
@@ -33,7 +33,7 @@ test_expect_success 'setup' '
 		git add foo &&
 		git commit -m "Add foo"
 	) &&
-	git submodule add git://example.com/submod submod &&
+	git submodule add /dev/null submod &&
 	git add file1 "spaced name" file1[1-4] subdir/file3 .gitmodules submod &&
 	git commit -m "add initial versions" &&
 
@@ -614,7 +614,7 @@ test_expect_success 'submodule in subdirectory' '
 		)
 	) &&
 	test_when_finished "rm -rf subdir/subdir_module" &&
-	git submodule add git://example.com/subsubmodule subdir/subdir_module &&
+	git submodule add /dev/null subdir/subdir_module &&
 	git add subdir/subdir_module &&
 	git commit -m "add submodule in subdirectory" &&
 
-- 
2.38.0.1452.g710f45c7951
Taylor Blau· Nov 9, 2022, 22:47 UTC · re: Ævar Arnfjörð Bjarmason · lore

Re: [PATCH] t7610: fix flaky timeout issue, don't clone from example.com

On Sat, Nov 05, 2022 at 12:54:21PM +0100, Ævar Arnfjörð Bjarmason wrote:
Show 5 quoted lines
> The behavior of "-N" here might be surprising to some, since it's
> explained as "[if you use -N we] don’t fetch new objects from the
> remote site". But (perhaps counter-intuitively) it's only talking
> about if it needs to do so via "git fetch". In this case we'll end up
> spawning a "git clone", as we have no submodule set up.

Makes sense, though I'm not sure I agree this is worth patching as I currently understand it.

If I run t7610 today with '--run=2-' (IOW, skipping the setup test), I am definitely going to get failures. And I don't think we should have every subsequent test depend on having run anything containing "setup" before it. That is, it is not surprising that we will see some test failures when carving up and running portions of the test instead of the whole file.

I don't think this is substantively any different than that. Whether we don't complete the "setup" test because we had some leak (and ran the test suite with the appropriate LSan options), or because we neglected to run it at all, I don't think there is a significant difference.

Either way, the end-state of the test repository isn't guaranteed to match the intent of the "setup" test.

If this is the only such problem in-tree, sure, I think it's fine to patch. But I would wager that there are *many* more than just this one lurking, and patching all of them would be less straightforward than this one.

So... I don't know. I'm certainly leaning negative on this patch, but if you have some compelling reason that I'm missing, I'm all-ears.

Thanks, Taylor

Ævar Arnfjörð Bjarmason· Nov 9, 2022, 23:55 UTC · re: Taylor Blau · lore

Re: [PATCH] t7610: fix flaky timeout issue, don't clone from example.com

On Wed, Nov 09 2022, Taylor Blau wrote:
Show 32 quoted lines
> On Sat, Nov 05, 2022 at 12:54:21PM +0100, Ævar Arnfjörð Bjarmason wrote:
>> The behavior of "-N" here might be surprising to some, since it's
>> explained as "[if you use -N we] don’t fetch new objects from the
>> remote site". But (perhaps counter-intuitively) it's only talking
>> about if it needs to do so via "git fetch". In this case we'll end up
>> spawning a "git clone", as we have no submodule set up.
>
> Makes sense, though I'm not sure I agree this is worth patching as I
> currently understand it.
>
> If I run t7610 today with '--run=2-' (IOW, skipping the setup test), I
> am definitely going to get failures. And I don't think we should have
> every subsequent test depend on having run anything containing "setup"
> before it. That is, it is not surprising that we will see some test
> failures when carving up and running portions of the test instead of the
> whole file.
>
> I don't think this is substantively any different than that. Whether we
> don't complete the "setup" test because we had some leak (and ran the
> test suite with the appropriate LSan options), or because we neglected
> to run it at all, I don't think there is a significant difference.
>
> Either way, the end-state of the test repository isn't guaranteed to
> match the intent of the "setup" test.
>
> If this is the only such problem in-tree, sure, I think it's fine to
> patch. But I would wager that there are *many* more than just this one
> lurking, and patching all of them would be less straightforward than
> this one.
>
> So... I don't know. I'm certainly leaning negative on this patch, but if
> you have some compelling reason that I'm missing, I'm all-ears.

I agree with that in general, but the expected failure case for all those other things that are broken with the missing setup is just that we'll predictably error out immediately.

Whereas in this case we'll end up hanging on a connect() to some box that IANA's maintaining for the example.com landing page, and after we reach the connect() timeout we move onto the next failing test, which will do that again. The test takes a long time to finally report an overall failure.

I think that's worth fixing in general, aside from the leak detection. I.e. yes in general we aren't good about running tests 3..9 successfully if the 2nd test fails.

But we generally just fail some or all of 3..9 pretty fast, and don't start taking 20 minutes to run the test, when it took 10s before (or whatever).

Taylor Blau· Nov 10, 2022, 02:17 UTC · re: Ævar Arnfjörð Bjarmason · lore

Re: [PATCH] t7610: fix flaky timeout issue, don't clone from example.com

On Thu, Nov 10, 2022 at 12:55:30AM +0100, Ævar Arnfjörð Bjarmason wrote:
> But we generally just fail some or all of 3..9 pretty fast, and don't
> start taking 20 minutes to run the test, when it took 10s before (or
> whatever).

OK. I still think that in principle this is indistinguishable from not running a setup test to completion.

*But*: I'm willing to treat them differently since instead of manifesting in an immediate failure later on in the suite, we hang for a substantial period of time.

So I'm content to merge this down, but I don't think it's worth searching out for more of these in the future.

Thanks, Taylor

Taylor Blau· Nov 15, 2022, 19:42 UTC · re: Taylor Blau · lore

Re: [PATCH] t7610: fix flaky timeout issue, don't clone from example.com

On Wed, Nov 09, 2022 at 09:17:25PM -0500, Taylor Blau wrote:
Show 14 quoted lines
> On Thu, Nov 10, 2022 at 12:55:30AM +0100, Ævar Arnfjörð Bjarmason wrote:
> > But we generally just fail some or all of 3..9 pretty fast, and don't
> > start taking 20 minutes to run the test, when it took 10s before (or
> > whatever).
>
> OK. I still think that in principle this is indistinguishable from not
> running a setup test to completion.
>
> *But*: I'm willing to treat them differently since instead of
> manifesting in an immediate failure later on in the suite, we hang for a
> substantial period of time.
>
> So I'm content to merge this down, but I don't think it's worth
> searching out for more of these in the future.

Having merged this down to 'next', it looks like there is some CI fallout on the Windows tests, see:

  https://github.com/ttaylorr/git/actions/runs/3473324797/jobs/5805324776

I am not sure how I might have caught this earlier not having a Windows machine myself. Regardless, let's make sure that it is fixed up before this graduates.

Thanks, Taylor

Ævar Arnfjörð Bjarmason· Nov 15, 2022, 23:40 UTC · re: Taylor Blau · lore

[PATCH] t7610: use "file:///dev/null", not "/dev/null", fixes MinGW

On MinGW the "/dev/null" is translated to "nul" on command-lines, even though as in this case it'll never end up referring to an actual file.

So on Windows the fix for the previous "example.com" timeout issue in 8354cf752ec (t7610: fix flaky timeout issue, don't clone from example.com, 2022-11-05) would yield:

  fatal: repo URL: 'nul' must be absolute or begin with ./|../

Let's evade this yet again by prefixing this with "file://", which makes this pass in the Windows CI.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
On Tue, Nov 15 2022, Taylor Blau wrote:
Show 24 quoted lines
> On Wed, Nov 09, 2022 at 09:17:25PM -0500, Taylor Blau wrote:
>> On Thu, Nov 10, 2022 at 12:55:30AM +0100, Ævar Arnfjörð Bjarmason wrote:
>> > But we generally just fail some or all of 3..9 pretty fast, and don't
>> > start taking 20 minutes to run the test, when it took 10s before (or
>> > whatever).
>>
>> OK. I still think that in principle this is indistinguishable from not
>> running a setup test to completion.
>>
>> *But*: I'm willing to treat them differently since instead of
>> manifesting in an immediate failure later on in the suite, we hang for a
>> substantial period of time.
>>
>> So I'm content to merge this down, but I don't think it's worth
>> searching out for more of these in the future.
>
> Having merged this down to 'next', it looks like there is some CI
> fallout on the Windows tests, see:
>
>   https://github.com/ttaylorr/git/actions/runs/3473324797/jobs/5805324776
>
> I am not sure how I might have caught this earlier not having a Windows
> machine myself. Regardless, let's make sure that it is fixed up before
> this graduates.

Sorry again :( I think my CI was quite queued up at the time, and I figured surely *this* won't have any porability issues, but forgot about MinGW's sneaky /dev/null string-replacement behavior.

Windows CI passes with this: https://github.com/avar/git/actions/runs/3473817195

 t/t7610-mergetool.sh | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
Show changes to t/t7610-mergetool.sh +2 −2
diff --git a/t/t7610-mergetool.sh b/t/t7610-mergetool.sh
index b1ba0d9a088..7b957022f1a 100755
--- a/t/t7610-mergetool.sh
+++ b/t/t7610-mergetool.sh
@@ -33,7 +33,7 @@ test_expect_success 'setup' '
 		git add foo &&
 		git commit -m "Add foo"
 	) &&
-	git submodule add /dev/null submod &&
+	git submodule add file:///dev/null submod &&
 	git add file1 "spaced name" file1[1-4] subdir/file3 .gitmodules submod &&
 	git commit -m "add initial versions" &&
 
@@ -614,7 +614,7 @@ test_expect_success 'submodule in subdirectory' '
 		)
 	) &&
 	test_when_finished "rm -rf subdir/subdir_module" &&
-	git submodule add /dev/null subdir/subdir_module &&
+	git submodule add file:///dev/null subdir/subdir_module &&
 	git add subdir/subdir_module &&
 	git commit -m "add submodule in subdirectory" &&
 
-- 
2.38.0.1473.g172bcc0511c
Taylor Blau· Nov 16, 2022, 01:07 UTC · re: Ævar Arnfjörð Bjarmason · lore

Re: [PATCH] t7610: use "file:///dev/null", not "/dev/null", fixes MinGW

On Wed, Nov 16, 2022 at 12:40:14AM +0100, Ævar Arnfjörð Bjarmason wrote:
Show 11 quoted lines
> On MinGW the "/dev/null" is translated to "nul" on command-lines, even
> though as in this case it'll never end up referring to an actual file.
>
> So on Windows the fix for the previous "example.com" timeout issue in
> 8354cf752ec (t7610: fix flaky timeout issue, don't clone from
> example.com, 2022-11-05) would yield:
>
>   fatal: repo URL: 'nul' must be absolute or begin with ./|../
>
> Let's evade this yet again by prefixing this with "file://", which
> makes this pass in the Windows CI.

Thanks very much. I just picked this up on top of my 'ab/t7610-timeout' branch and pushed it out to ttaylorr/git to double-check your results against CI. (Not that I don't trust you, of course, but I'm feeling like I should be extra-cautious ;-)).

Assuming that all looks good, then I'll push out a new version of 'next' and dependent branches to the usual spots so that we can get 'next' back to green.

Thanks for working on this so quickly.

Thanks, Taylor

Ævar Arnfjörð Bjarmason· Nov 17, 2022, 10:08 UTC · re: Taylor Blau · lore

Re: [PATCH] t7610: use "file:///dev/null", not "/dev/null", fixes MinGW

On Tue, Nov 15 2022, Taylor Blau wrote:
Show 23 quoted lines
> On Wed, Nov 16, 2022 at 12:40:14AM +0100, Ævar Arnfjörð Bjarmason wrote:
>> On MinGW the "/dev/null" is translated to "nul" on command-lines, even
>> though as in this case it'll never end up referring to an actual file.
>>
>> So on Windows the fix for the previous "example.com" timeout issue in
>> 8354cf752ec (t7610: fix flaky timeout issue, don't clone from
>> example.com, 2022-11-05) would yield:
>>
>>   fatal: repo URL: 'nul' must be absolute or begin with ./|../
>>
>> Let's evade this yet again by prefixing this with "file://", which
>> makes this pass in the Windows CI.
>
> Thanks very much. I just picked this up on top of my 'ab/t7610-timeout'
> branch and pushed it out to ttaylorr/git to double-check your results
> against CI. (Not that I don't trust you, of course, but I'm feeling like
> I should be extra-cautious ;-)).
>
> Assuming that all looks good, then I'll push out a new version of 'next'
> and dependent branches to the usual spots so that we can get 'next' back
> to green.
>
> Thanks for working on this so quickly.

And thank for picking it up! Looks like "next" is passing CI now: https://github.com/git/git/actions/runs/3475751137/jobs/5810308794

Taylor Blau· Nov 17, 2022, 21:50 UTC · re: Ævar Arnfjörð Bjarmason · lore

Re: [PATCH] t7610: use "file:///dev/null", not "/dev/null", fixes MinGW

On Thu, Nov 17, 2022 at 11:08:08AM +0100, Ævar Arnfjörð Bjarmason wrote:
> And thank for picking it up! Looks like "next" is passing CI now:
> https://github.com/git/git/actions/runs/3475751137/jobs/5810308794

No problem. I dislike pushing out 'next' without a corresponding WC report, but I dislike 'next' failing CI even more. So I felt like here it was OK.

Thanks, Taylor

← back to recent threads