threads / discuss / 51251

Re: Git self test failure on Solaris 11.3

Subject: Re: Git self test failure on Solaris 11.3

## tl;dr

5 messages between Jun 6, 2019 and Jun 7, 2019.

replies: 4people: 3as markdown or json

Eric Sunshine· Jun 6, 2019, 17:18 UTC · lore
[forwarding to the Git list]
On Sun, Jun 2, 2019 at 6:23 AM Jeffrey Walton <noloader@gmail.com> wrote:
Show 23 quoted lines
> On Sun, Jun 2, 2019 at 5:09 AM Jeffrey Walton <noloader@gmail.com> wrote:
> > I'm catching a self test failure on Solaris 11.3. Git 2.21 from sources.
> >
> > ok 8 - check normal git operations: two packs
> > ok 9 - add more packs
> > ok 10 - check normal git operations: mixed mode (two packs + extra)
> > ok 11 - write midx with twelve packs
> > not ok 12 - check normal git operations: twelve packs
> > #
> > #                       midx_git_two_modes "rev-list --objects --all" &&
> > #                       midx_git_two_modes "log --raw" &&
> > #                       midx_git_two_modes "count-objects --verbose" &&
> > #                       midx_git_two_modes "cat-file
> > --batch-all-objects --buffer --batch-check" &&
> > #                       midx_git_two_modes "cat-file
> > --batch-all-objects --buffer --batch-check --unsorted" sorted
> > #
> > ok 13 - verify multi-pack-index success
> > ok 14 - verify bad signature
> > ok 15 - verify bad version
> > ...
>
> Ick, this is bad. Intermittent failure. The second time around it passed.

Jeff Walton reported this to me privately. I'm not familiar with this code and don't have time presently to investigate it, so I'm forwarding it to the list in the hope that someone who knows the code intimately (Stolee) or someone who has touched this code (Peff) might have some idea about it. The failure is intermittent, so perhaps the output of some command is not stable sort-wise(?).

Jeff King· Jun 6, 2019, 17:35 UTC · re: Eric Sunshine · lore
On Thu, Jun 06, 2019 at 01:18:01PM -0400, Eric Sunshine wrote:
Show 18 quoted lines
> > > not ok 12 - check normal git operations: twelve packs
> > > #
> > > #                       midx_git_two_modes "rev-list --objects --all" &&
> > > #                       midx_git_two_modes "log --raw" &&
> > > #                       midx_git_two_modes "count-objects --verbose" &&
> > > #                       midx_git_two_modes "cat-file
> > > --batch-all-objects --buffer --batch-check" &&
> > > #                       midx_git_two_modes "cat-file
> > > --batch-all-objects --buffer --batch-check --unsorted" sorted
> > > #
> [...]
> 
> Jeff Walton reported this to me privately. I'm not familiar with this
> code and don't have time presently to investigate it, so I'm
> forwarding it to the list in the hope that someone who knows the code
> intimately (Stolee) or someone who has touched this code (Peff) might
> have some idea about it. The failure is intermittent, so perhaps the
> output of some command is not stable sort-wise(?).

I can't seem to get it to fail for me, even under high load. Do you know which of those commands is failing (i.e., what does "-v -x" say)?

If it's the rev-list or log output, it could be due to traversal order. I notice that the commit creation doesn't increment the timestamps. This might help:

diff --git a/t/t5319-multi-pack-index.sh b/t/t5319-multi-pack-index.sh
index 70926b5bc0..79aa97efce 100755
--- a/t/t5319-multi-pack-index.sh
+++ b/t/t5319-multi-pack-index.sh
@@ -60,6 +60,7 @@ commit_and_list_objects () {
 	} >file_101 &&
 	git update-index --add file_101 &&
 	tree=$(git write-tree) &&
+	test_tick &&
 	commit=$(git commit-tree $tree -p HEAD</dev/null) &&
 	{
 		echo $tree &&

Though I think in each case that the results should be deterministic for
a given set of commits (and the tests here are just running over the
same set of commits with and without midx; so we should see the same
output in both cases).

The normal "cat-file" output should be sorted by sha1, so should be
stable. The unsorted output will be in whatever order we happen to hit
the packfiles, which _would_ be subject to timestamp variations.
However, Jeffrey's email mentions Git 2.21, which predates my b4a14394af
(t5319: fix bogus cat-file argument, 2019-04-05). So before that,
"cat-file --unsorted" should always be producing no output. :)

After the switch to correctly use --unordered, I could believe there
might be a timestamp issue, but the "sorted" flag there means that we
should be sorting the output before comparing.

I can't reproduce the intermittent failure either on 2.21.0, or with
v2.22.0-rc3.

-Peff
Eric Sunshine· Jun 6, 2019, 19:00 UTC · re: Jeff King · lore
On Thu, Jun 6, 2019 at 1:35 PM Jeff King <peff@peff.net> wrote:
Show 12 quoted lines
> On Thu, Jun 06, 2019 at 01:18:01PM -0400, Eric Sunshine wrote:
> > > > not ok 12 - check normal git operations: twelve packs
> >
> > Jeff Walton reported this to me privately. I'm not familiar with this
> > code and don't have time presently to investigate it, so I'm
> > forwarding it to the list in the hope that someone who knows the code
> > intimately (Stolee) or someone who has touched this code (Peff) might
> > have some idea about it. The failure is intermittent, so perhaps the
> > output of some command is not stable sort-wise(?).
>
> I can't seem to get it to fail for me, even under high load. Do you know
> which of those commands is failing (i.e., what does "-v -x" say)?
I don't know.
> I can't reproduce the intermittent failure either on 2.21.0, or with
> v2.22.0-rc3.

I can't reproduce it either on Jeff's Solaris box. Perhaps Jeff can add "-v -x" to his automated build/test script in order to help diagnose the problem if it occurs again.

Thanks for looking into it.
Jeff King· Jun 6, 2019, 19:05 UTC · re: Eric Sunshine · lore
On Thu, Jun 06, 2019 at 03:00:00PM -0400, Eric Sunshine wrote:
Show 6 quoted lines
> > I can't reproduce the intermittent failure either on 2.21.0, or with
> > v2.22.0-rc3.
> 
> I can't reproduce it either on Jeff's Solaris box. Perhaps Jeff can
> add "-v -x" to his automated build/test script in order to help
> diagnose the problem if it occurs again.
Jeff: try "--verbose-log -x" if you're doing this as part of an automated
run. That will leave you nice logs in t/test-results without dumping it
all (and in particular, if you run the tests in parallel, it will avoid
interleaving output from various tests).
-Peff
Derrick Stolee· Jun 7, 2019, 01:03 UTC · re: Jeff King · lore
On 6/6/2019 3:05 PM, Jeff King wrote:
Show 13 quoted lines
> On Thu, Jun 06, 2019 at 03:00:00PM -0400, Eric Sunshine wrote:
> 
>>> I can't reproduce the intermittent failure either on 2.21.0, or with
>>> v2.22.0-rc3.
>>
>> I can't reproduce it either on Jeff's Solaris box. Perhaps Jeff can
>> add "-v -x" to his automated build/test script in order to help
>> diagnose the problem if it occurs again.
> 
> Jeff: try "--verbose-log -x" if you're doing this as part of an automated
> run. That will leave you nice logs in t/test-results without dumping it
> all (and in particular, if you run the tests in parallel, it will avoid
> interleaving output from various tests).

Please keep me posted with any updates with exactly which command failed. These comparison tests are intended to catch surprising interactions, but are not very narrow. We'll need to know the exact failure spot, and the failure could take a few forms:

* The git command failed with an error.
* The midx-enabled and midx-disabled commands differ in output.

Between these two options and the different commands that could have failed, it's hard to dig in without more to go on.

Thanks, -Stolee

← back to recent threads