Re: [PATCH v2 9/9] gpg-interface t: extend the existing GPG tests with GPGSM

> On Wed, Jul 11, 2018 at 12:38:24PM +0200, Henning Schild wrote:
> 
> > > Can we save a dummy generated key and just import it? That's what
> > > we do for the regular gpg case.  
> > 
> > I will look into storing a binary and leaving notes how it was
> > generated, just like regular gpg does. The reason i did not do that
> > in the first place is that x509 certs have a validity and we
> > introduce time into the picture. But i will see if i can generate
> > epoch->infinity to get the host clock or just the future out of the
> > picture.  
> 
> That would be preferable. But even if we can just get something like a
> 10-year expiration, that may be enough. Somebody dealing with failing
> tests and regenerating keys in ten years is probably not the end of
> the world.
> 
> It could hurt people with drastically incorrect system clocks, but I
> suspect there are other tests with similar problems (especially if
> your clock is in the past).

> > > We're going to have a lot of duplicated tests here. That's a
> > > maintenance burden when one of them needs fixes later. And when
> > > new tests are added, we won't automatically get them tested under
> > > each format.
> > > 
> > > Can we move the battery of tests into a function that takes a few
> > > parameters (prereq name, branch to look at, etc) and then call it
> > > for both the gpg/gpgsm cases?  
> > 
> > I guess this is part of the earlier "allow GPGSM without GPG" and i
> > can ignore it if we agree that this is not needed?  
> 
> I think it's orthogonal. Even if GPGSM requires GPG, you'd still want
> to make sure that whatever exercise we give to the GPG code is also
> exercised using GPGSM.
> 
> I will note, though, that _some_ tests are not really exercising
> gpg-specific bits, but more how we react to it (e.g., how we format %G
> placeholders). And it's probably OK for those to just be run once.
> 
> So in an ideal world, the test script would probably look something
> like:
> 
>   # this function holds tests which exercise the interactions
>   # with the gpg binary itself
>   type_specific_tests() {
> 	prereq=$1
> 	branch=$2
> 
> 	test_expect_success $prereq "test whatever ($prereq)" '
> 		some test using $branch here
> 	'
>   }
> 
>   test_expect_success 'setup' '
> 	set up both openpgp and x509 branches here
>   '
> 
>   type_specific_tests GPG openpgp
>   type_specific_tests GPGSM x509
> 
>   # and now tests that generically care about getting _some_ signature
>   # result (e.g., the way we format signature info)
> 
>   # and then probably a few tests specific to how the config is
> handled, # like your new gpg.format coverage
> 
> But in practice, the type-specific bits are often muddled together
> with the type-independent ones (e.g., we happen to test the parsing
> of a failed signature from gpg by checking how %G? is formatted or
> similar).
> 
> So it may be simplest to just run most of the tests twice, once with
> gpg and once with gpgsm. I kind of wonder if all of t7510 could just
> be bumped into a function. Or even into a sourced file and run from
> two different scripts. See the way that t8001 and t8002 use
> annotate-tests.sh for an example.