Re: [PATCH 05/15] t/Makefile: optimize chainlint self-test

From: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Date: Dec 16, 2021, 13:17 UTC
Message-ID: <211216.86zgp0adls.gmgdl@evledraar.gmail.com>
In-Reply-To: <20211213154327.pmhopjbdlkz7dgjh@fs>

On Mon, Dec 13 2021, Fabian Stelzer wrote:

Show 40 quoted lines

> On 13.12.2021 09:27, Eric Sunshine wrote:
>>On Mon, Dec 13, 2021 at 5:22 AM Fabian Stelzer <fs@gigacodes.de> wrote:
>>> On 13.12.2021 01:30, Eric Sunshine wrote:
>>> > check-chainlint:
>>> >+      sed -e '/^# LINT: /d' $(patsubst %,chainlint/%.test,$(CHAINLINTTESTS)) >'$(CHAINLINTTMP_SQ)'/tests && \
>>> >+      cat $(patsubst %,chainlint/%.expect,$(CHAINLINTTESTS)) >'$(CHAINLINTTMP_SQ)'/expect && \
>>> >+      $(CHAINLINT) '$(CHAINLINTTMP_SQ)'/tests >'$(CHAINLINTTMP_SQ)'/actual && \
>>> >+      diff -u '$(CHAINLINTTMP_SQ)'/expect '$(CHAINLINTTMP_SQ)'/actual
>>>
>>> If I read this right you are relying on the order of the .test & .expect
>>> files to match. I did something similar in a test prereq which resulted in a
>>> bug when setting the test_output_dir to something residing in /dev/shm,
>>> since the order of files in /dev/shm is reversed (at least on some
>>> platforms). Even though this should work as is I could see this leading to a
>>> similar bug in the future.
>>
>>It's not seen in the patch context, but earlier in the file we have:
>>
>>    CHAINLINTTESTS = $(sort $(...,$(wildcard chainlint/*.test)))
>>
>>which provides stability via `sort`, thus ensures that the order of
>>the ".test" and ".expect" match.
>>
>>I think that addresses your concern (unless I misunderstand your observation).
>
> Yes, thats what i meant. I didn't realize $CHAINLINTTESTS is already
> the sorted glob. Thanks for clarifying.
>
> Personally i find the initial for loop variant to be the most
> readable.  Ævars makefile targets could be very nice too, but
> especially:
>
> +$(BUILT_CHAINLINTTESTS): | .build/chainlint
> +$(BUILT_CHAINLINTTESTS): .build/%.actual: %
> +       $(CHAINLINT) <$< | \
> +	 sed -e '/^# LINT: /d' >$@ && \
> +       diff -u $(basename $<).expect $@
>
> i find very hard to grasp :/
> I have no idea what is going on here: `<$< |` ?

It's a minor point, and not relevant to this series proceeding.

But just FWIW I think both of you are wrong about the potenital for a ".test" and ".expect" bug here.

I.e. yes the CHAINLINTTESTS variable is sorted:

    CHAINLINTTESTS = $(sort $(...,$(wildcard chainlint/*.test)))

But in Eric's patch we just have this relevant to this concern of (paraphrased) "would it not be sorted break it?":

	+	sed -e '/^# LINT: /d' $(patsubst %,chainlint/%.test,$(CHAINLINTTESTS)) >'$(CHAINLINTTMP_SQ)'/tests && \
	+	cat $(patsubst %,chainlint/%.expect,$(CHAINLINTTESTS)) >'$(CHAINLINTTMP_SQ)'/expect && \

So it doesn't matter if it's sorted our not.

I.e. we've got {A,B,C}.{test,expect} files in a directory, and we're constructing a "A.test" and "A.expect" via "$(patsubst)".

So if it's "A B C", "C B A", "A C B" etc. won't matter. We'll always get ".test" files corresponding to ".expect".

If it's not sorted we'll get failure output in an unexpected order, but it won't matter to whether we detect a failure or not.