Re: [RFC][PATCH] index-pack: add testcases found using AFL

> Vegard Nossum <vegard.nossum@oracle.com> writes:
>
>> However, I think it's more useful to think of these testcases not as
>> "binary test that nobody knows what they are doing", but as "(sometimes
>> invalid) packfiles which tickle interesting code paths in the packfile
>> parser".
>>
>> With this perspective it becomes clearer that while they were generated
>> from the code, they also in a sense describe the packfile format itself.
>
> I do agree with these two paragraphs (that is why I said that
> continuously running fuzzer tests on the codebase would have value),
> and I really appreciate the effort.
>
>> I did a few experiments in changing the code of the packfile reader in
>> various small ways (e.g. deleting a check, reordering some code) to see
>> the effects of the testcases found by fuzzing, and I have to admit it
>> was fairly disappointing. The testcases I added did not catch a single
>> buggy change, whereas the other testcases did catch many of them.
>
> In short, the summary of the above three paragraphs is that we still
> do believe the general approach of using fuzzer has value, but your
> experiment indicates that data presented in the patch in this thread
> weren't particularly good examples to demonstrate the merit?