Re: [PATCH v1 3/3] travis-ci: run Git bisect on failed tests

From: Junio C Hamano <gitster@pobox.com>
Date: May 23, 2016, 01:22 UTC
Message-ID: <xmqq1t4tbmbp.fsf@gitster.mtv.corp.google.com>
In-Reply-To: <1463914856-64745-4-git-send-email-larsxschneider@gmail.com>

larsxschneider@gmail.com writes:

Show 7 quoted lines

> Junio usually pushes many commits at once to the public "pu"/"next"/
> "master" branches. If a test fails then it is not obvious what commit
> caused the failure. Therefore we run Git bisect with the merge base
> between the failing rev and its more stable branch ("next" for "pu",
> "master" for "next", and "maint" for "master") as good ref to find the
> offending commit. This is only enabled on "github.com/git/git" because
> there we can assume that all relevant branches are up to date.

Just FYI, 'git log next..pu' is not a very interesting range unless you also use the "--no-merges" option.

Is there a good way to tell the CI that, instead of testing 'pu' (or a specific branch in general), test all new commits that appear on the first-parent chain between 'master'..'pu'?

It would be ideal if CI can do this every time 'master' and 'pu' gets updated.

 - Find all commits that appear on the first-parent chain between
   'master..pu'.

 - For each of them:

   - Test it.  This tests the merge result and can catch breakages
     introduced by mismerging.

   - Also test its second parent if it is a merge.  This tests the
     tip of each topic branch in isolation.  CI gets a bonus point
     if it can remember that the tip of the topic did not move and
     the commit already passed the test to skip it ;-)

 - Optionally, for those topics that failed the "tip of the topic
   branch" test, "bisect master..$topic" to see where it breaks.

The thing is, "bisect" works only when you have a _single_ cause of breakage, but because 'pu' is a pile of unsorted raw material yet to be sifted into dirt and gem, once you see a failure at the tip of 'pu', mechanical 'bisect run' may not be a very useful tool, as you do not know if there is only one breakage you are looking for.

Because the unit of integration we use is a topic branch (not an individual commit on a topic branch), "this topic is broken", and "this topic itself may be good, but merging it breaks 'pu'" are far more interesting and relevant piece of information than "this commit is what broke 'pu'".