Git bisect run should check for the existence of the script

Hi,

When using 'git bisect run', if the name of the script is misspelled or if the script was not made executable, 'git bisect' considers it to be a failure and stops at the first revision after the one claimed good. It would be better in my eyes to error out.

$ git bisect start
$ git bisect bad HEAD
$ git bisect good HEAD~10
Bisecting: 4 revisions left to test after this (roughly 2 steps)
[344dce312a0cf86d5a5772d54843cc179acaf6e3] bpo-41228: Fix /a/are/ in 
monthcalendar() descripton (GH-21372)
$ git bisect run ./non-existent.sh
running ./non-existent.sh
/usr/lib/git-core/git-bisect: 247: ./non-existent.sh: not found
Bisecting: 2 revisions left to test after this (roughly 1 step)
[6fc732a2116e2c42b0431bb7e2a21719351af755] Fix typo in docs: 'created by 
th' -> 'created by the' (GH-21384)
running ./non-existent.sh
/usr/lib/git-core/git-bisect: 247: ./non-existent.sh: not found
Bisecting: 0 revisions left to test after this (roughly 0 steps)
[8182cc2e68a3c6ea5d5342fed3f1c76b0521fbc1] bpo-39573: Use the Py_TYPE() 
macro (GH-21433)
running ./non-existent.sh
/usr/lib/git-core/git-bisect: 247: ./non-existent.sh: not found
8182cc2e68a3c6ea5d5342fed3f1c76b0521fbc1 is the first bad commit
commit 8182cc2e68a3c6ea5d5342fed3f1c76b0521fbc1
Author: Victor Stinner <vstinner@python.org>
Date:   Fri Jul 10 12:40:38 2020 +0200

     bpo-39573: Use the Py_TYPE() macro (GH-21433)

     Replace obj->ob_type with Py_TYPE(obj).

  Modules/_elementtree.c       | 2 +-
  Objects/abstract.c           | 4 ++--
  Objects/genericaliasobject.c | 2 +-
  Objects/unicodeobject.c      | 4 ++--
  PC/_msi.c                    | 6 +++---
  PC/winreg.c                  | 4 ++--
  Tools/scripts/combinerefs.py | 2 +-
  7 files changed, 12 insertions(+), 12 deletions(-)
bisect run success

Best regards, Jean Abou Samra

Jean Abou Samra <jean@abou-samra.fr> writes:

Yes, it would be nice if "git bisect run" can reliably tell that it got a "not found" error and not a "test performed by the script did not pass" and stop at the first failure.

On the other hand, the "./non-existent.sh" script could be part of the tracked contents (i.e. some revisions have it and the working tree has it when they get checked out, some revisions don't and the working tree does not have it), and the user is trying to find the first revision that stopped having a working script in its tree. In such a case, the script that does not exist and the script that fails need to be treated the same way by "git bisect run" as failures.

So... I dunno.

Le 15/07/2020 à 16:55, Junio C Hamano a écrit :

Perhapsa --not-found-as-failure option could help?

Best, Jean Abou Samra

Jean Abou Samra <jean@abou-samra.fr> writes:

Do you mean a new option must be passed if the end-user expects the script to always exist across revisions, or the script is not tracked to begin with? It feels somewhat backwards and the effort by the end-user to always type the option is better spent to make sure there is no typo on the command line.

Besides, we need to take into account that "bisect run" takes an arbitrary shell snippet, e.g. the user may try to switch between two scripts, test-1 and test-2, based on some condition, and wrote this:

  $ git bisect run "if some-condition; then ./test-1; else ./test-3; fi"

but made a typo in the name of test-2. It would be noticed after attempting to run the "if ... fi" as a scriptlet by the returned status from it being 127 (command not found), which is not treated any specially by "git bisect run". So a typo in the script name in the above example cannot be distinguished from this error where the user wanted to run test-1 under some condition, but wanted to declare that the revision is bad if that condition did not hold, i.e.

  $ git bisect run "if some-condition; then ./test-1; else (exit 127); fi"

Even if we somehow could tell these two apart, we cannot pick the first scriptlet apart and guess which substring in it were meant as a filename with a typo.

It may be possible to forbid the general use of exit code 127 and instead reserve it to signal "we know that the test script is so broken (this includes the case where it does not even exist) that using it with 'bisect run' is meaningless; please stop immediately." and nothing else, just like exit code 125 is reserved for "this revision is not testable and cannot say good or bad", but making such a change retroactively is not to be done very lightly.

So, again, I dunno.

 git-bisect.sh | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

Junio C Hamano <gitster@pobox.com> writes:

I forgot to follow up on this part.

It does not change the conclusion that it needs to be done carefully if we chose to retroactivelyreserve return code 127 for our own use, but such a backward incompatible change can easily be worked around if users relied on the current behaviour that a missing (tracked) script would mark the revision "bad" without being a fatal error in "git bisect run". Instead of

    git bisect run "./tracked-script"

they can just do

    git bisect run "test -f ./tracked-script && ./tracked-script"

and the problem is solved ;-)