Re: bug?: git grep HEAD with exclude in pathspec not taken into account

>
> Hi,
>
> I observed an unexpected behavior while using git grep with both git
> 2.19.1 and 2.14.3. Here is how to reproduce it:
>
> > git init
> Initialized empty Git repository in /private/tmp/hello/.git/
> > echo foo > fileA
> > echo 'foo is false+' > fileB
> > git add fileA
> > git commit -m fileA
> [master (root-commit) f2c83e7] fileA
>  1 file changed, 1 insertion(+)
>  create mode 100644 fileA
> > git add fileB
> > git commit -m fileB
> [master e35df26] fileB
>  1 file changed, 1 insertion(+)
>  create mode 100644 fileB
> > git --no-pager grep foo HEAD -- ':!fileA'
> HEAD:fileB:foo is false+
> > git --no-pager grep foo HEAD -- ':!fileB'
> HEAD:fileA:foo
> HEAD:fileB:foo is false+
>
> In the last command, fileB appears in grep results but it should have
> been excluded.
>
> If the HEAD parameter is removed, it works as expected:

>
> On Wed, Oct 24, 2018 at 4:55 PM Christophe Bliard
> <christophe.bliard@trux.info> wrote:
> >
> > Hi,
> >
> > I observed an unexpected behavior while using git grep with both git
> > 2.19.1 and 2.14.3. Here is how to reproduce it:
> >
> > > git init
> > Initialized empty Git repository in /private/tmp/hello/.git/
> > > echo foo > fileA
> > > echo 'foo is false+' > fileB
> > > git add fileA
> > > git commit -m fileA
> > [master (root-commit) f2c83e7] fileA
> >  1 file changed, 1 insertion(+)
> >  create mode 100644 fileA
> > > git add fileB
> > > git commit -m fileB
> > [master e35df26] fileB
> >  1 file changed, 1 insertion(+)
> >  create mode 100644 fileB
> > > git --no-pager grep foo HEAD -- ':!fileA'
> > HEAD:fileB:foo is false+
> > > git --no-pager grep foo HEAD -- ':!fileB'
> > HEAD:fileA:foo
> > HEAD:fileB:foo is false+
> >
> > In the last command, fileB appears in grep results but it should have
> > been excluded.
> >
> > If the HEAD parameter is removed, it works as expected:
>
> It's probably a bug. We have different code paths for matching things
> with or without HEAD, roughly speaking. I'll look into to this more on
> the weekend, unless somebody (is welcome to) beats me first.
>
> Another thing that might also be a problem is, negative patterns are
> supposed to exclude stuff from "something" but you don't specify that
> "something" (i.e. positive patterns) in your grep commands above. If
> "grep foo HEAD -- :/ ':!fileB'" works, then you probably just run into
> some undefined corner case.
> --
> Duy

>
> Hi,
>
> I observed an unexpected behavior while using git grep with both git
> 2.19.1 and 2.14.3.

> Rules 8 and 18 are now updated to be less eager. We conclude that the
> current entry is positively matched and included. But we say nothing
> about remaining entries. tree_entry_interesting() will be called again
> for those entries where we will determine entries individually.
>
> Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
> ---
> diff --git a/t/t6132-pathspec-exclude.sh b/t/t6132-pathspec-exclude.sh
> @@ -194,4 +194,21 @@ test_expect_success 'multiple exclusions' '
> +test_expect_success 't_e_i() exclude case #8' '
> +       git init case8 &&
> +       (
> +               cd case8 &&
> +               echo file >file1 &&
> +               echo file >file2 &&
> +               git add . &&

> +               git commit -m twofiles &&
> +               git grep -l file HEAD :^file2 >actual &&
> +               echo HEAD:file1 >expected &&
> +               test_cmp expected actual &&
> +               git grep -l file HEAD :^file1 >actual &&
> +               echo HEAD:file2 >expected &&
> +               test_cmp expected actual
> +       )
> +'

> On Sat, Nov 3, 2018 at 11:31 AM Nguyễn Thái Ngọc Duy <pclouds@gmail.com> wrote:
> > +test_expect_success 't_e_i() exclude case #8' '
> > +       git init case8 &&
> > +       (
> > +               cd case8 &&
> > +               echo file >file1 &&
> > +               echo file >file2 &&
> > +               git add . &&
>
> Won't this loose git-add invocation end up adding all the junk files
> from earlier tests? One might have expected to see the more restricted
> invocation:
>     git add file1 file2 &&
> to make it easier to reason about the test and not worry about someone
> later inserting tests above this one which might interfere with it.

>
> On Sat, Nov 3, 2018 at 8:25 PM Eric Sunshine <sunshine@sunshineco.com> wrote:
> > On Sat, Nov 3, 2018 at 11:31 AM Nguyễn Thái Ngọc Duy <pclouds@gmail.com> wrote:
> > > +test_expect_success 't_e_i() exclude case #8' '
> > > +       git init case8 &&
> > > +       (
> > > +               cd case8 &&
> > > +               echo file >file1 &&
> > > +               echo file >file2 &&
> > > +               git add . &&
> >
> > Won't this loose git-add invocation end up adding all the junk files
> > from earlier tests? One might have expected to see the more restricted
> > invocation:
> >     git add file1 file2 &&
> > to make it easier to reason about the test and not worry about someone
> > later inserting tests above this one which might interfere with it.
>
> Upon reflection, there shouldn't be any junk files since this test is
> creating a new repository and "file1" and "file2" are the only files
> present. Apparently, I wasn't paying close enough attention when I
> made the earlier observation.

> tree_entry_interesting() is used for matching pathspec on a tree. The
> interesting thing about this function is that, because the tree
> entries are known to be sorted, this function can return more than
> just "yes, matched" and "no, not matched". It can also say "yes, this
> entry is matched and so is the remaining entries in the tree".
>
> This is where I made a mistake when matching exclude pathspec. For
> exclude pathspec, we do matching twice, one with positive patterns and
> one with negative ones, then a rule table is applied to determine the
> final "include or exclude" result. Note that "matched" does not
> necessarily mean include. For negative patterns, "matched" means
> exclude.
>
> This particular rule is too eager to include everything. Rule 8 says
> that "if all entries are positively matched" and the current entry is
> not negatively matched (i.e. not excluded), then all entries are
> positively matched and therefore included. But this is not true. If
> the _current_ entry is not negatively matched, it does not mean the
> next one will not be and we cannot conclude right away that all
> remaining entries are positively matched and can be included.
>
> Rules 8 and 18 are now updated to be less eager. We conclude that the
> current entry is positively matched and included. But we say nothing
> about remaining entries. tree_entry_interesting() will be called again
> for those entries where we will determine entries individually.

> Reported-by: Christophe Bliard <christophe.bliard@trux.info>
> Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
> ---
>  v2 fixes the too broad "git add ." in the test
>
>  t/t6132-pathspec-exclude.sh | 17 +++++++++++++++++
>  tree-walk.c                 | 11 ++++++++---
>  2 files changed, 25 insertions(+), 3 deletions(-)
>
> diff --git a/t/t6132-pathspec-exclude.sh b/t/t6132-pathspec-exclude.sh
> index eb829fce97..2462b19ddd 100755
> --- a/t/t6132-pathspec-exclude.sh
> +++ b/t/t6132-pathspec-exclude.sh
> @@ -194,4 +194,21 @@ test_expect_success 'multiple exclusions' '
>  	test_cmp expect actual
>  '
>  
> +test_expect_success 't_e_i() exclude case #8' '
> +	git init case8 &&
> +	(
> +		cd case8 &&
> +		echo file >file1 &&
> +		echo file >file2 &&
> +		git add file1 file2 &&
> +		git commit -m twofiles &&
> +		git grep -l file HEAD :^file2 >actual &&
> +		echo HEAD:file1 >expected &&
> +		test_cmp expected actual &&
> +		git grep -l file HEAD :^file1 >actual &&
> +		echo HEAD:file2 >expected &&
> +		test_cmp expected actual
> +	)
> +'
> +
>  test_done
> diff --git a/tree-walk.c b/tree-walk.c
> index 77b37f36fa..79bafbd1a2 100644
> --- a/tree-walk.c
> +++ b/tree-walk.c
> @@ -1107,7 +1107,7 @@ enum interesting tree_entry_interesting(const struct name_entry *entry,
>  	 *   5  |  file |    1     |    1     |   0
>  	 *   6  |  file |    1     |    2     |   0
>  	 *   7  |  file |    2     |   -1     |   2
> -	 *   8  |  file |    2     |    0     |   2
> +	 *   8  |  file |    2     |    0     |   1
>  	 *   9  |  file |    2     |    1     |   0
>  	 *  10  |  file |    2     |    2     |  -1
>  	 * -----+-------+----------+----------+-------
> @@ -1118,7 +1118,7 @@ enum interesting tree_entry_interesting(const struct name_entry *entry,
>  	 *  15  |  dir  |    1     |    1     |   1 (*)
>  	 *  16  |  dir  |    1     |    2     |   0
>  	 *  17  |  dir  |    2     |   -1     |   2
> -	 *  18  |  dir  |    2     |    0     |   2
> +	 *  18  |  dir  |    2     |    0     |   1
>  	 *  19  |  dir  |    2     |    1     |   1 (*)
>  	 *  20  |  dir  |    2     |    2     |  -1
>  	 *
> @@ -1134,7 +1134,12 @@ enum interesting tree_entry_interesting(const struct name_entry *entry,
>  
>  	negative = do_match(entry, base, base_offset, ps, 1);
>  
> -	/* #3, #4, #7, #8, #13, #14, #17, #18 */
> +	/* #8, #18 */
> +	if (positive == all_entries_interesting &&
> +	    negative == entry_not_interesting)
> +		return entry_interesting;
> +
> +	/* #3, #4, #7, #13, #14, #17 */
>  	if (negative <= entry_not_interesting)
>  		return positive;