Gitignore matching "git add" wildcard prevents operation

It seems that that there is a bug related to how Git handles the add command with wildcards if the wildcard also matches a file that is ignored.

E.g (I activate the *.[ao] rule when editing .git/info/exclude):

mep@Blackbird:~$ cd /tmp mep@Blackbird:/tmp$ mkdir git mep@Blackbird:/tmp$ cd git mep@Blackbird:/tmp/git$ git init Initialized empty Git repository in /tmp/git/.git/ mep@Blackbird:/tmp/git$ nano -w .git/info/exclude mep@Blackbird:/tmp/git$ touch a.o a.c a.h mep@Blackbird:/tmp/git$ git add a.* The following paths are ignored by one of your .gitignore files: a.o Use -f if you really want to add them. fatal: no files added mep@Blackbird:/tmp/git$ git status # On branch master # # Initial commit # # Untracked files: # (use "git add <file>..." to include in what will be committed) # # a.c # a.h # a.o nothing added to commit but untracked files present (use "git add" to track)

Furthermore:

mep@Blackbird:/tmp/git$ git add .
mep@Blackbird:/tmp/git$ git commit -m "foo"
[master (root-commit) 43da825] foo
 0 files changed, 0 insertions(+), 0 deletions(-)
 create mode 100644 a.c
 create mode 100644 a.h
mep@Blackbird:/tmp/git$ nano a.c
mep@Blackbird:/tmp/git$ nano a.h
mep@Blackbird:/tmp/git$ git add a.*
The following paths are ignored by one of your .gitignore files:
a.o
Use -f if you really want to add them.
fatal: no files added
mep@Blackbird:/tmp/git$ git status
# On branch master
# Changed but not updated:
#   (use "git add <file>..." to update what will be committed)
#   (use "git checkout -- <file>..." to discard changes in working
directory)
#
#       modified:   a.c
#       modified:   a.h
#
no changes added to commit (use "git add" and/or "git commit -a")

On Thu, Jan 14, 2010 at 08:18:40PM +0200, Marko Poutiainen wrote:

It's not a bug. That is working as designed. If "git add" encounters a problem with any of the files you gave it on the command line (and remember, the shell expands the wildcard, so git literally sees the three files on the command line) then it aborts the entire operation.

Now whether that is useful behavior in this case, I don't know. If you were truly manually specifying the files, then it would probably not be a big deal to simply remove the offending file. But there is no way to use the shell wildcard and get the behavior you want (not even a "git add --really-ignore-my-ignores a.*").

I don't think it would be right to silently ignore them, because it may be the sign of an error. It could be downgraded to a warning, like:

  $ git add *.a
  warning: a.o is ignored, not adding

The downside would be that somebody who really _did_ want to add such a file but forgot to use "-f" is going to be less likely to notice it, as the command does not barf. So maybe such behavior should be triggered with a command-line option. I dunno.

-Peff

Jeff King <peff@peff.net> writes:

Perhaps you want to run

	$ git add 'a.*'

Notice that the wildcard is protected from the shell.

On Thu, Jan 14, 2010 at 12:21:34PM -0800, Junio C Hamano wrote:

Ugh. You're right that it does work, but I don't expect users to make the intuitive jump from the OP's problem to this solution (I certainly didn't). In particular:

  1. Most programs don't take their own globs. Without knowing that git
     can do so, there is no reason to discover it in this instance. I
     can see searching the manpage for options, but not for a discussion
     of globbing behavior.

  2. They would have to know that using a git-glob will magically change
     the error-checking behavior.

-Peff

The 14/01/10, Jeff King wrote:

Not sure. This isn't a Git-particular issue.

Users may hit this with a lot of other unix tools (sed, grep, find,
etc). So, we can expect either
  they already know the issue;
or
  they are discovering it using Git.

Most of the tools I talk about do have a manual section about globbing. Users could learn globs with Git too and expect the same behaviour somewhere else.

On Fri, Jan 15, 2010 at 04:34:19PM +0100, Nicolas Sebrecht wrote:

I don't understand what you mean. How does "sed" do its own globbing of the command line?

Sure. git-add(1) talks about globbing, too, and it even has a sentence that explains exactly what is happening in Marko's case:

   The git add command will not add ignored files by default. If any
   ignored files were explicitly specified on the command line, git add
   will fail with a list of ignored files. Ignored files reached by
   directory recursion or filename globbing performed by Git (quote your
   globs before the shell) will be silently ignored.

My complaint was more that as a user, I am not likely to see this problem and think "I'll bet git-specific globbing can solve it." And when I look in the manual, I am more likely to look for a command-line option that helps me rather than to read all of the text (though the fact that we specifically mention "ignore" in the same paragraph means that at least a user searching for "ignore" in the man page will come up with this paragraph).

-Peff

The 15/01/10, Jeff King wrote:

Well, we are in the same dilemma as the other tools. The internal globbing rules are explained in the related man page.

Oops, I was missing that; thanks.

Yes. My point is that we are not talking about a Git specific issue. What you're raising here is true whatever the command is. So, as long as we clearly explain how 'git add' works, we are fine. Don't we?

True. All I can see is to improve the man page with a dedicated section "Globbing" instead of loosing it in a "random" place.

On Fri, Jan 15, 2010 at 05:11:07PM +0100, Nicolas Sebrecht wrote:

Maybe I wasn't clear: to my knowledge, "sed" does not do any globbing itself. How is this the same situation?

Of course other commands like sed will be fed the expansion of a shell glob, and there may be times when you want to feed a subset of an expansion. But that is not my complaint; my complaint was mainly that git's solution to this is not easily discoverable by an uninformed user. Most other commands don't even have a solution (you would have to solve it in the shell to pass the desired expansion to the program).

I don't think that would help. The problem is that the user knows they have an issue with ignored files. The solution is custom globbing, but they don't know that. So making globbing more prominent doesn't help, since they will be looking for ignores. You would need to have an "ignore" section that mentions globbing.

To be clear: I do not have an actual solution, and my initial message was mostly just grumbling. We _do_ mention globbing and ignores in the same paragraph, as I quoted earlier. So that is probably enough for a diligent user to come up with the solution, or at least enough that trying to improve on it will have diminishing returns.

You could even argue that I was not being such a diligent user in my initial response. :)

-Peff

The 15/01/10, Jeff King wrote:

"sed" does /pattern matching/ which is the meaning I give to /globbing/.

Sure, "sed" doesn't do /shell globbing/ but is fed from it. We obviously agree here. :-)

I agree in that it is not easily discoverable. But I don't think Git's solution is worse than any other tool. The "discoverabiliy" looks equal to any other tool for me. As you said, to do a good job we should be able to know what was the original command-line, which isn't possible.

Nicolas Sebrecht <nicolas.s.dev@gmx.fr> writes:

Actually it is. The thing is, git subcommands do *not* take individual filenames in general from their command line. They _always_ take pathspecs, find files in the work tree that match the pathspecs (but excludes ignored ones), and work on them [*1*].

Other commands usually don't do that.

So "git add ." and "git add 'a.*'" behave exactly the same way. Both find files that match their respective pathspecs. The former knows that "." matches "all files that exist in the cwd (excluding ignored ones)" and the latter knows that "a.*" matches "all files that fnmatch(3) 'a.*' glob (excluding ignored ones)". Hence you will not get "don't try to add a file that you said you want to ignore; use -f if you mean it".

[Footnote]

*1* There are exceptions, such as parameters to the "--file" option, and "git blame".

Jeff King wrote:

Well, it then the interface just isn't consistent, because if that's the reason, then why does

$git add .

then work in this case? . is still just a type of wildcard, so it fail in a similar way, shouldn't it?

Your idea of giving a warning is good as is the command-line option. In any case, in my opinion this feature can make some operations more complex than they should be. For instance, in my example I would have to separately do git add to both files. What if there are more than two files to add and I get this error every time I try to add the files?

The 14/01/10, Marko Poutiainen wrote:

But it is...

...because the dot isn't a wildcard. It's a path.

So, we expect 'git add' to recusively work and resolve ignored paths from .gitignore and so.