git ls-files wildcard behavior considered harmful

I was very surprised to learn the other day that git ls-files 'foo*' will expand wildcards (including character classes), in the absence of expansion by the shell. (git version 2.1.4)

joey@darkstar:~/tmp/aaa>git ls-files 'foo*bar' foo*bar foobazbar joey@darkstar:~/tmp/aaa>git ls-files '[abc]' [abc] a b

As far as I can see this behavior is not documented on the man page, except for a tiny mention in the --with-tree documentation, where it says "<file> (i.e. path pattern)".

Since I wanted to avoid this wildcard expension, I tried slash-escaping the wildcard characters. This works:

joey@darkstar:~/tmp/aaa>git ls-files 'foo\*bar' foo*bar joey@darkstar:~/tmp/aaa>git ls-files '\[abc\]' [abc]

But, there is a weird behavior here with subdirectories. While normally ls-files would recurse, slash-escaped wildcard characters in the directory name prevent recursion.

joey@darkstar:~/tmp/aaa>git ls-files 'foo[d]' foo[d]/subfile food joey@darkstar:~/tmp/aaa>git ls-files 'foo\[d\]' joey@darkstar:~/tmp/aaa>

The above example shows a case where it's impossible to get ls-files to only list files in a directory and not other files that match the wildcard. This seems like it must be a bug, and it means it's impossible to reliably work around the wildcard expansion behavior.

I suspect that this wildcard expansion behavior is useful somewhere. But from the perspective of using ls-files as plumbing, where you want to get out some subset of what was put in, it's not nice.

On Tue, Mar 31, 2015 at 6:04 AM, Joey Hess <id@joeyh.name> wrote:

You can do "git --literal-pathspecs ls-files ..." or set GIT_LITERAL_PATHSPECS.

Duy Nguyen wrote:

Thanks! --literal-pathspecs does allow getting around this.

Now I'm wondering what other parts of plumbing might be doing globbing that I did not anticipate. Maybe I should set the environment variable so I don't need to worry about it..

On Mon, Mar 30, 2015 at 07:36:14PM -0400, Joey Hess wrote:

Pretty much everything takes magic patterns[1]. Pathspecs given to diff, path limiters for pruning, etc. If you are scripting with raw filenames (e.g., feeding the filenames out of another tool), I recommend setting GIT_LITERAL_PATHSPECS everywhere. That's what we do on the server side at GitHub (and is the reason I implemented --literal-pathspecs in the first place).

-Peff

[1] Note that globbing is just part of this. Names starting with colon
    are also magical, e.g. ":/foo", or even ":(literal)foo".  These are
    documented in the "pathspec" definition of "git help glossary".

On Tue, Mar 31, 2015 at 6:36 AM, Joey Hess <id@joeyh.name> wrote:

We're just a bit sloppy in documentation. ls-files learned about wildcards since 56fc510 ([PATCH] git-ls-files: generalized pathspecs - 2005-08-21). Two months later, git-ls-files.txt is updated to say "<file>" (which is not entirely correct even if we ignore wildcards, ls-files can take directories as well). With latest git, I think only git-mv and git-read-tree go with plain paths. It's probably best to set GIT_LITERAL_PATHSPECS for scripting like Jeff said.

Hi Joey,

Joey Hess wrote:

Does 'git --noglob-pathspecs' help?

Curious, Jonathan