How do I qualify paths in the .gitignore file w.r.t. the repo root directory?

Say I have these files and directories [2]:

  /home/smart_tator/misc_files/.gitignore
  /home/smart_tator/misc_files/foo/
  /home/smart_tator/misc_files/bar/
  /home/smart_tator/misc_files/bar/baz/foo/
  /home/smart_tator/misc_files/bar/baz/real/

then I do:

  cd /home/smart_tator/misc_files/; git init

and say I have this line in that .gitignore file:

  foo/

And then I naively execute:

  git add bar/

then the bar/baz/real/ is added, but these are dutifully ignored:

  /home/smart_tator/misc_files/foo/
  /home/smart_tator/misc_files/bar/baz/foo/

But consider in my real repo, I have thousands of files, versus the Tinker Toy example shown above. And consider that I don't know about that bar/baz/foo/ exists ahead of time, because I'm not the only developer checking in content to the repo that might contain precious "foo/"'s to keep. I can't move my top level foo/, as I have tools that rely upon that foo/ being in its place.

That means that we can't solve this problem efficiently/effectively with the negation operator in the .gitignore file:

  foo/
  !bar/baz/foo/

because sooner or later someone is going to get surprised as to why their foo/ isn't being added in some other subdirectory, and they will "fix it" by committing a change where they have removed the foo/ from the .gitignore file, thus causing the top level foo/ to show up as candidates for addition in git status output.

Is there some way to express that the foo/ that is to be ignored is always the one in /home/smart_tator/misc_files/ directory, but all other foo/ subdirectories in any other directory under consideration should still continue to participate in adds and merges, all without having to "over-express" the exceptions with negation operators?

Maybe the imaginary .gitignore syntax I am thinking of would be one of the following (most of these are just silly/fun):

  </>foo/  # <-- Huh?
  <top>foo/  # <-- What the ...?
  //foo/  # <-- Smells like Perforce or Windows UNC paths
  /foo/ # <-- No! Matches UNIX root filesystem directory paths!
  >foo/  # <-- You can't have a > character in DOS or Unix paths, can you?
  $root/foo/  # <-- I like this syntax the best [1]

Thanks, bg

[1] Maybe there are other "variables" besides $root that might be
    useful to be added in the future, like $HOME.
[2] By "repo root directory", I mean whatever "$GIT_DIR/.." resolves to.

Brent Goodrick <bgoodr@gmail.com> writes:

I think you are looking for "/foo/".  From Documentation/gitignore.txt:

 - If the pattern ends with a slash, it is removed for the
   purpose of the following description, but it would only find
   a match with a directory.  In other words, `foo/` will match a
   directory `foo` and paths underneath it, but will not match a
   regular file or a symbolic link `foo` (this is consistent
   with the way how pathspec works in general in git).

With this rule, (1) the trailing slash in your "foo/" tells git to match only with directories, but (2) it behaves as if you said "foo" for all the other rules.

With "/foo/", you tell git to match only with a directory, and it is as if you said "/foo".

 - If the pattern does not contain a slash '/', git treats it as
   a shell glob pattern and checks for a match against the
   pathname without leading directories.

Your "foo/" now behaves the same way as "foo" behaves. You are telling git to match directory foo anywhere in the tree. "/foo/" (now behaving the same way as "/foo") does not satisfy this criteria so we would skip this rule.

 - Otherwise, git treats the pattern as a shell glob suitable
   for consumption by fnmatch(3) with the FNM_PATHNAME flag:
   wildcards in the pattern will not match a / in the pathname.
   For example, "Documentation/\*.html" matches
   "Documentation/git.html" but not
   "Documentation/ppc/ppc.html".  A leading slash matches the
   beginning of the pathname; for example, "/*.c" matches
   "cat-file.c" but not "mozilla-sha1/sha1.c".

Your "foo/" does not survive to this rule, but "/foo/" does. It now behaves as "/foo" and its leading slash makes it match the beginning.

On 2009-02-24, Junio C Hamano <gitster@pobox.com> wrote:

[lots of very clear and detailed explanation snipped for brevity...]

I'd been sort of struggling with the part of 'man gitignore' that describes the rules for the exclusion patterns; it just didn't seem as clear as it could have been. It's very accurate, but I (and I noticed a few others on irc) had to read very carefully to do anything moderately complex.

A few days ago, 'doener' (Björn Steinbrink) came up with some much simpler rules that said the same thing, and -- building on the insight that his rules gave me -- I came up with these:

----->8-----

Note that rule 1 merely *modifies* rules 2 and 3, it does not supercede or preclude them.

1.  If you pattern ends with a slash, it matches only
    directories (and their contents)
2.  If there is no slash otherwise, it matches that name, at
    any depth in the tree
3.  If there is a slash anywhere else, it matches that name,
    relative to the .gitignore (or $GIT_WORK_TREE if the
    pattern is from one of the other pattern sources like
    `.git/info/exclude` etc)

The wildcards (`*` and `?`) do not match slashes, but otherwise the patterns are normal shell globs as defined by fnmatch(3) with the FNM_PATHNAME flag set.

----->8-----

Those rules are meant to clarify the following lines from the gitignore man page:

 - If the pattern ends with a slash, it is removed for the
   purpose of the following description, but it would only find
   a match with a directory.  In other words, `foo/` will match a
   directory `foo` and paths underneath it, but will not match a
   regular file or a symbolic link `foo` (this is consistent
   with the way how pathspec works in general in git).

 - If the pattern does not contain a slash '/', git treats it as
   a shell glob pattern and checks for a match against the
   pathname without leading directories.

 - Otherwise, git treats the pattern as a shell glob suitable
   for consumption by fnmatch(3) with the FNM_PATHNAME flag:
   wildcards in the pattern will not match a / in the pathname.

Sitaram Chamarty <sitaramc@gmail.com> writes:

The existing documentation has an unfortunate history behind it and the "if it ends with a slash it matches only directories" was bolted on as an afterthought; see d6b8fc3 (gitignore(5): Allow "foo/" in ignore list to match directory "foo", 2008-01-31).

Nicely written, except that as a non-native speaker I fear "otherwise" and "anywhere else" _might_ leave ambiguity for a pattern that has slash only at the end [*1*], but I dunno. It certainly is much better than what I wrote in the current documentation.

Please send it in a patch form (possibly addressing my ambiguity concern if it is real for other people) with a one-liner log message that says "The existing documentation is unreadable even though it may be precise", and I'll apply.

Thanks.

[Footnote]

*1* That is why I wrote "it is removed for the purpose of..." in the description, even though I do agree that paragraph is hard to read. Perhaps I should have said "it only matches directories. The slash at the end is ignored for the purpose of the following rules." which probably would have flowed a bit more naturally.

On 2009-02-24, Junio C Hamano <gitster@pobox.com> wrote:

Thank you for the encouragement! (And to doener as well of course for the inspiration)

I will send in a suitably amended patch with in the next day or so -- I'm half asleep right now, :-(

Regards,

Sitaram

On 2009-02-24, Junio C Hamano <gitster@pobox.com> wrote:

I couldn't think of an easy way to clear that up without making it far more verbose.

The ambiguity is partly because we're overloading the slash to control both "what matches" (only a directory, versus directory / file / symlink) and "where it matches" (anchored at the directory in which the current .gitignore is found or work tree, versus at any depth underneath).

So I came up with this (see below). It keeps the "what" and the "where" clearly separate, so now the "otherwise" applies to only one preceding clause, and there is no "anywhere else".

It's a somewhat larger change, replacing all 6 bullets and the line preceding them. I think it looks nicer but since I wrote it, I can't vote ;-)

Please tell me what you think. If you like it, I'll send in this patch. If you prefer the previous one, I'll send that in.

[I've also changed $GIT_WORK_TREE in my previous attempt to 'root of the working tree' because that variable is not normally set, and I don't want to imply that it does]

----->8-----

The files containing the patterns have the following format:

 - A blank line matches no files, so it can serve as a
   separator for readability.

 - A line starting with # serves as a comment.

This is _how_ the patterns match:

 - The wildcards (`*` and `?`) do not match slashes, but
   otherwise the patterns are normal shell globs as defined
   by fnmatch(3) with the FNM_PATHNAME flag set.

 - An optional prefix '!' negates the pattern; any matching
   file excluded by a previous pattern will become included
   again.  If a negated pattern matches, this will override
   lower precedence patterns sources.

This is _what_ the patterns match:

 - If the pattern ends with a slash, it matches only
   directories (and their contents), otherwise it matches
   regular files and symlinks also.

This is _where_ the patterns match (a trailing slash is ignored for these rules):

 - If there is a slash at the start or within the pattern,
   it matches paths relative to the .gitignore file in which
   the pattern is found, or to the root of the working tree
   if the pattern is from one of the other pattern sources
   (i.e., `.git/info/exclude`, `core.excludesfile`)

 - Otherwise, it matches a path at any depth in the tree

----->8-----

Regards,

Sitaram

Sitaram Chamarty <sitaramc@gmail.com> writes:

I had to read this twice and run "man 3 fnmatch" to clear my head.

 - In normal shell globs, wildcards '*' and '?' do not match slashes;

 - fnmatch(3) with FNM_PATHNAME implements the normal shell globs;

 - wildcards do not match slashes in gitignore either.

Given these three, I am very confused why you say "but otherwise". I would understand it if it were:

    The patterns are treated as normal shell globs defined by fnmatch(3) with
    FNM_PATHNAME; in other words, the wildcards (`*` and `?`) do not match
    slashes.

'!' is not part of _how_ the patterns match. It is _what happens_ when a pattern marked as such matches (meaning, the syntax for a line in gitignore file is "an optional '!' followed by a pattern").

    An optional prefix '!' is not a part of the pattern and it does not
    affect the match.  When a path matches such a pattern, instead of
    being ignored, it is unignored.

It would be good to clarify that '!' is not part of the pattern, as I'd like to take J6t's patch that says gitattributes uses the same pattern as gitignore uses.

Do we want "(and their contents)" here? Once a directory is ignored like this, none of its contents, including .gitignore file in it, is examined because we do not even descend into it.

"at the start or within but not at the end of the pattern"?

On 2009-02-25, Junio C Hamano <gitster@pobox.com> wrote:

I'll use that.

This confusing statement existed in the previous (shorter) version also. My fault; I use [[ path == patt ]] far more often than plain globs so I had a thinko. (Perversely, that one does *not* use FNM_PATHNAME... go figure!)

I can use this. Can we keep it in the same section, despite being technically not a '_how_'? It fits the other sections even less, and the sectioning is the main thing in all this.

Do we not want to specify that we don't descend? The original text does say '...will match a directory foo and paths underneath it'.

Isn't that confusing? 'if there is a slash ... not at the end of the pattern' can easily sound like "there *should not* be a trailing slash", which is quite different from "we don't care if there *is* a trailing slash; it's the *other* slashes that matter here".

And it'll _seem to_ contradict what we say, just above, that a trailing slash is ignored for these rules.

Sitaram Chamarty <sitaramc@gmail.com> writes:

I thought making the text easier to follow was the main thing.

Sectioning could be part of the solution, but if we find that the boundary between sections are blurry, or if there are too many sections compared to the number of rules, perhaps dividing them into sections and giving each a separate section header may make them even harder to follow.

I am actually very tempted to say that the correct description of the gitignore language is:

    - an optional ! sign whose meaning is "unignore paths that matches
      this pattern, instead of ignoring them"; followed by

    - an optional / sign whose meaning is "a match with this pattern must
      be made at this directory and not in its subdirectories"; followed
      by

    - a pattern that never begins nor ends with a slash whose meaning is
      "this is a shell glob pattern to test paths against"; followed by

    - an optional / sign whose meaning is "this pattern matches only with
      a directory".

We'll need to tweak the language a bit in J6t's patch that talks about the gitattributes pattern if we go this route, though. The attribute system uses the latter three that specify how a match is made, but does not use the first one that specifies what happens once a match is found, because the latter is done by the attributes part that follows the pattern in the gitattributes file.

Ok. If we unignore a directory that does not mean all paths inside it are now unignored --- they are still subject to .gitignore rules read from it and its subdirectories. So "will match it and paths inside it" is correct but "will ignore it and paths inside it" is not.

You are absolutely right.  Please scratch my comment on this item.

On 2009.02.25 13:25:24 -0800, Junio C Hamano wrote:

I don't think that's good enough. The case in which we ignore directories is impressively hard to describe correctly. Consider this:

.gitignore # Contains foo foo/.gitignore # Contains !bar foo/bar

The current man page says that for "foo/bar" foo/.gitignore has the highest precedence, overriding the patterns from .gitignore. So without further information, one could think that while foo/bar is matched by the "foo" pattern, this is invalidated by the "!bar" rule and thus foo/bar would not be ignored. But of course we don't bother looking into foo at all, so the foo/.gitignore has no effect.

And it gets more interesting:

.gitignore # Contains "foo" and "!foo/bar" foo/bar

Even in this case, foo/bar is ignored. Because again, we don't look into foo at all, so we never even see foo/bar and thus we won't notice that it is to be unignored. So saying "match it and paths inside it" is misleading, because we'll never even see any path inside it that could match, and it might lead to failing gitignore setups like the above.

So maybe instead of "... and paths inside it", we could have something like this:

    If a directory is ignored, git won't look into it at all when
    searching for untracked files. This means that all paths inside it
    are implicitly ignored and that you cannot unignore these paths.

Björn

On 2009-02-25, Junio C Hamano <gitster@pobox.com> wrote:

I meant the sectioning was the main change to make the text easier to follow, since the words are mostly the same otherwise.

I'm ambivalent on this, so I appreciate the tie-breaker.

I see you've used 'pattern' and 'sign' to break the overloading of '/'.

I wish[1]. But in reality, a slash 'inside' anchors the match the same as a leading slash does.

Boy this is tough :-) and I'm almost tempted to relook at my first attempt, where your only concerns were the words 'otherwise' and 'anywhere else' for non-native speakers.

I'll think about this some more and get back to you.

Regards,

Sitaram

[1] Would have been much simpler, plus you'd be able to specify a non-anchored pattern that contains a slash, which you currently cannot.

On 2009-02-26, Sitaram Chamarty <sitaramc@gmail.com> wrote:

How about this:

----8<----

    - an optional leading ! symbol meaning "unignore paths
      that match this pattern, instead of ignoring them"

    - an optional trailing / symbol meaning "this pattern
      matches only with a directory (i.e., files and
      symlinks won't match)"

    - the above two symbols (if present) are then removed.
      What remains is treated as a normal shell glob
      pattern, with the additional restriction that if the
      pattern still contains a slash, it matches only at the
      current directory and not in its subdirectories

----8<----

I'm not going into the 'descend' thing; based on how the email from Björn Steinbrink plays out, we can describe that later. Same for 'current directory', which stands for the root of the work-tree if the pattern came from .git/info/exclude or a core.excludesfile; this also can be added as a footnote.

Sitaram Chamarty <sitaramc@gmail.com> writes:

Sure, but then you are not stripping the leading / from the pattern but you do not use it for the purpose of matching, right?

I think your original before I wondered about the ambiguity is the best rewrite so far.

Thanks James and Junio. I'll try the leading slash. I assumed that the beginning slash means what it means for other tools (meaning a fully-qualified path to some file somewhere on the file system), but apparently such is not the case with git (and I conclude from this that it is actually not possible to store fully qualified paths _unless_ the $GIT_DIR is right under the root filesystem). This isn't a problem for me, and probably not for most folks, but it was confusing to me the way it was written in the man page.

Brent

On Mon, Feb 23, 2009 at 11:06 PM, Junio C Hamano <gitster@pobox.com> wrote:

On 2009-02-25, Brent Goodrick <bgoodr@gmail.com> wrote:

I suspect it's not _useful_ also, because it would prevent you from relocating your git work tree to some other path while getting predictable results. I'm unable to think of a use-case where that would be desirable, but I admit I haven't thought about it too much.

If you could look at either of my two submissions in this thread, and comment on whether they would have helped you in this regard, I'd appreciate it very much.

I prefer the second one that I sent, but don't let that prejudice you ;-)