{"thread":{"id":"38007","subject":".gitignore sub-dir exclusions not overriding '*'","startedAt":"2014-11-19T03:40:11Z","lastAt":"2014-11-20T01:26:42Z","messageCount":7,"participants":["Phil Pennock","Duy Nguyen","Nguyễn Thái Ngọc Duy","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"252177","messageId":"20141119034010.GA18145@tower.spodhuis.org","threadId":"38007","inReplyTo":null,"subject":".gitignore sub-dir exclusions not overriding '*'","fromName":"Phil Pennock","fromEmail":"phil-gitml@phil.spodhuis.org","sentAt":"2014-11-19T03:40:11Z","receivedAt":"2014-11-19T03:40:11Z","isPatch":false,"sender":{"key":"phil-gitml@phil.spodhuis.org","avatar":null},"body":"The behaviour below is seen with both git 1.8.3.2 and git 2.1.3; I am\nnot subscribed to the vger list, please keep me in the CC list.\n\nUse-case: git repo which only holds PGP-encrypted files with a .asc\nextension, no matter which sub-directory they're in, or if in the top\ndirectory.\n\nSimple layout; for demo purposes names starting 'incl' should end up\nincluded, those starting 'excl' should end up excluded, but not based on\nthose prefices: they're success result indicators; so:\n\n    cd wherever\n    git init\n    mkdir foo\n    touch incl.asc excl excl.txt foo/incl.asc foo/excl.txt foo/excl\n    $EDITOR .gitignore\n\nExpected to work as .gitignore in top-level of repo:\n\n    *\n    !**/*.asc\n    !.gitignore\n\nWith that, `git status` ignores the contents of foo/ thusly:\n\n    $ git check-ignore -v foo/incl.asc\n    .gitignore:1:*\tfoo/incl.asc\n\nCommenting out the '*' line and removing the '!' from the second, the\n**/*.asc clearly matches.  The only way I can make this style work is to\nset the first line to '**/*.*' which fails to exclude the plain 'excl'\nfiles (no extension).\n\nIt seems that there's some magic around '*' as the entire final path\ncomponent of a pattern which causes it to match against the entire\ndirectory, and excludes of the directory can not be overriden by matches\nagainst '*.ext' within the directory, even when they come later in the\nsame config file at the same precedence.\n\nThis does not seem to my reading to match the behaviour described by\n`git help gitignore` (checked in both versions of git) and so seems to\nme to be a bug, but if it's a failure of my understanding, please help\nme to understand where I messed up.\n\nThanks,\n-Phil\n"},{"id":"252178","messageId":"CACsJy8BHvucnEAW065OXOe5NBQkxp7+8HJb7XCGYgOa=pkxLVA@mail.gmail.com","threadId":"38007","inReplyTo":"20141119034010.GA18145@tower.spodhuis.org","subject":"Re: .gitignore sub-dir exclusions not overriding '*'","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2014-11-19T09:48:16Z","receivedAt":"2014-11-19T09:48:16Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Wed, Nov 19, 2014 at 10:40 AM, Phil Pennock\n<phil-gitml@phil.spodhuis.org> wrote:\n> Expected to work as .gitignore in top-level of repo:\n>\n>     *\n>     !**/*.asc\n>     !.gitignore\n>\n\ngitignore man page has this \"It is not possible to re-include a file\nif a parent directory of that file is excluded\". In this case,\ndirectory \"foo\" is ignored by \"*\". Although it makes sense for this\nparticular case to re-include something in foo because we can clearly\nsee there are rules to re-include. It's on my todo list, but I don't\nknow when it will be implemented.\n-- \nDuy\n"},{"id":"252179","messageId":"1416393058-5497-1-git-send-email-pclouds@gmail.com","threadId":"38007","inReplyTo":"CACsJy8BHvucnEAW065OXOe5NBQkxp7+8HJb7XCGYgOa=pkxLVA@mail.gmail.com","subject":"[PATCH] dir.c: allow re-include after a dir is excluded in some cases","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-11-19T10:30:58Z","receivedAt":"2014-11-19T10:30:58Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"If we exclude a directory and have no knowledge in advance if there\nwill be any negative rules on that directory, then it makes no sense\nto enter the directory and search for those negative rules. That\ndefeats the purpose of excluding in the first place.\n\nHowever there are cases where we know in advance about such negative\nrules. One of them is when the rule to exclude a dir and negative\nrules on that dir appear on the same .gitignore file. This case be\neasily detected.\n\nNote that I do not support this case\n\n  *\n  !bar\n\nbecause I fear of performance degradation because people may not\nrealize \"!bar\" will force recursing indefinitely. So you need at least\na slash some where to nail your pattern to some directory. \"!/bar\" is\nfine (but pointless), \"bar/*.c\" or \"**/foo\" may be seen more often.\n\nThis is a quick prototype. I know the two statements in add_exclude()\nis not good enough. I should not count the last trailing slash because\nthat one means different thing.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>\n---\n On Wed, Nov 19, 2014 at 4:48 PM, Duy Nguyen <pclouds@gmail.com> wrote:\n > On Wed, Nov 19, 2014 at 10:40 AM, Phil Pennock\n > <phil-gitml@phil.spodhuis.org> wrote:\n >> Expected to work as .gitignore in top-level of repo:\n >>\n >>     *\n >>     !**/*.asc\n >>     !.gitignore\n >>\n >\n > gitignore man page has this \"It is not possible to re-include a file\n > if a parent directory of that file is excluded\". In this case,\n > directory \"foo\" is ignored by \"*\". Although it makes sense for this\n > particular case to re-include something in foo because we can clearly\n > see there are rules to re-include. It's on my todo list, but I don't\n > know when it will be implemented.\n\n It turns out not so hard to do. I just needed a push like your\n report. I'll try to find some time to finish this up, if people don't\n object this idea.\n\n Documentation/gitignore.txt | 13 +++++++++----\n dir.c                       | 27 +++++++++++++++++++++------\n dir.h                       |  2 ++\n 3 files changed, 32 insertions(+), 10 deletions(-)\n\ndiff --git a/Documentation/gitignore.txt b/Documentation/gitignore.txt\nindex 09e82c3..0340c44 100644\n--- a/Documentation/gitignore.txt\n+++ b/Documentation/gitignore.txt\n@@ -82,10 +82,9 @@ PATTERN FORMAT\n \n  - An optional prefix \"`!`\" which negates the pattern; any\n    matching file excluded by a previous pattern will become\n-   included again. It is not possible to re-include a file if a parent\n-   directory of that file is excluded. Git doesn't list excluded\n-   directories for performance reasons, so any patterns on contained\n-   files have no effect, no matter where they are defined.\n+   included again.\n+   It is usually not possible to re-include a file if a parent\n+   directory of that file is excluded. See NOTES for details.\n    Put a backslash (\"`\\`\") in front of the first \"`!`\" for patterns\n    that begin with a literal \"`!`\", for example, \"`\\!important!.txt`\".\n \n@@ -144,6 +143,12 @@ use 'git update-index {litdd}assume-unchanged'.\n To stop tracking a file that is currently tracked, use\n 'git rm --cached'.\n \n+It is usually not possible to re-include a file if a parent directory\n+of that file is excluded because of performance reasons. However, if\n+there are negative rules in the same .gitignore file that contains the\n+rule to ignore a specific directory, and those negative rules contain\n+a slash, then re-inclusion is possible.\n+\n EXAMPLES\n --------\n \ndiff --git a/dir.c b/dir.c\nindex 3f7a025..df6d940 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -443,6 +443,8 @@ void add_exclude(const char *string, const char *base,\n \tint flags;\n \tint nowildcardlen;\n \n+\tif (string[0] == '!' && strchr(string + 1, '/'))\n+\t\tel->flags |= EXC_FLAG_MAY_REINCLUDE;\n \tparse_exclude_pattern(&string, &patternlen, &flags, &nowildcardlen);\n \tif (flags & EXC_FLAG_MUSTBEDIR) {\n \t\tchar *s;\n@@ -710,14 +712,17 @@ static struct exclude *last_exclude_matching_from_list(const char *pathname,\n \t\t\t\t\t\t       struct exclude_list *el)\n {\n \tint i;\n+\tstruct exclude *x;\n+\tconst char *exclude;\n+\tint prefix;\n \n \tif (!el->nr)\n \t\treturn NULL;\t/* undefined */\n \n \tfor (i = el->nr - 1; 0 <= i; i--) {\n-\t\tstruct exclude *x = el->excludes[i];\n-\t\tconst char *exclude = x->pattern;\n-\t\tint prefix = x->nowildcardlen;\n+\t\tx = el->excludes[i];\n+\t\texclude = x->pattern;\n+\t\tprefix = x->nowildcardlen;\n \n \t\tif (x->flags & EXC_FLAG_MUSTBEDIR) {\n \t\t\tif (*dtype == DT_UNKNOWN)\n@@ -731,7 +736,7 @@ static struct exclude *last_exclude_matching_from_list(const char *pathname,\n \t\t\t\t\t   pathlen - (basename - pathname),\n \t\t\t\t\t   exclude, prefix, x->patternlen,\n \t\t\t\t\t   x->flags))\n-\t\t\t\treturn x;\n+\t\t\t\tgoto done;\n \t\t\tcontinue;\n \t\t}\n \n@@ -739,9 +744,19 @@ static struct exclude *last_exclude_matching_from_list(const char *pathname,\n \t\tif (match_pathname(pathname, pathlen,\n \t\t\t\t   x->base, x->baselen ? x->baselen - 1 : 0,\n \t\t\t\t   exclude, prefix, x->patternlen, x->flags))\n-\t\t\treturn x;\n+\t\t\tgoto done;\n+\t}\n+\tx = NULL;\t\t/* undecided */\n+done:\n+\tif (x && !(x->flags & EXC_FLAG_NEGATIVE) &&\n+\t    (el->flags & EXC_FLAG_MAY_REINCLUDE)) {\n+\t\tif (*dtype == DT_UNKNOWN)\n+\t\t\t*dtype = get_dtype(NULL, pathname, pathlen);\n+\t\tif (*dtype == DT_DIR)\n+\t\t\tx = NULL;\n \t}\n-\treturn NULL; /* undecided */\n+\n+\treturn x;\n }\n \n /*\ndiff --git a/dir.h b/dir.h\nindex 6c45e9d..1ace1cf 100644\n--- a/dir.h\n+++ b/dir.h\n@@ -14,6 +14,7 @@ struct dir_entry {\n #define EXC_FLAG_ENDSWITH 4\n #define EXC_FLAG_MUSTBEDIR 8\n #define EXC_FLAG_NEGATIVE 16\n+#define EXC_FLAG_MAY_REINCLUDE 32\n \n struct exclude {\n \t/*\n@@ -46,6 +47,7 @@ struct exclude {\n struct exclude_list {\n \tint nr;\n \tint alloc;\n+\tint flags;\n \n \t/* remember pointer to exclude file contents so we can free() */\n \tchar *filebuf;\n-- \n2.1.0.rc0.78.gc0d8480\n"},{"id":"252191","messageId":"xmqqmw7nrpqh.fsf@gitster.dls.corp.google.com","threadId":"38007","inReplyTo":"1416393058-5497-1-git-send-email-pclouds@gmail.com","subject":"Re: [PATCH] dir.c: allow re-include after a dir is excluded in some cases","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2014-11-19T18:48:22Z","receivedAt":"2014-11-19T18:48:22Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Nguyễn Thái Ngọc Duy  <pclouds@gmail.com> writes:\n\n> diff --git a/Documentation/gitignore.txt b/Documentation/gitignore.txt\n> index 09e82c3..0340c44 100644\n> --- a/Documentation/gitignore.txt\n> +++ b/Documentation/gitignore.txt\n> @@ -82,10 +82,9 @@ PATTERN FORMAT\n>  \n>   - An optional prefix \"`!`\" which negates the pattern; any\n>     matching file excluded by a previous pattern will become\n> -   included again. It is not possible to re-include a file if a parent\n> -   directory of that file is excluded. Git doesn't list excluded\n> -   directories for performance reasons, so any patterns on contained\n> -   files have no effect, no matter where they are defined.\n> +   included again.\n> +   It is usually not possible to re-include a file if a parent\n> +   directory of that file is excluded. See NOTES for details.\n>     Put a backslash (\"`\\`\") in front of the first \"`!`\" for patterns\n>     that begin with a literal \"`!`\", for example, \"`\\!important!.txt`\".\n>  \n> @@ -144,6 +143,12 @@ use 'git update-index {litdd}assume-unchanged'.\n>  To stop tracking a file that is currently tracked, use\n>  'git rm --cached'.\n>  \n> +It is usually not possible to re-include a file if a parent directory\n> +of that file is excluded because of performance reasons. However, if\n> +there are negative rules in the same .gitignore file that contains the\n> +rule to ignore a specific directory, and those negative rules contain\n> +a slash, then re-inclusion is possible.\n\nDoes that mean \"performance reasons\" goes out the window???\n\nWhat trade-off are the users making by choosing to do so?  Is it\nexplained in the documentation well enough to allow them to make an\ninformed decision?\n"},{"id":"252225","messageId":"20141119234136.GA53731@tower.spodhuis.org","threadId":"38007","inReplyTo":"CACsJy8BHvucnEAW065OXOe5NBQkxp7+8HJb7XCGYgOa=pkxLVA@mail.gmail.com","subject":"Re: .gitignore sub-dir exclusions not overriding '*'","fromName":"Phil Pennock","fromEmail":"phil-gitml@phil.spodhuis.org","sentAt":"2014-11-19T23:41:36Z","receivedAt":"2014-11-19T23:41:36Z","isPatch":false,"sender":{"key":"phil-gitml@phil.spodhuis.org","avatar":null},"body":"On 2014-11-19 at 16:48 +0700, Duy Nguyen wrote:\n> On Wed, Nov 19, 2014 at 10:40 AM, Phil Pennock\n> <phil-gitml@phil.spodhuis.org> wrote:\n> > Expected to work as .gitignore in top-level of repo:\n> >\n> >     *\n> >     !**/*.asc\n> >     !.gitignore\n> >\n> \n> gitignore man page has this \"It is not possible to re-include a file\n> if a parent directory of that file is excluded\". In this case,\n> directory \"foo\" is ignored by \"*\". Although it makes sense for this\n> particular case to re-include something in foo because we can clearly\n> see there are rules to re-include. It's on my todo list, but I don't\n> know when it will be implemented.\n\nThanks for this and the patches and discussion which follow.\n\nI didn't cover it in my report, but one of the scenarios I tried was to\nexplicitly re-include directories, to make them candidates again, and\neither use directory-matching patterns in the top-level .gitignore or to\nuse per-directory .gitignore to handle those directories.\n\nLooking fresh today, I see that I failed to compare baseline behaviour\nwithout a .gitignore when using `git status` as a baseline for\ncomparison.  So a .gitignore like this:\n\n    *\n    !*/\n    !*.asc\n\nappeared to not work; even within the `foo/` sub-directory, `git status`\nshows no candidates for inclusion.  But this is true even without a\n.gitignore.  *sigh*\n\nIn fact, it looks like the simple three lines above work, without any\n.gitignore in sub-directories.\n\nThe behaviour which confused me between this simplified test-case and\nthe original was that `git status` shows files in the top-level\ndirectory which are untracked, and in untracked files sub-directories\nwhere some other file in that directory is already tracked, but if no\nfile in the sub-directory is already tracked, then `git status` does not\nreport the files for inclusion, even if the cwd is inside that\ndirectory.\n\nI tied myself in knots trying to avoid adding unencrypted files to the\nrepo.\n\nThanks,\n-Phil\n"},{"id":"252226","messageId":"CACsJy8DyPoXwAKF4h1sj2QOevN96QSUcJqOBmm1FSim8MKT4_A@mail.gmail.com","threadId":"38007","inReplyTo":"xmqqmw7nrpqh.fsf@gitster.dls.corp.google.com","subject":"Re: [PATCH] dir.c: allow re-include after a dir is excluded in some cases","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2014-11-20T00:27:06Z","receivedAt":"2014-11-20T00:27:06Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Thu, Nov 20, 2014 at 1:48 AM, Junio C Hamano <gitster@pobox.com> wrote:\n> Nguyễn Thái Ngọc Duy  <pclouds@gmail.com> writes:\n>\n>> diff --git a/Documentation/gitignore.txt b/Documentation/gitignore.txt\n>> index 09e82c3..0340c44 100644\n>> --- a/Documentation/gitignore.txt\n>> +++ b/Documentation/gitignore.txt\n>> @@ -82,10 +82,9 @@ PATTERN FORMAT\n>>\n>>   - An optional prefix \"`!`\" which negates the pattern; any\n>>     matching file excluded by a previous pattern will become\n>> -   included again. It is not possible to re-include a file if a parent\n>> -   directory of that file is excluded. Git doesn't list excluded\n>> -   directories for performance reasons, so any patterns on contained\n>> -   files have no effect, no matter where they are defined.\n>> +   included again.\n>> +   It is usually not possible to re-include a file if a parent\n>> +   directory of that file is excluded. See NOTES for details.\n>>     Put a backslash (\"`\\`\") in front of the first \"`!`\" for patterns\n>>     that begin with a literal \"`!`\", for example, \"`\\!important!.txt`\".\n>>\n>> @@ -144,6 +143,12 @@ use 'git update-index {litdd}assume-unchanged'.\n>>  To stop tracking a file that is currently tracked, use\n>>  'git rm --cached'.\n>>\n>> +It is usually not possible to re-include a file if a parent directory\n>> +of that file is excluded because of performance reasons. However, if\n>> +there are negative rules in the same .gitignore file that contains the\n>> +rule to ignore a specific directory, and those negative rules contain\n>> +a slash, then re-inclusion is possible.\n>\n> Does that mean \"performance reasons\" goes out the window???\n>\n> What trade-off are the users making by choosing to do so?  Is it\n> explained in the documentation well enough to allow them to make an\n> informed decision?\n\nIf they put \"!**/foo\" there, I think it's obvious for users that all\ndirs must be looked at. \"!a*/**/foo\" may be expected to only look at a\nsubset of dirs recursively, instead of everything. \"!*/def\" may be\nexpected that only one more level of directories are looked at. I\ndidn't cover the last two cases well and I don't think it'll be easy\nto do. Which makes this patch less appealing. (so much for a\nnot-thoroughly-thought-through quick prototype)\n\nPerhaps I look at it from a wrong angle. If we have a way to say\n\"these patterns do not apply to directories\", then the user can\nselectively exclude certain directories and let others through. It\nwould give the user more control and make the travelling cost more\napparent (assuming we mention somewhere that the more dirs we examine,\nthe longer it will take).\n-- \nDuy\n"},{"id":"252233","messageId":"CACsJy8Cfg5eEoJb3zvPKoTjC8ZX-2ptggTUo_6yFz+q+-rqAnA@mail.gmail.com","threadId":"38007","inReplyTo":"20141119234136.GA53731@tower.spodhuis.org","subject":"Re: .gitignore sub-dir exclusions not overriding '*'","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2014-11-20T01:26:42Z","receivedAt":"2014-11-20T01:26:42Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Thu, Nov 20, 2014 at 6:41 AM, Phil Pennock\n<phil-gitml@phil.spodhuis.org> wrote:\n> On 2014-11-19 at 16:48 +0700, Duy Nguyen wrote:\n>> On Wed, Nov 19, 2014 at 10:40 AM, Phil Pennock\n>> <phil-gitml@phil.spodhuis.org> wrote:\n>> > Expected to work as .gitignore in top-level of repo:\n>> >\n>> >     *\n>> >     !**/*.asc\n>> >     !.gitignore\n>> >\n>>\n>> gitignore man page has this \"It is not possible to re-include a file\n>> if a parent directory of that file is excluded\". In this case,\n>> directory \"foo\" is ignored by \"*\". Although it makes sense for this\n>> particular case to re-include something in foo because we can clearly\n>> see there are rules to re-include. It's on my todo list, but I don't\n>> know when it will be implemented.\n>\n> Thanks for this and the patches and discussion which follow.\n>\n> I didn't cover it in my report, but one of the scenarios I tried was to\n> explicitly re-include directories, to make them candidates again, and\n> either use directory-matching patterns in the top-level .gitignore or to\n> use per-directory .gitignore to handle those directories.\n>\n> Looking fresh today, I see that I failed to compare baseline behaviour\n> without a .gitignore when using `git status` as a baseline for\n> comparison.  So a .gitignore like this:\n>\n>     *\n>     !*/\n>     !*.asc\n>\n> appeared to not work; even within the `foo/` sub-directory, `git status`\n> shows no candidates for inclusion.  But this is true even without a\n> .gitignore.  *sigh*\n\nI should have read this mail before replying to Junio in the other\nemail :( Yeah the \"!*/\" would re-include dirs back. I'm not sure if\nthere are any side effects by doing this, no time to think about this\nyet. Maybe we can put this in the example section in gitignore man\npage with more explanation.\n\n> In fact, it looks like the simple three lines above work, without any\n> .gitignore in sub-directories.\n>\n> The behaviour which confused me between this simplified test-case and\n> the original was that `git status` shows files in the top-level\n> directory which are untracked, and in untracked files sub-directories\n> where some other file in that directory is already tracked, but if no\n> file in the sub-directory is already tracked, then `git status` does not\n> report the files for inclusion, even if the cwd is inside that\n> directory.\n>\n> I tied myself in knots trying to avoid adding unencrypted files to the\n> repo.\n>\n> Thanks,\n> -Phil\n\n\n\n-- \nDuy\n"}]}