{"thread":{"id":"27863","subject":"[RFC/PATCH] grep --no-index: allow to grep without git exclusions","startedAt":"2011-07-20T12:50:53Z","lastAt":"2011-07-21T17:23:25Z","messageCount":7,"participants":["Bert Wesarg","Junio C Hamano","Nguyen Thai Ngoc Duy"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"171732","messageId":"82218b89c89f733dc0759d648b3a60bca6e20f3e.1311165328.git.bert.wesarg@googlemail.com","threadId":"27863","inReplyTo":null,"subject":"[RFC/PATCH] grep --no-index: allow to grep without git exclusions","fromName":"Bert Wesarg","fromEmail":"bert.wesarg@googlemail.com","sentAt":"2011-07-20T12:50:53Z","receivedAt":"2011-07-20T12:50:53Z","isPatch":true,"sender":{"key":"bert.wesarg@googlemail.com","avatar":"https://avatars.githubusercontent.com/u/111934?v=4"},"body":"It is currently not possible to grep in arbitrary files with --no-index,\nbecause the standard git exclusions are active when using --no-index.\n\nSuppress this when the new option --no-exclude-standard was provided.\n\nSigned-off-by: Bert Wesarg \n---\n\nThe name comes from git-ls-files, and I think it is worthwhile to add the full\nrange of exclusion flags to git-grep. Namely --exclude=,\n--exclude-from=, and --exclude-per-directory=. Which should \nonly be honored when using --no-index, obviously.\n\n Documentation/git-grep.txt | 7 +++++\n builtin/grep.c | 17 ++++++++++---\n t/t7810-grep.sh | 55 ++++++++++++++++++++++++++++++++++++++++++++\n 3 files changed, 75 insertions(+), 4 deletions(-)\n\ndiff --git a/Documentation/git-grep.txt b/Documentation/git-grep.txt\nindex 07b3c6a..be9cf8b 100644\n--- a/Documentation/git-grep.txt\n+++ b/Documentation/git-grep.txt\n@@ -24,6 +24,7 @@ SYNOPSIS\n \t [-f ] [-e] \n \t [--and|--or|--not|(|)|-e ...]\n \t [--cached | --no-index | ...]\n+\t [--no-exclude-standard]\n \t [--] [...]\n \n DESCRIPTION\n@@ -200,6 +201,12 @@ OPTIONS\n \tDo not output matched lines; instead, exit with status 0 when\n \tthere is a match and with non-zero status when there isn't.\n \n+--exclude-standard::\n+--[no-]exclude-standard::\n+\tUse the standard git exclusions: .git/info/exclude, .gitignore\n+\tin each directory, and the user's global exclusion file when\n+\tsearching in the work tree with `--no-index`.\n+\n ...::\n \tInstead of searching tracked files in the working tree, search\n \tblobs in the given trees.\ndiff --git a/builtin/grep.c b/builtin/grep.c\nindex cccf8da..f2fce73 100644\n--- a/builtin/grep.c\n+++ b/builtin/grep.c\n@@ -637,13 +637,15 @@ static int grep_objects(struct grep_opt *opt, const struct pathspec *pathspec,\n \treturn hit;\n }\n \n-static int grep_directory(struct grep_opt *opt, const struct pathspec *pathspec)\n+static int grep_directory(struct grep_opt *opt, const struct pathspec *pathspec,\n+\t\t\t int exclude_standard)\n {\n \tstruct dir_struct dir;\n \tint i, hit = 0;\n \n \tmemset(&dir, 0, sizeof(dir));\n-\tsetup_standard_excludes(&dir);\n+\tif (exclude_standard)\n+\t\tsetup_standard_excludes(&dir);\n \n \tfill_directory(&dir, pathspec->raw);\n \tfor (i = 0; i < dir.nr; i++) {\n@@ -761,7 +763,7 @@ int cmd_grep(int argc, const char **argv, const char *prefix)\n \tstruct string_list path_list = STRING_LIST_INIT_NODUP;\n \tint i;\n \tint dummy;\n-\tint use_index = 1;\n+\tint use_index = 1, exclude_standard = 1;\n \tenum {\n \t\tpattern_type_unspecified = 0,\n \t\tpattern_type_bre,\n@@ -839,6 +841,9 @@ int cmd_grep(int argc, const char **argv, const char *prefix)\n \t\tOPT_BOOLEAN('p', \"show-function\", &opt.funcname,\n \t\t\t\"show a line with the function name before matches\"),\n \t\tOPT_GROUP(\"\"),\n+\t\tOPT_BOOLEAN(0, \"exclude-standard\", &exclude_standard,\n+\t\t\t\"don't use standard excludes, needs --no-index\"),\n+\t\tOPT_GROUP(\"\"),\n \t\tOPT_CALLBACK('f', NULL, &opt, \"file\",\n \t\t\t\"read patterns from file\", file_callback),\n \t\t{ OPTION_CALLBACK, 'e', NULL, &opt, \"pattern\",\n@@ -936,6 +941,10 @@ int cmd_grep(int argc, const char **argv, const char *prefix)\n \t\tbreak; /* nothing */\n \t}\n \n+\t/* --no-exclude-standard needs --no-index */\n+\tif (use_index && !exclude_standard)\n+\t\tdie(_(\"--no-exclude-standard does not make sense without --no-index.\"));\n+\n \tif (use_index && !startup_info->have_repository)\n \t\t/* die the same way as if we did it at the beginning */\n \t\tsetup_git_directory();\n@@ -1050,7 +1059,7 @@ int cmd_grep(int argc, const char **argv, const char *prefix)\n \t\t\tdie(_(\"--cached cannot be used with --no-index.\"));\n \t\tif (list.nr)\n \t\t\tdie(_(\"--no-index cannot be used with revs.\"));\n-\t\thit = grep_directory(&opt, &pathspec);\n+\t\thit = grep_directory(&opt, &pathspec, exclude_standard);\n \t} else if (!list.nr) {\n \t\tif (!cached)\n \t\t\tsetup_work_tree();\ndiff --git a/t/t7810-grep.sh b/t/t7810-grep.sh\nindex a29ae45..15eb52b 100755\n--- a/t/t7810-grep.sh\n+++ b/t/t7810-grep.sh\n@@ -587,6 +587,61 @@ test_expect_success 'inside git repository but with --no-index' '\n \t)\n '\n \n+test_expect_success 'outside of git repository and --no-exclude-standard' '\n+\trm -fr non &&\n+\tmkdir -p non/git/sub &&\n+\techo hello >non/git/file1 &&\n+\techo world >non/git/sub/file2 &&\n+\techo \".*o*\" >non/git/.gitignore &&\n+\techo \"fi*\" >>non/git/.gitignore &&\n+\t{\n+\t\techo file1:hello &&\n+\t\techo sub/file2:world\n+\t} >non/expect.full &&\n+\t: >non/expect.empty &&\n+\techo file2:world >non/expect.sub &&\n+\t(\n+\t\tGIT_CEILING_DIRECTORIES=\"$(pwd)/non/git\" &&\n+\t\texport GIT_CEILING_DIRECTORIES &&\n+\t\tcd non/git &&\n+\t\ttest_must_fail git grep l &&\n+\t\tgit grep --no-index --no-exclude-standard l >../actual.full &&\n+\t\ttest_cmp ../expect.full ../actual.full\n+\t\tcd sub &&\n+\t\ttest_must_fail git grep l &&\n+\t\tgit grep --no-index l >../../actual.sub &&\n+\t\ttest_cmp ../../expect.sub ../../actual.sub\n+\t)\n+'\n+\n+test_expect_success 'inside git repository but with --no-index and --no-exclude-standard' '\n+\trm -fr is &&\n+\tmkdir -p is/git/sub &&\n+\techo hello >is/git/file1 &&\n+\techo world >is/git/sub/file2 &&\n+\techo \".*o*\" >is/git/.gitignore &&\n+\techo \"fi*\" >>is/git/.gitignore &&\n+\t{\n+\t\techo file1:hello &&\n+\t\techo sub/file2:world\n+\t} >is/expect.full &&\n+\t: >is/expect.empty &&\n+\techo file2:world >is/expect.sub &&\n+\t(\n+\t\tcd is/git &&\n+\t\tgit init &&\n+\t\ttest_must_fail git grep l >../actual.empty1 &&\n+\t\ttest_cmp ../expect.empty ../actual.empty1 &&\n+\t\tgit grep --no-index --no-exclude-standard l >../actual.full &&\n+\t\ttest_cmp ../expect.full ../actual.full &&\n+\t\tcd sub &&\n+\t\ttest_must_fail git grep l >../../actual.sub &&\n+\t\ttest_cmp ../../expect.empty ../../actual.sub &&\n+\t\tgit grep --no-index --no-exclude-standard l >../../actual.sub &&\n+\t\ttest_cmp ../../expect.sub ../../actual.sub\n+\t)\n+'\n+\n test_expect_success 'setup double-dash tests' '\n cat >double-dash < +\t/* --no-exclude-standard needs --no-index */\n> +\tif (use_index && !exclude_standard)\n> +\t\tdie(_(\"--no-exclude-standard does not make sense without --no-index.\"));\n\nFor that matter,\n\n $ git grep --no-exclude-standard --exclude-standard --cached -e foo\n\nshould be an error, no?\n"},{"id":"171782","messageId":"CACsJy8CYJzfAWAB5yRsX6ug6uMOq27POk8EyC8TDmVTbOdoFaA@mail.gmail.com","threadId":"27863","inReplyTo":"7vzkk86577.fsf@alter.siamese.dyndns.org","subject":"Re: [RFC/PATCH] grep --no-index: allow to grep without git exclusions","fromName":"Nguyen Thai Ngoc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2011-07-21T03:47:58Z","receivedAt":"2011-07-21T03:47:58Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Thu, Jul 21, 2011 at 3:57 AM, Junio C Hamano wrote:\n>  - It might be useful to be able to \"git grep\" both tracked and untracked\n>   (i.e. new files you may want to \"git add\") paths, but there is no good\n>   way to do so. Introduce a new option --untracked-too (or more suitable\n>   name --- I am bad at naming and not married to this one) to allow\n>   this. This mode always takes \"exclude\" into account.\n\n--ls-files ([c|d|o|i|s|u|k|m])+? This opens door for grepping only\nuntracked files, for instant.\n-- \nDuy\n"},{"id":"171800","messageId":"CAKPyHN2TMu2yO4sZDAqCce9P-5==Z2jKQVoU=zUsmUQJhHoeQg@mail.gmail.com","threadId":"27863","inReplyTo":"7vzkk86577.fsf@alter.siamese.dyndns.org","subject":"Re: [RFC/PATCH] grep --no-index: allow to grep without git exclusions","fromName":"Bert Wesarg","fromEmail":"bert.wesarg@googlemail.com","sentAt":"2011-07-21T07:11:25Z","receivedAt":"2011-07-21T07:11:25Z","isPatch":true,"sender":{"key":"bert.wesarg@googlemail.com","avatar":"https://avatars.githubusercontent.com/u/111934?v=4"},"body":"On Wed, Jul 20, 2011 at 22:57, Junio C Hamano wrote:\n> Taken in total isolation, this patch does allow a use case where we did\n> not allow, but when considered in a larger picture of what \"grep\" is used\n> for and how different use cases the command should support, a few random\n> thoughts come to mind:\n>\n>  - Like \"diff --no-index\", \"grep --no-index\" is about a directory that is\n>   not managed by git at all with random collection of files. Do we even\n>   want to be using any \"exclude\" in such a use case? Wouldn't it actually\n>   be a bug that we pay attention to standard-exlcludes in the current\n>   code, as .gitignore files scattered in such a directory should _not_\n>   mean anything, as it is not a git working tree at all?\n>\n>  - Even in a git managed directory, you _could_ use \"grep --no-index\" to\n>   find hits from both tracked and untracked files. In this particular use\n>   case, it makes some sense to pay attention to \"exclude\", as that would\n>   catch what _could_ be committed, and paths that would be excluded won't\n>   be part of that set (unless you use \"add -f\"). But wouldn't that use\n>   case better be covered by a switch that is different from --no-index\n>   (which means \"These are not managed by git at all\")? It is still about\n>   files in a git working tree, it is just that the user wants us to pay\n>   attention also to untracked files, e.g. \"grep --untracked-too\"?\n>\n> So I think the patch identified a good problem to solve, but it might be a\n> wrong solution that encourages a use of a wrong option (i.e. --no-index)\n> only because we do not have the right one (i.e. \"I am in the working tree,\n> but I want untracked ones also considered.\").\n>\n> What do people think if we did this a bit differently?\n>\n>  - Since 3081623 (grep --no-index: allow use of \"git grep\" outside a git\n>   repository, 2010-01-15) and 59332d1 (Resurrect \"git grep --no-index\",\n>   2010-02-06), \"grep --no-index\" incorrectly paid attention to the\n>   exclude patterns. We shouldn't have, and we'd fix that bug.\n\nI considered this when I noticed this, but feared the\nbackward-incompatibility. If you call it a bug, than we can change for\nthe better.\n\n>\n>  - It might be useful to be able to \"git grep\" both tracked and untracked\n>   (i.e. new files you may want to \"git add\") paths, but there is no good\n>   way to do so. Introduce a new option --untracked-too (or more suitable\n>   name --- I am bad at naming and not married to this one) to allow\n>   this. This mode always takes \"exclude\" into account.\n\nMy proposal would be to name it --include-untracked.\n\n>\n> Opinions?\n>\n> Regarding the patch:\n>\n>> +     /* --no-exclude-standard needs --no-index */\n>> +     if (use_index && !exclude_standard)\n>> +             die(_(\"--no-exclude-standard does not make sense without --no-index.\"));\n>\n> For that matter,\n>\n>    $ git grep --no-exclude-standard --exclude-standard --cached -e foo\n>\n> should be an error, no?\n>\n\nIt should be. But I think that unveils one of the shortcomings of the\n(any) option parser: You wont get notified when an option was given,\nregardless of its value. To handle the above I would have to use\nOPTION_CALLBACK to set an addition flag exc_given (like it is done in\ngit-ls-files) and test against this. The same problem is possible for\na number option, if you want to know whether the option was actually\ngiven on the command line, one need to invent an invalid value (which\nisn't always possible) and use this as the initializer or use\nOPTION_CALLBACK again.\n\nBert\n"},{"id":"171774","messageId":"CAKPyHN3JT1xmJMnSYS2q=AYCjKnodM8iLRjdnU+FA8K_zSO0-Q@mail.gmail.com","threadId":"27863","inReplyTo":"CAKPyHN2TMu2yO4sZDAqCce9P-5==Z2jKQVoU=zUsmUQJhHoeQg@mail.gmail.com","subject":"Re: [RFC/PATCH] grep --no-index: allow to grep without git exclusions","fromName":"Bert Wesarg","fromEmail":"bert.wesarg@googlemail.com","sentAt":"2011-07-21T07:25:19Z","receivedAt":"2011-07-21T07:25:19Z","isPatch":true,"sender":{"key":"bert.wesarg@googlemail.com","avatar":"https://avatars.githubusercontent.com/u/111934?v=4"},"body":"2011/7/21 Bert Wesarg :\n>\n> It should be. But I think that unveils one of the shortcomings of the\n> (any) option parser: You wont get notified when an option was given,\n> regardless of its value. To handle the above I would have to use\n> OPTION_CALLBACK to set an addition flag exc_given (like it is done in\n> git-ls-files) and test against this. The same problem is possible for\n> a number option, if you want to know whether the option was actually\n> given on the command line, one need to invent an invalid value (which\n> isn't always possible) and use this as the initializer or use\n> OPTION_CALLBACK again.\n\nHere is a proof-of-concept patch (probably whitespace damaged) to make\nit more clear what I meant:\n\n--- >8 ---\nSubject: [PoC/RFC/PATCH] parse-option: flag the presence of an option regardless\n of its value\n\nSigned-off-by: Bert Wesarg \n\n---\n parse-options.c | 15 +++++++++++++--\n parse-options.h | 20 ++++++++++++++++++++\n 2 files changed, 33 insertions(+), 2 deletions(-)\n\ndiff --git a/parse-options.c b/parse-options.c\nindex 73bd28a..df0f483 100644\n--- a/parse-options.c\n+++ b/parse-options.c\n@@ -156,7 +156,10 @@ static int parse_short_opt(struct parse_opt_ctx_t\n*p, const struct option *optio\n \tfor (; options->type != OPTION_END; options++) {\n \t\tif (options->short_name == *p->opt) {\n \t\t\tp->opt = p->opt[1] ? p->opt + 1 : NULL;\n-\t\t\treturn get_value(p, options, OPT_SHORT);\n+\t\t\tif (options->given) {\n+\t\t\t\t*options->given = 1;\n+\t\t\t\treturn get_value(p, options, OPT_SHORT);\n+\t\t\t}\n \t\t}\n\n \t\t/*\n@@ -171,6 +174,9 @@ static int parse_short_opt(struct parse_opt_ctx_t\n*p, const struct option *optio\n \t\tchar *arg;\n \t\tint rc;\n\n+\t\tif (options->given)\n+\t\t\t*options->given = 1;\n+\n \t\twhile (isdigit(p->opt[len]))\n \t\t\tlen++;\n \t\targ = xmemdupz(p->opt, len);\n@@ -254,6 +260,8 @@ is_abbreviated:\n \t\t\t\tcontinue;\n \t\t\tp->opt = rest + 1;\n \t\t}\n+\t\tif (options->given)\n+\t\t\t*options->given = 1;\n \t\treturn get_value(p, options, flags);\n \t}\n\n@@ -276,8 +284,11 @@ static int parse_nodash_opt(struct\nparse_opt_ctx_t *p, const char *arg,\n \tfor (; options->type != OPTION_END; options++) {\n \t\tif (!(options->flags & PARSE_OPT_NODASH))\n \t\t\tcontinue;\n-\t\tif (options->short_name == arg[0] && arg[1] == '\\0')\n+\t\tif (options->short_name == arg[0] && arg[1] == '\\0') {\n+\t\t\tif (options->given)\n+\t\t\t\t*options->given = 1;\n \t\t\treturn get_value(p, options, OPT_SHORT);\n+\t\t}\n \t}\n \treturn -2;\n }\ndiff --git a/parse-options.h b/parse-options.h\nindex d1b12fe..7fe3d49 100644\n--- a/parse-options.h\n+++ b/parse-options.h\n@@ -100,6 +100,10 @@ typedef int parse_opt_ll_cb(struct parse_opt_ctx_t *ctx,\n * OPTION_{BIT,SET_INT,SET_PTR} store the {mask,integer,pointer} to put in\n * the value when met.\n * CALLBACKS can use it like they want.\n+ *\n+ * `given`::\n+ * Pointer to a flag variable which is set to 1 if this option was given.\n+ * The flag should be 0 initialized, to make sense.\n */\n struct option {\n \tenum parse_opt_type type;\n@@ -112,6 +116,7 @@ struct option {\n \tint flags;\n \tparse_opt_cb *callback;\n \tintptr_t defval;\n+\tint *given;\n };\n\n #define OPT_END() { OPTION_END }\n@@ -146,6 +151,21 @@ struct option {\n \t{ OPTION_CALLBACK, (s), (l), (v), \"when\", (h), PARSE_OPT_OPTARG, \\\n \t\tparse_opt_color_flag_cb, (intptr_t)\"always\" }\n\n+/* same as above but with given flag pointer */\n+#define OPT_BIT_GIVEN(s, l, v, h, b, g) \\\n+\t{ OPTION_BIT, (s), (l), (v), NULL, (h), PARSE_OPT_NOARG, NULL, (b), (g) }\n+#define OPT_NEGBIT_GIVEN(s, l, v, h, b, g) \\\n+\t{ OPTION_NEGBIT, (s), (l), (v), NULL, (h), PARSE_OPT_NOARG, NULL, (b), (g) }\n+#define OPT_BOOLEAN_GIVEN(s, l, v, h, g) \\\n+\t{ OPTION_BOOLEAN, (s), (l), (v), NULL, (h), PARSE_OPT_NOARG, NULL, 0, (g) }\n+#define OPT_SET_INT_GIVEN(s, l, v, h, i, g) \\\n+\t{ OPTION_SET_INT, (s), (l), (v), NULL, (h), PARSE_OPT_NOARG, NULL, (i), (g) }\n+#define OPT_SET_PTR_GIVEN(s, l, v, h, p, g) \\\n+\t{ OPTION_SET_PTR, (s), (l), (v), NULL, (h), PARSE_OPT_NOARG, NULL, (p), (g) }\n+#define OPT_INTEGER_GIVEN(s, l, v, h, g) \\\n+\t{ OPTION_INTEGER, (s), (l), (v), \"n\", (h), 0, NULL, 0, (g) }\n+#define OPT_STRING_GIVEN(s, l, v, a, h, g) \\\n+\t{ OPTION_STRING, (s), (l), (v), (a), (h), 0, NULL, 0, (g) }\n\n /* parse_options() will filter out the processed options and leave the\n * non-option arguments in argv[].\n-- \n1.7.6.588.g8d735\n"},{"id":"171796","messageId":"7v62mv4n93.fsf@alter.siamese.dyndns.org","threadId":"27863","inReplyTo":"CAKPyHN2TMu2yO4sZDAqCce9P-5==Z2jKQVoU=zUsmUQJhHoeQg@mail.gmail.com","subject":"Re: [RFC/PATCH] grep --no-index: allow to grep without git exclusions","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-07-21T16:22:32Z","receivedAt":"2011-07-21T16:22:32Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Bert Wesarg writes:\n\n> It should be. But I think that unveils one of the shortcomings of the\n> (any) option parser: You wont get notified when an option was given,\n> regardless of its value. To handle the above I would have to use\n> OPTION_CALLBACK to set an addition flag exc_given (like it is done in\n> git-ls-files) and test against this.\n\nPrepare a three-value variable, initialized to -1, set it to 0 on --no-foo\nand set it to 1 on --foo. Use the default if the variable is still -1.\n"},{"id":"171785","messageId":"CAKPyHN3dRyrpW_3emC1s8eDWTF_hSGEY9p+Ojio3NzvNP2hOsg@mail.gmail.com","threadId":"27863","inReplyTo":"7v62mv4n93.fsf@alter.siamese.dyndns.org","subject":"Re: [RFC/PATCH] grep --no-index: allow to grep without git exclusions","fromName":"Bert Wesarg","fromEmail":"bert.wesarg@googlemail.com","sentAt":"2011-07-21T17:23:25Z","receivedAt":"2011-07-21T17:23:25Z","isPatch":true,"sender":{"key":"bert.wesarg@googlemail.com","avatar":"https://avatars.githubusercontent.com/u/111934?v=4"},"body":"2011/7/21 Junio C Hamano :\n> Bert Wesarg writes:\n>\n>> It should be. But I think that unveils one of the shortcomings of the\n>> (any) option parser: You wont get notified when an option was given,\n>> regardless of its value. To handle the above I would have to use\n>> OPTION_CALLBACK to set an addition flag exc_given (like it is done in\n>> git-ls-files) and test against this.\n>\n> Prepare a three-value variable, initialized to -1, set it to 0 on --no-foo\n> and set it to 1 on --foo. Use the default if the variable is still -1.\n>\n\nThats the 'invent an invalid value'-case I described. Which does not\nnecessarily exist.\n\nHaving the proposed 'given' flag available I would need only one\nvariable for all the options (these coming from ls-files) to check if\nany of them where given, and could decide if the given options would\nmake sense.\n\nBert\n"}]}