{"thread":{"id":"32742","subject":"[PATCH v4 0/2] for-each-repo: new command for multi-repo operations","startedAt":"2013-01-27T12:46:15Z","lastAt":"2013-02-04T06:41:41Z","messageCount":18,"participants":["Lars Hjemli","Junio C Hamano","John Keeping","Jonathan Nieder","Jens Lehmann"],"isPatch":true,"patchVersion":4,"patchTotal":2},"messages":[{"id":"207949","messageId":"1359290777-5483-1-git-send-email-hjemli@gmail.com","threadId":"32742","inReplyTo":null,"subject":"[PATCH v4 0/2] for-each-repo: new command for multi-repo operations","fromName":"Lars Hjemli","fromEmail":"hjemli@gmail.com","sentAt":"2013-01-27T12:46:15Z","receivedAt":"2013-01-27T12:46:15Z","isPatch":true,"sender":{"key":"hjemli@gmail.com","avatar":null},"body":"Changes since v3:\n* option -x used to execute non-git commands\n* option -z used to NUL-terminate paths\n* write_name_quoted() used to print repo paths\n* repos are handled in sorted order (as defined by strcmp(3)) to get\n  predictable output from the command\n* unsetenv() reintroduced to avoid problems from GIT_DIR/WORK_TREE\n* more tests\n\nLars Hjemli (2):\n  for-each-repo: new command used for multi-repo operations\n  git: rewrite `git -a` to become a git-for-each-repo command\n\n .gitignore                          |   1 +\n Documentation/git-for-each-repo.txt |  71 ++++++++++++\n Makefile                            |   1 +\n builtin.h                           |   1 +\n builtin/for-each-repo.c             | 145 ++++++++++++++++++++++++\n git.c                               |  37 +++++++\n t/t6400-for-each-repo.sh            | 213 ++++++++++++++++++++++++++++++++++++\n 7 files changed, 469 insertions(+)\n create mode 100644 Documentation/git-for-each-repo.txt\n create mode 100644 builtin/for-each-repo.c\n create mode 100755 t/t6400-for-each-repo.sh\n\n-- \n1.8.1.1.349.g4cdd23e\n"},{"id":"207951","messageId":"1359290777-5483-2-git-send-email-hjemli@gmail.com","threadId":"32742","inReplyTo":"1359290777-5483-1-git-send-email-hjemli@gmail.com","subject":"[PATCH v4 1/2] for-each-repo: new command used for multi-repo operations","fromName":"Lars Hjemli","fromEmail":"hjemli@gmail.com","sentAt":"2013-01-27T12:46:16Z","receivedAt":"2013-01-27T12:46:16Z","isPatch":true,"sender":{"key":"hjemli@gmail.com","avatar":null},"body":"When working with multiple, unrelated (or loosly related) git repos,\nthere is often a need to locate all repos with uncommitted work and\nperform some action on them (say, commit and push). Before this patch,\nsuch tasks would require manually visiting all repositories, running\n`git status` within each one and then decide what to do next.\n\nThis mundane task can now be automated by e.g. `git for-each-repo --dirty\nstatus`, which will find all non-bare git repositories below the current\ndirectory (even nested ones), check if they are dirty (as defined by\n`git diff --quiet && git diff --cached --quiet`), and for each dirty repo\nprint the path to the repo and then execute `git status` within the repo.\n\nThe command also honours the option '--clean' which restricts the set of\nrepos to those which '--dirty' would skip, and '-x' which is used to\nexecute non-git commands.\n\nFinally, the command to execute within each repo is optional. If none is\ngiven, git-for-each-repo will just print the path to each repo found. And\nsince the command supports -z, this can be used for more advanced scripting\nneeds.\n\nNote: since git-for-each-repo can execute both git- and nongit commands, it\nmust cd into the worktree of each repository before executing the command.\nIt is then no need for the environment variables $GIT_WORK_TREE and $GIT_DIR\nto be specified, so git-for-each-repo will instead unset these variables to\nstop them from interfering with the executed commands.\n\nSigned-off-by: Lars Hjemli <hjemli@gmail.com>\n---\n .gitignore                          |   1 +\n Documentation/git-for-each-repo.txt |  71 +++++++++++++++++\n Makefile                            |   1 +\n builtin.h                           |   1 +\n builtin/for-each-repo.c             | 145 ++++++++++++++++++++++++++++++++++\n git.c                               |   1 +\n t/t6400-for-each-repo.sh            | 150 ++++++++++++++++++++++++++++++++++++\n 7 files changed, 370 insertions(+)\n create mode 100644 Documentation/git-for-each-repo.txt\n create mode 100644 builtin/for-each-repo.c\n create mode 100755 t/t6400-for-each-repo.sh\n\ndiff --git a/.gitignore b/.gitignore\nindex aa258a6..0c27981 100644\n--- a/.gitignore\n+++ b/.gitignore\n@@ -56,6 +56,7 @@\n /git-filter-branch\n /git-fmt-merge-msg\n /git-for-each-ref\n+/git-for-each-repo\n /git-format-patch\n /git-fsck\n /git-fsck-objects\ndiff --git a/Documentation/git-for-each-repo.txt b/Documentation/git-for-each-repo.txt\nnew file mode 100644\nindex 0000000..fb12b3f\n--- /dev/null\n+++ b/Documentation/git-for-each-repo.txt\n@@ -0,0 +1,71 @@\n+git-for-each-repo(1)\n+====================\n+\n+NAME\n+----\n+git-for-each-repo - Execute a git command in multiple non-bare repositories\n+\n+SYNOPSIS\n+--------\n+[verse]\n+'git for-each-repo' [-acdxz] [command]\n+\n+DESCRIPTION\n+-----------\n+The git-for-each-repo command is used to locate all non-bare git\n+repositories within the current directory tree, and optionally\n+execute a git command in each of the found repos.\n+\n+OPTIONS\n+-------\n+-a::\n+--all::\n+\tInclude both clean and dirty repositories (this is the default\n+\tbehaviour of `git-for-each-repo`).\n+\n+-c::\n+--clean::\n+\tOnly include repositories with a clean worktree.\n+\n+-d::\n+--dirty::\n+\tOnly include repositories with a dirty worktree.\n+\n+-x::\n+\tExecute a genric (non-git) command in each repo.\n+\n+-z::\n+\tTerminate each path name with the NUL character.\n+\n+EXAMPLES\n+--------\n+\n+Various ways to exploit this command::\n++\n+------------\n+$ git for-each-repo            <1>\n+$ git for-each-repo fetch      <2>\n+$ git for-each-repo -d gui     <3>\n+$ git for-each-repo -c push    <4>\n+$ git for-each-repo -x du -sh  <5>\n+------------\n++\n+<1> Print the path to all repos found below the current directory.\n+\n+<2> Fetch updates from default remote in all repos.\n+\n+<3> Start linkgit:git-gui[1] in each repo containing uncommitted changes.\n+\n+<4> Push the current branch in each repo with no uncommited changes.\n+\n+<5> Print disk-usage for each repository.\n+\n+NOTES\n+-----\n+\n+For the purpose of `git-for-each-repo`, a dirty worktree is defined as a\n+worktree with uncommitted changes.\n+\n+GIT\n+---\n+Part of the linkgit:git[1] suite\ndiff --git a/Makefile b/Makefile\nindex a786d4c..8c42c17 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -870,6 +870,7 @@ BUILTIN_OBJS += builtin/fetch-pack.o\n BUILTIN_OBJS += builtin/fetch.o\n BUILTIN_OBJS += builtin/fmt-merge-msg.o\n BUILTIN_OBJS += builtin/for-each-ref.o\n+BUILTIN_OBJS += builtin/for-each-repo.o\n BUILTIN_OBJS += builtin/fsck.o\n BUILTIN_OBJS += builtin/gc.o\n BUILTIN_OBJS += builtin/grep.o\ndiff --git a/builtin.h b/builtin.h\nindex 7e7bbd6..02fc712 100644\n--- a/builtin.h\n+++ b/builtin.h\n@@ -73,6 +73,7 @@ extern int cmd_fetch(int argc, const char **argv, const char *prefix);\n extern int cmd_fetch_pack(int argc, const char **argv, const char *prefix);\n extern int cmd_fmt_merge_msg(int argc, const char **argv, const char *prefix);\n extern int cmd_for_each_ref(int argc, const char **argv, const char *prefix);\n+extern int cmd_for_each_repo(int argc, const char **argv, const char *prefix);\n extern int cmd_format_patch(int argc, const char **argv, const char *prefix);\n extern int cmd_fsck(int argc, const char **argv, const char *prefix);\n extern int cmd_gc(int argc, const char **argv, const char *prefix);\ndiff --git a/builtin/for-each-repo.c b/builtin/for-each-repo.c\nnew file mode 100644\nindex 0000000..9333ae0\n--- /dev/null\n+++ b/builtin/for-each-repo.c\n@@ -0,0 +1,145 @@\n+/*\n+ * \"git for-each-repo\" builtin command.\n+ *\n+ * Copyright (c) 2013 Lars Hjemli <hjemli@gmail.com>\n+ */\n+#include \"cache.h\"\n+#include \"color.h\"\n+#include \"quote.h\"\n+#include \"builtin.h\"\n+#include \"run-command.h\"\n+#include \"parse-options.h\"\n+\n+#define ALL 0\n+#define DIRTY 1\n+#define CLEAN 2\n+\n+static char *color = GIT_COLOR_NORMAL;\n+static int eol = '\\n';\n+static int match;\n+static int runopt = RUN_GIT_CMD;\n+\n+static const char * const builtin_foreachrepo_usage[] = {\n+\tN_(\"git for-each-repo [-acdxz] [cmd]\"),\n+\tNULL\n+};\n+\n+static struct option builtin_foreachrepo_options[] = {\n+\tOPT_SET_INT('a', \"all\", &match, N_(\"match both clean and dirty repositories\"), ALL),\n+\tOPT_SET_INT('c', \"clean\", &match, N_(\"only show clean repositories\"), CLEAN),\n+\tOPT_SET_INT('d', \"dirty\", &match, N_(\"only show dirty repositories\"), DIRTY),\n+\tOPT_SET_INT('x', NULL, &runopt, N_(\"execute generic (non-git) command\"), 0),\n+\tOPT_SET_INT('z', NULL, &eol, N_(\"terminate each repo path with NUL character\"), 0),\n+\tOPT_END(),\n+};\n+\n+static int get_repo_state(const char *dir)\n+{\n+\tconst char *diffidx[] = {\"diff\", \"--quiet\", \"--cached\", NULL};\n+\tconst char *diffwd[] = {\"diff\", \"--quiet\", NULL};\n+\n+\tif (run_command_v_opt_cd_env(diffidx, RUN_GIT_CMD, dir, NULL) != 0)\n+\t\treturn DIRTY;\n+\tif (run_command_v_opt_cd_env(diffwd, RUN_GIT_CMD, dir, NULL) != 0)\n+\t\treturn DIRTY;\n+\treturn CLEAN;\n+}\n+\n+static void print_repo_path(const char *path, unsigned pretty)\n+{\n+\tif (path[0] == '.' && path[1] == '/')\n+\t\tpath += 2;\n+\tif (pretty)\n+\t\tcolor_fprintf_ln(stdout, color, \"[%s]\", path);\n+\telse\n+\t\twrite_name_quoted(path, stdout, eol);\n+}\n+\n+static void handle_repo(struct strbuf *path, const char **argv)\n+{\n+\tconst char *gitdir;\n+\tint len;\n+\n+\tlen = path->len;\n+\tstrbuf_addstr(path, \".git\");\n+\tgitdir = resolve_gitdir(path->buf);\n+\tstrbuf_setlen(path, len - 1);\n+\tif (!gitdir)\n+\t\tgoto done;\n+\tif (match != ALL && match != get_repo_state(path->buf))\n+\t\tgoto done;\n+\tprint_repo_path(path->buf, *argv != NULL);\n+\tif (*argv)\n+\t\trun_command_v_opt_cd_env(argv, runopt, path->buf, NULL);\n+done:\n+\tstrbuf_addstr(path, \"/\");\n+}\n+\n+static int walk(struct strbuf *path, int argc, const char **argv)\n+{\n+\tDIR *dir;\n+\tstruct dirent *ent;\n+\tstruct stat st;\n+\tsize_t len;\n+\tint has_dotgit = 0;\n+\tstruct string_list list = STRING_LIST_INIT_DUP;\n+\tstruct string_list_item *item;\n+\n+\tdir = opendir(path->buf);\n+\tif (!dir)\n+\t\treturn errno;\n+\tstrbuf_addstr(path, \"/\");\n+\tlen = path->len;\n+\twhile ((ent = readdir(dir))) {\n+\t\tif (!strcmp(ent->d_name, \".\") || !strcmp(ent->d_name, \"..\"))\n+\t\t\tcontinue;\n+\t\tif (!strcmp(ent->d_name, \".git\")) {\n+\t\t\thas_dotgit = 1;\n+\t\t\tcontinue;\n+\t\t}\n+\t\tswitch (DTYPE(ent)) {\n+\t\tcase DT_UNKNOWN:\n+\t\tcase DT_LNK:\n+\t\t\t/* Use stat() to figure out if this path leads\n+\t\t\t * to a directory - it's  not important if it's\n+\t\t\t * a symlink which gets us there.\n+\t\t\t */\n+\t\t\tstrbuf_setlen(path, len);\n+\t\t\tstrbuf_addstr(path, ent->d_name);\n+\t\t\tif (stat(path->buf, &st) || !S_ISDIR(st.st_mode))\n+\t\t\t\tbreak;\n+\t\t\t/* fallthrough */\n+\t\tcase DT_DIR:\n+\t\t\tstring_list_append(&list, ent->d_name);\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\tclosedir(dir);\n+\tstrbuf_setlen(path, len);\n+\tif (has_dotgit)\n+\t\thandle_repo(path, argv);\n+\tsort_string_list(&list);\n+\tfor_each_string_list_item(item, &list) {\n+\t\tstrbuf_setlen(path, len);\n+\t\tstrbuf_addstr(path, item->string);\n+\t\twalk(path, argc, argv);\n+\t}\n+\tstring_list_clear(&list, 0);\n+\treturn 0;\n+}\n+\n+int cmd_for_each_repo(int argc, const char **argv, const char *prefix)\n+{\n+\tstruct strbuf path = STRBUF_INIT;\n+\n+\tunsetenv(GIT_DIR_ENVIRONMENT);\n+\tunsetenv(GIT_WORK_TREE_ENVIRONMENT);\n+\targc = parse_options(argc, argv, prefix,\n+\t\t\t     builtin_foreachrepo_options,\n+\t\t\t     builtin_foreachrepo_usage,\n+\t\t\t     PARSE_OPT_STOP_AT_NON_OPTION);\n+\tif (want_color(GIT_COLOR_AUTO))\n+\t\tcolor = GIT_COLOR_YELLOW;\n+\tstrbuf_addstr(&path, \".\");\n+\treturn walk(&path, argc, argv);\n+}\ndiff --git a/git.c b/git.c\nindex ed66c66..6b53169 100644\n--- a/git.c\n+++ b/git.c\n@@ -337,6 +337,7 @@ static void handle_internal_command(int argc, const char **argv)\n \t\t{ \"fetch-pack\", cmd_fetch_pack, RUN_SETUP },\n \t\t{ \"fmt-merge-msg\", cmd_fmt_merge_msg, RUN_SETUP },\n \t\t{ \"for-each-ref\", cmd_for_each_ref, RUN_SETUP },\n+\t\t{ \"for-each-repo\", cmd_for_each_repo },\n \t\t{ \"format-patch\", cmd_format_patch, RUN_SETUP },\n \t\t{ \"fsck\", cmd_fsck, RUN_SETUP },\n \t\t{ \"fsck-objects\", cmd_fsck, RUN_SETUP },\ndiff --git a/t/t6400-for-each-repo.sh b/t/t6400-for-each-repo.sh\nnew file mode 100755\nindex 0000000..af02c0c\n--- /dev/null\n+++ b/t/t6400-for-each-repo.sh\n@@ -0,0 +1,150 @@\n+#!/bin/sh\n+#\n+# Copyright (c) 2013 Lars Hjemli\n+#\n+\n+test_description='Test the git-for-each-repo command'\n+\n+. ./test-lib.sh\n+\n+qname=\"with\\\"quote\"\n+qqname=\"\\\"with\\\\\\\"quote\\\"\"\n+\n+test_expect_success \"setup\" '\n+\ttest_create_repo clean &&\n+\t(cd clean && test_commit foo1) &&\n+\tgit init --separate-git-dir=.cleansub clean/gitfile &&\n+\t(cd clean/gitfile && test_commit foo2 && echo bar >>foo2.t) &&\n+\ttest_create_repo dirty-idx &&\n+\t(cd dirty-idx && test_commit foo3 && git rm foo3.t) &&\n+\ttest_create_repo dirty-wt &&\n+\t(cd dirty-wt && mv .git .linkedgit && ln -s .linkedgit .git &&\n+\t  test_commit foo4 && rm foo4.t) &&\n+\ttest_create_repo \"$qname\" &&\n+\t(cd \"$qname\" && test_commit foo5) &&\n+\tmkdir fakedir && mkdir fakedir/.git\n+'\n+\n+test_expect_success \"without filtering, all repos are included\" '\n+\techo \".\" >expect &&\n+\techo \"clean\" >>expect &&\n+\techo \"clean/gitfile\" >>expect &&\n+\techo \"dirty-idx\" >>expect &&\n+\techo \"dirty-wt\" >>expect &&\n+\techo \"$qqname\" >>expect &&\n+\tgit for-each-repo >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success \"-z NUL-terminates each path\" '\n+\techo \"(.)\" >expect &&\n+\techo \"(clean)\" >>expect &&\n+\techo \"(clean/gitfile)\" >>expect &&\n+\techo \"(dirty-idx)\" >>expect &&\n+\techo \"(dirty-wt)\" >>expect &&\n+\techo \"($qname)\" >>expect &&\n+\tgit for-each-repo -z | xargs -0 printf \"(%s)\\n\"  >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success \"--dirty only includes dirty repos\" '\n+\techo \"clean/gitfile\" >expect &&\n+\techo \"dirty-idx\" >>expect &&\n+\techo \"dirty-wt\" >>expect &&\n+\tgit for-each-repo --dirty >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success \"--clean only includes clean repos\" '\n+\techo \".\" >expect &&\n+\techo \"clean\" >>expect &&\n+\techo \"$qqname\" >>expect &&\n+\tgit for-each-repo --clean >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success \"run a git-command in all repos\" '\n+\techo \"[.]\" >expect &&\n+\techo \"[clean]\" >>expect &&\n+\techo \"[clean/gitfile]\" >>expect &&\n+\techo \" M foo2.t\" >>expect &&\n+\techo \"[dirty-idx]\" >>expect &&\n+\techo \"D  foo3.t\" >>expect &&\n+\techo \"[dirty-wt]\" >>expect &&\n+\techo \" D foo4.t\" >> expect\n+\techo \"[$qname]\" >>expect &&\n+\tgit for-each-repo status -suno >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success \"run a git-command in dirty repos only\" '\n+\techo \"[clean/gitfile]\" >expect &&\n+\techo \" M foo2.t\" >>expect &&\n+\techo \"[dirty-idx]\" >>expect &&\n+\techo \"D  foo3.t\" >>expect &&\n+\techo \"[dirty-wt]\" >>expect &&\n+\techo \" D foo4.t\" >> expect\n+\tgit for-each-repo -d status -suno >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success \"run a git-command in clean repos only\" '\n+\techo \"[.]\" >expect &&\n+\techo \"[clean]\" >>expect &&\n+\techo \"foo1.t\" >>expect &&\n+\techo \"[$qname]\" >>expect &&\n+\techo \"foo5.t\" >>expect &&\n+\tgit for-each-repo -c ls-files >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success \"-z is disabled when a command is run\" '\n+\techo \"[.]\" >expect &&\n+\techo \"[clean]\" >>expect &&\n+\techo \"foo1.t\" >>expect &&\n+\techo \"[$qname]\" >>expect &&\n+\techo \"foo5.t\" >>expect &&\n+\tgit for-each-repo -cz ls-files >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success \"-x executes any command in each repo\" '\n+\techo \"[.]\" >expect &&\n+\techo \"$HOME\" >>expect &&\n+\techo \"[clean]\" >>expect &&\n+\techo \"$HOME/clean\" >>expect &&\n+\techo \"[clean/gitfile]\" >>expect &&\n+\techo \"$HOME/clean/gitfile\" >>expect &&\n+\techo \"[dirty-idx]\" >>expect &&\n+\techo \"$HOME/dirty-idx\" >>expect &&\n+\techo \"[dirty-wt]\" >>expect &&\n+\techo \"$HOME/dirty-wt\" >> expect\n+\techo \"[$qname]\" >>expect &&\n+\techo \"$HOME/$qname\" >>expect &&\n+\tgit for-each-repo -x pwd >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success \"-cx executes any command in clean repos\" '\n+\techo \"[.]\" >expect &&\n+\techo \"$HOME\" >>expect &&\n+\techo \"[clean]\" >>expect &&\n+\techo \"$HOME/clean\" >>expect &&\n+\techo \"[$qname]\" >>expect &&\n+\techo \"$HOME/$qname\" >>expect &&\n+\tgit for-each-repo -cx pwd >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success \"-dx executes any command in dirty repos\" '\n+\techo \"[clean/gitfile]\" >expect &&\n+\techo \"$HOME/clean/gitfile\" >>expect &&\n+\techo \"[dirty-idx]\" >>expect &&\n+\techo \"$HOME/dirty-idx\" >>expect &&\n+\techo \"[dirty-wt]\" >>expect &&\n+\techo \"$HOME/dirty-wt\" >> expect\n+\tgit for-each-repo -dx pwd >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_done\n-- \n1.8.1.1.349.g4cdd23e\n"},{"id":"207950","messageId":"1359290777-5483-3-git-send-email-hjemli@gmail.com","threadId":"32742","inReplyTo":"1359290777-5483-1-git-send-email-hjemli@gmail.com","subject":"[PATCH v4 2/2] git: rewrite `git -a` to become a git-for-each-repo command","fromName":"Lars Hjemli","fromEmail":"hjemli@gmail.com","sentAt":"2013-01-27T12:46:17Z","receivedAt":"2013-01-27T12:46:17Z","isPatch":true,"sender":{"key":"hjemli@gmail.com","avatar":null},"body":"With this rewriting, it is now possible to run e.g. `git -ad gui` to\nstart up git-gui in each repo within the current directory which\ncontains uncommited work.\n\nSigned-off-by: Lars Hjemli <hjemli@gmail.com>\n---\n git.c                    | 36 +++++++++++++++++++++++++++\n t/t6400-for-each-repo.sh | 63 ++++++++++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 99 insertions(+)\n\ndiff --git a/git.c b/git.c\nindex 6b53169..f933b5d 100644\n--- a/git.c\n+++ b/git.c\n@@ -31,8 +31,42 @@ static void commit_pager_choice(void) {\n \t}\n }\n \n+/*\n+ * Rewrite 'git -ad status' to 'git for-each-repo -d status'\n+ */\n+static int rewrite_foreach_repo(const char ***orig_argv,\n+\t\t\t\tconst char **curr_argv,\n+\t\t\t\tint *curr_argc)\n+{\n+\tconst char **new_argv;\n+\tchar *tmp;\n+\tint new_argc, curr_pos, i, j;\n+\n+\tcurr_pos = curr_argv - *orig_argv;\n+\tif (strlen(curr_argv[0]) == 2) {\n+\t\tcurr_argv[0] = \"for-each-repo\";\n+\t\treturn curr_pos - 1;\n+\t}\n+\n+\tnew_argc = curr_pos + *curr_argc + 1;\n+\tnew_argv = xmalloc(new_argc * sizeof(void *));\n+\tfor (i = j = 0; j < new_argc; i++, j++) {\n+\t\tif (i == curr_pos) {\n+\t\t\tasprintf(&tmp, \"-%s\", (*orig_argv)[i] + 2);\n+\t\t\tnew_argv[j] = \"for-each-repo\";\n+\t\t\tnew_argv[++j] = tmp;\n+\t\t} else {\n+\t\t\tnew_argv[j] = (*orig_argv)[i];\n+\t\t}\n+\t}\n+\t*orig_argv = new_argv;\n+\t(*curr_argc)++;\n+\treturn curr_pos;\n+}\n+\n static int handle_options(const char ***argv, int *argc, int *envchanged)\n {\n+\tconst char ***pargv = argv;\n \tconst char **orig_argv = *argv;\n \n \twhile (*argc > 0) {\n@@ -143,6 +177,8 @@ static int handle_options(const char ***argv, int *argc, int *envchanged)\n \t\t\tsetenv(GIT_LITERAL_PATHSPECS_ENVIRONMENT, \"0\", 1);\n \t\t\tif (envchanged)\n \t\t\t\t*envchanged = 1;\n+\t\t} else if (!strncmp(cmd, \"-a\", 2)) {\n+\t\t\treturn rewrite_foreach_repo(pargv, *argv, argc);\n \t\t} else {\n \t\t\tfprintf(stderr, \"Unknown option: %s\\n\", cmd);\n \t\t\tusage(git_usage_string);\ndiff --git a/t/t6400-for-each-repo.sh b/t/t6400-for-each-repo.sh\nindex af02c0c..eaa4518 100755\n--- a/t/t6400-for-each-repo.sh\n+++ b/t/t6400-for-each-repo.sh\n@@ -147,4 +147,67 @@ test_expect_success \"-dx executes any command in dirty repos\" '\n \ttest_cmp expect actual\n '\n \n+test_expect_success \"rewrite 'git -a'\" '\n+\techo \".\" >expect &&\n+\techo \"clean\" >>expect &&\n+\techo \"clean/gitfile\" >>expect &&\n+\techo \"dirty-idx\" >>expect &&\n+\techo \"dirty-wt\" >>expect &&\n+\techo \"$qqname\" >>expect &&\n+\tgit -a >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success \"rewrite 'git -az'\" '\n+\techo \"(.)\" >expect &&\n+\techo \"(clean)\" >>expect &&\n+\techo \"(clean/gitfile)\" >>expect &&\n+\techo \"(dirty-idx)\" >>expect &&\n+\techo \"(dirty-wt)\" >>expect &&\n+\techo \"($qname)\" >>expect &&\n+\tgit -az | xargs -0 printf \"(%s)\\n\"  >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success \"rewrite 'git -ad'\" '\n+\techo \"clean/gitfile\" >expect &&\n+\techo \"dirty-idx\" >>expect &&\n+\techo \"dirty-wt\" >>expect &&\n+\tgit -ad >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success \"rewrite 'git -ac'\" '\n+\techo \".\" >expect &&\n+\techo \"clean\" >>expect &&\n+\techo \"$qqname\" >>expect &&\n+\tgit -ac >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success \"rewrite 'git -a status -suno'\" '\n+\techo \"[.]\" >expect &&\n+\techo \"[clean]\" >>expect &&\n+\techo \"[clean/gitfile]\" >>expect &&\n+\techo \" M foo2.t\" >>expect &&\n+\techo \"[dirty-idx]\" >>expect &&\n+\techo \"D  foo3.t\" >>expect &&\n+\techo \"[dirty-wt]\" >>expect &&\n+\techo \" D foo4.t\" >> expect\n+\techo \"[$qname]\" >>expect &&\n+\tgit -a status -suno >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success \"rewrite 'git -acx pwd'\" '\n+\techo \"[.]\" >expect &&\n+\techo \"$HOME\" >>expect &&\n+\techo \"[clean]\" >>expect &&\n+\techo \"$HOME/clean\" >>expect &&\n+\techo \"[$qname]\" >>expect &&\n+\techo \"$HOME/$qname\" >>expect &&\n+\tgit -acx pwd >actual &&\n+\ttest_cmp expect actual\n+'\n+\n test_done\n-- \n1.8.1.1.349.g4cdd23e\n"},{"id":"207968","messageId":"7vk3qywiqf.fsf@alter.siamese.dyndns.org","threadId":"32742","inReplyTo":"1359290777-5483-2-git-send-email-hjemli@gmail.com","subject":"Re: [PATCH v4 1/2] for-each-repo: new command used for multi-repo operations","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-01-27T19:04:08Z","receivedAt":"2013-01-27T19:04:08Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Lars Hjemli <hjemli@gmail.com> writes:\n\n> When working with multiple, unrelated (or loosly related) git repos,\n> there is often a need to locate all repos with uncommitted work and\n> perform some action on them (say, commit and push). Before this patch,\n> such tasks would require manually visiting all repositories, running\n> `git status` within each one and then decide what to do next.\n>\n> This mundane task can now be automated by e.g. `git for-each-repo --dirty\n> status`, which will find all non-bare git repositories below the current\n> directory (even nested ones), check if they are dirty (as defined by\n> `git diff --quiet && git diff --cached --quiet`), and for each dirty repo\n> print the path to the repo and then execute `git status` within the repo.\n>\n> The command also honours the option '--clean' which restricts the set of\n> repos to those which '--dirty' would skip, and '-x' which is used to\n> execute non-git commands.\n\nIt might make sense to internally use RUN_GIT_CMD flag when the\nfirst word of the command line is 'git' as an optimization, but \nI am not sure it is a good idea to force the end users to think\nwhen to use -x and when not to is a good idea.\n\nIn other words, I think\n\n     git for-each-repo -d diff --name-only\n     git for-each-repo -d -x ls '*.c'\n\nis less nice than letting the user say\n\n     git for-each-repo -d git diff --name-only\n     git for-each-repo -d ls '*.c'\n\n> Finally, the command to execute within each repo is optional. If none is\n> given, git-for-each-repo will just print the path to each repo found. And\n> since the command supports -z, this can be used for more advanced scripting\n> needs.\n\nIt amounts to the same thing, but I would rather describe it as:\n\n    To allow scripts to handle paths with shell-unsafe characters,\n    support \"-z\" to show paths with NUL termination.  Otherwise,\n    such paths are shown with the usual c-quoting.\n\nOne more thing that nobody brought up during the previous reviews is\nif we want to support subset of repositories by allowing the\nstandard pathspec match mechanism.  For example,\n\n\tgit for-each-repo -d git diff --name-only -- foo/ bar/b\\*z\n\nmight be a way to ask \"please find repositories match the given\npathspecs (i.e. foo/ bar/b\\*z) and run the command in the ones that\nare dirty\".  We would need to think about how to mark the end of the\ncommand though---we could borrow \\; from find(1), even though find\nis not the best example of the UI design.  I.e.\n\n\tgit for-each-repo -d git diff --name-only \\; [--] foo/ bar/b\\*z\n\nwith or without \"--\".\n\n> diff --git a/Documentation/git-for-each-repo.txt b/Documentation/git-for-each-repo.txt\n> new file mode 100644\n> index 0000000..fb12b3f\n> --- /dev/null\n> +++ b/Documentation/git-for-each-repo.txt\n> @@ -0,0 +1,71 @@\n> +git-for-each-repo(1)\n> +====================\n> +\n> +NAME\n> +----\n> +git-for-each-repo - Execute a git command in multiple non-bare repositories\n\nThere is a separate topic in flight that turns s/git/Git/ when we\nrefer to the system as a whole.  In any case, this is no longer\nlimited to \"execute a Git command\".\n\n\tFind non-bare Git repositories in subdirectories\n\nor\n\n\tFind or execute a command in non-bare Git repositories in subdirectories\n\n\nperhaps?\n\n> +SYNOPSIS\n> +--------\n> +[verse]\n> +'git for-each-repo' [-acdxz] [command]\n> +\n> +DESCRIPTION\n> +-----------\n> +The git-for-each-repo command is used to locate all non-bare git\n\nShould be sufficient to say s/is used to locate/locates/.\n\n> +repositories within the current directory tree, and optionally\n> +execute a git command in each of the found repos.\n\ns/a git command/a command/;\n\n> +OPTIONS\n> +-------\n> ...\n> +-x::\n> +\tExecute a genric (non-git) command in each repo.\n\nDrop this option.\n\n> +NOTES\n> +-----\n> +\n> +For the purpose of `git-for-each-repo`, a dirty worktree is defined as a\n> +worktree with uncommitted changes.\n\nIs it a definition that is different from usual?  If so why does it\nneed to be inconsistent with the rest of the system?\n\n> diff --git a/builtin/for-each-repo.c b/builtin/for-each-repo.c\n> new file mode 100644\n> index 0000000..9333ae0\n> --- /dev/null\n> +++ b/builtin/for-each-repo.c\n> @@ -0,0 +1,145 @@\n> +/*\n> + * \"git for-each-repo\" builtin command.\n> + *\n> + * Copyright (c) 2013 Lars Hjemli <hjemli@gmail.com>\n> + */\n> +#include \"cache.h\"\n> +#include \"color.h\"\n> +#include \"quote.h\"\n> +#include \"builtin.h\"\n> +#include \"run-command.h\"\n> +#include \"parse-options.h\"\n> +\n> +#define ALL 0\n> +#define DIRTY 1\n> +#define CLEAN 2\n> +\n> +static char *color = GIT_COLOR_NORMAL;\n> +static int eol = '\\n';\n> +static int match;\n> +static int runopt = RUN_GIT_CMD;\n> +\n> +static const char * const builtin_foreachrepo_usage[] = {\n> +\tN_(\"git for-each-repo [-acdxz] [cmd]\"),\n> +\tNULL\n> +};\n> +\n> +static struct option builtin_foreachrepo_options[] = {\n> +\tOPT_SET_INT('a', \"all\", &match, N_(\"match both clean and dirty repositories\"), ALL),\n> +\tOPT_SET_INT('c', \"clean\", &match, N_(\"only show clean repositories\"), CLEAN),\n> +\tOPT_SET_INT('d', \"dirty\", &match, N_(\"only show dirty repositories\"), DIRTY),\n> +\tOPT_SET_INT('x', NULL, &runopt, N_(\"execute generic (non-git) command\"), 0),\n> +\tOPT_SET_INT('z', NULL, &eol, N_(\"terminate each repo path with NUL character\"), 0),\n> +\tOPT_END(),\n> +};\n> +\n> +static int get_repo_state(const char *dir)\n> +{\n> +\tconst char *diffidx[] = {\"diff\", \"--quiet\", \"--cached\", NULL};\n> +\tconst char *diffwd[] = {\"diff\", \"--quiet\", NULL};\n> +\n> +\tif (run_command_v_opt_cd_env(diffidx, RUN_GIT_CMD, dir, NULL) != 0)\n> +\t\treturn DIRTY;\n> +\tif (run_command_v_opt_cd_env(diffwd, RUN_GIT_CMD, dir, NULL) != 0)\n> +\t\treturn DIRTY;\n> +\treturn CLEAN;\n> +}\n> +\n> +static void print_repo_path(const char *path, unsigned pretty)\n> +{\n> +\tif (path[0] == '.' && path[1] == '/')\n> +\t\tpath += 2;\n> +\tif (pretty)\n> +\t\tcolor_fprintf_ln(stdout, color, \"[%s]\", path);\n\nThis is shown before running a command in that repository.  I am of\ntwo minds.  It certainly is nice to be able to tell which repository\neach block of output lines comes from, and not requiring the command\nto do this themselves is a good default.  However, I wonder if people\nwould want to do something like this:\n\n\tgit for-each-repo sh -c '\n\t\tgit diff --name-only |\n\t\tsed -e \"s|^|$path/|\"\n        '\n\nto get a consolidated view, in a way similar to how \"submodule\nforeach\" can be used.  This unconditional output will get in the way\nfor such a use case.\n\nOh, that reminds me of another thing.  Perhaps we would want to\nexport the (relative) path to the found repository in some way to\nallow the commands to do this kind of thing in the first place?\n\"submodule foreach\" does this with $path, I think.\n\n> +\telse\n> +\t\twrite_name_quoted(path, stdout, eol);\n> +}\n\nNice.  Doubly nice that you do not hardcode \"color\" at this point\nbut made it into a separate variable.\n\n> +static void handle_repo(struct strbuf *path, const char **argv)\n> +{\n> +\tconst char *gitdir;\n> +\tint len;\n> +\n> +\tlen = path->len;\n> +\tstrbuf_addstr(path, \".git\");\n> +\tgitdir = resolve_gitdir(path->buf);\n> +\tstrbuf_setlen(path, len - 1);\n> +\tif (!gitdir)\n> +\t\tgoto done;\n> +\tif (match != ALL && match != get_repo_state(path->buf))\n> +\t\tgoto done;\n> +\tprint_repo_path(path->buf, *argv != NULL);\n> +\tif (*argv)\n> +\t\trun_command_v_opt_cd_env(argv, runopt, path->buf, NULL);\n> +done:\n> +\tstrbuf_addstr(path, \"/\");\n\nOK, you get \"$D/\" from the caller, make it \"$D/.git\" to call\nresolve_gitdir() with, turn it to \"$D\" before printing and runnning,\nand then add \"/\" back.  Slightly tricky but correct.\n\n> +static int walk(struct strbuf *path, int argc, const char **argv)\n> +{\n> +\tDIR *dir;\n> +\tstruct dirent *ent;\n> +\tstruct stat st;\n> +\tsize_t len;\n> +\tint has_dotgit = 0;\n> +\tstruct string_list list = STRING_LIST_INIT_DUP;\n> +\tstruct string_list_item *item;\n> +\n> +\tdir = opendir(path->buf);\n> +\tif (!dir)\n> +\t\treturn errno;\n> +\tstrbuf_addstr(path, \"/\");\n> +\tlen = path->len;\n> +\twhile ((ent = readdir(dir))) {\n> +\t\tif (!strcmp(ent->d_name, \".\") || !strcmp(ent->d_name, \"..\"))\n> +\t\t\tcontinue;\n> +\t\tif (!strcmp(ent->d_name, \".git\")) {\n> +\t\t\thas_dotgit = 1;\n> +\t\t\tcontinue;\n> +\t\t}\n> +\t\tswitch (DTYPE(ent)) {\n> +\t\tcase DT_UNKNOWN:\n> +\t\tcase DT_LNK:\n> +\t\t\t/* Use stat() to figure out if this path leads\n> +\t\t\t * to a directory - it's  not important if it's\n> +\t\t\t * a symlink which gets us there.\n> +\t\t\t */\n> +\t\t\tstrbuf_setlen(path, len);\n> +\t\t\tstrbuf_addstr(path, ent->d_name);\n> +\t\t\tif (stat(path->buf, &st) || !S_ISDIR(st.st_mode))\n> +\t\t\t\tbreak;\n> +\t\t\t/* fallthrough */\n> +\t\tcase DT_DIR:\n> +\t\t\tstring_list_append(&list, ent->d_name);\n> +\t\t\tbreak;\n> +\t\t}\n> +\t}\n> +\tclosedir(dir);\n> +\tstrbuf_setlen(path, len);\n> +\tif (has_dotgit)\n> +\t\thandle_repo(path, argv);\n> +\tsort_string_list(&list);\n> +\tfor_each_string_list_item(item, &list) {\n> +\t\tstrbuf_setlen(path, len);\n> +\t\tstrbuf_addstr(path, item->string);\n> +\t\twalk(path, argc, argv);\n> +\t}\n> +\tstring_list_clear(&list, 0);\n> +\treturn 0;\n> +}\n\nIs the \"collect-first-and-then-sort\" done so that the repositories\nare shown in a stable order regardless of the order in which\nreaddir() returns he entries?  I am not complaining, but being\ncurious.\n\n> diff --git a/t/t6400-for-each-repo.sh b/t/t6400-for-each-repo.sh\n\nThis command does not look like \"6 - the revision tree commands\" to\nme. \"7 - the porcelainish commands concerning the working tree\" or\n\"9 - the git tools\" may be a better match?\n\n> new file mode 100755\n> index 0000000..af02c0c\n> --- /dev/null\n> +++ b/t/t6400-for-each-repo.sh\n> @@ -0,0 +1,150 @@\n> +#!/bin/sh\n> +#\n> +# Copyright (c) 2013 Lars Hjemli\n> +#\n> +\n> +test_description='Test the git-for-each-repo command'\n> +\n> +. ./test-lib.sh\n> +\n> +qname=\"with\\\"quote\"\n> +qqname=\"\\\"with\\\\\\\"quote\\\"\"\n\nIf Windows does not have problems with paths with dq in it, then\nthis is fine, but I dunno.  Otherwise, you may want to exclude the\nc-quote testing from the main part of the test, and have a single\ntest that has prerequisite for filesystems that can do this at the\nend of the script.\n\n> +test_expect_success \"setup\" '\n> +\ttest_create_repo clean &&\n> +\t(cd clean && test_commit foo1) &&\n> +\tgit init --separate-git-dir=.cleansub clean/gitfile &&\n> +\t(cd clean/gitfile && test_commit foo2 && echo bar >>foo2.t) &&\n> +\ttest_create_repo dirty-idx &&\n> +\t(cd dirty-idx && test_commit foo3 && git rm foo3.t) &&\n> +\ttest_create_repo dirty-wt &&\n> +\t(cd dirty-wt && mv .git .linkedgit && ln -s .linkedgit .git &&\n\nSome platforms are symlink-challenged.  Can we do this test without\n\"ln -s\"?  SYMLINKS prereq wouldn't be very useful for the setup\nstep, as all the remaining tests won't work without setting up the\ntest scenario.\n\n> +\t  test_commit foo4 && rm foo4.t) &&\n> +\ttest_create_repo \"$qname\" &&\n> +\t(cd \"$qname\" && test_commit foo5) &&\n> +\tmkdir fakedir && mkdir fakedir/.git\n> +'\n> +\n> +test_expect_success \"without filtering, all repos are included\" '\n> +\techo \".\" >expect &&\n> +\techo \"clean\" >>expect &&\n> +\techo \"clean/gitfile\" >>expect &&\n> +\techo \"dirty-idx\" >>expect &&\n> +\techo \"dirty-wt\" >>expect &&\n> +\techo \"$qqname\" >>expect &&\n\nA single\n\n\tcat >expect <<-EOF\n        .\n        clean\n        clean/gitfile\n        ...\n\t$qqname\n\tEOF\n\nmay be a lot easier to read (likewise for all the \"expect\"\npreparation in the rest of the script).\n\n> +test_expect_success \"-z NUL-terminates each path\" '\n> +\techo \"(.)\" >expect &&\n> +\techo \"(clean)\" >>expect &&\n> +\techo \"(clean/gitfile)\" >>expect &&\n> +\techo \"(dirty-idx)\" >>expect &&\n> +\techo \"(dirty-wt)\" >>expect &&\n> +\techo \"($qname)\" >>expect &&\n> +\tgit for-each-repo -z | xargs -0 printf \"(%s)\\n\"  >actual &&\n\nThis needs prereq on \"xargs -0\", but because we know we do not have\nany string with Q in it in the expected list of repositories, it may\nbe simpler to do something like this:\n\n\techo \".QcleanQclean/gitfileQ...$qname\" >expect &&\n\tgit for-each-repo -z | tr \"\\0\" Q >actual &&\n\ttest_cmp expect actual\n\nThanks.\n"},{"id":"207971","messageId":"20130127194223.GR7498@serenity.lan","threadId":"32742","inReplyTo":"7vk3qywiqf.fsf@alter.siamese.dyndns.org","subject":"Re: [PATCH v4 1/2] for-each-repo: new command used for multi-repo operations","fromName":"John Keeping","fromEmail":"john@keeping.me.uk","sentAt":"2013-01-27T19:42:23Z","receivedAt":"2013-01-27T19:42:23Z","isPatch":true,"sender":{"key":"john@keeping.me.uk","avatar":"https://avatars.githubusercontent.com/u/1702081?v=4"},"body":"On Sun, Jan 27, 2013 at 11:04:08AM -0800, Junio C Hamano wrote:\n> One more thing that nobody brought up during the previous reviews is\n> if we want to support subset of repositories by allowing the\n> standard pathspec match mechanism.  For example,\n> \n> \tgit for-each-repo -d git diff --name-only -- foo/ bar/b\\*z\n> \n> might be a way to ask \"please find repositories match the given\n> pathspecs (i.e. foo/ bar/b\\*z) and run the command in the ones that\n> are dirty\".  We would need to think about how to mark the end of the\n> command though---we could borrow \\; from find(1), even though find\n> is not the best example of the UI design.  I.e.\n> \n> \tgit for-each-repo -d git diff --name-only \\; [--] foo/ bar/b\\*z\n> \n> with or without \"--\".\n\nWould it be better to make this a (multi-valued) option?\n\n    git for-each-repo -d --filter=foo/ --filter=bar/b\\*z git diff --name-only\n\nIt seems a lot simpler than trying to figure out how the command is\ngoing to handle '--' arguments.\n\n> Oh, that reminds me of another thing.  Perhaps we would want to\n> export the (relative) path to the found repository in some way to\n> allow the commands to do this kind of thing in the first place?\n> \"submodule foreach\" does this with $path, I think.\n\nI think $path is the only variable exported by \"submodule foreach\" which\nis applicable here, but it doesn't work on Windows, where environment\nvariables are case-insensitive.\n\nCommit 64394e3 (git-submodule.sh: Don't use $path variable in\neval_gettext string) changed \"submodule foreach\" to use $sm_path\ninternally although I notice that the documentation still uses $path.\n\nPerhaps $repo_path in this case?\n\n\nJohn\n"},{"id":"207972","messageId":"7v4ni2wgto.fsf@alter.siamese.dyndns.org","threadId":"32742","inReplyTo":"20130127194223.GR7498@serenity.lan","subject":"Re: [PATCH v4 1/2] for-each-repo: new command used for multi-repo operations","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-01-27T19:45:23Z","receivedAt":"2013-01-27T19:45:23Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"John Keeping <john@keeping.me.uk> writes:\n\n> On Sun, Jan 27, 2013 at 11:04:08AM -0800, Junio C Hamano wrote:\n>> One more thing that nobody brought up during the previous reviews is\n>> if we want to support subset of repositories by allowing the\n>> standard pathspec match mechanism.  For example,\n>> \n>> \tgit for-each-repo -d git diff --name-only -- foo/ bar/b\\*z\n>> \n>> might be a way to ask \"please find repositories match the given\n>> pathspecs (i.e. foo/ bar/b\\*z) and run the command in the ones that\n>> are dirty\".  We would need to think about how to mark the end of the\n>> command though---we could borrow \\; from find(1), even though find\n>> is not the best example of the UI design.  I.e.\n>> \n>> \tgit for-each-repo -d git diff --name-only \\; [--] foo/ bar/b\\*z\n>> \n>> with or without \"--\".\n>\n> Would it be better to make this a (multi-valued) option?\n>\n>     git for-each-repo -d --filter=foo/ --filter=bar/b\\*z git diff --name-only\n\nThe standard way to use filtering based on paths we have is to use\nthe pathspec parameters at the end of the commmand line.\n\nI see no reason for such an inconsistency with an option like --filter.\n\n>> Oh, that reminds me of another thing.  Perhaps we would want to\n>> export the (relative) path to the found repository in some way to\n>> allow the commands to do this kind of thing in the first place?\n>> \"submodule foreach\" does this with $path, I think.\n>\n> I think $path is the only variable exported by \"submodule foreach\" which\n> is applicable here, but it doesn't work on Windows, where environment\n> variables are case-insensitive.\n>\n> Commit 64394e3 (git-submodule.sh: Don't use $path variable in\n> eval_gettext string) changed \"submodule foreach\" to use $sm_path\n> internally although I notice that the documentation still uses $path.\n>\n> Perhaps $repo_path in this case?\n\nI do not care too deeply about the name, as long as the names used\nby both mechanisms are the same.\n"},{"id":"208067","messageId":"CAFXTnz6GTVgY4DK-FLELGF-Cb1=iNYyWcUsUiaUytGRx9Tr4Ow@mail.gmail.com","threadId":"32742","inReplyTo":"7vk3qywiqf.fsf@alter.siamese.dyndns.org","subject":"Re: [PATCH v4 1/2] for-each-repo: new command used for multi-repo operations","fromName":"Lars Hjemli","fromEmail":"hjemli@gmail.com","sentAt":"2013-01-28T07:50:41Z","receivedAt":"2013-01-28T07:50:41Z","isPatch":true,"sender":{"key":"hjemli@gmail.com","avatar":null},"body":"On Sun, Jan 27, 2013 at 8:04 PM, Junio C Hamano <gitster@pobox.com> wrote:\n> Lars Hjemli <hjemli@gmail.com> writes:\n>\n>> The command also honours the option '--clean' which restricts the set of\n>> repos to those which '--dirty' would skip, and '-x' which is used to\n>> execute non-git commands.\n>\n> It might make sense to internally use RUN_GIT_CMD flag when the\n> first word of the command line is 'git' as an optimization, but\n> I am not sure it is a good idea to force the end users to think\n> when to use -x and when not to is a good idea.\n>\n> In other words, I think\n>\n>      git for-each-repo -d diff --name-only\n>      git for-each-repo -d -x ls '*.c'\n>\n> is less nice than letting the user say\n>\n>      git for-each-repo -d git diff --name-only\n>      git for-each-repo -d ls '*.c'\n>\n\nThe 'git-for-each-repo' command was made to allow any git command to\nbe executed in all discovered repositories, and I've used it that way\nfor two years (in the form of a shell-script called 'git-all'). During\nthis time, I've occasionally thought about forking non-git commands\nbut the itch hasn't been strong enough for me to scratch. The point\nI'm trying to make is that to me, this command acts as a modifier for\nother git commands[1]. Having the possibility to execute non-git\ncommands would be nice, but it is not the main objective of this\ncommand.\n\n[1] The 'git -a' rewrite patch shows how I think about this command -\nit's just an option to the 'git' command, modifying the way any\nsubcommand is invoked (btw: I don't expect that patch to be applied\nsince 'git-all' was deemed to generic, so I'll just carry the patch in\nmy own tree).\n\n>> Finally, the command to execute within each repo is optional. If none is\n>> given, git-for-each-repo will just print the path to each repo found. And\n>> since the command supports -z, this can be used for more advanced scripting\n>> needs.\n>\n> It amounts to the same thing, but I would rather describe it as:\n>\n>     To allow scripts to handle paths with shell-unsafe characters,\n>     support \"-z\" to show paths with NUL termination.  Otherwise,\n>     such paths are shown with the usual c-quoting.\n>\n\nMuch better, thanks.\n\n\n> One more thing that nobody brought up during the previous reviews is\n> if we want to support subset of repositories by allowing the\n> standard pathspec match mechanism.  For example,\n>\n>         git for-each-repo -d git diff --name-only -- foo/ bar/b\\*z\n>\n> might be a way to ask \"please find repositories match the given\n> pathspecs (i.e. foo/ bar/b\\*z) and run the command in the ones that\n> are dirty\".  We would need to think about how to mark the end of the\n> command though---we could borrow \\; from find(1), even though find\n> is not the best example of the UI design.  I.e.\n>\n>         git for-each-repo -d git diff --name-only \\; [--] foo/ bar/b\\*z\n>\n> with or without \"--\".\n\nI don't think this would be very nice to end users, and would prefer\n--include and --exclude options (the latter is actually already a part\nof git-all, added by one of my coworkers).\n\n>> +NOTES\n>> +-----\n>> +\n>> +For the purpose of `git-for-each-repo`, a dirty worktree is defined as a\n>> +worktree with uncommitted changes.\n>\n> Is it a definition that is different from usual?  If so why does it\n> need to be inconsistent with the rest of the system?\n\nI just wanted to clarify what condition --dirty and --clean will\ncheck. In particular, the lack of checking for untracked files (which\ncould be added as yet another option).\n\n>> +static void print_repo_path(const char *path, unsigned pretty)\n>> +{\n>> +     if (path[0] == '.' && path[1] == '/')\n>> +             path += 2;\n>> +     if (pretty)\n>> +             color_fprintf_ln(stdout, color, \"[%s]\", path);\n>\n> This is shown before running a command in that repository.  I am of\n> two minds.  It certainly is nice to be able to tell which repository\n> each block of output lines comes from, and not requiring the command\n> to do this themselves is a good default.  However, I wonder if people\n> would want to do something like this:\n>\n>         git for-each-repo sh -c '\n>                 git diff --name-only |\n>                 sed -e \"s|^|$path/|\"\n>         '\n>\n> to get a consolidated view, in a way similar to how \"submodule\n> foreach\" can be used.  This unconditional output will get in the way\n> for such a use case.\n\nI guess -q/--quiet could be useful.\n\n>> +static int walk(struct strbuf *path, int argc, const char **argv)\n>> +{\n>> +     DIR *dir;\n>> +     struct dirent *ent;\n>> +     struct stat st;\n>> +     size_t len;\n>> +     int has_dotgit = 0;\n>> +     struct string_list list = STRING_LIST_INIT_DUP;\n>> +     struct string_list_item *item;\n>> +\n>> +     dir = opendir(path->buf);\n>> +     if (!dir)\n>> +             return errno;\n>> +     strbuf_addstr(path, \"/\");\n>> +     len = path->len;\n>> +     while ((ent = readdir(dir))) {\n>> +             if (!strcmp(ent->d_name, \".\") || !strcmp(ent->d_name, \"..\"))\n>> +                     continue;\n>> +             if (!strcmp(ent->d_name, \".git\")) {\n>> +                     has_dotgit = 1;\n>> +                     continue;\n>> +             }\n>> +             switch (DTYPE(ent)) {\n>> +             case DT_UNKNOWN:\n>> +             case DT_LNK:\n>> +                     /* Use stat() to figure out if this path leads\n>> +                      * to a directory - it's  not important if it's\n>> +                      * a symlink which gets us there.\n>> +                      */\n>> +                     strbuf_setlen(path, len);\n>> +                     strbuf_addstr(path, ent->d_name);\n>> +                     if (stat(path->buf, &st) || !S_ISDIR(st.st_mode))\n>> +                             break;\n>> +                     /* fallthrough */\n>> +             case DT_DIR:\n>> +                     string_list_append(&list, ent->d_name);\n>> +                     break;\n>> +             }\n>> +     }\n>> +     closedir(dir);\n>> +     strbuf_setlen(path, len);\n>> +     if (has_dotgit)\n>> +             handle_repo(path, argv);\n>> +     sort_string_list(&list);\n>> +     for_each_string_list_item(item, &list) {\n>> +             strbuf_setlen(path, len);\n>> +             strbuf_addstr(path, item->string);\n>> +             walk(path, argc, argv);\n>> +     }\n>> +     string_list_clear(&list, 0);\n>> +     return 0;\n>> +}\n>\n> Is the \"collect-first-and-then-sort\" done so that the repositories\n> are shown in a stable order regardless of the order in which\n> readdir() returns he entries?\n\nYes (writing the testcases demonstrated a need for predictable output).\n\n\n>> diff --git a/t/t6400-for-each-repo.sh b/t/t6400-for-each-repo.sh\n>\n> This command does not look like \"6 - the revision tree commands\" to\n> me. \"7 - the porcelainish commands concerning the working tree\" or\n> \"9 - the git tools\" may be a better match?\n\nOk, how about t9003?\n\n>> new file mode 100755\n>> index 0000000..af02c0c\n>> --- /dev/null\n>> +++ b/t/t6400-for-each-repo.sh\n>> @@ -0,0 +1,150 @@\n>> +#!/bin/sh\n>> +#\n>> +# Copyright (c) 2013 Lars Hjemli\n>> +#\n>> +\n>> +test_description='Test the git-for-each-repo command'\n>> +\n>> +. ./test-lib.sh\n>> +\n>> +qname=\"with\\\"quote\"\n>> +qqname=\"\\\"with\\\\\\\"quote\\\"\"\n>\n> If Windows does not have problems with paths with dq in it, then\n> this is fine, but I dunno.  Otherwise, you may want to exclude the\n> c-quote testing from the main part of the test, and have a single\n> test that has prerequisite for filesystems that can do this at the\n> end of the script.\n\nI'll check my patch on msysgit before resending.\n\n\n>> +test_expect_success \"setup\" '\n>> +     test_create_repo clean &&\n>> +     (cd clean && test_commit foo1) &&\n>> +     git init --separate-git-dir=.cleansub clean/gitfile &&\n>> +     (cd clean/gitfile && test_commit foo2 && echo bar >>foo2.t) &&\n>> +     test_create_repo dirty-idx &&\n>> +     (cd dirty-idx && test_commit foo3 && git rm foo3.t) &&\n>> +     test_create_repo dirty-wt &&\n>> +     (cd dirty-wt && mv .git .linkedgit && ln -s .linkedgit .git &&\n>\n> Some platforms are symlink-challenged.  Can we do this test without\n> \"ln -s\"?  SYMLINKS prereq wouldn't be very useful for the setup\n> step, as all the remaining tests won't work without setting up the\n> test scenario.\n\nI added this test to check the DT_UNKNOWN/DT_LINK case in walk() so\nI'd rather not drop it, but it can be moved into a standalone,\nSYMLINKS-enabled testcase.\n\nThanks for the review.\n\n-- \nlarsh\n"},{"id":"208069","messageId":"20130128081006.GA2434@elie.Belkin","threadId":"32742","inReplyTo":"CAFXTnz6GTVgY4DK-FLELGF-Cb1=iNYyWcUsUiaUytGRx9Tr4Ow@mail.gmail.com","subject":"Re: [PATCH v4 1/2] for-each-repo: new command used for multi-repo operations","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2013-01-28T08:10:06Z","receivedAt":"2013-01-28T08:10:06Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi,\n\nLars Hjemli wrote:\n\n> [1] The 'git -a' rewrite patch shows how I think about this command -\n> it's just an option to the 'git' command, modifying the way any\n> subcommand is invoked (btw: I don't expect that patch to be applied\n> since 'git-all' was deemed to generic, so I'll just carry the patch in\n> my own tree).\n\nAs one data point, 'git all' also seems too generic to me but 'git -a'\ndoesn't.  Intuition can be weird.\n\nSo if I ran the world, then having commands\n\n\tgit -a diff\n\nand\n\n\tgit for-each-repo git diff\n\ndo the same thing would be fine.  Of course I don't run the world. ;-)\n\n[...]\n>> One more thing that nobody brought up during the previous reviews is\n>> if we want to support subset of repositories by allowing the\n>> standard pathspec match mechanism.  For example,\n>>\n>>         git for-each-repo -d git diff --name-only -- foo/ bar/b\\*z\n>>\n>> might be a way to ask \"please find repositories match the given\n>> pathspecs (i.e. foo/ bar/b\\*z) and run the command in the ones that\n>> are dirty\".  We would need to think about how to mark the end of the\n>> command though---we could borrow \\; from find(1), even though find\n>> is not the best example of the UI design.\n\nIn most non-git commands, \"--\" represents an end-of-options marker,\nallowing arbitrary options afterward without having to worry about\nescaping minus signs.  So in that spirit, if this weren't a git\ncommand, I'd expect to be able to do\n\n\tfor-each-repo -- git diff -- '*.c'\n\nand have the second '--' passed verbatim to \"git diff\".\n\nUnfortunately in git (imitating commands like \"grep\", I suppose), \"--\"\nmeans \"paths start here\".  That means that with the git convention,\nthere is only one place to pass paths to a given command.\n\nTracing backwards: it would be really nice to be able to do\n\n\tgit for-each-repo git grep -e foo -- '*.c'\n\nor\n\n\tgit -a grep -e foo -- '*.c'\n\nFor this practical reason, it seems that paths listed after the '--'\nshould go to the command being run.  On the other hand, if I wanted to\nlimit my for-each-repo run to repositories in two subdirectories of\nthe cwd, I'd be tempted to try\n\n\tgit for-each-repo git grep -e foo -- src/ doc/\n\nAnd if I wanted to limit to different file types in the repositories\nunder each directory, it would be tempting to use\n\n\tgit for-each-repo git grep -e foo -- 'src/*.c' 'doc/*.txt'\n\nIs there a convention that would be usable today that is roughly\nforward-compatible with that?  (To throw an example out, requiring\nthat each pathspec passed to for-each-repo either starts with '*' or\ncontains no wildcards.)\n\nThanks,\nJonathan\n"},{"id":"208090","messageId":"CAFXTnz6zN0izx8S23JFww5niVD6x-r2e7TSthqZnempUrvAEWw@mail.gmail.com","threadId":"32742","inReplyTo":"20130128081006.GA2434@elie.Belkin","subject":"Re: [PATCH v4 1/2] for-each-repo: new command used for multi-repo operations","fromName":"Lars Hjemli","fromEmail":"hjemli@gmail.com","sentAt":"2013-01-28T17:11:06Z","receivedAt":"2013-01-28T17:11:06Z","isPatch":true,"sender":{"key":"hjemli@gmail.com","avatar":null},"body":"On Mon, Jan 28, 2013 at 9:10 AM, Jonathan Nieder <jrnieder@gmail.com> wrote:\n>\n> Lars Hjemli wrote:\n>\n>> [1] The 'git -a' rewrite patch shows how I think about this command -\n>> it's just an option to the 'git' command, modifying the way any\n>> subcommand is invoked (btw: I don't expect that patch to be applied\n>> since 'git-all' was deemed to generic, so I'll just carry the patch in\n>> my own tree).\n>\n> As one data point, 'git all' also seems too generic to me but 'git -a'\n> doesn't.  Intuition can be weird.\n>\n> So if I ran the world, then having commands\n>\n>         git -a diff\n>\n> and\n>\n>         git for-each-repo git diff\n>\n> do the same thing would be fine.  Of course I don't run the world. ;-)\n\nThis would make me very happy. Junio?\n\n--\nlarsh\n"},{"id":"208092","messageId":"7vham1xktx.fsf@alter.siamese.dyndns.org","threadId":"32742","inReplyTo":"20130128081006.GA2434@elie.Belkin","subject":"Re: [PATCH v4 1/2] for-each-repo: new command used for multi-repo operations","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-01-28T17:45:46Z","receivedAt":"2013-01-28T17:45:46Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jonathan Nieder <jrnieder@gmail.com> writes:\n\n> Tracing backwards: it would be really nice to be able to do\n>\n> \tgit for-each-repo git grep -e foo -- '*.c'\n\nThis is a very good example that shows the command that is run in\nthe repositories found may want pathspecs passed, but at the same\ntime, makes me realize that these repositories have to be fairly\nuniform for this command to be useful.  For example, 'src/*.c' or\n'inc/*.h' pathspecs wouldn't be useful unless majority if not all\nprojects the loop finds follow that layout convention.  This is not\nnecessarily limited to pathspecs, of course.  Unless they all have\nthe 'next' branch \"git for-each-repo checkout next\" would not work,\netc. etc.\n\nAs to the pathspec limiting to affect the loop itself, not the\nargument given to the command that is run, I don't think it is\nabsolutely needed; I am perfectly fine with declaring that\nfor-each-repo goes to repositories in all subdirectories without\nlimit, especially if doing so will make the UI issues we have to\ndeal with simpler.\n\nAs to the \"option to the command, not to the subcommand, -a option\",\nI have been assuming that it was a joke patch, but if \"git -a grep\"\nturns out to be really useful, \"submodule foreach\" that iterates\nover the submodules may also want to have such a short and sweet\nmechanism.  Between \"for-each-repo\" and \"submodule foreach\", I do\nnot yet have a strong opinion on which one deserves it more.\n\nCome to think of it, is there a reason why \"for-each-repo\" should\nnot be an extention to \"submodule foreach\"?  We can view this as\nvisiting repositories that _could_ be registered as a submodule, in\naddition to iterating over the registered submodules, no?\n\nIf these two are unified, then we do not have to even worry about\nwhich one deserves \"git -a\" more.\n"},{"id":"208104","messageId":"7vvcahw3zf.fsf@alter.siamese.dyndns.org","threadId":"32742","inReplyTo":"CAFXTnz6zN0izx8S23JFww5niVD6x-r2e7TSthqZnempUrvAEWw@mail.gmail.com","subject":"Re: [PATCH v4 1/2] for-each-repo: new command used for multi-repo operations","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-01-28T18:35:00Z","receivedAt":"2013-01-28T18:35:00Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Lars Hjemli <hjemli@gmail.com> writes:\n\n> On Mon, Jan 28, 2013 at 9:10 AM, Jonathan Nieder <jrnieder@gmail.com> wrote:\n>> ...\n>> So if I ran the world, then having commands\n>>\n>>         git -a diff\n>>\n>> and\n>>\n>>         git for-each-repo git diff\n>>\n>> do the same thing would be fine.  Of course I don't run the world. ;-)\n>\n> This would make me very happy. Junio?\n\nAhh, our mails crossed (rather, I responded to the other message I\nsaw before I saw this one).  I am not completely sold on \"git -a\"\nyet, but another worry I have is which one between \"submodule\nforeach\" and \"for-each-repo\" should use \"git -a\", if we decide that\nit is useful to the users to add it.\n"},{"id":"208105","messageId":"CAFXTnz6xBMo42jWdqahYX-bnTBucVmQpFPN29X8tGRd7L=g2wQ@mail.gmail.com","threadId":"32742","inReplyTo":"7vham1xktx.fsf@alter.siamese.dyndns.org","subject":"Re: [PATCH v4 1/2] for-each-repo: new command used for multi-repo operations","fromName":"Lars Hjemli","fromEmail":"hjemli@gmail.com","sentAt":"2013-01-28T18:35:21Z","receivedAt":"2013-01-28T18:35:21Z","isPatch":true,"sender":{"key":"hjemli@gmail.com","avatar":null},"body":"On Mon, Jan 28, 2013 at 6:45 PM, Junio C Hamano <gitster@pobox.com> wrote:\n> As to the pathspec limiting to affect the loop itself, not the\n> argument given to the command that is run, I don't think it is\n> absolutely needed; I am perfectly fine with declaring that\n> for-each-repo goes to repositories in all subdirectories without\n> limit, especially if doing so will make the UI issues we have to\n> deal with simpler.\n\nGood (since the relative path of each repo will be exported to the\nchild process, that process can perform path limiting when needed).\n\n\n> As to the \"option to the command, not to the subcommand, -a option\",\n> I have been assuming that it was a joke patch, but if \"git -a grep\"\n> turns out to be really useful, \"submodule foreach\" that iterates\n> over the submodules may also want to have such a short and sweet\n> mechanism.  Between \"for-each-repo\" and \"submodule foreach\", I do\n> not yet have a strong opinion on which one deserves it more.\n>\n> Come to think of it, is there a reason why \"for-each-repo\" should\n> not be an extention to \"submodule foreach\"?  We can view this as\n> visiting repositories that _could_ be registered as a submodule, in\n> addition to iterating over the registered submodules, no?\n\nYes, but I see some possible problems with that approach:\n-'git for-each-repo' does not need to be started from within a git worktree\n-'git for-each-repo' and 'git submodule foreach' have different\nsemantics for --dirty and --clean\n-'git for-each-repo' is in C because my 'git-all' shell script was\nhorribly slow on large directory trees (especially on windows)\n\nAll of these problems are probably solvable, but it would require\nquite some reworking of git-submodule.sh\n\n-- \nlarsh\n"},{"id":"208106","messageId":"7vr4l5w385.fsf@alter.siamese.dyndns.org","threadId":"32742","inReplyTo":"CAFXTnz6xBMo42jWdqahYX-bnTBucVmQpFPN29X8tGRd7L=g2wQ@mail.gmail.com","subject":"Re: [PATCH v4 1/2] for-each-repo: new command used for multi-repo operations","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-01-28T18:51:22Z","receivedAt":"2013-01-28T18:51:22Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Lars Hjemli <hjemli@gmail.com> writes:\n\n>> Come to think of it, is there a reason why \"for-each-repo\" should\n>> not be an extention to \"submodule foreach\"?  We can view this as\n>> visiting repositories that _could_ be registered as a submodule, in\n>> addition to iterating over the registered submodules, no?\n>\n> Yes, but I see some possible problems with that approach:\n> -'git for-each-repo' does not need to be started from within a git worktree\n\nTrue, but \"git submodule foreach --untracked\" can be told that it is\nOK not (yet) to be in any superproject, no?\n\n> -'git for-each-repo' and 'git submodule foreach' have different\n> semantics for --dirty and --clean\n\nThat could be a problem.  Is there a good reason why they should use\ndifferent definitions of dirtyness?\n\n> -'git for-each-repo' is in C because my 'git-all' shell script was\n> horribly slow on large directory trees (especially on windows)\n\nYour for-each-repo could be a good basis to build a new builtin\n\"submodule--foreach\" that is a pure helper hidden from the end users\nthat does both; cmd_foreach() in git-submodule.sh can simply delegate\nto it.\n\n> All of these problems are probably solvable, but it would require\n> quite some reworking of git-submodule.sh\n\nOf course some work is needed, but we do not have to convert all the\ncmd_foo in git-submodule.sh in one step.  For the purpose of\nunifying for-each-repo and submodule foreach to deliver the\nfunctionality sooner to the end users, we can go the route to add\nonly the submodule--foreach builtin, out of which we will get\nreusable implementation of module_list and other helper functions we\ncan leverage later to do other cmd_foo functions.\n"},{"id":"208110","messageId":"CAFXTnz4x1K1cwYKWUJK1ExCjGti8vaW_endoJg20wTYUf4C5NQ@mail.gmail.com","threadId":"32742","inReplyTo":"7vr4l5w385.fsf@alter.siamese.dyndns.org","subject":"Re: [PATCH v4 1/2] for-each-repo: new command used for multi-repo operations","fromName":"Lars Hjemli","fromEmail":"hjemli@gmail.com","sentAt":"2013-01-28T19:42:16Z","receivedAt":"2013-01-28T19:42:16Z","isPatch":true,"sender":{"key":"hjemli@gmail.com","avatar":null},"body":"On Mon, Jan 28, 2013 at 7:51 PM, Junio C Hamano <gitster@pobox.com> wrote:\n> Lars Hjemli <hjemli@gmail.com> writes:\n>\n>>> Come to think of it, is there a reason why \"for-each-repo\" should\n>>> not be an extention to \"submodule foreach\"?  We can view this as\n>>> visiting repositories that _could_ be registered as a submodule, in\n>>> addition to iterating over the registered submodules, no?\n>>\n>> Yes, but I see some possible problems with that approach:\n>> -'git for-each-repo' does not need to be started from within a git worktree\n>\n> True, but \"git submodule foreach --untracked\" can be told that it is\n> OK not (yet) to be in any superproject, no?\n\nYes.\n\n>\n>> -'git for-each-repo' and 'git submodule foreach' have different\n>> semantics for --dirty and --clean\n>\n> That could be a problem.  Is there a good reason why they should use\n> different definitions of dirtyness?\n\nI suspected that 'submodule foreach --dirty' might want to compare the\nHEAD sha1 in the submodule against the one recorded in the\nsuperproject (similar to what 'git submodule status' does), but such a\ncheck could be triggered by a different flag (e.g. --behind/--ahead or\nsomething similar).\n\n>> -'git for-each-repo' is in C because my 'git-all' shell script was\n>> horribly slow on large directory trees (especially on windows)\n>\n> Your for-each-repo could be a good basis to build a new builtin\n> \"submodule--foreach\" that is a pure helper hidden from the end users\n> that does both; cmd_foreach() in git-submodule.sh can simply delegate\n> to it.\n\nOk, I'll rework my patches in this direction. Thanks.\n\n--\nlarsh\n"},{"id":"208112","messageId":"5106DBB7.70007@web.de","threadId":"32742","inReplyTo":"7vr4l5w385.fsf@alter.siamese.dyndns.org","subject":"Re: [PATCH v4 1/2] for-each-repo: new command used for multi-repo operations","fromName":"Jens Lehmann","fromEmail":"jens.lehmann@web.de","sentAt":"2013-01-28T20:12:39Z","receivedAt":"2013-01-28T20:12:39Z","isPatch":true,"sender":{"key":"jens.lehmann@web.de","avatar":"https://avatars.githubusercontent.com/u/135220?v=4"},"body":"Am 28.01.2013 19:51, schrieb Junio C Hamano:\n> Lars Hjemli <hjemli@gmail.com> writes:\n> \n>>> Come to think of it, is there a reason why \"for-each-repo\" should\n>>> not be an extention to \"submodule foreach\"?  We can view this as\n>>> visiting repositories that _could_ be registered as a submodule, in\n>>> addition to iterating over the registered submodules, no?\n>>\n>> Yes, but I see some possible problems with that approach:\n>> -'git for-each-repo' does not need to be started from within a git worktree\n> \n> True, but \"git submodule foreach --untracked\" can be told that it is\n> OK not (yet) to be in any superproject, no?\n\nHmm, I'm not sure how that would work as it looks for gitlinks\nin the index which point to work tree paths.\n\n>> -'git for-each-repo' and 'git submodule foreach' have different\n>> semantics for --dirty and --clean\n\nI'm confused, what semantics of --dirty and --clean does current\n'git submodule foreach' have? I can't find any sign of it in the\ncurrent code ... did I miss something while skimming through this\nthread? Or are you talking about status and diff here?\n\n> That could be a problem.  Is there a good reason why they should use\n> different definitions of dirtyness?\n\nI don't see any (except of course for comparing a gitlink with the\nHEAD of the submodule, which is an additional condition that only\napplies to submodules). But I think the current for-each-repo\nproposal doesn't allow to traverse repos which contain untracked\ncontent (and it would be nice if the user could somehow combine\nthat with the current --dirty flag to have both in one go).\n\n>> -'git for-each-repo' is in C because my 'git-all' shell script was\n>> horribly slow on large directory trees (especially on windows)\n> \n> Your for-each-repo could be a good basis to build a new builtin\n> \"submodule--foreach\" that is a pure helper hidden from the end users\n> that does both; cmd_foreach() in git-submodule.sh can simply delegate\n> to it.\n\nI like that approach, because the operations are very similar from\nthe user's point of view. But please remember that internally they\nwould work differently, as submodule foreach walks the index and\nonly descends into those submodules that are populated (and contain\na .git directory or file) while for-each-repo scans the whole work\ntree, which makes it a more expensive operation.\n\n>> All of these problems are probably solvable, but it would require\n>> quite some reworking of git-submodule.sh\n> \n> Of course some work is needed, but we do not have to convert all the\n> cmd_foo in git-submodule.sh in one step.  For the purpose of\n> unifying for-each-repo and submodule foreach to deliver the\n> functionality sooner to the end users, we can go the route to add\n> only the submodule--foreach builtin, out of which we will get\n> reusable implementation of module_list and other helper functions we\n> can leverage later to do other cmd_foo functions.\n\nI really like that idea!\n"},{"id":"208115","messageId":"7vlibdvyh3.fsf@alter.siamese.dyndns.org","threadId":"32742","inReplyTo":"5106DBB7.70007@web.de","subject":"Re: [PATCH v4 1/2] for-each-repo: new command used for multi-repo operations","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-01-28T20:34:00Z","receivedAt":"2013-01-28T20:34:00Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jens Lehmann <Jens.Lehmann@web.de> writes:\n\n> Am 28.01.2013 19:51, schrieb Junio C Hamano:\n>> Lars Hjemli <hjemli@gmail.com> writes:\n>> \n>>>> Come to think of it, is there a reason why \"for-each-repo\" should\n>>>> not be an extention to \"submodule foreach\"?  We can view this as\n>>>> visiting repositories that _could_ be registered as a submodule, in\n>>>> addition to iterating over the registered submodules, no?\n>>>\n>>> Yes, but I see some possible problems with that approach:\n>>> -'git for-each-repo' does not need to be started from within a git worktree\n>> \n>> True, but \"git submodule foreach --untracked\" can be told that it is\n>> OK not (yet) to be in any superproject, no?\n>\n> Hmm, I'm not sure how that would work as it looks for gitlinks\n> in the index which point to work tree paths.\n\nI was imagining that \"foreach --untracked\" could go something like this:\n\n * If you are inside an existing git repository, read its index to\n   learn the gitlinks in the directory and its subdirectories.\n\n * Start from the current directory and recursively apply the\n   procedure in this step:\n\n   * Scan the directory and iterate over the ones that has \".git\" in\n     it:\n\n     * If it is a gitlinked one, show it, but do not descend into it\n       unless --recursive is given (e.g. you start from /home/jens,\n       find /home/jens/proj/ directory that has /home/jens/proj/.git\n       in it.  /home/jens/.git/index knows that it is a submodule of\n       the top-level superproject.  \"proj\" is handled, and it is up\n       to the --recursive option if its submodules are handled).\n\n     * If it is _not_ a gitlinked one, show it and descend into it\n       (e.g. /home/jens/ is not a repository or /home/jens/proj is\n       not a tracked submodule) to apply this procedure recursively.\n\nOf course, without --untracked, we have no need to iterate over the\nreaddir() return values; instead we just scan the index of the\ntop-level superproject.\n\n>>> -'git for-each-repo' and 'git submodule foreach' have different\n>>> semantics for --dirty and --clean\n>\n> I'm confused, what semantics of --dirty and --clean does current\n> 'git submodule foreach' have? I can't find any sign of it in the\n> current code ... did I miss something while skimming through this\n> thread? Or are you talking about status and diff here?\n\nI think Lars is hinting that \"submodule foreach\" could restrict its\noperation to a similar --dirty/--clean/--both option he has.  Of\ncourse, the command given to foreach can decide to become no-op by\ninspecting the submodule itself, so in that sense, --dirty/--clean\ncan be done without, but I think it would make sense to have it in\n\"submodule foreach\" even without the \"--untracked\" option.\n\n> But I think the current for-each-repo\n> proposal doesn't allow to traverse repos which contain untracked\n> content (and it would be nice if the user could somehow combine\n> that with the current --dirty flag to have both in one go).\n\nPerhaps.  I personally felt it was really strange that submodule\ndiff and status consider that it is a sin to have untracked and\nunignored cruft in the submodule working tree, though.\n"},{"id":"208126","messageId":"5106ECB6.9010801@web.de","threadId":"32742","inReplyTo":"7vlibdvyh3.fsf@alter.siamese.dyndns.org","subject":"Re: [PATCH v4 1/2] for-each-repo: new command used for multi-repo operations","fromName":"Jens Lehmann","fromEmail":"jens.lehmann@web.de","sentAt":"2013-01-28T21:25:10Z","receivedAt":"2013-01-28T21:25:10Z","isPatch":true,"sender":{"key":"jens.lehmann@web.de","avatar":"https://avatars.githubusercontent.com/u/135220?v=4"},"body":"Am 28.01.2013 21:34, schrieb Junio C Hamano:\n> Jens Lehmann <Jens.Lehmann@web.de> writes:\n> \n>> Am 28.01.2013 19:51, schrieb Junio C Hamano:\n>>> Lars Hjemli <hjemli@gmail.com> writes:\n>>>\n>>>>> Come to think of it, is there a reason why \"for-each-repo\" should\n>>>>> not be an extention to \"submodule foreach\"?  We can view this as\n>>>>> visiting repositories that _could_ be registered as a submodule, in\n>>>>> addition to iterating over the registered submodules, no?\n>>>>\n>>>> Yes, but I see some possible problems with that approach:\n>>>> -'git for-each-repo' does not need to be started from within a git worktree\n>>>\n>>> True, but \"git submodule foreach --untracked\" can be told that it is\n>>> OK not (yet) to be in any superproject, no?\n>>\n>> Hmm, I'm not sure how that would work as it looks for gitlinks\n>> in the index which point to work tree paths.\n> \n> I was imagining that \"foreach --untracked\" could go something like this:\n> \n>  * If you are inside an existing git repository, read its index to\n>    learn the gitlinks in the directory and its subdirectories.\n> \n>  * Start from the current directory and recursively apply the\n>    procedure in this step:\n> \n>    * Scan the directory and iterate over the ones that has \".git\" in\n>      it:\n> \n>      * If it is a gitlinked one, show it, but do not descend into it\n>        unless --recursive is given (e.g. you start from /home/jens,\n>        find /home/jens/proj/ directory that has /home/jens/proj/.git\n>        in it.  /home/jens/.git/index knows that it is a submodule of\n>        the top-level superproject.  \"proj\" is handled, and it is up\n>        to the --recursive option if its submodules are handled).\n> \n>      * If it is _not_ a gitlinked one, show it and descend into it\n>        (e.g. /home/jens/ is not a repository or /home/jens/proj is\n>        not a tracked submodule) to apply this procedure recursively.\n> \n> Of course, without --untracked, we have no need to iterate over the\n> readdir() return values; instead we just scan the index of the\n> top-level superproject.\n\nThanks for explaining, that makes tons of sense.\n\n>>>> -'git for-each-repo' and 'git submodule foreach' have different\n>>>> semantics for --dirty and --clean\n>>\n>> I'm confused, what semantics of --dirty and --clean does current\n>> 'git submodule foreach' have? I can't find any sign of it in the\n>> current code ... did I miss something while skimming through this\n>> thread? Or are you talking about status and diff here?\n> \n> I think Lars is hinting that \"submodule foreach\" could restrict its\n> operation to a similar --dirty/--clean/--both option he has.  Of\n> course, the command given to foreach can decide to become no-op by\n> inspecting the submodule itself, so in that sense, --dirty/--clean\n> can be done without, but I think it would make sense to have it in\n> \"submodule foreach\" even without the \"--untracked\" option.\n\nNice idea. E.g. that would help submodule users to easily script\na workflow which descends only into modified submodules to create\nbranches and push them there. Or to remove branches which were\ncreated everywhere only in those submodules that weren't changed.\n\n>> But I think the current for-each-repo\n>> proposal doesn't allow to traverse repos which contain untracked\n>> content (and it would be nice if the user could somehow combine\n>> that with the current --dirty flag to have both in one go).\n> \n> Perhaps.  I personally felt it was really strange that submodule\n> diff and status consider that it is a sin to have untracked and\n> unignored cruft in the submodule working tree, though.\n\nThe VCS we used at work before Git didn't show us any untracked\nfiles, which caused trouble on a regular basis as people were\nbreaking builds for others because they forgot to check in new\nfiles. That didn't happen with Git anymore, which was very cool.\nBut the problem reappeared as we started using submodules. Since\nI taught status and diff to show that we're happy again. So for\nus it was everything but strange ;-)\n\nBut for for-each-repo I would rather propose that modifications of\ntracked files can optionally and/or solely be used to pick the\nrepos. Maybe: --dirty=modified, --dirty=untracked and --dirty=both\nwith --dirty defaulting to modified?\n"},{"id":"208588","messageId":"7vip6860nu.fsf@alter.siamese.dyndns.org","threadId":"32742","inReplyTo":"5106ECB6.9010801@web.de","subject":"Re: [PATCH v4 1/2] for-each-repo: new command used for multi-repo operations","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-02-04T06:41:41Z","receivedAt":"2013-02-04T06:41:41Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jens Lehmann <Jens.Lehmann@web.de> writes:\n\n> Am 28.01.2013 21:34, schrieb Junio C Hamano:\n> ...\n>> I was imagining that \"foreach --untracked\" could go something like this:\n>> \n>>  * If you are inside an existing git repository, read its index to\n>>    learn the gitlinks in the directory and its subdirectories.\n>> \n>>  * Start from the current directory and recursively apply the\n>>    procedure in this step:\n>> \n>>    * Scan the directory and iterate over the ones that has \".git\" in\n>>      it:\n>> \n>>      * If it is a gitlinked one, show it, but do not descend into it\n>>        unless --recursive is given (e.g. you start from /home/jens,\n>>        find /home/jens/proj/ directory that has /home/jens/proj/.git\n>>        in it.  /home/jens/.git/index knows that it is a submodule of\n>>        the top-level superproject.  \"proj\" is handled, and it is up\n>>        to the --recursive option if its submodules are handled).\n>> \n>>      * If it is _not_ a gitlinked one, show it and descend into it\n>>        (e.g. /home/jens/ is not a repository or /home/jens/proj is\n>>        not a tracked submodule) to apply this procedure recursively.\n>> \n>> Of course, without --untracked, we have no need to iterate over the\n>> readdir() return values; instead we just scan the index of the\n>> top-level superproject.\n>\n> Thanks for explaining, that makes tons of sense.\n\nThere is a small thinko above, though, and I'd like to correct it\nbefore anybody takes the above too seriously as _the_ outline of the\ndesign and implements it to the letter.\n\nThe --recursive option should govern both a tracked submodule and an\nuntracked one.  When asking to list both existing submodules and\ndirectories that could become submodules, you should be able to say\n\n\t$ git submodule foreach --untracked\n\nto list the direct submodules and the directories with .git in them\nthat are not yet submodules of the top-level superproject, but the\nlatter is limited to those with no parent directories with .git in\nthem (other than the top-level of the working tree of the\nsuperproject).  With\n\n\t$ git submodule foreach --untracked --recursive\n\nyou would see submodules and their submodules recursively, and also\ndirectories with .git in them (i.e. candidates to become direct\nsubmodules of the superproject) and the directories with .git in\nthem inside such submodule candidates (i.e. candidates to become\ndirect submodules of the directories that could become direct\nsubmodules of the superproject) recursively.\n\nIf we set things up this way:\n\n\tmkdir -p a/b c/d &&\n\tfor d in . a a/b c c/d\n        do\n\t\tgit init $d &&\n                ( cd $d && git commit --allow-empty -m initial )\n\tdone &&\n        git add a &&\n        ( cd a && git add b )\n\nThe expected results for various combinations are:\n\n * \"git submodule foreach\" would visit 'a' and nothing else;\n * \"git submodule foreach --recursive\" would visit 'a' and 'a/b';\n * \"git submodule foreach --untracked\" would visit 'a' and 'c'; and\n * \"git submodule foreach --untracked --recursive\" would visit all four.\n"}]}