{"thread":{"id":"59034","subject":"[PATCH v5 0/6] submodule: parallelize diff","startedAt":"2023-01-04T21:54:46Z","lastAt":"2023-03-17T20:42:11Z","messageCount":86,"participants":["Calvin Wan","Glen Choo","Ævar Arnfjörð Bjarmason","Phillip Wood","Junio C Hamano"],"isPatch":true,"patchVersion":5,"patchTotal":6},"messages":[{"id":"469792","messageId":"20230104215415.1083526-1-calvinwan@google.com","threadId":"59034","inReplyTo":"https://lore.kernel.org/git/20221108184200.2813458-1-calvinwan@google.com/","subject":"[PATCH v5 0/6] submodule: parallelize diff","fromName":"Calvin Wan","fromEmail":"calvinwan@google.com","sentAt":"2023-01-04T21:54:09Z","receivedAt":"2023-01-04T21:54:46Z","isPatch":true,"sender":{"key":"calvinwan@google.com","avatar":"https://avatars.githubusercontent.com/u/92547554?v=4"},"body":"Original cover letter for context:\nhttps://lore.kernel.org/git/20221011232604.839941-1-calvinwan@google.com/\n\nThank you again everyone for the numerous reviews! For this reroll, I\nincorporated most of the feedback given, fixed a bug I found, and made\nsome stylistic refactors. I also added a new patch at the end that swaps\nthe serial implementation in is_submodule_modified for the new parallel\none. While I had patch 6 originally smushed with the previous one,\nthe diff came out not very reviewer friendly so it has been separated\nout.\n\nChanges since v4\n\n(Patch 1)\nThe code in run-command.c that calls duplicate_output_fn has been\ncleaned up and no longer passes a separate strbuf for the output. It\ninstead passes an offset that represents the starting point in the\noriginal strbuf.\n\n(Patch 5)\nMoved status parsing from status_duplicate_output to status_finish. In\npp_buffer_stderr::run-command.c, output is gathered by strbuf_read_once\nwhich reads 8192 bytes at once so a longer status message would error\nout during status parsing since part of it would be cut off. Therefore,\nstatus parsing must happen at the end of the process rather than in\nduplicate_output_fn (and has subsequently been moved).\n\n(Patch 6)\nNew patch swapping serial implementation in is_submodule_modified for\nthe new parallel one.\n\nCalvin Wan (6):\n run-command: add duplicate_output_fn to run_processes_parallel_opts\n submodule: strbuf variable rename\n submodule: move status parsing into function\n diff-lib: refactor match_stat_with_submodule\n diff-lib: parallelize run_diff_files for submodules\n submodule: call parallel code from serial status\n\n Documentation/config/submodule.txt | 12 ++\n diff-lib.c | 104 ++++++++++--\n run-command.c | 16 +-\n run-command.h | 27 ++++\n submodule.c | 250 ++++++++++++++++++++++-------\n submodule.h | 9 ++\n t/helper/test-run-command.c | 21 +++\n t/t0061-run-command.sh | 39 +++++\n t/t4027-diff-submodule.sh | 19 +++\n t/t7506-status-submodule.sh | 19 +++\n 10 files changed, 441 insertions(+), 75 deletions(-)\n\n-- \n2.39.0.314.g84b9a713c41-goog\n\n"},{"id":"469793","messageId":"20230104215415.1083526-2-calvinwan@google.com","threadId":"59034","inReplyTo":"https://lore.kernel.org/git/20221108184200.2813458-1-calvinwan@google.com/","subject":"[PATCH v5 1/6] run-command: add duplicate_output_fn to run_processes_parallel_opts","fromName":"Calvin Wan","fromEmail":"calvinwan@google.com","sentAt":"2023-01-04T21:54:10Z","receivedAt":"2023-01-04T21:54:53Z","isPatch":true,"sender":{"key":"calvinwan@google.com","avatar":"https://avatars.githubusercontent.com/u/92547554?v=4"},"body":"Add duplicate_output_fn as an optionally set function in\nrun_process_parallel_opts. If set, output from each child process is\ncopied and passed to the callback function whenever output from the\nchild process is buffered to allow for separate parsing.\n\nSigned-off-by: Calvin Wan \n---\n run-command.c | 16 ++++++++++++---\n run-command.h | 27 +++++++++++++++++++++++++\n t/helper/test-run-command.c | 21 ++++++++++++++++++++\n t/t0061-run-command.sh | 39 +++++++++++++++++++++++++++++++++++++\n 4 files changed, 100 insertions(+), 3 deletions(-)\n\ndiff --git a/run-command.c b/run-command.c\nindex 756f1839aa..cad88befe0 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -1526,6 +1526,9 @@ static void pp_init(struct parallel_processes *pp,\n \tif (!opts->get_next_task)\n \t\tBUG(\"you need to specify a get_next_task function\");\n \n+\tif (opts->duplicate_output && opts->ungroup)\n+\t\tBUG(\"duplicate_output and ungroup are incompatible with each other\");\n+\n \tCALLOC_ARRAY(pp->children, n);\n \tif (!opts->ungroup)\n \t\tCALLOC_ARRAY(pp->pfd, n);\n@@ -1645,14 +1648,21 @@ static void pp_buffer_stderr(struct parallel_processes *pp,\n \tfor (size_t i = 0; i < opts->processes; i++) {\n \t\tif (pp->children[i].state == GIT_CP_WORKING &&\n \t\t pp->pfd[i].revents & (POLLIN | POLLHUP)) {\n-\t\t\tint n = strbuf_read_once(&pp->children[i].err,\n-\t\t\t\t\t\t pp->children[i].process.err, 0);\n+\t\t\tssize_t n = strbuf_read_once(&pp->children[i].err,\n+\t\t\t\t\t\t pp->children[i].process.err, 0);\n \t\t\tif (n == 0) {\n \t\t\t\tclose(pp->children[i].process.err);\n \t\t\t\tpp->children[i].state = GIT_CP_WAIT_CLEANUP;\n-\t\t\t} else if (n < 0)\n+\t\t\t} else if (n < 0) {\n \t\t\t\tif (errno != EAGAIN)\n \t\t\t\t\tdie_errno(\"read\");\n+\t\t\t} else {\n+\t\t\t\tif (opts->duplicate_output)\n+\t\t\t\t\topts->duplicate_output(&pp->children[i].err,\n+\t\t\t\t\t strlen(pp->children[i].err.buf) - n,\n+\t\t\t\t\t opts->data,\n+\t\t\t\t\t pp->children[i].data);\n+\t\t\t}\n \t\t}\n \t}\n }\ndiff --git a/run-command.h b/run-command.h\nindex 072db56a4d..6dcf999f6c 100644\n--- a/run-command.h\n+++ b/run-command.h\n@@ -408,6 +408,27 @@ typedef int (*start_failure_fn)(struct strbuf *out,\n \t\t\t\tvoid *pp_cb,\n \t\t\t\tvoid *pp_task_cb);\n \n+/**\n+ * This callback is called whenever output from a child process is buffered\n+ * \n+ * See run_processes_parallel() below for a discussion of the \"struct\n+ * strbuf *out\" parameter.\n+ * \n+ * The offset refers to the number of bytes originally in \"out\" before\n+ * the output from the child process was buffered. Therefore, the buffer\n+ * range, \"out + buf\" to the end of \"out\", would contain the buffer of\n+ * the child process output.\n+ *\n+ * pp_cb is the callback cookie as passed into run_processes_parallel,\n+ * pp_task_cb is the callback cookie as passed into get_next_task_fn.\n+ *\n+ * This function is incompatible with \"ungroup\"\n+ */\n+typedef void (*duplicate_output_fn)(struct strbuf *out,\n+\t\t\t\t size_t offset,\n+\t\t\t\t void *pp_cb,\n+\t\t\t\t void *pp_task_cb);\n+\n /**\n * This callback is called on every child process that finished processing.\n *\n@@ -461,6 +482,12 @@ struct run_process_parallel_opts\n \t */\n \tstart_failure_fn start_failure;\n \n+\t/**\n+\t * duplicate_output: See duplicate_output_fn() above. This should be\n+\t * NULL unless process specific output is needed\n+\t */\n+\tduplicate_output_fn duplicate_output;\n+\n \t/**\n \t * task_finished: See task_finished_fn() above. This can be\n \t * NULL to omit any special handling.\ndiff --git a/t/helper/test-run-command.c b/t/helper/test-run-command.c\nindex 3ecb830f4a..ffd3cd0045 100644\n--- a/t/helper/test-run-command.c\n+++ b/t/helper/test-run-command.c\n@@ -52,6 +52,21 @@ static int no_job(struct child_process *cp,\n \treturn 0;\n }\n \n+static void duplicate_output(struct strbuf *out,\n+\t\t\tsize_t offset,\n+\t\t\tvoid *pp_cb UNUSED,\n+\t\t\tvoid *pp_task_cb UNUSED)\n+{\n+\tstruct string_list list = STRING_LIST_INIT_DUP;\n+\n+\tstring_list_split(&list, out->buf + offset, '\\n', -1);\n+\tfor (size_t i = 0; i < list.nr; i++) {\n+\t\tif (strlen(list.items[i].string) > 0)\n+\t\t\tfprintf(stderr, \"duplicate_output: %s\\n\", list.items[i].string);\n+\t}\n+\tstring_list_clear(&list, 0);\n+}\n+\n static int task_finished(int result,\n \t\t\t struct strbuf *err,\n \t\t\t void *pp_cb,\n@@ -439,6 +454,12 @@ int cmd__run_command(int argc, const char **argv)\n \t\topts.ungroup = 1;\n \t}\n \n+\tif (!strcmp(argv[1], \"--duplicate-output\")) {\n+\t\targv += 1;\n+\t\targc -= 1;\n+\t\topts.duplicate_output = duplicate_output;\n+\t}\n+\n \tjobs = atoi(argv[2]);\n \tstrvec_clear(&proc.args);\n \tstrvec_pushv(&proc.args, (const char **)argv + 3);\ndiff --git a/t/t0061-run-command.sh b/t/t0061-run-command.sh\nindex e2411f6a9b..879e536638 100755\n--- a/t/t0061-run-command.sh\n+++ b/t/t0061-run-command.sh\n@@ -135,6 +135,15 @@ test_expect_success 'run_command runs in parallel with more jobs available than\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'run_command runs in parallel with more jobs available than tasks --duplicate-output' '\n+\ttest-tool run-command --duplicate-output run-command-parallel 5 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n+\ttest_must_be_empty out &&\n+\ttest 4 = $(grep -c \"duplicate_output: Hello\" err) &&\n+\ttest 4 = $(grep -c \"duplicate_output: World\" err) &&\n+\tsed \"/duplicate_output/d\" err > err1 &&\n+\ttest_cmp expect err1\n+'\n+\n test_expect_success 'run_command runs ungrouped in parallel with more jobs available than tasks' '\n \ttest-tool run-command --ungroup run-command-parallel 5 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n \ttest_line_count = 8 out &&\n@@ -147,6 +156,15 @@ test_expect_success 'run_command runs in parallel with as many jobs as tasks' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'run_command runs in parallel with as many jobs as tasks --duplicate-output' '\n+\ttest-tool run-command --duplicate-output run-command-parallel 4 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n+\ttest_must_be_empty out &&\n+\ttest 4 = $(grep -c \"duplicate_output: Hello\" err) &&\n+\ttest 4 = $(grep -c \"duplicate_output: World\" err) &&\n+\tsed \"/duplicate_output/d\" err > err1 &&\n+\ttest_cmp expect err1\n+'\n+\n test_expect_success 'run_command runs ungrouped in parallel with as many jobs as tasks' '\n \ttest-tool run-command --ungroup run-command-parallel 4 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n \ttest_line_count = 8 out &&\n@@ -159,6 +177,15 @@ test_expect_success 'run_command runs in parallel with more tasks than jobs avai\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'run_command runs in parallel with more tasks than jobs available --duplicate-output' '\n+\ttest-tool run-command --duplicate-output run-command-parallel 3 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n+\ttest_must_be_empty out &&\n+\ttest 4 = $(grep -c \"duplicate_output: Hello\" err) &&\n+\ttest 4 = $(grep -c \"duplicate_output: World\" err) &&\n+\tsed \"/duplicate_output/d\" err > err1 &&\n+\ttest_cmp expect err1\n+'\n+\n test_expect_success 'run_command runs ungrouped in parallel with more tasks than jobs available' '\n \ttest-tool run-command --ungroup run-command-parallel 3 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n \ttest_line_count = 8 out &&\n@@ -180,6 +207,12 @@ test_expect_success 'run_command is asked to abort gracefully' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'run_command is asked to abort gracefully --duplicate-output' '\n+\ttest-tool run-command --duplicate-output run-command-abort 3 false >out 2>err &&\n+\ttest_must_be_empty out &&\n+\ttest_cmp expect err\n+'\n+\n test_expect_success 'run_command is asked to abort gracefully (ungroup)' '\n \ttest-tool run-command --ungroup run-command-abort 3 false >out 2>err &&\n \ttest_must_be_empty out &&\n@@ -196,6 +229,12 @@ test_expect_success 'run_command outputs ' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'run_command outputs --duplicate-output' '\n+\ttest-tool run-command --duplicate-output run-command-no-jobs 3 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n+\ttest_must_be_empty out &&\n+\ttest_cmp expect err\n+'\n+\n test_expect_success 'run_command outputs (ungroup) ' '\n \ttest-tool run-command --ungroup run-command-no-jobs 3 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n \ttest_must_be_empty out &&\n-- \n2.39.0.314.g84b9a713c41-goog\n\n"},{"id":"469794","messageId":"20230104215415.1083526-3-calvinwan@google.com","threadId":"59034","inReplyTo":"https://lore.kernel.org/git/20221108184200.2813458-1-calvinwan@google.com/","subject":"[PATCH v5 2/6] submodule: strbuf variable rename","fromName":"Calvin Wan","fromEmail":"calvinwan@google.com","sentAt":"2023-01-04T21:54:11Z","receivedAt":"2023-01-04T21:54:56Z","isPatch":true,"sender":{"key":"calvinwan@google.com","avatar":"https://avatars.githubusercontent.com/u/92547554?v=4"},"body":"A prepatory change for a future patch that moves the status parsing\nlogic to a separate function.\n\nSigned-off-by: Calvin Wan \n---\n submodule.c | 23 +++++++++++++----------\n 1 file changed, 13 insertions(+), 10 deletions(-)\n\ndiff --git a/submodule.c b/submodule.c\nindex fae24ef34a..faf37c1101 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -1906,25 +1906,28 @@ unsigned is_submodule_modified(const char *path, int ignore_untracked)\n \n \tfp = xfdopen(cp.out, \"r\");\n \twhile (strbuf_getwholeline(&buf, fp, '\\n') != EOF) {\n+\t\tchar *str = buf.buf;\n+\t\tconst size_t len = buf.len;\n+\n \t\t/* regular untracked files */\n-\t\tif (buf.buf[0] == '?')\n+\t\tif (str[0] == '?')\n \t\t\tdirty_submodule |= DIRTY_SUBMODULE_UNTRACKED;\n \n-\t\tif (buf.buf[0] == 'u' ||\n-\t\t buf.buf[0] == '1' ||\n-\t\t buf.buf[0] == '2') {\n+\t\tif (str[0] == 'u' ||\n+\t\t str[0] == '1' ||\n+\t\t str[0] == '2') {\n \t\t\t/* T = line type, XY = status, SSSS = submodule state */\n-\t\t\tif (buf.len < strlen(\"T XY SSSS\"))\n+\t\t\tif (len < strlen(\"T XY SSSS\"))\n \t\t\t\tBUG(\"invalid status --porcelain=2 line %s\",\n-\t\t\t\t buf.buf);\n+\t\t\t\t str);\n \n-\t\t\tif (buf.buf[5] == 'S' && buf.buf[8] == 'U')\n+\t\t\tif (str[5] == 'S' && str[8] == 'U')\n \t\t\t\t/* nested untracked file */\n \t\t\t\tdirty_submodule |= DIRTY_SUBMODULE_UNTRACKED;\n \n-\t\t\tif (buf.buf[0] == 'u' ||\n-\t\t\t buf.buf[0] == '2' ||\n-\t\t\t memcmp(buf.buf + 5, \"S..U\", 4))\n+\t\t\tif (str[0] == 'u' ||\n+\t\t\t str[0] == '2' ||\n+\t\t\t memcmp(str + 5, \"S..U\", 4))\n \t\t\t\t/* other change */\n \t\t\t\tdirty_submodule |= DIRTY_SUBMODULE_MODIFIED;\n \t\t}\n-- \n2.39.0.314.g84b9a713c41-goog\n\n"},{"id":"469795","messageId":"20230104215415.1083526-4-calvinwan@google.com","threadId":"59034","inReplyTo":"https://lore.kernel.org/git/20221108184200.2813458-1-calvinwan@google.com/","subject":"[PATCH v5 3/6] submodule: move status parsing into function","fromName":"Calvin Wan","fromEmail":"calvinwan@google.com","sentAt":"2023-01-04T21:54:12Z","receivedAt":"2023-01-04T21:55:04Z","isPatch":true,"sender":{"key":"calvinwan@google.com","avatar":"https://avatars.githubusercontent.com/u/92547554?v=4"},"body":"A future patch requires the ability to parse the output of git\nstatus --porcelain=2. Move parsing code from is_submodule_modified to\nparse_status_porcelain.\n\nSigned-off-by: Calvin Wan \n---\n submodule.c | 74 ++++++++++++++++++++++++++++++-----------------------\n 1 file changed, 42 insertions(+), 32 deletions(-)\n\ndiff --git a/submodule.c b/submodule.c\nindex faf37c1101..768d4b4cd7 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -1870,6 +1870,45 @@ int fetch_submodules(struct repository *r,\n \treturn spf.result;\n }\n \n+static int parse_status_porcelain(char *str, size_t len,\n+\t\t\t\t unsigned *dirty_submodule,\n+\t\t\t\t int ignore_untracked)\n+{\n+\t/* regular untracked files */\n+\tif (str[0] == '?')\n+\t\t*dirty_submodule |= DIRTY_SUBMODULE_UNTRACKED;\n+\n+\tif (str[0] == 'u' ||\n+\t str[0] == '1' ||\n+\t str[0] == '2') {\n+\t\t/* T = line type, XY = status, SSSS = submodule state */\n+\t\tif (len < strlen(\"T XY SSSS\"))\n+\t\t\tBUG(\"invalid status --porcelain=2 line %s\",\n+\t\t\t str);\n+\n+\t\tif (str[5] == 'S' && str[8] == 'U')\n+\t\t\t/* nested untracked file */\n+\t\t\t*dirty_submodule |= DIRTY_SUBMODULE_UNTRACKED;\n+\n+\t\tif (str[0] == 'u' ||\n+\t\t str[0] == '2' ||\n+\t\t memcmp(str + 5, \"S..U\", 4))\n+\t\t\t/* other change */\n+\t\t\t*dirty_submodule |= DIRTY_SUBMODULE_MODIFIED;\n+\t}\n+\n+\tif ((*dirty_submodule & DIRTY_SUBMODULE_MODIFIED) &&\n+\t ((*dirty_submodule & DIRTY_SUBMODULE_UNTRACKED) ||\n+\t ignore_untracked)) {\n+\t\t/*\n+\t\t* We're not interested in any further information from\n+\t\t* the child any more, neither output nor its exit code.\n+\t\t*/\n+\t\treturn 1;\n+\t}\n+\treturn 0;\n+}\n+\n unsigned is_submodule_modified(const char *path, int ignore_untracked)\n {\n \tstruct child_process cp = CHILD_PROCESS_INIT;\n@@ -1909,39 +1948,10 @@ unsigned is_submodule_modified(const char *path, int ignore_untracked)\n \t\tchar *str = buf.buf;\n \t\tconst size_t len = buf.len;\n \n-\t\t/* regular untracked files */\n-\t\tif (str[0] == '?')\n-\t\t\tdirty_submodule |= DIRTY_SUBMODULE_UNTRACKED;\n-\n-\t\tif (str[0] == 'u' ||\n-\t\t str[0] == '1' ||\n-\t\t str[0] == '2') {\n-\t\t\t/* T = line type, XY = status, SSSS = submodule state */\n-\t\t\tif (len < strlen(\"T XY SSSS\"))\n-\t\t\t\tBUG(\"invalid status --porcelain=2 line %s\",\n-\t\t\t\t str);\n-\n-\t\t\tif (str[5] == 'S' && str[8] == 'U')\n-\t\t\t\t/* nested untracked file */\n-\t\t\t\tdirty_submodule |= DIRTY_SUBMODULE_UNTRACKED;\n-\n-\t\t\tif (str[0] == 'u' ||\n-\t\t\t str[0] == '2' ||\n-\t\t\t memcmp(str + 5, \"S..U\", 4))\n-\t\t\t\t/* other change */\n-\t\t\t\tdirty_submodule |= DIRTY_SUBMODULE_MODIFIED;\n-\t\t}\n-\n-\t\tif ((dirty_submodule & DIRTY_SUBMODULE_MODIFIED) &&\n-\t\t ((dirty_submodule & DIRTY_SUBMODULE_UNTRACKED) ||\n-\t\t ignore_untracked)) {\n-\t\t\t/*\n-\t\t\t * We're not interested in any further information from\n-\t\t\t * the child any more, neither output nor its exit code.\n-\t\t\t */\n-\t\t\tignore_cp_exit_code = 1;\n+\t\tignore_cp_exit_code = parse_status_porcelain(str, len, &dirty_submodule,\n+\t\t\t\t\t\t\t ignore_untracked);\n+\t\tif (ignore_cp_exit_code)\n \t\t\tbreak;\n-\t\t}\n \t}\n \tfclose(fp);\n \n-- \n2.39.0.314.g84b9a713c41-goog\n\n"},{"id":"469796","messageId":"20230104215415.1083526-5-calvinwan@google.com","threadId":"59034","inReplyTo":"https://lore.kernel.org/git/20221108184200.2813458-1-calvinwan@google.com/","subject":"[PATCH v5 4/6] diff-lib: refactor match_stat_with_submodule","fromName":"Calvin Wan","fromEmail":"calvinwan@google.com","sentAt":"2023-01-04T21:54:13Z","receivedAt":"2023-01-04T21:55:06Z","isPatch":true,"sender":{"key":"calvinwan@google.com","avatar":"https://avatars.githubusercontent.com/u/92547554?v=4"},"body":"Flatten out the if statements in match_stat_with_submodule so the\nlogic is more readable and easier for future patches to add to.\norig_flags didn't need to be set if the cache entry wasn't a\nGITLINK so defer setting it.\n\nSigned-off-by: Calvin Wan \n---\n diff-lib.c | 28 +++++++++++++++++-----------\n 1 file changed, 17 insertions(+), 11 deletions(-)\n\ndiff --git a/diff-lib.c b/diff-lib.c\nindex dec040c366..64583fded0 100644\n--- a/diff-lib.c\n+++ b/diff-lib.c\n@@ -73,18 +73,24 @@ static int match_stat_with_submodule(struct diff_options *diffopt,\n \t\t\t\t unsigned *dirty_submodule)\n {\n \tint changed = ie_match_stat(diffopt->repo->index, ce, st, ce_option);\n-\tif (S_ISGITLINK(ce->ce_mode)) {\n-\t\tstruct diff_flags orig_flags = diffopt->flags;\n-\t\tif (!diffopt->flags.override_submodule_config)\n-\t\t\tset_diffopt_flags_from_submodule_config(diffopt, ce->name);\n-\t\tif (diffopt->flags.ignore_submodules)\n-\t\t\tchanged = 0;\n-\t\telse if (!diffopt->flags.ignore_dirty_submodules &&\n-\t\t\t (!changed || diffopt->flags.dirty_submodules))\n-\t\t\t*dirty_submodule = is_submodule_modified(ce->name,\n-\t\t\t\t\t\t\t\t diffopt->flags.ignore_untracked_in_submodules);\n-\t\tdiffopt->flags = orig_flags;\n+\tstruct diff_flags orig_flags;\n+\n+\tif (!S_ISGITLINK(ce->ce_mode))\n+\t\treturn changed;\n+\n+\torig_flags = diffopt->flags;\n+\tif (!diffopt->flags.override_submodule_config)\n+\t\tset_diffopt_flags_from_submodule_config(diffopt, ce->name);\n+\tif (diffopt->flags.ignore_submodules) {\n+\t\tchanged = 0;\n+\t\tgoto cleanup;\n \t}\n+\tif (!diffopt->flags.ignore_dirty_submodules &&\n+\t (!changed || diffopt->flags.dirty_submodules))\n+\t\t*dirty_submodule = is_submodule_modified(ce->name,\n+\t\t\t\t\t diffopt->flags.ignore_untracked_in_submodules);\n+cleanup:\n+\tdiffopt->flags = orig_flags;\n \treturn changed;\n }\n \n-- \n2.39.0.314.g84b9a713c41-goog\n\n"},{"id":"469797","messageId":"20230104215415.1083526-6-calvinwan@google.com","threadId":"59034","inReplyTo":"https://lore.kernel.org/git/20221108184200.2813458-1-calvinwan@google.com/","subject":"[PATCH v5 5/6] diff-lib: parallelize run_diff_files for submodules","fromName":"Calvin Wan","fromEmail":"calvinwan@google.com","sentAt":"2023-01-04T21:54:14Z","receivedAt":"2023-01-04T21:55:08Z","isPatch":true,"sender":{"key":"calvinwan@google.com","avatar":"https://avatars.githubusercontent.com/u/92547554?v=4"},"body":"During the iteration of the index entries in run_diff_files, whenever\na submodule is found and needs its status checked, a subprocess is\nspawned for it. Instead of spawning the subprocess immediately and\nwaiting for its completion to continue, hold onto all submodules and\nrelevant information in a list. Then use that list to create tasks for\nrun_processes_parallel. Subprocess output is duplicated and passed to\nstatus_pipe_output which stores it to be parsed on completion of the\nsubprocess.\n\nAdd config option submodule.diffJobs to set the maximum number\nof parallel jobs. The option defaults to 1 if unset. If set to 0, the\nnumber of jobs is set to online_cpus().\n\nSince run_diff_files is called from many different commands, I chose\nto grab the config option in the function rather than adding variables\nto every git command and then figuring out how to pass them all in.\n\nSigned-off-by: Calvin Wan \n---\n Documentation/config/submodule.txt | 12 +++\n diff-lib.c | 84 +++++++++++++--\n submodule.c | 168 +++++++++++++++++++++++++++++\n submodule.h | 9 ++\n t/t4027-diff-submodule.sh | 19 ++++\n t/t7506-status-submodule.sh | 19 ++++\n 6 files changed, 304 insertions(+), 7 deletions(-)\n\ndiff --git a/Documentation/config/submodule.txt b/Documentation/config/submodule.txt\nindex 6490527b45..3209eb8117 100644\n--- a/Documentation/config/submodule.txt\n+++ b/Documentation/config/submodule.txt\n@@ -93,6 +93,18 @@ submodule.fetchJobs::\n \tin parallel. A value of 0 will give some reasonable default.\n \tIf unset, it defaults to 1.\n \n+submodule.diffJobs::\n+\tSpecifies how many submodules are diffed at the same time. A\n+\tpositive integer allows up to that number of submodules diffed\n+\tin parallel. A value of 0 will give some reasonable default.\n+\tIf unset, it defaults to 1. The diff operation is used by many\n+\tother git commands such as add, merge, diff, status, stash and\n+\tmore. Note that the expensive part of the diff operation is\n+\treading the index from cache or memory. Therefore multiple jobs\n+\tmay be detrimental to performance if your hardware does not\n+\tsupport parallel reads or if the number of jobs greatly exceeds\n+\tthe amount of supported reads.\n+\n submodule.alternateLocation::\n \tSpecifies how the submodules obtain alternates when submodules are\n \tcloned. Possible values are `no`, `superproject`.\ndiff --git a/diff-lib.c b/diff-lib.c\nindex 64583fded0..f51ea07f36 100644\n--- a/diff-lib.c\n+++ b/diff-lib.c\n@@ -14,6 +14,7 @@\n #include \"dir.h\"\n #include \"fsmonitor.h\"\n #include \"commit-reach.h\"\n+#include \"config.h\"\n \n /*\n * diff-files\n@@ -65,18 +66,23 @@ static int check_removed(const struct index_state *istate, const struct cache_en\n * Return 1 when changes are detected, 0 otherwise. If the DIRTY_SUBMODULES\n * option is set, the caller does not only want to know if a submodule is\n * modified at all but wants to know all the conditions that are met (new\n- * commits, untracked content and/or modified content).\n+ * commits, untracked content and/or modified content). If\n+ * defer_submodule_status bit is set, dirty_submodule will be left to the\n+ * caller to set. defer_submodule_status can also be set to 0 in this\n+ * function if there is no need to check if the submodule is modified.\n */\n static int match_stat_with_submodule(struct diff_options *diffopt,\n \t\t\t\t const struct cache_entry *ce,\n \t\t\t\t struct stat *st, unsigned ce_option,\n-\t\t\t\t unsigned *dirty_submodule)\n+\t\t\t\t unsigned *dirty_submodule, int *defer_submodule_status,\n+\t\t\t\t unsigned *ignore_untracked)\n {\n \tint changed = ie_match_stat(diffopt->repo->index, ce, st, ce_option);\n \tstruct diff_flags orig_flags;\n+\tint defer = 0;\n \n \tif (!S_ISGITLINK(ce->ce_mode))\n-\t\treturn changed;\n+\t\tgoto ret;\n \n \torig_flags = diffopt->flags;\n \tif (!diffopt->flags.override_submodule_config)\n@@ -86,11 +92,20 @@ static int match_stat_with_submodule(struct diff_options *diffopt,\n \t\tgoto cleanup;\n \t}\n \tif (!diffopt->flags.ignore_dirty_submodules &&\n-\t (!changed || diffopt->flags.dirty_submodules))\n-\t\t*dirty_submodule = is_submodule_modified(ce->name,\n+\t (!changed || diffopt->flags.dirty_submodules)) {\n+\t\tif (defer_submodule_status && *defer_submodule_status) {\n+\t\t\tdefer = 1;\n+\t\t\t*ignore_untracked = diffopt->flags.ignore_untracked_in_submodules;\n+\t\t} else {\n+\t\t\t*dirty_submodule = is_submodule_modified(ce->name,\n \t\t\t\t\t diffopt->flags.ignore_untracked_in_submodules);\n+\t\t}\n+\t}\n cleanup:\n \tdiffopt->flags = orig_flags;\n+ret:\n+\tif (defer_submodule_status)\n+\t\t*defer_submodule_status = defer;\n \treturn changed;\n }\n \n@@ -102,6 +117,7 @@ int run_diff_files(struct rev_info *revs, unsigned int option)\n \t\t\t ? CE_MATCH_RACY_IS_DIRTY : 0);\n \tuint64_t start = getnanotime();\n \tstruct index_state *istate = revs->diffopt.repo->index;\n+\tstruct string_list submodules = STRING_LIST_INIT_NODUP;\n \n \tdiff_set_mnemonic_prefix(&revs->diffopt, \"i/\", \"w/\");\n \n@@ -226,6 +242,8 @@ int run_diff_files(struct rev_info *revs, unsigned int option)\n \t\t\tnewmode = ce->ce_mode;\n \t\t} else {\n \t\t\tstruct stat st;\n+\t\t\tunsigned ignore_untracked = 0;\n+\t\t\tint defer_submodule_status = !!revs->repo;\n \n \t\t\tchanged = check_removed(istate, ce, &st);\n \t\t\tif (changed) {\n@@ -247,8 +265,26 @@ int run_diff_files(struct rev_info *revs, unsigned int option)\n \t\t\t}\n \n \t\t\tchanged = match_stat_with_submodule(&revs->diffopt, ce, &st,\n-\t\t\t\t\t\t\t ce_option, &dirty_submodule);\n+\t\t\t\t\t\t\t ce_option, &dirty_submodule,\n+\t\t\t\t\t\t\t &defer_submodule_status,\n+\t\t\t\t\t\t\t &ignore_untracked);\n \t\t\tnewmode = ce_mode_from_stat(ce, st.st_mode);\n+\t\t\tif (defer_submodule_status) {\n+\t\t\t\tstruct submodule_status_util tmp = {\n+\t\t\t\t\t.changed = changed,\n+\t\t\t\t\t.dirty_submodule = 0,\n+\t\t\t\t\t.ignore_untracked = ignore_untracked,\n+\t\t\t\t\t.newmode = newmode,\n+\t\t\t\t\t.ce = ce,\n+\t\t\t\t\t.path = ce->name,\n+\t\t\t\t};\n+\t\t\t\tstruct string_list_item *item;\n+\n+\t\t\t\titem = string_list_append(&submodules, ce->name);\n+\t\t\t\titem->util = xmalloc(sizeof(tmp));\n+\t\t\t\tmemcpy(item->util, &tmp, sizeof(tmp));\n+\t\t\t\tcontinue;\n+\t\t\t}\n \t\t}\n \n \t\tif (!changed && !dirty_submodule) {\n@@ -267,6 +303,40 @@ int run_diff_files(struct rev_info *revs, unsigned int option)\n \t\t\t ce->name, 0, dirty_submodule);\n \n \t}\n+\tif (submodules.nr > 0) {\n+\t\tint parallel_jobs;\n+\t\tif (git_config_get_int(\"submodule.diffjobs\", ¶llel_jobs))\n+\t\t\tparallel_jobs = 1;\n+\t\telse if (!parallel_jobs)\n+\t\t\tparallel_jobs = online_cpus();\n+\t\telse if (parallel_jobs < 0)\n+\t\t\tdie(_(\"submodule.diffjobs cannot be negative\"));\n+\n+\t\tif (get_submodules_status(&submodules, parallel_jobs))\n+\t\t\tdie(_(\"submodule status failed\"));\n+\t\tfor (size_t i = 0; i < submodules.nr; i++) {\n+\t\t\tstruct submodule_status_util *util = submodules.items[i].util;\n+\t\t\tstruct cache_entry *ce = util->ce;\n+\t\t\tunsigned int oldmode;\n+\t\t\tconst struct object_id *old_oid, *new_oid;\n+\n+\t\t\tif (!util->changed && !util->dirty_submodule) {\n+\t\t\t\tce_mark_uptodate(ce);\n+\t\t\t\tmark_fsmonitor_valid(istate, ce);\n+\t\t\t\tif (!revs->diffopt.flags.find_copies_harder)\n+\t\t\t\t\tcontinue;\n+\t\t\t}\n+\t\t\toldmode = ce->ce_mode;\n+\t\t\told_oid = &ce->oid;\n+\t\t\tnew_oid = util->changed ? null_oid() : &ce->oid;\n+\t\t\tdiff_change(&revs->diffopt, oldmode, util->newmode,\n+\t\t\t\t old_oid, new_oid,\n+\t\t\t\t !is_null_oid(old_oid),\n+\t\t\t\t !is_null_oid(new_oid),\n+\t\t\t\t ce->name, 0, util->dirty_submodule);\n+\t\t}\n+\t}\n+\tstring_list_clear(&submodules, 1);\n \tdiffcore_std(&revs->diffopt);\n \tdiff_flush(&revs->diffopt);\n \ttrace_performance_since(start, \"diff-files\");\n@@ -314,7 +384,7 @@ static int get_stat_data(const struct index_state *istate,\n \t\t\treturn -1;\n \t\t}\n \t\tchanged = match_stat_with_submodule(diffopt, ce, &st,\n-\t\t\t\t\t\t 0, dirty_submodule);\n+\t\t\t\t\t\t 0, dirty_submodule, NULL, NULL);\n \t\tif (changed) {\n \t\t\tmode = ce_mode_from_stat(ce, st.st_mode);\n \t\t\toid = null_oid();\ndiff --git a/submodule.c b/submodule.c\nindex 768d4b4cd7..a0ca646d9b 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -1369,6 +1369,17 @@ int submodule_touches_in_range(struct repository *r,\n \treturn ret;\n }\n \n+struct submodule_parallel_status {\n+\tsize_t index_count;\n+\tint result;\n+\n+\tstruct string_list *submodule_names;\n+\n+\t/* Pending statuses by OIDs */\n+\tstruct status_task **oid_status_tasks;\n+\tint oid_status_tasks_nr, oid_status_tasks_alloc;\n+};\n+\n struct submodule_parallel_fetch {\n \t/*\n \t * The index of the last index entry processed by\n@@ -1451,6 +1462,12 @@ struct fetch_task {\n \tstruct oid_array *commits; /* Ensure these commits are fetched */\n };\n \n+struct status_task {\n+\tconst char *path;\n+\tstruct strbuf out;\n+\tint ignore_untracked;\n+};\n+\n /**\n * When a submodule is not defined in .gitmodules, we cannot access it\n * via the regular submodule-config. Create a fake submodule, which we can\n@@ -1909,6 +1926,25 @@ static int parse_status_porcelain(char *str, size_t len,\n \treturn 0;\n }\n \n+static void parse_status_porcelain_strbuf(struct strbuf *buf,\n+\t\t\t\t unsigned *dirty_submodule,\n+\t\t\t\t int ignore_untracked)\n+{\n+\tstruct string_list list = STRING_LIST_INIT_DUP;\n+\tstruct string_list_item *item;\n+\n+\tstring_list_split(&list, buf->buf, '\\n', -1);\n+\n+\tfor_each_string_list_item(item, &list) {\n+\t\tif (parse_status_porcelain(item->string,\n+\t\t\t\t\t strlen(item->string),\n+\t\t\t\t\t dirty_submodule,\n+\t\t\t\t\t ignore_untracked))\n+\t\t\tbreak;\n+\t}\n+\tstring_list_clear(&list, 0);\n+}\n+\n unsigned is_submodule_modified(const char *path, int ignore_untracked)\n {\n \tstruct child_process cp = CHILD_PROCESS_INIT;\n@@ -1962,6 +1998,138 @@ unsigned is_submodule_modified(const char *path, int ignore_untracked)\n \treturn dirty_submodule;\n }\n \n+static struct status_task *\n+get_status_task_from_index(struct submodule_parallel_status *sps,\n+\t\t\t struct strbuf *err)\n+{\n+\tfor (; sps->index_count < sps->submodule_names->nr; sps->index_count++) {\n+\t\tstruct submodule_status_util *util = sps->submodule_names->items[sps->index_count].util;\n+\t\tstruct status_task *task;\n+\t\tstruct strbuf buf = STRBUF_INIT;\n+\t\tconst char *git_dir;\n+\n+\t\tstrbuf_addf(&buf, \"%s/.git\", util->path);\n+\t\tgit_dir = read_gitfile(buf.buf);\n+\t\tif (!git_dir)\n+\t\t\tgit_dir = buf.buf;\n+\t\tif (!is_git_directory(git_dir)) {\n+\t\t\tif (is_directory(git_dir))\n+\t\t\t\tdie(_(\"'%s' not recognized as a git repository\"), git_dir);\n+\t\t\tstrbuf_release(&buf);\n+\t\t\t/* The submodule is not checked out, so it is not modified */\n+\t\t\tutil->dirty_submodule = 0;\n+\t\t\tcontinue;\n+\t\t}\n+\t\tstrbuf_release(&buf);\n+\n+\t\ttask = xmalloc(sizeof(*task));\n+\t\ttask->path = util->path;\n+\t\ttask->ignore_untracked = util->ignore_untracked;\n+\t\tstrbuf_init(&task->out, 0);\n+\t\tsps->index_count++;\n+\t\treturn task;\n+\t}\n+\treturn NULL;\n+}\n+\n+static int get_next_submodule_status(struct child_process *cp,\n+\t\t\t\t struct strbuf *err, void *data,\n+\t\t\t\t void **task_cb)\n+{\n+\tstruct submodule_parallel_status *sps = data;\n+\tstruct status_task *task = get_status_task_from_index(sps, err);\n+\n+\tif (!task)\n+\t\treturn 0;\n+\n+\tchild_process_init(cp);\n+\tprepare_submodule_repo_env_in_gitdir(&cp->env);\n+\n+\tstrvec_init(&cp->args);\n+\tstrvec_pushl(&cp->args, \"status\", \"--porcelain=2\", NULL);\n+\tif (task->ignore_untracked)\n+\t\tstrvec_push(&cp->args, \"-uno\");\n+\n+\tprepare_submodule_repo_env(&cp->env);\n+\tcp->git_cmd = 1;\n+\tcp->dir = task->path;\n+\t*task_cb = task;\n+\treturn 1;\n+}\n+\n+static int status_start_failure(struct strbuf *err,\n+\t\t\t\tvoid *cb, void *task_cb)\n+{\n+\tstruct submodule_parallel_status *sps = cb;\n+\tstruct status_task *task = task_cb;\n+\n+\tsps->result = 1;\n+\tstrbuf_addf(err,\n+\t _(\"could not run 'git status --porcelain=2' in submodule %s\"),\n+\t task->path);\n+\treturn 0;\n+}\n+\n+static void status_duplicate_output(struct strbuf *out,\n+\t\t\t\t size_t offset,\n+\t\t\t\t void *cb, void *task_cb)\n+{\n+\tstruct status_task *task = task_cb;\n+\n+\tstrbuf_add(&task->out, out->buf + offset, out->len - offset);\n+\tstrbuf_setlen(out, offset);\n+}\n+\n+static int status_finish(int retvalue, struct strbuf *err,\n+\t\t\t void *cb, void *task_cb)\n+{\n+\tstruct submodule_parallel_status *sps = cb;\n+\tstruct status_task *task = task_cb;\n+\tstruct string_list_item *it =\n+\t\tstring_list_lookup(sps->submodule_names, task->path);\n+\tstruct submodule_status_util *util = it->util;\n+\n+\tif (retvalue) {\n+\t\tsps->result = 1;\n+\t\tstrbuf_addf(err,\n+\t\t _(\"'git status --porcelain=2' failed in submodule %s\"),\n+\t\t task->path);\n+\t}\n+\n+\tparse_status_porcelain_strbuf(&task->out,\n+\t\t\t &util->dirty_submodule,\n+\t\t\t util->ignore_untracked);\n+\n+\tfree(task);\n+\n+\treturn 0;\n+}\n+\n+int get_submodules_status(struct string_list *submodules,\n+\t\t\t int max_parallel_jobs)\n+{\n+\tstruct submodule_parallel_status sps = {\n+\t\t.submodule_names = submodules,\n+\t};\n+\tconst struct run_process_parallel_opts opts = {\n+\t\t.tr2_category = \"submodule\",\n+\t\t.tr2_label = \"parallel/status\",\n+\n+\t\t.processes = max_parallel_jobs,\n+\n+\t\t.get_next_task = get_next_submodule_status,\n+\t\t.start_failure = status_start_failure,\n+\t\t.duplicate_output = status_duplicate_output,\n+\t\t.task_finished = status_finish,\n+\t\t.data = &sps,\n+\t};\n+\n+\tstring_list_sort(sps.submodule_names);\n+\trun_processes_parallel(&opts);\n+\n+\treturn sps.result;\n+}\n+\n int submodule_uses_gitfile(const char *path)\n {\n \tstruct child_process cp = CHILD_PROCESS_INIT;\ndiff --git a/submodule.h b/submodule.h\nindex b52a4ff1e7..08d278a414 100644\n--- a/submodule.h\n+++ b/submodule.h\n@@ -41,6 +41,13 @@ struct submodule_update_strategy {\n \t.type = SM_UPDATE_UNSPECIFIED, \\\n }\n \n+struct submodule_status_util {\n+\tint changed, ignore_untracked;\n+\tunsigned dirty_submodule, newmode;\n+\tstruct cache_entry *ce;\n+\tconst char *path;\n+};\n+\n int is_gitmodules_unmerged(struct index_state *istate);\n int is_writing_gitmodules_ok(void);\n int is_staging_gitmodules_ok(struct index_state *istate);\n@@ -94,6 +101,8 @@ int fetch_submodules(struct repository *r,\n \t\t int command_line_option,\n \t\t int default_option,\n \t\t int quiet, int max_parallel_jobs);\n+int get_submodules_status(struct string_list *submodules,\n+\t\t\t int max_parallel_jobs);\n unsigned is_submodule_modified(const char *path, int ignore_untracked);\n int submodule_uses_gitfile(const char *path);\n \ndiff --git a/t/t4027-diff-submodule.sh b/t/t4027-diff-submodule.sh\nindex 40164ae07d..e08ee315a7 100755\n--- a/t/t4027-diff-submodule.sh\n+++ b/t/t4027-diff-submodule.sh\n@@ -34,6 +34,25 @@ test_expect_success setup '\n \tsubtip=$3 subprev=$2\n '\n \n+test_expect_success 'diff in superproject with submodules respects parallel settings' '\n+\ttest_when_finished \"rm -f trace.out\" &&\n+\t(\n+\t\tGIT_TRACE=$(pwd)/trace.out git diff &&\n+\t\tgrep \"1 tasks\" trace.out &&\n+\t\t>trace.out &&\n+\n+\t\tgit config submodule.diffJobs 8 &&\n+\t\tGIT_TRACE=$(pwd)/trace.out git diff &&\n+\t\tgrep \"8 tasks\" trace.out &&\n+\t\t>trace.out &&\n+\n+\t\tGIT_TRACE=$(pwd)/trace.out git -c submodule.diffJobs=0 diff &&\n+\t\tgrep \"preparing to run up to [0-9]* tasks\" trace.out &&\n+\t\t! grep \"up to 0 tasks\" trace.out &&\n+\t\t>trace.out\n+\t)\n+'\n+\n test_expect_success 'git diff --raw HEAD' '\n \thexsz=$(test_oid hexsz) &&\n \tgit diff --raw --abbrev=$hexsz HEAD >actual &&\ndiff --git a/t/t7506-status-submodule.sh b/t/t7506-status-submodule.sh\nindex d050091345..52a82b703f 100755\n--- a/t/t7506-status-submodule.sh\n+++ b/t/t7506-status-submodule.sh\n@@ -412,4 +412,23 @@ test_expect_success 'status with added file in nested submodule (short)' '\n \tEOF\n '\n \n+test_expect_success 'status in superproject with submodules respects parallel settings' '\n+\ttest_when_finished \"rm -f trace.out\" &&\n+\t(\n+\t\tGIT_TRACE=$(pwd)/trace.out git status &&\n+\t\tgrep \"1 tasks\" trace.out &&\n+\t\t>trace.out &&\n+\n+\t\tgit config submodule.diffJobs 8 &&\n+\t\tGIT_TRACE=$(pwd)/trace.out git status &&\n+\t\tgrep \"8 tasks\" trace.out &&\n+\t\t>trace.out &&\n+\n+\t\tGIT_TRACE=$(pwd)/trace.out git -c submodule.diffJobs=0 status &&\n+\t\tgrep \"preparing to run up to [0-9]* tasks\" trace.out &&\n+\t\t! grep \"up to 0 tasks\" trace.out &&\n+\t\t>trace.out\n+\t)\n+'\n+\n test_done\n-- \n2.39.0.314.g84b9a713c41-goog\n\n"},{"id":"469798","messageId":"20230104215415.1083526-7-calvinwan@google.com","threadId":"59034","inReplyTo":"https://lore.kernel.org/git/20221108184200.2813458-1-calvinwan@google.com/","subject":"[PATCH v5 6/6] submodule: call parallel code from serial status","fromName":"Calvin Wan","fromEmail":"calvinwan@google.com","sentAt":"2023-01-04T21:54:15Z","receivedAt":"2023-01-04T21:55:10Z","isPatch":true,"sender":{"key":"calvinwan@google.com","avatar":"https://avatars.githubusercontent.com/u/92547554?v=4"},"body":"Remove the serial implementation of status inside of\nis_submodule_modified since the parallel implementation of status with\none job accomplishes the same task.\n\nCombine parse_status_porcelain and parse_status_porcelain_strbuf since\nthe only other caller of parse_status_porcelain was in\nis_submodule_modified\n\nSigned-off-by: Calvin Wan \n---\n submodule.c | 143 ++++++++++++++++++----------------------------------\n 1 file changed, 48 insertions(+), 95 deletions(-)\n\ndiff --git a/submodule.c b/submodule.c\nindex a0ca646d9b..042e26137f 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -1887,46 +1887,7 @@ int fetch_submodules(struct repository *r,\n \treturn spf.result;\n }\n \n-static int parse_status_porcelain(char *str, size_t len,\n-\t\t\t\t unsigned *dirty_submodule,\n-\t\t\t\t int ignore_untracked)\n-{\n-\t/* regular untracked files */\n-\tif (str[0] == '?')\n-\t\t*dirty_submodule |= DIRTY_SUBMODULE_UNTRACKED;\n-\n-\tif (str[0] == 'u' ||\n-\t str[0] == '1' ||\n-\t str[0] == '2') {\n-\t\t/* T = line type, XY = status, SSSS = submodule state */\n-\t\tif (len < strlen(\"T XY SSSS\"))\n-\t\t\tBUG(\"invalid status --porcelain=2 line %s\",\n-\t\t\t str);\n-\n-\t\tif (str[5] == 'S' && str[8] == 'U')\n-\t\t\t/* nested untracked file */\n-\t\t\t*dirty_submodule |= DIRTY_SUBMODULE_UNTRACKED;\n-\n-\t\tif (str[0] == 'u' ||\n-\t\t str[0] == '2' ||\n-\t\t memcmp(str + 5, \"S..U\", 4))\n-\t\t\t/* other change */\n-\t\t\t*dirty_submodule |= DIRTY_SUBMODULE_MODIFIED;\n-\t}\n-\n-\tif ((*dirty_submodule & DIRTY_SUBMODULE_MODIFIED) &&\n-\t ((*dirty_submodule & DIRTY_SUBMODULE_UNTRACKED) ||\n-\t ignore_untracked)) {\n-\t\t/*\n-\t\t* We're not interested in any further information from\n-\t\t* the child any more, neither output nor its exit code.\n-\t\t*/\n-\t\treturn 1;\n-\t}\n-\treturn 0;\n-}\n-\n-static void parse_status_porcelain_strbuf(struct strbuf *buf,\n+static void parse_status_porcelain(struct strbuf *buf,\n \t\t\t\t unsigned *dirty_submodule,\n \t\t\t\t int ignore_untracked)\n {\n@@ -1936,66 +1897,58 @@ static void parse_status_porcelain_strbuf(struct strbuf *buf,\n \tstring_list_split(&list, buf->buf, '\\n', -1);\n \n \tfor_each_string_list_item(item, &list) {\n-\t\tif (parse_status_porcelain(item->string,\n-\t\t\t\t\t strlen(item->string),\n-\t\t\t\t\t dirty_submodule,\n-\t\t\t\t\t ignore_untracked))\n+\t\tchar *str = item->string;\n+\t\t/* regular untracked files */\n+\t\tif (str[0] == '?')\n+\t\t\t*dirty_submodule |= DIRTY_SUBMODULE_UNTRACKED;\n+\n+\t\tif (str[0] == 'u' ||\n+\t\tstr[0] == '1' ||\n+\t\tstr[0] == '2') {\n+\t\t\t/* T = line type, XY = status, SSSS = submodule state */\n+\t\t\tif (strlen(str) < strlen(\"T XY SSSS\"))\n+\t\t\t\tBUG(\"invalid status --porcelain=2 line %s\",\n+\t\t\t\tstr);\n+\n+\t\t\tif (str[5] == 'S' && str[8] == 'U')\n+\t\t\t\t/* nested untracked file */\n+\t\t\t\t*dirty_submodule |= DIRTY_SUBMODULE_UNTRACKED;\n+\n+\t\t\tif (str[0] == 'u' ||\n+\t\t\tstr[0] == '2' ||\n+\t\t\tmemcmp(str + 5, \"S..U\", 4))\n+\t\t\t\t/* other change */\n+\t\t\t\t*dirty_submodule |= DIRTY_SUBMODULE_MODIFIED;\n+\t\t}\n+\n+\t\tif ((*dirty_submodule & DIRTY_SUBMODULE_MODIFIED) &&\n+\t\t ((*dirty_submodule & DIRTY_SUBMODULE_UNTRACKED) ||\n+\t\t ignore_untracked)) {\n+\t\t\t/*\n+\t\t\t* We're not interested in any further information from\n+\t\t\t* the child any more, neither output nor its exit code.\n+\t\t\t*/\n \t\t\tbreak;\n+\t\t}\n \t}\n \tstring_list_clear(&list, 0);\n }\n \n unsigned is_submodule_modified(const char *path, int ignore_untracked)\n {\n-\tstruct child_process cp = CHILD_PROCESS_INIT;\n-\tstruct strbuf buf = STRBUF_INIT;\n-\tFILE *fp;\n-\tunsigned dirty_submodule = 0;\n-\tconst char *git_dir;\n-\tint ignore_cp_exit_code = 0;\n-\n-\tstrbuf_addf(&buf, \"%s/.git\", path);\n-\tgit_dir = read_gitfile(buf.buf);\n-\tif (!git_dir)\n-\t\tgit_dir = buf.buf;\n-\tif (!is_git_directory(git_dir)) {\n-\t\tif (is_directory(git_dir))\n-\t\t\tdie(_(\"'%s' not recognized as a git repository\"), git_dir);\n-\t\tstrbuf_release(&buf);\n-\t\t/* The submodule is not checked out, so it is not modified */\n-\t\treturn 0;\n-\t}\n-\tstrbuf_reset(&buf);\n-\n-\tstrvec_pushl(&cp.args, \"status\", \"--porcelain=2\", NULL);\n-\tif (ignore_untracked)\n-\t\tstrvec_push(&cp.args, \"-uno\");\n-\n-\tprepare_submodule_repo_env(&cp.env);\n-\tcp.git_cmd = 1;\n-\tcp.no_stdin = 1;\n-\tcp.out = -1;\n-\tcp.dir = path;\n-\tif (start_command(&cp))\n-\t\tdie(_(\"Could not run 'git status --porcelain=2' in submodule %s\"), path);\n-\n-\tfp = xfdopen(cp.out, \"r\");\n-\twhile (strbuf_getwholeline(&buf, fp, '\\n') != EOF) {\n-\t\tchar *str = buf.buf;\n-\t\tconst size_t len = buf.len;\n-\n-\t\tignore_cp_exit_code = parse_status_porcelain(str, len, &dirty_submodule,\n-\t\t\t\t\t\t\t ignore_untracked);\n-\t\tif (ignore_cp_exit_code)\n-\t\t\tbreak;\n-\t}\n-\tfclose(fp);\n-\n-\tif (finish_command(&cp) && !ignore_cp_exit_code)\n-\t\tdie(_(\"'git status --porcelain=2' failed in submodule %s\"), path);\n+\tstruct submodule_status_util util = {\n+\t\t.dirty_submodule = 0,\n+\t\t.ignore_untracked = ignore_untracked,\n+\t\t.path = path,\n+\t};\n+\tstruct string_list sub = STRING_LIST_INIT_NODUP;\n+\tstruct string_list_item *item;\n \n-\tstrbuf_release(&buf);\n-\treturn dirty_submodule;\n+\titem = string_list_append(&sub, path);\n+\titem->util = &util;\n+\tif (get_submodules_status(&sub, 1))\n+\t\tdie(_(\"submodule status failed\"));\n+\treturn util.dirty_submodule;\n }\n \n static struct status_task *\n@@ -2096,9 +2049,9 @@ static int status_finish(int retvalue, struct strbuf *err,\n \t\t task->path);\n \t}\n \n-\tparse_status_porcelain_strbuf(&task->out,\n-\t\t\t &util->dirty_submodule,\n-\t\t\t util->ignore_untracked);\n+\tparse_status_porcelain(&task->out,\n+\t\t\t &util->dirty_submodule,\n+\t\t\t util->ignore_untracked);\n \n \tfree(task);\n \n-- \n2.39.0.314.g84b9a713c41-goog\n\n"},{"id":"469815","messageId":"CAFySSZA6EnWJr8N_G2dtt8=PKL8qtJnnm65x0yzWXuTBFfj4cw@mail.gmail.com","threadId":"59034","inReplyTo":"20230104215415.1083526-1-calvinwan@google.com","subject":"Re: [PATCH v5 0/6] submodule: parallelize diff","fromName":"Calvin Wan","fromEmail":"calvinwan@google.com","sentAt":"2023-01-05T23:23:48Z","receivedAt":"2023-01-05T23:24:05Z","isPatch":true,"sender":{"key":"calvinwan@google.com","avatar":"https://avatars.githubusercontent.com/u/92547554?v=4"},"body":"Apologies for the broken link to the previous versions. Looks like I had\nsome encoding issues with copy/paste. Here are the previous versions\n\nv4: https://lore.kernel.org/git/20221108184200.2813458-1-calvinwan@google.com/\nv3: https://lore.kernel.org/git/20221020232532.1128326-1-calvinwan@google.com/\nv2: https://lore.kernel.org/git/20221011232604.839941-1-calvinwan@google.com/\nv1: https://lore.kernel.org/git/20220922232947.631309-1-calvinwan@google.com/\n\nOn Wed, Jan 4, 2023 at 1:54 PM Calvin Wan wrote:\n>\n> Original cover letter for context:\n> https://lore.kernel.org/git/20221011232604.839941-1-calvinwan@google.com/\n>\n> Thank you again everyone for the numerous reviews! For this reroll, I\n> incorporated most of the feedback given, fixed a bug I found, and made\n> some stylistic refactors. I also added a new patch at the end that swaps\n> the serial implementation in is_submodule_modified for the new parallel\n> one. While I had patch 6 originally smushed with the previous one,\n> the diff came out not very reviewer friendly so it has been separated\n> out.\n>\n> Changes since v4\n>\n> (Patch 1)\n> The code in run-command.c that calls duplicate_output_fn has been\n> cleaned up and no longer passes a separate strbuf for the output. It\n> instead passes an offset that represents the starting point in the\n> original strbuf.\n>\n> (Patch 5)\n> Moved status parsing from status_duplicate_output to status_finish. In\n> pp_buffer_stderr::run-command.c, output is gathered by strbuf_read_once\n> which reads 8192 bytes at once so a longer status message would error\n> out during status parsing since part of it would be cut off. Therefore,\n> status parsing must happen at the end of the process rather than in\n> duplicate_output_fn (and has subsequently been moved).\n>\n> (Patch 6)\n> New patch swapping serial implementation in is_submodule_modified for\n> the new parallel one.\n>\n> Calvin Wan (6):\n> run-command: add duplicate_output_fn to run_processes_parallel_opts\n> submodule: strbuf variable rename\n> submodule: move status parsing into function\n> diff-lib: refactor match_stat_with_submodule\n> diff-lib: parallelize run_diff_files for submodules\n> submodule: call parallel code from serial status\n>\n> Documentation/config/submodule.txt | 12 ++\n> diff-lib.c | 104 ++++++++++--\n> run-command.c | 16 +-\n> run-command.h | 27 ++++\n> submodule.c | 250 ++++++++++++++++++++++-------\n> submodule.h | 9 ++\n> t/helper/test-run-command.c | 21 +++\n> t/t0061-run-command.sh | 39 +++++\n> t/t4027-diff-submodule.sh | 19 +++\n> t/t7506-status-submodule.sh | 19 +++\n> 10 files changed, 441 insertions(+), 75 deletions(-)\n>\n> --\n> 2.39.0.314.g84b9a713c41-goog\n>\n"},{"id":"470530","messageId":"20230117193041.708692-1-calvinwan@google.com","threadId":"59034","inReplyTo":"20230104215415.1083526-1-calvinwan@google.com","subject":"[PATCH v6 0/6] submodule: parallelize diff","fromName":"Calvin Wan","fromEmail":"calvinwan@google.com","sentAt":"2023-01-17T19:30:35Z","receivedAt":"2023-01-17T21:05:26Z","isPatch":true,"sender":{"key":"calvinwan@google.com","avatar":"https://avatars.githubusercontent.com/u/92547554?v=4"},"body":"Original cover letter for context:\nhttps://lore.kernel.org/git/20221011232604.839941-1-calvinwan@google.com/\n\n(Quick reroll to fix leaks from v5)\n\nThank you again everyone for the numerous reviews! For this reroll, I\nincorporated most of the feedback given, fixed a bug I found, and made\nsome stylistic refactors. I also added a new patch at the end that swaps\nthe serial implementation in is_submodule_modified for the new parallel\none. While I had patch 6 originally smushed with the previous one,\nthe diff came out not very reviewer friendly so it has been separated\nout.\n\nChanges since v4\n\n(Patch 1)\nThe code in run-command.c that calls duplicate_output_fn has been\ncleaned up and no longer passes a separate strbuf for the output. It\ninstead passes an offset that represents the starting point in the\noriginal strbuf.\n\n(Patch 5)\nMoved status parsing from status_duplicate_output to status_finish. In\npp_buffer_stderr::run-command.c, output is gathered by strbuf_read_once\nwhich reads 8192 bytes at once so a longer status message would error\nout during status parsing since part of it would be cut off. Therefore,\nstatus parsing must happen at the end of the process rather than in\nduplicate_output_fn (and has subsequently been moved).\n\n(Patch 6)\nNew patch swapping serial implementation in is_submodule_modified for\nthe new parallel one.\n\nCalvin Wan (6):\n run-command: add duplicate_output_fn to run_processes_parallel_opts\n submodule: strbuf variable rename\n submodule: move status parsing into function\n diff-lib: refactor match_stat_with_submodule\n diff-lib: parallelize run_diff_files for submodules\n submodule: call parallel code from serial status\n\n Documentation/config/submodule.txt | 12 ++\n diff-lib.c | 104 ++++++++++--\n run-command.c | 16 +-\n run-command.h | 27 +++\n submodule.c | 254 ++++++++++++++++++++++-------\n submodule.h | 9 +\n t/helper/test-run-command.c | 21 +++\n t/t0061-run-command.sh | 39 +++++\n t/t4027-diff-submodule.sh | 19 +++\n t/t7506-status-submodule.sh | 19 +++\n 10 files changed, 445 insertions(+), 75 deletions(-)\n\n-- \n2.39.0.314.g84b9a713c41-goog\n\n"},{"id":"470531","messageId":"20230117193041.708692-2-calvinwan@google.com","threadId":"59034","inReplyTo":"20230104215415.1083526-1-calvinwan@google.com","subject":"[PATCH v6 1/6] run-command: add duplicate_output_fn to run_processes_parallel_opts","fromName":"Calvin Wan","fromEmail":"calvinwan@google.com","sentAt":"2023-01-17T19:30:36Z","receivedAt":"2023-01-17T21:06:35Z","isPatch":true,"sender":{"key":"calvinwan@google.com","avatar":"https://avatars.githubusercontent.com/u/92547554?v=4"},"body":"Add duplicate_output_fn as an optionally set function in\nrun_process_parallel_opts. If set, output from each child process is\ncopied and passed to the callback function whenever output from the\nchild process is buffered to allow for separate parsing.\n\nSigned-off-by: Calvin Wan \n---\n run-command.c | 16 ++++++++++++---\n run-command.h | 27 +++++++++++++++++++++++++\n t/helper/test-run-command.c | 21 ++++++++++++++++++++\n t/t0061-run-command.sh | 39 +++++++++++++++++++++++++++++++++++++\n 4 files changed, 100 insertions(+), 3 deletions(-)\n\ndiff --git a/run-command.c b/run-command.c\nindex 756f1839aa..cad88befe0 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -1526,6 +1526,9 @@ static void pp_init(struct parallel_processes *pp,\n \tif (!opts->get_next_task)\n \t\tBUG(\"you need to specify a get_next_task function\");\n \n+\tif (opts->duplicate_output && opts->ungroup)\n+\t\tBUG(\"duplicate_output and ungroup are incompatible with each other\");\n+\n \tCALLOC_ARRAY(pp->children, n);\n \tif (!opts->ungroup)\n \t\tCALLOC_ARRAY(pp->pfd, n);\n@@ -1645,14 +1648,21 @@ static void pp_buffer_stderr(struct parallel_processes *pp,\n \tfor (size_t i = 0; i < opts->processes; i++) {\n \t\tif (pp->children[i].state == GIT_CP_WORKING &&\n \t\t pp->pfd[i].revents & (POLLIN | POLLHUP)) {\n-\t\t\tint n = strbuf_read_once(&pp->children[i].err,\n-\t\t\t\t\t\t pp->children[i].process.err, 0);\n+\t\t\tssize_t n = strbuf_read_once(&pp->children[i].err,\n+\t\t\t\t\t\t pp->children[i].process.err, 0);\n \t\t\tif (n == 0) {\n \t\t\t\tclose(pp->children[i].process.err);\n \t\t\t\tpp->children[i].state = GIT_CP_WAIT_CLEANUP;\n-\t\t\t} else if (n < 0)\n+\t\t\t} else if (n < 0) {\n \t\t\t\tif (errno != EAGAIN)\n \t\t\t\t\tdie_errno(\"read\");\n+\t\t\t} else {\n+\t\t\t\tif (opts->duplicate_output)\n+\t\t\t\t\topts->duplicate_output(&pp->children[i].err,\n+\t\t\t\t\t strlen(pp->children[i].err.buf) - n,\n+\t\t\t\t\t opts->data,\n+\t\t\t\t\t pp->children[i].data);\n+\t\t\t}\n \t\t}\n \t}\n }\ndiff --git a/run-command.h b/run-command.h\nindex 072db56a4d..6dcf999f6c 100644\n--- a/run-command.h\n+++ b/run-command.h\n@@ -408,6 +408,27 @@ typedef int (*start_failure_fn)(struct strbuf *out,\n \t\t\t\tvoid *pp_cb,\n \t\t\t\tvoid *pp_task_cb);\n \n+/**\n+ * This callback is called whenever output from a child process is buffered\n+ * \n+ * See run_processes_parallel() below for a discussion of the \"struct\n+ * strbuf *out\" parameter.\n+ * \n+ * The offset refers to the number of bytes originally in \"out\" before\n+ * the output from the child process was buffered. Therefore, the buffer\n+ * range, \"out + buf\" to the end of \"out\", would contain the buffer of\n+ * the child process output.\n+ *\n+ * pp_cb is the callback cookie as passed into run_processes_parallel,\n+ * pp_task_cb is the callback cookie as passed into get_next_task_fn.\n+ *\n+ * This function is incompatible with \"ungroup\"\n+ */\n+typedef void (*duplicate_output_fn)(struct strbuf *out,\n+\t\t\t\t size_t offset,\n+\t\t\t\t void *pp_cb,\n+\t\t\t\t void *pp_task_cb);\n+\n /**\n * This callback is called on every child process that finished processing.\n *\n@@ -461,6 +482,12 @@ struct run_process_parallel_opts\n \t */\n \tstart_failure_fn start_failure;\n \n+\t/**\n+\t * duplicate_output: See duplicate_output_fn() above. This should be\n+\t * NULL unless process specific output is needed\n+\t */\n+\tduplicate_output_fn duplicate_output;\n+\n \t/**\n \t * task_finished: See task_finished_fn() above. This can be\n \t * NULL to omit any special handling.\ndiff --git a/t/helper/test-run-command.c b/t/helper/test-run-command.c\nindex 3ecb830f4a..ffd3cd0045 100644\n--- a/t/helper/test-run-command.c\n+++ b/t/helper/test-run-command.c\n@@ -52,6 +52,21 @@ static int no_job(struct child_process *cp,\n \treturn 0;\n }\n \n+static void duplicate_output(struct strbuf *out,\n+\t\t\tsize_t offset,\n+\t\t\tvoid *pp_cb UNUSED,\n+\t\t\tvoid *pp_task_cb UNUSED)\n+{\n+\tstruct string_list list = STRING_LIST_INIT_DUP;\n+\n+\tstring_list_split(&list, out->buf + offset, '\\n', -1);\n+\tfor (size_t i = 0; i < list.nr; i++) {\n+\t\tif (strlen(list.items[i].string) > 0)\n+\t\t\tfprintf(stderr, \"duplicate_output: %s\\n\", list.items[i].string);\n+\t}\n+\tstring_list_clear(&list, 0);\n+}\n+\n static int task_finished(int result,\n \t\t\t struct strbuf *err,\n \t\t\t void *pp_cb,\n@@ -439,6 +454,12 @@ int cmd__run_command(int argc, const char **argv)\n \t\topts.ungroup = 1;\n \t}\n \n+\tif (!strcmp(argv[1], \"--duplicate-output\")) {\n+\t\targv += 1;\n+\t\targc -= 1;\n+\t\topts.duplicate_output = duplicate_output;\n+\t}\n+\n \tjobs = atoi(argv[2]);\n \tstrvec_clear(&proc.args);\n \tstrvec_pushv(&proc.args, (const char **)argv + 3);\ndiff --git a/t/t0061-run-command.sh b/t/t0061-run-command.sh\nindex e2411f6a9b..879e536638 100755\n--- a/t/t0061-run-command.sh\n+++ b/t/t0061-run-command.sh\n@@ -135,6 +135,15 @@ test_expect_success 'run_command runs in parallel with more jobs available than\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'run_command runs in parallel with more jobs available than tasks --duplicate-output' '\n+\ttest-tool run-command --duplicate-output run-command-parallel 5 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n+\ttest_must_be_empty out &&\n+\ttest 4 = $(grep -c \"duplicate_output: Hello\" err) &&\n+\ttest 4 = $(grep -c \"duplicate_output: World\" err) &&\n+\tsed \"/duplicate_output/d\" err > err1 &&\n+\ttest_cmp expect err1\n+'\n+\n test_expect_success 'run_command runs ungrouped in parallel with more jobs available than tasks' '\n \ttest-tool run-command --ungroup run-command-parallel 5 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n \ttest_line_count = 8 out &&\n@@ -147,6 +156,15 @@ test_expect_success 'run_command runs in parallel with as many jobs as tasks' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'run_command runs in parallel with as many jobs as tasks --duplicate-output' '\n+\ttest-tool run-command --duplicate-output run-command-parallel 4 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n+\ttest_must_be_empty out &&\n+\ttest 4 = $(grep -c \"duplicate_output: Hello\" err) &&\n+\ttest 4 = $(grep -c \"duplicate_output: World\" err) &&\n+\tsed \"/duplicate_output/d\" err > err1 &&\n+\ttest_cmp expect err1\n+'\n+\n test_expect_success 'run_command runs ungrouped in parallel with as many jobs as tasks' '\n \ttest-tool run-command --ungroup run-command-parallel 4 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n \ttest_line_count = 8 out &&\n@@ -159,6 +177,15 @@ test_expect_success 'run_command runs in parallel with more tasks than jobs avai\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'run_command runs in parallel with more tasks than jobs available --duplicate-output' '\n+\ttest-tool run-command --duplicate-output run-command-parallel 3 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n+\ttest_must_be_empty out &&\n+\ttest 4 = $(grep -c \"duplicate_output: Hello\" err) &&\n+\ttest 4 = $(grep -c \"duplicate_output: World\" err) &&\n+\tsed \"/duplicate_output/d\" err > err1 &&\n+\ttest_cmp expect err1\n+'\n+\n test_expect_success 'run_command runs ungrouped in parallel with more tasks than jobs available' '\n \ttest-tool run-command --ungroup run-command-parallel 3 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n \ttest_line_count = 8 out &&\n@@ -180,6 +207,12 @@ test_expect_success 'run_command is asked to abort gracefully' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'run_command is asked to abort gracefully --duplicate-output' '\n+\ttest-tool run-command --duplicate-output run-command-abort 3 false >out 2>err &&\n+\ttest_must_be_empty out &&\n+\ttest_cmp expect err\n+'\n+\n test_expect_success 'run_command is asked to abort gracefully (ungroup)' '\n \ttest-tool run-command --ungroup run-command-abort 3 false >out 2>err &&\n \ttest_must_be_empty out &&\n@@ -196,6 +229,12 @@ test_expect_success 'run_command outputs ' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'run_command outputs --duplicate-output' '\n+\ttest-tool run-command --duplicate-output run-command-no-jobs 3 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n+\ttest_must_be_empty out &&\n+\ttest_cmp expect err\n+'\n+\n test_expect_success 'run_command outputs (ungroup) ' '\n \ttest-tool run-command --ungroup run-command-no-jobs 3 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n \ttest_must_be_empty out &&\n-- \n2.39.0.314.g84b9a713c41-goog\n\n"},{"id":"470532","messageId":"20230117193041.708692-3-calvinwan@google.com","threadId":"59034","inReplyTo":"20230104215415.1083526-1-calvinwan@google.com","subject":"[PATCH v6 2/6] submodule: strbuf variable rename","fromName":"Calvin Wan","fromEmail":"calvinwan@google.com","sentAt":"2023-01-17T19:30:37Z","receivedAt":"2023-01-17T21:06:40Z","isPatch":true,"sender":{"key":"calvinwan@google.com","avatar":"https://avatars.githubusercontent.com/u/92547554?v=4"},"body":"A prepatory change for a future patch that moves the status parsing\nlogic to a separate function.\n\nSigned-off-by: Calvin Wan \n---\n submodule.c | 23 +++++++++++++----------\n 1 file changed, 13 insertions(+), 10 deletions(-)\n\ndiff --git a/submodule.c b/submodule.c\nindex fae24ef34a..faf37c1101 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -1906,25 +1906,28 @@ unsigned is_submodule_modified(const char *path, int ignore_untracked)\n \n \tfp = xfdopen(cp.out, \"r\");\n \twhile (strbuf_getwholeline(&buf, fp, '\\n') != EOF) {\n+\t\tchar *str = buf.buf;\n+\t\tconst size_t len = buf.len;\n+\n \t\t/* regular untracked files */\n-\t\tif (buf.buf[0] == '?')\n+\t\tif (str[0] == '?')\n \t\t\tdirty_submodule |= DIRTY_SUBMODULE_UNTRACKED;\n \n-\t\tif (buf.buf[0] == 'u' ||\n-\t\t buf.buf[0] == '1' ||\n-\t\t buf.buf[0] == '2') {\n+\t\tif (str[0] == 'u' ||\n+\t\t str[0] == '1' ||\n+\t\t str[0] == '2') {\n \t\t\t/* T = line type, XY = status, SSSS = submodule state */\n-\t\t\tif (buf.len < strlen(\"T XY SSSS\"))\n+\t\t\tif (len < strlen(\"T XY SSSS\"))\n \t\t\t\tBUG(\"invalid status --porcelain=2 line %s\",\n-\t\t\t\t buf.buf);\n+\t\t\t\t str);\n \n-\t\t\tif (buf.buf[5] == 'S' && buf.buf[8] == 'U')\n+\t\t\tif (str[5] == 'S' && str[8] == 'U')\n \t\t\t\t/* nested untracked file */\n \t\t\t\tdirty_submodule |= DIRTY_SUBMODULE_UNTRACKED;\n \n-\t\t\tif (buf.buf[0] == 'u' ||\n-\t\t\t buf.buf[0] == '2' ||\n-\t\t\t memcmp(buf.buf + 5, \"S..U\", 4))\n+\t\t\tif (str[0] == 'u' ||\n+\t\t\t str[0] == '2' ||\n+\t\t\t memcmp(str + 5, \"S..U\", 4))\n \t\t\t\t/* other change */\n \t\t\t\tdirty_submodule |= DIRTY_SUBMODULE_MODIFIED;\n \t\t}\n-- \n2.39.0.314.g84b9a713c41-goog\n\n"},{"id":"470533","messageId":"20230117193041.708692-4-calvinwan@google.com","threadId":"59034","inReplyTo":"20230104215415.1083526-1-calvinwan@google.com","subject":"[PATCH v6 3/6] submodule: move status parsing into function","fromName":"Calvin Wan","fromEmail":"calvinwan@google.com","sentAt":"2023-01-17T19:30:38Z","receivedAt":"2023-01-17T21:06:49Z","isPatch":true,"sender":{"key":"calvinwan@google.com","avatar":"https://avatars.githubusercontent.com/u/92547554?v=4"},"body":"A future patch requires the ability to parse the output of git\nstatus --porcelain=2. Move parsing code from is_submodule_modified to\nparse_status_porcelain.\n\nSigned-off-by: Calvin Wan \n---\n submodule.c | 74 ++++++++++++++++++++++++++++++-----------------------\n 1 file changed, 42 insertions(+), 32 deletions(-)\n\ndiff --git a/submodule.c b/submodule.c\nindex faf37c1101..768d4b4cd7 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -1870,6 +1870,45 @@ int fetch_submodules(struct repository *r,\n \treturn spf.result;\n }\n \n+static int parse_status_porcelain(char *str, size_t len,\n+\t\t\t\t unsigned *dirty_submodule,\n+\t\t\t\t int ignore_untracked)\n+{\n+\t/* regular untracked files */\n+\tif (str[0] == '?')\n+\t\t*dirty_submodule |= DIRTY_SUBMODULE_UNTRACKED;\n+\n+\tif (str[0] == 'u' ||\n+\t str[0] == '1' ||\n+\t str[0] == '2') {\n+\t\t/* T = line type, XY = status, SSSS = submodule state */\n+\t\tif (len < strlen(\"T XY SSSS\"))\n+\t\t\tBUG(\"invalid status --porcelain=2 line %s\",\n+\t\t\t str);\n+\n+\t\tif (str[5] == 'S' && str[8] == 'U')\n+\t\t\t/* nested untracked file */\n+\t\t\t*dirty_submodule |= DIRTY_SUBMODULE_UNTRACKED;\n+\n+\t\tif (str[0] == 'u' ||\n+\t\t str[0] == '2' ||\n+\t\t memcmp(str + 5, \"S..U\", 4))\n+\t\t\t/* other change */\n+\t\t\t*dirty_submodule |= DIRTY_SUBMODULE_MODIFIED;\n+\t}\n+\n+\tif ((*dirty_submodule & DIRTY_SUBMODULE_MODIFIED) &&\n+\t ((*dirty_submodule & DIRTY_SUBMODULE_UNTRACKED) ||\n+\t ignore_untracked)) {\n+\t\t/*\n+\t\t* We're not interested in any further information from\n+\t\t* the child any more, neither output nor its exit code.\n+\t\t*/\n+\t\treturn 1;\n+\t}\n+\treturn 0;\n+}\n+\n unsigned is_submodule_modified(const char *path, int ignore_untracked)\n {\n \tstruct child_process cp = CHILD_PROCESS_INIT;\n@@ -1909,39 +1948,10 @@ unsigned is_submodule_modified(const char *path, int ignore_untracked)\n \t\tchar *str = buf.buf;\n \t\tconst size_t len = buf.len;\n \n-\t\t/* regular untracked files */\n-\t\tif (str[0] == '?')\n-\t\t\tdirty_submodule |= DIRTY_SUBMODULE_UNTRACKED;\n-\n-\t\tif (str[0] == 'u' ||\n-\t\t str[0] == '1' ||\n-\t\t str[0] == '2') {\n-\t\t\t/* T = line type, XY = status, SSSS = submodule state */\n-\t\t\tif (len < strlen(\"T XY SSSS\"))\n-\t\t\t\tBUG(\"invalid status --porcelain=2 line %s\",\n-\t\t\t\t str);\n-\n-\t\t\tif (str[5] == 'S' && str[8] == 'U')\n-\t\t\t\t/* nested untracked file */\n-\t\t\t\tdirty_submodule |= DIRTY_SUBMODULE_UNTRACKED;\n-\n-\t\t\tif (str[0] == 'u' ||\n-\t\t\t str[0] == '2' ||\n-\t\t\t memcmp(str + 5, \"S..U\", 4))\n-\t\t\t\t/* other change */\n-\t\t\t\tdirty_submodule |= DIRTY_SUBMODULE_MODIFIED;\n-\t\t}\n-\n-\t\tif ((dirty_submodule & DIRTY_SUBMODULE_MODIFIED) &&\n-\t\t ((dirty_submodule & DIRTY_SUBMODULE_UNTRACKED) ||\n-\t\t ignore_untracked)) {\n-\t\t\t/*\n-\t\t\t * We're not interested in any further information from\n-\t\t\t * the child any more, neither output nor its exit code.\n-\t\t\t */\n-\t\t\tignore_cp_exit_code = 1;\n+\t\tignore_cp_exit_code = parse_status_porcelain(str, len, &dirty_submodule,\n+\t\t\t\t\t\t\t ignore_untracked);\n+\t\tif (ignore_cp_exit_code)\n \t\t\tbreak;\n-\t\t}\n \t}\n \tfclose(fp);\n \n-- \n2.39.0.314.g84b9a713c41-goog\n\n"},{"id":"470534","messageId":"20230117193041.708692-5-calvinwan@google.com","threadId":"59034","inReplyTo":"20230104215415.1083526-1-calvinwan@google.com","subject":"[PATCH v6 4/6] diff-lib: refactor match_stat_with_submodule","fromName":"Calvin Wan","fromEmail":"calvinwan@google.com","sentAt":"2023-01-17T19:30:39Z","receivedAt":"2023-01-17T21:07:03Z","isPatch":true,"sender":{"key":"calvinwan@google.com","avatar":"https://avatars.githubusercontent.com/u/92547554?v=4"},"body":"Flatten out the if statements in match_stat_with_submodule so the\nlogic is more readable and easier for future patches to add to.\norig_flags didn't need to be set if the cache entry wasn't a\nGITLINK so defer setting it.\n\nSigned-off-by: Calvin Wan \n---\n diff-lib.c | 28 +++++++++++++++++-----------\n 1 file changed, 17 insertions(+), 11 deletions(-)\n\ndiff --git a/diff-lib.c b/diff-lib.c\nindex dec040c366..64583fded0 100644\n--- a/diff-lib.c\n+++ b/diff-lib.c\n@@ -73,18 +73,24 @@ static int match_stat_with_submodule(struct diff_options *diffopt,\n \t\t\t\t unsigned *dirty_submodule)\n {\n \tint changed = ie_match_stat(diffopt->repo->index, ce, st, ce_option);\n-\tif (S_ISGITLINK(ce->ce_mode)) {\n-\t\tstruct diff_flags orig_flags = diffopt->flags;\n-\t\tif (!diffopt->flags.override_submodule_config)\n-\t\t\tset_diffopt_flags_from_submodule_config(diffopt, ce->name);\n-\t\tif (diffopt->flags.ignore_submodules)\n-\t\t\tchanged = 0;\n-\t\telse if (!diffopt->flags.ignore_dirty_submodules &&\n-\t\t\t (!changed || diffopt->flags.dirty_submodules))\n-\t\t\t*dirty_submodule = is_submodule_modified(ce->name,\n-\t\t\t\t\t\t\t\t diffopt->flags.ignore_untracked_in_submodules);\n-\t\tdiffopt->flags = orig_flags;\n+\tstruct diff_flags orig_flags;\n+\n+\tif (!S_ISGITLINK(ce->ce_mode))\n+\t\treturn changed;\n+\n+\torig_flags = diffopt->flags;\n+\tif (!diffopt->flags.override_submodule_config)\n+\t\tset_diffopt_flags_from_submodule_config(diffopt, ce->name);\n+\tif (diffopt->flags.ignore_submodules) {\n+\t\tchanged = 0;\n+\t\tgoto cleanup;\n \t}\n+\tif (!diffopt->flags.ignore_dirty_submodules &&\n+\t (!changed || diffopt->flags.dirty_submodules))\n+\t\t*dirty_submodule = is_submodule_modified(ce->name,\n+\t\t\t\t\t diffopt->flags.ignore_untracked_in_submodules);\n+cleanup:\n+\tdiffopt->flags = orig_flags;\n \treturn changed;\n }\n \n-- \n2.39.0.314.g84b9a713c41-goog\n\n"},{"id":"470535","messageId":"20230117193041.708692-6-calvinwan@google.com","threadId":"59034","inReplyTo":"20230104215415.1083526-1-calvinwan@google.com","subject":"[PATCH v6 5/6] diff-lib: parallelize run_diff_files for submodules","fromName":"Calvin Wan","fromEmail":"calvinwan@google.com","sentAt":"2023-01-17T19:30:40Z","receivedAt":"2023-01-17T21:07:13Z","isPatch":true,"sender":{"key":"calvinwan@google.com","avatar":"https://avatars.githubusercontent.com/u/92547554?v=4"},"body":"During the iteration of the index entries in run_diff_files, whenever\na submodule is found and needs its status checked, a subprocess is\nspawned for it. Instead of spawning the subprocess immediately and\nwaiting for its completion to continue, hold onto all submodules and\nrelevant information in a list. Then use that list to create tasks for\nrun_processes_parallel. Subprocess output is duplicated and passed to\nstatus_pipe_output which stores it to be parsed on completion of the\nsubprocess.\n\nAdd config option submodule.diffJobs to set the maximum number\nof parallel jobs. The option defaults to 1 if unset. If set to 0, the\nnumber of jobs is set to online_cpus().\n\nSince run_diff_files is called from many different commands, I chose\nto grab the config option in the function rather than adding variables\nto every git command and then figuring out how to pass them all in.\n\nSigned-off-by: Calvin Wan \n---\n Documentation/config/submodule.txt | 12 ++\n diff-lib.c | 84 ++++++++++++--\n submodule.c | 169 +++++++++++++++++++++++++++++\n submodule.h | 9 ++\n t/t4027-diff-submodule.sh | 19 ++++\n t/t7506-status-submodule.sh | 19 ++++\n 6 files changed, 305 insertions(+), 7 deletions(-)\n\ndiff --git a/Documentation/config/submodule.txt b/Documentation/config/submodule.txt\nindex 6490527b45..3209eb8117 100644\n--- a/Documentation/config/submodule.txt\n+++ b/Documentation/config/submodule.txt\n@@ -93,6 +93,18 @@ submodule.fetchJobs::\n \tin parallel. A value of 0 will give some reasonable default.\n \tIf unset, it defaults to 1.\n \n+submodule.diffJobs::\n+\tSpecifies how many submodules are diffed at the same time. A\n+\tpositive integer allows up to that number of submodules diffed\n+\tin parallel. A value of 0 will give some reasonable default.\n+\tIf unset, it defaults to 1. The diff operation is used by many\n+\tother git commands such as add, merge, diff, status, stash and\n+\tmore. Note that the expensive part of the diff operation is\n+\treading the index from cache or memory. Therefore multiple jobs\n+\tmay be detrimental to performance if your hardware does not\n+\tsupport parallel reads or if the number of jobs greatly exceeds\n+\tthe amount of supported reads.\n+\n submodule.alternateLocation::\n \tSpecifies how the submodules obtain alternates when submodules are\n \tcloned. Possible values are `no`, `superproject`.\ndiff --git a/diff-lib.c b/diff-lib.c\nindex 64583fded0..f51ea07f36 100644\n--- a/diff-lib.c\n+++ b/diff-lib.c\n@@ -14,6 +14,7 @@\n #include \"dir.h\"\n #include \"fsmonitor.h\"\n #include \"commit-reach.h\"\n+#include \"config.h\"\n \n /*\n * diff-files\n@@ -65,18 +66,23 @@ static int check_removed(const struct index_state *istate, const struct cache_en\n * Return 1 when changes are detected, 0 otherwise. If the DIRTY_SUBMODULES\n * option is set, the caller does not only want to know if a submodule is\n * modified at all but wants to know all the conditions that are met (new\n- * commits, untracked content and/or modified content).\n+ * commits, untracked content and/or modified content). If\n+ * defer_submodule_status bit is set, dirty_submodule will be left to the\n+ * caller to set. defer_submodule_status can also be set to 0 in this\n+ * function if there is no need to check if the submodule is modified.\n */\n static int match_stat_with_submodule(struct diff_options *diffopt,\n \t\t\t\t const struct cache_entry *ce,\n \t\t\t\t struct stat *st, unsigned ce_option,\n-\t\t\t\t unsigned *dirty_submodule)\n+\t\t\t\t unsigned *dirty_submodule, int *defer_submodule_status,\n+\t\t\t\t unsigned *ignore_untracked)\n {\n \tint changed = ie_match_stat(diffopt->repo->index, ce, st, ce_option);\n \tstruct diff_flags orig_flags;\n+\tint defer = 0;\n \n \tif (!S_ISGITLINK(ce->ce_mode))\n-\t\treturn changed;\n+\t\tgoto ret;\n \n \torig_flags = diffopt->flags;\n \tif (!diffopt->flags.override_submodule_config)\n@@ -86,11 +92,20 @@ static int match_stat_with_submodule(struct diff_options *diffopt,\n \t\tgoto cleanup;\n \t}\n \tif (!diffopt->flags.ignore_dirty_submodules &&\n-\t (!changed || diffopt->flags.dirty_submodules))\n-\t\t*dirty_submodule = is_submodule_modified(ce->name,\n+\t (!changed || diffopt->flags.dirty_submodules)) {\n+\t\tif (defer_submodule_status && *defer_submodule_status) {\n+\t\t\tdefer = 1;\n+\t\t\t*ignore_untracked = diffopt->flags.ignore_untracked_in_submodules;\n+\t\t} else {\n+\t\t\t*dirty_submodule = is_submodule_modified(ce->name,\n \t\t\t\t\t diffopt->flags.ignore_untracked_in_submodules);\n+\t\t}\n+\t}\n cleanup:\n \tdiffopt->flags = orig_flags;\n+ret:\n+\tif (defer_submodule_status)\n+\t\t*defer_submodule_status = defer;\n \treturn changed;\n }\n \n@@ -102,6 +117,7 @@ int run_diff_files(struct rev_info *revs, unsigned int option)\n \t\t\t ? CE_MATCH_RACY_IS_DIRTY : 0);\n \tuint64_t start = getnanotime();\n \tstruct index_state *istate = revs->diffopt.repo->index;\n+\tstruct string_list submodules = STRING_LIST_INIT_NODUP;\n \n \tdiff_set_mnemonic_prefix(&revs->diffopt, \"i/\", \"w/\");\n \n@@ -226,6 +242,8 @@ int run_diff_files(struct rev_info *revs, unsigned int option)\n \t\t\tnewmode = ce->ce_mode;\n \t\t} else {\n \t\t\tstruct stat st;\n+\t\t\tunsigned ignore_untracked = 0;\n+\t\t\tint defer_submodule_status = !!revs->repo;\n \n \t\t\tchanged = check_removed(istate, ce, &st);\n \t\t\tif (changed) {\n@@ -247,8 +265,26 @@ int run_diff_files(struct rev_info *revs, unsigned int option)\n \t\t\t}\n \n \t\t\tchanged = match_stat_with_submodule(&revs->diffopt, ce, &st,\n-\t\t\t\t\t\t\t ce_option, &dirty_submodule);\n+\t\t\t\t\t\t\t ce_option, &dirty_submodule,\n+\t\t\t\t\t\t\t &defer_submodule_status,\n+\t\t\t\t\t\t\t &ignore_untracked);\n \t\t\tnewmode = ce_mode_from_stat(ce, st.st_mode);\n+\t\t\tif (defer_submodule_status) {\n+\t\t\t\tstruct submodule_status_util tmp = {\n+\t\t\t\t\t.changed = changed,\n+\t\t\t\t\t.dirty_submodule = 0,\n+\t\t\t\t\t.ignore_untracked = ignore_untracked,\n+\t\t\t\t\t.newmode = newmode,\n+\t\t\t\t\t.ce = ce,\n+\t\t\t\t\t.path = ce->name,\n+\t\t\t\t};\n+\t\t\t\tstruct string_list_item *item;\n+\n+\t\t\t\titem = string_list_append(&submodules, ce->name);\n+\t\t\t\titem->util = xmalloc(sizeof(tmp));\n+\t\t\t\tmemcpy(item->util, &tmp, sizeof(tmp));\n+\t\t\t\tcontinue;\n+\t\t\t}\n \t\t}\n \n \t\tif (!changed && !dirty_submodule) {\n@@ -267,6 +303,40 @@ int run_diff_files(struct rev_info *revs, unsigned int option)\n \t\t\t ce->name, 0, dirty_submodule);\n \n \t}\n+\tif (submodules.nr > 0) {\n+\t\tint parallel_jobs;\n+\t\tif (git_config_get_int(\"submodule.diffjobs\", ¶llel_jobs))\n+\t\t\tparallel_jobs = 1;\n+\t\telse if (!parallel_jobs)\n+\t\t\tparallel_jobs = online_cpus();\n+\t\telse if (parallel_jobs < 0)\n+\t\t\tdie(_(\"submodule.diffjobs cannot be negative\"));\n+\n+\t\tif (get_submodules_status(&submodules, parallel_jobs))\n+\t\t\tdie(_(\"submodule status failed\"));\n+\t\tfor (size_t i = 0; i < submodules.nr; i++) {\n+\t\t\tstruct submodule_status_util *util = submodules.items[i].util;\n+\t\t\tstruct cache_entry *ce = util->ce;\n+\t\t\tunsigned int oldmode;\n+\t\t\tconst struct object_id *old_oid, *new_oid;\n+\n+\t\t\tif (!util->changed && !util->dirty_submodule) {\n+\t\t\t\tce_mark_uptodate(ce);\n+\t\t\t\tmark_fsmonitor_valid(istate, ce);\n+\t\t\t\tif (!revs->diffopt.flags.find_copies_harder)\n+\t\t\t\t\tcontinue;\n+\t\t\t}\n+\t\t\toldmode = ce->ce_mode;\n+\t\t\told_oid = &ce->oid;\n+\t\t\tnew_oid = util->changed ? null_oid() : &ce->oid;\n+\t\t\tdiff_change(&revs->diffopt, oldmode, util->newmode,\n+\t\t\t\t old_oid, new_oid,\n+\t\t\t\t !is_null_oid(old_oid),\n+\t\t\t\t !is_null_oid(new_oid),\n+\t\t\t\t ce->name, 0, util->dirty_submodule);\n+\t\t}\n+\t}\n+\tstring_list_clear(&submodules, 1);\n \tdiffcore_std(&revs->diffopt);\n \tdiff_flush(&revs->diffopt);\n \ttrace_performance_since(start, \"diff-files\");\n@@ -314,7 +384,7 @@ static int get_stat_data(const struct index_state *istate,\n \t\t\treturn -1;\n \t\t}\n \t\tchanged = match_stat_with_submodule(diffopt, ce, &st,\n-\t\t\t\t\t\t 0, dirty_submodule);\n+\t\t\t\t\t\t 0, dirty_submodule, NULL, NULL);\n \t\tif (changed) {\n \t\t\tmode = ce_mode_from_stat(ce, st.st_mode);\n \t\t\toid = null_oid();\ndiff --git a/submodule.c b/submodule.c\nindex 768d4b4cd7..da95ea1f5e 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -1369,6 +1369,17 @@ int submodule_touches_in_range(struct repository *r,\n \treturn ret;\n }\n \n+struct submodule_parallel_status {\n+\tsize_t index_count;\n+\tint result;\n+\n+\tstruct string_list *submodule_names;\n+\n+\t/* Pending statuses by OIDs */\n+\tstruct status_task **oid_status_tasks;\n+\tint oid_status_tasks_nr, oid_status_tasks_alloc;\n+};\n+\n struct submodule_parallel_fetch {\n \t/*\n \t * The index of the last index entry processed by\n@@ -1451,6 +1462,12 @@ struct fetch_task {\n \tstruct oid_array *commits; /* Ensure these commits are fetched */\n };\n \n+struct status_task {\n+\tconst char *path;\n+\tstruct strbuf out;\n+\tint ignore_untracked;\n+};\n+\n /**\n * When a submodule is not defined in .gitmodules, we cannot access it\n * via the regular submodule-config. Create a fake submodule, which we can\n@@ -1909,6 +1926,25 @@ static int parse_status_porcelain(char *str, size_t len,\n \treturn 0;\n }\n \n+static void parse_status_porcelain_strbuf(struct strbuf *buf,\n+\t\t\t\t unsigned *dirty_submodule,\n+\t\t\t\t int ignore_untracked)\n+{\n+\tstruct string_list list = STRING_LIST_INIT_DUP;\n+\tstruct string_list_item *item;\n+\n+\tstring_list_split(&list, buf->buf, '\\n', -1);\n+\n+\tfor_each_string_list_item(item, &list) {\n+\t\tif (parse_status_porcelain(item->string,\n+\t\t\t\t\t strlen(item->string),\n+\t\t\t\t\t dirty_submodule,\n+\t\t\t\t\t ignore_untracked))\n+\t\t\tbreak;\n+\t}\n+\tstring_list_clear(&list, 0);\n+}\n+\n unsigned is_submodule_modified(const char *path, int ignore_untracked)\n {\n \tstruct child_process cp = CHILD_PROCESS_INIT;\n@@ -1962,6 +1998,139 @@ unsigned is_submodule_modified(const char *path, int ignore_untracked)\n \treturn dirty_submodule;\n }\n \n+static struct status_task *\n+get_status_task_from_index(struct submodule_parallel_status *sps,\n+\t\t\t struct strbuf *err)\n+{\n+\tfor (; sps->index_count < sps->submodule_names->nr; sps->index_count++) {\n+\t\tstruct submodule_status_util *util = sps->submodule_names->items[sps->index_count].util;\n+\t\tstruct status_task *task;\n+\t\tstruct strbuf buf = STRBUF_INIT;\n+\t\tconst char *git_dir;\n+\n+\t\tstrbuf_addf(&buf, \"%s/.git\", util->path);\n+\t\tgit_dir = read_gitfile(buf.buf);\n+\t\tif (!git_dir)\n+\t\t\tgit_dir = buf.buf;\n+\t\tif (!is_git_directory(git_dir)) {\n+\t\t\tif (is_directory(git_dir))\n+\t\t\t\tdie(_(\"'%s' not recognized as a git repository\"), git_dir);\n+\t\t\tstrbuf_release(&buf);\n+\t\t\t/* The submodule is not checked out, so it is not modified */\n+\t\t\tutil->dirty_submodule = 0;\n+\t\t\tcontinue;\n+\t\t}\n+\t\tstrbuf_release(&buf);\n+\n+\t\ttask = xmalloc(sizeof(*task));\n+\t\ttask->path = util->path;\n+\t\ttask->ignore_untracked = util->ignore_untracked;\n+\t\tstrbuf_init(&task->out, 0);\n+\t\tsps->index_count++;\n+\t\treturn task;\n+\t}\n+\treturn NULL;\n+}\n+\n+static int get_next_submodule_status(struct child_process *cp,\n+\t\t\t\t struct strbuf *err, void *data,\n+\t\t\t\t void **task_cb)\n+{\n+\tstruct submodule_parallel_status *sps = data;\n+\tstruct status_task *task = get_status_task_from_index(sps, err);\n+\n+\tif (!task)\n+\t\treturn 0;\n+\n+\tchild_process_init(cp);\n+\tprepare_submodule_repo_env_in_gitdir(&cp->env);\n+\n+\tstrvec_init(&cp->args);\n+\tstrvec_pushl(&cp->args, \"status\", \"--porcelain=2\", NULL);\n+\tif (task->ignore_untracked)\n+\t\tstrvec_push(&cp->args, \"-uno\");\n+\n+\tprepare_submodule_repo_env(&cp->env);\n+\tcp->git_cmd = 1;\n+\tcp->dir = task->path;\n+\t*task_cb = task;\n+\treturn 1;\n+}\n+\n+static int status_start_failure(struct strbuf *err,\n+\t\t\t\tvoid *cb, void *task_cb)\n+{\n+\tstruct submodule_parallel_status *sps = cb;\n+\tstruct status_task *task = task_cb;\n+\n+\tsps->result = 1;\n+\tstrbuf_addf(err,\n+\t _(\"could not run 'git status --porcelain=2' in submodule %s\"),\n+\t task->path);\n+\treturn 0;\n+}\n+\n+static void status_duplicate_output(struct strbuf *out,\n+\t\t\t\t size_t offset,\n+\t\t\t\t void *cb, void *task_cb)\n+{\n+\tstruct status_task *task = task_cb;\n+\n+\tstrbuf_add(&task->out, out->buf + offset, out->len - offset);\n+\tstrbuf_setlen(out, offset);\n+}\n+\n+static int status_finish(int retvalue, struct strbuf *err,\n+\t\t\t void *cb, void *task_cb)\n+{\n+\tstruct submodule_parallel_status *sps = cb;\n+\tstruct status_task *task = task_cb;\n+\tstruct string_list_item *it =\n+\t\tstring_list_lookup(sps->submodule_names, task->path);\n+\tstruct submodule_status_util *util = it->util;\n+\n+\tif (retvalue) {\n+\t\tsps->result = 1;\n+\t\tstrbuf_addf(err,\n+\t\t _(\"'git status --porcelain=2' failed in submodule %s\"),\n+\t\t task->path);\n+\t}\n+\n+\tparse_status_porcelain_strbuf(&task->out,\n+\t\t\t &util->dirty_submodule,\n+\t\t\t util->ignore_untracked);\n+\n+\tstrbuf_release(&task->out);\n+\tfree(task);\n+\n+\treturn 0;\n+}\n+\n+int get_submodules_status(struct string_list *submodules,\n+\t\t\t int max_parallel_jobs)\n+{\n+\tstruct submodule_parallel_status sps = {\n+\t\t.submodule_names = submodules,\n+\t};\n+\tconst struct run_process_parallel_opts opts = {\n+\t\t.tr2_category = \"submodule\",\n+\t\t.tr2_label = \"parallel/status\",\n+\n+\t\t.processes = max_parallel_jobs,\n+\n+\t\t.get_next_task = get_next_submodule_status,\n+\t\t.start_failure = status_start_failure,\n+\t\t.duplicate_output = status_duplicate_output,\n+\t\t.task_finished = status_finish,\n+\t\t.data = &sps,\n+\t};\n+\n+\tstring_list_sort(sps.submodule_names);\n+\trun_processes_parallel(&opts);\n+\n+\treturn sps.result;\n+}\n+\n int submodule_uses_gitfile(const char *path)\n {\n \tstruct child_process cp = CHILD_PROCESS_INIT;\ndiff --git a/submodule.h b/submodule.h\nindex b52a4ff1e7..08d278a414 100644\n--- a/submodule.h\n+++ b/submodule.h\n@@ -41,6 +41,13 @@ struct submodule_update_strategy {\n \t.type = SM_UPDATE_UNSPECIFIED, \\\n }\n \n+struct submodule_status_util {\n+\tint changed, ignore_untracked;\n+\tunsigned dirty_submodule, newmode;\n+\tstruct cache_entry *ce;\n+\tconst char *path;\n+};\n+\n int is_gitmodules_unmerged(struct index_state *istate);\n int is_writing_gitmodules_ok(void);\n int is_staging_gitmodules_ok(struct index_state *istate);\n@@ -94,6 +101,8 @@ int fetch_submodules(struct repository *r,\n \t\t int command_line_option,\n \t\t int default_option,\n \t\t int quiet, int max_parallel_jobs);\n+int get_submodules_status(struct string_list *submodules,\n+\t\t\t int max_parallel_jobs);\n unsigned is_submodule_modified(const char *path, int ignore_untracked);\n int submodule_uses_gitfile(const char *path);\n \ndiff --git a/t/t4027-diff-submodule.sh b/t/t4027-diff-submodule.sh\nindex 40164ae07d..e08ee315a7 100755\n--- a/t/t4027-diff-submodule.sh\n+++ b/t/t4027-diff-submodule.sh\n@@ -34,6 +34,25 @@ test_expect_success setup '\n \tsubtip=$3 subprev=$2\n '\n \n+test_expect_success 'diff in superproject with submodules respects parallel settings' '\n+\ttest_when_finished \"rm -f trace.out\" &&\n+\t(\n+\t\tGIT_TRACE=$(pwd)/trace.out git diff &&\n+\t\tgrep \"1 tasks\" trace.out &&\n+\t\t>trace.out &&\n+\n+\t\tgit config submodule.diffJobs 8 &&\n+\t\tGIT_TRACE=$(pwd)/trace.out git diff &&\n+\t\tgrep \"8 tasks\" trace.out &&\n+\t\t>trace.out &&\n+\n+\t\tGIT_TRACE=$(pwd)/trace.out git -c submodule.diffJobs=0 diff &&\n+\t\tgrep \"preparing to run up to [0-9]* tasks\" trace.out &&\n+\t\t! grep \"up to 0 tasks\" trace.out &&\n+\t\t>trace.out\n+\t)\n+'\n+\n test_expect_success 'git diff --raw HEAD' '\n \thexsz=$(test_oid hexsz) &&\n \tgit diff --raw --abbrev=$hexsz HEAD >actual &&\ndiff --git a/t/t7506-status-submodule.sh b/t/t7506-status-submodule.sh\nindex d050091345..52a82b703f 100755\n--- a/t/t7506-status-submodule.sh\n+++ b/t/t7506-status-submodule.sh\n@@ -412,4 +412,23 @@ test_expect_success 'status with added file in nested submodule (short)' '\n \tEOF\n '\n \n+test_expect_success 'status in superproject with submodules respects parallel settings' '\n+\ttest_when_finished \"rm -f trace.out\" &&\n+\t(\n+\t\tGIT_TRACE=$(pwd)/trace.out git status &&\n+\t\tgrep \"1 tasks\" trace.out &&\n+\t\t>trace.out &&\n+\n+\t\tgit config submodule.diffJobs 8 &&\n+\t\tGIT_TRACE=$(pwd)/trace.out git status &&\n+\t\tgrep \"8 tasks\" trace.out &&\n+\t\t>trace.out &&\n+\n+\t\tGIT_TRACE=$(pwd)/trace.out git -c submodule.diffJobs=0 status &&\n+\t\tgrep \"preparing to run up to [0-9]* tasks\" trace.out &&\n+\t\t! grep \"up to 0 tasks\" trace.out &&\n+\t\t>trace.out\n+\t)\n+'\n+\n test_done\n-- \n2.39.0.314.g84b9a713c41-goog\n\n"},{"id":"470536","messageId":"20230117193041.708692-7-calvinwan@google.com","threadId":"59034","inReplyTo":"20230104215415.1083526-1-calvinwan@google.com","subject":"[PATCH v6 6/6] submodule: call parallel code from serial status","fromName":"Calvin Wan","fromEmail":"calvinwan@google.com","sentAt":"2023-01-17T19:30:41Z","receivedAt":"2023-01-17T21:07:16Z","isPatch":true,"sender":{"key":"calvinwan@google.com","avatar":"https://avatars.githubusercontent.com/u/92547554?v=4"},"body":"Remove the serial implementation of status inside of\nis_submodule_modified since the parallel implementation of status with\none job accomplishes the same task.\n\nCombine parse_status_porcelain and parse_status_porcelain_strbuf since\nthe only other caller of parse_status_porcelain was in\nis_submodule_modified\n\nSigned-off-by: Calvin Wan \n---\n submodule.c | 146 ++++++++++++++++++----------------------------------\n 1 file changed, 51 insertions(+), 95 deletions(-)\n\ndiff --git a/submodule.c b/submodule.c\nindex da95ea1f5e..2009748d9f 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -1887,46 +1887,7 @@ int fetch_submodules(struct repository *r,\n \treturn spf.result;\n }\n \n-static int parse_status_porcelain(char *str, size_t len,\n-\t\t\t\t unsigned *dirty_submodule,\n-\t\t\t\t int ignore_untracked)\n-{\n-\t/* regular untracked files */\n-\tif (str[0] == '?')\n-\t\t*dirty_submodule |= DIRTY_SUBMODULE_UNTRACKED;\n-\n-\tif (str[0] == 'u' ||\n-\t str[0] == '1' ||\n-\t str[0] == '2') {\n-\t\t/* T = line type, XY = status, SSSS = submodule state */\n-\t\tif (len < strlen(\"T XY SSSS\"))\n-\t\t\tBUG(\"invalid status --porcelain=2 line %s\",\n-\t\t\t str);\n-\n-\t\tif (str[5] == 'S' && str[8] == 'U')\n-\t\t\t/* nested untracked file */\n-\t\t\t*dirty_submodule |= DIRTY_SUBMODULE_UNTRACKED;\n-\n-\t\tif (str[0] == 'u' ||\n-\t\t str[0] == '2' ||\n-\t\t memcmp(str + 5, \"S..U\", 4))\n-\t\t\t/* other change */\n-\t\t\t*dirty_submodule |= DIRTY_SUBMODULE_MODIFIED;\n-\t}\n-\n-\tif ((*dirty_submodule & DIRTY_SUBMODULE_MODIFIED) &&\n-\t ((*dirty_submodule & DIRTY_SUBMODULE_UNTRACKED) ||\n-\t ignore_untracked)) {\n-\t\t/*\n-\t\t* We're not interested in any further information from\n-\t\t* the child any more, neither output nor its exit code.\n-\t\t*/\n-\t\treturn 1;\n-\t}\n-\treturn 0;\n-}\n-\n-static void parse_status_porcelain_strbuf(struct strbuf *buf,\n+static void parse_status_porcelain(struct strbuf *buf,\n \t\t\t\t unsigned *dirty_submodule,\n \t\t\t\t int ignore_untracked)\n {\n@@ -1936,65 +1897,60 @@ static void parse_status_porcelain_strbuf(struct strbuf *buf,\n \tstring_list_split(&list, buf->buf, '\\n', -1);\n \n \tfor_each_string_list_item(item, &list) {\n-\t\tif (parse_status_porcelain(item->string,\n-\t\t\t\t\t strlen(item->string),\n-\t\t\t\t\t dirty_submodule,\n-\t\t\t\t\t ignore_untracked))\n+\t\tchar *str = item->string;\n+\t\t/* regular untracked files */\n+\t\tif (str[0] == '?')\n+\t\t\t*dirty_submodule |= DIRTY_SUBMODULE_UNTRACKED;\n+\n+\t\tif (str[0] == 'u' ||\n+\t\tstr[0] == '1' ||\n+\t\tstr[0] == '2') {\n+\t\t\t/* T = line type, XY = status, SSSS = submodule state */\n+\t\t\tif (strlen(str) < strlen(\"T XY SSSS\"))\n+\t\t\t\tBUG(\"invalid status --porcelain=2 line %s\",\n+\t\t\t\tstr);\n+\n+\t\t\tif (str[5] == 'S' && str[8] == 'U')\n+\t\t\t\t/* nested untracked file */\n+\t\t\t\t*dirty_submodule |= DIRTY_SUBMODULE_UNTRACKED;\n+\n+\t\t\tif (str[0] == 'u' ||\n+\t\t\tstr[0] == '2' ||\n+\t\t\tmemcmp(str + 5, \"S..U\", 4))\n+\t\t\t\t/* other change */\n+\t\t\t\t*dirty_submodule |= DIRTY_SUBMODULE_MODIFIED;\n+\t\t}\n+\n+\t\tif ((*dirty_submodule & DIRTY_SUBMODULE_MODIFIED) &&\n+\t\t ((*dirty_submodule & DIRTY_SUBMODULE_UNTRACKED) ||\n+\t\t ignore_untracked)) {\n+\t\t\t/*\n+\t\t\t* We're not interested in any further information from\n+\t\t\t* the child any more, neither output nor its exit code.\n+\t\t\t*/\n \t\t\tbreak;\n+\t\t}\n \t}\n \tstring_list_clear(&list, 0);\n }\n \n unsigned is_submodule_modified(const char *path, int ignore_untracked)\n {\n-\tstruct child_process cp = CHILD_PROCESS_INIT;\n-\tstruct strbuf buf = STRBUF_INIT;\n-\tFILE *fp;\n-\tunsigned dirty_submodule = 0;\n-\tconst char *git_dir;\n-\tint ignore_cp_exit_code = 0;\n-\n-\tstrbuf_addf(&buf, \"%s/.git\", path);\n-\tgit_dir = read_gitfile(buf.buf);\n-\tif (!git_dir)\n-\t\tgit_dir = buf.buf;\n-\tif (!is_git_directory(git_dir)) {\n-\t\tif (is_directory(git_dir))\n-\t\t\tdie(_(\"'%s' not recognized as a git repository\"), git_dir);\n-\t\tstrbuf_release(&buf);\n-\t\t/* The submodule is not checked out, so it is not modified */\n-\t\treturn 0;\n-\t}\n-\tstrbuf_reset(&buf);\n-\n-\tstrvec_pushl(&cp.args, \"status\", \"--porcelain=2\", NULL);\n-\tif (ignore_untracked)\n-\t\tstrvec_push(&cp.args, \"-uno\");\n-\n-\tprepare_submodule_repo_env(&cp.env);\n-\tcp.git_cmd = 1;\n-\tcp.no_stdin = 1;\n-\tcp.out = -1;\n-\tcp.dir = path;\n-\tif (start_command(&cp))\n-\t\tdie(_(\"Could not run 'git status --porcelain=2' in submodule %s\"), path);\n-\n-\tfp = xfdopen(cp.out, \"r\");\n-\twhile (strbuf_getwholeline(&buf, fp, '\\n') != EOF) {\n-\t\tchar *str = buf.buf;\n-\t\tconst size_t len = buf.len;\n-\n-\t\tignore_cp_exit_code = parse_status_porcelain(str, len, &dirty_submodule,\n-\t\t\t\t\t\t\t ignore_untracked);\n-\t\tif (ignore_cp_exit_code)\n-\t\t\tbreak;\n-\t}\n-\tfclose(fp);\n-\n-\tif (finish_command(&cp) && !ignore_cp_exit_code)\n-\t\tdie(_(\"'git status --porcelain=2' failed in submodule %s\"), path);\n-\n-\tstrbuf_release(&buf);\n+\tstruct submodule_status_util util = {\n+\t\t.dirty_submodule = 0,\n+\t\t.ignore_untracked = ignore_untracked,\n+\t\t.path = path,\n+\t};\n+\tstruct string_list sub = STRING_LIST_INIT_NODUP;\n+\tstruct string_list_item *item;\n+\tint dirty_submodule;\n+\n+\titem = string_list_append(&sub, path);\n+\titem->util = &util;\n+\tif (get_submodules_status(&sub, 1))\n+\t\tdie(_(\"submodule status failed\"));\n+\tdirty_submodule = util.dirty_submodule;\n+\tstring_list_clear(&sub, 0);\n \treturn dirty_submodule;\n }\n \n@@ -2096,9 +2052,9 @@ static int status_finish(int retvalue, struct strbuf *err,\n \t\t task->path);\n \t}\n \n-\tparse_status_porcelain_strbuf(&task->out,\n-\t\t\t &util->dirty_submodule,\n-\t\t\t util->ignore_untracked);\n+\tparse_status_porcelain(&task->out,\n+\t\t\t &util->dirty_submodule,\n+\t\t\t util->ignore_untracked);\n \n \tstrbuf_release(&task->out);\n \tfree(task);\n-- \n2.39.0.314.g84b9a713c41-goog\n\n"},{"id":"471051","messageId":"kl6lv8ktvhrp.fsf@chooglen-macbookpro.roam.corp.google.com","threadId":"59034","inReplyTo":"20230117193041.708692-7-calvinwan@google.com","subject":"Re: [PATCH v6 6/6] submodule: call parallel code from serial status","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2023-01-26T08:09:46Z","receivedAt":"2023-01-26T08:09:57Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"Calvin Wan writes:\n\n> Remove the serial implementation of status inside of\n> is_submodule_modified since the parallel implementation of status with\n> one job accomplishes the same task.\n>\n> Combine parse_status_porcelain and parse_status_porcelain_strbuf since\n> the only other caller of parse_status_porcelain was in\n> is_submodule_modified\n\nI see that this is in direct response to Jonathan's earlier comment [1]\nthat we should have only one implementation. Thanks, this is helpful.\nDefinitely a step in the right direction.\n\nThat said, I don't think this patch's position in the series makes\nsense. I would have expected a patch like this to come before 5/6. I.e.\nthis series duplicates code in 5/6 and deletes it in 6/6 so that we only\nhave one implementation for both serial and parallel submodule status.\n\nInstead, I would have expected we would refactor out the serial\nimplementation, then use the refactored code for the parallel\nimplementation. Not having duplicated code in 5/6 would shrink the line\ncount a lot and make it easier to review.\n\n[1] https://lore.kernel.org/git/20221128210125.2751300-1-jonathantanmy@google.com/\n"},{"id":"471052","messageId":"kl6lo7qlvg4h.fsf@chooglen-macbookpro.roam.corp.google.com","threadId":"59034","inReplyTo":"kl6lv8ktvhrp.fsf@chooglen-macbookpro.roam.corp.google.com","subject":"Re: [PATCH v6 6/6] submodule: call parallel code from serial status","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2023-01-26T08:45:18Z","receivedAt":"2023-01-26T08:45:23Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"Glen Choo writes:\n\n> Calvin Wan writes:\n>\n>> Remove the serial implementation of status inside of\n>> is_submodule_modified since the parallel implementation of status with\n>> one job accomplishes the same task.\n>\n> I see that this is in direct response to Jonathan's earlier comment [1]\n> that we should have only one implementation. Thanks, this is helpful.\n> Definitely a step in the right direction.\n>\n> That said, I don't think this patch's position in the series makes\n> sense. I would have expected a patch like this to come before 5/6. I.e.\n> this series duplicates code in 5/6 and deletes it in 6/6 so that we only\n> have one implementation for both serial and parallel submodule status.\n>\n> Instead, I would have expected we would refactor out the serial\n> implementation, then use the refactored code for the parallel\n> implementation. Not having duplicated code in 5/6 would shrink the line\n> count a lot and make it easier to review.\n>\n> [1] https://lore.kernel.org/git/20221128210125.2751300-1-jonathantanmy@google.com/\n\nAh, I realize I completely misunderstood this patch. I thought that this\nwas deleting code that was duplicated between the serial and parallel\nimplementations in 5/6 such that both ended up sharing just one copy of\nthe code.\n\nInstead, this patch deletes the serial implementation altogether and\nreplaces it with the parallel one. As such, this patch can't come\nearlier than 5/6, because we need the parallel implementation to exist\nbefore we can use it.\n\nFor reviewability of 5/6, I'd still strongly prefer that we refactor out\nfunctions (I'll leave more specific comments on that patch). We could\nstill consider replacing the serial implementation with \"parallel with a\nsingle job\", though I suspect that it will be unnecessary if we do the\nrefactoring well. I'm also not sure how idiomatic it is to call\nrun_processes_parallel() with a hardcoded value of 1, but I don't feel\ntoo strongly about that.\n"},{"id":"471054","messageId":"kl6llelpvf0i.fsf@chooglen-macbookpro.roam.corp.google.com","threadId":"59034","inReplyTo":"20230117193041.708692-6-calvinwan@google.com","subject":"Re: [PATCH v6 5/6] diff-lib: parallelize run_diff_files for submodules","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2023-01-26T09:09:17Z","receivedAt":"2023-01-26T09:09:23Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"As Jonathan mentioned in [1], I think we should refactor functions out\nfrom the serial implementation in a preparatory patch, then use those\nfunctions to implement the parallel version in this patch. In its\ncurrent form, there is a fair amount of duplicated code, which makes it\ntricky to review because of the additional overhead of checking what the\nduplicated code does and whether we've copied it correcly.\n\nFor cleanliness, I'll only point out the duplicated code in this email;\nI'll comment on other things I spotted in another one.\n\n[1] https://lore.kernel.org/git/20221128210125.2751300-1-jonathantanmy@google.com/\n\nCalvin Wan writes:\n\n> +\t\tfor (size_t i = 0; i < submodules.nr; i++) {\n> +\t\t\tstruct submodule_status_util *util = submodules.items[i].util;\n> +\t\t\tstruct cache_entry *ce = util->ce;\n> +\t\t\tunsigned int oldmode;\n> +\t\t\tconst struct object_id *old_oid, *new_oid;\n> +\n> +\t\t\tif (!util->changed && !util->dirty_submodule) {\n> +\t\t\t\tce_mark_uptodate(ce);\n> +\t\t\t\tmark_fsmonitor_valid(istate, ce);\n> +\t\t\t\tif (!revs->diffopt.flags.find_copies_harder)\n> +\t\t\t\t\tcontinue;\n> +\t\t\t}\n> +\t\t\toldmode = ce->ce_mode;\n> +\t\t\told_oid = &ce->oid;\n> +\t\t\tnew_oid = util->changed ? null_oid() : &ce->oid;\n> +\t\t\tdiff_change(&revs->diffopt, oldmode, util->newmode,\n> +\t\t\t\t old_oid, new_oid,\n> +\t\t\t\t !is_null_oid(old_oid),\n> +\t\t\t\t !is_null_oid(new_oid),\n> +\t\t\t\t ce->name, 0, util->dirty_submodule);\n> +\t\t}\n> +\t}\n\nThe lines from \"if (!util->changed && !util->dirty_submodule)\" onwards\nare copied from earlier in run_diff_files(). This might be refactored\ninto something like diff_submodule_change().\n\n> +static struct status_task *\n> +get_status_task_from_index(struct submodule_parallel_status *sps,\n> +\t\t\t struct strbuf *err)\n> +{\n> +\tfor (; sps->index_count < sps->submodule_names->nr; sps->index_count++) {\n> +\t\tstruct submodule_status_util *util = sps->submodule_names->items[sps->index_count].util;\n> +\t\tstruct status_task *task;\n> +\t\tstruct strbuf buf = STRBUF_INIT;\n> +\t\tconst char *git_dir;\n> +\n> +\t\tstrbuf_addf(&buf, \"%s/.git\", util->path);\n> +\t\tgit_dir = read_gitfile(buf.buf);\n\nThis...\n\n> +static int get_next_submodule_status(struct child_process *cp,\n> +\t\t\t\t struct strbuf *err, void *data,\n> +\t\t\t\t void **task_cb)\n> +{\n> +\tstruct submodule_parallel_status *sps = data;\n> +\tstruct status_task *task = get_status_task_from_index(sps, err);\n> +\n> +\tif (!task)\n> +\t\treturn 0;\n> +\n> +\tchild_process_init(cp);\n> +\tprepare_submodule_repo_env_in_gitdir(&cp->env);\n> +\n> +\tstrvec_init(&cp->args);\n> +\tstrvec_pushl(&cp->args, \"status\", \"--porcelain=2\", NULL);\n> +\tif (task->ignore_untracked)\n> +\t\tstrvec_push(&cp->args, \"-uno\");\n> +\n> +\tprepare_submodule_repo_env(&cp->env);\n> +\tcp->git_cmd = 1;\n\nthis...\n\n> +static int status_start_failure(struct strbuf *err,\n> +\t\t\t\tvoid *cb, void *task_cb)\n> +{\n> +\tstruct submodule_parallel_status *sps = cb;\n> +\tstruct status_task *task = task_cb;\n> +\n> +\tsps->result = 1;\n> +\tstrbuf_addf(err,\n> +\t _(\"could not run 'git status --porcelain=2' in submodule %s\"),\n> +\t task->path);\n> +\treturn 0;\n> +}\n\nthis...\n\n> +static int status_finish(int retvalue, struct strbuf *err,\n> +\t\t\t void *cb, void *task_cb)\n> +{\n> +\tstruct submodule_parallel_status *sps = cb;\n> +\tstruct status_task *task = task_cb;\n> +\tstruct string_list_item *it =\n> +\t\tstring_list_lookup(sps->submodule_names, task->path);\n> +\tstruct submodule_status_util *util = it->util;\n> +\n> +\tif (retvalue) {\n> +\t\tsps->result = 1;\n> +\t\tstrbuf_addf(err,\n> +\t\t _(\"'git status --porcelain=2' failed in submodule %s\"),\n> +\t\t task->path);\n> +\t}\n\nand this are all copied from different parts of is_submodule_modified().\nTo refactor them out, I think we could combine the first two into\n\"setup_submodule_status()\". The last one could be moved into\n\"process_submodule_status_result()\" or perhaps we could find a way to\ncombine it into parse_status_porcelain().\n"},{"id":"471055","messageId":"kl6lilgtveoe.fsf@chooglen-macbookpro.roam.corp.google.com","threadId":"59034","inReplyTo":"20230117193041.708692-6-calvinwan@google.com","subject":"Re: [PATCH v6 5/6] diff-lib: parallelize run_diff_files for submodules","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2023-01-26T09:16:33Z","receivedAt":"2023-01-26T09:16:39Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"\nCalvin Wan writes:\n\n> @@ -226,6 +242,8 @@ int run_diff_files(struct rev_info *revs, unsigned int option)\n> \t\t\tnewmode = ce->ce_mode;\n> \t\t} else {\n> \t\t\tstruct stat st;\n> +\t\t\tunsigned ignore_untracked = 0;\n> +\t\t\tint defer_submodule_status = !!revs->repo;\n\nWhat is the reasoning behind this condition? I would expect revs->repo\nto always be set, and we would always end up deferring.\n\n> \t\t\tnewmode = ce_mode_from_stat(ce, st.st_mode);\n> +\t\t\tif (defer_submodule_status) {\n> +\t\t\t\tstruct submodule_status_util tmp = {\n> +\t\t\t\t\t.changed = changed,\n> +\t\t\t\t\t.dirty_submodule = 0,\n> +\t\t\t\t\t.ignore_untracked = ignore_untracked,\n> +\t\t\t\t\t.newmode = newmode,\n> +\t\t\t\t\t.ce = ce,\n> +\t\t\t\t\t.path = ce->name,\n> +\t\t\t\t};\n> +\t\t\t\tstruct string_list_item *item;\n> +\n> +\t\t\t\titem = string_list_append(&submodules, ce->name);\n> +\t\t\t\titem->util = xmalloc(sizeof(tmp));\n> +\t\t\t\tmemcpy(item->util, &tmp, sizeof(tmp));\n\n(Not a C expert) Since we don't return the string list, I wonder if we\ncan avoid the memcpy() by using &tmp like so:\n\n struct string_list_item *item;\n item = string_list_append(&submodules, ce->name);\n item->util = &tmp;\n\nAnd then when we call string_list_clear(), we wouldn't need to free the\nutil since we exit the stack frame.\n\n> +test_expect_success 'diff in superproject with submodules respects parallel settings' '\n> +\ttest_when_finished \"rm -f trace.out\" &&\n> +\t(\n> +\t\tGIT_TRACE=$(pwd)/trace.out git diff &&\n> +\t\tgrep \"1 tasks\" trace.out &&\n> +\t\t>trace.out &&\n> +\n> +\t\tgit config submodule.diffJobs 8 &&\n> +\t\tGIT_TRACE=$(pwd)/trace.out git diff &&\n> +\t\tgrep \"8 tasks\" trace.out &&\n> +\t\t>trace.out &&\n> +\n> +\t\tGIT_TRACE=$(pwd)/trace.out git -c submodule.diffJobs=0 diff &&\n> +\t\tgrep \"preparing to run up to [0-9]* tasks\" trace.out &&\n> +\t\t! grep \"up to 0 tasks\" trace.out &&\n> +\t\t>trace.out\n> +\t)\n> +'\n> +\n\nCould we get tests to check that the output of git diff isn't changed by\nsetting parallelism? This might not be feasible for submodule.diffJobs >\n1 due to raciness, but it would be good to see for submodule.diffJobs =\n1 at least.\n\n> test_expect_success 'git diff --raw HEAD' '\n> \thexsz=$(test_oid hexsz) &&\n> \tgit diff --raw --abbrev=$hexsz HEAD >actual &&\n> diff --git a/t/t7506-status-submodule.sh b/t/t7506-status-submodule.sh\n> index d050091345..52a82b703f 100755\n> --- a/t/t7506-status-submodule.sh\n> +++ b/t/t7506-status-submodule.sh\n> @@ -412,4 +412,23 @@ test_expect_success 'status with added file in nested submodule (short)' '\n> \tEOF\n> '\n> \n> +test_expect_success 'status in superproject with submodules respects parallel settings' '\n> +\ttest_when_finished \"rm -f trace.out\" &&\n> +\t(\n> +\t\tGIT_TRACE=$(pwd)/trace.out git status &&\n> +\t\tgrep \"1 tasks\" trace.out &&\n> +\t\t>trace.out &&\n> +\n> +\t\tgit config submodule.diffJobs 8 &&\n> +\t\tGIT_TRACE=$(pwd)/trace.out git status &&\n> +\t\tgrep \"8 tasks\" trace.out &&\n> +\t\t>trace.out &&\n> +\n> +\t\tGIT_TRACE=$(pwd)/trace.out git -c submodule.diffJobs=0 status &&\n> +\t\tgrep \"preparing to run up to [0-9]* tasks\" trace.out &&\n> +\t\t! grep \"up to 0 tasks\" trace.out &&\n> +\t\t>trace.out\n> +\t)\n> +'\n> +\n\nDitto for \"status\".\n"},{"id":"471076","messageId":"CAFySSZBiW7=ZTmXRaLzCoKUi0Jd=fzvW5PJ6=Ka0jKHoP2ddSw@mail.gmail.com","threadId":"59034","inReplyTo":"kl6lilgtveoe.fsf@chooglen-macbookpro.roam.corp.google.com","subject":"Re: [PATCH v6 5/6] diff-lib: parallelize run_diff_files for submodules","fromName":"Calvin Wan","fromEmail":"calvinwan@google.com","sentAt":"2023-01-26T18:52:17Z","receivedAt":"2023-01-26T18:52:32Z","isPatch":true,"sender":{"key":"calvinwan@google.com","avatar":"https://avatars.githubusercontent.com/u/92547554?v=4"},"body":"On Thu, Jan 26, 2023 at 1:16 AM Glen Choo wrote:\n>\n>\n> Calvin Wan writes:\n>\n> > @@ -226,6 +242,8 @@ int run_diff_files(struct rev_info *revs, unsigned int option)\n> > newmode = ce->ce_mode;\n> > } else {\n> > struct stat st;\n> > + unsigned ignore_untracked = 0;\n> > + int defer_submodule_status = !!revs->repo;\n>\n> What is the reasoning behind this condition? I would expect revs->repo\n> to always be set, and we would always end up deferring.\n\nAh looks like a vestigial sanity check. You're correct that we would\nalways be deferring anyways.\n\n>\n> > newmode = ce_mode_from_stat(ce, st.st_mode);\n> > + if (defer_submodule_status) {\n> > + struct submodule_status_util tmp = {\n> > + .changed = changed,\n> > + .dirty_submodule = 0,\n> > + .ignore_untracked = ignore_untracked,\n> > + .newmode = newmode,\n> > + .ce = ce,\n> > + .path = ce->name,\n> > + };\n> > + struct string_list_item *item;\n> > +\n> > + item = string_list_append(&submodules, ce->name);\n> > + item->util = xmalloc(sizeof(tmp));\n> > + memcpy(item->util, &tmp, sizeof(tmp));\n>\n> (Not a C expert) Since we don't return the string list, I wonder if we\n> can avoid the memcpy() by using &tmp like so:\n>\n> struct string_list_item *item;\n> item = string_list_append(&submodules, ce->name);\n> item->util = &tmp;\n>\n> And then when we call string_list_clear(), we wouldn't need to free the\n> util since we exit the stack frame.\n\nUnfortunately this doesn't work because tmp is deallocated off the stack\nafter changing scope.\n\n> > +test_expect_success 'diff in superproject with submodules respects parallel settings' '\n> > + test_when_finished \"rm -f trace.out\" &&\n> > + (\n> > + GIT_TRACE=$(pwd)/trace.out git diff &&\n> > + grep \"1 tasks\" trace.out &&\n> > + >trace.out &&\n> > +\n> > + git config submodule.diffJobs 8 &&\n> > + GIT_TRACE=$(pwd)/trace.out git diff &&\n> > + grep \"8 tasks\" trace.out &&\n> > + >trace.out &&\n> > +\n> > + GIT_TRACE=$(pwd)/trace.out git -c submodule.diffJobs=0 diff &&\n> > + grep \"preparing to run up to [0-9]* tasks\" trace.out &&\n> > + ! grep \"up to 0 tasks\" trace.out &&\n> > + >trace.out\n> > + )\n> > +'\n> > +\n>\n> Could we get tests to check that the output of git diff isn't changed by\n> setting parallelism? This might not be feasible for submodule.diffJobs >\n> 1 due to raciness, but it would be good to see for submodule.diffJobs =\n> 1 at least.\n\nack.\n\n>\n> > test_expect_success 'git diff --raw HEAD' '\n> > hexsz=$(test_oid hexsz) &&\n> > git diff --raw --abbrev=$hexsz HEAD >actual &&\n> > diff --git a/t/t7506-status-submodule.sh b/t/t7506-status-submodule.sh\n> > index d050091345..52a82b703f 100755\n> > --- a/t/t7506-status-submodule.sh\n> > +++ b/t/t7506-status-submodule.sh\n> > @@ -412,4 +412,23 @@ test_expect_success 'status with added file in nested submodule (short)' '\n> > EOF\n> > '\n> >\n> > +test_expect_success 'status in superproject with submodules respects parallel settings' '\n> > + test_when_finished \"rm -f trace.out\" &&\n> > + (\n> > + GIT_TRACE=$(pwd)/trace.out git status &&\n> > + grep \"1 tasks\" trace.out &&\n> > + >trace.out &&\n> > +\n> > + git config submodule.diffJobs 8 &&\n> > + GIT_TRACE=$(pwd)/trace.out git status &&\n> > + grep \"8 tasks\" trace.out &&\n> > + >trace.out &&\n> > +\n> > + GIT_TRACE=$(pwd)/trace.out git -c submodule.diffJobs=0 status &&\n> > + grep \"preparing to run up to [0-9]* tasks\" trace.out &&\n> > + ! grep \"up to 0 tasks\" trace.out &&\n> > + >trace.out\n> > + )\n> > +'\n> > +\n>\n> Ditto for \"status\".\n\nack.\n"},{"id":"471706","messageId":"20230207181706.363453-2-calvinwan@google.com","threadId":"59034","inReplyTo":"20230117193041.708692-1-calvinwan@google.com","subject":"[PATCH v7 1/7] run-command: add duplicate_output_fn to run_processes_parallel_opts","fromName":"Calvin Wan","fromEmail":"calvinwan@google.com","sentAt":"2023-02-07T18:17:00Z","receivedAt":"2023-02-07T18:18:03Z","isPatch":true,"sender":{"key":"calvinwan@google.com","avatar":"https://avatars.githubusercontent.com/u/92547554?v=4"},"body":"Add duplicate_output_fn as an optionally set function in\nrun_process_parallel_opts. If set, output from each child process is\ncopied and passed to the callback function whenever output from the\nchild process is buffered to allow for separate parsing.\n\nSigned-off-by: Calvin Wan \n---\n run-command.c | 16 ++++++++++++---\n run-command.h | 27 +++++++++++++++++++++++++\n t/helper/test-run-command.c | 21 ++++++++++++++++++++\n t/t0061-run-command.sh | 39 +++++++++++++++++++++++++++++++++++++\n 4 files changed, 100 insertions(+), 3 deletions(-)\n\ndiff --git a/run-command.c b/run-command.c\nindex 756f1839aa..cad88befe0 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -1526,6 +1526,9 @@ static void pp_init(struct parallel_processes *pp,\n \tif (!opts->get_next_task)\n \t\tBUG(\"you need to specify a get_next_task function\");\n \n+\tif (opts->duplicate_output && opts->ungroup)\n+\t\tBUG(\"duplicate_output and ungroup are incompatible with each other\");\n+\n \tCALLOC_ARRAY(pp->children, n);\n \tif (!opts->ungroup)\n \t\tCALLOC_ARRAY(pp->pfd, n);\n@@ -1645,14 +1648,21 @@ static void pp_buffer_stderr(struct parallel_processes *pp,\n \tfor (size_t i = 0; i < opts->processes; i++) {\n \t\tif (pp->children[i].state == GIT_CP_WORKING &&\n \t\t pp->pfd[i].revents & (POLLIN | POLLHUP)) {\n-\t\t\tint n = strbuf_read_once(&pp->children[i].err,\n-\t\t\t\t\t\t pp->children[i].process.err, 0);\n+\t\t\tssize_t n = strbuf_read_once(&pp->children[i].err,\n+\t\t\t\t\t\t pp->children[i].process.err, 0);\n \t\t\tif (n == 0) {\n \t\t\t\tclose(pp->children[i].process.err);\n \t\t\t\tpp->children[i].state = GIT_CP_WAIT_CLEANUP;\n-\t\t\t} else if (n < 0)\n+\t\t\t} else if (n < 0) {\n \t\t\t\tif (errno != EAGAIN)\n \t\t\t\t\tdie_errno(\"read\");\n+\t\t\t} else {\n+\t\t\t\tif (opts->duplicate_output)\n+\t\t\t\t\topts->duplicate_output(&pp->children[i].err,\n+\t\t\t\t\t strlen(pp->children[i].err.buf) - n,\n+\t\t\t\t\t opts->data,\n+\t\t\t\t\t pp->children[i].data);\n+\t\t\t}\n \t\t}\n \t}\n }\ndiff --git a/run-command.h b/run-command.h\nindex 072db56a4d..6dcf999f6c 100644\n--- a/run-command.h\n+++ b/run-command.h\n@@ -408,6 +408,27 @@ typedef int (*start_failure_fn)(struct strbuf *out,\n \t\t\t\tvoid *pp_cb,\n \t\t\t\tvoid *pp_task_cb);\n \n+/**\n+ * This callback is called whenever output from a child process is buffered\n+ * \n+ * See run_processes_parallel() below for a discussion of the \"struct\n+ * strbuf *out\" parameter.\n+ * \n+ * The offset refers to the number of bytes originally in \"out\" before\n+ * the output from the child process was buffered. Therefore, the buffer\n+ * range, \"out + buf\" to the end of \"out\", would contain the buffer of\n+ * the child process output.\n+ *\n+ * pp_cb is the callback cookie as passed into run_processes_parallel,\n+ * pp_task_cb is the callback cookie as passed into get_next_task_fn.\n+ *\n+ * This function is incompatible with \"ungroup\"\n+ */\n+typedef void (*duplicate_output_fn)(struct strbuf *out,\n+\t\t\t\t size_t offset,\n+\t\t\t\t void *pp_cb,\n+\t\t\t\t void *pp_task_cb);\n+\n /**\n * This callback is called on every child process that finished processing.\n *\n@@ -461,6 +482,12 @@ struct run_process_parallel_opts\n \t */\n \tstart_failure_fn start_failure;\n \n+\t/**\n+\t * duplicate_output: See duplicate_output_fn() above. This should be\n+\t * NULL unless process specific output is needed\n+\t */\n+\tduplicate_output_fn duplicate_output;\n+\n \t/**\n \t * task_finished: See task_finished_fn() above. This can be\n \t * NULL to omit any special handling.\ndiff --git a/t/helper/test-run-command.c b/t/helper/test-run-command.c\nindex 3ecb830f4a..ffd3cd0045 100644\n--- a/t/helper/test-run-command.c\n+++ b/t/helper/test-run-command.c\n@@ -52,6 +52,21 @@ static int no_job(struct child_process *cp,\n \treturn 0;\n }\n \n+static void duplicate_output(struct strbuf *out,\n+\t\t\tsize_t offset,\n+\t\t\tvoid *pp_cb UNUSED,\n+\t\t\tvoid *pp_task_cb UNUSED)\n+{\n+\tstruct string_list list = STRING_LIST_INIT_DUP;\n+\n+\tstring_list_split(&list, out->buf + offset, '\\n', -1);\n+\tfor (size_t i = 0; i < list.nr; i++) {\n+\t\tif (strlen(list.items[i].string) > 0)\n+\t\t\tfprintf(stderr, \"duplicate_output: %s\\n\", list.items[i].string);\n+\t}\n+\tstring_list_clear(&list, 0);\n+}\n+\n static int task_finished(int result,\n \t\t\t struct strbuf *err,\n \t\t\t void *pp_cb,\n@@ -439,6 +454,12 @@ int cmd__run_command(int argc, const char **argv)\n \t\topts.ungroup = 1;\n \t}\n \n+\tif (!strcmp(argv[1], \"--duplicate-output\")) {\n+\t\targv += 1;\n+\t\targc -= 1;\n+\t\topts.duplicate_output = duplicate_output;\n+\t}\n+\n \tjobs = atoi(argv[2]);\n \tstrvec_clear(&proc.args);\n \tstrvec_pushv(&proc.args, (const char **)argv + 3);\ndiff --git a/t/t0061-run-command.sh b/t/t0061-run-command.sh\nindex e2411f6a9b..879e536638 100755\n--- a/t/t0061-run-command.sh\n+++ b/t/t0061-run-command.sh\n@@ -135,6 +135,15 @@ test_expect_success 'run_command runs in parallel with more jobs available than\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'run_command runs in parallel with more jobs available than tasks --duplicate-output' '\n+\ttest-tool run-command --duplicate-output run-command-parallel 5 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n+\ttest_must_be_empty out &&\n+\ttest 4 = $(grep -c \"duplicate_output: Hello\" err) &&\n+\ttest 4 = $(grep -c \"duplicate_output: World\" err) &&\n+\tsed \"/duplicate_output/d\" err > err1 &&\n+\ttest_cmp expect err1\n+'\n+\n test_expect_success 'run_command runs ungrouped in parallel with more jobs available than tasks' '\n \ttest-tool run-command --ungroup run-command-parallel 5 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n \ttest_line_count = 8 out &&\n@@ -147,6 +156,15 @@ test_expect_success 'run_command runs in parallel with as many jobs as tasks' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'run_command runs in parallel with as many jobs as tasks --duplicate-output' '\n+\ttest-tool run-command --duplicate-output run-command-parallel 4 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n+\ttest_must_be_empty out &&\n+\ttest 4 = $(grep -c \"duplicate_output: Hello\" err) &&\n+\ttest 4 = $(grep -c \"duplicate_output: World\" err) &&\n+\tsed \"/duplicate_output/d\" err > err1 &&\n+\ttest_cmp expect err1\n+'\n+\n test_expect_success 'run_command runs ungrouped in parallel with as many jobs as tasks' '\n \ttest-tool run-command --ungroup run-command-parallel 4 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n \ttest_line_count = 8 out &&\n@@ -159,6 +177,15 @@ test_expect_success 'run_command runs in parallel with more tasks than jobs avai\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'run_command runs in parallel with more tasks than jobs available --duplicate-output' '\n+\ttest-tool run-command --duplicate-output run-command-parallel 3 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n+\ttest_must_be_empty out &&\n+\ttest 4 = $(grep -c \"duplicate_output: Hello\" err) &&\n+\ttest 4 = $(grep -c \"duplicate_output: World\" err) &&\n+\tsed \"/duplicate_output/d\" err > err1 &&\n+\ttest_cmp expect err1\n+'\n+\n test_expect_success 'run_command runs ungrouped in parallel with more tasks than jobs available' '\n \ttest-tool run-command --ungroup run-command-parallel 3 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n \ttest_line_count = 8 out &&\n@@ -180,6 +207,12 @@ test_expect_success 'run_command is asked to abort gracefully' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'run_command is asked to abort gracefully --duplicate-output' '\n+\ttest-tool run-command --duplicate-output run-command-abort 3 false >out 2>err &&\n+\ttest_must_be_empty out &&\n+\ttest_cmp expect err\n+'\n+\n test_expect_success 'run_command is asked to abort gracefully (ungroup)' '\n \ttest-tool run-command --ungroup run-command-abort 3 false >out 2>err &&\n \ttest_must_be_empty out &&\n@@ -196,6 +229,12 @@ test_expect_success 'run_command outputs ' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'run_command outputs --duplicate-output' '\n+\ttest-tool run-command --duplicate-output run-command-no-jobs 3 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n+\ttest_must_be_empty out &&\n+\ttest_cmp expect err\n+'\n+\n test_expect_success 'run_command outputs (ungroup) ' '\n \ttest-tool run-command --ungroup run-command-no-jobs 3 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n \ttest_must_be_empty out &&\n-- \n2.39.1.519.gcb327c4b5f-goog\n\n"},{"id":"471705","messageId":"20230207181706.363453-1-calvinwan@google.com","threadId":"59034","inReplyTo":"20230117193041.708692-1-calvinwan@google.com","subject":"[PATCH v7 0/7] submodule: parallelize diff","fromName":"Calvin Wan","fromEmail":"calvinwan@google.com","sentAt":"2023-02-07T18:16:59Z","receivedAt":"2023-02-07T18:18:05Z","isPatch":true,"sender":{"key":"calvinwan@google.com","avatar":"https://avatars.githubusercontent.com/u/92547554?v=4"},"body":"Original cover letter for context:\nhttps://lore.kernel.org/git/20221011232604.839941-1-calvinwan@google.com/\n\nChanges since v6\n\nAdded patches 4 and 5 to refactor out more functionality so that it is\nclear what changes my final patch makes. Since the large majority of\nthe functionality between the serial and parallel implementation is now\nshared, I no longer remove the serial implementation.\n\nAdded additional tests to verify setting parallelism doesn't alter\noutput\n\nCalvin Wan (7):\n run-command: add duplicate_output_fn to run_processes_parallel_opts\n submodule: strbuf variable rename\n submodule: move status parsing into function\n submodule: refactor is_submodule_modified()\n diff-lib: refactor out diff_change logic\n diff-lib: refactor match_stat_with_submodule\n diff-lib: parallelize run_diff_files for submodules\n\n Documentation/config/submodule.txt | 12 ++\n diff-lib.c | 133 +++++++++++---\n run-command.c | 16 +-\n run-command.h | 27 +++\n submodule.c | 274 ++++++++++++++++++++++++-----\n submodule.h | 9 +\n t/helper/test-run-command.c | 21 +++\n t/t0061-run-command.sh | 39 ++++\n t/t4027-diff-submodule.sh | 31 ++++\n t/t7506-status-submodule.sh | 25 +++\n 10 files changed, 508 insertions(+), 79 deletions(-)\n\n-- \n2.39.1.519.gcb327c4b5f-goog\n\n"},{"id":"471707","messageId":"20230207181706.363453-3-calvinwan@google.com","threadId":"59034","inReplyTo":"20230117193041.708692-1-calvinwan@google.com","subject":"[PATCH v7 2/7] submodule: strbuf variable rename","fromName":"Calvin Wan","fromEmail":"calvinwan@google.com","sentAt":"2023-02-07T18:17:01Z","receivedAt":"2023-02-07T18:18:06Z","isPatch":true,"sender":{"key":"calvinwan@google.com","avatar":"https://avatars.githubusercontent.com/u/92547554?v=4"},"body":"A prepatory change for a future patch that moves the status parsing\nlogic to a separate function.\n\nSigned-off-by: Calvin Wan \n---\n submodule.c | 23 +++++++++++++----------\n 1 file changed, 13 insertions(+), 10 deletions(-)\n\ndiff --git a/submodule.c b/submodule.c\nindex fae24ef34a..faf37c1101 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -1906,25 +1906,28 @@ unsigned is_submodule_modified(const char *path, int ignore_untracked)\n \n \tfp = xfdopen(cp.out, \"r\");\n \twhile (strbuf_getwholeline(&buf, fp, '\\n') != EOF) {\n+\t\tchar *str = buf.buf;\n+\t\tconst size_t len = buf.len;\n+\n \t\t/* regular untracked files */\n-\t\tif (buf.buf[0] == '?')\n+\t\tif (str[0] == '?')\n \t\t\tdirty_submodule |= DIRTY_SUBMODULE_UNTRACKED;\n \n-\t\tif (buf.buf[0] == 'u' ||\n-\t\t buf.buf[0] == '1' ||\n-\t\t buf.buf[0] == '2') {\n+\t\tif (str[0] == 'u' ||\n+\t\t str[0] == '1' ||\n+\t\t str[0] == '2') {\n \t\t\t/* T = line type, XY = status, SSSS = submodule state */\n-\t\t\tif (buf.len < strlen(\"T XY SSSS\"))\n+\t\t\tif (len < strlen(\"T XY SSSS\"))\n \t\t\t\tBUG(\"invalid status --porcelain=2 line %s\",\n-\t\t\t\t buf.buf);\n+\t\t\t\t str);\n \n-\t\t\tif (buf.buf[5] == 'S' && buf.buf[8] == 'U')\n+\t\t\tif (str[5] == 'S' && str[8] == 'U')\n \t\t\t\t/* nested untracked file */\n \t\t\t\tdirty_submodule |= DIRTY_SUBMODULE_UNTRACKED;\n \n-\t\t\tif (buf.buf[0] == 'u' ||\n-\t\t\t buf.buf[0] == '2' ||\n-\t\t\t memcmp(buf.buf + 5, \"S..U\", 4))\n+\t\t\tif (str[0] == 'u' ||\n+\t\t\t str[0] == '2' ||\n+\t\t\t memcmp(str + 5, \"S..U\", 4))\n \t\t\t\t/* other change */\n \t\t\t\tdirty_submodule |= DIRTY_SUBMODULE_MODIFIED;\n \t\t}\n-- \n2.39.1.519.gcb327c4b5f-goog\n\n"},{"id":"471708","messageId":"20230207181706.363453-4-calvinwan@google.com","threadId":"59034","inReplyTo":"20230117193041.708692-1-calvinwan@google.com","subject":"[PATCH v7 3/7] submodule: move status parsing into function","fromName":"Calvin Wan","fromEmail":"calvinwan@google.com","sentAt":"2023-02-07T18:17:02Z","receivedAt":"2023-02-07T18:18:10Z","isPatch":true,"sender":{"key":"calvinwan@google.com","avatar":"https://avatars.githubusercontent.com/u/92547554?v=4"},"body":"A future patch requires the ability to parse the output of git\nstatus --porcelain=2. Move parsing code from is_submodule_modified to\nparse_status_porcelain.\n\nSigned-off-by: Calvin Wan \n---\n submodule.c | 74 ++++++++++++++++++++++++++++++-----------------------\n 1 file changed, 42 insertions(+), 32 deletions(-)\n\ndiff --git a/submodule.c b/submodule.c\nindex faf37c1101..768d4b4cd7 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -1870,6 +1870,45 @@ int fetch_submodules(struct repository *r,\n \treturn spf.result;\n }\n \n+static int parse_status_porcelain(char *str, size_t len,\n+\t\t\t\t unsigned *dirty_submodule,\n+\t\t\t\t int ignore_untracked)\n+{\n+\t/* regular untracked files */\n+\tif (str[0] == '?')\n+\t\t*dirty_submodule |= DIRTY_SUBMODULE_UNTRACKED;\n+\n+\tif (str[0] == 'u' ||\n+\t str[0] == '1' ||\n+\t str[0] == '2') {\n+\t\t/* T = line type, XY = status, SSSS = submodule state */\n+\t\tif (len < strlen(\"T XY SSSS\"))\n+\t\t\tBUG(\"invalid status --porcelain=2 line %s\",\n+\t\t\t str);\n+\n+\t\tif (str[5] == 'S' && str[8] == 'U')\n+\t\t\t/* nested untracked file */\n+\t\t\t*dirty_submodule |= DIRTY_SUBMODULE_UNTRACKED;\n+\n+\t\tif (str[0] == 'u' ||\n+\t\t str[0] == '2' ||\n+\t\t memcmp(str + 5, \"S..U\", 4))\n+\t\t\t/* other change */\n+\t\t\t*dirty_submodule |= DIRTY_SUBMODULE_MODIFIED;\n+\t}\n+\n+\tif ((*dirty_submodule & DIRTY_SUBMODULE_MODIFIED) &&\n+\t ((*dirty_submodule & DIRTY_SUBMODULE_UNTRACKED) ||\n+\t ignore_untracked)) {\n+\t\t/*\n+\t\t* We're not interested in any further information from\n+\t\t* the child any more, neither output nor its exit code.\n+\t\t*/\n+\t\treturn 1;\n+\t}\n+\treturn 0;\n+}\n+\n unsigned is_submodule_modified(const char *path, int ignore_untracked)\n {\n \tstruct child_process cp = CHILD_PROCESS_INIT;\n@@ -1909,39 +1948,10 @@ unsigned is_submodule_modified(const char *path, int ignore_untracked)\n \t\tchar *str = buf.buf;\n \t\tconst size_t len = buf.len;\n \n-\t\t/* regular untracked files */\n-\t\tif (str[0] == '?')\n-\t\t\tdirty_submodule |= DIRTY_SUBMODULE_UNTRACKED;\n-\n-\t\tif (str[0] == 'u' ||\n-\t\t str[0] == '1' ||\n-\t\t str[0] == '2') {\n-\t\t\t/* T = line type, XY = status, SSSS = submodule state */\n-\t\t\tif (len < strlen(\"T XY SSSS\"))\n-\t\t\t\tBUG(\"invalid status --porcelain=2 line %s\",\n-\t\t\t\t str);\n-\n-\t\t\tif (str[5] == 'S' && str[8] == 'U')\n-\t\t\t\t/* nested untracked file */\n-\t\t\t\tdirty_submodule |= DIRTY_SUBMODULE_UNTRACKED;\n-\n-\t\t\tif (str[0] == 'u' ||\n-\t\t\t str[0] == '2' ||\n-\t\t\t memcmp(str + 5, \"S..U\", 4))\n-\t\t\t\t/* other change */\n-\t\t\t\tdirty_submodule |= DIRTY_SUBMODULE_MODIFIED;\n-\t\t}\n-\n-\t\tif ((dirty_submodule & DIRTY_SUBMODULE_MODIFIED) &&\n-\t\t ((dirty_submodule & DIRTY_SUBMODULE_UNTRACKED) ||\n-\t\t ignore_untracked)) {\n-\t\t\t/*\n-\t\t\t * We're not interested in any further information from\n-\t\t\t * the child any more, neither output nor its exit code.\n-\t\t\t */\n-\t\t\tignore_cp_exit_code = 1;\n+\t\tignore_cp_exit_code = parse_status_porcelain(str, len, &dirty_submodule,\n+\t\t\t\t\t\t\t ignore_untracked);\n+\t\tif (ignore_cp_exit_code)\n \t\t\tbreak;\n-\t\t}\n \t}\n \tfclose(fp);\n \n-- \n2.39.1.519.gcb327c4b5f-goog\n\n"},{"id":"471709","messageId":"20230207181706.363453-5-calvinwan@google.com","threadId":"59034","inReplyTo":"20230117193041.708692-1-calvinwan@google.com","subject":"[PATCH v7 4/7] submodule: refactor is_submodule_modified()","fromName":"Calvin Wan","fromEmail":"calvinwan@google.com","sentAt":"2023-02-07T18:17:03Z","receivedAt":"2023-02-07T18:18:33Z","isPatch":true,"sender":{"key":"calvinwan@google.com","avatar":"https://avatars.githubusercontent.com/u/92547554?v=4"},"body":"Refactor out submodule status logic and error messages that will be\nused in a future patch.\n\nSigned-off-by: Calvin Wan \n---\n submodule.c | 65 ++++++++++++++++++++++++++++++++++-------------------\n 1 file changed, 42 insertions(+), 23 deletions(-)\n\ndiff --git a/submodule.c b/submodule.c\nindex 768d4b4cd7..d88aa2c573 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -28,6 +28,10 @@ static int config_update_recurse_submodules = RECURSE_SUBMODULES_OFF;\n static int initialized_fetch_ref_tips;\n static struct oid_array ref_tips_before_fetch;\n static struct oid_array ref_tips_after_fetch;\n+static const char *status_porcelain_start_error =\n+\tN_(\"could not run 'git status --porcelain=2' in submodule %s\");\n+static const char *status_porcelain_fail_error =\n+\tN_(\"'git status --porcelain=2' failed in submodule %s\");\n \n /*\n * Check if the .gitmodules file is unmerged. Parsing of the .gitmodules file\n@@ -1870,6 +1874,40 @@ int fetch_submodules(struct repository *r,\n \treturn spf.result;\n }\n \n+static int verify_submodule_git_directory(const char *path)\n+{\n+\tconst char *git_dir;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\n+\tstrbuf_addf(&buf, \"%s/.git\", path);\n+\tgit_dir = read_gitfile(buf.buf);\n+\tif (!git_dir)\n+\t\tgit_dir = buf.buf;\n+\tif (!is_git_directory(git_dir)) {\n+\t\tif (is_directory(git_dir))\n+\t\t\tdie(_(\"'%s' not recognized as a git repository\"), git_dir);\n+\t\tstrbuf_release(&buf);\n+\t\t/* The submodule is not checked out, so it is not modified */\n+\t\treturn 0;\n+\t}\n+\tstrbuf_release(&buf);\n+\treturn 1;\n+}\n+\n+static void prepare_status_porcelain(struct child_process *cp,\n+\t\t\t const char *path, int ignore_untracked)\n+{\n+\tstrvec_pushl(&cp->args, \"status\", \"--porcelain=2\", NULL);\n+\tif (ignore_untracked)\n+\t\tstrvec_push(&cp->args, \"-uno\");\n+\n+\tprepare_submodule_repo_env(&cp->env);\n+\tcp->git_cmd = 1;\n+\tcp->no_stdin = 1;\n+\tcp->out = -1;\n+\tcp->dir = path;\n+}\n+\n static int parse_status_porcelain(char *str, size_t len,\n \t\t\t\t unsigned *dirty_submodule,\n \t\t\t\t int ignore_untracked)\n@@ -1915,33 +1953,14 @@ unsigned is_submodule_modified(const char *path, int ignore_untracked)\n \tstruct strbuf buf = STRBUF_INIT;\n \tFILE *fp;\n \tunsigned dirty_submodule = 0;\n-\tconst char *git_dir;\n \tint ignore_cp_exit_code = 0;\n \n-\tstrbuf_addf(&buf, \"%s/.git\", path);\n-\tgit_dir = read_gitfile(buf.buf);\n-\tif (!git_dir)\n-\t\tgit_dir = buf.buf;\n-\tif (!is_git_directory(git_dir)) {\n-\t\tif (is_directory(git_dir))\n-\t\t\tdie(_(\"'%s' not recognized as a git repository\"), git_dir);\n-\t\tstrbuf_release(&buf);\n-\t\t/* The submodule is not checked out, so it is not modified */\n+\tif (!verify_submodule_git_directory(path))\n \t\treturn 0;\n-\t}\n-\tstrbuf_reset(&buf);\n-\n-\tstrvec_pushl(&cp.args, \"status\", \"--porcelain=2\", NULL);\n-\tif (ignore_untracked)\n-\t\tstrvec_push(&cp.args, \"-uno\");\n \n-\tprepare_submodule_repo_env(&cp.env);\n-\tcp.git_cmd = 1;\n-\tcp.no_stdin = 1;\n-\tcp.out = -1;\n-\tcp.dir = path;\n+\tprepare_status_porcelain(&cp, path, ignore_untracked);\n \tif (start_command(&cp))\n-\t\tdie(_(\"Could not run 'git status --porcelain=2' in submodule %s\"), path);\n+\t\tdie(_(status_porcelain_start_error), path);\n \n \tfp = xfdopen(cp.out, \"r\");\n \twhile (strbuf_getwholeline(&buf, fp, '\\n') != EOF) {\n@@ -1956,7 +1975,7 @@ unsigned is_submodule_modified(const char *path, int ignore_untracked)\n \tfclose(fp);\n \n \tif (finish_command(&cp) && !ignore_cp_exit_code)\n-\t\tdie(_(\"'git status --porcelain=2' failed in submodule %s\"), path);\n+\t\tdie(_(status_porcelain_fail_error), path);\n \n \tstrbuf_release(&buf);\n \treturn dirty_submodule;\n-- \n2.39.1.519.gcb327c4b5f-goog\n\n"},{"id":"471710","messageId":"20230207181706.363453-6-calvinwan@google.com","threadId":"59034","inReplyTo":"20230117193041.708692-1-calvinwan@google.com","subject":"[PATCH v7 5/7] diff-lib: refactor out diff_change logic","fromName":"Calvin Wan","fromEmail":"calvinwan@google.com","sentAt":"2023-02-07T18:17:04Z","receivedAt":"2023-02-07T18:18:34Z","isPatch":true,"sender":{"key":"calvinwan@google.com","avatar":"https://avatars.githubusercontent.com/u/92547554?v=4"},"body":"Refactor out logic that sets up the diff_change call into a helper\nfunction for a future patch.\n\nSigned-off-by: Calvin Wan \n---\n diff-lib.c | 46 +++++++++++++++++++++++++++++-----------------\n 1 file changed, 29 insertions(+), 17 deletions(-)\n\ndiff --git a/diff-lib.c b/diff-lib.c\nindex dec040c366..7101cfda3f 100644\n--- a/diff-lib.c\n+++ b/diff-lib.c\n@@ -88,6 +88,31 @@ static int match_stat_with_submodule(struct diff_options *diffopt,\n \treturn changed;\n }\n \n+static int diff_change_helper(struct diff_options *options,\n+\t unsigned newmode, unsigned dirty_submodule,\n+\t int changed, struct index_state *istate,\n+\t struct cache_entry *ce)\n+{\n+\tunsigned int oldmode;\n+\tconst struct object_id *old_oid, *new_oid;\n+\n+\tif (!changed && !dirty_submodule) {\n+\t\tce_mark_uptodate(ce);\n+\t\tmark_fsmonitor_valid(istate, ce);\n+\t\tif (!options->flags.find_copies_harder)\n+\t\t\treturn 1;\n+\t}\n+\toldmode = ce->ce_mode;\n+\told_oid = &ce->oid;\n+\tnew_oid = changed ? null_oid() : &ce->oid;\n+\tdiff_change(options, oldmode, newmode,\n+\t\t\told_oid, new_oid,\n+\t\t\t!is_null_oid(old_oid),\n+\t\t\t!is_null_oid(new_oid),\n+\t\t\tce->name, 0, dirty_submodule);\n+\treturn 0;\n+}\n+\n int run_diff_files(struct rev_info *revs, unsigned int option)\n {\n \tint entries, i;\n@@ -105,11 +130,10 @@ int run_diff_files(struct rev_info *revs, unsigned int option)\n \t\tdiff_unmerged_stage = 2;\n \tentries = istate->cache_nr;\n \tfor (i = 0; i < entries; i++) {\n-\t\tunsigned int oldmode, newmode;\n+\t\tunsigned int newmode;\n \t\tstruct cache_entry *ce = istate->cache[i];\n \t\tint changed;\n \t\tunsigned dirty_submodule = 0;\n-\t\tconst struct object_id *old_oid, *new_oid;\n \n \t\tif (diff_can_quit_early(&revs->diffopt))\n \t\t\tbreak;\n@@ -245,21 +269,9 @@ int run_diff_files(struct rev_info *revs, unsigned int option)\n \t\t\tnewmode = ce_mode_from_stat(ce, st.st_mode);\n \t\t}\n \n-\t\tif (!changed && !dirty_submodule) {\n-\t\t\tce_mark_uptodate(ce);\n-\t\t\tmark_fsmonitor_valid(istate, ce);\n-\t\t\tif (!revs->diffopt.flags.find_copies_harder)\n-\t\t\t\tcontinue;\n-\t\t}\n-\t\toldmode = ce->ce_mode;\n-\t\told_oid = &ce->oid;\n-\t\tnew_oid = changed ? null_oid() : &ce->oid;\n-\t\tdiff_change(&revs->diffopt, oldmode, newmode,\n-\t\t\t old_oid, new_oid,\n-\t\t\t !is_null_oid(old_oid),\n-\t\t\t !is_null_oid(new_oid),\n-\t\t\t ce->name, 0, dirty_submodule);\n-\n+\t\tif (diff_change_helper(&revs->diffopt, newmode, dirty_submodule,\n+\t\t\t\t changed, istate, ce))\n+\t\t\tcontinue;\n \t}\n \tdiffcore_std(&revs->diffopt);\n \tdiff_flush(&revs->diffopt);\n-- \n2.39.1.519.gcb327c4b5f-goog\n\n"},{"id":"471711","messageId":"20230207181706.363453-7-calvinwan@google.com","threadId":"59034","inReplyTo":"20230117193041.708692-1-calvinwan@google.com","subject":"[PATCH v7 6/7] diff-lib: refactor match_stat_with_submodule","fromName":"Calvin Wan","fromEmail":"calvinwan@google.com","sentAt":"2023-02-07T18:17:05Z","receivedAt":"2023-02-07T18:18:41Z","isPatch":true,"sender":{"key":"calvinwan@google.com","avatar":"https://avatars.githubusercontent.com/u/92547554?v=4"},"body":"Flatten out the if statements in match_stat_with_submodule so the\nlogic is more readable and easier for future patches to add to.\norig_flags didn't need to be set if the cache entry wasn't a\nGITLINK so defer setting it.\n\nSigned-off-by: Calvin Wan \n---\n diff-lib.c | 28 +++++++++++++++++-----------\n 1 file changed, 17 insertions(+), 11 deletions(-)\n\ndiff --git a/diff-lib.c b/diff-lib.c\nindex 7101cfda3f..e18c886a80 100644\n--- a/diff-lib.c\n+++ b/diff-lib.c\n@@ -73,18 +73,24 @@ static int match_stat_with_submodule(struct diff_options *diffopt,\n \t\t\t\t unsigned *dirty_submodule)\n {\n \tint changed = ie_match_stat(diffopt->repo->index, ce, st, ce_option);\n-\tif (S_ISGITLINK(ce->ce_mode)) {\n-\t\tstruct diff_flags orig_flags = diffopt->flags;\n-\t\tif (!diffopt->flags.override_submodule_config)\n-\t\t\tset_diffopt_flags_from_submodule_config(diffopt, ce->name);\n-\t\tif (diffopt->flags.ignore_submodules)\n-\t\t\tchanged = 0;\n-\t\telse if (!diffopt->flags.ignore_dirty_submodules &&\n-\t\t\t (!changed || diffopt->flags.dirty_submodules))\n-\t\t\t*dirty_submodule = is_submodule_modified(ce->name,\n-\t\t\t\t\t\t\t\t diffopt->flags.ignore_untracked_in_submodules);\n-\t\tdiffopt->flags = orig_flags;\n+\tstruct diff_flags orig_flags;\n+\n+\tif (!S_ISGITLINK(ce->ce_mode))\n+\t\treturn changed;\n+\n+\torig_flags = diffopt->flags;\n+\tif (!diffopt->flags.override_submodule_config)\n+\t\tset_diffopt_flags_from_submodule_config(diffopt, ce->name);\n+\tif (diffopt->flags.ignore_submodules) {\n+\t\tchanged = 0;\n+\t\tgoto cleanup;\n \t}\n+\tif (!diffopt->flags.ignore_dirty_submodules &&\n+\t (!changed || diffopt->flags.dirty_submodules))\n+\t\t*dirty_submodule = is_submodule_modified(ce->name,\n+\t\t\t\t\t diffopt->flags.ignore_untracked_in_submodules);\n+cleanup:\n+\tdiffopt->flags = orig_flags;\n \treturn changed;\n }\n \n-- \n2.39.1.519.gcb327c4b5f-goog\n\n"},{"id":"471712","messageId":"20230207181706.363453-8-calvinwan@google.com","threadId":"59034","inReplyTo":"20230117193041.708692-1-calvinwan@google.com","subject":"[PATCH v7 7/7] diff-lib: parallelize run_diff_files for submodules","fromName":"Calvin Wan","fromEmail":"calvinwan@google.com","sentAt":"2023-02-07T18:17:06Z","receivedAt":"2023-02-07T18:18:42Z","isPatch":true,"sender":{"key":"calvinwan@google.com","avatar":"https://avatars.githubusercontent.com/u/92547554?v=4"},"body":"During the iteration of the index entries in run_diff_files, whenever\na submodule is found and needs its status checked, a subprocess is\nspawned for it. Instead of spawning the subprocess immediately and\nwaiting for its completion to continue, hold onto all submodules and\nrelevant information in a list. Then use that list to create tasks for\nrun_processes_parallel. Subprocess output is duplicated and passed to\nstatus_pipe_output which stores it to be parsed on completion of the\nsubprocess.\n\nAdd config option submodule.diffJobs to set the maximum number\nof parallel jobs. The option defaults to 1 if unset. If set to 0, the\nnumber of jobs is set to online_cpus().\n\nSince run_diff_files is called from many different commands, I chose\nto grab the config option in the function rather than adding variables\nto every git command and then figuring out how to pass them all in.\n\nSigned-off-by: Calvin Wan \n---\n Documentation/config/submodule.txt | 12 +++\n diff-lib.c | 71 ++++++++++++--\n submodule.c | 148 +++++++++++++++++++++++++++++\n submodule.h | 9 ++\n t/t4027-diff-submodule.sh | 31 ++++++\n t/t7506-status-submodule.sh | 25 +++++\n 6 files changed, 289 insertions(+), 7 deletions(-)\n\ndiff --git a/Documentation/config/submodule.txt b/Documentation/config/submodule.txt\nindex 6490527b45..3209eb8117 100644\n--- a/Documentation/config/submodule.txt\n+++ b/Documentation/config/submodule.txt\n@@ -93,6 +93,18 @@ submodule.fetchJobs::\n \tin parallel. A value of 0 will give some reasonable default.\n \tIf unset, it defaults to 1.\n \n+submodule.diffJobs::\n+\tSpecifies how many submodules are diffed at the same time. A\n+\tpositive integer allows up to that number of submodules diffed\n+\tin parallel. A value of 0 will give some reasonable default.\n+\tIf unset, it defaults to 1. The diff operation is used by many\n+\tother git commands such as add, merge, diff, status, stash and\n+\tmore. Note that the expensive part of the diff operation is\n+\treading the index from cache or memory. Therefore multiple jobs\n+\tmay be detrimental to performance if your hardware does not\n+\tsupport parallel reads or if the number of jobs greatly exceeds\n+\tthe amount of supported reads.\n+\n submodule.alternateLocation::\n \tSpecifies how the submodules obtain alternates when submodules are\n \tcloned. Possible values are `no`, `superproject`.\ndiff --git a/diff-lib.c b/diff-lib.c\nindex e18c886a80..f91cd73ae7 100644\n--- a/diff-lib.c\n+++ b/diff-lib.c\n@@ -14,6 +14,7 @@\n #include \"dir.h\"\n #include \"fsmonitor.h\"\n #include \"commit-reach.h\"\n+#include \"config.h\"\n \n /*\n * diff-files\n@@ -65,18 +66,23 @@ static int check_removed(const struct index_state *istate, const struct cache_en\n * Return 1 when changes are detected, 0 otherwise. If the DIRTY_SUBMODULES\n * option is set, the caller does not only want to know if a submodule is\n * modified at all but wants to know all the conditions that are met (new\n- * commits, untracked content and/or modified content).\n+ * commits, untracked content and/or modified content). If\n+ * defer_submodule_status bit is set, dirty_submodule will be left to the\n+ * caller to set. defer_submodule_status can also be set to 0 in this\n+ * function if there is no need to check if the submodule is modified.\n */\n static int match_stat_with_submodule(struct diff_options *diffopt,\n \t\t\t\t const struct cache_entry *ce,\n \t\t\t\t struct stat *st, unsigned ce_option,\n-\t\t\t\t unsigned *dirty_submodule)\n+\t\t\t\t unsigned *dirty_submodule, int *defer_submodule_status,\n+\t\t\t\t unsigned *ignore_untracked)\n {\n \tint changed = ie_match_stat(diffopt->repo->index, ce, st, ce_option);\n \tstruct diff_flags orig_flags;\n+\tint defer = 0;\n \n \tif (!S_ISGITLINK(ce->ce_mode))\n-\t\treturn changed;\n+\t\tgoto ret;\n \n \torig_flags = diffopt->flags;\n \tif (!diffopt->flags.override_submodule_config)\n@@ -86,11 +92,20 @@ static int match_stat_with_submodule(struct diff_options *diffopt,\n \t\tgoto cleanup;\n \t}\n \tif (!diffopt->flags.ignore_dirty_submodules &&\n-\t (!changed || diffopt->flags.dirty_submodules))\n-\t\t*dirty_submodule = is_submodule_modified(ce->name,\n+\t (!changed || diffopt->flags.dirty_submodules)) {\n+\t\tif (defer_submodule_status && *defer_submodule_status) {\n+\t\t\tdefer = 1;\n+\t\t\t*ignore_untracked = diffopt->flags.ignore_untracked_in_submodules;\n+\t\t} else {\n+\t\t\t*dirty_submodule = is_submodule_modified(ce->name,\n \t\t\t\t\t diffopt->flags.ignore_untracked_in_submodules);\n+\t\t}\n+\t}\n cleanup:\n \tdiffopt->flags = orig_flags;\n+ret:\n+\tif (defer_submodule_status)\n+\t\t*defer_submodule_status = defer;\n \treturn changed;\n }\n \n@@ -127,6 +142,7 @@ int run_diff_files(struct rev_info *revs, unsigned int option)\n \t\t\t ? CE_MATCH_RACY_IS_DIRTY : 0);\n \tuint64_t start = getnanotime();\n \tstruct index_state *istate = revs->diffopt.repo->index;\n+\tstruct string_list submodules = STRING_LIST_INIT_NODUP;\n \n \tdiff_set_mnemonic_prefix(&revs->diffopt, \"i/\", \"w/\");\n \n@@ -250,6 +266,8 @@ int run_diff_files(struct rev_info *revs, unsigned int option)\n \t\t\tnewmode = ce->ce_mode;\n \t\t} else {\n \t\t\tstruct stat st;\n+\t\t\tunsigned ignore_untracked = 0;\n+\t\t\tint defer_submodule_status = 1;\n \n \t\t\tchanged = check_removed(istate, ce, &st);\n \t\t\tif (changed) {\n@@ -271,14 +289,53 @@ int run_diff_files(struct rev_info *revs, unsigned int option)\n \t\t\t}\n \n \t\t\tchanged = match_stat_with_submodule(&revs->diffopt, ce, &st,\n-\t\t\t\t\t\t\t ce_option, &dirty_submodule);\n+\t\t\t\t\t\t\t ce_option, &dirty_submodule,\n+\t\t\t\t\t\t\t &defer_submodule_status,\n+\t\t\t\t\t\t\t &ignore_untracked);\n \t\t\tnewmode = ce_mode_from_stat(ce, st.st_mode);\n+\t\t\tif (defer_submodule_status) {\n+\t\t\t\tstruct submodule_status_util tmp = {\n+\t\t\t\t\t.changed = changed,\n+\t\t\t\t\t.dirty_submodule = 0,\n+\t\t\t\t\t.ignore_untracked = ignore_untracked,\n+\t\t\t\t\t.newmode = newmode,\n+\t\t\t\t\t.ce = ce,\n+\t\t\t\t\t.path = ce->name,\n+\t\t\t\t};\n+\t\t\t\tstruct string_list_item *item;\n+\n+\t\t\t\titem = string_list_append(&submodules, ce->name);\n+\t\t\t\titem->util = xmalloc(sizeof(tmp));\n+\t\t\t\tmemcpy(item->util, &tmp, sizeof(tmp));\n+\t\t\t\tcontinue;\n+\t\t\t}\n \t\t}\n \n \t\tif (diff_change_helper(&revs->diffopt, newmode, dirty_submodule,\n \t\t\t\t changed, istate, ce))\n \t\t\tcontinue;\n \t}\n+\tif (submodules.nr > 0) {\n+\t\tint parallel_jobs;\n+\t\tif (git_config_get_int(\"submodule.diffjobs\", ¶llel_jobs))\n+\t\t\tparallel_jobs = 1;\n+\t\telse if (!parallel_jobs)\n+\t\t\tparallel_jobs = online_cpus();\n+\t\telse if (parallel_jobs < 0)\n+\t\t\tdie(_(\"submodule.diffjobs cannot be negative\"));\n+\n+\t\tif (get_submodules_status(&submodules, parallel_jobs))\n+\t\t\tdie(_(\"submodule status failed\"));\n+\t\tfor (size_t i = 0; i < submodules.nr; i++) {\n+\t\t\tstruct submodule_status_util *util = submodules.items[i].util;\n+\n+\t\t\tif (diff_change_helper(&revs->diffopt, util->newmode,\n+\t\t\t\t util->dirty_submodule, util->changed,\n+\t\t\t\t istate, util->ce))\n+\t\t\t\tcontinue;\n+\t\t}\n+\t}\n+\tstring_list_clear(&submodules, 1);\n \tdiffcore_std(&revs->diffopt);\n \tdiff_flush(&revs->diffopt);\n \ttrace_performance_since(start, \"diff-files\");\n@@ -326,7 +383,7 @@ static int get_stat_data(const struct index_state *istate,\n \t\t\treturn -1;\n \t\t}\n \t\tchanged = match_stat_with_submodule(diffopt, ce, &st,\n-\t\t\t\t\t\t 0, dirty_submodule);\n+\t\t\t\t\t\t 0, dirty_submodule, NULL, NULL);\n \t\tif (changed) {\n \t\t\tmode = ce_mode_from_stat(ce, st.st_mode);\n \t\t\toid = null_oid();\ndiff --git a/submodule.c b/submodule.c\nindex d88aa2c573..3e1811691a 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -1373,6 +1373,17 @@ int submodule_touches_in_range(struct repository *r,\n \treturn ret;\n }\n \n+struct submodule_parallel_status {\n+\tsize_t index_count;\n+\tint result;\n+\n+\tstruct string_list *submodule_names;\n+\n+\t/* Pending statuses by OIDs */\n+\tstruct status_task **oid_status_tasks;\n+\tint oid_status_tasks_nr, oid_status_tasks_alloc;\n+};\n+\n struct submodule_parallel_fetch {\n \t/*\n \t * The index of the last index entry processed by\n@@ -1455,6 +1466,12 @@ struct fetch_task {\n \tstruct oid_array *commits; /* Ensure these commits are fetched */\n };\n \n+struct status_task {\n+\tconst char *path;\n+\tstruct strbuf out;\n+\tint ignore_untracked;\n+};\n+\n /**\n * When a submodule is not defined in .gitmodules, we cannot access it\n * via the regular submodule-config. Create a fake submodule, which we can\n@@ -1947,6 +1964,25 @@ static int parse_status_porcelain(char *str, size_t len,\n \treturn 0;\n }\n \n+static void parse_status_porcelain_strbuf(struct strbuf *buf,\n+\t\t\t\t unsigned *dirty_submodule,\n+\t\t\t\t int ignore_untracked)\n+{\n+\tstruct string_list list = STRING_LIST_INIT_DUP;\n+\tstruct string_list_item *item;\n+\n+\tstring_list_split(&list, buf->buf, '\\n', -1);\n+\n+\tfor_each_string_list_item(item, &list) {\n+\t\tif (parse_status_porcelain(item->string,\n+\t\t\t\t\t strlen(item->string),\n+\t\t\t\t\t dirty_submodule,\n+\t\t\t\t\t ignore_untracked))\n+\t\t\tbreak;\n+\t}\n+\tstring_list_clear(&list, 0);\n+}\n+\n unsigned is_submodule_modified(const char *path, int ignore_untracked)\n {\n \tstruct child_process cp = CHILD_PROCESS_INIT;\n@@ -1981,6 +2017,118 @@ unsigned is_submodule_modified(const char *path, int ignore_untracked)\n \treturn dirty_submodule;\n }\n \n+static struct status_task *\n+get_status_task_from_index(struct submodule_parallel_status *sps,\n+\t\t\t struct strbuf *err)\n+{\n+\tfor (; sps->index_count < sps->submodule_names->nr; sps->index_count++) {\n+\t\tstruct submodule_status_util *util = sps->submodule_names->items[sps->index_count].util;\n+\t\tstruct status_task *task;\n+\n+\t\tif (!verify_submodule_git_directory(util->path))\n+\t\t\tcontinue;\n+\n+\t\ttask = xmalloc(sizeof(*task));\n+\t\ttask->path = util->path;\n+\t\ttask->ignore_untracked = util->ignore_untracked;\n+\t\tstrbuf_init(&task->out, 0);\n+\t\tsps->index_count++;\n+\t\treturn task;\n+\t}\n+\treturn NULL;\n+}\n+\n+static int get_next_submodule_status(struct child_process *cp,\n+\t\t\t\t struct strbuf *err, void *data,\n+\t\t\t\t void **task_cb)\n+{\n+\tstruct submodule_parallel_status *sps = data;\n+\tstruct status_task *task = get_status_task_from_index(sps, err);\n+\n+\tif (!task)\n+\t\treturn 0;\n+\n+\tchild_process_init(cp);\n+\tprepare_submodule_repo_env_in_gitdir(&cp->env);\n+\tprepare_status_porcelain(cp, task->path, task->ignore_untracked);\n+\t*task_cb = task;\n+\treturn 1;\n+}\n+\n+static int status_start_failure(struct strbuf *err,\n+\t\t\t\tvoid *cb, void *task_cb)\n+{\n+\tstruct submodule_parallel_status *sps = cb;\n+\tstruct status_task *task = task_cb;\n+\n+\tsps->result = 1;\n+\tstrbuf_addf(err,\n+\t _(status_porcelain_start_error),\n+\t task->path);\n+\treturn 0;\n+}\n+\n+static void status_duplicate_output(struct strbuf *out,\n+\t\t\t\t size_t offset,\n+\t\t\t\t void *cb, void *task_cb)\n+{\n+\tstruct status_task *task = task_cb;\n+\n+\tstrbuf_add(&task->out, out->buf + offset, out->len - offset);\n+\tstrbuf_setlen(out, offset);\n+}\n+\n+static int status_finish(int retvalue, struct strbuf *err,\n+\t\t\t void *cb, void *task_cb)\n+{\n+\tstruct submodule_parallel_status *sps = cb;\n+\tstruct status_task *task = task_cb;\n+\tstruct string_list_item *it =\n+\t\tstring_list_lookup(sps->submodule_names, task->path);\n+\tstruct submodule_status_util *util = it->util;\n+\n+\tif (retvalue) {\n+\t\tsps->result = 1;\n+\t\tstrbuf_addf(err,\n+\t\t _(status_porcelain_fail_error),\n+\t\t task->path);\n+\t}\n+\n+\tparse_status_porcelain_strbuf(&task->out,\n+\t\t\t &util->dirty_submodule,\n+\t\t\t util->ignore_untracked);\n+\n+\tstrbuf_release(&task->out);\n+\tfree(task);\n+\n+\treturn 0;\n+}\n+\n+int get_submodules_status(struct string_list *submodules,\n+\t\t\t int max_parallel_jobs)\n+{\n+\tstruct submodule_parallel_status sps = {\n+\t\t.submodule_names = submodules,\n+\t};\n+\tconst struct run_process_parallel_opts opts = {\n+\t\t.tr2_category = \"submodule\",\n+\t\t.tr2_label = \"parallel/status\",\n+\n+\t\t.processes = max_parallel_jobs,\n+\n+\t\t.get_next_task = get_next_submodule_status,\n+\t\t.start_failure = status_start_failure,\n+\t\t.duplicate_output = status_duplicate_output,\n+\t\t.task_finished = status_finish,\n+\t\t.data = &sps,\n+\t};\n+\n+\tstring_list_sort(sps.submodule_names);\n+\trun_processes_parallel(&opts);\n+\n+\treturn sps.result;\n+}\n+\n int submodule_uses_gitfile(const char *path)\n {\n \tstruct child_process cp = CHILD_PROCESS_INIT;\ndiff --git a/submodule.h b/submodule.h\nindex b52a4ff1e7..08d278a414 100644\n--- a/submodule.h\n+++ b/submodule.h\n@@ -41,6 +41,13 @@ struct submodule_update_strategy {\n \t.type = SM_UPDATE_UNSPECIFIED, \\\n }\n \n+struct submodule_status_util {\n+\tint changed, ignore_untracked;\n+\tunsigned dirty_submodule, newmode;\n+\tstruct cache_entry *ce;\n+\tconst char *path;\n+};\n+\n int is_gitmodules_unmerged(struct index_state *istate);\n int is_writing_gitmodules_ok(void);\n int is_staging_gitmodules_ok(struct index_state *istate);\n@@ -94,6 +101,8 @@ int fetch_submodules(struct repository *r,\n \t\t int command_line_option,\n \t\t int default_option,\n \t\t int quiet, int max_parallel_jobs);\n+int get_submodules_status(struct string_list *submodules,\n+\t\t\t int max_parallel_jobs);\n unsigned is_submodule_modified(const char *path, int ignore_untracked);\n int submodule_uses_gitfile(const char *path);\n \ndiff --git a/t/t4027-diff-submodule.sh b/t/t4027-diff-submodule.sh\nindex 40164ae07d..1c747cc325 100755\n--- a/t/t4027-diff-submodule.sh\n+++ b/t/t4027-diff-submodule.sh\n@@ -34,6 +34,25 @@ test_expect_success setup '\n \tsubtip=$3 subprev=$2\n '\n \n+test_expect_success 'diff in superproject with submodules respects parallel settings' '\n+\ttest_when_finished \"rm -f trace.out\" &&\n+\t(\n+\t\tGIT_TRACE=$(pwd)/trace.out git diff &&\n+\t\tgrep \"1 tasks\" trace.out &&\n+\t\t>trace.out &&\n+\n+\t\tgit config submodule.diffJobs 8 &&\n+\t\tGIT_TRACE=$(pwd)/trace.out git diff &&\n+\t\tgrep \"8 tasks\" trace.out &&\n+\t\t>trace.out &&\n+\n+\t\tGIT_TRACE=$(pwd)/trace.out git -c submodule.diffJobs=0 diff &&\n+\t\tgrep \"preparing to run up to [0-9]* tasks\" trace.out &&\n+\t\t! grep \"up to 0 tasks\" trace.out &&\n+\t\t>trace.out\n+\t)\n+'\n+\n test_expect_success 'git diff --raw HEAD' '\n \thexsz=$(test_oid hexsz) &&\n \tgit diff --raw --abbrev=$hexsz HEAD >actual &&\n@@ -70,6 +89,18 @@ test_expect_success 'git diff HEAD with dirty submodule (work tree)' '\n \ttest_cmp expect.body actual.body\n '\n \n+test_expect_success 'git diff HEAD with dirty submodule (work tree, parallel)' '\n+\t(\n+\t\tcd sub &&\n+\t\tgit reset --hard &&\n+\t\techo >>world\n+\t) &&\n+\tgit -c submodule.diffJobs=8 diff HEAD >actual &&\n+\tsed -e \"1,/^@@/d\" actual >actual.body &&\n+\texpect_from_to >expect.body $subtip $subprev-dirty &&\n+\ttest_cmp expect.body actual.body\n+'\n+\n test_expect_success 'git diff HEAD with dirty submodule (index)' '\n \t(\n \t\tcd sub &&\ndiff --git a/t/t7506-status-submodule.sh b/t/t7506-status-submodule.sh\nindex d050091345..7da64e4c4c 100755\n--- a/t/t7506-status-submodule.sh\n+++ b/t/t7506-status-submodule.sh\n@@ -412,4 +412,29 @@ test_expect_success 'status with added file in nested submodule (short)' '\n \tEOF\n '\n \n+test_expect_success 'status in superproject with submodules respects parallel settings' '\n+\ttest_when_finished \"rm -f trace.out\" &&\n+\t(\n+\t\tGIT_TRACE=$(pwd)/trace.out git status &&\n+\t\tgrep \"1 tasks\" trace.out &&\n+\t\t>trace.out &&\n+\n+\t\tgit config submodule.diffJobs 8 &&\n+\t\tGIT_TRACE=$(pwd)/trace.out git status &&\n+\t\tgrep \"8 tasks\" trace.out &&\n+\t\t>trace.out &&\n+\n+\t\tGIT_TRACE=$(pwd)/trace.out git -c submodule.diffJobs=0 status &&\n+\t\tgrep \"preparing to run up to [0-9]* tasks\" trace.out &&\n+\t\t! grep \"up to 0 tasks\" trace.out &&\n+\t\t>trace.out\n+\t)\n+'\n+\n+test_expect_success 'status in superproject with submodules (parallel)' '\n+\tgit -C super status --porcelain >output &&\n+\tgit -C super -c submodule.diffJobs=8 status --porcelain >output_parallel &&\n+\tdiff output output_parallel\n+'\n+\n test_done\n-- \n2.39.1.519.gcb327c4b5f-goog\n\n"},{"id":"471733","messageId":"230207.86y1p914ci.gmgdl@evledraar.gmail.com","threadId":"59034","inReplyTo":"20230207181706.363453-2-calvinwan@google.com","subject":"Re: [PATCH v7 1/7] run-command: add duplicate_output_fn to run_processes_parallel_opts","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2023-02-07T22:16:20Z","receivedAt":"2023-02-07T22:46:28Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Feb 07 2023, Calvin Wan wrote:\n\n> diff --git a/run-command.c b/run-command.c\n> index 756f1839aa..cad88befe0 100644\n> --- a/run-command.c\n> +++ b/run-command.c\n> @@ -1526,6 +1526,9 @@ static void pp_init(struct parallel_processes *pp,\n> \tif (!opts->get_next_task)\n> \t\tBUG(\"you need to specify a get_next_task function\");\n> \n> +\tif (opts->duplicate_output && opts->ungroup)\n> +\t\tBUG(\"duplicate_output and ungroup are incompatible with each other\");\n> +\n> \tCALLOC_ARRAY(pp->children, n);\n> \tif (!opts->ungroup)\n> \t\tCALLOC_ARRAY(pp->pfd, n);\n\nA trivial request, not worth a re-roll in itself: The \"prep\" topic[1] I\nhave for Emily's eventual config-based hooks doesn't need to add new\nrun-command.c modes that are incompatible with ungroup, but that happens\nin the next stage of that saga.\n\nWhen I merge your topic here with that, the end result here is:\n\n\tif (opts->ungroup) {\n\t\tif (opts->feed_pipe)\n\t\t\tBUG(\".ungroup=1 is incompatible with .feed_pipe != NULL\");\n\t\tif (opts->consume_sideband)\n\t\t\tBUG(\".ungroup=1 is incompatible with .consume_sideband != NULL\");\n\t}\n\n\tif (!opts->get_next_task)\n\t\tBUG(\"you need to specify a get_next_task function\");\n\n\tif (opts->duplicate_output && opts->ungroup)\n\t\tBUG(\"duplicate_output and ungroup are incompatible with each other\");\n\nSo, whether do the incompatibility check before or after\n\"get_next_task\" is arbitrary. If I had to pick, I think doing it after as you're\ndoing here probably makes more sense.\n\nBut would ou mind if this addition of yours were instead:\n\n\tif (opts->ungroup) {\n\t\tif (opts->duplicate_output)\n\t\t\tBUG(\"duplicate_output and ungroup are incompatible with each other\")\n\t}\n\nLike I said, a trivial request.\n\nBut it will save us the eventual refactoring of that into nested checks\nas we add more of these options.\n\nTo the extent that we need to mention the seemingly odd looking pattern\nwe could just say that we're future-proofing this for future\nincompatible modes.\n\n1. https://lore.kernel.org/git/cover-0.5-00000000000-20230123T170550Z-avarab@gmail.com/#t\n\n> @@ -1645,14 +1648,21 @@ static void pp_buffer_stderr(struct parallel_processes *pp,\n> \tfor (size_t i = 0; i < opts->processes; i++) {\n> \t\tif (pp->children[i].state == GIT_CP_WORKING &&\n> \t\t pp->pfd[i].revents & (POLLIN | POLLHUP)) {\n> -\t\t\tint n = strbuf_read_once(&pp->children[i].err,\n> -\t\t\t\t\t\t pp->children[i].process.err, 0);\n> +\t\t\tssize_t n = strbuf_read_once(&pp->children[i].err,\n> +\t\t\t\t\t\t pp->children[i].process.err, 0);\n\nThis s/int/ssize_t/ change is a good on, but not mentioned in the commit\nmessage. Maybe worth splitting out?\n\nIf I revert that back to \"int\" on top of this entire topic our tests\nstill pass, so while it's a good change it seems entirely unrelated to\nthe \"duplicate_output\" subject of this patch.\n\n> \t\t\tif (n == 0) {\n> \t\t\t\tclose(pp->children[i].process.err);\n> \t\t\t\tpp->children[i].state = GIT_CP_WAIT_CLEANUP;\n> -\t\t\t} else if (n < 0)\n> +\t\t\t} else if (n < 0) {\n\nHere you're adding braces, which is an otherwise good change (but maybe\nworth splitting up, I haven't read the rest of this topic to see if\nthere's even more style changes).\n\nIn this case we should/could have done this change with the pre-image,\nbefore \"duplicate_output\".\n\n> \t\t\t\tif (errno != EAGAIN)\n> \t\t\t\t\tdie_errno(\"read\");\n> +\t\t\t} else {\n> +\t\t\t\tif (opts->duplicate_output)\n\nI've read ahead and this topic adds nothing new to this \"else\" block, so\nwhy the extra indentation instead of:\n\n\t} else if (opts->duplicate_output) {\n\t\t[...];\n\n> +\t\t\t\t\topts->duplicate_output(&pp->children[i].err,\n> +\t\t\t\t\t strlen(pp->children[i].err.buf) - n,\n\nUh, why are we getting the length of strbuf with strlen()? Am I missing\nsomething obvious here, or should this be:\n\n\tpp->children[i].err.len - n\n\n?\n\n> +\t\t\t\t\t opts->data,\n> +\t\t\t\t\t pp->children[i].data);\n\nEspecially with how otherwise painful the wrapping is here (well, not\nvery, but we can easily save a \\t-indent here).\n\n> +\t\t\t}\n> \t\t}\n> \t}\n> }\n> diff --git a/run-command.h b/run-command.h\n> index 072db56a4d..6dcf999f6c 100644\n> --- a/run-command.h\n> +++ b/run-command.h\n> @@ -408,6 +408,27 @@ typedef int (*start_failure_fn)(struct strbuf *out,\n> \t\t\t\tvoid *pp_cb,\n> \t\t\t\tvoid *pp_task_cb);\n> \n> +/**\n> + * This callback is called whenever output from a child process is buffered\n> + * \n> + * See run_processes_parallel() below for a discussion of the \"struct\n> + * strbuf *out\" parameter.\n> + * \n> + * The offset refers to the number of bytes originally in \"out\" before\n> + * the output from the child process was buffered. Therefore, the buffer\n> + * range, \"out + buf\" to the end of \"out\", would contain the buffer of\n> + * the child process output.\n> + *\n> + * pp_cb is the callback cookie as passed into run_processes_parallel,\n> + * pp_task_cb is the callback cookie as passed into get_next_task_fn.\n> + *\n> + * This function is incompatible with \"ungroup\"\n> + */\n> +typedef void (*duplicate_output_fn)(struct strbuf *out,\n> +\t\t\t\t size_t offset,\n> +\t\t\t\t void *pp_cb,\n> +\t\t\t\t void *pp_task_cb);\n\nThere's some over-wrapping here, I see some existing code does it, but\nfor new code we could follow our usual style, which would put this on\ntwo lines.\n\n> +\n> /**\n> * This callback is called on every child process that finished processing.\n> *\n> @@ -461,6 +482,12 @@ struct run_process_parallel_opts\n> \t */\n> \tstart_failure_fn start_failure;\n> \n> +\t/**\n> +\t * duplicate_output: See duplicate_output_fn() above. This should be\n> +\t * NULL unless process specific output is needed\n> +\t */\n\nHere we mostly refer to the previous docs, but the \"unless process\nspecific output is neeed\" is very confusing. Without seeing the name or\nhaving read the above I'd think this were some \"do_not_pipe_to_dev_null\"\nfeature.\n\nShouldn't we say \"Unless you need to capture the output... leave this at\nNULL\" or something?\n\n> +static void duplicate_output(struct strbuf *out,\n> +\t\t\tsize_t offset,\n> +\t\t\tvoid *pp_cb UNUSED,\n> +\t\t\tvoid *pp_task_cb UNUSED)\n> +{\n> +\tstruct string_list list = STRING_LIST_INIT_DUP;\n> +\n> +\tstring_list_split(&list, out->buf + offset, '\\n', -1);\n> +\tfor (size_t i = 0; i < list.nr; i++) {\n> +\t\tif (strlen(list.items[i].string) > 0)\n\nFirst, you can use for_each_string_list_item() here to make this look\nmuch nicer/simpler.\n\nSecond, don't use strlen(s) > 0, just use strlen(s).\n\nThird, you can git rid of the {} braces for the \"for\" here.\n\nBut just getting rid of that strlen() check and printing makes all your\ntests pass.\n\nAnd why is this thing that wants to prove to us that we're capturing the\noutput wanting to strip successive newlines?\n\nUsing a struct string_list for this is also pretty wasteful, we could\njust make this a while-loop that printed this string when it sees \"\\n\".\n\nBut it's just test code, so we don't care, I think it's fine for it to\nbe wastful, I just don't see why it's doing what it's doing, and what\nit's going out of its way to do isn't tested for here.\n\n> +test_expect_success 'run_command runs in parallel with more jobs available than tasks --duplicate-output' '\n> +\ttest-tool run-command --duplicate-output run-command-parallel 5 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n> +\ttest_must_be_empty out &&\n> +\ttest 4 = $(grep -c \"duplicate_output: Hello\" err) &&\n> +\ttest 4 = $(grep -c \"duplicate_output: World\" err) &&\n> +\tsed \"/duplicate_output/d\" err > err1 &&\n\nStyle: \">f\" not \"> f\".\n"},{"id":"471734","messageId":"230207.86ttzx13xc.gmgdl@evledraar.gmail.com","threadId":"59034","inReplyTo":"20230207181706.363453-3-calvinwan@google.com","subject":"Re: [PATCH v7 2/7] submodule: strbuf variable rename","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2023-02-07T22:47:53Z","receivedAt":"2023-02-07T22:55:41Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Feb 07 2023, Calvin Wan wrote:\n\n> A prepatory change for a future patch that moves the status parsing\n> logic to a separate function.\n\nAh, I think I suggested splitting this up in some previous round, and\ncoming back to this this + the next patch look very nice with the move\ndetection, thanks!\n> \tfp = xfdopen(cp.out, \"r\");\n> \twhile (strbuf_getwholeline(&buf, fp, '\\n') != EOF) {\n> +\t\tchar *str = buf.buf;\n> +\t\tconst size_t len = buf.len;\n> +\n> \t\t/* regular untracked files */\n> -\t\tif (buf.buf[0] == '?')\n> +\t\tif (str[0] == '?')\n> \t\t\tdirty_submodule |= DIRTY_SUBMODULE_UNTRACKED;\n\nI'll only add that we could also do this on top:\n\t\n\tdiff --git a/submodule.c b/submodule.c\n\tindex c7c6bfb2e26..eeb940d96a0 100644\n\t--- a/submodule.c\n\t+++ b/submodule.c\n\t@@ -1875,7 +1875,7 @@ unsigned is_submodule_modified(const char *path, int ignore_untracked)\n\t \tstruct child_process cp = CHILD_PROCESS_INIT;\n\t \tstruct strbuf buf = STRBUF_INIT;\n\t \tFILE *fp;\n\t-\tunsigned dirty_submodule = 0;\n\t+\tunsigned dirty_submodule0 = 0;\n\t \tconst char *git_dir;\n\t \tint ignore_cp_exit_code = 0;\n\t \n\t@@ -1908,10 +1908,11 @@ unsigned is_submodule_modified(const char *path, int ignore_untracked)\n\t \twhile (strbuf_getwholeline(&buf, fp, '\\n') != EOF) {\n\t \t\tchar *str = buf.buf;\n\t \t\tconst size_t len = buf.len;\n\t+\t\tunsigned *dirty_submodule = &dirty_submodule0;\n\t \n\t \t\t/* regular untracked files */\n\t \t\tif (str[0] == '?')\n\t-\t\t\tdirty_submodule |= DIRTY_SUBMODULE_UNTRACKED;\n\t+\t\t\t*dirty_submodule |= DIRTY_SUBMODULE_UNTRACKED;\n\t \n\t \t\tif (str[0] == 'u' ||\n\t \t\t str[0] == '1' ||\n\t@@ -1923,17 +1924,17 @@ unsigned is_submodule_modified(const char *path, int ignore_untracked)\n\t \n\t \t\t\tif (str[5] == 'S' && str[8] == 'U')\n\t \t\t\t\t/* nested untracked file */\n\t-\t\t\t\tdirty_submodule |= DIRTY_SUBMODULE_UNTRACKED;\n\t+\t\t\t\t*dirty_submodule |= DIRTY_SUBMODULE_UNTRACKED;\n\t \n\t \t\t\tif (str[0] == 'u' ||\n\t \t\t\t str[0] == '2' ||\n\t \t\t\t memcmp(str + 5, \"S..U\", 4))\n\t \t\t\t\t/* other change */\n\t-\t\t\t\tdirty_submodule |= DIRTY_SUBMODULE_MODIFIED;\n\t+\t\t\t\t*dirty_submodule |= DIRTY_SUBMODULE_MODIFIED;\n\t \t\t}\n\t \n\t-\t\tif ((dirty_submodule & DIRTY_SUBMODULE_MODIFIED) &&\n\t-\t\t ((dirty_submodule & DIRTY_SUBMODULE_UNTRACKED) ||\n\t+\t\tif ((*dirty_submodule & DIRTY_SUBMODULE_MODIFIED) &&\n\t+\t\t ((*dirty_submodule & DIRTY_SUBMODULE_UNTRACKED) ||\n\t \t\t ignore_untracked)) {\n\t \t\t\t/*\n\t \t\t\t * We're not interested in any further information from\n\t@@ -1949,7 +1950,7 @@ unsigned is_submodule_modified(const char *path, int ignore_untracked)\n\t \t\tdie(_(\"'git status --porcelain=2' failed in submodule %s\"), path);\n\t \n\t \tstrbuf_release(&buf);\n\t-\treturn dirty_submodule;\n\t+\treturn dirty_submodule0;\n\t }\n\t \n\t int submodule_uses_gitfile(const char *path)\n\nWhich, if we're massaging this for a subsequent smaller diff we can do\nto make only the comment adjustment part of this be a non-moved line.\n"},{"id":"471736","messageId":"230208.86pmal13jo.gmgdl@evledraar.gmail.com","threadId":"59034","inReplyTo":"20230207181706.363453-5-calvinwan@google.com","subject":"Re: [PATCH v7 4/7] submodule: refactor is_submodule_modified()","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2023-02-07T22:59:05Z","receivedAt":"2023-02-07T23:03:51Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Feb 07 2023, Calvin Wan wrote:\n\n> diff --git a/submodule.c b/submodule.c\n> index 768d4b4cd7..d88aa2c573 100644\n> --- a/submodule.c\n> +++ b/submodule.c\n> @@ -28,6 +28,10 @@ static int config_update_recurse_submodules = RECURSE_SUBMODULES_OFF;\n> static int initialized_fetch_ref_tips;\n> static struct oid_array ref_tips_before_fetch;\n> static struct oid_array ref_tips_after_fetch;\n> +static const char *status_porcelain_start_error =\n> +\tN_(\"could not run 'git status --porcelain=2' in submodule %s\");\n> +static const char *status_porcelain_fail_error =\n> +\tN_(\"'git status --porcelain=2' failed in submodule %s\");\n\nLet's instead do:\n\n\t#define STATUS_PORCELAIN_START_ERROR \\\n \tN_(\"could not run 'git status --porcelain=2' in submodule %s\")\n\t#define STATUS_PORCELAIN_FAIL_ERROR \\\n \tN_(\"'git status --porcelain=2' failed in submodule %s\")\n\nBecause a thing you're not discussing in the commit message is that the\ndisadvantage of doing this sort of thing is that we lose the checking\nthat -Wformat gives us (try to add an extra \"%s\" to these in your\nversion, then the macro version, with gcc and/or clang).\n\nPersonally I'd prefer just copy/pasting over losing that, but using a\nmacro instead of a variable allows us to have our cake and eat it too.\n"},{"id":"471747","messageId":"230208.86ilgd0yej.gmgdl@evledraar.gmail.com","threadId":"59034","inReplyTo":"20230207181706.363453-8-calvinwan@google.com","subject":"Re: [PATCH v7 7/7] diff-lib: parallelize run_diff_files for submodules","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2023-02-07T23:06:48Z","receivedAt":"2023-02-08T00:54:57Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Feb 07 2023, Calvin Wan wrote:\n\n> [...]\n> +\tsps->result = 1;\n> +\tstrbuf_addf(err,\n> +\t _(status_porcelain_start_error),\n> +\t task->path);\n> +\treturn 0;\n> [...]\n> +\tif (retvalue) {\n> +\t\tsps->result = 1;\n> +\t\tstrbuf_addf(err,\n> +\t\t _(status_porcelain_fail_error),\n> +\t\t task->path);\n> [...]\n\nThis is nitpicky, but what's with the short lines and over-wrapping?\n\nIf you change these two to (just using my macro version on top, but it's\nthe same with yours):\n\n\tstrbuf_addf(err, _(STATUS_PORCELAIN_START_ERROR), task->path);\n\nAnd:\n\n\tstrbuf_addf(err, _(STATUS_PORCELAIN_FAIL_ERROR), task->path);\n\nBoth of these are under our usual line limit at their respective\nindentation (the latter at 77, rule of thumb is to wrap at 79-80).\n\n> +\tif (submodules.nr > 0) {\n\nDon't compare unsigned to >0, just use \"submodules.nr\".\n\n> +\t\tint parallel_jobs;\n\nnit: add extra \\n, or maybe just call this \"int v\", as it's clear from\nthe scope what it's about...\n\n> +\t\tif (git_config_get_int(\"submodule.diffjobs\", ¶llel_jobs))\n> +\t\t\tparallel_jobs = 1;\n> +\t\telse if (!parallel_jobs)\n> +\t\t\tparallel_jobs = online_cpus();\n> +\t\telse if (parallel_jobs < 0)\n> +\t\t\tdie(_(\"submodule.diffjobs cannot be negative\"));\n\nCan't you use the \"ulong\" instead of \"int\" and have it handle this \"is\nnegative?\" error check for you?\n\n> +\n> +\t\tif (get_submodules_status(&submodules, parallel_jobs))\n> +\t\t\tdie(_(\"submodule status failed\"));\n> +\t\tfor (size_t i = 0; i < submodules.nr; i++) {\n\nAnother case that can use for_each_string_list_item().\n\n> +struct submodule_parallel_status {\n> +\tsize_t index_count;\n> +\tint result;\n> +\n> +\tstruct string_list *submodule_names;\n> +\n> +\t/* Pending statuses by OIDs */\n> +\tstruct status_task **oid_status_tasks;\n> +\tint oid_status_tasks_nr, oid_status_tasks_alloc;\n\nFor new structs, let's use size_t, not \"int\" for alloc/nr.\n\nAlso, as this is 7/7 and we're not adding another such pattern for the\nforseeable future, can we just call these \"size_t nr\", \"size_t alloc\"\nand \"tasks\"?\n\nAnd having said all that, it turns out this is just dead code that can\nbe removed? Blindly copied from submodule_parallel_fetch?\n"},{"id":"471748","messageId":"230208.86edr10y9h.gmgdl@evledraar.gmail.com","threadId":"59034","inReplyTo":"20230207181706.363453-1-calvinwan@google.com","subject":"Re: [PATCH v7 0/7] submodule: parallelize diff","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2023-02-08T00:55:35Z","receivedAt":"2023-02-08T00:57:52Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Feb 07 2023, Calvin Wan wrote:\n\n> Original cover letter for context:\n> https://lore.kernel.org/git/20221011232604.839941-1-calvinwan@google.com/\n\nI went over this, noticed some issues, some nits, but definitely some\nthings worth an eventual re-roll.\n\n> Changes since v6\n\nI would very much appreciate for future iterations if you can start\nincluding a range-diff to the previous version.\n\n> Added patches 4 and 5 to refactor out more functionality so that it is\n> clear what changes my final patch makes. Since the large majority of\n> the functionality between the serial and parallel implementation is now\n> shared, I no longer remove the serial implementation.\n>\n> Added additional tests to verify setting parallelism doesn't alter\n> output\n\nI could have, but didn't manually apply both v6 and v7 and produce a\nrange-diff, having it in the CL would really help to track the changes\nacross re-rolls.\n"},{"id":"471756","messageId":"230208.861qn01s4g.gmgdl@evledraar.gmail.com","threadId":"59034","inReplyTo":"20230207181706.363453-7-calvinwan@google.com","subject":"Re: [PATCH v7 6/7] diff-lib: refactor match_stat_with_submodule","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2023-02-08T08:18:54Z","receivedAt":"2023-02-08T08:25:15Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Feb 07 2023, Calvin Wan wrote:\n\n> diff --git a/diff-lib.c b/diff-lib.c\n> index 7101cfda3f..e18c886a80 100644\n> --- a/diff-lib.c\n> +++ b/diff-lib.c\n> @@ -73,18 +73,24 @@ static int match_stat_with_submodule(struct diff_options *diffopt,\n> \t\t\t\t unsigned *dirty_submodule)\n> {\n> \tint changed = ie_match_stat(diffopt->repo->index, ce, st, ce_option);\n> -\tif (S_ISGITLINK(ce->ce_mode)) {\n> -\t\tstruct diff_flags orig_flags = diffopt->flags;\n> -\t\tif (!diffopt->flags.override_submodule_config)\n> -\t\t\tset_diffopt_flags_from_submodule_config(diffopt, ce->name);\n> -\t\tif (diffopt->flags.ignore_submodules)\n> -\t\t\tchanged = 0;\n> -\t\telse if (!diffopt->flags.ignore_dirty_submodules &&\n> -\t\t\t (!changed || diffopt->flags.dirty_submodules))\n> -\t\t\t*dirty_submodule = is_submodule_modified(ce->name,\n> -\t\t\t\t\t\t\t\t diffopt->flags.ignore_untracked_in_submodules);\n> -\t\tdiffopt->flags = orig_flags;\n> +\tstruct diff_flags orig_flags;\n> +\n> +\tif (!S_ISGITLINK(ce->ce_mode))\n> +\t\treturn changed;\n> +\n> +\torig_flags = diffopt->flags;\n> +\tif (!diffopt->flags.override_submodule_config)\n> +\t\tset_diffopt_flags_from_submodule_config(diffopt, ce->name);\n> +\tif (diffopt->flags.ignore_submodules) {\n> +\t\tchanged = 0;\n> +\t\tgoto cleanup;\n> \t}\n> +\tif (!diffopt->flags.ignore_dirty_submodules &&\n> +\t (!changed || diffopt->flags.dirty_submodules))\n> +\t\t*dirty_submodule = is_submodule_modified(ce->name,\n> +\t\t\t\t\t diffopt->flags.ignore_untracked_in_submodules);\n> +cleanup:\n> +\tdiffopt->flags = orig_flags;\n> \treturn changed;\n> }\n\nParallel to reviewing your topic I started wondering if we couldn't get\nrid of this \"orig_flags\" flip-flopping, i.e. can't we just set the\nspecific flags we want in output parameters.\n\nAnyway, having looked at this closely I think this patch should be\ndropped entirely. I don't understand how this refactoring is meant to\nmake the end result easier to read, reason about, or how it helps the\nsubsequent patch.\n\nIn addition to the above diff in 7/7 you do (and that's the change this\nis meant to help):\n\t\n\t static int match_stat_with_submodule(struct diff_options *diffopt,\n\t \t\t\t\t const struct cache_entry *ce,\n\t \t\t\t\t struct stat *st, unsigned ce_option,\n\t-\t\t\t\t unsigned *dirty_submodule)\n\t+\t\t\t\t unsigned *dirty_submodule, int *defer_submodule_status,\n\t+\t\t\t\t unsigned *ignore_untracked)\n\t {\n\t \tint changed = ie_match_stat(diffopt->repo->index, ce, st, ce_option);\n\t \tstruct diff_flags orig_flags;\n\t+\tint defer = 0;\n\t \n\t \tif (!S_ISGITLINK(ce->ce_mode))\n\t-\t\treturn changed;\n\t+\t\tgoto ret;\n\t \n\t \torig_flags = diffopt->flags;\n\t \tif (!diffopt->flags.override_submodule_config)\n\t@@ -86,11 +92,20 @@ static int match_stat_with_submodule(struct diff_options *diffopt,\n\t \t\tgoto cleanup;\n\t \t}\n\t \tif (!diffopt->flags.ignore_dirty_submodules &&\n\t-\t (!changed || diffopt->flags.dirty_submodules))\n\t-\t\t*dirty_submodule = is_submodule_modified(ce->name,\n\t+\t (!changed || diffopt->flags.dirty_submodules)) {\n\t+\t\tif (defer_submodule_status && *defer_submodule_status) {\n\t+\t\t\tdefer = 1;\n\t+\t\t\t*ignore_untracked = diffopt->flags.ignore_untracked_in_submodules;\n\t+\t\t} else {\n\t+\t\t\t*dirty_submodule = is_submodule_modified(ce->name,\n\t \t\t\t\t\t diffopt->flags.ignore_untracked_in_submodules);\n\t+\t\t}\n\t+\t}\n\t cleanup:\n\t \tdiffopt->flags = orig_flags;\n\t+ret:\n\t+\tif (defer_submodule_status)\n\t+\t\t*defer_submodule_status = defer;\n\t \treturn changed;\n\t }\n\nBut if I rebase out this 6/7 patch and solve the conflict for 7/7 it\nbecomes:\n\t\n\t@@ -65,14 +66,20 @@ static int check_removed(const struct index_state *istate, const struct cache_en\n\t * Return 1 when changes are detected, 0 otherwise. If the DIRTY_SUBMODULES\n\t * option is set, the caller does not only want to know if a submodule is\n\t * modified at all but wants to know all the conditions that are met (new\n\t- * commits, untracked content and/or modified content).\n\t+ * commits, untracked content and/or modified content). If\n\t+ * defer_submodule_status bit is set, dirty_submodule will be left to the\n\t+ * caller to set. defer_submodule_status can also be set to 0 in this\n\t+ * function if there is no need to check if the submodule is modified.\n\t */\n\t static int match_stat_with_submodule(struct diff_options *diffopt,\n\t \t\t\t\t const struct cache_entry *ce,\n\t \t\t\t\t struct stat *st, unsigned ce_option,\n\t-\t\t\t\t unsigned *dirty_submodule)\n\t+\t\t\t\t unsigned *dirty_submodule, int *defer_submodule_status,\n\t+\t\t\t\t unsigned *ignore_untracked)\n\t {\n\t \tint changed = ie_match_stat(diffopt->repo->index, ce, st, ce_option);\n\t+\tint defer = 0;\n\t+\n\t \tif (S_ISGITLINK(ce->ce_mode)) {\n\t \t\tstruct diff_flags orig_flags = diffopt->flags;\n\t \t\tif (!diffopt->flags.override_submodule_config)\n\t@@ -80,11 +87,20 @@ static int match_stat_with_submodule(struct diff_options *diffopt,\n\t \t\tif (diffopt->flags.ignore_submodules)\n\t \t\t\tchanged = 0;\n\t \t\telse if (!diffopt->flags.ignore_dirty_submodules &&\n\t-\t\t\t (!changed || diffopt->flags.dirty_submodules))\n\t-\t\t\t*dirty_submodule = is_submodule_modified(ce->name,\n\t-\t\t\t\t\t\t\t\t diffopt->flags.ignore_untracked_in_submodules);\n\t+\t\t\t (!changed || diffopt->flags.dirty_submodules)) {\n\t+\t\t\tif (defer_submodule_status && *defer_submodule_status) {\n\t+\t\t\t\tdefer = 1;\n\t+\t\t\t\t*ignore_untracked = diffopt->flags.ignore_untracked_in_submodules;\n\t+\t\t\t} else {\n\t+\t\t\t\t*dirty_submodule = is_submodule_modified(ce->name,\n\t+\t\t\t\t\t\t\t\t\t diffopt->flags.ignore_untracked_in_submodules);\n\t+\t\t\t}\n\t+\t\t}\n\t \t\tdiffopt->flags = orig_flags;\n\t \t}\n\t+\n\t+\tif (defer_submodule_status)\n\t+\t\t*defer_submodule_status = defer;\n\t \treturn changed;\n\t }\n\t \n\nI can see how there's some room for *a* refactoring to reduce the\nsubsequent diff, but not by mutch.\n\nBut this commit didn't help at all. This whole \"goto ret\", and \"goto\ncleanup\" is just working around the fact that you pulled \"orig_flags\"\nout of the \"if\" scope. Normally the de-indentation would be worth it,\nbut here it's not. The control flow becomes more complex to reason about\nas a result.\n\n"},{"id":"471762","messageId":"cd528776-caa8-6372-0ca5-db0545947fba@dunelm.org.uk","threadId":"59034","inReplyTo":"20230207181706.363453-2-calvinwan@google.com","subject":"Re: [PATCH v7 1/7] run-command: add duplicate_output_fn to run_processes_parallel_opts","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2023-02-08T14:19:55Z","receivedAt":"2023-02-08T14:20:27Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"Hi Calvin\n\nOn 07/02/2023 18:17, Calvin Wan wrote:\n> Add duplicate_output_fn as an optionally set function in\n> run_process_parallel_opts. If set, output from each child process is\n> copied and passed to the callback function whenever output from the\n> child process is buffered to allow for separate parsing.\n> \n> Signed-off-by: Calvin Wan \n> ---\n> run-command.c | 16 ++++++++++++---\n> run-command.h | 27 +++++++++++++++++++++++++\n> t/helper/test-run-command.c | 21 ++++++++++++++++++++\n> t/t0061-run-command.sh | 39 +++++++++++++++++++++++++++++++++++++\n> 4 files changed, 100 insertions(+), 3 deletions(-)\n> \n> diff --git a/run-command.c b/run-command.c\n> index 756f1839aa..cad88befe0 100644\n> --- a/run-command.c\n> +++ b/run-command.c\n> @@ -1526,6 +1526,9 @@ static void pp_init(struct parallel_processes *pp,\n> \tif (!opts->get_next_task)\n> \t\tBUG(\"you need to specify a get_next_task function\");\n> \n> +\tif (opts->duplicate_output && opts->ungroup)\n> +\t\tBUG(\"duplicate_output and ungroup are incompatible with each other\");\n> +\n> \tCALLOC_ARRAY(pp->children, n);\n> \tif (!opts->ungroup)\n> \t\tCALLOC_ARRAY(pp->pfd, n);\n> @@ -1645,14 +1648,21 @@ static void pp_buffer_stderr(struct parallel_processes *pp,\n> \tfor (size_t i = 0; i < opts->processes; i++) {\n> \t\tif (pp->children[i].state == GIT_CP_WORKING &&\n> \t\t pp->pfd[i].revents & (POLLIN | POLLHUP)) {\n> -\t\t\tint n = strbuf_read_once(&pp->children[i].err,\n> -\t\t\t\t\t\t pp->children[i].process.err, 0);\n> +\t\t\tssize_t n = strbuf_read_once(&pp->children[i].err,\n> +\t\t\t\t\t\t pp->children[i].process.err, 0);\n> \t\t\tif (n == 0) {\n> \t\t\t\tclose(pp->children[i].process.err);\n> \t\t\t\tpp->children[i].state = GIT_CP_WAIT_CLEANUP;\n> -\t\t\t} else if (n < 0)\n> +\t\t\t} else if (n < 0) {\n> \t\t\t\tif (errno != EAGAIN)\n> \t\t\t\t\tdie_errno(\"read\");\n> +\t\t\t} else {\n> +\t\t\t\tif (opts->duplicate_output)\n> +\t\t\t\t\topts->duplicate_output(&pp->children[i].err,\n> +\t\t\t\t\t strlen(pp->children[i].err.buf) - n,\n\nLooking at how this is used in patch 7 I think it would be better to \npass a const char*, length pair rather than a struct strbuf*, offset pair.\ni.e.\n\topts->duplicate_output(pp->children[i].err.buf + \npp->children[i].err.len - n, n, ...)\n\nThat would make it clear that we do not expect duplicate_output() to \nalter the buffer and would avoid the duplicate_output() having to add \nthe offset to the start of the buffer to find the new data.\n\nBest Wishes\n\nPhillip\n\n\n> +\t\t\t\t\t opts->data,\n> +\t\t\t\t\t pp->children[i].data);\n> +\t\t\t}\n> \t\t}\n> \t}\n> }\n> diff --git a/run-command.h b/run-command.h\n> index 072db56a4d..6dcf999f6c 100644\n> --- a/run-command.h\n> +++ b/run-command.h\n> @@ -408,6 +408,27 @@ typedef int (*start_failure_fn)(struct strbuf *out,\n> \t\t\t\tvoid *pp_cb,\n> \t\t\t\tvoid *pp_task_cb);\n> \n> +/**\n> + * This callback is called whenever output from a child process is buffered\n> + *\n> + * See run_processes_parallel() below for a discussion of the \"struct\n> + * strbuf *out\" parameter.\n> + *\n> + * The offset refers to the number of bytes originally in \"out\" before\n> + * the output from the child process was buffered. Therefore, the buffer\n> + * range, \"out + buf\" to the end of \"out\", would contain the buffer of\n> + * the child process output.\n> + *\n> + * pp_cb is the callback cookie as passed into run_processes_parallel,\n> + * pp_task_cb is the callback cookie as passed into get_next_task_fn.\n> + *\n> + * This function is incompatible with \"ungroup\"\n> + */\n> +typedef void (*duplicate_output_fn)(struct strbuf *out,\n> +\t\t\t\t size_t offset,\n> +\t\t\t\t void *pp_cb,\n> +\t\t\t\t void *pp_task_cb);\n> +\n> /**\n> * This callback is called on every child process that finished processing.\n> *\n> @@ -461,6 +482,12 @@ struct run_process_parallel_opts\n> \t */\n> \tstart_failure_fn start_failure;\n> \n> +\t/**\n> +\t * duplicate_output: See duplicate_output_fn() above. This should be\n> +\t * NULL unless process specific output is needed\n> +\t */\n> +\tduplicate_output_fn duplicate_output;\n> +\n> \t/**\n> \t * task_finished: See task_finished_fn() above. This can be\n> \t * NULL to omit any special handling.\n> diff --git a/t/helper/test-run-command.c b/t/helper/test-run-command.c\n> index 3ecb830f4a..ffd3cd0045 100644\n> --- a/t/helper/test-run-command.c\n> +++ b/t/helper/test-run-command.c\n> @@ -52,6 +52,21 @@ static int no_job(struct child_process *cp,\n> \treturn 0;\n> }\n> \n> +static void duplicate_output(struct strbuf *out,\n> +\t\t\tsize_t offset,\n> +\t\t\tvoid *pp_cb UNUSED,\n> +\t\t\tvoid *pp_task_cb UNUSED)\n> +{\n> +\tstruct string_list list = STRING_LIST_INIT_DUP;\n> +\n> +\tstring_list_split(&list, out->buf + offset, '\\n', -1);\n> +\tfor (size_t i = 0; i < list.nr; i++) {\n> +\t\tif (strlen(list.items[i].string) > 0)\n> +\t\t\tfprintf(stderr, \"duplicate_output: %s\\n\", list.items[i].string);\n> +\t}\n> +\tstring_list_clear(&list, 0);\n> +}\n> +\n> static int task_finished(int result,\n> \t\t\t struct strbuf *err,\n> \t\t\t void *pp_cb,\n> @@ -439,6 +454,12 @@ int cmd__run_command(int argc, const char **argv)\n> \t\topts.ungroup = 1;\n> \t}\n> \n> +\tif (!strcmp(argv[1], \"--duplicate-output\")) {\n> +\t\targv += 1;\n> +\t\targc -= 1;\n> +\t\topts.duplicate_output = duplicate_output;\n> +\t}\n> +\n> \tjobs = atoi(argv[2]);\n> \tstrvec_clear(&proc.args);\n> \tstrvec_pushv(&proc.args, (const char **)argv + 3);\n> diff --git a/t/t0061-run-command.sh b/t/t0061-run-command.sh\n> index e2411f6a9b..879e536638 100755\n> --- a/t/t0061-run-command.sh\n> +++ b/t/t0061-run-command.sh\n> @@ -135,6 +135,15 @@ test_expect_success 'run_command runs in parallel with more jobs available than\n> \ttest_cmp expect actual\n> '\n> \n> +test_expect_success 'run_command runs in parallel with more jobs available than tasks --duplicate-output' '\n> +\ttest-tool run-command --duplicate-output run-command-parallel 5 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n> +\ttest_must_be_empty out &&\n> +\ttest 4 = $(grep -c \"duplicate_output: Hello\" err) &&\n> +\ttest 4 = $(grep -c \"duplicate_output: World\" err) &&\n> +\tsed \"/duplicate_output/d\" err > err1 &&\n> +\ttest_cmp expect err1\n> +'\n> +\n> test_expect_success 'run_command runs ungrouped in parallel with more jobs available than tasks' '\n> \ttest-tool run-command --ungroup run-command-parallel 5 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n> \ttest_line_count = 8 out &&\n> @@ -147,6 +156,15 @@ test_expect_success 'run_command runs in parallel with as many jobs as tasks' '\n> \ttest_cmp expect actual\n> '\n> \n> +test_expect_success 'run_command runs in parallel with as many jobs as tasks --duplicate-output' '\n> +\ttest-tool run-command --duplicate-output run-command-parallel 4 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n> +\ttest_must_be_empty out &&\n> +\ttest 4 = $(grep -c \"duplicate_output: Hello\" err) &&\n> +\ttest 4 = $(grep -c \"duplicate_output: World\" err) &&\n> +\tsed \"/duplicate_output/d\" err > err1 &&\n> +\ttest_cmp expect err1\n> +'\n> +\n> test_expect_success 'run_command runs ungrouped in parallel with as many jobs as tasks' '\n> \ttest-tool run-command --ungroup run-command-parallel 4 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n> \ttest_line_count = 8 out &&\n> @@ -159,6 +177,15 @@ test_expect_success 'run_command runs in parallel with more tasks than jobs avai\n> \ttest_cmp expect actual\n> '\n> \n> +test_expect_success 'run_command runs in parallel with more tasks than jobs available --duplicate-output' '\n> +\ttest-tool run-command --duplicate-output run-command-parallel 3 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n> +\ttest_must_be_empty out &&\n> +\ttest 4 = $(grep -c \"duplicate_output: Hello\" err) &&\n> +\ttest 4 = $(grep -c \"duplicate_output: World\" err) &&\n> +\tsed \"/duplicate_output/d\" err > err1 &&\n> +\ttest_cmp expect err1\n> +'\n> +\n> test_expect_success 'run_command runs ungrouped in parallel with more tasks than jobs available' '\n> \ttest-tool run-command --ungroup run-command-parallel 3 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n> \ttest_line_count = 8 out &&\n> @@ -180,6 +207,12 @@ test_expect_success 'run_command is asked to abort gracefully' '\n> \ttest_cmp expect actual\n> '\n> \n> +test_expect_success 'run_command is asked to abort gracefully --duplicate-output' '\n> +\ttest-tool run-command --duplicate-output run-command-abort 3 false >out 2>err &&\n> +\ttest_must_be_empty out &&\n> +\ttest_cmp expect err\n> +'\n> +\n> test_expect_success 'run_command is asked to abort gracefully (ungroup)' '\n> \ttest-tool run-command --ungroup run-command-abort 3 false >out 2>err &&\n> \ttest_must_be_empty out &&\n> @@ -196,6 +229,12 @@ test_expect_success 'run_command outputs ' '\n> \ttest_cmp expect actual\n> '\n> \n> +test_expect_success 'run_command outputs --duplicate-output' '\n> +\ttest-tool run-command --duplicate-output run-command-no-jobs 3 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n> +\ttest_must_be_empty out &&\n> +\ttest_cmp expect err\n> +'\n> +\n> test_expect_success 'run_command outputs (ungroup) ' '\n> \ttest-tool run-command --ungroup run-command-no-jobs 3 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n> \ttest_must_be_empty out &&\n"},{"id":"471763","messageId":"c9b10e8b-641c-ff29-95a4-2ac3f3219c0d@dunelm.org.uk","threadId":"59034","inReplyTo":"20230207181706.363453-7-calvinwan@google.com","subject":"Re: [PATCH v7 6/7] diff-lib: refactor match_stat_with_submodule","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2023-02-08T14:22:33Z","receivedAt":"2023-02-08T14:22:44Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"Hi Calvin\n\nOn 07/02/2023 18:17, Calvin Wan wrote:\n> Flatten out the if statements in match_stat_with_submodule so the\n> logic is more readable and easier for future patches to add to.\n> orig_flags didn't need to be set if the cache entry wasn't a\n> GITLINK so defer setting it.\n> \n> Signed-off-by: Calvin Wan \n> ---\n> diff-lib.c | 28 +++++++++++++++++-----------\n> 1 file changed, 17 insertions(+), 11 deletions(-)\n> \n> diff --git a/diff-lib.c b/diff-lib.c\n> index 7101cfda3f..e18c886a80 100644\n> --- a/diff-lib.c\n> +++ b/diff-lib.c\n> @@ -73,18 +73,24 @@ static int match_stat_with_submodule(struct diff_options *diffopt,\n> \t\t\t\t unsigned *dirty_submodule)\n> {\n> \tint changed = ie_match_stat(diffopt->repo->index, ce, st, ce_option);\n> -\tif (S_ISGITLINK(ce->ce_mode)) {\n> -\t\tstruct diff_flags orig_flags = diffopt->flags;\n> -\t\tif (!diffopt->flags.override_submodule_config)\n> -\t\t\tset_diffopt_flags_from_submodule_config(diffopt, ce->name);\n> -\t\tif (diffopt->flags.ignore_submodules)\n> -\t\t\tchanged = 0;\n> -\t\telse if (!diffopt->flags.ignore_dirty_submodules &&\n> -\t\t\t (!changed || diffopt->flags.dirty_submodules))\n> -\t\t\t*dirty_submodule = is_submodule_modified(ce->name,\n> -\t\t\t\t\t\t\t\t diffopt->flags.ignore_untracked_in_submodules);\n> -\t\tdiffopt->flags = orig_flags;\n> +\tstruct diff_flags orig_flags;\n> +\n> +\tif (!S_ISGITLINK(ce->ce_mode))\n> +\t\treturn changed;\n> +\n> +\torig_flags = diffopt->flags;\n> +\tif (!diffopt->flags.override_submodule_config)\n> +\t\tset_diffopt_flags_from_submodule_config(diffopt, ce->name);\n> +\tif (diffopt->flags.ignore_submodules) {\n> +\t\tchanged = 0;\n> +\t\tgoto cleanup;\n\nLooking ahead to patch 7 there are no new uses of the \"cleanup\" label so \nI think it would be simpler to leave the code as it was, rather than \nchanging the \"else if\" below to \"if\" and adding the goto here.\n\nBest Wishes\n\nPhillip\n\n> \t}\n> +\tif (!diffopt->flags.ignore_dirty_submodules &&\n> +\t (!changed || diffopt->flags.dirty_submodules))\n> +\t\t*dirty_submodule = is_submodule_modified(ce->name,\n> +\t\t\t\t\t diffopt->flags.ignore_untracked_in_submodules);\n> +cleanup:\n> +\tdiffopt->flags = orig_flags;\n> \treturn changed;\n> }\n> \n"},{"id":"471764","messageId":"4ae106ed-bfa8-2824-c0c7-6cde32dbe369@dunelm.org.uk","threadId":"59034","inReplyTo":"20230207181706.363453-6-calvinwan@google.com","subject":"Re: [PATCH v7 5/7] diff-lib: refactor out diff_change logic","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2023-02-08T14:28:32Z","receivedAt":"2023-02-08T14:28:39Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"Hi Calvin\n\nOn 07/02/2023 18:17, Calvin Wan wrote:\n> Refactor out logic that sets up the diff_change call into a helper\n> function for a future patch.\n> \n> Signed-off-by: Calvin Wan \n> ---\n> diff-lib.c | 46 +++++++++++++++++++++++++++++-----------------\n> 1 file changed, 29 insertions(+), 17 deletions(-)\n> \n> diff --git a/diff-lib.c b/diff-lib.c\n> index dec040c366..7101cfda3f 100644\n> --- a/diff-lib.c\n> +++ b/diff-lib.c\n> @@ -88,6 +88,31 @@ static int match_stat_with_submodule(struct diff_options *diffopt,\n> \treturn changed;\n> }\n> \n> +static int diff_change_helper(struct diff_options *options,\n> +\t unsigned newmode, unsigned dirty_submodule,\n> +\t int changed,\n\nI worry that having three integer parameters next to each other makes it \nvery easy to mix them up with out getting any errors from the compiler \nbecause the types are all compatible. Could the last two be combined \ninto a flags argument? A similar issues occurs in \nmatch_stat_with_submodule() in patch 7\n\nBest Wishes\n\nPhillip\n\n struct index_state *istate,\n> +\t struct cache_entry *ce)\n> +{\n> +\tunsigned int oldmode;\n> +\tconst struct object_id *old_oid, *new_oid;\n> +\n> +\tif (!changed && !dirty_submodule) {\n> +\t\tce_mark_uptodate(ce);\n> +\t\tmark_fsmonitor_valid(istate, ce);\n> +\t\tif (!options->flags.find_copies_harder)\n> +\t\t\treturn 1;\n> +\t}\n> +\toldmode = ce->ce_mode;\n> +\told_oid = &ce->oid;\n> +\tnew_oid = changed ? null_oid() : &ce->oid;\n> +\tdiff_change(options, oldmode, newmode,\n> +\t\t\told_oid, new_oid,\n> +\t\t\t!is_null_oid(old_oid),\n> +\t\t\t!is_null_oid(new_oid),\n> +\t\t\tce->name, 0, dirty_submodule);\n> +\treturn 0;\n> +}\n> +\n> int run_diff_files(struct rev_info *revs, unsigned int option)\n> {\n> \tint entries, i;\n> @@ -105,11 +130,10 @@ int run_diff_files(struct rev_info *revs, unsigned int option)\n> \t\tdiff_unmerged_stage = 2;\n> \tentries = istate->cache_nr;\n> \tfor (i = 0; i < entries; i++) {\n> -\t\tunsigned int oldmode, newmode;\n> +\t\tunsigned int newmode;\n> \t\tstruct cache_entry *ce = istate->cache[i];\n> \t\tint changed;\n> \t\tunsigned dirty_submodule = 0;\n> -\t\tconst struct object_id *old_oid, *new_oid;\n> \n> \t\tif (diff_can_quit_early(&revs->diffopt))\n> \t\t\tbreak;\n> @@ -245,21 +269,9 @@ int run_diff_files(struct rev_info *revs, unsigned int option)\n> \t\t\tnewmode = ce_mode_from_stat(ce, st.st_mode);\n> \t\t}\n> \n> -\t\tif (!changed && !dirty_submodule) {\n> -\t\t\tce_mark_uptodate(ce);\n> -\t\t\tmark_fsmonitor_valid(istate, ce);\n> -\t\t\tif (!revs->diffopt.flags.find_copies_harder)\n> -\t\t\t\tcontinue;\n> -\t\t}\n> -\t\toldmode = ce->ce_mode;\n> -\t\told_oid = &ce->oid;\n> -\t\tnew_oid = changed ? null_oid() : &ce->oid;\n> -\t\tdiff_change(&revs->diffopt, oldmode, newmode,\n> -\t\t\t old_oid, new_oid,\n> -\t\t\t !is_null_oid(old_oid),\n> -\t\t\t !is_null_oid(new_oid),\n> -\t\t\t ce->name, 0, dirty_submodule);\n> -\n> +\t\tif (diff_change_helper(&revs->diffopt, newmode, dirty_submodule,\n> +\t\t\t\t changed, istate, ce))\n> +\t\t\tcontinue;\n> \t}\n> \tdiffcore_std(&revs->diffopt);\n> \tdiff_flush(&revs->diffopt);\n"},{"id":"471780","messageId":"66bf75bd-b14f-2658-5f6f-8a368c1d429b@dunelm.org.uk","threadId":"59034","inReplyTo":"230208.861qn01s4g.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH v7 6/7] diff-lib: refactor match_stat_with_submodule","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2023-02-08T17:07:44Z","receivedAt":"2023-02-08T17:07:50Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"On 08/02/2023 08:18, Ævar Arnfjörð Bjarmason wrote:\n> \n> On Tue, Feb 07 2023, Calvin Wan wrote:\n\n> Anyway, having looked at this closely I think this patch should be\n> dropped entirely. I don't understand how this refactoring is meant to\n> make the end result easier to read, reason about, or how it helps the\n> subsequent patch.\n\nThat's my feeling too c.f. \n<19f91fea-a2a9-7dc6-d940-cc10f384fe76@dunelm.org.uk>. This patch has \nimproved since that comment on v4 but I still think we'd be better off \nwithout it.\n\nBest Wishes\n\nPhillip\n\n\n> In addition to the above diff in 7/7 you do (and that's the change this\n> is meant to help):\n> \t\n> \t static int match_stat_with_submodule(struct diff_options *diffopt,\n> \t \t\t\t\t const struct cache_entry *ce,\n> \t \t\t\t\t struct stat *st, unsigned ce_option,\n> \t-\t\t\t\t unsigned *dirty_submodule)\n> \t+\t\t\t\t unsigned *dirty_submodule, int *defer_submodule_status,\n> \t+\t\t\t\t unsigned *ignore_untracked)\n> \t {\n> \t \tint changed = ie_match_stat(diffopt->repo->index, ce, st, ce_option);\n> \t \tstruct diff_flags orig_flags;\n> \t+\tint defer = 0;\n> \t\n> \t \tif (!S_ISGITLINK(ce->ce_mode))\n> \t-\t\treturn changed;\n> \t+\t\tgoto ret;\n> \t\n> \t \torig_flags = diffopt->flags;\n> \t \tif (!diffopt->flags.override_submodule_config)\n> \t@@ -86,11 +92,20 @@ static int match_stat_with_submodule(struct diff_options *diffopt,\n> \t \t\tgoto cleanup;\n> \t \t}\n> \t \tif (!diffopt->flags.ignore_dirty_submodules &&\n> \t-\t (!changed || diffopt->flags.dirty_submodules))\n> \t-\t\t*dirty_submodule = is_submodule_modified(ce->name,\n> \t+\t (!changed || diffopt->flags.dirty_submodules)) {\n> \t+\t\tif (defer_submodule_status && *defer_submodule_status) {\n> \t+\t\t\tdefer = 1;\n> \t+\t\t\t*ignore_untracked = diffopt->flags.ignore_untracked_in_submodules;\n> \t+\t\t} else {\n> \t+\t\t\t*dirty_submodule = is_submodule_modified(ce->name,\n> \t \t\t\t\t\t diffopt->flags.ignore_untracked_in_submodules);\n> \t+\t\t}\n> \t+\t}\n> \t cleanup:\n> \t \tdiffopt->flags = orig_flags;\n> \t+ret:\n> \t+\tif (defer_submodule_status)\n> \t+\t\t*defer_submodule_status = defer;\n> \t \treturn changed;\n> \t }\n> \n> But if I rebase out this 6/7 patch and solve the conflict for 7/7 it\n> becomes:\n> \t\n> \t@@ -65,14 +66,20 @@ static int check_removed(const struct index_state *istate, const struct cache_en\n> \t * Return 1 when changes are detected, 0 otherwise. If the DIRTY_SUBMODULES\n> \t * option is set, the caller does not only want to know if a submodule is\n> \t * modified at all but wants to know all the conditions that are met (new\n> \t- * commits, untracked content and/or modified content).\n> \t+ * commits, untracked content and/or modified content). If\n> \t+ * defer_submodule_status bit is set, dirty_submodule will be left to the\n> \t+ * caller to set. defer_submodule_status can also be set to 0 in this\n> \t+ * function if there is no need to check if the submodule is modified.\n> \t */\n> \t static int match_stat_with_submodule(struct diff_options *diffopt,\n> \t \t\t\t\t const struct cache_entry *ce,\n> \t \t\t\t\t struct stat *st, unsigned ce_option,\n> \t-\t\t\t\t unsigned *dirty_submodule)\n> \t+\t\t\t\t unsigned *dirty_submodule, int *defer_submodule_status,\n> \t+\t\t\t\t unsigned *ignore_untracked)\n> \t {\n> \t \tint changed = ie_match_stat(diffopt->repo->index, ce, st, ce_option);\n> \t+\tint defer = 0;\n> \t+\n> \t \tif (S_ISGITLINK(ce->ce_mode)) {\n> \t \t\tstruct diff_flags orig_flags = diffopt->flags;\n> \t \t\tif (!diffopt->flags.override_submodule_config)\n> \t@@ -80,11 +87,20 @@ static int match_stat_with_submodule(struct diff_options *diffopt,\n> \t \t\tif (diffopt->flags.ignore_submodules)\n> \t \t\t\tchanged = 0;\n> \t \t\telse if (!diffopt->flags.ignore_dirty_submodules &&\n> \t-\t\t\t (!changed || diffopt->flags.dirty_submodules))\n> \t-\t\t\t*dirty_submodule = is_submodule_modified(ce->name,\n> \t-\t\t\t\t\t\t\t\t diffopt->flags.ignore_untracked_in_submodules);\n> \t+\t\t\t (!changed || diffopt->flags.dirty_submodules)) {\n> \t+\t\t\tif (defer_submodule_status && *defer_submodule_status) {\n> \t+\t\t\t\tdefer = 1;\n> \t+\t\t\t\t*ignore_untracked = diffopt->flags.ignore_untracked_in_submodules;\n> \t+\t\t\t} else {\n> \t+\t\t\t\t*dirty_submodule = is_submodule_modified(ce->name,\n> \t+\t\t\t\t\t\t\t\t\t diffopt->flags.ignore_untracked_in_submodules);\n> \t+\t\t\t}\n> \t+\t\t}\n> \t \t\tdiffopt->flags = orig_flags;\n> \t \t}\n> \t+\n> \t+\tif (defer_submodule_status)\n> \t+\t\t*defer_submodule_status = defer;\n> \t \treturn changed;\n> \t }\n> \t\n> \n> I can see how there's some room for *a* refactoring to reduce the\n> subsequent diff, but not by mutch.\n> \n> But this commit didn't help at all. This whole \"goto ret\", and \"goto\n> cleanup\" is just working around the fact that you pulled \"orig_flags\"\n> out of the \"if\" scope. Normally the de-indentation would be worth it,\n> but here it's not. The control flow becomes more complex to reason about\n> as a result.\n> \n"},{"id":"471809","messageId":"CAFySSZCX=tySZf1xSd59GH0qS9PWtUC6ekLB1bWtZR3c09+ttw@mail.gmail.com","threadId":"59034","inReplyTo":"230207.86y1p914ci.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH v7 1/7] run-command: add duplicate_output_fn to run_processes_parallel_opts","fromName":"Calvin Wan","fromEmail":"calvinwan@google.com","sentAt":"2023-02-08T22:50:11Z","receivedAt":"2023-02-08T22:50:27Z","isPatch":true,"sender":{"key":"calvinwan@google.com","avatar":"https://avatars.githubusercontent.com/u/92547554?v=4"},"body":"> But would ou mind if this addition of yours were instead:\n>\n> if (opts->ungroup) {\n> if (opts->duplicate_output)\n> BUG(\"duplicate_output and ungroup are incompatible with each other\")\n> }\n\nI don't see why not -- will change.\n\n> > @@ -1645,14 +1648,21 @@ static void pp_buffer_stderr(struct parallel_processes *pp,\n> > for (size_t i = 0; i < opts->processes; i++) {\n> > if (pp->children[i].state == GIT_CP_WORKING &&\n> > pp->pfd[i].revents & (POLLIN | POLLHUP)) {\n> > - int n = strbuf_read_once(&pp->children[i].err,\n> > - pp->children[i].process.err, 0);\n> > + ssize_t n = strbuf_read_once(&pp->children[i].err,\n> > + pp->children[i].process.err, 0);\n>\n> This s/int/ssize_t/ change is a good on, but not mentioned in the commit\n> message. Maybe worth splitting out?\n\nI'll call this and the style change out in the commit message instead of\nsplitting it out.\n\n> And why is this thing that wants to prove to us that we're capturing the\n> output wanting to strip successive newlines?\n\nI added it as a sanity check originally, but you're right that this is\nunnecessary. Thanks for your comments on the other stylistic nits. I've\ngone ahead and fixed them all.\n"},{"id":"471810","messageId":"CAFySSZDeC-zc7wS7WXE+bAijVQkuin5GwLRnJ2ok7C9PRaH3+g@mail.gmail.com","threadId":"59034","inReplyTo":"cd528776-caa8-6372-0ca5-db0545947fba@dunelm.org.uk","subject":"Re: [PATCH v7 1/7] run-command: add duplicate_output_fn to run_processes_parallel_opts","fromName":"Calvin Wan","fromEmail":"calvinwan@google.com","sentAt":"2023-02-08T22:54:55Z","receivedAt":"2023-02-08T22:55:10Z","isPatch":true,"sender":{"key":"calvinwan@google.com","avatar":"https://avatars.githubusercontent.com/u/92547554?v=4"},"body":"> > + } else {\n> > + if (opts->duplicate_output)\n> > + opts->duplicate_output(&pp->children[i].err,\n> > + strlen(pp->children[i].err.buf) - n,\n>\n> Looking at how this is used in patch 7 I think it would be better to\n> pass a const char*, length pair rather than a struct strbuf*, offset pair.\n> i.e.\n> opts->duplicate_output(pp->children[i].err.buf +\n> pp->children[i].err.len - n, n, ...)\n>\n> That would make it clear that we do not expect duplicate_output() to\n> alter the buffer and would avoid the duplicate_output() having to add\n> the offset to the start of the buffer to find the new data.\n\nI don't think that would work since\npp->children[i].err.buf + pp->children[i].err.len - n\nwouldn't end up as a const char* unless I'm missing something?\n"},{"id":"471811","messageId":"CAFySSZD8OdTJjvdoT1nLaOq_Jjq=JF2rSSn_612JMXgD8+rycQ@mail.gmail.com","threadId":"59034","inReplyTo":"230207.86ttzx13xc.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH v7 2/7] submodule: strbuf variable rename","fromName":"Calvin Wan","fromEmail":"calvinwan@google.com","sentAt":"2023-02-08T22:59:57Z","receivedAt":"2023-02-08T23:00:12Z","isPatch":true,"sender":{"key":"calvinwan@google.com","avatar":"https://avatars.githubusercontent.com/u/92547554?v=4"},"body":"> I'll only add that we could also do this on top:\n>\n> diff --git a/submodule.c b/submodule.c\n> index c7c6bfb2e26..eeb940d96a0 100644\n> --- a/submodule.c\n> +++ b/submodule.c\n> @@ -1875,7 +1875,7 @@ unsigned is_submodule_modified(const char *path, int ignore_untracked)\n> struct child_process cp = CHILD_PROCESS_INIT;\n> struct strbuf buf = STRBUF_INIT;\n> FILE *fp;\n> - unsigned dirty_submodule = 0;\n> + unsigned dirty_submodule0 = 0;\n> const char *git_dir;\n> int ignore_cp_exit_code = 0;\n>\n> @@ -1908,10 +1908,11 @@ unsigned is_submodule_modified(const char *path, int ignore_untracked)\n> while (strbuf_getwholeline(&buf, fp, '\\n') != EOF) {\n> char *str = buf.buf;\n> const size_t len = buf.len;\n> + unsigned *dirty_submodule = &dirty_submodule0;\n>\n> /* regular untracked files */\n> if (str[0] == '?')\n> - dirty_submodule |= DIRTY_SUBMODULE_UNTRACKED;\n> + *dirty_submodule |= DIRTY_SUBMODULE_UNTRACKED;\n>\n> if (str[0] == 'u' ||\n> str[0] == '1' ||\n> @@ -1923,17 +1924,17 @@ unsigned is_submodule_modified(const char *path, int ignore_untracked)\n>\n> if (str[5] == 'S' && str[8] == 'U')\n> /* nested untracked file */\n> - dirty_submodule |= DIRTY_SUBMODULE_UNTRACKED;\n> + *dirty_submodule |= DIRTY_SUBMODULE_UNTRACKED;\n>\n> if (str[0] == 'u' ||\n> str[0] == '2' ||\n> memcmp(str + 5, \"S..U\", 4))\n> /* other change */\n> - dirty_submodule |= DIRTY_SUBMODULE_MODIFIED;\n> + *dirty_submodule |= DIRTY_SUBMODULE_MODIFIED;\n> }\n>\n> - if ((dirty_submodule & DIRTY_SUBMODULE_MODIFIED) &&\n> - ((dirty_submodule & DIRTY_SUBMODULE_UNTRACKED) ||\n> + if ((*dirty_submodule & DIRTY_SUBMODULE_MODIFIED) &&\n> + ((*dirty_submodule & DIRTY_SUBMODULE_UNTRACKED) ||\n> ignore_untracked)) {\n> /*\n> * We're not interested in any further information from\n> @@ -1949,7 +1950,7 @@ unsigned is_submodule_modified(const char *path, int ignore_untracked)\n> die(_(\"'git status --porcelain=2' failed in submodule %s\"), path);\n>\n> strbuf_release(&buf);\n> - return dirty_submodule;\n> + return dirty_submodule0;\n> }\n>\n> int submodule_uses_gitfile(const char *path)\n>\n> Which, if we're massaging this for a subsequent smaller diff we can do\n> to make only the comment adjustment part of this be a non-moved line.\n\nAh that's a neat little trick -- I'll save this one for the next time I do a\nrefactor like this :)\n"},{"id":"471812","messageId":"CAFySSZC31aT4zu=-y1vBBd_Z=KWWYn3-7yziLfVbeK9Foc9c3w@mail.gmail.com","threadId":"59034","inReplyTo":"4ae106ed-bfa8-2824-c0c7-6cde32dbe369@dunelm.org.uk","subject":"Re: [PATCH v7 5/7] diff-lib: refactor out diff_change logic","fromName":"Calvin Wan","fromEmail":"calvinwan@google.com","sentAt":"2023-02-08T23:12:20Z","receivedAt":"2023-02-08T23:12:36Z","isPatch":true,"sender":{"key":"calvinwan@google.com","avatar":"https://avatars.githubusercontent.com/u/92547554?v=4"},"body":"> I worry that having three integer parameters next to each other makes it\n> very easy to mix them up with out getting any errors from the compiler\n> because the types are all compatible. Could the last two be combined\n> into a flags argument? A similar issues occurs in\n> match_stat_with_submodule() in patch 7\n\nI'm not sure how much more I want to engineer a static helper function\nthat is only being called in one other place. I also don't understand what\nyou mean by combining the last two into paramters a flags argument.\n"},{"id":"471813","messageId":"CAFySSZC1N4Jq9-nSHnX+GqxxU190Eq=PDod1GdCFirEhvyhbVg@mail.gmail.com","threadId":"59034","inReplyTo":"66bf75bd-b14f-2658-5f6f-8a368c1d429b@dunelm.org.uk","subject":"Re: [PATCH v7 6/7] diff-lib: refactor match_stat_with_submodule","fromName":"Calvin Wan","fromEmail":"calvinwan@google.com","sentAt":"2023-02-08T23:13:41Z","receivedAt":"2023-02-08T23:14:02Z","isPatch":true,"sender":{"key":"calvinwan@google.com","avatar":"https://avatars.githubusercontent.com/u/92547554?v=4"},"body":"I agree that this patch should be dropped. Thanks for catching this.\n"},{"id":"471816","messageId":"20230209000212.1892457-1-calvinwan@google.com","threadId":"59034","inReplyTo":"20230207181706.363453-1-calvinwan@google.com","subject":"[PATCH v8 0/6] submodule: parallelize diff","fromName":"Calvin Wan","fromEmail":"calvinwan@google.com","sentAt":"2023-02-09T00:02:06Z","receivedAt":"2023-02-09T00:02:27Z","isPatch":true,"sender":{"key":"calvinwan@google.com","avatar":"https://avatars.githubusercontent.com/u/92547554?v=4"},"body":"Original cover letter for context:\nhttps://lore.kernel.org/git/20221011232604.839941-1-calvinwan@google.com/\n\nThis reroll contains stylistic changes suggested by Avar and Phillip,\nand includes a range-diff below.\n\nCalvin Wan (6):\n run-command: add duplicate_output_fn to run_processes_parallel_opts\n submodule: strbuf variable rename\n submodule: move status parsing into function\n submodule: refactor is_submodule_modified()\n diff-lib: refactor out diff_change logic\n diff-lib: parallelize run_diff_files for submodules\n\n Documentation/config/submodule.txt | 12 ++\n diff-lib.c | 133 +++++++++++----\n run-command.c | 16 +-\n run-command.h | 25 +++\n submodule.c | 266 ++++++++++++++++++++++++-----\n submodule.h | 9 +\n t/helper/test-run-command.c | 20 +++\n t/t0061-run-command.sh | 39 +++++\n t/t4027-diff-submodule.sh | 31 ++++\n t/t7506-status-submodule.sh | 25 +++\n 10 files changed, 497 insertions(+), 79 deletions(-)\n\nRange-diff against v7:\n1: 311b1abfbe ! 1: 5d51250c67 run-command: add duplicate_output_fn to run_processes_parallel_opts\n @@ run-command.c: static void pp_init(struct parallel_processes *pp,\n \tif (!opts->get_next_task)\n \t\tBUG(\"you need to specify a get_next_task function\");\n \n -+\tif (opts->duplicate_output && opts->ungroup)\n -+\t\tBUG(\"duplicate_output and ungroup are incompatible with each other\");\n ++\tif (opts->ungroup) {\n ++\t\tif (opts->duplicate_output)\n ++\t\t\tBUG(\"duplicate_output and ungroup are incompatible with each other\");\n ++\t}\n +\n \tCALLOC_ARRAY(pp->children, n);\n \tif (!opts->ungroup)\n @@ run-command.c: static void pp_buffer_stderr(struct parallel_processes *pp,\n +\t\t\t} else if (n < 0) {\n \t\t\t\tif (errno != EAGAIN)\n \t\t\t\t\tdie_errno(\"read\");\n -+\t\t\t} else {\n -+\t\t\t\tif (opts->duplicate_output)\n -+\t\t\t\t\topts->duplicate_output(&pp->children[i].err,\n -+\t\t\t\t\t strlen(pp->children[i].err.buf) - n,\n -+\t\t\t\t\t opts->data,\n -+\t\t\t\t\t pp->children[i].data);\n ++\t\t\t} else if (opts->duplicate_output) {\n ++\t\t\t\topts->duplicate_output(&pp->children[i].err,\n ++\t\t\t\t\tpp->children[i].err.len - n,\n ++\t\t\t\t\topts->data, pp->children[i].data);\n +\t\t\t}\n \t\t}\n \t}\n @@ run-command.h: typedef int (*start_failure_fn)(struct strbuf *out,\n + *\n + * This function is incompatible with \"ungroup\"\n + */\n -+typedef void (*duplicate_output_fn)(struct strbuf *out,\n -+\t\t\t\t size_t offset,\n -+\t\t\t\t void *pp_cb,\n -+\t\t\t\t void *pp_task_cb);\n ++typedef void (*duplicate_output_fn)(struct strbuf *out, size_t offset,\n ++\t\t\t\t void *pp_cb, void *pp_task_cb);\n +\n /**\n * This callback is called on every child process that finished processing.\n @@ run-command.h: struct run_process_parallel_opts\n \tstart_failure_fn start_failure;\n \n +\t/**\n -+\t * duplicate_output: See duplicate_output_fn() above. This should be\n -+\t * NULL unless process specific output is needed\n ++\t * duplicate_output: See duplicate_output_fn() above. Unless you need\n ++\t * to capture output from child processes, leave this as NULL.\n +\t */\n +\tduplicate_output_fn duplicate_output;\n +\n @@ t/helper/test-run-command.c: static int no_job(struct child_process *cp,\n +\t\t\tvoid *pp_task_cb UNUSED)\n +{\n +\tstruct string_list list = STRING_LIST_INIT_DUP;\n ++\tstruct string_list_item *item;\n +\n +\tstring_list_split(&list, out->buf + offset, '\\n', -1);\n -+\tfor (size_t i = 0; i < list.nr; i++) {\n -+\t\tif (strlen(list.items[i].string) > 0)\n -+\t\t\tfprintf(stderr, \"duplicate_output: %s\\n\", list.items[i].string);\n -+\t}\n ++\tfor_each_string_list_item(item, &list)\n ++\t\tfprintf(stderr, \"duplicate_output: %s\\n\", item->string);\n +\tstring_list_clear(&list, 0);\n +}\n +\n @@ t/t0061-run-command.sh: test_expect_success 'run_command runs in parallel with m\n +\ttest_must_be_empty out &&\n +\ttest 4 = $(grep -c \"duplicate_output: Hello\" err) &&\n +\ttest 4 = $(grep -c \"duplicate_output: World\" err) &&\n -+\tsed \"/duplicate_output/d\" err > err1 &&\n ++\tsed \"/duplicate_output/d\" err >err1 &&\n +\ttest_cmp expect err1\n +'\n +\n @@ t/t0061-run-command.sh: test_expect_success 'run_command runs in parallel with a\n +\ttest_must_be_empty out &&\n +\ttest 4 = $(grep -c \"duplicate_output: Hello\" err) &&\n +\ttest 4 = $(grep -c \"duplicate_output: World\" err) &&\n -+\tsed \"/duplicate_output/d\" err > err1 &&\n ++\tsed \"/duplicate_output/d\" err >err1 &&\n +\ttest_cmp expect err1\n +'\n +\n @@ t/t0061-run-command.sh: test_expect_success 'run_command runs in parallel with m\n +\ttest_must_be_empty out &&\n +\ttest 4 = $(grep -c \"duplicate_output: Hello\" err) &&\n +\ttest 4 = $(grep -c \"duplicate_output: World\" err) &&\n -+\tsed \"/duplicate_output/d\" err > err1 &&\n ++\tsed \"/duplicate_output/d\" err >err1 &&\n +\ttest_cmp expect err1\n +'\n +\n2: d00a18dd84 = 2: 6ded5b6788 submodule: strbuf variable rename\n3: dcda518922 = 3: 0c71cea8cd submodule: move status parsing into function\n4: c6fc5ba13b ! 4: 5c8cc93f9f submodule: refactor is_submodule_modified()\n @@ submodule.c: static int config_update_recurse_submodules = RECURSE_SUBMODULES_OF\n static int initialized_fetch_ref_tips;\n static struct oid_array ref_tips_before_fetch;\n static struct oid_array ref_tips_after_fetch;\n -+static const char *status_porcelain_start_error =\n -+\tN_(\"could not run 'git status --porcelain=2' in submodule %s\");\n -+static const char *status_porcelain_fail_error =\n -+\tN_(\"'git status --porcelain=2' failed in submodule %s\");\n ++#define STATUS_PORCELAIN_START_ERROR \\\n ++\tN_(\"could not run 'git status --porcelain=2' in submodule %s\")\n ++#define STATUS_PORCELAIN_FAIL_ERROR \\\n ++\tN_(\"'git status --porcelain=2' failed in submodule %s\")\n \n /*\n * Check if the .gitmodules file is unmerged. Parsing of the .gitmodules file\n @@ submodule.c: unsigned is_submodule_modified(const char *path, int ignore_untrack\n +\tprepare_status_porcelain(&cp, path, ignore_untracked);\n \tif (start_command(&cp))\n -\t\tdie(_(\"Could not run 'git status --porcelain=2' in submodule %s\"), path);\n -+\t\tdie(_(status_porcelain_start_error), path);\n ++\t\tdie(_(STATUS_PORCELAIN_START_ERROR), path);\n \n \tfp = xfdopen(cp.out, \"r\");\n \twhile (strbuf_getwholeline(&buf, fp, '\\n') != EOF) {\n @@ submodule.c: unsigned is_submodule_modified(const char *path, int ignore_untrack\n \n \tif (finish_command(&cp) && !ignore_cp_exit_code)\n -\t\tdie(_(\"'git status --porcelain=2' failed in submodule %s\"), path);\n -+\t\tdie(_(status_porcelain_fail_error), path);\n ++\t\tdie(_(STATUS_PORCELAIN_FAIL_ERROR), path);\n \n \tstrbuf_release(&buf);\n \treturn dirty_submodule;\n5: 1ea8eae9c9 = 5: 6c2b62abc8 diff-lib: refactor out diff_change logic\n6: 0d35fcc38d < -: ---------- diff-lib: refactor match_stat_with_submodule\n7: fd1eec974d ! 6: bb25dadbe5 diff-lib: parallelize run_diff_files for submodules\n @@ diff-lib.c: static int check_removed(const struct index_state *istate, const str\n +\t\t\t\t unsigned *ignore_untracked)\n {\n \tint changed = ie_match_stat(diffopt->repo->index, ce, st, ce_option);\n - \tstruct diff_flags orig_flags;\n +-\tif (S_ISGITLINK(ce->ce_mode)) {\n +-\t\tstruct diff_flags orig_flags = diffopt->flags;\n +-\t\tif (!diffopt->flags.override_submodule_config)\n +-\t\t\tset_diffopt_flags_from_submodule_config(diffopt, ce->name);\n +-\t\tif (diffopt->flags.ignore_submodules)\n +-\t\t\tchanged = 0;\n +-\t\telse if (!diffopt->flags.ignore_dirty_submodules &&\n +-\t\t\t (!changed || diffopt->flags.dirty_submodules))\n ++\tstruct diff_flags orig_flags;\n +\tint defer = 0;\n - \n - \tif (!S_ISGITLINK(ce->ce_mode))\n --\t\treturn changed;\n ++\n ++\tif (!S_ISGITLINK(ce->ce_mode))\n +\t\tgoto ret;\n - \n - \torig_flags = diffopt->flags;\n - \tif (!diffopt->flags.override_submodule_config)\n -@@ diff-lib.c: static int match_stat_with_submodule(struct diff_options *diffopt,\n - \t\tgoto cleanup;\n - \t}\n - \tif (!diffopt->flags.ignore_dirty_submodules &&\n --\t (!changed || diffopt->flags.dirty_submodules))\n --\t\t*dirty_submodule = is_submodule_modified(ce->name,\n ++\n ++\torig_flags = diffopt->flags;\n ++\tif (!diffopt->flags.override_submodule_config)\n ++\t\tset_diffopt_flags_from_submodule_config(diffopt, ce->name);\n ++\tif (diffopt->flags.ignore_submodules) {\n ++\t\tchanged = 0;\n ++\t\tgoto cleanup;\n ++\t}\n ++\tif (!diffopt->flags.ignore_dirty_submodules &&\n +\t (!changed || diffopt->flags.dirty_submodules)) {\n +\t\tif (defer_submodule_status && *defer_submodule_status) {\n +\t\t\tdefer = 1;\n +\t\t\t*ignore_untracked = diffopt->flags.ignore_untracked_in_submodules;\n +\t\t} else {\n -+\t\t\t*dirty_submodule = is_submodule_modified(ce->name,\n - \t\t\t\t\t diffopt->flags.ignore_untracked_in_submodules);\n + \t\t\t*dirty_submodule = is_submodule_modified(ce->name,\n +-\t\t\t\t\t\t\t\t diffopt->flags.ignore_untracked_in_submodules);\n +-\t\tdiffopt->flags = orig_flags;\n ++\t\t\t\t\t diffopt->flags.ignore_untracked_in_submodules);\n +\t\t}\n -+\t}\n - cleanup:\n - \tdiffopt->flags = orig_flags;\n + \t}\n ++cleanup:\n ++\tdiffopt->flags = orig_flags;\n +ret:\n +\tif (defer_submodule_status)\n +\t\t*defer_submodule_status = defer;\n @@ diff-lib.c: int run_diff_files(struct rev_info *revs, unsigned int option)\n \t\t\t\t changed, istate, ce))\n \t\t\tcontinue;\n \t}\n -+\tif (submodules.nr > 0) {\n -+\t\tint parallel_jobs;\n -+\t\tif (git_config_get_int(\"submodule.diffjobs\", ¶llel_jobs))\n ++\tif (submodules.nr) {\n ++\t\tunsigned long parallel_jobs;\n ++\t\tstruct string_list_item *item;\n ++\n ++\t\tif (git_config_get_ulong(\"submodule.diffjobs\", ¶llel_jobs))\n +\t\t\tparallel_jobs = 1;\n +\t\telse if (!parallel_jobs)\n +\t\t\tparallel_jobs = online_cpus();\n -+\t\telse if (parallel_jobs < 0)\n -+\t\t\tdie(_(\"submodule.diffjobs cannot be negative\"));\n +\n +\t\tif (get_submodules_status(&submodules, parallel_jobs))\n +\t\t\tdie(_(\"submodule status failed\"));\n -+\t\tfor (size_t i = 0; i < submodules.nr; i++) {\n -+\t\t\tstruct submodule_status_util *util = submodules.items[i].util;\n ++\t\tfor_each_string_list_item(item, &submodules) {\n ++\t\t\tstruct submodule_status_util *util = item->util;\n +\n +\t\t\tif (diff_change_helper(&revs->diffopt, util->newmode,\n +\t\t\t\t util->dirty_submodule, util->changed,\n @@ submodule.c: int submodule_touches_in_range(struct repository *r,\n +\tint result;\n +\n +\tstruct string_list *submodule_names;\n -+\n -+\t/* Pending statuses by OIDs */\n -+\tstruct status_task **oid_status_tasks;\n -+\tint oid_status_tasks_nr, oid_status_tasks_alloc;\n +};\n +\n struct submodule_parallel_fetch {\n @@ submodule.c: unsigned is_submodule_modified(const char *path, int ignore_untrack\n +\tstruct status_task *task = task_cb;\n +\n +\tsps->result = 1;\n -+\tstrbuf_addf(err,\n -+\t _(status_porcelain_start_error),\n -+\t task->path);\n ++\tstrbuf_addf(err, _(STATUS_PORCELAIN_START_ERROR), task->path);\n +\treturn 0;\n +}\n +\n @@ submodule.c: unsigned is_submodule_modified(const char *path, int ignore_untrack\n +\n +\tif (retvalue) {\n +\t\tsps->result = 1;\n -+\t\tstrbuf_addf(err,\n -+\t\t _(status_porcelain_fail_error),\n -+\t\t task->path);\n ++\t\tstrbuf_addf(err, _(STATUS_PORCELAIN_FAIL_ERROR), task->path);\n +\t}\n +\n +\tparse_status_porcelain_strbuf(&task->out,\n-- \n2.39.1.519.gcb327c4b5f-goog\n\n"},{"id":"471817","messageId":"20230209000212.1892457-2-calvinwan@google.com","threadId":"59034","inReplyTo":"20230207181706.363453-1-calvinwan@google.com","subject":"[PATCH v8 1/6] run-command: add duplicate_output_fn to run_processes_parallel_opts","fromName":"Calvin Wan","fromEmail":"calvinwan@google.com","sentAt":"2023-02-09T00:02:07Z","receivedAt":"2023-02-09T00:02:28Z","isPatch":true,"sender":{"key":"calvinwan@google.com","avatar":"https://avatars.githubusercontent.com/u/92547554?v=4"},"body":"Add duplicate_output_fn as an optionally set function in\nrun_process_parallel_opts. If set, output from each child process is\ncopied and passed to the callback function whenever output from the\nchild process is buffered to allow for separate parsing.\n\nFix two items in pp_buffer_stderr:\n * strbuf_read_once returns a ssize_t but the variable it is set to is\n an int so fix that.\n * Add missing brackets to \"else if\" statement\n\nThe ungroup/duplicate_output incompatibility check is nested to\nprepare for future imcompatibles modes with ungroup.\n\nSigned-off-by: Calvin Wan \n---\n run-command.c | 16 ++++++++++++---\n run-command.h | 25 ++++++++++++++++++++++++\n t/helper/test-run-command.c | 20 +++++++++++++++++++\n t/t0061-run-command.sh | 39 +++++++++++++++++++++++++++++++++++++\n 4 files changed, 97 insertions(+), 3 deletions(-)\n\ndiff --git a/run-command.c b/run-command.c\nindex 756f1839aa..50f741f2ab 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -1526,6 +1526,11 @@ static void pp_init(struct parallel_processes *pp,\n \tif (!opts->get_next_task)\n \t\tBUG(\"you need to specify a get_next_task function\");\n \n+\tif (opts->ungroup) {\n+\t\tif (opts->duplicate_output)\n+\t\t\tBUG(\"duplicate_output and ungroup are incompatible with each other\");\n+\t}\n+\n \tCALLOC_ARRAY(pp->children, n);\n \tif (!opts->ungroup)\n \t\tCALLOC_ARRAY(pp->pfd, n);\n@@ -1645,14 +1650,19 @@ static void pp_buffer_stderr(struct parallel_processes *pp,\n \tfor (size_t i = 0; i < opts->processes; i++) {\n \t\tif (pp->children[i].state == GIT_CP_WORKING &&\n \t\t pp->pfd[i].revents & (POLLIN | POLLHUP)) {\n-\t\t\tint n = strbuf_read_once(&pp->children[i].err,\n-\t\t\t\t\t\t pp->children[i].process.err, 0);\n+\t\t\tssize_t n = strbuf_read_once(&pp->children[i].err,\n+\t\t\t\t\t\t pp->children[i].process.err, 0);\n \t\t\tif (n == 0) {\n \t\t\t\tclose(pp->children[i].process.err);\n \t\t\t\tpp->children[i].state = GIT_CP_WAIT_CLEANUP;\n-\t\t\t} else if (n < 0)\n+\t\t\t} else if (n < 0) {\n \t\t\t\tif (errno != EAGAIN)\n \t\t\t\t\tdie_errno(\"read\");\n+\t\t\t} else if (opts->duplicate_output) {\n+\t\t\t\topts->duplicate_output(&pp->children[i].err,\n+\t\t\t\t\tpp->children[i].err.len - n,\n+\t\t\t\t\topts->data, pp->children[i].data);\n+\t\t\t}\n \t\t}\n \t}\n }\ndiff --git a/run-command.h b/run-command.h\nindex 072db56a4d..0c16d7f251 100644\n--- a/run-command.h\n+++ b/run-command.h\n@@ -408,6 +408,25 @@ typedef int (*start_failure_fn)(struct strbuf *out,\n \t\t\t\tvoid *pp_cb,\n \t\t\t\tvoid *pp_task_cb);\n \n+/**\n+ * This callback is called whenever output from a child process is buffered\n+ * \n+ * See run_processes_parallel() below for a discussion of the \"struct\n+ * strbuf *out\" parameter.\n+ * \n+ * The offset refers to the number of bytes originally in \"out\" before\n+ * the output from the child process was buffered. Therefore, the buffer\n+ * range, \"out + buf\" to the end of \"out\", would contain the buffer of\n+ * the child process output.\n+ *\n+ * pp_cb is the callback cookie as passed into run_processes_parallel,\n+ * pp_task_cb is the callback cookie as passed into get_next_task_fn.\n+ *\n+ * This function is incompatible with \"ungroup\"\n+ */\n+typedef void (*duplicate_output_fn)(struct strbuf *out, size_t offset,\n+\t\t\t\t void *pp_cb, void *pp_task_cb);\n+\n /**\n * This callback is called on every child process that finished processing.\n *\n@@ -461,6 +480,12 @@ struct run_process_parallel_opts\n \t */\n \tstart_failure_fn start_failure;\n \n+\t/**\n+\t * duplicate_output: See duplicate_output_fn() above. Unless you need\n+\t * to capture output from child processes, leave this as NULL.\n+\t */\n+\tduplicate_output_fn duplicate_output;\n+\n \t/**\n \t * task_finished: See task_finished_fn() above. This can be\n \t * NULL to omit any special handling.\ndiff --git a/t/helper/test-run-command.c b/t/helper/test-run-command.c\nindex 3ecb830f4a..4596ba68a8 100644\n--- a/t/helper/test-run-command.c\n+++ b/t/helper/test-run-command.c\n@@ -52,6 +52,20 @@ static int no_job(struct child_process *cp,\n \treturn 0;\n }\n \n+static void duplicate_output(struct strbuf *out,\n+\t\t\tsize_t offset,\n+\t\t\tvoid *pp_cb UNUSED,\n+\t\t\tvoid *pp_task_cb UNUSED)\n+{\n+\tstruct string_list list = STRING_LIST_INIT_DUP;\n+\tstruct string_list_item *item;\n+\n+\tstring_list_split(&list, out->buf + offset, '\\n', -1);\n+\tfor_each_string_list_item(item, &list)\n+\t\tfprintf(stderr, \"duplicate_output: %s\\n\", item->string);\n+\tstring_list_clear(&list, 0);\n+}\n+\n static int task_finished(int result,\n \t\t\t struct strbuf *err,\n \t\t\t void *pp_cb,\n@@ -439,6 +453,12 @@ int cmd__run_command(int argc, const char **argv)\n \t\topts.ungroup = 1;\n \t}\n \n+\tif (!strcmp(argv[1], \"--duplicate-output\")) {\n+\t\targv += 1;\n+\t\targc -= 1;\n+\t\topts.duplicate_output = duplicate_output;\n+\t}\n+\n \tjobs = atoi(argv[2]);\n \tstrvec_clear(&proc.args);\n \tstrvec_pushv(&proc.args, (const char **)argv + 3);\ndiff --git a/t/t0061-run-command.sh b/t/t0061-run-command.sh\nindex e2411f6a9b..31f1db96fc 100755\n--- a/t/t0061-run-command.sh\n+++ b/t/t0061-run-command.sh\n@@ -135,6 +135,15 @@ test_expect_success 'run_command runs in parallel with more jobs available than\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'run_command runs in parallel with more jobs available than tasks --duplicate-output' '\n+\ttest-tool run-command --duplicate-output run-command-parallel 5 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n+\ttest_must_be_empty out &&\n+\ttest 4 = $(grep -c \"duplicate_output: Hello\" err) &&\n+\ttest 4 = $(grep -c \"duplicate_output: World\" err) &&\n+\tsed \"/duplicate_output/d\" err >err1 &&\n+\ttest_cmp expect err1\n+'\n+\n test_expect_success 'run_command runs ungrouped in parallel with more jobs available than tasks' '\n \ttest-tool run-command --ungroup run-command-parallel 5 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n \ttest_line_count = 8 out &&\n@@ -147,6 +156,15 @@ test_expect_success 'run_command runs in parallel with as many jobs as tasks' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'run_command runs in parallel with as many jobs as tasks --duplicate-output' '\n+\ttest-tool run-command --duplicate-output run-command-parallel 4 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n+\ttest_must_be_empty out &&\n+\ttest 4 = $(grep -c \"duplicate_output: Hello\" err) &&\n+\ttest 4 = $(grep -c \"duplicate_output: World\" err) &&\n+\tsed \"/duplicate_output/d\" err >err1 &&\n+\ttest_cmp expect err1\n+'\n+\n test_expect_success 'run_command runs ungrouped in parallel with as many jobs as tasks' '\n \ttest-tool run-command --ungroup run-command-parallel 4 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n \ttest_line_count = 8 out &&\n@@ -159,6 +177,15 @@ test_expect_success 'run_command runs in parallel with more tasks than jobs avai\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'run_command runs in parallel with more tasks than jobs available --duplicate-output' '\n+\ttest-tool run-command --duplicate-output run-command-parallel 3 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n+\ttest_must_be_empty out &&\n+\ttest 4 = $(grep -c \"duplicate_output: Hello\" err) &&\n+\ttest 4 = $(grep -c \"duplicate_output: World\" err) &&\n+\tsed \"/duplicate_output/d\" err >err1 &&\n+\ttest_cmp expect err1\n+'\n+\n test_expect_success 'run_command runs ungrouped in parallel with more tasks than jobs available' '\n \ttest-tool run-command --ungroup run-command-parallel 3 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n \ttest_line_count = 8 out &&\n@@ -180,6 +207,12 @@ test_expect_success 'run_command is asked to abort gracefully' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'run_command is asked to abort gracefully --duplicate-output' '\n+\ttest-tool run-command --duplicate-output run-command-abort 3 false >out 2>err &&\n+\ttest_must_be_empty out &&\n+\ttest_cmp expect err\n+'\n+\n test_expect_success 'run_command is asked to abort gracefully (ungroup)' '\n \ttest-tool run-command --ungroup run-command-abort 3 false >out 2>err &&\n \ttest_must_be_empty out &&\n@@ -196,6 +229,12 @@ test_expect_success 'run_command outputs ' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'run_command outputs --duplicate-output' '\n+\ttest-tool run-command --duplicate-output run-command-no-jobs 3 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n+\ttest_must_be_empty out &&\n+\ttest_cmp expect err\n+'\n+\n test_expect_success 'run_command outputs (ungroup) ' '\n \ttest-tool run-command --ungroup run-command-no-jobs 3 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n \ttest_must_be_empty out &&\n-- \n2.39.1.519.gcb327c4b5f-goog\n\n"},{"id":"471818","messageId":"20230209000212.1892457-3-calvinwan@google.com","threadId":"59034","inReplyTo":"20230207181706.363453-1-calvinwan@google.com","subject":"[PATCH v8 2/6] submodule: strbuf variable rename","fromName":"Calvin Wan","fromEmail":"calvinwan@google.com","sentAt":"2023-02-09T00:02:08Z","receivedAt":"2023-02-09T00:02:30Z","isPatch":true,"sender":{"key":"calvinwan@google.com","avatar":"https://avatars.githubusercontent.com/u/92547554?v=4"},"body":"A prepatory change for a future patch that moves the status parsing\nlogic to a separate function.\n\nSigned-off-by: Calvin Wan \n---\n submodule.c | 23 +++++++++++++----------\n 1 file changed, 13 insertions(+), 10 deletions(-)\n\ndiff --git a/submodule.c b/submodule.c\nindex fae24ef34a..faf37c1101 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -1906,25 +1906,28 @@ unsigned is_submodule_modified(const char *path, int ignore_untracked)\n \n \tfp = xfdopen(cp.out, \"r\");\n \twhile (strbuf_getwholeline(&buf, fp, '\\n') != EOF) {\n+\t\tchar *str = buf.buf;\n+\t\tconst size_t len = buf.len;\n+\n \t\t/* regular untracked files */\n-\t\tif (buf.buf[0] == '?')\n+\t\tif (str[0] == '?')\n \t\t\tdirty_submodule |= DIRTY_SUBMODULE_UNTRACKED;\n \n-\t\tif (buf.buf[0] == 'u' ||\n-\t\t buf.buf[0] == '1' ||\n-\t\t buf.buf[0] == '2') {\n+\t\tif (str[0] == 'u' ||\n+\t\t str[0] == '1' ||\n+\t\t str[0] == '2') {\n \t\t\t/* T = line type, XY = status, SSSS = submodule state */\n-\t\t\tif (buf.len < strlen(\"T XY SSSS\"))\n+\t\t\tif (len < strlen(\"T XY SSSS\"))\n \t\t\t\tBUG(\"invalid status --porcelain=2 line %s\",\n-\t\t\t\t buf.buf);\n+\t\t\t\t str);\n \n-\t\t\tif (buf.buf[5] == 'S' && buf.buf[8] == 'U')\n+\t\t\tif (str[5] == 'S' && str[8] == 'U')\n \t\t\t\t/* nested untracked file */\n \t\t\t\tdirty_submodule |= DIRTY_SUBMODULE_UNTRACKED;\n \n-\t\t\tif (buf.buf[0] == 'u' ||\n-\t\t\t buf.buf[0] == '2' ||\n-\t\t\t memcmp(buf.buf + 5, \"S..U\", 4))\n+\t\t\tif (str[0] == 'u' ||\n+\t\t\t str[0] == '2' ||\n+\t\t\t memcmp(str + 5, \"S..U\", 4))\n \t\t\t\t/* other change */\n \t\t\t\tdirty_submodule |= DIRTY_SUBMODULE_MODIFIED;\n \t\t}\n-- \n2.39.1.519.gcb327c4b5f-goog\n\n"},{"id":"471819","messageId":"20230209000212.1892457-4-calvinwan@google.com","threadId":"59034","inReplyTo":"20230207181706.363453-1-calvinwan@google.com","subject":"[PATCH v8 3/6] submodule: move status parsing into function","fromName":"Calvin Wan","fromEmail":"calvinwan@google.com","sentAt":"2023-02-09T00:02:09Z","receivedAt":"2023-02-09T00:02:33Z","isPatch":true,"sender":{"key":"calvinwan@google.com","avatar":"https://avatars.githubusercontent.com/u/92547554?v=4"},"body":"A future patch requires the ability to parse the output of git\nstatus --porcelain=2. Move parsing code from is_submodule_modified to\nparse_status_porcelain.\n\nSigned-off-by: Calvin Wan \n---\n submodule.c | 74 ++++++++++++++++++++++++++++++-----------------------\n 1 file changed, 42 insertions(+), 32 deletions(-)\n\ndiff --git a/submodule.c b/submodule.c\nindex faf37c1101..768d4b4cd7 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -1870,6 +1870,45 @@ int fetch_submodules(struct repository *r,\n \treturn spf.result;\n }\n \n+static int parse_status_porcelain(char *str, size_t len,\n+\t\t\t\t unsigned *dirty_submodule,\n+\t\t\t\t int ignore_untracked)\n+{\n+\t/* regular untracked files */\n+\tif (str[0] == '?')\n+\t\t*dirty_submodule |= DIRTY_SUBMODULE_UNTRACKED;\n+\n+\tif (str[0] == 'u' ||\n+\t str[0] == '1' ||\n+\t str[0] == '2') {\n+\t\t/* T = line type, XY = status, SSSS = submodule state */\n+\t\tif (len < strlen(\"T XY SSSS\"))\n+\t\t\tBUG(\"invalid status --porcelain=2 line %s\",\n+\t\t\t str);\n+\n+\t\tif (str[5] == 'S' && str[8] == 'U')\n+\t\t\t/* nested untracked file */\n+\t\t\t*dirty_submodule |= DIRTY_SUBMODULE_UNTRACKED;\n+\n+\t\tif (str[0] == 'u' ||\n+\t\t str[0] == '2' ||\n+\t\t memcmp(str + 5, \"S..U\", 4))\n+\t\t\t/* other change */\n+\t\t\t*dirty_submodule |= DIRTY_SUBMODULE_MODIFIED;\n+\t}\n+\n+\tif ((*dirty_submodule & DIRTY_SUBMODULE_MODIFIED) &&\n+\t ((*dirty_submodule & DIRTY_SUBMODULE_UNTRACKED) ||\n+\t ignore_untracked)) {\n+\t\t/*\n+\t\t* We're not interested in any further information from\n+\t\t* the child any more, neither output nor its exit code.\n+\t\t*/\n+\t\treturn 1;\n+\t}\n+\treturn 0;\n+}\n+\n unsigned is_submodule_modified(const char *path, int ignore_untracked)\n {\n \tstruct child_process cp = CHILD_PROCESS_INIT;\n@@ -1909,39 +1948,10 @@ unsigned is_submodule_modified(const char *path, int ignore_untracked)\n \t\tchar *str = buf.buf;\n \t\tconst size_t len = buf.len;\n \n-\t\t/* regular untracked files */\n-\t\tif (str[0] == '?')\n-\t\t\tdirty_submodule |= DIRTY_SUBMODULE_UNTRACKED;\n-\n-\t\tif (str[0] == 'u' ||\n-\t\t str[0] == '1' ||\n-\t\t str[0] == '2') {\n-\t\t\t/* T = line type, XY = status, SSSS = submodule state */\n-\t\t\tif (len < strlen(\"T XY SSSS\"))\n-\t\t\t\tBUG(\"invalid status --porcelain=2 line %s\",\n-\t\t\t\t str);\n-\n-\t\t\tif (str[5] == 'S' && str[8] == 'U')\n-\t\t\t\t/* nested untracked file */\n-\t\t\t\tdirty_submodule |= DIRTY_SUBMODULE_UNTRACKED;\n-\n-\t\t\tif (str[0] == 'u' ||\n-\t\t\t str[0] == '2' ||\n-\t\t\t memcmp(str + 5, \"S..U\", 4))\n-\t\t\t\t/* other change */\n-\t\t\t\tdirty_submodule |= DIRTY_SUBMODULE_MODIFIED;\n-\t\t}\n-\n-\t\tif ((dirty_submodule & DIRTY_SUBMODULE_MODIFIED) &&\n-\t\t ((dirty_submodule & DIRTY_SUBMODULE_UNTRACKED) ||\n-\t\t ignore_untracked)) {\n-\t\t\t/*\n-\t\t\t * We're not interested in any further information from\n-\t\t\t * the child any more, neither output nor its exit code.\n-\t\t\t */\n-\t\t\tignore_cp_exit_code = 1;\n+\t\tignore_cp_exit_code = parse_status_porcelain(str, len, &dirty_submodule,\n+\t\t\t\t\t\t\t ignore_untracked);\n+\t\tif (ignore_cp_exit_code)\n \t\t\tbreak;\n-\t\t}\n \t}\n \tfclose(fp);\n \n-- \n2.39.1.519.gcb327c4b5f-goog\n\n"},{"id":"471820","messageId":"20230209000212.1892457-5-calvinwan@google.com","threadId":"59034","inReplyTo":"20230207181706.363453-1-calvinwan@google.com","subject":"[PATCH v8 4/6] submodule: refactor is_submodule_modified()","fromName":"Calvin Wan","fromEmail":"calvinwan@google.com","sentAt":"2023-02-09T00:02:10Z","receivedAt":"2023-02-09T00:02:34Z","isPatch":true,"sender":{"key":"calvinwan@google.com","avatar":"https://avatars.githubusercontent.com/u/92547554?v=4"},"body":"Refactor out submodule status logic and error messages that will be\nused in a future patch.\n\nSigned-off-by: Calvin Wan \n---\n submodule.c | 65 ++++++++++++++++++++++++++++++++++-------------------\n 1 file changed, 42 insertions(+), 23 deletions(-)\n\ndiff --git a/submodule.c b/submodule.c\nindex 768d4b4cd7..426074cebb 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -28,6 +28,10 @@ static int config_update_recurse_submodules = RECURSE_SUBMODULES_OFF;\n static int initialized_fetch_ref_tips;\n static struct oid_array ref_tips_before_fetch;\n static struct oid_array ref_tips_after_fetch;\n+#define STATUS_PORCELAIN_START_ERROR \\\n+\tN_(\"could not run 'git status --porcelain=2' in submodule %s\")\n+#define STATUS_PORCELAIN_FAIL_ERROR \\\n+\tN_(\"'git status --porcelain=2' failed in submodule %s\")\n \n /*\n * Check if the .gitmodules file is unmerged. Parsing of the .gitmodules file\n@@ -1870,6 +1874,40 @@ int fetch_submodules(struct repository *r,\n \treturn spf.result;\n }\n \n+static int verify_submodule_git_directory(const char *path)\n+{\n+\tconst char *git_dir;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\n+\tstrbuf_addf(&buf, \"%s/.git\", path);\n+\tgit_dir = read_gitfile(buf.buf);\n+\tif (!git_dir)\n+\t\tgit_dir = buf.buf;\n+\tif (!is_git_directory(git_dir)) {\n+\t\tif (is_directory(git_dir))\n+\t\t\tdie(_(\"'%s' not recognized as a git repository\"), git_dir);\n+\t\tstrbuf_release(&buf);\n+\t\t/* The submodule is not checked out, so it is not modified */\n+\t\treturn 0;\n+\t}\n+\tstrbuf_release(&buf);\n+\treturn 1;\n+}\n+\n+static void prepare_status_porcelain(struct child_process *cp,\n+\t\t\t const char *path, int ignore_untracked)\n+{\n+\tstrvec_pushl(&cp->args, \"status\", \"--porcelain=2\", NULL);\n+\tif (ignore_untracked)\n+\t\tstrvec_push(&cp->args, \"-uno\");\n+\n+\tprepare_submodule_repo_env(&cp->env);\n+\tcp->git_cmd = 1;\n+\tcp->no_stdin = 1;\n+\tcp->out = -1;\n+\tcp->dir = path;\n+}\n+\n static int parse_status_porcelain(char *str, size_t len,\n \t\t\t\t unsigned *dirty_submodule,\n \t\t\t\t int ignore_untracked)\n@@ -1915,33 +1953,14 @@ unsigned is_submodule_modified(const char *path, int ignore_untracked)\n \tstruct strbuf buf = STRBUF_INIT;\n \tFILE *fp;\n \tunsigned dirty_submodule = 0;\n-\tconst char *git_dir;\n \tint ignore_cp_exit_code = 0;\n \n-\tstrbuf_addf(&buf, \"%s/.git\", path);\n-\tgit_dir = read_gitfile(buf.buf);\n-\tif (!git_dir)\n-\t\tgit_dir = buf.buf;\n-\tif (!is_git_directory(git_dir)) {\n-\t\tif (is_directory(git_dir))\n-\t\t\tdie(_(\"'%s' not recognized as a git repository\"), git_dir);\n-\t\tstrbuf_release(&buf);\n-\t\t/* The submodule is not checked out, so it is not modified */\n+\tif (!verify_submodule_git_directory(path))\n \t\treturn 0;\n-\t}\n-\tstrbuf_reset(&buf);\n-\n-\tstrvec_pushl(&cp.args, \"status\", \"--porcelain=2\", NULL);\n-\tif (ignore_untracked)\n-\t\tstrvec_push(&cp.args, \"-uno\");\n \n-\tprepare_submodule_repo_env(&cp.env);\n-\tcp.git_cmd = 1;\n-\tcp.no_stdin = 1;\n-\tcp.out = -1;\n-\tcp.dir = path;\n+\tprepare_status_porcelain(&cp, path, ignore_untracked);\n \tif (start_command(&cp))\n-\t\tdie(_(\"Could not run 'git status --porcelain=2' in submodule %s\"), path);\n+\t\tdie(_(STATUS_PORCELAIN_START_ERROR), path);\n \n \tfp = xfdopen(cp.out, \"r\");\n \twhile (strbuf_getwholeline(&buf, fp, '\\n') != EOF) {\n@@ -1956,7 +1975,7 @@ unsigned is_submodule_modified(const char *path, int ignore_untracked)\n \tfclose(fp);\n \n \tif (finish_command(&cp) && !ignore_cp_exit_code)\n-\t\tdie(_(\"'git status --porcelain=2' failed in submodule %s\"), path);\n+\t\tdie(_(STATUS_PORCELAIN_FAIL_ERROR), path);\n \n \tstrbuf_release(&buf);\n \treturn dirty_submodule;\n-- \n2.39.1.519.gcb327c4b5f-goog\n\n"},{"id":"471821","messageId":"20230209000212.1892457-6-calvinwan@google.com","threadId":"59034","inReplyTo":"20230207181706.363453-1-calvinwan@google.com","subject":"[PATCH v8 5/6] diff-lib: refactor out diff_change logic","fromName":"Calvin Wan","fromEmail":"calvinwan@google.com","sentAt":"2023-02-09T00:02:11Z","receivedAt":"2023-02-09T00:02:35Z","isPatch":true,"sender":{"key":"calvinwan@google.com","avatar":"https://avatars.githubusercontent.com/u/92547554?v=4"},"body":"Refactor out logic that sets up the diff_change call into a helper\nfunction for a future patch.\n\nSigned-off-by: Calvin Wan \n---\n diff-lib.c | 46 +++++++++++++++++++++++++++++-----------------\n 1 file changed, 29 insertions(+), 17 deletions(-)\n\ndiff --git a/diff-lib.c b/diff-lib.c\nindex dec040c366..7101cfda3f 100644\n--- a/diff-lib.c\n+++ b/diff-lib.c\n@@ -88,6 +88,31 @@ static int match_stat_with_submodule(struct diff_options *diffopt,\n \treturn changed;\n }\n \n+static int diff_change_helper(struct diff_options *options,\n+\t unsigned newmode, unsigned dirty_submodule,\n+\t int changed, struct index_state *istate,\n+\t struct cache_entry *ce)\n+{\n+\tunsigned int oldmode;\n+\tconst struct object_id *old_oid, *new_oid;\n+\n+\tif (!changed && !dirty_submodule) {\n+\t\tce_mark_uptodate(ce);\n+\t\tmark_fsmonitor_valid(istate, ce);\n+\t\tif (!options->flags.find_copies_harder)\n+\t\t\treturn 1;\n+\t}\n+\toldmode = ce->ce_mode;\n+\told_oid = &ce->oid;\n+\tnew_oid = changed ? null_oid() : &ce->oid;\n+\tdiff_change(options, oldmode, newmode,\n+\t\t\told_oid, new_oid,\n+\t\t\t!is_null_oid(old_oid),\n+\t\t\t!is_null_oid(new_oid),\n+\t\t\tce->name, 0, dirty_submodule);\n+\treturn 0;\n+}\n+\n int run_diff_files(struct rev_info *revs, unsigned int option)\n {\n \tint entries, i;\n@@ -105,11 +130,10 @@ int run_diff_files(struct rev_info *revs, unsigned int option)\n \t\tdiff_unmerged_stage = 2;\n \tentries = istate->cache_nr;\n \tfor (i = 0; i < entries; i++) {\n-\t\tunsigned int oldmode, newmode;\n+\t\tunsigned int newmode;\n \t\tstruct cache_entry *ce = istate->cache[i];\n \t\tint changed;\n \t\tunsigned dirty_submodule = 0;\n-\t\tconst struct object_id *old_oid, *new_oid;\n \n \t\tif (diff_can_quit_early(&revs->diffopt))\n \t\t\tbreak;\n@@ -245,21 +269,9 @@ int run_diff_files(struct rev_info *revs, unsigned int option)\n \t\t\tnewmode = ce_mode_from_stat(ce, st.st_mode);\n \t\t}\n \n-\t\tif (!changed && !dirty_submodule) {\n-\t\t\tce_mark_uptodate(ce);\n-\t\t\tmark_fsmonitor_valid(istate, ce);\n-\t\t\tif (!revs->diffopt.flags.find_copies_harder)\n-\t\t\t\tcontinue;\n-\t\t}\n-\t\toldmode = ce->ce_mode;\n-\t\told_oid = &ce->oid;\n-\t\tnew_oid = changed ? null_oid() : &ce->oid;\n-\t\tdiff_change(&revs->diffopt, oldmode, newmode,\n-\t\t\t old_oid, new_oid,\n-\t\t\t !is_null_oid(old_oid),\n-\t\t\t !is_null_oid(new_oid),\n-\t\t\t ce->name, 0, dirty_submodule);\n-\n+\t\tif (diff_change_helper(&revs->diffopt, newmode, dirty_submodule,\n+\t\t\t\t changed, istate, ce))\n+\t\t\tcontinue;\n \t}\n \tdiffcore_std(&revs->diffopt);\n \tdiff_flush(&revs->diffopt);\n-- \n2.39.1.519.gcb327c4b5f-goog\n\n"},{"id":"471822","messageId":"20230209000212.1892457-7-calvinwan@google.com","threadId":"59034","inReplyTo":"20230207181706.363453-1-calvinwan@google.com","subject":"[PATCH v8 6/6] diff-lib: parallelize run_diff_files for submodules","fromName":"Calvin Wan","fromEmail":"calvinwan@google.com","sentAt":"2023-02-09T00:02:12Z","receivedAt":"2023-02-09T00:02:36Z","isPatch":true,"sender":{"key":"calvinwan@google.com","avatar":"https://avatars.githubusercontent.com/u/92547554?v=4"},"body":"During the iteration of the index entries in run_diff_files, whenever\na submodule is found and needs its status checked, a subprocess is\nspawned for it. Instead of spawning the subprocess immediately and\nwaiting for its completion to continue, hold onto all submodules and\nrelevant information in a list. Then use that list to create tasks for\nrun_processes_parallel. Subprocess output is duplicated and passed to\nstatus_pipe_output which stores it to be parsed on completion of the\nsubprocess.\n\nAdd config option submodule.diffJobs to set the maximum number\nof parallel jobs. The option defaults to 1 if unset. If set to 0, the\nnumber of jobs is set to online_cpus().\n\nSince run_diff_files is called from many different commands, I chose\nto grab the config option in the function rather than adding variables\nto every git command and then figuring out how to pass them all in.\n\nSigned-off-by: Calvin Wan \n---\n Documentation/config/submodule.txt | 12 +++\n diff-lib.c | 91 ++++++++++++++++---\n submodule.c | 140 +++++++++++++++++++++++++++++\n submodule.h | 9 ++\n t/t4027-diff-submodule.sh | 31 +++++++\n t/t7506-status-submodule.sh | 25 ++++++\n 6 files changed, 294 insertions(+), 14 deletions(-)\n\ndiff --git a/Documentation/config/submodule.txt b/Documentation/config/submodule.txt\nindex 6490527b45..3209eb8117 100644\n--- a/Documentation/config/submodule.txt\n+++ b/Documentation/config/submodule.txt\n@@ -93,6 +93,18 @@ submodule.fetchJobs::\n \tin parallel. A value of 0 will give some reasonable default.\n \tIf unset, it defaults to 1.\n \n+submodule.diffJobs::\n+\tSpecifies how many submodules are diffed at the same time. A\n+\tpositive integer allows up to that number of submodules diffed\n+\tin parallel. A value of 0 will give some reasonable default.\n+\tIf unset, it defaults to 1. The diff operation is used by many\n+\tother git commands such as add, merge, diff, status, stash and\n+\tmore. Note that the expensive part of the diff operation is\n+\treading the index from cache or memory. Therefore multiple jobs\n+\tmay be detrimental to performance if your hardware does not\n+\tsupport parallel reads or if the number of jobs greatly exceeds\n+\tthe amount of supported reads.\n+\n submodule.alternateLocation::\n \tSpecifies how the submodules obtain alternates when submodules are\n \tcloned. Possible values are `no`, `superproject`.\ndiff --git a/diff-lib.c b/diff-lib.c\nindex 7101cfda3f..2dde575ec6 100644\n--- a/diff-lib.c\n+++ b/diff-lib.c\n@@ -14,6 +14,7 @@\n #include \"dir.h\"\n #include \"fsmonitor.h\"\n #include \"commit-reach.h\"\n+#include \"config.h\"\n \n /*\n * diff-files\n@@ -65,26 +66,46 @@ static int check_removed(const struct index_state *istate, const struct cache_en\n * Return 1 when changes are detected, 0 otherwise. If the DIRTY_SUBMODULES\n * option is set, the caller does not only want to know if a submodule is\n * modified at all but wants to know all the conditions that are met (new\n- * commits, untracked content and/or modified content).\n+ * commits, untracked content and/or modified content). If\n+ * defer_submodule_status bit is set, dirty_submodule will be left to the\n+ * caller to set. defer_submodule_status can also be set to 0 in this\n+ * function if there is no need to check if the submodule is modified.\n */\n static int match_stat_with_submodule(struct diff_options *diffopt,\n \t\t\t\t const struct cache_entry *ce,\n \t\t\t\t struct stat *st, unsigned ce_option,\n-\t\t\t\t unsigned *dirty_submodule)\n+\t\t\t\t unsigned *dirty_submodule, int *defer_submodule_status,\n+\t\t\t\t unsigned *ignore_untracked)\n {\n \tint changed = ie_match_stat(diffopt->repo->index, ce, st, ce_option);\n-\tif (S_ISGITLINK(ce->ce_mode)) {\n-\t\tstruct diff_flags orig_flags = diffopt->flags;\n-\t\tif (!diffopt->flags.override_submodule_config)\n-\t\t\tset_diffopt_flags_from_submodule_config(diffopt, ce->name);\n-\t\tif (diffopt->flags.ignore_submodules)\n-\t\t\tchanged = 0;\n-\t\telse if (!diffopt->flags.ignore_dirty_submodules &&\n-\t\t\t (!changed || diffopt->flags.dirty_submodules))\n+\tstruct diff_flags orig_flags;\n+\tint defer = 0;\n+\n+\tif (!S_ISGITLINK(ce->ce_mode))\n+\t\tgoto ret;\n+\n+\torig_flags = diffopt->flags;\n+\tif (!diffopt->flags.override_submodule_config)\n+\t\tset_diffopt_flags_from_submodule_config(diffopt, ce->name);\n+\tif (diffopt->flags.ignore_submodules) {\n+\t\tchanged = 0;\n+\t\tgoto cleanup;\n+\t}\n+\tif (!diffopt->flags.ignore_dirty_submodules &&\n+\t (!changed || diffopt->flags.dirty_submodules)) {\n+\t\tif (defer_submodule_status && *defer_submodule_status) {\n+\t\t\tdefer = 1;\n+\t\t\t*ignore_untracked = diffopt->flags.ignore_untracked_in_submodules;\n+\t\t} else {\n \t\t\t*dirty_submodule = is_submodule_modified(ce->name,\n-\t\t\t\t\t\t\t\t diffopt->flags.ignore_untracked_in_submodules);\n-\t\tdiffopt->flags = orig_flags;\n+\t\t\t\t\t diffopt->flags.ignore_untracked_in_submodules);\n+\t\t}\n \t}\n+cleanup:\n+\tdiffopt->flags = orig_flags;\n+ret:\n+\tif (defer_submodule_status)\n+\t\t*defer_submodule_status = defer;\n \treturn changed;\n }\n \n@@ -121,6 +142,7 @@ int run_diff_files(struct rev_info *revs, unsigned int option)\n \t\t\t ? CE_MATCH_RACY_IS_DIRTY : 0);\n \tuint64_t start = getnanotime();\n \tstruct index_state *istate = revs->diffopt.repo->index;\n+\tstruct string_list submodules = STRING_LIST_INIT_NODUP;\n \n \tdiff_set_mnemonic_prefix(&revs->diffopt, \"i/\", \"w/\");\n \n@@ -244,6 +266,8 @@ int run_diff_files(struct rev_info *revs, unsigned int option)\n \t\t\tnewmode = ce->ce_mode;\n \t\t} else {\n \t\t\tstruct stat st;\n+\t\t\tunsigned ignore_untracked = 0;\n+\t\t\tint defer_submodule_status = 1;\n \n \t\t\tchanged = check_removed(istate, ce, &st);\n \t\t\tif (changed) {\n@@ -265,14 +289,53 @@ int run_diff_files(struct rev_info *revs, unsigned int option)\n \t\t\t}\n \n \t\t\tchanged = match_stat_with_submodule(&revs->diffopt, ce, &st,\n-\t\t\t\t\t\t\t ce_option, &dirty_submodule);\n+\t\t\t\t\t\t\t ce_option, &dirty_submodule,\n+\t\t\t\t\t\t\t &defer_submodule_status,\n+\t\t\t\t\t\t\t &ignore_untracked);\n \t\t\tnewmode = ce_mode_from_stat(ce, st.st_mode);\n+\t\t\tif (defer_submodule_status) {\n+\t\t\t\tstruct submodule_status_util tmp = {\n+\t\t\t\t\t.changed = changed,\n+\t\t\t\t\t.dirty_submodule = 0,\n+\t\t\t\t\t.ignore_untracked = ignore_untracked,\n+\t\t\t\t\t.newmode = newmode,\n+\t\t\t\t\t.ce = ce,\n+\t\t\t\t\t.path = ce->name,\n+\t\t\t\t};\n+\t\t\t\tstruct string_list_item *item;\n+\n+\t\t\t\titem = string_list_append(&submodules, ce->name);\n+\t\t\t\titem->util = xmalloc(sizeof(tmp));\n+\t\t\t\tmemcpy(item->util, &tmp, sizeof(tmp));\n+\t\t\t\tcontinue;\n+\t\t\t}\n \t\t}\n \n \t\tif (diff_change_helper(&revs->diffopt, newmode, dirty_submodule,\n \t\t\t\t changed, istate, ce))\n \t\t\tcontinue;\n \t}\n+\tif (submodules.nr) {\n+\t\tunsigned long parallel_jobs;\n+\t\tstruct string_list_item *item;\n+\n+\t\tif (git_config_get_ulong(\"submodule.diffjobs\", ¶llel_jobs))\n+\t\t\tparallel_jobs = 1;\n+\t\telse if (!parallel_jobs)\n+\t\t\tparallel_jobs = online_cpus();\n+\n+\t\tif (get_submodules_status(&submodules, parallel_jobs))\n+\t\t\tdie(_(\"submodule status failed\"));\n+\t\tfor_each_string_list_item(item, &submodules) {\n+\t\t\tstruct submodule_status_util *util = item->util;\n+\n+\t\t\tif (diff_change_helper(&revs->diffopt, util->newmode,\n+\t\t\t\t util->dirty_submodule, util->changed,\n+\t\t\t\t istate, util->ce))\n+\t\t\t\tcontinue;\n+\t\t}\n+\t}\n+\tstring_list_clear(&submodules, 1);\n \tdiffcore_std(&revs->diffopt);\n \tdiff_flush(&revs->diffopt);\n \ttrace_performance_since(start, \"diff-files\");\n@@ -320,7 +383,7 @@ static int get_stat_data(const struct index_state *istate,\n \t\t\treturn -1;\n \t\t}\n \t\tchanged = match_stat_with_submodule(diffopt, ce, &st,\n-\t\t\t\t\t\t 0, dirty_submodule);\n+\t\t\t\t\t\t 0, dirty_submodule, NULL, NULL);\n \t\tif (changed) {\n \t\t\tmode = ce_mode_from_stat(ce, st.st_mode);\n \t\t\toid = null_oid();\ndiff --git a/submodule.c b/submodule.c\nindex 426074cebb..e175fb8d3f 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -1373,6 +1373,13 @@ int submodule_touches_in_range(struct repository *r,\n \treturn ret;\n }\n \n+struct submodule_parallel_status {\n+\tsize_t index_count;\n+\tint result;\n+\n+\tstruct string_list *submodule_names;\n+};\n+\n struct submodule_parallel_fetch {\n \t/*\n \t * The index of the last index entry processed by\n@@ -1455,6 +1462,12 @@ struct fetch_task {\n \tstruct oid_array *commits; /* Ensure these commits are fetched */\n };\n \n+struct status_task {\n+\tconst char *path;\n+\tstruct strbuf out;\n+\tint ignore_untracked;\n+};\n+\n /**\n * When a submodule is not defined in .gitmodules, we cannot access it\n * via the regular submodule-config. Create a fake submodule, which we can\n@@ -1947,6 +1960,25 @@ static int parse_status_porcelain(char *str, size_t len,\n \treturn 0;\n }\n \n+static void parse_status_porcelain_strbuf(struct strbuf *buf,\n+\t\t\t\t unsigned *dirty_submodule,\n+\t\t\t\t int ignore_untracked)\n+{\n+\tstruct string_list list = STRING_LIST_INIT_DUP;\n+\tstruct string_list_item *item;\n+\n+\tstring_list_split(&list, buf->buf, '\\n', -1);\n+\n+\tfor_each_string_list_item(item, &list) {\n+\t\tif (parse_status_porcelain(item->string,\n+\t\t\t\t\t strlen(item->string),\n+\t\t\t\t\t dirty_submodule,\n+\t\t\t\t\t ignore_untracked))\n+\t\t\tbreak;\n+\t}\n+\tstring_list_clear(&list, 0);\n+}\n+\n unsigned is_submodule_modified(const char *path, int ignore_untracked)\n {\n \tstruct child_process cp = CHILD_PROCESS_INIT;\n@@ -1981,6 +2013,114 @@ unsigned is_submodule_modified(const char *path, int ignore_untracked)\n \treturn dirty_submodule;\n }\n \n+static struct status_task *\n+get_status_task_from_index(struct submodule_parallel_status *sps,\n+\t\t\t struct strbuf *err)\n+{\n+\tfor (; sps->index_count < sps->submodule_names->nr; sps->index_count++) {\n+\t\tstruct submodule_status_util *util = sps->submodule_names->items[sps->index_count].util;\n+\t\tstruct status_task *task;\n+\n+\t\tif (!verify_submodule_git_directory(util->path))\n+\t\t\tcontinue;\n+\n+\t\ttask = xmalloc(sizeof(*task));\n+\t\ttask->path = util->path;\n+\t\ttask->ignore_untracked = util->ignore_untracked;\n+\t\tstrbuf_init(&task->out, 0);\n+\t\tsps->index_count++;\n+\t\treturn task;\n+\t}\n+\treturn NULL;\n+}\n+\n+static int get_next_submodule_status(struct child_process *cp,\n+\t\t\t\t struct strbuf *err, void *data,\n+\t\t\t\t void **task_cb)\n+{\n+\tstruct submodule_parallel_status *sps = data;\n+\tstruct status_task *task = get_status_task_from_index(sps, err);\n+\n+\tif (!task)\n+\t\treturn 0;\n+\n+\tchild_process_init(cp);\n+\tprepare_submodule_repo_env_in_gitdir(&cp->env);\n+\tprepare_status_porcelain(cp, task->path, task->ignore_untracked);\n+\t*task_cb = task;\n+\treturn 1;\n+}\n+\n+static int status_start_failure(struct strbuf *err,\n+\t\t\t\tvoid *cb, void *task_cb)\n+{\n+\tstruct submodule_parallel_status *sps = cb;\n+\tstruct status_task *task = task_cb;\n+\n+\tsps->result = 1;\n+\tstrbuf_addf(err, _(STATUS_PORCELAIN_START_ERROR), task->path);\n+\treturn 0;\n+}\n+\n+static void status_duplicate_output(struct strbuf *out,\n+\t\t\t\t size_t offset,\n+\t\t\t\t void *cb, void *task_cb)\n+{\n+\tstruct status_task *task = task_cb;\n+\n+\tstrbuf_add(&task->out, out->buf + offset, out->len - offset);\n+\tstrbuf_setlen(out, offset);\n+}\n+\n+static int status_finish(int retvalue, struct strbuf *err,\n+\t\t\t void *cb, void *task_cb)\n+{\n+\tstruct submodule_parallel_status *sps = cb;\n+\tstruct status_task *task = task_cb;\n+\tstruct string_list_item *it =\n+\t\tstring_list_lookup(sps->submodule_names, task->path);\n+\tstruct submodule_status_util *util = it->util;\n+\n+\tif (retvalue) {\n+\t\tsps->result = 1;\n+\t\tstrbuf_addf(err, _(STATUS_PORCELAIN_FAIL_ERROR), task->path);\n+\t}\n+\n+\tparse_status_porcelain_strbuf(&task->out,\n+\t\t\t &util->dirty_submodule,\n+\t\t\t util->ignore_untracked);\n+\n+\tstrbuf_release(&task->out);\n+\tfree(task);\n+\n+\treturn 0;\n+}\n+\n+int get_submodules_status(struct string_list *submodules,\n+\t\t\t int max_parallel_jobs)\n+{\n+\tstruct submodule_parallel_status sps = {\n+\t\t.submodule_names = submodules,\n+\t};\n+\tconst struct run_process_parallel_opts opts = {\n+\t\t.tr2_category = \"submodule\",\n+\t\t.tr2_label = \"parallel/status\",\n+\n+\t\t.processes = max_parallel_jobs,\n+\n+\t\t.get_next_task = get_next_submodule_status,\n+\t\t.start_failure = status_start_failure,\n+\t\t.duplicate_output = status_duplicate_output,\n+\t\t.task_finished = status_finish,\n+\t\t.data = &sps,\n+\t};\n+\n+\tstring_list_sort(sps.submodule_names);\n+\trun_processes_parallel(&opts);\n+\n+\treturn sps.result;\n+}\n+\n int submodule_uses_gitfile(const char *path)\n {\n \tstruct child_process cp = CHILD_PROCESS_INIT;\ndiff --git a/submodule.h b/submodule.h\nindex b52a4ff1e7..08d278a414 100644\n--- a/submodule.h\n+++ b/submodule.h\n@@ -41,6 +41,13 @@ struct submodule_update_strategy {\n \t.type = SM_UPDATE_UNSPECIFIED, \\\n }\n \n+struct submodule_status_util {\n+\tint changed, ignore_untracked;\n+\tunsigned dirty_submodule, newmode;\n+\tstruct cache_entry *ce;\n+\tconst char *path;\n+};\n+\n int is_gitmodules_unmerged(struct index_state *istate);\n int is_writing_gitmodules_ok(void);\n int is_staging_gitmodules_ok(struct index_state *istate);\n@@ -94,6 +101,8 @@ int fetch_submodules(struct repository *r,\n \t\t int command_line_option,\n \t\t int default_option,\n \t\t int quiet, int max_parallel_jobs);\n+int get_submodules_status(struct string_list *submodules,\n+\t\t\t int max_parallel_jobs);\n unsigned is_submodule_modified(const char *path, int ignore_untracked);\n int submodule_uses_gitfile(const char *path);\n \ndiff --git a/t/t4027-diff-submodule.sh b/t/t4027-diff-submodule.sh\nindex 40164ae07d..1c747cc325 100755\n--- a/t/t4027-diff-submodule.sh\n+++ b/t/t4027-diff-submodule.sh\n@@ -34,6 +34,25 @@ test_expect_success setup '\n \tsubtip=$3 subprev=$2\n '\n \n+test_expect_success 'diff in superproject with submodules respects parallel settings' '\n+\ttest_when_finished \"rm -f trace.out\" &&\n+\t(\n+\t\tGIT_TRACE=$(pwd)/trace.out git diff &&\n+\t\tgrep \"1 tasks\" trace.out &&\n+\t\t>trace.out &&\n+\n+\t\tgit config submodule.diffJobs 8 &&\n+\t\tGIT_TRACE=$(pwd)/trace.out git diff &&\n+\t\tgrep \"8 tasks\" trace.out &&\n+\t\t>trace.out &&\n+\n+\t\tGIT_TRACE=$(pwd)/trace.out git -c submodule.diffJobs=0 diff &&\n+\t\tgrep \"preparing to run up to [0-9]* tasks\" trace.out &&\n+\t\t! grep \"up to 0 tasks\" trace.out &&\n+\t\t>trace.out\n+\t)\n+'\n+\n test_expect_success 'git diff --raw HEAD' '\n \thexsz=$(test_oid hexsz) &&\n \tgit diff --raw --abbrev=$hexsz HEAD >actual &&\n@@ -70,6 +89,18 @@ test_expect_success 'git diff HEAD with dirty submodule (work tree)' '\n \ttest_cmp expect.body actual.body\n '\n \n+test_expect_success 'git diff HEAD with dirty submodule (work tree, parallel)' '\n+\t(\n+\t\tcd sub &&\n+\t\tgit reset --hard &&\n+\t\techo >>world\n+\t) &&\n+\tgit -c submodule.diffJobs=8 diff HEAD >actual &&\n+\tsed -e \"1,/^@@/d\" actual >actual.body &&\n+\texpect_from_to >expect.body $subtip $subprev-dirty &&\n+\ttest_cmp expect.body actual.body\n+'\n+\n test_expect_success 'git diff HEAD with dirty submodule (index)' '\n \t(\n \t\tcd sub &&\ndiff --git a/t/t7506-status-submodule.sh b/t/t7506-status-submodule.sh\nindex d050091345..7da64e4c4c 100755\n--- a/t/t7506-status-submodule.sh\n+++ b/t/t7506-status-submodule.sh\n@@ -412,4 +412,29 @@ test_expect_success 'status with added file in nested submodule (short)' '\n \tEOF\n '\n \n+test_expect_success 'status in superproject with submodules respects parallel settings' '\n+\ttest_when_finished \"rm -f trace.out\" &&\n+\t(\n+\t\tGIT_TRACE=$(pwd)/trace.out git status &&\n+\t\tgrep \"1 tasks\" trace.out &&\n+\t\t>trace.out &&\n+\n+\t\tgit config submodule.diffJobs 8 &&\n+\t\tGIT_TRACE=$(pwd)/trace.out git status &&\n+\t\tgrep \"8 tasks\" trace.out &&\n+\t\t>trace.out &&\n+\n+\t\tGIT_TRACE=$(pwd)/trace.out git -c submodule.diffJobs=0 status &&\n+\t\tgrep \"preparing to run up to [0-9]* tasks\" trace.out &&\n+\t\t! grep \"up to 0 tasks\" trace.out &&\n+\t\t>trace.out\n+\t)\n+'\n+\n+test_expect_success 'status in superproject with submodules (parallel)' '\n+\tgit -C super status --porcelain >output &&\n+\tgit -C super -c submodule.diffJobs=8 status --porcelain >output_parallel &&\n+\tdiff output output_parallel\n+'\n+\n test_done\n-- \n2.39.1.519.gcb327c4b5f-goog\n\n"},{"id":"471828","messageId":"230209.867cwrzk1l.gmgdl@evledraar.gmail.com","threadId":"59034","inReplyTo":"20230209000212.1892457-1-calvinwan@google.com","subject":"Re: [PATCH v8 0/6] submodule: parallelize diff","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2023-02-09T01:42:15Z","receivedAt":"2023-02-09T01:47:56Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, Feb 09 2023, Calvin Wan wrote:\n\n> 6: 0d35fcc38d < -: ---------- diff-lib: refactor match_stat_with_submodule\n> 7: fd1eec974d ! 6: bb25dadbe5 diff-lib: parallelize run_diff_files for submodules\n> @@ diff-lib.c: static int check_removed(const struct index_state *istate, const str\n> +\t\t\t\t unsigned *ignore_untracked)\n> {\n> \tint changed = ie_match_stat(diffopt->repo->index, ce, st, ce_option);\n> - \tstruct diff_flags orig_flags;\n> +-\tif (S_ISGITLINK(ce->ce_mode)) {\n> +-\t\tstruct diff_flags orig_flags = diffopt->flags;\n> +-\t\tif (!diffopt->flags.override_submodule_config)\n> +-\t\t\tset_diffopt_flags_from_submodule_config(diffopt, ce->name);\n> +-\t\tif (diffopt->flags.ignore_submodules)\n> +-\t\t\tchanged = 0;\n> +-\t\telse if (!diffopt->flags.ignore_dirty_submodules &&\n> +-\t\t\t (!changed || diffopt->flags.dirty_submodules))\n> ++\tstruct diff_flags orig_flags;\n> +\tint defer = 0;\n> - \n> - \tif (!S_ISGITLINK(ce->ce_mode))\n> --\t\treturn changed;\n> ++\n> ++\tif (!S_ISGITLINK(ce->ce_mode))\n> +\t\tgoto ret;\n> - \n> - \torig_flags = diffopt->flags;\n> - \tif (!diffopt->flags.override_submodule_config)\n> -@@ diff-lib.c: static int match_stat_with_submodule(struct diff_options *diffopt,\n> - \t\tgoto cleanup;\n> - \t}\n> - \tif (!diffopt->flags.ignore_dirty_submodules &&\n> --\t (!changed || diffopt->flags.dirty_submodules))\n> --\t\t*dirty_submodule = is_submodule_modified(ce->name,\n> ++\n> ++\torig_flags = diffopt->flags;\n> ++\tif (!diffopt->flags.override_submodule_config)\n> ++\t\tset_diffopt_flags_from_submodule_config(diffopt, ce->name);\n> ++\tif (diffopt->flags.ignore_submodules) {\n> ++\t\tchanged = 0;\n> ++\t\tgoto cleanup;\n> ++\t}\n> ++\tif (!diffopt->flags.ignore_dirty_submodules &&\n> +\t (!changed || diffopt->flags.dirty_submodules)) {\n> +\t\tif (defer_submodule_status && *defer_submodule_status) {\n> +\t\t\tdefer = 1;\n> +\t\t\t*ignore_untracked = diffopt->flags.ignore_untracked_in_submodules;\n> +\t\t} else {\n> -+\t\t\t*dirty_submodule = is_submodule_modified(ce->name,\n> - \t\t\t\t\t diffopt->flags.ignore_untracked_in_submodules);\n> + \t\t\t*dirty_submodule = is_submodule_modified(ce->name,\n> +-\t\t\t\t\t\t\t\t diffopt->flags.ignore_untracked_in_submodules);\n> +-\t\tdiffopt->flags = orig_flags;\n> ++\t\t\t\t\t diffopt->flags.ignore_untracked_in_submodules);\n> +\t\t}\n> -+\t}\n> - cleanup:\n> - \tdiffopt->flags = orig_flags;\n> + \t}\n> ++cleanup:\n> ++\tdiffopt->flags = orig_flags;\n> +ret:\n> +\tif (defer_submodule_status)\n> +\t\t*defer_submodule_status = defer;\n> @@ diff-lib.c: int run_diff_files(struct rev_info *revs, unsigned int option)\n> \t\t\t\t changed, istate, ce))\n\nI think you dropped the 7/8 per my suggestion in [1]. I think this 6/6\nis actually worse than the v6.\n\nI.e. it seems you dropped the previous refactoring commit by squashing\nthe refactoring+functional change together.\n\nWhat I was pointing out in [1] was that you don't need the refactoring,\nand that both the change itself and the end-state is much easier to look\nat and reason about as a result\n\nI.e. I think the diff in your 6/6 should just be what's after \"it\nbecomes\" in [1] (maybe with some pre-refactoring, e.g. we could add the\nbraces first or whatever).\n\nBut in case you strongly prefer the current end-state I think having\nyour previous refactoring prep would be better, because it would at\nleast split off some of the refactoring & functional change.\n\nI haven't looked as deeply at this v8 as v7 for the rest, but from\nskimming the range-diff it all looked good otherwise.\n\n1. https://lore.kernel.org/git/230208.861qn01s4g.gmgdl@evledraar.gmail.com/\n"},{"id":"471829","messageId":"230209.86357fzjt5.gmgdl@evledraar.gmail.com","threadId":"59034","inReplyTo":"20230209000212.1892457-6-calvinwan@google.com","subject":"Re: [PATCH v8 5/6] diff-lib: refactor out diff_change logic","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2023-02-09T01:48:42Z","receivedAt":"2023-02-09T01:53:03Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, Feb 09 2023, Calvin Wan wrote:\n\n> +\tdiff_change(options, oldmode, newmode,\n> +\t\t\told_oid, new_oid,\n> +\t\t\t!is_null_oid(old_oid),\n> +\t\t\t!is_null_oid(new_oid),\n> +\t\t\tce->name, 0, dirty_submodule);\n\nNit: This has odd not-our-usual-style indentation (to align with the\n\"(\"). I didn't spot it before, but I vaguely recall seeing something\nlike this in another one of your patches, but maybe I misrecall. In case\nnot maybe some editor settings need tweaking?\n\nI haven't looked carefully at the rest to see if the same issue occurs\nin other code here.\n\n> -\t\tif (!changed && !dirty_submodule) {\n> -\t\t\tce_mark_uptodate(ce);\n> -\t\t\tmark_fsmonitor_valid(istate, ce);\n> -\t\t\tif (!revs->diffopt.flags.find_copies_harder)\n> -\t\t\t\tcontinue;\n> -\t\t}\n> -\t\toldmode = ce->ce_mode;\n> -\t\told_oid = &ce->oid;\n> -\t\tnew_oid = changed ? null_oid() : &ce->oid;\n> -\t\tdiff_change(&revs->diffopt, oldmode, newmode,\n> -\t\t\t old_oid, new_oid,\n> -\t\t\t !is_null_oid(old_oid),\n> -\t\t\t !is_null_oid(new_oid),\n> -\t\t\t ce->name, 0, dirty_submodule);\n\nSo in this case it's not new code, but code moving, note the four spaces\nafter the sequence of tabs that aren't in your version.\n\nSo perhaps your editor on re-indentation is configured not to just strip\noff the leading \\t to re-indent (which is all that's needed here) but\nstrips all whitespace, then re-indents after its own mind?\n"},{"id":"471873","messageId":"xmqqwn4qmxds.fsf@gitster.g","threadId":"59034","inReplyTo":"20230209000212.1892457-1-calvinwan@google.com","subject":"Re: [PATCH v8 0/6] submodule: parallelize diff","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-02-09T19:50:23Z","receivedAt":"2023-02-09T19:50:27Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Calvin Wan writes:\n\n> Original cover letter for context:\n> https://lore.kernel.org/git/20221011232604.839941-1-calvinwan@google.com/\n\nThanks. I'll try to take a look at this today.\n\nBy the way, how are you driving send-email when sending a\nmulti-patch series with a cover letter? It seems that all\nmessages in this series including its cover are marked as if they\nare replies to the cover letter of the previous round, which is a\nbit harder to follow than making only [v8 0/6] as a reply to [v7 0/X]\nand all [v8 n/6] (n > 0) to be replies to [v8 0/6].\n"},{"id":"471875","messageId":"a86cda2c-71be-a19a-f269-92d095b2428d@dunelm.org.uk","threadId":"59034","inReplyTo":"CAFySSZDeC-zc7wS7WXE+bAijVQkuin5GwLRnJ2ok7C9PRaH3+g@mail.gmail.com","subject":"Re: [PATCH v7 1/7] run-command: add duplicate_output_fn to run_processes_parallel_opts","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2023-02-09T20:37:16Z","receivedAt":"2023-02-09T20:37:25Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"Hi Calvin\n\nOn 08/02/2023 22:54, Calvin Wan wrote:\n>>> + } else {\n>>> + if (opts->duplicate_output)\n>>> + opts->duplicate_output(&pp->children[i].err,\n>>> + strlen(pp->children[i].err.buf) - n,\n>>\n>> Looking at how this is used in patch 7 I think it would be better to\n>> pass a const char*, length pair rather than a struct strbuf*, offset pair.\n>> i.e.\n>> opts->duplicate_output(pp->children[i].err.buf +\n>> pp->children[i].err.len - n, n, ...)\n>>\n>> That would make it clear that we do not expect duplicate_output() to\n>> alter the buffer and would avoid the duplicate_output() having to add\n>> the offset to the start of the buffer to find the new data.\n> \n> I don't think that would work since\n> pp->children[i].err.buf + pp->children[i].err.len - n\n> wouldn't end up as a const char* unless I'm missing something?\n\nYou can still pass it to a function that takes a const char* though and \nchange type of the callback to\n\ntypedef void (*duplicate_output_fn)(const char *out, size_t offset,\n\t\t\t\t void *pp_cb, void *pp_task_cb);\n\nBest Wishes\n\nPhillip\n"},{"id":"471876","messageId":"0e1858f1-8772-30eb-b6c9-5b0854d082a5@dunelm.org.uk","threadId":"59034","inReplyTo":"20230209000212.1892457-1-calvinwan@google.com","subject":"Re: [PATCH v8 0/6] submodule: parallelize diff","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2023-02-09T20:50:37Z","receivedAt":"2023-02-09T20:51:44Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"Hi Calvin\n\nOn 09/02/2023 00:02, Calvin Wan wrote:\n> Original cover letter for context:\n> https://lore.kernel.org/git/20221011232604.839941-1-calvinwan@google.com/\n> \n> This reroll contains stylistic changes suggested by Avar and Phillip,\n> and includes a range-diff below.\n> \n> Calvin Wan (6):\n> run-command: add duplicate_output_fn to run_processes_parallel_opts\n> submodule: strbuf variable rename\n> submodule: move status parsing into function\n> submodule: refactor is_submodule_modified()\n> diff-lib: refactor out diff_change logic\n> diff-lib: parallelize run_diff_files for submodules\n> \n> Documentation/config/submodule.txt | 12 ++\n> diff-lib.c | 133 +++++++++++----\n> run-command.c | 16 +-\n> run-command.h | 25 +++\n> submodule.c | 266 ++++++++++++++++++++++++-----\n> submodule.h | 9 +\n> t/helper/test-run-command.c | 20 +++\n> t/t0061-run-command.sh | 39 +++++\n> t/t4027-diff-submodule.sh | 31 ++++\n> t/t7506-status-submodule.sh | 25 +++\n> 10 files changed, 497 insertions(+), 79 deletions(-)\n> \n> Range-diff against v7:\n> 6: 0d35fcc38d < -: ---------- diff-lib: refactor match_stat_with_submodule\n> 7: fd1eec974d ! 6: bb25dadbe5 diff-lib: parallelize run_diff_files for submodules\n> @@ diff-lib.c: static int check_removed(const struct index_state *istate, const str\n> +\t\t\t\t unsigned *ignore_untracked)\n> {\n> \tint changed = ie_match_stat(diffopt->repo->index, ce, st, ce_option);\n> - \tstruct diff_flags orig_flags;\n> +-\tif (S_ISGITLINK(ce->ce_mode)) {\n> +-\t\tstruct diff_flags orig_flags = diffopt->flags;\n> +-\t\tif (!diffopt->flags.override_submodule_config)\n> +-\t\t\tset_diffopt_flags_from_submodule_config(diffopt, ce->name);\n> +-\t\tif (diffopt->flags.ignore_submodules)\n> +-\t\t\tchanged = 0;\n> +-\t\telse if (!diffopt->flags.ignore_dirty_submodules &&\n> +-\t\t\t (!changed || diffopt->flags.dirty_submodules))\n> ++\tstruct diff_flags orig_flags;\n> +\tint defer = 0;\n> -\n> - \tif (!S_ISGITLINK(ce->ce_mode))\n> --\t\treturn changed;\n> ++\n> ++\tif (!S_ISGITLINK(ce->ce_mode))\n> +\t\tgoto ret;\n> -\n> - \torig_flags = diffopt->flags;\n> - \tif (!diffopt->flags.override_submodule_config)\n> -@@ diff-lib.c: static int match_stat_with_submodule(struct diff_options *diffopt,\n> - \t\tgoto cleanup;\n> - \t}\n> - \tif (!diffopt->flags.ignore_dirty_submodules &&\n> --\t (!changed || diffopt->flags.dirty_submodules))\n> --\t\t*dirty_submodule = is_submodule_modified(ce->name,\n> ++\n> ++\torig_flags = diffopt->flags;\n> ++\tif (!diffopt->flags.override_submodule_config)\n> ++\t\tset_diffopt_flags_from_submodule_config(diffopt, ce->name);\n> ++\tif (diffopt->flags.ignore_submodules) {\n> ++\t\tchanged = 0;\n> ++\t\tgoto cleanup;\n> ++\t}\n> ++\tif (!diffopt->flags.ignore_dirty_submodules &&\n> +\t (!changed || diffopt->flags.dirty_submodules)) {\n> +\t\tif (defer_submodule_status && *defer_submodule_status) {\n> +\t\t\tdefer = 1;\n> +\t\t\t*ignore_untracked = diffopt->flags.ignore_untracked_in_submodules;\n> +\t\t} else {\n> -+\t\t\t*dirty_submodule = is_submodule_modified(ce->name,\n> - \t\t\t\t\t diffopt->flags.ignore_untracked_in_submodules);\n> + \t\t\t*dirty_submodule = is_submodule_modified(ce->name,\n> +-\t\t\t\t\t\t\t\t diffopt->flags.ignore_untracked_in_submodules);\n> +-\t\tdiffopt->flags = orig_flags;\n> ++\t\t\t\t\t diffopt->flags.ignore_untracked_in_submodules);\n> +\t\t}\n> -+\t}\n> - cleanup:\n> - \tdiffopt->flags = orig_flags;\n> + \t}\n> ++cleanup:\n> ++\tdiffopt->flags = orig_flags;\n> +ret:\n> +\tif (defer_submodule_status)\n\nThe idea behind the suggestion to drop the previous patch from the last \nversion was to stop refactoring the if block and get away from having \nthese labels. Can't you just add the \"if (defer_submodule_status && \n...)\" into the existing code?\n\nBest Wishes\n\nPhillip\n\n> +\t\t*defer_submodule_status = defer;\n> @@ diff-lib.c: int run_diff_files(struct rev_info *revs, unsigned int option)\n> \t\t\t\t changed, istate, ce))\n> \t\t\tcontinue;\n> \t}\n> -+\tif (submodules.nr > 0) {\n> -+\t\tint parallel_jobs;\n> -+\t\tif (git_config_get_int(\"submodule.diffjobs\", ¶llel_jobs))\n> ++\tif (submodules.nr) {\n> ++\t\tunsigned long parallel_jobs;\n> ++\t\tstruct string_list_item *item;\n> ++\n> ++\t\tif (git_config_get_ulong(\"submodule.diffjobs\", ¶llel_jobs))\n> +\t\t\tparallel_jobs = 1;\n> +\t\telse if (!parallel_jobs)\n> +\t\t\tparallel_jobs = online_cpus();\n> -+\t\telse if (parallel_jobs < 0)\n> -+\t\t\tdie(_(\"submodule.diffjobs cannot be negative\"));\n> +\n> +\t\tif (get_submodules_status(&submodules, parallel_jobs))\n> +\t\t\tdie(_(\"submodule status failed\"));\n> -+\t\tfor (size_t i = 0; i < submodules.nr; i++) {\n> -+\t\t\tstruct submodule_status_util *util = submodules.items[i].util;\n> ++\t\tfor_each_string_list_item(item, &submodules) {\n> ++\t\t\tstruct submodule_status_util *util = item->util;\n> +\n> +\t\t\tif (diff_change_helper(&revs->diffopt, util->newmode,\n> +\t\t\t\t util->dirty_submodule, util->changed,\n> @@ submodule.c: int submodule_touches_in_range(struct repository *r,\n> +\tint result;\n> +\n> +\tstruct string_list *submodule_names;\n> -+\n> -+\t/* Pending statuses by OIDs */\n> -+\tstruct status_task **oid_status_tasks;\n> -+\tint oid_status_tasks_nr, oid_status_tasks_alloc;\n> +};\n> +\n> struct submodule_parallel_fetch {\n> @@ submodule.c: unsigned is_submodule_modified(const char *path, int ignore_untrack\n> +\tstruct status_task *task = task_cb;\n> +\n> +\tsps->result = 1;\n> -+\tstrbuf_addf(err,\n> -+\t _(status_porcelain_start_error),\n> -+\t task->path);\n> ++\tstrbuf_addf(err, _(STATUS_PORCELAIN_START_ERROR), task->path);\n> +\treturn 0;\n> +}\n> +\n> @@ submodule.c: unsigned is_submodule_modified(const char *path, int ignore_untrack\n> +\n> +\tif (retvalue) {\n> +\t\tsps->result = 1;\n> -+\t\tstrbuf_addf(err,\n> -+\t\t _(status_porcelain_fail_error),\n> -+\t\t task->path);\n> ++\t\tstrbuf_addf(err, _(STATUS_PORCELAIN_FAIL_ERROR), task->path);\n> +\t}\n> +\n> +\tparse_status_porcelain_strbuf(&task->out,\n"},{"id":"471877","messageId":"4a09db46-5c45-c953-d3ff-383499ab0e21@dunelm.org.uk","threadId":"59034","inReplyTo":"CAFySSZC31aT4zu=-y1vBBd_Z=KWWYn3-7yziLfVbeK9Foc9c3w@mail.gmail.com","subject":"Re: [PATCH v7 5/7] diff-lib: refactor out diff_change logic","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2023-02-09T20:53:16Z","receivedAt":"2023-02-09T20:53:22Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"Hi Calvin\n\nOn 08/02/2023 23:12, Calvin Wan wrote:\n>> I worry that having three integer parameters next to each other makes it\n>> very easy to mix them up with out getting any errors from the compiler\n>> because the types are all compatible. Could the last two be combined\n>> into a flags argument? A similar issues occurs in\n>> match_stat_with_submodule() in patch 7\n> \n> I'm not sure how much more I want to engineer a static helper function\n> that is only being called in one other place. I also don't understand what\n> you mean by combining the last two into paramters a flags argument.\n\nAre `dirty_submodule` and `changed` booleans? If so then you can have a \nsingle bit flags argument made up of\n\n#define SUBMODULE_DIRTY 1\n#define SUBMODULE_CHANGED 2\n\nBest Wishes\n\nPhillip\n"},{"id":"471882","messageId":"CAFySSZAy=k+hgvg9r2NmroUs=BviZjS_KMW_+yO2nPs1dUsc-A@mail.gmail.com","threadId":"59034","inReplyTo":"xmqqwn4qmxds.fsf@gitster.g","subject":"Re: [PATCH v8 0/6] submodule: parallelize diff","fromName":"Calvin Wan","fromEmail":"calvinwan@google.com","sentAt":"2023-02-09T21:52:31Z","receivedAt":"2023-02-09T21:52:46Z","isPatch":true,"sender":{"key":"calvinwan@google.com","avatar":"https://avatars.githubusercontent.com/u/92547554?v=4"},"body":"> By the way, how are you driving send-email when sending a\n> multi-patch series with a cover letter? It seems that all\n> messages in this series including its cover are marked as if they\n> are replies to the cover letter of the previous round, which is a\n> bit harder to follow than making only [v8 0/6] as a reply to [v7 0/X]\n> and all [v8 n/6] (n > 0) to be replies to [v8 0/6].\n\nI'll do that from now on -- didn't realize that make it harder to follow\n"},{"id":"471884","messageId":"xmqqttzulbmq.fsf@gitster.g","threadId":"59034","inReplyTo":"CAFySSZAy=k+hgvg9r2NmroUs=BviZjS_KMW_+yO2nPs1dUsc-A@mail.gmail.com","subject":"Re: [PATCH v8 0/6] submodule: parallelize diff","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-02-09T22:25:33Z","receivedAt":"2023-02-09T22:25:37Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Calvin Wan writes:\n\n>> By the way, how are you driving send-email when sending a\n>> multi-patch series with a cover letter? It seems that all\n>> messages in this series including its cover are marked as if they\n>> are replies to the cover letter of the previous round, which is a\n>> bit harder to follow than making only [v8 0/6] as a reply to [v7 0/X]\n>> and all [v8 n/6] (n > 0) to be replies to [v8 0/6].\n>\n> I'll do that from now on -- didn't realize that make it harder to follow\n\n\"a bit harder\" may even have been an exaggeration. It is just being\ndifferent from how other topics by many other people are formatted.\n\nThanks.\n"},{"id":"471919","messageId":"230210.86sffdveio.gmgdl@evledraar.gmail.com","threadId":"59034","inReplyTo":"CAFySSZAy=k+hgvg9r2NmroUs=BviZjS_KMW_+yO2nPs1dUsc-A@mail.gmail.com","subject":"Re: [PATCH v8 0/6] submodule: parallelize diff","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2023-02-10T13:24:33Z","receivedAt":"2023-02-10T13:25:25Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, Feb 09 2023, Calvin Wan wrote:\n\n>> By the way, how are you driving send-email when sending a\n>> multi-patch series with a cover letter? It seems that all\n>> messages in this series including its cover are marked as if they\n>> are replies to the cover letter of the previous round, which is a\n>> bit harder to follow than making only [v8 0/6] as a reply to [v7 0/X]\n>> and all [v8 n/6] (n > 0) to be replies to [v8 0/6].\n>\n> I'll do that from now on -- didn't realize that make it harder to follow\n\nWelcome to the club :)\n\nThis came up before when I'd been sending mails like this for years,\nwithout realizing the difference:\nhttps://lore.kernel.org/git/nycvar.QRO.7.76.6.2103191540330.57@tvgsbejvaqbjf.bet/\n& https://lore.kernel.org/git/xmqqr1k9k2w7.fsf@gitster.g/\n"},{"id":"471932","messageId":"xmqq7cwpju1p.fsf@gitster.g","threadId":"59034","inReplyTo":"230210.86sffdveio.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH v8 0/6] submodule: parallelize diff","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-02-10T17:42:58Z","receivedAt":"2023-02-10T17:43:03Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ævar Arnfjörð Bjarmason writes:\n\n> Welcome to the club :)\n>\n> This came up before when I'd been sending mails like this for years,\n> without realizing the difference:\n> https://lore.kernel.org/git/nycvar.QRO.7.76.6.2103191540330.57@tvgsbejvaqbjf.bet/\n> & https://lore.kernel.org/git/xmqqr1k9k2w7.fsf@gitster.g/\n\nThe organization makes it easier to identify the cover letter,\nmechanically from the thread structure without relying on the\nsubject line [*], and that is one of the things that the procedure\nto prepare the \"What's cooking\" report needs to do.\n\n\tSide note: When the \"What's cooking\" report is updated, it\n\tknows individual commits on a topic, and the message ID of\n\tthe patch for each of these commits (they are recorded in\n\trefs/notes/amlog). But the message ID of the cover letter\n\tis not recorded anywhere because it does not become any\n\tcommit, so it looks at these messages to find references\n\tand/or in-reply-to. A flat \"everything including cover is\n\treply to previous cover\" organization would not help to find\n\tthe cover of this iteration at all.\n\nIn Documentation/SubmittingPatches, we give unnecessary either-or\nrecommendation. We should clearly spell it out instead, perhaps\nsomething like this.\n\n Documentation/SubmittingPatches | 5 +++--\n 1 file changed, 3 insertions(+), 2 deletions(-)\n\ndiff --git c/Documentation/SubmittingPatches w/Documentation/SubmittingPatches\nindex 927f7329a5..af7f2a4045 100644\n--- c/Documentation/SubmittingPatches\n+++ w/Documentation/SubmittingPatches\n@@ -346,8 +346,9 @@ your code. For this reason, each patch should be submitted\n \n Multiple related patches should be grouped into their own e-mail\n thread to help readers find all parts of the series. To that end,\n-send them as replies to either an additional \"cover letter\" message\n-(see below), the first patch, or the respective preceding patch.\n+send them as replies to an additional \"cover letter\" message\n+(see below), which should be a reply to the \"cover letter\" of\n+the previous iteration.\n \n If your log message (including your name on the\n `Signed-off-by` trailer) is not writable in ASCII, make sure that\n"},{"id":"471999","messageId":"kl6lo7pyax9q.fsf@chooglen-macbookpro.roam.corp.google.com","threadId":"59034","inReplyTo":"20230209000212.1892457-2-calvinwan@google.com","subject":"Re: [PATCH v8 1/6] run-command: add duplicate_output_fn to run_processes_parallel_opts","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2023-02-13T06:34:57Z","receivedAt":"2023-02-13T06:35:02Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"Calvin Wan writes:\n\n> @@ -1645,14 +1650,19 @@ static void pp_buffer_stderr(struct parallel_processes *pp,\n> \tfor (size_t i = 0; i < opts->processes; i++) {\n> \t\tif (pp->children[i].state == GIT_CP_WORKING &&\n> \t\t pp->pfd[i].revents & (POLLIN | POLLHUP)) {\n> -\t\t\tint n = strbuf_read_once(&pp->children[i].err,\n> -\t\t\t\t\t\t pp->children[i].process.err, 0);\n> +\t\t\tssize_t n = strbuf_read_once(&pp->children[i].err,\n> +\t\t\t\t\t\t pp->children[i].process.err, 0);\n> \t\t\tif (n == 0) {\n> \t\t\t\tclose(pp->children[i].process.err);\n> \t\t\t\tpp->children[i].state = GIT_CP_WAIT_CLEANUP;\n> -\t\t\t} else if (n < 0)\n> +\t\t\t} else if (n < 0) {\n> \t\t\t\tif (errno != EAGAIN)\n> \t\t\t\t\tdie_errno(\"read\");\n> +\t\t\t} else if (opts->duplicate_output) {\n> +\t\t\t\topts->duplicate_output(&pp->children[i].err,\n> +\t\t\t\t\tpp->children[i].err.len - n,\n> +\t\t\t\t\topts->data, pp->children[i].data);\n> +\t\t\t}\n> \t\t}\n> \t}\n> }\n\nWhat do we think of the name \"duplicate_output\"? IMO it made sense in\nearlier versions when we were copying the output to a separate buffer (I\nbelieve it was renamed in response to [1]), but now that we're just\ncalling a callback on the main buffer, it seems misleading. Maybe\n\"output_buffered\" would be better?\n\nSidenote: One convention from JS that I like is to name such event\nlisteners as \"on_\", e.g. \"on_output_buffered\". This makes\nnaming a lot easier sometimes because you don't have to worry about\nhaving your event listener being mistaken for something else. It\nwouldn't be idiomatic for Git today, but I wonder what others think\nabout adopting this.\n\n[1] https://lore.kernel.org/git/xmqq4jvxpw46.fsf@gitster.g/\n\n> +/**\n> + * This callback is called whenever output from a child process is buffered\n> + * \n> + * See run_processes_parallel() below for a discussion of the \"struct\n> + * strbuf *out\" parameter.\n> + * \n> + * The offset refers to the number of bytes originally in \"out\" before\n> + * the output from the child process was buffered. Therefore, the buffer\n> + * range, \"out + buf\" to the end of \"out\", would contain the buffer of\n> + * the child process output.\n\nLooks like there's extra whitespace on the 'blank' lines.\n\n"},{"id":"472001","messageId":"kl6lk00mavsu.fsf@chooglen-macbookpro.roam.corp.google.com","threadId":"59034","inReplyTo":"20230209000212.1892457-5-calvinwan@google.com","subject":"Re: [PATCH v8 4/6] submodule: refactor is_submodule_modified()","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2023-02-13T07:06:41Z","receivedAt":"2023-02-13T07:06:46Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"Calvin Wan writes:\n\n> Refactor out submodule status logic and error messages that will be\n> used in a future patch.\n\nThis improves the readability of the last patch by quite a lot. Thanks\nfor taking the suggestion :)\n\n(This patch was actually introduced in the previous round, but I missed\nthat, sorry.)\n"},{"id":"472002","messageId":"kl6lh6vqarnh.fsf@chooglen-macbookpro.roam.corp.google.com","threadId":"59034","inReplyTo":"20230209000212.1892457-7-calvinwan@google.com","subject":"Re: [PATCH v8 6/6] diff-lib: parallelize run_diff_files for submodules","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2023-02-13T08:36:18Z","receivedAt":"2023-02-13T08:36:28Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"Calvin Wan writes:\n\n> @@ -244,6 +266,8 @@ int run_diff_files(struct rev_info *revs, unsigned int option)\n> \t\t\tnewmode = ce->ce_mode;\n> \t\t} else {\n> \t\t\tstruct stat st;\n> +\t\t\tunsigned ignore_untracked = 0;\n> +\t\t\tint defer_submodule_status = 1;\n> \n> \t\t\tchanged = check_removed(istate, ce, &st);\n> \t\t\tif (changed) {\n\nPreviously [1] it wasn't entirely clear whether we intended to always\nparallelize submodule diffing, but now it seems that we always try to\nparallelize. In essence, this means that we don't have a serial\nimplementation any more, but maybe that's okay.\n\n[1] https://lore.kernel.org/git/kl6lilgtveoe.fsf@chooglen-macbookpro.roam.corp.google.com/\n\n> @@ -265,14 +289,53 @@ int run_diff_files(struct rev_info *revs, unsigned int option)\n> \t\t\t}\n> \n> \t\t\tchanged = match_stat_with_submodule(&revs->diffopt, ce, &st,\n> -\t\t\t\t\t\t\t ce_option, &dirty_submodule);\n> +\t\t\t\t\t\t\t ce_option, &dirty_submodule,\n> +\t\t\t\t\t\t\t &defer_submodule_status,\n> +\t\t\t\t\t\t\t &ignore_untracked);\n\nHere we get the 'changed' bit of the submodule. Because we always defer,\nwe never call is_submodule_modified() inside\nmatch_stat_with_submodule() and as such, we never set \"dirty_submodule\"\nhere. If so, could we remove the variable altogether?\n\n> \t\t\tnewmode = ce_mode_from_stat(ce, st.st_mode);\n> +\t\t\tif (defer_submodule_status) {\n> +\t\t\t\tstruct submodule_status_util tmp = {\n> +\t\t\t\t\t.changed = changed,\n> +\t\t\t\t\t.dirty_submodule = 0,\n> +\t\t\t\t\t.ignore_untracked = ignore_untracked,\n> +\t\t\t\t\t.newmode = newmode,\n> +\t\t\t\t\t.ce = ce,\n> +\t\t\t\t\t.path = ce->name,\n> +\t\t\t\t};\n> +\t\t\t\tstruct string_list_item *item;\n> +\n> +\t\t\t\titem = string_list_append(&submodules, ce->name);\n> +\t\t\t\titem->util = xmalloc(sizeof(tmp));\n> +\t\t\t\tmemcpy(item->util, &tmp, sizeof(tmp));\n> +\t\t\t\tcontinue;\n> +\t\t\t}\n> \t\t}\n> \n> \t\tif (diff_change_helper(&revs->diffopt, newmode, dirty_submodule,\n> \t\t\t\t changed, istate, ce))\n\nI'm surprised to see that we still call \"diff_change_helper()\" even\nthough we've 'deferred' the submodule diff, especially since \"changed\"\nis set and \"dirty_submodule\" is unset. Even if this is safe, I think we\nshouldn't do this because...\n\n> +\tif (submodules.nr) {\n> +\t\tunsigned long parallel_jobs;\n> +\t\tstruct string_list_item *item;\n> +\n> +\t\tif (git_config_get_ulong(\"submodule.diffjobs\", ¶llel_jobs))\n> +\t\t\tparallel_jobs = 1;\n> +\t\telse if (!parallel_jobs)\n> +\t\t\tparallel_jobs = online_cpus();\n> +\n> +\t\tif (get_submodules_status(&submodules, parallel_jobs))\n> +\t\t\tdie(_(\"submodule status failed\"));\n> +\t\tfor_each_string_list_item(item, &submodules) {\n> +\t\t\tstruct submodule_status_util *util = item->util;\n> +\n> +\t\t\tif (diff_change_helper(&revs->diffopt, util->newmode,\n> +\t\t\t\t util->dirty_submodule, util->changed,\n> +\t\t\t\t istate, util->ce))\n\nHere we call \"diff_change_helper()\" again on the deferred submodule, but\nnow with the \"dirty_submodule\" value we expected. At best this is\nwasteful, but at worst this is possibly wrong.\n\nFor good measure, I applied this patch to see if we needed either\n\"dirty_submodule\" or the second \"diff_change_helper()\" call; our\ntest suite still passes after I remove both of them.\n\n diff --git a/diff-lib.c b/diff-lib.c\n index 2dde575ec6..21adcc7fd6 100644\n --- a/diff-lib.c\n +++ b/diff-lib.c\n @@ -156,6 +156,7 @@ int run_diff_files(struct rev_info *revs, unsigned int option)\n struct cache_entry *ce = istate->cache[i];\n int changed;\n unsigned dirty_submodule = 0;\n +\t\tint defer_submodule_status = 1;\n\n if (diff_can_quit_early(&revs->diffopt))\n break;\n @@ -267,7 +268,6 @@ int run_diff_files(struct rev_info *revs, unsigned int option)\n } else {\n struct stat st;\n unsigned ignore_untracked = 0;\n -\t\t\tint defer_submodule_status = 1;\n\n changed = check_removed(istate, ce, &st);\n if (changed) {\n @@ -311,9 +311,9 @@ int run_diff_files(struct rev_info *revs, unsigned int option)\n }\n }\n\n -\t\tif (diff_change_helper(&revs->diffopt, newmode, dirty_submodule,\n -\t\t\t\t changed, istate, ce))\n -\t\t\tcontinue;\n +\t\tif (!defer_submodule_status)\n +\t\t\tdiff_change_helper(&revs->diffopt, newmode, 0,\n +\t\t\t\t\t changed, istate, ce);\n }\n if (submodules.nr) {\n unsigned long parallel_jobs;\n\n\n> +static void parse_status_porcelain_strbuf(struct strbuf *buf,\n> +\t\t\t\t unsigned *dirty_submodule,\n> +\t\t\t\t int ignore_untracked)\n> +{\n> +\tstruct string_list list = STRING_LIST_INIT_DUP;\n> +\tstruct string_list_item *item;\n> +\n> +\tstring_list_split(&list, buf->buf, '\\n', -1);\n> +\n> +\tfor_each_string_list_item(item, &list) {\n> +\t\tif (parse_status_porcelain(item->string,\n> +\t\t\t\t\t strlen(item->string),\n> +\t\t\t\t\t dirty_submodule,\n> +\t\t\t\t\t ignore_untracked))\n> +\t\t\tbreak;\n> +\t}\n> +\tstring_list_clear(&list, 0);\n> +}\n\nGiven that this function only has one caller, is quite simple, and isn't\nactually a strbuf version of \"parse_status_porcelain()\" (it's actually a\nmultiline version that also happens to accept a strbuf), I think this\nmight be better inlined.\n\n> +test_expect_success 'status in superproject with submodules (parallel)' '\n> +\tgit -C super status --porcelain >output &&\n> +\tgit -C super -c submodule.diffJobs=8 status --porcelain >output_parallel &&\n> +\tdiff output output_parallel\n> +'\n> +\n> test_done\n\nWhen I first suggested this test, I thought that we would sometimes\ndefer submodule status and sometimes not, so this would be a good way to\ncheck between both modes. Now this is less useful, since this is only\nchecking that parallelism > 1 doesn't affect the output, but it's still\na useful reasonableness check IMO. Thanks.\n"},{"id":"472003","messageId":"kl6lcz6earkp.fsf@chooglen-macbookpro.roam.corp.google.com","threadId":"59034","inReplyTo":"20230209000212.1892457-3-calvinwan@google.com","subject":"Re: [PATCH v8 2/6] submodule: strbuf variable rename","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2023-02-13T08:37:58Z","receivedAt":"2023-02-13T08:38:04Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"Calvin Wan writes:\n\n> Subject: [PATCH v8 2/6] submodule: strbuf variable rename\n\nThis should probably be \"submodule: rename strbuf variable\".\n"},{"id":"472004","messageId":"kl6la61iardd.fsf@chooglen-macbookpro.roam.corp.google.com","threadId":"59034","inReplyTo":"20230209000212.1892457-6-calvinwan@google.com","subject":"Re: [PATCH v8 5/6] diff-lib: refactor out diff_change logic","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2023-02-13T08:42:22Z","receivedAt":"2023-02-13T08:42:28Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"Calvin Wan writes:\n\n> Refactor out logic that sets up the diff_change call into a helper\n> function for a future patch.\n\nThis seems underspecified; there are two diff_change calls in diff-lib,\nand the call in show_modified() is not changed in this patch.\n\n> +static int diff_change_helper(struct diff_options *options,\n> +\t unsigned newmode, unsigned dirty_submodule,\n> +\t int changed, struct index_state *istate,\n> +\t struct cache_entry *ce)\n\nThe function name is very generic, and it's not clear:\n\n- What this does besides calling \"diff_change()\".\n- When I should call this instead of \"diff_change()\".\n- What the return value means.\n\nBoth of these should be documented in a comment, and I also suggest\nrenaming the function.\n\n\n> @@ -245,21 +269,9 @@ int run_diff_files(struct rev_info *revs, unsigned int option)\n> \t\t\tnewmode = ce_mode_from_stat(ce, st.st_mode);\n> \t\t}\n> \n> -\t\tif (!changed && !dirty_submodule) {\n> -\t\t\tce_mark_uptodate(ce);\n> -\t\t\tmark_fsmonitor_valid(istate, ce);\n> -\t\t\tif (!revs->diffopt.flags.find_copies_harder)\n> -\t\t\t\tcontinue;\n> -\t\t}\n> -\t\toldmode = ce->ce_mode;\n> -\t\told_oid = &ce->oid;\n> -\t\tnew_oid = changed ? null_oid() : &ce->oid;\n> -\t\tdiff_change(&revs->diffopt, oldmode, newmode,\n> -\t\t\t old_oid, new_oid,\n> -\t\t\t !is_null_oid(old_oid),\n> -\t\t\t !is_null_oid(new_oid),\n> -\t\t\t ce->name, 0, dirty_submodule);\n> -\n> +\t\tif (diff_change_helper(&revs->diffopt, newmode, dirty_submodule,\n> +\t\t\t\t changed, istate, ce))\n> +\t\t\tcontinue;\n> \t}\n\nIf I'm reading the indentation correctly, the \"continue\" comes right\nbefore the end of the for-loop block, so it's a no-op and should be\nremoved.\n"},{"id":"472011","messageId":"xmqqbklxcv1v.fsf@gitster.g","threadId":"59034","inReplyTo":"kl6lo7pyax9q.fsf@chooglen-macbookpro.roam.corp.google.com","subject":"Re: [PATCH v8 1/6] run-command: add duplicate_output_fn to run_processes_parallel_opts","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-02-13T17:52:12Z","receivedAt":"2023-02-13T17:52:21Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Glen Choo writes:\n\n> What do we think of the name \"duplicate_output\"? IMO it made sense in\n> earlier versions when we were copying the output to a separate buffer (I\n> believe it was renamed in response to [1]), but now that we're just\n> calling a callback on the main buffer, it seems misleading. Maybe\n> \"output_buffered\" would be better?\n\nYeah, we do not even know what the callback does to the data we are\ngiving it. The only thing we know is that we have output from the\nchild, and in addition to the usual buffering we do ourselves, we\nare allowing the callback to peek into the buffered data in advance.\n\nIf the callback does consume it *and* remove the buffered data it\nconsumed right away, then as you say, \"duplicate\" becomes a word\nthat totally misses the point. There is no duplication, as the\ncallback consumed and we no longer has our own copy, either.\n\nIf the callback consumes it but leaves the buffered data as-is, and\nwe would show that once the child finishes anyway, you can say that\nwe are feeding a duplicate of buffered data to the callback. The\nmechanism could be used merely to count how much output we have\naccumulated so far to update the progress-bar, for example, and the\noutput may be given after the process is done. But note that we are\nnot doing an \"output\" of \"buffered\" data in such a case.\n\nTo me, both \"duplicate_output\" and \"output_buffered\" sound like they\nare names that are quite specific to the expected use case the\nperson who proposed the names had in mind, yet it is a bit hard to\nguess exactly what the expected use cases they had in mind were,\nbecause the names are not quite specific enough.\n\n> Sidenote: One convention from JS that I like is to name such event\n> listeners as \"on_\", e.g. \"on_output_buffered\".\n\nThanks for bringing this up. I agree that \"Upon X happening, do\nthis\" is a very good convention to follow. I think the callback is\nmade whenever the child emits to the standard error stream, so\n\"on_error_output\" (if we are worried that \"error\" has a too strong\n\"something bad happend\" connotation, then perhaps \"on_stderr_output\"\nmay dampen it) perhaps?\n"},{"id":"472020","messageId":"CAFySSZDMO6=YL6wVxyfj6h8piTFg7OygMdqdELspnQWpm4yJ7A@mail.gmail.com","threadId":"59034","inReplyTo":"xmqqbklxcv1v.fsf@gitster.g","subject":"Re: [PATCH v8 1/6] run-command: add duplicate_output_fn to run_processes_parallel_opts","fromName":"Calvin Wan","fromEmail":"calvinwan@google.com","sentAt":"2023-02-13T18:26:50Z","receivedAt":"2023-02-13T18:28:04Z","isPatch":true,"sender":{"key":"calvinwan@google.com","avatar":"https://avatars.githubusercontent.com/u/92547554?v=4"},"body":"> > Sidenote: One convention from JS that I like is to name such event\n> > listeners as \"on_\", e.g. \"on_output_buffered\".\n>\n> Thanks for bringing this up. I agree that \"Upon X happening, do\n> this\" is a very good convention to follow. I think the callback is\n> made whenever the child emits to the standard error stream, so\n> \"on_error_output\" (if we are worried that \"error\" has a too strong\n> \"something bad happend\" connotation, then perhaps \"on_stderr_output\"\n> may dampen it) perhaps?\n\n\"on_stderr_output\" sounds much better than \"duplicate_output\". I\ndid spend much time trying to come up with a better name, but\ncouldn't find anything that conveyed what the expected use case\nof this function was. Thanks, I'll rename it on my next reroll.\n"},{"id":"472021","messageId":"CAFySSZDz2BE4yrWKrWZhBx_yFBTUcWYtM9Fie-Npa84Ln0RTZA@mail.gmail.com","threadId":"59034","inReplyTo":"kl6la61iardd.fsf@chooglen-macbookpro.roam.corp.google.com","subject":"Re: [PATCH v8 5/6] diff-lib: refactor out diff_change logic","fromName":"Calvin Wan","fromEmail":"calvinwan@google.com","sentAt":"2023-02-13T18:29:55Z","receivedAt":"2023-02-13T18:30:27Z","isPatch":true,"sender":{"key":"calvinwan@google.com","avatar":"https://avatars.githubusercontent.com/u/92547554?v=4"},"body":"On Mon, Feb 13, 2023 at 12:42 AM Glen Choo wrote:\n>\n> Calvin Wan writes:\n>\n> > Refactor out logic that sets up the diff_change call into a helper\n> > function for a future patch.\n>\n> This seems underspecified; there are two diff_change calls in diff-lib,\n> and the call in show_modified() is not changed in this patch.\n>\n> > +static int diff_change_helper(struct diff_options *options,\n> > + unsigned newmode, unsigned dirty_submodule,\n> > + int changed, struct index_state *istate,\n> > + struct cache_entry *ce)\n>\n> The function name is very generic, and it's not clear:\n>\n> - What this does besides calling \"diff_change()\".\n> - When I should call this instead of \"diff_change()\".\n> - What the return value means.\n>\n> Both of these should be documented in a comment, and I also suggest\n> renaming the function.\n\nack.\n\n> > @@ -245,21 +269,9 @@ int run_diff_files(struct rev_info *revs, unsigned int option)\n> > newmode = ce_mode_from_stat(ce, st.st_mode);\n> > }\n> >\n> > - if (!changed && !dirty_submodule) {\n> > - ce_mark_uptodate(ce);\n> > - mark_fsmonitor_valid(istate, ce);\n> > - if (!revs->diffopt.flags.find_copies_harder)\n> > - continue;\n> > - }\n> > - oldmode = ce->ce_mode;\n> > - old_oid = &ce->oid;\n> > - new_oid = changed ? null_oid() : &ce->oid;\n> > - diff_change(&revs->diffopt, oldmode, newmode,\n> > - old_oid, new_oid,\n> > - !is_null_oid(old_oid),\n> > - !is_null_oid(new_oid),\n> > - ce->name, 0, dirty_submodule);\n> > -\n> > + if (diff_change_helper(&revs->diffopt, newmode, dirty_submodule,\n> > + changed, istate, ce))\n> > + continue;\n> > }\n>\n> If I'm reading the indentation correctly, the \"continue\" comes right\n> before the end of the for-loop block, so it's a no-op and should be\n> removed.\n\nIt is a no-op, but I left it in as future-proofing in case more code is\nadded after that block later. I'm not sure whether that line of\nreasoning is enough to leave it in though.\n"},{"id":"472043","messageId":"kl6lwn4kopum.fsf@chooglen-macbookpro.roam.corp.google.com","threadId":"59034","inReplyTo":"CAFySSZDz2BE4yrWKrWZhBx_yFBTUcWYtM9Fie-Npa84Ln0RTZA@mail.gmail.com","subject":"Re: [PATCH v8 5/6] diff-lib: refactor out diff_change logic","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2023-02-14T04:03:45Z","receivedAt":"2023-02-14T04:03:52Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"Calvin Wan writes:\n\n>> > @@ -245,21 +269,9 @@ int run_diff_files(struct rev_info *revs, unsigned int option)\n>> > newmode = ce_mode_from_stat(ce, st.st_mode);\n>> > }\n>> >\n>> > - if (!changed && !dirty_submodule) {\n>> > - ce_mark_uptodate(ce);\n>> > - mark_fsmonitor_valid(istate, ce);\n>> > - if (!revs->diffopt.flags.find_copies_harder)\n>> > - continue;\n>> > - }\n>> > - oldmode = ce->ce_mode;\n>> > - old_oid = &ce->oid;\n>> > - new_oid = changed ? null_oid() : &ce->oid;\n>> > - diff_change(&revs->diffopt, oldmode, newmode,\n>> > - old_oid, new_oid,\n>> > - !is_null_oid(old_oid),\n>> > - !is_null_oid(new_oid),\n>> > - ce->name, 0, dirty_submodule);\n>> > -\n>> > + if (diff_change_helper(&revs->diffopt, newmode, dirty_submodule,\n>> > + changed, istate, ce))\n>> > + continue;\n>> > }\n>>\n>> If I'm reading the indentation correctly, the \"continue\" comes right\n>> before the end of the for-loop block, so it's a no-op and should be\n>> removed.\n>\n> It is a no-op, but I left it in as future-proofing in case more code is\n> added after that block later. I'm not sure whether that line of\n> reasoning is enough to leave it in though.\n\nI don't think it is. If we haven't thought of the reason why we would\nneed to skip code, that seems like YAGNI to me.\n\nAs a matter of personal taste, I wouldn't leave a trailing \"continue\" in\nan earlier patch even if I were going to change it in a later patch,\nbecause it looks too much like an unintentional mistake.\n"},{"id":"472934","messageId":"20230302215237.1473444-1-calvinwan@google.com","threadId":"59034","inReplyTo":"20230209000212.1892457-1-calvinwan@google.com","subject":"[PATCH v9 0/6] submodule: parallelize diff","fromName":"Calvin Wan","fromEmail":"calvinwan@google.com","sentAt":"2023-03-02T21:52:31Z","receivedAt":"2023-03-02T22:01:36Z","isPatch":true,"sender":{"key":"calvinwan@google.com","avatar":"https://avatars.githubusercontent.com/u/92547554?v=4"},"body":"Original cover letter for context:\nhttps://lore.kernel.org/git/20221011232604.839941-1-calvinwan@google.com/\n\nI appreciate all the reviewers that have stuck through this entire series!\nHoping this can be the final reroll as I believe I've addressed all feedback\nand personally am happy with the state of the patches.\n\nChanges from v8\n - renamed duplicate_output_fn to on_stderr_output_fn\n - renamed diff_change_helper() to record_file_diff() and added comments\n - reworded commit message for patch 5\n - removed the refactoring of match_stat_with_submodule()\n - inlined parse_status_porcelain_strbuf()\n - fixed stylistic nits and cleaned up unnecessary variables and logic\n\nCalvin Wan (6):\n run-command: add on_stderr_output_fn to run_processes_parallel_opts\n submodule: rename strbuf variable\n submodule: move status parsing into function\n submodule: refactor is_submodule_modified()\n diff-lib: refactor out diff_change logic\n diff-lib: parallelize run_diff_files for submodules\n\n Documentation/config/submodule.txt | 12 ++\n diff-lib.c | 123 +++++++++++---\n run-command.c | 16 +-\n run-command.h | 25 +++\n submodule.c | 254 +++++++++++++++++++++++------\n submodule.h | 9 +\n t/helper/test-run-command.c | 20 +++\n t/t0061-run-command.sh | 39 +++++\n t/t4027-diff-submodule.sh | 31 ++++\n t/t7506-status-submodule.sh | 25 +++\n 10 files changed, 478 insertions(+), 76 deletions(-)\n\nRange-diff against v8:\n1: 5d51250c67 ! 1: 49749ae3a5 run-command: add duplicate_output_fn to run_processes_parallel_opts\n @@ Metadata\n Author: Calvin Wan \n \n ## Commit message ##\n - run-command: add duplicate_output_fn to run_processes_parallel_opts\n + run-command: add on_stderr_output_fn to run_processes_parallel_opts\n \n ## run-command.c ##\n @@ run-command.c: static void pp_init(struct parallel_processes *pp,\n @@ run-command.c: static void pp_init(struct parallel_processes *pp,\n \t\tBUG(\"you need to specify a get_next_task function\");\n \n +\tif (opts->ungroup) {\n -+\t\tif (opts->duplicate_output)\n -+\t\t\tBUG(\"duplicate_output and ungroup are incompatible with each other\");\n ++\t\tif (opts->on_stderr_output)\n ++\t\t\tBUG(\"on_stderr_output and ungroup are incompatible with each other\");\n +\t}\n +\n \tCALLOC_ARRAY(pp->children, n);\n @@ run-command.c: static void pp_buffer_stderr(struct parallel_processes *pp,\n +\t\t\t} else if (n < 0) {\n \t\t\t\tif (errno != EAGAIN)\n \t\t\t\t\tdie_errno(\"read\");\n -+\t\t\t} else if (opts->duplicate_output) {\n -+\t\t\t\topts->duplicate_output(&pp->children[i].err,\n ++\t\t\t} else if (opts->on_stderr_output) {\n ++\t\t\t\topts->on_stderr_output(&pp->children[i].err,\n +\t\t\t\t\tpp->children[i].err.len - n,\n +\t\t\t\t\topts->data, pp->children[i].data);\n +\t\t\t}\n @@ run-command.h: typedef int (*start_failure_fn)(struct strbuf *out,\n \n +/**\n + * This callback is called whenever output from a child process is buffered\n -+ * \n ++ *\n + * See run_processes_parallel() below for a discussion of the \"struct\n + * strbuf *out\" parameter.\n -+ * \n ++ *\n + * The offset refers to the number of bytes originally in \"out\" before\n + * the output from the child process was buffered. Therefore, the buffer\n + * range, \"out + buf\" to the end of \"out\", would contain the buffer of\n @@ run-command.h: typedef int (*start_failure_fn)(struct strbuf *out,\n + *\n + * This function is incompatible with \"ungroup\"\n + */\n -+typedef void (*duplicate_output_fn)(struct strbuf *out, size_t offset,\n ++typedef void (*on_stderr_output_fn)(struct strbuf *out, size_t offset,\n +\t\t\t\t void *pp_cb, void *pp_task_cb);\n +\n /**\n @@ run-command.h: struct run_process_parallel_opts\n \tstart_failure_fn start_failure;\n \n +\t/**\n -+\t * duplicate_output: See duplicate_output_fn() above. Unless you need\n ++\t * on_stderr_output: See on_stderr_output_fn() above. Unless you need\n +\t * to capture output from child processes, leave this as NULL.\n +\t */\n -+\tduplicate_output_fn duplicate_output;\n ++\ton_stderr_output_fn on_stderr_output;\n +\n \t/**\n \t * task_finished: See task_finished_fn() above. This can be\n @@ t/helper/test-run-command.c: static int no_job(struct child_process *cp,\n \treturn 0;\n }\n \n -+static void duplicate_output(struct strbuf *out,\n ++static void on_stderr_output(struct strbuf *out,\n +\t\t\tsize_t offset,\n +\t\t\tvoid *pp_cb UNUSED,\n +\t\t\tvoid *pp_task_cb UNUSED)\n @@ t/helper/test-run-command.c: static int no_job(struct child_process *cp,\n +\n +\tstring_list_split(&list, out->buf + offset, '\\n', -1);\n +\tfor_each_string_list_item(item, &list)\n -+\t\tfprintf(stderr, \"duplicate_output: %s\\n\", item->string);\n ++\t\tfprintf(stderr, \"on_stderr_output: %s\\n\", item->string);\n +\tstring_list_clear(&list, 0);\n +}\n +\n @@ t/helper/test-run-command.c: int cmd__run_command(int argc, const char **argv)\n \t\topts.ungroup = 1;\n \t}\n \n -+\tif (!strcmp(argv[1], \"--duplicate-output\")) {\n ++\tif (!strcmp(argv[1], \"--on-stderr-output\")) {\n +\t\targv += 1;\n +\t\targc -= 1;\n -+\t\topts.duplicate_output = duplicate_output;\n ++\t\topts.on_stderr_output = on_stderr_output;\n +\t}\n +\n \tjobs = atoi(argv[2]);\n @@ t/t0061-run-command.sh: test_expect_success 'run_command runs in parallel with m\n \ttest_cmp expect actual\n '\n \n -+test_expect_success 'run_command runs in parallel with more jobs available than tasks --duplicate-output' '\n -+\ttest-tool run-command --duplicate-output run-command-parallel 5 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n ++test_expect_success 'run_command runs in parallel with more jobs available than tasks --on-stderr-output' '\n ++\ttest-tool run-command --on-stderr-output run-command-parallel 5 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n +\ttest_must_be_empty out &&\n -+\ttest 4 = $(grep -c \"duplicate_output: Hello\" err) &&\n -+\ttest 4 = $(grep -c \"duplicate_output: World\" err) &&\n -+\tsed \"/duplicate_output/d\" err >err1 &&\n ++\ttest 4 = $(grep -c \"on_stderr_output: Hello\" err) &&\n ++\ttest 4 = $(grep -c \"on_stderr_output: World\" err) &&\n ++\tsed \"/on_stderr_output/d\" err >err1 &&\n +\ttest_cmp expect err1\n +'\n +\n @@ t/t0061-run-command.sh: test_expect_success 'run_command runs in parallel with a\n \ttest_cmp expect actual\n '\n \n -+test_expect_success 'run_command runs in parallel with as many jobs as tasks --duplicate-output' '\n -+\ttest-tool run-command --duplicate-output run-command-parallel 4 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n ++test_expect_success 'run_command runs in parallel with as many jobs as tasks --on-stderr-output' '\n ++\ttest-tool run-command --on-stderr-output run-command-parallel 4 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n +\ttest_must_be_empty out &&\n -+\ttest 4 = $(grep -c \"duplicate_output: Hello\" err) &&\n -+\ttest 4 = $(grep -c \"duplicate_output: World\" err) &&\n -+\tsed \"/duplicate_output/d\" err >err1 &&\n ++\ttest 4 = $(grep -c \"on_stderr_output: Hello\" err) &&\n ++\ttest 4 = $(grep -c \"on_stderr_output: World\" err) &&\n ++\tsed \"/on_stderr_output/d\" err >err1 &&\n +\ttest_cmp expect err1\n +'\n +\n @@ t/t0061-run-command.sh: test_expect_success 'run_command runs in parallel with m\n \ttest_cmp expect actual\n '\n \n -+test_expect_success 'run_command runs in parallel with more tasks than jobs available --duplicate-output' '\n -+\ttest-tool run-command --duplicate-output run-command-parallel 3 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n ++test_expect_success 'run_command runs in parallel with more tasks than jobs available --on-stderr-output' '\n ++\ttest-tool run-command --on-stderr-output run-command-parallel 3 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n +\ttest_must_be_empty out &&\n -+\ttest 4 = $(grep -c \"duplicate_output: Hello\" err) &&\n -+\ttest 4 = $(grep -c \"duplicate_output: World\" err) &&\n -+\tsed \"/duplicate_output/d\" err >err1 &&\n ++\ttest 4 = $(grep -c \"on_stderr_output: Hello\" err) &&\n ++\ttest 4 = $(grep -c \"on_stderr_output: World\" err) &&\n ++\tsed \"/on_stderr_output/d\" err >err1 &&\n +\ttest_cmp expect err1\n +'\n +\n @@ t/t0061-run-command.sh: test_expect_success 'run_command is asked to abort grace\n \ttest_cmp expect actual\n '\n \n -+test_expect_success 'run_command is asked to abort gracefully --duplicate-output' '\n -+\ttest-tool run-command --duplicate-output run-command-abort 3 false >out 2>err &&\n ++test_expect_success 'run_command is asked to abort gracefully --on-stderr-output' '\n ++\ttest-tool run-command --on-stderr-output run-command-abort 3 false >out 2>err &&\n +\ttest_must_be_empty out &&\n +\ttest_cmp expect err\n +'\n @@ t/t0061-run-command.sh: test_expect_success 'run_command outputs ' '\n \ttest_cmp expect actual\n '\n \n -+test_expect_success 'run_command outputs --duplicate-output' '\n -+\ttest-tool run-command --duplicate-output run-command-no-jobs 3 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n ++test_expect_success 'run_command outputs --on-stderr-output' '\n ++\ttest-tool run-command --on-stderr-output run-command-no-jobs 3 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n +\ttest_must_be_empty out &&\n +\ttest_cmp expect err\n +'\n2: 6ded5b6788 ! 2: 6c62e670f9 submodule: strbuf variable rename\n @@ Metadata\n Author: Calvin Wan \n \n ## Commit message ##\n - submodule: strbuf variable rename\n + submodule: rename strbuf variable\n \n ## submodule.c ##\n @@ submodule.c: unsigned is_submodule_modified(const char *path, int ignore_untracked)\n3: 0c71cea8cd = 3: 24e02f2a24 submodule: move status parsing into function\n4: 5c8cc93f9f = 4: 86c1f734a0 submodule: refactor is_submodule_modified()\n5: 6c2b62abc8 ! 5: 811a1fee55 diff-lib: refactor out diff_change logic\n @@ diff-lib.c: static int match_stat_with_submodule(struct diff_options *diffopt,\n \treturn changed;\n }\n \n -+static int diff_change_helper(struct diff_options *options,\n -+\t unsigned newmode, unsigned dirty_submodule,\n -+\t int changed, struct index_state *istate,\n -+\t struct cache_entry *ce)\n ++/**\n ++ * Records diff_change if there is a change in the entry from run_diff_files.\n ++ * If there is no change, then the cache entry is marked CE_UPTODATE and\n ++ * CE_FSMONITOR_VALID. If there is no change and the find_copies_harder flag\n ++ * is not set, then the function returns early.\n ++ */\n ++static void record_file_diff(struct diff_options *options, unsigned newmode,\n ++\t\t\t unsigned dirty_submodule, int changed,\n ++\t\t\t struct index_state *istate,\n ++\t\t\t struct cache_entry *ce)\n +{\n +\tunsigned int oldmode;\n +\tconst struct object_id *old_oid, *new_oid;\n @@ diff-lib.c: static int match_stat_with_submodule(struct diff_options *diffopt,\n +\t\tce_mark_uptodate(ce);\n +\t\tmark_fsmonitor_valid(istate, ce);\n +\t\tif (!options->flags.find_copies_harder)\n -+\t\t\treturn 1;\n ++\t\t\treturn;\n +\t}\n +\toldmode = ce->ce_mode;\n +\told_oid = &ce->oid;\n +\tnew_oid = changed ? null_oid() : &ce->oid;\n -+\tdiff_change(options, oldmode, newmode,\n -+\t\t\told_oid, new_oid,\n -+\t\t\t!is_null_oid(old_oid),\n -+\t\t\t!is_null_oid(new_oid),\n -+\t\t\tce->name, 0, dirty_submodule);\n -+\treturn 0;\n ++\tdiff_change(options, oldmode, newmode, old_oid, new_oid,\n ++\t\t !is_null_oid(old_oid), !is_null_oid(new_oid),\n ++\t\t ce->name, 0, dirty_submodule);\n +}\n +\n int run_diff_files(struct rev_info *revs, unsigned int option)\n @@ diff-lib.c: int run_diff_files(struct rev_info *revs, unsigned int option)\n -\t\t\t !is_null_oid(new_oid),\n -\t\t\t ce->name, 0, dirty_submodule);\n -\n -+\t\tif (diff_change_helper(&revs->diffopt, newmode, dirty_submodule,\n -+\t\t\t\t changed, istate, ce))\n -+\t\t\tcontinue;\n ++\t\trecord_file_diff(&revs->diffopt, newmode, dirty_submodule,\n ++\t\t\t\t changed, istate, ce);\n \t}\n \tdiffcore_std(&revs->diffopt);\n \tdiff_flush(&revs->diffopt);\n6: bb25dadbe5 ! 6: 17010fc179 diff-lib: parallelize run_diff_files for submodules\n @@ diff-lib.c: static int check_removed(const struct index_state *istate, const str\n +\t\t\t\t unsigned *ignore_untracked)\n {\n \tint changed = ie_match_stat(diffopt->repo->index, ce, st, ce_option);\n --\tif (S_ISGITLINK(ce->ce_mode)) {\n --\t\tstruct diff_flags orig_flags = diffopt->flags;\n --\t\tif (!diffopt->flags.override_submodule_config)\n --\t\t\tset_diffopt_flags_from_submodule_config(diffopt, ce->name);\n ++\tint defer = 0;\n ++\n + \tif (S_ISGITLINK(ce->ce_mode)) {\n + \t\tstruct diff_flags orig_flags = diffopt->flags;\n + \t\tif (!diffopt->flags.override_submodule_config)\n + \t\t\tset_diffopt_flags_from_submodule_config(diffopt, ce->name);\n -\t\tif (diffopt->flags.ignore_submodules)\n --\t\t\tchanged = 0;\n ++\t\tif (diffopt->flags.ignore_submodules) {\n + \t\t\tchanged = 0;\n -\t\telse if (!diffopt->flags.ignore_dirty_submodules &&\n -\t\t\t (!changed || diffopt->flags.dirty_submodules))\n -+\tstruct diff_flags orig_flags;\n -+\tint defer = 0;\n -+\n -+\tif (!S_ISGITLINK(ce->ce_mode))\n -+\t\tgoto ret;\n -+\n -+\torig_flags = diffopt->flags;\n -+\tif (!diffopt->flags.override_submodule_config)\n -+\t\tset_diffopt_flags_from_submodule_config(diffopt, ce->name);\n -+\tif (diffopt->flags.ignore_submodules) {\n -+\t\tchanged = 0;\n -+\t\tgoto cleanup;\n -+\t}\n -+\tif (!diffopt->flags.ignore_dirty_submodules &&\n -+\t (!changed || diffopt->flags.dirty_submodules)) {\n -+\t\tif (defer_submodule_status && *defer_submodule_status) {\n -+\t\t\tdefer = 1;\n -+\t\t\t*ignore_untracked = diffopt->flags.ignore_untracked_in_submodules;\n -+\t\t} else {\n - \t\t\t*dirty_submodule = is_submodule_modified(ce->name,\n +-\t\t\t*dirty_submodule = is_submodule_modified(ce->name,\n -\t\t\t\t\t\t\t\t diffopt->flags.ignore_untracked_in_submodules);\n --\t\tdiffopt->flags = orig_flags;\n ++\t\t} else if (!diffopt->flags.ignore_dirty_submodules &&\n ++\t\t\t (!changed || diffopt->flags.dirty_submodules)) {\n ++\t\t\tif (defer_submodule_status && *defer_submodule_status) {\n ++\t\t\t\tdefer = 1;\n ++\t\t\t\t*ignore_untracked = diffopt->flags.ignore_untracked_in_submodules;\n ++\t\t\t} else {\n ++\t\t\t\t*dirty_submodule = is_submodule_modified(ce->name,\n +\t\t\t\t\t diffopt->flags.ignore_untracked_in_submodules);\n ++\t\t\t}\n +\t\t}\n + \t\tdiffopt->flags = orig_flags;\n \t}\n -+cleanup:\n -+\tdiffopt->flags = orig_flags;\n -+ret:\n ++\n +\tif (defer_submodule_status)\n +\t\t*defer_submodule_status = defer;\n \treturn changed;\n @@ diff-lib.c: int run_diff_files(struct rev_info *revs, unsigned int option)\n \n \tdiff_set_mnemonic_prefix(&revs->diffopt, \"i/\", \"w/\");\n \n +@@ diff-lib.c: int run_diff_files(struct rev_info *revs, unsigned int option)\n + \t\tunsigned int newmode;\n + \t\tstruct cache_entry *ce = istate->cache[i];\n + \t\tint changed;\n +-\t\tunsigned dirty_submodule = 0;\n ++\t\tint defer_submodule_status = 1;\n + \n + \t\tif (diff_can_quit_early(&revs->diffopt))\n + \t\t\tbreak;\n @@ diff-lib.c: int run_diff_files(struct rev_info *revs, unsigned int option)\n \t\t\tnewmode = ce->ce_mode;\n \t\t} else {\n \t\t\tstruct stat st;\n +\t\t\tunsigned ignore_untracked = 0;\n -+\t\t\tint defer_submodule_status = 1;\n \n \t\t\tchanged = check_removed(istate, ce, &st);\n \t\t\tif (changed) {\n @@ diff-lib.c: int run_diff_files(struct rev_info *revs, unsigned int option)\n \n \t\t\tchanged = match_stat_with_submodule(&revs->diffopt, ce, &st,\n -\t\t\t\t\t\t\t ce_option, &dirty_submodule);\n -+\t\t\t\t\t\t\t ce_option, &dirty_submodule,\n ++\t\t\t\t\t\t\t ce_option, NULL,\n +\t\t\t\t\t\t\t &defer_submodule_status,\n +\t\t\t\t\t\t\t &ignore_untracked);\n \t\t\tnewmode = ce_mode_from_stat(ce, st.st_mode);\n @@ diff-lib.c: int run_diff_files(struct rev_info *revs, unsigned int option)\n +\t\t\t}\n \t\t}\n \n - \t\tif (diff_change_helper(&revs->diffopt, newmode, dirty_submodule,\n - \t\t\t\t changed, istate, ce))\n - \t\t\tcontinue;\n - \t}\n +-\t\trecord_file_diff(&revs->diffopt, newmode, dirty_submodule,\n +-\t\t\t\t changed, istate, ce);\n ++\t\tif (!defer_submodule_status)\n ++\t\t\trecord_file_diff(&revs->diffopt, newmode, 0,\n ++\t\t\t\t\t changed,istate, ce);\n ++\t}\n +\tif (submodules.nr) {\n +\t\tunsigned long parallel_jobs;\n +\t\tstruct string_list_item *item;\n @@ diff-lib.c: int run_diff_files(struct rev_info *revs, unsigned int option)\n +\t\tfor_each_string_list_item(item, &submodules) {\n +\t\t\tstruct submodule_status_util *util = item->util;\n +\n -+\t\t\tif (diff_change_helper(&revs->diffopt, util->newmode,\n -+\t\t\t\t util->dirty_submodule, util->changed,\n -+\t\t\t\t istate, util->ce))\n -+\t\t\t\tcontinue;\n ++\t\t\trecord_file_diff(&revs->diffopt, util->newmode,\n ++\t\t\t\t\t util->dirty_submodule, util->changed,\n ++\t\t\t\t\t istate, util->ce);\n +\t\t}\n -+\t}\n + \t}\n +\tstring_list_clear(&submodules, 1);\n \tdiffcore_std(&revs->diffopt);\n \tdiff_flush(&revs->diffopt);\n @@ submodule.c: struct fetch_task {\n /**\n * When a submodule is not defined in .gitmodules, we cannot access it\n * via the regular submodule-config. Create a fake submodule, which we can\n -@@ submodule.c: static int parse_status_porcelain(char *str, size_t len,\n - \treturn 0;\n - }\n - \n -+static void parse_status_porcelain_strbuf(struct strbuf *buf,\n -+\t\t\t\t unsigned *dirty_submodule,\n -+\t\t\t\t int ignore_untracked)\n -+{\n -+\tstruct string_list list = STRING_LIST_INIT_DUP;\n -+\tstruct string_list_item *item;\n -+\n -+\tstring_list_split(&list, buf->buf, '\\n', -1);\n -+\n -+\tfor_each_string_list_item(item, &list) {\n -+\t\tif (parse_status_porcelain(item->string,\n -+\t\t\t\t\t strlen(item->string),\n -+\t\t\t\t\t dirty_submodule,\n -+\t\t\t\t\t ignore_untracked))\n -+\t\t\tbreak;\n -+\t}\n -+\tstring_list_clear(&list, 0);\n -+}\n -+\n - unsigned is_submodule_modified(const char *path, int ignore_untracked)\n - {\n - \tstruct child_process cp = CHILD_PROCESS_INIT;\n @@ submodule.c: unsigned is_submodule_modified(const char *path, int ignore_untracked)\n \treturn dirty_submodule;\n }\n @@ submodule.c: unsigned is_submodule_modified(const char *path, int ignore_untrack\n +\treturn 0;\n +}\n +\n -+static void status_duplicate_output(struct strbuf *out,\n ++static void status_on_stderr_output(struct strbuf *out,\n +\t\t\t\t size_t offset,\n +\t\t\t\t void *cb, void *task_cb)\n +{\n @@ submodule.c: unsigned is_submodule_modified(const char *path, int ignore_untrack\n +\tstruct string_list_item *it =\n +\t\tstring_list_lookup(sps->submodule_names, task->path);\n +\tstruct submodule_status_util *util = it->util;\n ++\tstruct string_list list = STRING_LIST_INIT_DUP;\n ++\tstruct string_list_item *item;\n +\n +\tif (retvalue) {\n +\t\tsps->result = 1;\n +\t\tstrbuf_addf(err, _(STATUS_PORCELAIN_FAIL_ERROR), task->path);\n +\t}\n +\n -+\tparse_status_porcelain_strbuf(&task->out,\n -+\t\t\t &util->dirty_submodule,\n -+\t\t\t util->ignore_untracked);\n -+\n ++\tstring_list_split(&list, task->out.buf, '\\n', -1);\n ++\tfor_each_string_list_item(item, &list) {\n ++\t\tif (parse_status_porcelain(item->string,\n ++\t\t\t\t\t strlen(item->string),\n ++\t\t\t\t\t &util->dirty_submodule,\n ++\t\t\t\t\t util->ignore_untracked))\n ++\t\t\tbreak;\n ++\t}\n ++\tstring_list_clear(&list, 0);\n +\tstrbuf_release(&task->out);\n +\tfree(task);\n +\n @@ submodule.c: unsigned is_submodule_modified(const char *path, int ignore_untrack\n +\n +\t\t.get_next_task = get_next_submodule_status,\n +\t\t.start_failure = status_start_failure,\n -+\t\t.duplicate_output = status_duplicate_output,\n ++\t\t.on_stderr_output = status_on_stderr_output,\n +\t\t.task_finished = status_finish,\n +\t\t.data = &sps,\n +\t};\n-- \n2.40.0.rc0.216.gc4246ad0f0-goog\n\n"},{"id":"472936","messageId":"20230302220251.1474923-1-calvinwan@google.com","threadId":"59034","inReplyTo":"20230302215237.1473444-1-calvinwan@google.com","subject":"[PATCH v9 1/6] run-command: add on_stderr_output_fn to run_processes_parallel_opts","fromName":"Calvin Wan","fromEmail":"calvinwan@google.com","sentAt":"2023-03-02T22:02:46Z","receivedAt":"2023-03-02T22:03:33Z","isPatch":true,"sender":{"key":"calvinwan@google.com","avatar":"https://avatars.githubusercontent.com/u/92547554?v=4"},"body":"Add duplicate_output_fn as an optionally set function in\nrun_process_parallel_opts. If set, output from each child process is\ncopied and passed to the callback function whenever output from the\nchild process is buffered to allow for separate parsing.\n\nFix two items in pp_buffer_stderr:\n * strbuf_read_once returns a ssize_t but the variable it is set to is\n an int so fix that.\n * Add missing brackets to \"else if\" statement\n\nThe ungroup/duplicate_output incompatibility check is nested to\nprepare for future imcompatibles modes with ungroup.\n\nSigned-off-by: Calvin Wan \n---\n run-command.c | 16 ++++++++++++---\n run-command.h | 25 ++++++++++++++++++++++++\n t/helper/test-run-command.c | 20 +++++++++++++++++++\n t/t0061-run-command.sh | 39 +++++++++++++++++++++++++++++++++++++\n 4 files changed, 97 insertions(+), 3 deletions(-)\n\ndiff --git a/run-command.c b/run-command.c\nindex 756f1839aa..7eed4e98c2 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -1526,6 +1526,11 @@ static void pp_init(struct parallel_processes *pp,\n \tif (!opts->get_next_task)\n \t\tBUG(\"you need to specify a get_next_task function\");\n \n+\tif (opts->ungroup) {\n+\t\tif (opts->on_stderr_output)\n+\t\t\tBUG(\"on_stderr_output and ungroup are incompatible with each other\");\n+\t}\n+\n \tCALLOC_ARRAY(pp->children, n);\n \tif (!opts->ungroup)\n \t\tCALLOC_ARRAY(pp->pfd, n);\n@@ -1645,14 +1650,19 @@ static void pp_buffer_stderr(struct parallel_processes *pp,\n \tfor (size_t i = 0; i < opts->processes; i++) {\n \t\tif (pp->children[i].state == GIT_CP_WORKING &&\n \t\t pp->pfd[i].revents & (POLLIN | POLLHUP)) {\n-\t\t\tint n = strbuf_read_once(&pp->children[i].err,\n-\t\t\t\t\t\t pp->children[i].process.err, 0);\n+\t\t\tssize_t n = strbuf_read_once(&pp->children[i].err,\n+\t\t\t\t\t\t pp->children[i].process.err, 0);\n \t\t\tif (n == 0) {\n \t\t\t\tclose(pp->children[i].process.err);\n \t\t\t\tpp->children[i].state = GIT_CP_WAIT_CLEANUP;\n-\t\t\t} else if (n < 0)\n+\t\t\t} else if (n < 0) {\n \t\t\t\tif (errno != EAGAIN)\n \t\t\t\t\tdie_errno(\"read\");\n+\t\t\t} else if (opts->on_stderr_output) {\n+\t\t\t\topts->on_stderr_output(&pp->children[i].err,\n+\t\t\t\t\tpp->children[i].err.len - n,\n+\t\t\t\t\topts->data, pp->children[i].data);\n+\t\t\t}\n \t\t}\n \t}\n }\ndiff --git a/run-command.h b/run-command.h\nindex 072db56a4d..8f08e41fae 100644\n--- a/run-command.h\n+++ b/run-command.h\n@@ -408,6 +408,25 @@ typedef int (*start_failure_fn)(struct strbuf *out,\n \t\t\t\tvoid *pp_cb,\n \t\t\t\tvoid *pp_task_cb);\n \n+/**\n+ * This callback is called whenever output from a child process is buffered\n+ *\n+ * See run_processes_parallel() below for a discussion of the \"struct\n+ * strbuf *out\" parameter.\n+ *\n+ * The offset refers to the number of bytes originally in \"out\" before\n+ * the output from the child process was buffered. Therefore, the buffer\n+ * range, \"out + buf\" to the end of \"out\", would contain the buffer of\n+ * the child process output.\n+ *\n+ * pp_cb is the callback cookie as passed into run_processes_parallel,\n+ * pp_task_cb is the callback cookie as passed into get_next_task_fn.\n+ *\n+ * This function is incompatible with \"ungroup\"\n+ */\n+typedef void (*on_stderr_output_fn)(struct strbuf *out, size_t offset,\n+\t\t\t\t void *pp_cb, void *pp_task_cb);\n+\n /**\n * This callback is called on every child process that finished processing.\n *\n@@ -461,6 +480,12 @@ struct run_process_parallel_opts\n \t */\n \tstart_failure_fn start_failure;\n \n+\t/**\n+\t * on_stderr_output: See on_stderr_output_fn() above. Unless you need\n+\t * to capture output from child processes, leave this as NULL.\n+\t */\n+\ton_stderr_output_fn on_stderr_output;\n+\n \t/**\n \t * task_finished: See task_finished_fn() above. This can be\n \t * NULL to omit any special handling.\ndiff --git a/t/helper/test-run-command.c b/t/helper/test-run-command.c\nindex 3ecb830f4a..a2fac6f762 100644\n--- a/t/helper/test-run-command.c\n+++ b/t/helper/test-run-command.c\n@@ -52,6 +52,20 @@ static int no_job(struct child_process *cp,\n \treturn 0;\n }\n \n+static void on_stderr_output(struct strbuf *out,\n+\t\t\tsize_t offset,\n+\t\t\tvoid *pp_cb UNUSED,\n+\t\t\tvoid *pp_task_cb UNUSED)\n+{\n+\tstruct string_list list = STRING_LIST_INIT_DUP;\n+\tstruct string_list_item *item;\n+\n+\tstring_list_split(&list, out->buf + offset, '\\n', -1);\n+\tfor_each_string_list_item(item, &list)\n+\t\tfprintf(stderr, \"on_stderr_output: %s\\n\", item->string);\n+\tstring_list_clear(&list, 0);\n+}\n+\n static int task_finished(int result,\n \t\t\t struct strbuf *err,\n \t\t\t void *pp_cb,\n@@ -439,6 +453,12 @@ int cmd__run_command(int argc, const char **argv)\n \t\topts.ungroup = 1;\n \t}\n \n+\tif (!strcmp(argv[1], \"--on-stderr-output\")) {\n+\t\targv += 1;\n+\t\targc -= 1;\n+\t\topts.on_stderr_output = on_stderr_output;\n+\t}\n+\n \tjobs = atoi(argv[2]);\n \tstrvec_clear(&proc.args);\n \tstrvec_pushv(&proc.args, (const char **)argv + 3);\ndiff --git a/t/t0061-run-command.sh b/t/t0061-run-command.sh\nindex e2411f6a9b..883d871dfb 100755\n--- a/t/t0061-run-command.sh\n+++ b/t/t0061-run-command.sh\n@@ -135,6 +135,15 @@ test_expect_success 'run_command runs in parallel with more jobs available than\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'run_command runs in parallel with more jobs available than tasks --on-stderr-output' '\n+\ttest-tool run-command --on-stderr-output run-command-parallel 5 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n+\ttest_must_be_empty out &&\n+\ttest 4 = $(grep -c \"on_stderr_output: Hello\" err) &&\n+\ttest 4 = $(grep -c \"on_stderr_output: World\" err) &&\n+\tsed \"/on_stderr_output/d\" err >err1 &&\n+\ttest_cmp expect err1\n+'\n+\n test_expect_success 'run_command runs ungrouped in parallel with more jobs available than tasks' '\n \ttest-tool run-command --ungroup run-command-parallel 5 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n \ttest_line_count = 8 out &&\n@@ -147,6 +156,15 @@ test_expect_success 'run_command runs in parallel with as many jobs as tasks' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'run_command runs in parallel with as many jobs as tasks --on-stderr-output' '\n+\ttest-tool run-command --on-stderr-output run-command-parallel 4 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n+\ttest_must_be_empty out &&\n+\ttest 4 = $(grep -c \"on_stderr_output: Hello\" err) &&\n+\ttest 4 = $(grep -c \"on_stderr_output: World\" err) &&\n+\tsed \"/on_stderr_output/d\" err >err1 &&\n+\ttest_cmp expect err1\n+'\n+\n test_expect_success 'run_command runs ungrouped in parallel with as many jobs as tasks' '\n \ttest-tool run-command --ungroup run-command-parallel 4 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n \ttest_line_count = 8 out &&\n@@ -159,6 +177,15 @@ test_expect_success 'run_command runs in parallel with more tasks than jobs avai\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'run_command runs in parallel with more tasks than jobs available --on-stderr-output' '\n+\ttest-tool run-command --on-stderr-output run-command-parallel 3 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n+\ttest_must_be_empty out &&\n+\ttest 4 = $(grep -c \"on_stderr_output: Hello\" err) &&\n+\ttest 4 = $(grep -c \"on_stderr_output: World\" err) &&\n+\tsed \"/on_stderr_output/d\" err >err1 &&\n+\ttest_cmp expect err1\n+'\n+\n test_expect_success 'run_command runs ungrouped in parallel with more tasks than jobs available' '\n \ttest-tool run-command --ungroup run-command-parallel 3 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n \ttest_line_count = 8 out &&\n@@ -180,6 +207,12 @@ test_expect_success 'run_command is asked to abort gracefully' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'run_command is asked to abort gracefully --on-stderr-output' '\n+\ttest-tool run-command --on-stderr-output run-command-abort 3 false >out 2>err &&\n+\ttest_must_be_empty out &&\n+\ttest_cmp expect err\n+'\n+\n test_expect_success 'run_command is asked to abort gracefully (ungroup)' '\n \ttest-tool run-command --ungroup run-command-abort 3 false >out 2>err &&\n \ttest_must_be_empty out &&\n@@ -196,6 +229,12 @@ test_expect_success 'run_command outputs ' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'run_command outputs --on-stderr-output' '\n+\ttest-tool run-command --on-stderr-output run-command-no-jobs 3 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n+\ttest_must_be_empty out &&\n+\ttest_cmp expect err\n+'\n+\n test_expect_success 'run_command outputs (ungroup) ' '\n \ttest-tool run-command --ungroup run-command-no-jobs 3 sh -c \"printf \\\"%s\\n%s\\n\\\" Hello World\" >out 2>err &&\n \ttest_must_be_empty out &&\n-- \n2.40.0.rc0.216.gc4246ad0f0-goog\n\n"},{"id":"472937","messageId":"20230302220251.1474923-2-calvinwan@google.com","threadId":"59034","inReplyTo":"20230302215237.1473444-1-calvinwan@google.com","subject":"[PATCH v9 2/6] submodule: rename strbuf variable","fromName":"Calvin Wan","fromEmail":"calvinwan@google.com","sentAt":"2023-03-02T22:02:47Z","receivedAt":"2023-03-02T22:03:35Z","isPatch":true,"sender":{"key":"calvinwan@google.com","avatar":"https://avatars.githubusercontent.com/u/92547554?v=4"},"body":"A prepatory change for a future patch that moves the status parsing\nlogic to a separate function.\n\nSigned-off-by: Calvin Wan \n---\n submodule.c | 23 +++++++++++++----------\n 1 file changed, 13 insertions(+), 10 deletions(-)\n\ndiff --git a/submodule.c b/submodule.c\nindex fae24ef34a..faf37c1101 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -1906,25 +1906,28 @@ unsigned is_submodule_modified(const char *path, int ignore_untracked)\n \n \tfp = xfdopen(cp.out, \"r\");\n \twhile (strbuf_getwholeline(&buf, fp, '\\n') != EOF) {\n+\t\tchar *str = buf.buf;\n+\t\tconst size_t len = buf.len;\n+\n \t\t/* regular untracked files */\n-\t\tif (buf.buf[0] == '?')\n+\t\tif (str[0] == '?')\n \t\t\tdirty_submodule |= DIRTY_SUBMODULE_UNTRACKED;\n \n-\t\tif (buf.buf[0] == 'u' ||\n-\t\t buf.buf[0] == '1' ||\n-\t\t buf.buf[0] == '2') {\n+\t\tif (str[0] == 'u' ||\n+\t\t str[0] == '1' ||\n+\t\t str[0] == '2') {\n \t\t\t/* T = line type, XY = status, SSSS = submodule state */\n-\t\t\tif (buf.len < strlen(\"T XY SSSS\"))\n+\t\t\tif (len < strlen(\"T XY SSSS\"))\n \t\t\t\tBUG(\"invalid status --porcelain=2 line %s\",\n-\t\t\t\t buf.buf);\n+\t\t\t\t str);\n \n-\t\t\tif (buf.buf[5] == 'S' && buf.buf[8] == 'U')\n+\t\t\tif (str[5] == 'S' && str[8] == 'U')\n \t\t\t\t/* nested untracked file */\n \t\t\t\tdirty_submodule |= DIRTY_SUBMODULE_UNTRACKED;\n \n-\t\t\tif (buf.buf[0] == 'u' ||\n-\t\t\t buf.buf[0] == '2' ||\n-\t\t\t memcmp(buf.buf + 5, \"S..U\", 4))\n+\t\t\tif (str[0] == 'u' ||\n+\t\t\t str[0] == '2' ||\n+\t\t\t memcmp(str + 5, \"S..U\", 4))\n \t\t\t\t/* other change */\n \t\t\t\tdirty_submodule |= DIRTY_SUBMODULE_MODIFIED;\n \t\t}\n-- \n2.40.0.rc0.216.gc4246ad0f0-goog\n\n"},{"id":"472938","messageId":"20230302220251.1474923-3-calvinwan@google.com","threadId":"59034","inReplyTo":"20230302215237.1473444-1-calvinwan@google.com","subject":"[PATCH v9 3/6] submodule: move status parsing into function","fromName":"Calvin Wan","fromEmail":"calvinwan@google.com","sentAt":"2023-03-02T22:02:48Z","receivedAt":"2023-03-02T22:03:37Z","isPatch":true,"sender":{"key":"calvinwan@google.com","avatar":"https://avatars.githubusercontent.com/u/92547554?v=4"},"body":"A future patch requires the ability to parse the output of git\nstatus --porcelain=2. Move parsing code from is_submodule_modified to\nparse_status_porcelain.\n\nSigned-off-by: Calvin Wan \n---\n submodule.c | 74 ++++++++++++++++++++++++++++++-----------------------\n 1 file changed, 42 insertions(+), 32 deletions(-)\n\ndiff --git a/submodule.c b/submodule.c\nindex faf37c1101..768d4b4cd7 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -1870,6 +1870,45 @@ int fetch_submodules(struct repository *r,\n \treturn spf.result;\n }\n \n+static int parse_status_porcelain(char *str, size_t len,\n+\t\t\t\t unsigned *dirty_submodule,\n+\t\t\t\t int ignore_untracked)\n+{\n+\t/* regular untracked files */\n+\tif (str[0] == '?')\n+\t\t*dirty_submodule |= DIRTY_SUBMODULE_UNTRACKED;\n+\n+\tif (str[0] == 'u' ||\n+\t str[0] == '1' ||\n+\t str[0] == '2') {\n+\t\t/* T = line type, XY = status, SSSS = submodule state */\n+\t\tif (len < strlen(\"T XY SSSS\"))\n+\t\t\tBUG(\"invalid status --porcelain=2 line %s\",\n+\t\t\t str);\n+\n+\t\tif (str[5] == 'S' && str[8] == 'U')\n+\t\t\t/* nested untracked file */\n+\t\t\t*dirty_submodule |= DIRTY_SUBMODULE_UNTRACKED;\n+\n+\t\tif (str[0] == 'u' ||\n+\t\t str[0] == '2' ||\n+\t\t memcmp(str + 5, \"S..U\", 4))\n+\t\t\t/* other change */\n+\t\t\t*dirty_submodule |= DIRTY_SUBMODULE_MODIFIED;\n+\t}\n+\n+\tif ((*dirty_submodule & DIRTY_SUBMODULE_MODIFIED) &&\n+\t ((*dirty_submodule & DIRTY_SUBMODULE_UNTRACKED) ||\n+\t ignore_untracked)) {\n+\t\t/*\n+\t\t* We're not interested in any further information from\n+\t\t* the child any more, neither output nor its exit code.\n+\t\t*/\n+\t\treturn 1;\n+\t}\n+\treturn 0;\n+}\n+\n unsigned is_submodule_modified(const char *path, int ignore_untracked)\n {\n \tstruct child_process cp = CHILD_PROCESS_INIT;\n@@ -1909,39 +1948,10 @@ unsigned is_submodule_modified(const char *path, int ignore_untracked)\n \t\tchar *str = buf.buf;\n \t\tconst size_t len = buf.len;\n \n-\t\t/* regular untracked files */\n-\t\tif (str[0] == '?')\n-\t\t\tdirty_submodule |= DIRTY_SUBMODULE_UNTRACKED;\n-\n-\t\tif (str[0] == 'u' ||\n-\t\t str[0] == '1' ||\n-\t\t str[0] == '2') {\n-\t\t\t/* T = line type, XY = status, SSSS = submodule state */\n-\t\t\tif (len < strlen(\"T XY SSSS\"))\n-\t\t\t\tBUG(\"invalid status --porcelain=2 line %s\",\n-\t\t\t\t str);\n-\n-\t\t\tif (str[5] == 'S' && str[8] == 'U')\n-\t\t\t\t/* nested untracked file */\n-\t\t\t\tdirty_submodule |= DIRTY_SUBMODULE_UNTRACKED;\n-\n-\t\t\tif (str[0] == 'u' ||\n-\t\t\t str[0] == '2' ||\n-\t\t\t memcmp(str + 5, \"S..U\", 4))\n-\t\t\t\t/* other change */\n-\t\t\t\tdirty_submodule |= DIRTY_SUBMODULE_MODIFIED;\n-\t\t}\n-\n-\t\tif ((dirty_submodule & DIRTY_SUBMODULE_MODIFIED) &&\n-\t\t ((dirty_submodule & DIRTY_SUBMODULE_UNTRACKED) ||\n-\t\t ignore_untracked)) {\n-\t\t\t/*\n-\t\t\t * We're not interested in any further information from\n-\t\t\t * the child any more, neither output nor its exit code.\n-\t\t\t */\n-\t\t\tignore_cp_exit_code = 1;\n+\t\tignore_cp_exit_code = parse_status_porcelain(str, len, &dirty_submodule,\n+\t\t\t\t\t\t\t ignore_untracked);\n+\t\tif (ignore_cp_exit_code)\n \t\t\tbreak;\n-\t\t}\n \t}\n \tfclose(fp);\n \n-- \n2.40.0.rc0.216.gc4246ad0f0-goog\n\n"},{"id":"472939","messageId":"20230302220251.1474923-5-calvinwan@google.com","threadId":"59034","inReplyTo":"20230302215237.1473444-1-calvinwan@google.com","subject":"[PATCH v9 5/6] diff-lib: refactor out diff_change logic","fromName":"Calvin Wan","fromEmail":"calvinwan@google.com","sentAt":"2023-03-02T22:02:50Z","receivedAt":"2023-03-02T22:03:39Z","isPatch":true,"sender":{"key":"calvinwan@google.com","avatar":"https://avatars.githubusercontent.com/u/92547554?v=4"},"body":"In run_diff_files, there is logic that records the diff and updates\nrelevant bits at the end of each entry iteration. Refactor out that\nlogic into a helper function so a future patch can call it.\n\nSigned-off-by: Calvin Wan \n---\n diff-lib.c | 48 +++++++++++++++++++++++++++++++-----------------\n 1 file changed, 31 insertions(+), 17 deletions(-)\n\ndiff --git a/diff-lib.c b/diff-lib.c\nindex dec040c366..744ae98a69 100644\n--- a/diff-lib.c\n+++ b/diff-lib.c\n@@ -88,6 +88,34 @@ static int match_stat_with_submodule(struct diff_options *diffopt,\n \treturn changed;\n }\n \n+/**\n+ * Records diff_change if there is a change in the entry from run_diff_files.\n+ * If there is no change, then the cache entry is marked CE_UPTODATE and\n+ * CE_FSMONITOR_VALID. If there is no change and the find_copies_harder flag\n+ * is not set, then the function returns early.\n+ */\n+static void record_file_diff(struct diff_options *options, unsigned newmode,\n+\t\t\t unsigned dirty_submodule, int changed,\n+\t\t\t struct index_state *istate,\n+\t\t\t struct cache_entry *ce)\n+{\n+\tunsigned int oldmode;\n+\tconst struct object_id *old_oid, *new_oid;\n+\n+\tif (!changed && !dirty_submodule) {\n+\t\tce_mark_uptodate(ce);\n+\t\tmark_fsmonitor_valid(istate, ce);\n+\t\tif (!options->flags.find_copies_harder)\n+\t\t\treturn;\n+\t}\n+\toldmode = ce->ce_mode;\n+\told_oid = &ce->oid;\n+\tnew_oid = changed ? null_oid() : &ce->oid;\n+\tdiff_change(options, oldmode, newmode, old_oid, new_oid,\n+\t\t !is_null_oid(old_oid), !is_null_oid(new_oid),\n+\t\t ce->name, 0, dirty_submodule);\n+}\n+\n int run_diff_files(struct rev_info *revs, unsigned int option)\n {\n \tint entries, i;\n@@ -105,11 +133,10 @@ int run_diff_files(struct rev_info *revs, unsigned int option)\n \t\tdiff_unmerged_stage = 2;\n \tentries = istate->cache_nr;\n \tfor (i = 0; i < entries; i++) {\n-\t\tunsigned int oldmode, newmode;\n+\t\tunsigned int newmode;\n \t\tstruct cache_entry *ce = istate->cache[i];\n \t\tint changed;\n \t\tunsigned dirty_submodule = 0;\n-\t\tconst struct object_id *old_oid, *new_oid;\n \n \t\tif (diff_can_quit_early(&revs->diffopt))\n \t\t\tbreak;\n@@ -245,21 +272,8 @@ int run_diff_files(struct rev_info *revs, unsigned int option)\n \t\t\tnewmode = ce_mode_from_stat(ce, st.st_mode);\n \t\t}\n \n-\t\tif (!changed && !dirty_submodule) {\n-\t\t\tce_mark_uptodate(ce);\n-\t\t\tmark_fsmonitor_valid(istate, ce);\n-\t\t\tif (!revs->diffopt.flags.find_copies_harder)\n-\t\t\t\tcontinue;\n-\t\t}\n-\t\toldmode = ce->ce_mode;\n-\t\told_oid = &ce->oid;\n-\t\tnew_oid = changed ? null_oid() : &ce->oid;\n-\t\tdiff_change(&revs->diffopt, oldmode, newmode,\n-\t\t\t old_oid, new_oid,\n-\t\t\t !is_null_oid(old_oid),\n-\t\t\t !is_null_oid(new_oid),\n-\t\t\t ce->name, 0, dirty_submodule);\n-\n+\t\trecord_file_diff(&revs->diffopt, newmode, dirty_submodule,\n+\t\t\t\t changed, istate, ce);\n \t}\n \tdiffcore_std(&revs->diffopt);\n \tdiff_flush(&revs->diffopt);\n-- \n2.40.0.rc0.216.gc4246ad0f0-goog\n\n"},{"id":"472940","messageId":"20230302220251.1474923-4-calvinwan@google.com","threadId":"59034","inReplyTo":"20230302215237.1473444-1-calvinwan@google.com","subject":"[PATCH v9 4/6] submodule: refactor is_submodule_modified()","fromName":"Calvin Wan","fromEmail":"calvinwan@google.com","sentAt":"2023-03-02T22:02:49Z","receivedAt":"2023-03-02T22:03:41Z","isPatch":true,"sender":{"key":"calvinwan@google.com","avatar":"https://avatars.githubusercontent.com/u/92547554?v=4"},"body":"Refactor out submodule status logic and error messages that will be\nused in a future patch.\n\nSigned-off-by: Calvin Wan \n---\n submodule.c | 65 ++++++++++++++++++++++++++++++++++-------------------\n 1 file changed, 42 insertions(+), 23 deletions(-)\n\ndiff --git a/submodule.c b/submodule.c\nindex 768d4b4cd7..426074cebb 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -28,6 +28,10 @@ static int config_update_recurse_submodules = RECURSE_SUBMODULES_OFF;\n static int initialized_fetch_ref_tips;\n static struct oid_array ref_tips_before_fetch;\n static struct oid_array ref_tips_after_fetch;\n+#define STATUS_PORCELAIN_START_ERROR \\\n+\tN_(\"could not run 'git status --porcelain=2' in submodule %s\")\n+#define STATUS_PORCELAIN_FAIL_ERROR \\\n+\tN_(\"'git status --porcelain=2' failed in submodule %s\")\n \n /*\n * Check if the .gitmodules file is unmerged. Parsing of the .gitmodules file\n@@ -1870,6 +1874,40 @@ int fetch_submodules(struct repository *r,\n \treturn spf.result;\n }\n \n+static int verify_submodule_git_directory(const char *path)\n+{\n+\tconst char *git_dir;\n+\tstruct strbuf buf = STRBUF_INIT;\n+\n+\tstrbuf_addf(&buf, \"%s/.git\", path);\n+\tgit_dir = read_gitfile(buf.buf);\n+\tif (!git_dir)\n+\t\tgit_dir = buf.buf;\n+\tif (!is_git_directory(git_dir)) {\n+\t\tif (is_directory(git_dir))\n+\t\t\tdie(_(\"'%s' not recognized as a git repository\"), git_dir);\n+\t\tstrbuf_release(&buf);\n+\t\t/* The submodule is not checked out, so it is not modified */\n+\t\treturn 0;\n+\t}\n+\tstrbuf_release(&buf);\n+\treturn 1;\n+}\n+\n+static void prepare_status_porcelain(struct child_process *cp,\n+\t\t\t const char *path, int ignore_untracked)\n+{\n+\tstrvec_pushl(&cp->args, \"status\", \"--porcelain=2\", NULL);\n+\tif (ignore_untracked)\n+\t\tstrvec_push(&cp->args, \"-uno\");\n+\n+\tprepare_submodule_repo_env(&cp->env);\n+\tcp->git_cmd = 1;\n+\tcp->no_stdin = 1;\n+\tcp->out = -1;\n+\tcp->dir = path;\n+}\n+\n static int parse_status_porcelain(char *str, size_t len,\n \t\t\t\t unsigned *dirty_submodule,\n \t\t\t\t int ignore_untracked)\n@@ -1915,33 +1953,14 @@ unsigned is_submodule_modified(const char *path, int ignore_untracked)\n \tstruct strbuf buf = STRBUF_INIT;\n \tFILE *fp;\n \tunsigned dirty_submodule = 0;\n-\tconst char *git_dir;\n \tint ignore_cp_exit_code = 0;\n \n-\tstrbuf_addf(&buf, \"%s/.git\", path);\n-\tgit_dir = read_gitfile(buf.buf);\n-\tif (!git_dir)\n-\t\tgit_dir = buf.buf;\n-\tif (!is_git_directory(git_dir)) {\n-\t\tif (is_directory(git_dir))\n-\t\t\tdie(_(\"'%s' not recognized as a git repository\"), git_dir);\n-\t\tstrbuf_release(&buf);\n-\t\t/* The submodule is not checked out, so it is not modified */\n+\tif (!verify_submodule_git_directory(path))\n \t\treturn 0;\n-\t}\n-\tstrbuf_reset(&buf);\n-\n-\tstrvec_pushl(&cp.args, \"status\", \"--porcelain=2\", NULL);\n-\tif (ignore_untracked)\n-\t\tstrvec_push(&cp.args, \"-uno\");\n \n-\tprepare_submodule_repo_env(&cp.env);\n-\tcp.git_cmd = 1;\n-\tcp.no_stdin = 1;\n-\tcp.out = -1;\n-\tcp.dir = path;\n+\tprepare_status_porcelain(&cp, path, ignore_untracked);\n \tif (start_command(&cp))\n-\t\tdie(_(\"Could not run 'git status --porcelain=2' in submodule %s\"), path);\n+\t\tdie(_(STATUS_PORCELAIN_START_ERROR), path);\n \n \tfp = xfdopen(cp.out, \"r\");\n \twhile (strbuf_getwholeline(&buf, fp, '\\n') != EOF) {\n@@ -1956,7 +1975,7 @@ unsigned is_submodule_modified(const char *path, int ignore_untracked)\n \tfclose(fp);\n \n \tif (finish_command(&cp) && !ignore_cp_exit_code)\n-\t\tdie(_(\"'git status --porcelain=2' failed in submodule %s\"), path);\n+\t\tdie(_(STATUS_PORCELAIN_FAIL_ERROR), path);\n \n \tstrbuf_release(&buf);\n \treturn dirty_submodule;\n-- \n2.40.0.rc0.216.gc4246ad0f0-goog\n\n"},{"id":"472941","messageId":"20230302220251.1474923-6-calvinwan@google.com","threadId":"59034","inReplyTo":"20230302215237.1473444-1-calvinwan@google.com","subject":"[PATCH v9 6/6] diff-lib: parallelize run_diff_files for submodules","fromName":"Calvin Wan","fromEmail":"calvinwan@google.com","sentAt":"2023-03-02T22:02:51Z","receivedAt":"2023-03-02T22:03:50Z","isPatch":true,"sender":{"key":"calvinwan@google.com","avatar":"https://avatars.githubusercontent.com/u/92547554?v=4"},"body":"During the iteration of the index entries in run_diff_files, whenever a\nsubmodule is found and needs its status checked, a subprocess is spawned\nfor it. Instead of spawning the subprocess immediately and waiting for\nits completion to continue, hold onto all submodules and relevant\ninformation in a list. Then use that list to create tasks for\nrun_processes_parallel. Subprocess output is passed to\nstatus_on_stderr_output which stores it to be parsed on completion of\nthe subprocess.\n\nAdd config option submodule.diffJobs to set the maximum number of\nparallel jobs. The option defaults to 1 if unset. If set to 0, the\nnumber of jobs is set to online_cpus().\n\nSince run_diff_files is called from many different commands, I chose to\ngrab the config option in the function rather than adding variables to\nevery git command and then figuring out how to pass them all in.\n\nSigned-off-by: Calvin Wan \n---\n Documentation/config/submodule.txt | 12 +++\n diff-lib.c | 81 +++++++++++++++---\n submodule.c | 128 +++++++++++++++++++++++++++++\n submodule.h | 9 ++\n t/t4027-diff-submodule.sh | 31 +++++++\n t/t7506-status-submodule.sh | 25 ++++++\n 6 files changed, 274 insertions(+), 12 deletions(-)\n\ndiff --git a/Documentation/config/submodule.txt b/Documentation/config/submodule.txt\nindex 6490527b45..3209eb8117 100644\n--- a/Documentation/config/submodule.txt\n+++ b/Documentation/config/submodule.txt\n@@ -93,6 +93,18 @@ submodule.fetchJobs::\n \tin parallel. A value of 0 will give some reasonable default.\n \tIf unset, it defaults to 1.\n \n+submodule.diffJobs::\n+\tSpecifies how many submodules are diffed at the same time. A\n+\tpositive integer allows up to that number of submodules diffed\n+\tin parallel. A value of 0 will give some reasonable default.\n+\tIf unset, it defaults to 1. The diff operation is used by many\n+\tother git commands such as add, merge, diff, status, stash and\n+\tmore. Note that the expensive part of the diff operation is\n+\treading the index from cache or memory. Therefore multiple jobs\n+\tmay be detrimental to performance if your hardware does not\n+\tsupport parallel reads or if the number of jobs greatly exceeds\n+\tthe amount of supported reads.\n+\n submodule.alternateLocation::\n \tSpecifies how the submodules obtain alternates when submodules are\n \tcloned. Possible values are `no`, `superproject`.\ndiff --git a/diff-lib.c b/diff-lib.c\nindex 744ae98a69..7fe6ced950 100644\n--- a/diff-lib.c\n+++ b/diff-lib.c\n@@ -14,6 +14,7 @@\n #include \"dir.h\"\n #include \"fsmonitor.h\"\n #include \"commit-reach.h\"\n+#include \"config.h\"\n \n /*\n * diff-files\n@@ -65,26 +66,41 @@ static int check_removed(const struct index_state *istate, const struct cache_en\n * Return 1 when changes are detected, 0 otherwise. If the DIRTY_SUBMODULES\n * option is set, the caller does not only want to know if a submodule is\n * modified at all but wants to know all the conditions that are met (new\n- * commits, untracked content and/or modified content).\n+ * commits, untracked content and/or modified content). If\n+ * defer_submodule_status bit is set, dirty_submodule will be left to the\n+ * caller to set. defer_submodule_status can also be set to 0 in this\n+ * function if there is no need to check if the submodule is modified.\n */\n static int match_stat_with_submodule(struct diff_options *diffopt,\n \t\t\t\t const struct cache_entry *ce,\n \t\t\t\t struct stat *st, unsigned ce_option,\n-\t\t\t\t unsigned *dirty_submodule)\n+\t\t\t\t unsigned *dirty_submodule, int *defer_submodule_status,\n+\t\t\t\t unsigned *ignore_untracked)\n {\n \tint changed = ie_match_stat(diffopt->repo->index, ce, st, ce_option);\n+\tint defer = 0;\n+\n \tif (S_ISGITLINK(ce->ce_mode)) {\n \t\tstruct diff_flags orig_flags = diffopt->flags;\n \t\tif (!diffopt->flags.override_submodule_config)\n \t\t\tset_diffopt_flags_from_submodule_config(diffopt, ce->name);\n-\t\tif (diffopt->flags.ignore_submodules)\n+\t\tif (diffopt->flags.ignore_submodules) {\n \t\t\tchanged = 0;\n-\t\telse if (!diffopt->flags.ignore_dirty_submodules &&\n-\t\t\t (!changed || diffopt->flags.dirty_submodules))\n-\t\t\t*dirty_submodule = is_submodule_modified(ce->name,\n-\t\t\t\t\t\t\t\t diffopt->flags.ignore_untracked_in_submodules);\n+\t\t} else if (!diffopt->flags.ignore_dirty_submodules &&\n+\t\t\t (!changed || diffopt->flags.dirty_submodules)) {\n+\t\t\tif (defer_submodule_status && *defer_submodule_status) {\n+\t\t\t\tdefer = 1;\n+\t\t\t\t*ignore_untracked = diffopt->flags.ignore_untracked_in_submodules;\n+\t\t\t} else {\n+\t\t\t\t*dirty_submodule = is_submodule_modified(ce->name,\n+\t\t\t\t\t diffopt->flags.ignore_untracked_in_submodules);\n+\t\t\t}\n+\t\t}\n \t\tdiffopt->flags = orig_flags;\n \t}\n+\n+\tif (defer_submodule_status)\n+\t\t*defer_submodule_status = defer;\n \treturn changed;\n }\n \n@@ -124,6 +140,7 @@ int run_diff_files(struct rev_info *revs, unsigned int option)\n \t\t\t ? CE_MATCH_RACY_IS_DIRTY : 0);\n \tuint64_t start = getnanotime();\n \tstruct index_state *istate = revs->diffopt.repo->index;\n+\tstruct string_list submodules = STRING_LIST_INIT_NODUP;\n \n \tdiff_set_mnemonic_prefix(&revs->diffopt, \"i/\", \"w/\");\n \n@@ -136,7 +153,7 @@ int run_diff_files(struct rev_info *revs, unsigned int option)\n \t\tunsigned int newmode;\n \t\tstruct cache_entry *ce = istate->cache[i];\n \t\tint changed;\n-\t\tunsigned dirty_submodule = 0;\n+\t\tint defer_submodule_status = 1;\n \n \t\tif (diff_can_quit_early(&revs->diffopt))\n \t\t\tbreak;\n@@ -247,6 +264,7 @@ int run_diff_files(struct rev_info *revs, unsigned int option)\n \t\t\tnewmode = ce->ce_mode;\n \t\t} else {\n \t\t\tstruct stat st;\n+\t\t\tunsigned ignore_untracked = 0;\n \n \t\t\tchanged = check_removed(istate, ce, &st);\n \t\t\tif (changed) {\n@@ -268,13 +286,52 @@ int run_diff_files(struct rev_info *revs, unsigned int option)\n \t\t\t}\n \n \t\t\tchanged = match_stat_with_submodule(&revs->diffopt, ce, &st,\n-\t\t\t\t\t\t\t ce_option, &dirty_submodule);\n+\t\t\t\t\t\t\t ce_option, NULL,\n+\t\t\t\t\t\t\t &defer_submodule_status,\n+\t\t\t\t\t\t\t &ignore_untracked);\n \t\t\tnewmode = ce_mode_from_stat(ce, st.st_mode);\n+\t\t\tif (defer_submodule_status) {\n+\t\t\t\tstruct submodule_status_util tmp = {\n+\t\t\t\t\t.changed = changed,\n+\t\t\t\t\t.dirty_submodule = 0,\n+\t\t\t\t\t.ignore_untracked = ignore_untracked,\n+\t\t\t\t\t.newmode = newmode,\n+\t\t\t\t\t.ce = ce,\n+\t\t\t\t\t.path = ce->name,\n+\t\t\t\t};\n+\t\t\t\tstruct string_list_item *item;\n+\n+\t\t\t\titem = string_list_append(&submodules, ce->name);\n+\t\t\t\titem->util = xmalloc(sizeof(tmp));\n+\t\t\t\tmemcpy(item->util, &tmp, sizeof(tmp));\n+\t\t\t\tcontinue;\n+\t\t\t}\n \t\t}\n \n-\t\trecord_file_diff(&revs->diffopt, newmode, dirty_submodule,\n-\t\t\t\t changed, istate, ce);\n+\t\tif (!defer_submodule_status)\n+\t\t\trecord_file_diff(&revs->diffopt, newmode, 0,\n+\t\t\t\t\t changed,istate, ce);\n+\t}\n+\tif (submodules.nr) {\n+\t\tunsigned long parallel_jobs;\n+\t\tstruct string_list_item *item;\n+\n+\t\tif (git_config_get_ulong(\"submodule.diffjobs\", ¶llel_jobs))\n+\t\t\tparallel_jobs = 1;\n+\t\telse if (!parallel_jobs)\n+\t\t\tparallel_jobs = online_cpus();\n+\n+\t\tif (get_submodules_status(&submodules, parallel_jobs))\n+\t\t\tdie(_(\"submodule status failed\"));\n+\t\tfor_each_string_list_item(item, &submodules) {\n+\t\t\tstruct submodule_status_util *util = item->util;\n+\n+\t\t\trecord_file_diff(&revs->diffopt, util->newmode,\n+\t\t\t\t\t util->dirty_submodule, util->changed,\n+\t\t\t\t\t istate, util->ce);\n+\t\t}\n \t}\n+\tstring_list_clear(&submodules, 1);\n \tdiffcore_std(&revs->diffopt);\n \tdiff_flush(&revs->diffopt);\n \ttrace_performance_since(start, \"diff-files\");\n@@ -322,7 +379,7 @@ static int get_stat_data(const struct index_state *istate,\n \t\t\treturn -1;\n \t\t}\n \t\tchanged = match_stat_with_submodule(diffopt, ce, &st,\n-\t\t\t\t\t\t 0, dirty_submodule);\n+\t\t\t\t\t\t 0, dirty_submodule, NULL, NULL);\n \t\tif (changed) {\n \t\t\tmode = ce_mode_from_stat(ce, st.st_mode);\n \t\t\toid = null_oid();\ndiff --git a/submodule.c b/submodule.c\nindex 426074cebb..6f6e150a3f 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -1373,6 +1373,13 @@ int submodule_touches_in_range(struct repository *r,\n \treturn ret;\n }\n \n+struct submodule_parallel_status {\n+\tsize_t index_count;\n+\tint result;\n+\n+\tstruct string_list *submodule_names;\n+};\n+\n struct submodule_parallel_fetch {\n \t/*\n \t * The index of the last index entry processed by\n@@ -1455,6 +1462,12 @@ struct fetch_task {\n \tstruct oid_array *commits; /* Ensure these commits are fetched */\n };\n \n+struct status_task {\n+\tconst char *path;\n+\tstruct strbuf out;\n+\tint ignore_untracked;\n+};\n+\n /**\n * When a submodule is not defined in .gitmodules, we cannot access it\n * via the regular submodule-config. Create a fake submodule, which we can\n@@ -1981,6 +1994,121 @@ unsigned is_submodule_modified(const char *path, int ignore_untracked)\n \treturn dirty_submodule;\n }\n \n+static struct status_task *\n+get_status_task_from_index(struct submodule_parallel_status *sps,\n+\t\t\t struct strbuf *err)\n+{\n+\tfor (; sps->index_count < sps->submodule_names->nr; sps->index_count++) {\n+\t\tstruct submodule_status_util *util = sps->submodule_names->items[sps->index_count].util;\n+\t\tstruct status_task *task;\n+\n+\t\tif (!verify_submodule_git_directory(util->path))\n+\t\t\tcontinue;\n+\n+\t\ttask = xmalloc(sizeof(*task));\n+\t\ttask->path = util->path;\n+\t\ttask->ignore_untracked = util->ignore_untracked;\n+\t\tstrbuf_init(&task->out, 0);\n+\t\tsps->index_count++;\n+\t\treturn task;\n+\t}\n+\treturn NULL;\n+}\n+\n+static int get_next_submodule_status(struct child_process *cp,\n+\t\t\t\t struct strbuf *err, void *data,\n+\t\t\t\t void **task_cb)\n+{\n+\tstruct submodule_parallel_status *sps = data;\n+\tstruct status_task *task = get_status_task_from_index(sps, err);\n+\n+\tif (!task)\n+\t\treturn 0;\n+\n+\tchild_process_init(cp);\n+\tprepare_submodule_repo_env_in_gitdir(&cp->env);\n+\tprepare_status_porcelain(cp, task->path, task->ignore_untracked);\n+\t*task_cb = task;\n+\treturn 1;\n+}\n+\n+static int status_start_failure(struct strbuf *err,\n+\t\t\t\tvoid *cb, void *task_cb)\n+{\n+\tstruct submodule_parallel_status *sps = cb;\n+\tstruct status_task *task = task_cb;\n+\n+\tsps->result = 1;\n+\tstrbuf_addf(err, _(STATUS_PORCELAIN_START_ERROR), task->path);\n+\treturn 0;\n+}\n+\n+static void status_on_stderr_output(struct strbuf *out,\n+\t\t\t\t size_t offset,\n+\t\t\t\t void *cb, void *task_cb)\n+{\n+\tstruct status_task *task = task_cb;\n+\n+\tstrbuf_add(&task->out, out->buf + offset, out->len - offset);\n+\tstrbuf_setlen(out, offset);\n+}\n+\n+static int status_finish(int retvalue, struct strbuf *err,\n+\t\t\t void *cb, void *task_cb)\n+{\n+\tstruct submodule_parallel_status *sps = cb;\n+\tstruct status_task *task = task_cb;\n+\tstruct string_list_item *it =\n+\t\tstring_list_lookup(sps->submodule_names, task->path);\n+\tstruct submodule_status_util *util = it->util;\n+\tstruct string_list list = STRING_LIST_INIT_DUP;\n+\tstruct string_list_item *item;\n+\n+\tif (retvalue) {\n+\t\tsps->result = 1;\n+\t\tstrbuf_addf(err, _(STATUS_PORCELAIN_FAIL_ERROR), task->path);\n+\t}\n+\n+\tstring_list_split(&list, task->out.buf, '\\n', -1);\n+\tfor_each_string_list_item(item, &list) {\n+\t\tif (parse_status_porcelain(item->string,\n+\t\t\t\t\t strlen(item->string),\n+\t\t\t\t\t &util->dirty_submodule,\n+\t\t\t\t\t util->ignore_untracked))\n+\t\t\tbreak;\n+\t}\n+\tstring_list_clear(&list, 0);\n+\tstrbuf_release(&task->out);\n+\tfree(task);\n+\n+\treturn 0;\n+}\n+\n+int get_submodules_status(struct string_list *submodules,\n+\t\t\t int max_parallel_jobs)\n+{\n+\tstruct submodule_parallel_status sps = {\n+\t\t.submodule_names = submodules,\n+\t};\n+\tconst struct run_process_parallel_opts opts = {\n+\t\t.tr2_category = \"submodule\",\n+\t\t.tr2_label = \"parallel/status\",\n+\n+\t\t.processes = max_parallel_jobs,\n+\n+\t\t.get_next_task = get_next_submodule_status,\n+\t\t.start_failure = status_start_failure,\n+\t\t.on_stderr_output = status_on_stderr_output,\n+\t\t.task_finished = status_finish,\n+\t\t.data = &sps,\n+\t};\n+\n+\tstring_list_sort(sps.submodule_names);\n+\trun_processes_parallel(&opts);\n+\n+\treturn sps.result;\n+}\n+\n int submodule_uses_gitfile(const char *path)\n {\n \tstruct child_process cp = CHILD_PROCESS_INIT;\ndiff --git a/submodule.h b/submodule.h\nindex b52a4ff1e7..08d278a414 100644\n--- a/submodule.h\n+++ b/submodule.h\n@@ -41,6 +41,13 @@ struct submodule_update_strategy {\n \t.type = SM_UPDATE_UNSPECIFIED, \\\n }\n \n+struct submodule_status_util {\n+\tint changed, ignore_untracked;\n+\tunsigned dirty_submodule, newmode;\n+\tstruct cache_entry *ce;\n+\tconst char *path;\n+};\n+\n int is_gitmodules_unmerged(struct index_state *istate);\n int is_writing_gitmodules_ok(void);\n int is_staging_gitmodules_ok(struct index_state *istate);\n@@ -94,6 +101,8 @@ int fetch_submodules(struct repository *r,\n \t\t int command_line_option,\n \t\t int default_option,\n \t\t int quiet, int max_parallel_jobs);\n+int get_submodules_status(struct string_list *submodules,\n+\t\t\t int max_parallel_jobs);\n unsigned is_submodule_modified(const char *path, int ignore_untracked);\n int submodule_uses_gitfile(const char *path);\n \ndiff --git a/t/t4027-diff-submodule.sh b/t/t4027-diff-submodule.sh\nindex 40164ae07d..1c747cc325 100755\n--- a/t/t4027-diff-submodule.sh\n+++ b/t/t4027-diff-submodule.sh\n@@ -34,6 +34,25 @@ test_expect_success setup '\n \tsubtip=$3 subprev=$2\n '\n \n+test_expect_success 'diff in superproject with submodules respects parallel settings' '\n+\ttest_when_finished \"rm -f trace.out\" &&\n+\t(\n+\t\tGIT_TRACE=$(pwd)/trace.out git diff &&\n+\t\tgrep \"1 tasks\" trace.out &&\n+\t\t>trace.out &&\n+\n+\t\tgit config submodule.diffJobs 8 &&\n+\t\tGIT_TRACE=$(pwd)/trace.out git diff &&\n+\t\tgrep \"8 tasks\" trace.out &&\n+\t\t>trace.out &&\n+\n+\t\tGIT_TRACE=$(pwd)/trace.out git -c submodule.diffJobs=0 diff &&\n+\t\tgrep \"preparing to run up to [0-9]* tasks\" trace.out &&\n+\t\t! grep \"up to 0 tasks\" trace.out &&\n+\t\t>trace.out\n+\t)\n+'\n+\n test_expect_success 'git diff --raw HEAD' '\n \thexsz=$(test_oid hexsz) &&\n \tgit diff --raw --abbrev=$hexsz HEAD >actual &&\n@@ -70,6 +89,18 @@ test_expect_success 'git diff HEAD with dirty submodule (work tree)' '\n \ttest_cmp expect.body actual.body\n '\n \n+test_expect_success 'git diff HEAD with dirty submodule (work tree, parallel)' '\n+\t(\n+\t\tcd sub &&\n+\t\tgit reset --hard &&\n+\t\techo >>world\n+\t) &&\n+\tgit -c submodule.diffJobs=8 diff HEAD >actual &&\n+\tsed -e \"1,/^@@/d\" actual >actual.body &&\n+\texpect_from_to >expect.body $subtip $subprev-dirty &&\n+\ttest_cmp expect.body actual.body\n+'\n+\n test_expect_success 'git diff HEAD with dirty submodule (index)' '\n \t(\n \t\tcd sub &&\ndiff --git a/t/t7506-status-submodule.sh b/t/t7506-status-submodule.sh\nindex d050091345..7da64e4c4c 100755\n--- a/t/t7506-status-submodule.sh\n+++ b/t/t7506-status-submodule.sh\n@@ -412,4 +412,29 @@ test_expect_success 'status with added file in nested submodule (short)' '\n \tEOF\n '\n \n+test_expect_success 'status in superproject with submodules respects parallel settings' '\n+\ttest_when_finished \"rm -f trace.out\" &&\n+\t(\n+\t\tGIT_TRACE=$(pwd)/trace.out git status &&\n+\t\tgrep \"1 tasks\" trace.out &&\n+\t\t>trace.out &&\n+\n+\t\tgit config submodule.diffJobs 8 &&\n+\t\tGIT_TRACE=$(pwd)/trace.out git status &&\n+\t\tgrep \"8 tasks\" trace.out &&\n+\t\t>trace.out &&\n+\n+\t\tGIT_TRACE=$(pwd)/trace.out git -c submodule.diffJobs=0 status &&\n+\t\tgrep \"preparing to run up to [0-9]* tasks\" trace.out &&\n+\t\t! grep \"up to 0 tasks\" trace.out &&\n+\t\t>trace.out\n+\t)\n+'\n+\n+test_expect_success 'status in superproject with submodules (parallel)' '\n+\tgit -C super status --porcelain >output &&\n+\tgit -C super -c submodule.diffJobs=8 status --porcelain >output_parallel &&\n+\tdiff output output_parallel\n+'\n+\n test_done\n-- \n2.40.0.rc0.216.gc4246ad0f0-goog\n\n"},{"id":"472947","messageId":"xmqqv8jiheao.fsf@gitster.g","threadId":"59034","inReplyTo":"20230302220251.1474923-2-calvinwan@google.com","subject":"Re: [PATCH v9 2/6] submodule: rename strbuf variable","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-03-03T00:25:03Z","receivedAt":"2023-03-03T00:25:08Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Calvin Wan writes:\n\n> A prepatory change for a future patch that moves the status parsing\n> logic to a separate function.\n>\n> Signed-off-by: Calvin Wan \n> ---\n> submodule.c | 23 +++++++++++++----------\n> 1 file changed, 13 insertions(+), 10 deletions(-)\n\n> Subject: Re: [PATCH v9 2/6] submodule: rename strbuf variable\n\nWhat strbuf variable renamed to what?\n\nI have a feeling that squashing this and 3/6 into a single patch,\nand pass buf.buf and buf.len to the new helper function without\nintroducing an intermediate variables in the caller, would make the\nresulting code easier to follow.\n\nIn any case, nice factoring out of a useful helper function.\n\n> diff --git a/submodule.c b/submodule.c\n> index fae24ef34a..faf37c1101 100644\n> --- a/submodule.c\n> +++ b/submodule.c\n> @@ -1906,25 +1906,28 @@ unsigned is_submodule_modified(const char *path, int ignore_untracked)\n> \n> \tfp = xfdopen(cp.out, \"r\");\n> \twhile (strbuf_getwholeline(&buf, fp, '\\n') != EOF) {\n> +\t\tchar *str = buf.buf;\n> +\t\tconst size_t len = buf.len;\n> +\n> \t\t/* regular untracked files */\n> -\t\tif (buf.buf[0] == '?')\n> +\t\tif (str[0] == '?')\n> \t\t\tdirty_submodule |= DIRTY_SUBMODULE_UNTRACKED;\n> \n> -\t\tif (buf.buf[0] == 'u' ||\n> -\t\t buf.buf[0] == '1' ||\n> -\t\t buf.buf[0] == '2') {\n> +\t\tif (str[0] == 'u' ||\n> +\t\t str[0] == '1' ||\n> +\t\t str[0] == '2') {\n> \t\t\t/* T = line type, XY = status, SSSS = submodule state */\n> -\t\t\tif (buf.len < strlen(\"T XY SSSS\"))\n> +\t\t\tif (len < strlen(\"T XY SSSS\"))\n> \t\t\t\tBUG(\"invalid status --porcelain=2 line %s\",\n> -\t\t\t\t buf.buf);\n> +\t\t\t\t str);\n> \n> -\t\t\tif (buf.buf[5] == 'S' && buf.buf[8] == 'U')\n> +\t\t\tif (str[5] == 'S' && str[8] == 'U')\n> \t\t\t\t/* nested untracked file */\n> \t\t\t\tdirty_submodule |= DIRTY_SUBMODULE_UNTRACKED;\n> \n> -\t\t\tif (buf.buf[0] == 'u' ||\n> -\t\t\t buf.buf[0] == '2' ||\n> -\t\t\t memcmp(buf.buf + 5, \"S..U\", 4))\n> +\t\t\tif (str[0] == 'u' ||\n> +\t\t\t str[0] == '2' ||\n> +\t\t\t memcmp(str + 5, \"S..U\", 4))\n> \t\t\t\t/* other change */\n> \t\t\t\tdirty_submodule |= DIRTY_SUBMODULE_MODIFIED;\n> \t\t}\n"},{"id":"473067","messageId":"CAFySSZABteFiyBYp_S7bur7_K1GkxL3A5DiTiV47iU_t8EpWKQ@mail.gmail.com","threadId":"59034","inReplyTo":"xmqqv8jiheao.fsf@gitster.g","subject":"Re: [PATCH v9 2/6] submodule: rename strbuf variable","fromName":"Calvin Wan","fromEmail":"calvinwan@google.com","sentAt":"2023-03-06T17:37:15Z","receivedAt":"2023-03-06T17:40:40Z","isPatch":true,"sender":{"key":"calvinwan@google.com","avatar":"https://avatars.githubusercontent.com/u/92547554?v=4"},"body":"On Thu, Mar 2, 2023 at 4:25 PM Junio C Hamano wrote:\n>\n> Calvin Wan writes:\n>\n> > A prepatory change for a future patch that moves the status parsing\n> > logic to a separate function.\n> >\n> > Signed-off-by: Calvin Wan \n> > ---\n> > submodule.c | 23 +++++++++++++----------\n> > 1 file changed, 13 insertions(+), 10 deletions(-)\n>\n> > Subject: Re: [PATCH v9 2/6] submodule: rename strbuf variable\n>\n> What strbuf variable renamed to what?\n>\n> I have a feeling that squashing this and 3/6 into a single patch,\n> and pass buf.buf and buf.len to the new helper function without\n> introducing an intermediate variables in the caller, would make the\n> resulting code easier to follow.\n>\n> In any case, nice factoring out of a useful helper function.\n>\n\nA much earlier version squashed those changes together, but it was\nrecommended to split those changes up; I think I am indifferent either way\nsince the refactoring is clear to me whether it is split up or not.\nhttps://lore.kernel.org/git/221012.868rllo545.gmgdl@evledraar.gmail.com/\n"},{"id":"473073","messageId":"xmqqa60pag10.fsf@gitster.g","threadId":"59034","inReplyTo":"CAFySSZABteFiyBYp_S7bur7_K1GkxL3A5DiTiV47iU_t8EpWKQ@mail.gmail.com","subject":"Re: [PATCH v9 2/6] submodule: rename strbuf variable","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-03-06T18:30:51Z","receivedAt":"2023-03-06T18:31:14Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Calvin Wan writes:\n\n> On Thu, Mar 2, 2023 at 4:25 PM Junio C Hamano wrote:\n>>\n>> Calvin Wan writes:\n>>\n>> > A prepatory change for a future patch that moves the status parsing\n>> > logic to a separate function.\n>> >\n>> > Signed-off-by: Calvin Wan \n>> > ---\n>> > submodule.c | 23 +++++++++++++----------\n>> > 1 file changed, 13 insertions(+), 10 deletions(-)\n>>\n>> > Subject: Re: [PATCH v9 2/6] submodule: rename strbuf variable\n>>\n>> What strbuf variable renamed to what?\n>>\n>> I have a feeling that squashing this and 3/6 into a single patch,\n>> and pass buf.buf and buf.len to the new helper function without\n>> introducing an intermediate variables in the caller, would make the\n>> resulting code easier to follow.\n>>\n>> In any case, nice factoring out of a useful helper function.\n>>\n>\n> A much earlier version squashed those changes together, but it was\n> recommended to split those changes up; I think I am indifferent either way\n> since the refactoring is clear to me whether it is split up or not.\n> https://lore.kernel.org/git/221012.868rllo545.gmgdl@evledraar.gmail.com/\n\nI am indifferent, either, but with or without them squashed into a\nsingle patch, \"rename strbuf\" would not be how you would describe\nthe value of this refactoring, which is to make the interface not\ndepend on strbuf. Some callers may have separate pair\nthat is not in strbuf, and with the current interface they are\nforced to wrap the pair in a throw-away strbuf which is not nice.\n\nAnd squashing them together into a single patch, it becomes a lot\nclear what the point of these two steps combined is.\n"},{"id":"473077","messageId":"CAFySSZDk05m6gU5-V1R+y3YnQ5PPduVW54+_gjBwD0rmacsLsw@mail.gmail.com","threadId":"59034","inReplyTo":"xmqqa60pag10.fsf@gitster.g","subject":"Re: [PATCH v9 2/6] submodule: rename strbuf variable","fromName":"Calvin Wan","fromEmail":"calvinwan@google.com","sentAt":"2023-03-06T19:00:49Z","receivedAt":"2023-03-06T19:01:28Z","isPatch":true,"sender":{"key":"calvinwan@google.com","avatar":"https://avatars.githubusercontent.com/u/92547554?v=4"},"body":"On Mon, Mar 6, 2023 at 10:30 AM Junio C Hamano wrote:\n>\n> Calvin Wan writes:\n>\n> > On Thu, Mar 2, 2023 at 4:25 PM Junio C Hamano wrote:\n> >>\n> >> Calvin Wan writes:\n> >>\n> >> > A prepatory change for a future patch that moves the status parsing\n> >> > logic to a separate function.\n> >> >\n> >> > Signed-off-by: Calvin Wan \n> >> > ---\n> >> > submodule.c | 23 +++++++++++++----------\n> >> > 1 file changed, 13 insertions(+), 10 deletions(-)\n> >>\n> >> > Subject: Re: [PATCH v9 2/6] submodule: rename strbuf variable\n> >>\n> >> What strbuf variable renamed to what?\n> >>\n> >> I have a feeling that squashing this and 3/6 into a single patch,\n> >> and pass buf.buf and buf.len to the new helper function without\n> >> introducing an intermediate variables in the caller, would make the\n> >> resulting code easier to follow.\n> >>\n> >> In any case, nice factoring out of a useful helper function.\n> >>\n> >\n> > A much earlier version squashed those changes together, but it was\n> > recommended to split those changes up; I think I am indifferent either way\n> > since the refactoring is clear to me whether it is split up or not.\n> > https://lore.kernel.org/git/221012.868rllo545.gmgdl@evledraar.gmail.com/\n>\n> I am indifferent, either, but with or without them squashed into a\n> single patch, \"rename strbuf\" would not be how you would describe\n> the value of this refactoring, which is to make the interface not\n> depend on strbuf. Some callers may have separate pair\n> that is not in strbuf, and with the current interface they are\n> forced to wrap the pair in a throw-away strbuf which is not nice.\n\nI see what you mean here; will reword the commit message, thanks!\n"},{"id":"473114","messageId":"230307.86a60o28f7.gmgdl@evledraar.gmail.com","threadId":"59034","inReplyTo":"20230302220251.1474923-6-calvinwan@google.com","subject":"Re: [PATCH v9 6/6] diff-lib: parallelize run_diff_files for submodules","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2023-03-07T08:41:28Z","receivedAt":"2023-03-07T09:55:32Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, Mar 02 2023, Calvin Wan wrote:\n\nSome of this is stuff I probably should have noted in earlier rounds,\nsorry, but then again the diff-churn in those made it harder to review,\nnow that that's mostly out of the way (yay!) ....\n\n> +submodule.diffJobs::\n> +\tSpecifies how many submodules are diffed at the same time. A\n> +\tpositive integer allows up to that number of submodules diffed\n> +\tin parallel. A value of 0 will give some reasonable default.\n> +\tIf unset, it defaults to 1. The diff operation is used by many\n\nNit: Maybe start a new paragraph as of \"The diff...\"?\n\n> +\tother git commands such as add, merge, diff, status, stash and\n> +\tmore. Note that the expensive part of the diff operation is\n\nNit: Maybe change 'add', 'merge' etc. to linkgit:git-add[1], or quote\nthem?\n\n> +\treading the index from cache or memory. Therefore multiple jobs\n\nWith how much we conflate \"the cache\" and \"index\" saying \"the index from\ncache\" might be especially confusing. I think we can just skip \" from\ncache or memory\" here.\n\n> static int match_stat_with_submodule(struct diff_options *diffopt,\n> \t\t\t\t const struct cache_entry *ce,\n> \t\t\t\t struct stat *st, unsigned ce_option,\n> -\t\t\t\t unsigned *dirty_submodule)\n> +\t\t\t\t unsigned *dirty_submodule, int *defer_submodule_status,\n\nNit: The other one is an \"unsigned\", shouldn't \"defer_submodule_status\"\nalso be (more on this below).\n\n> +\t\t\t\t unsigned *ignore_untracked)\n> {\n> \tint changed = ie_match_stat(diffopt->repo->index, ce, st, ce_option);\n> +\tint defer = 0;\n> +\n> \tif (S_ISGITLINK(ce->ce_mode)) {\n> \t\tstruct diff_flags orig_flags = diffopt->flags;\n> \t\tif (!diffopt->flags.override_submodule_config)\n> \t\t\tset_diffopt_flags_from_submodule_config(diffopt, ce->name);\n\nThe meaty functional change here looks *much* better, thanks! I.e. this\nis pretty much what I suggested in\nhttps://lore.kernel.org/git/230208.861qn01s4g.gmgdl@evledraar.gmail.com/\n\n> -\t\tif (diffopt->flags.ignore_submodules)\n> +\t\tif (diffopt->flags.ignore_submodules) {\n\nNot worth a re-roll in itself, but FWIW I think this change would be\nmarginally easier to follow with *a* preceding refactoring change, but\nper the above &\nhttps://lore.kernel.org/git/230209.867cwrzk1l.gmgdl@evledraar.gmail.com/\nI just didn't think v7's 6/7\n(https://lore.kernel.org/git/20230207181706.363453-7-calvinwan@google.com/)\nwas what we needed there.\n\nI.e. in this case a leading change that would add these braces would\nmake this a bit easier to read...\n\n> \t\t\tchanged = 0;\n> -\t\telse if (!diffopt->flags.ignore_dirty_submodules &&\n\n...ditto this line, which would stay the same.\n\n> -\t\t\t (!changed || diffopt->flags.dirty_submodules))\n> -\t\t\t*dirty_submodule = is_submodule_modified(ce->name,\n> -\t\t\t\t\t\t\t\t diffopt->flags.ignore_untracked_in_submodules);\n\nHere you are incorrectly changing the indentation of this away from our\nusual coding style, which...\n\n> +\t\t} else if (!diffopt->flags.ignore_dirty_submodules &&\n> +\t\t\t (!changed || diffopt->flags.dirty_submodules)) {\n> +\t\t\tif (defer_submodule_status && *defer_submodule_status) {\n\nHrm, if if I remove that \"&& *defer_submodule_status\" all of our tests\npass, the only two callers of this function are one where this is NULL,\nand where it's non-NULL but pre-initilized to 1, and the caller will\ncheck if it's then flipped to 0.\n\n> +\t\t\t\tdefer = 1;\n> +\t\t\t\t*ignore_untracked = diffopt->flags.ignore_untracked_in_submodules;\n> +\t\t\t} else {\n> +\t\t\t\t*dirty_submodule = is_submodule_modified(ce->name,\n> +\t\t\t\t\t diffopt->flags.ignore_untracked_in_submodules);\n\n...needlessly inflates the diff here, at least under -w and move\ndetection, as we correctly detect the \"*dirty_submodule\" line as the\nsame, but the \"diffopt->flags\" line also has a re-indentation change\nunrelated to adding the \"else\" scope.\n\n> +\t\t\t}\n> +\t\t}\n> \t\tdiffopt->flags = orig_flags;\n> \t}\n> +\n> +\tif (defer_submodule_status)\n> +\t\t*defer_submodule_status = defer;\n\nHaving read this whole thing to the end again I think this on top would\nbe much simpler (if I'm right about it being functionally equivalent),\nand would address some of the above:\n\t\n\tdiff --git a/diff-lib.c b/diff-lib.c\n\tindex 7fe6ced9501..d5c823f512a 100644\n\t--- a/diff-lib.c\n\t+++ b/diff-lib.c\n\t@@ -78,7 +78,6 @@ static int match_stat_with_submodule(struct diff_options *diffopt,\n\t \t\t\t\t unsigned *ignore_untracked)\n\t {\n\t \tint changed = ie_match_stat(diffopt->repo->index, ce, st, ce_option);\n\t-\tint defer = 0;\n\t \n\t \tif (S_ISGITLINK(ce->ce_mode)) {\n\t \t\tstruct diff_flags orig_flags = diffopt->flags;\n\t@@ -88,8 +87,8 @@ static int match_stat_with_submodule(struct diff_options *diffopt,\n\t \t\t\tchanged = 0;\n\t \t\t} else if (!diffopt->flags.ignore_dirty_submodules &&\n\t \t\t\t (!changed || diffopt->flags.dirty_submodules)) {\n\t-\t\t\tif (defer_submodule_status && *defer_submodule_status) {\n\t-\t\t\t\tdefer = 1;\n\t+\t\t\tif (defer_submodule_status) {\n\t+\t\t\t\t*defer_submodule_status = 1;\n\t \t\t\t\t*ignore_untracked = diffopt->flags.ignore_untracked_in_submodules;\n\t \t\t\t} else {\n\t \t\t\t\t*dirty_submodule = is_submodule_modified(ce->name,\n\t@@ -99,8 +98,6 @@ static int match_stat_with_submodule(struct diff_options *diffopt,\n\t \t\tdiffopt->flags = orig_flags;\n\t \t}\n\t \n\t-\tif (defer_submodule_status)\n\t-\t\t*defer_submodule_status = defer;\n\t \treturn changed;\n\t }\n\t \n\t@@ -153,7 +150,7 @@ int run_diff_files(struct rev_info *revs, unsigned int option)\n\t \t\tunsigned int newmode;\n\t \t\tstruct cache_entry *ce = istate->cache[i];\n\t \t\tint changed;\n\t-\t\tint defer_submodule_status = 1;\n\t+\t\tint defer_submodule_status = 0;\n\t \n\t \t\tif (diff_can_quit_early(&revs->diffopt))\n\t \t\t\tbreak;\n\nWe could also just leave it, but I for one found it a bit hard to follow\nthat this interface seems to be a tri-state (NULL, set to 0, set to 1),\nbut really it's dual-state, i.e. NULL or a \"tell me to defer this\" bit.\n\t\n> \treturn changed;\n> }\n> \n> @@ -124,6 +140,7 @@ int run_diff_files(struct rev_info *revs, unsigned int option)\n> \t\t\t ? CE_MATCH_RACY_IS_DIRTY : 0);\n> \tuint64_t start = getnanotime();\n> \tstruct index_state *istate = revs->diffopt.repo->index;\n> +\tstruct string_list submodules = STRING_LIST_INIT_NODUP;\n> \n> \tdiff_set_mnemonic_prefix(&revs->diffopt, \"i/\", \"w/\");\n> \n> @@ -136,7 +153,7 @@ int run_diff_files(struct rev_info *revs, unsigned int option)\n> \t\tunsigned int newmode;\n> \t\tstruct cache_entry *ce = istate->cache[i];\n> \t\tint changed;\n> -\t\tunsigned dirty_submodule = 0;\n> +\t\tint defer_submodule_status = 1;\n\nHrm, having suggested the diff above I just noticed this now, I ended up\ninverting this, but found the \"defer_submodule_status\" name a bit odd,\ncan't we just keep \"unsigned dirty_submodule\"? (that would also address\nthe change from \"unsigned\" to \"int\" noted above, which is seeminly\nunnecessary).\n\nBut maybe I'm missing a subtlety here, and we should have \"deferred\nstatus\" as apposed to \"dirty submodule\", but in any case the new one\nlooks like it doesn't need negative values.\n\n> +\t}\n> +\tif (submodules.nr) {\n> +\t\tunsigned long parallel_jobs;\n> +\t\tstruct string_list_item *item;\n> +\n> +\t\tif (git_config_get_ulong(\"submodule.diffjobs\", ¶llel_jobs))\n> +\t\t\tparallel_jobs = 1;\n> +\t\telse if (!parallel_jobs)\n> +\t\t\tparallel_jobs = online_cpus();\n\nGiven that online_cpus() returns int the \"unsigned long\" is slightly odd\nhere, but it's because git_config_get_ulong() exist, but we have no\ngit_config_get_uint(), so this is OK (but could be cleaned up as some\n#leftoverbits).\n\n> +\t\tif (get_submodules_status(&submodules, parallel_jobs))\n> +\t\t\tdie(_(\"submodule status failed\"));\n\nHere we're adding get_submodules_status(), and returning the actual\nerror code from \"status\", but then ignoring it here, and returning 128\nfor any non-zero.\n\nI think this would be better as either:\n\n\tcode = get_submodules_status(...);\n\tdie_message(...)\n\texit(code);\n\nOr to just have the function itself return !!status, i.e. a \"ok\" or \"not\nok\".\n\nAdmittedly a nit, but I have spent quite a bit of time chasing down\nvarious exit-code losses in the submodule code, and it would be nice if\nwe just carry the code up, or more explicitly ignore it, but don't add\ncode that seems to care about it, but really doesn't.\n\nI also changed this \"die\" to a \"BUG\" and our tests passed, so we have no\ntests for when \"status\" failed, will such a thing even happen in\npractice?\n\n> +\t\tfor_each_string_list_item(item, &submodules) {\n> +\t\t\tstruct submodule_status_util *util = item->util;\n> +\n> +\t\t\trecord_file_diff(&revs->diffopt, util->newmode,\n> +\t\t\t\t\t util->dirty_submodule, util->changed,\n> +\t\t\t\t\t istate, util->ce);\n> +\t\t}\n> \t}\n> +\tstring_list_clear(&submodules, 1);\n> \tdiffcore_std(&revs->diffopt);\n> \tdiff_flush(&revs->diffopt);\n> \ttrace_performance_since(start, \"diff-files\");\n> @@ -322,7 +379,7 @@ static int get_stat_data(const struct index_state *istate,\n> \t\t\treturn -1;\n> \t\t}\n> \t\tchanged = match_stat_with_submodule(diffopt, ce, &st,\n> -\t\t\t\t\t\t 0, dirty_submodule);\n> +\t\t\t\t\t\t 0, dirty_submodule, NULL, NULL);\n> \t\tif (changed) {\n> \t\t\tmode = ce_mode_from_stat(ce, st.st_mode);\n> \t\t\toid = null_oid();\n> diff --git a/submodule.c b/submodule.c\n> index 426074cebb..6f6e150a3f 100644\n> --- a/submodule.c\n> +++ b/submodule.c\n> @@ -1373,6 +1373,13 @@ int submodule_touches_in_range(struct repository *r,\n> \treturn ret;\n> }\n> \n> +struct submodule_parallel_status {\n> +\tsize_t index_count;\n> +\tint result;\n> +\n> +\tstruct string_list *submodule_names;\n> +};\n\nHrm, actually reading a bit more I think part of my comments above are\nincorrect, i.e. this \"result\" seems like an exit code, but really in the\nguts of the API we're ignoring the actual code we get, and just setting\nthis to 1.\n\nPer the above I think it might be OK to ignore the exit code (or not),\nbut I really wish we did this more explicitly, e.g. if you want to\nignore it call this something like \"failed\", not \"result\", and make it\nan \"unsigned int failed:1\" to firmly indicate that it's a boolean at the\nAPI level.\n\n> +struct status_task {\n> +\tconst char *path;\n\nI think we should call this \"ce_path\", but more on that below.\n\n> +\tstruct strbuf out;\n> +\tint ignore_untracked;\n\nContinued type mismatch commentary: Elsewhere in this diff this is\n\"unsigned\", and this compiles for me if I make it \"unsigned int\nignore_untracked:1\", so let's set it to such a flag instead?\n\n> +static int status_finish(int retvalue, struct strbuf *err,\n> +\t\t\t void *cb, void *task_cb)\n> +{\n> +\tstruct submodule_parallel_status *sps = cb;\n> +\tstruct status_task *task = task_cb;\n> +\tstruct string_list_item *it =\n> +\t\tstring_list_lookup(sps->submodule_names, task->path);\n> +\tstruct submodule_status_util *util = it->util;\n> +\tstruct string_list list = STRING_LIST_INIT_DUP;\n> +\tstruct string_list_item *item;\n> +\n> +\tif (retvalue) {\n> +\t\tsps->result = 1;\n> +\t\tstrbuf_addf(err, _(STATUS_PORCELAIN_FAIL_ERROR), task->path);\n> +\t}\n> +\n> +\tstring_list_split(&list, task->out.buf, '\\n', -1);\n\nI think I noted in some earlier round that taking a string and splitting\nit by \\n was a bit wasteful in the test code, but this uses the same\npattern.\n\nMaybe it's not a performance concern here either, but won't we\npotentially have to parse some very large statuses here?\n\nAside from that, I haven't tried or reviewed this bit in detail, but\nthis seems to be making things harder than they need to be. Why are we\nbuffering up all of the output into \"out\" here, only to split it by \"\\n\"\nlater on, and then consider each line as a status line?\n\nShouldn't we be allocating this string_list to begin with, and append to\nit in the \"status_on_stderr_output\" callback instead?\n\n> +\tfor_each_string_list_item(item, &list) {\n> +\t\tif (parse_status_porcelain(item->string,\n> +\t\t\t\t\t strlen(item->string),\n> +\t\t\t\t\t &util->dirty_submodule,\n> +\t\t\t\t\t util->ignore_untracked))\n\nOK, this seemingly buggy bit of error handling seems to actually be OK\non further review, because we'll BUG() out in the function if it fails,\nso the non-zero return here just means \"we're done here\".\n\n> +\t\t\tbreak;\n> +\t}\n\nStyle: drop the braces here, as this is just a for/if/body with a single\nbody line.\n\n> +int get_submodules_status(struct string_list *submodules,\n> +\t\t\t int max_parallel_jobs)\n> +{\n> +\tstruct submodule_parallel_status sps = {\n> +\t\t.submodule_names = submodules,\n> +\t};\n> +\tconst struct run_process_parallel_opts opts = {\n> +\t\t.tr2_category = \"submodule\",\n> +\t\t.tr2_label = \"parallel/status\",\n> +\n> +\t\t.processes = max_parallel_jobs,\n> +\n> +\t\t.get_next_task = get_next_submodule_status,\n> +\t\t.start_failure = status_start_failure,\n> +\t\t.on_stderr_output = status_on_stderr_output,\n> +\t\t.task_finished = status_finish,\n> +\t\t.data = &sps,\n> +\t};\n> +\n> +\tstring_list_sort(sps.submodule_names);\n> +\trun_processes_parallel(&opts);\n> +\n> +\treturn sps.result;\n\nAll OK, except as noted above the \"result\" here is just \"did we fail?\".\n\n> +}\n> +\n> int submodule_uses_gitfile(const char *path)\n> {\n> \tstruct child_process cp = CHILD_PROCESS_INIT;\n> diff --git a/submodule.h b/submodule.h\n> index b52a4ff1e7..08d278a414 100644\n> --- a/submodule.h\n> +++ b/submodule.h\n> @@ -41,6 +41,13 @@ struct submodule_update_strategy {\n> \t.type = SM_UPDATE_UNSPECIFIED, \\\n> }\n> \n> +struct submodule_status_util {\n> +\tint changed, ignore_untracked;\n> +\tunsigned dirty_submodule, newmode;\n> +\tstruct cache_entry *ce;\n> +\tconst char *path;\n\nRe \"ce_path\" above: What's the point of adding a \"path\" here if we\nalready have \"ce\"? You just seem to assign \"path\" to \"ce->name\"\nalways. I tried this fix-up on top & it worked:\n\t\n\tdiff --git a/diff-lib.c b/diff-lib.c\n\tindex d5c823f512a..39d8179f0ed 100644\n\t--- a/diff-lib.c\n\t+++ b/diff-lib.c\n\t@@ -294,7 +294,6 @@ int run_diff_files(struct rev_info *revs, unsigned int option)\n\t \t\t\t\t\t.ignore_untracked = ignore_untracked,\n\t \t\t\t\t\t.newmode = newmode,\n\t \t\t\t\t\t.ce = ce,\n\t-\t\t\t\t\t.path = ce->name,\n\t \t\t\t\t};\n\t \t\t\t\tstruct string_list_item *item;\n\t \n\tdiff --git a/submodule.c b/submodule.c\n\tindex 3eba00f1533..c220d85815a 100644\n\t--- a/submodule.c\n\t+++ b/submodule.c\n\t@@ -2002,11 +2002,11 @@ get_status_task_from_index(struct submodule_parallel_status *sps,\n\t \t\tstruct submodule_status_util *util = sps->submodule_names->items[sps->index_count].util;\n\t \t\tstruct status_task *task;\n\t \n\t-\t\tif (!verify_submodule_git_directory(util->path))\n\t+\t\tif (!verify_submodule_git_directory(util->ce->name))\n\t \t\t\tcontinue;\n\t \n\t \t\ttask = xmalloc(sizeof(*task));\n\t-\t\ttask->path = util->path;\n\t+\t\ttask->path = util->ce->name;\n\t \t\ttask->ignore_untracked = util->ignore_untracked;\n\t \t\tstrbuf_init(&task->out, 0);\n\t \t\tsps->index_count++;\n\tdiff --git a/submodule.h b/submodule.h\n\tindex 3b6abca05cd..3427c495573 100644\n\t--- a/submodule.h\n\t+++ b/submodule.h\n\t@@ -45,7 +45,6 @@ struct submodule_status_util {\n\t \tint changed, ignore_untracked;\n\t \tunsigned dirty_submodule, newmode;\n\t \tstruct cache_entry *ce;\n\t-\tconst char *path;\n\t };\n\t \n\t int is_gitmodules_unmerged(struct index_state *istate);\n\nI'd be all for actually narrowing the scope of data we get in general,\ni.e. do we need all of the \"ce\" members? I didn't check, but doing this\njust seems like needless duplication.\n\n> @@ -94,6 +101,8 @@ int fetch_submodules(struct repository *r,\n> \t\t int command_line_option,\n> \t\t int default_option,\n> \t\t int quiet, int max_parallel_jobs);\n> +int get_submodules_status(struct string_list *submodules,\n> +\t\t\t int max_parallel_jobs);\n\nIt would be nice to get some API docs for the new function, re its\n\"result\" behavior etc. noted above\n\n> unsigned is_submodule_modified(const char *path, int ignore_untracked);\n> int submodule_uses_gitfile(const char *path);\n> \n> diff --git a/t/t4027-diff-submodule.sh b/t/t4027-diff-submodule.sh\n> index 40164ae07d..1c747cc325 100755\n> --- a/t/t4027-diff-submodule.sh\n> +++ b/t/t4027-diff-submodule.sh\n> @@ -34,6 +34,25 @@ test_expect_success setup '\n> \tsubtip=$3 subprev=$2\n> '\n> \n> +test_expect_success 'diff in superproject with submodules respects parallel settings' '\n> +\ttest_when_finished \"rm -f trace.out\" &&\n> +\t(\n> +\t\tGIT_TRACE=$(pwd)/trace.out git diff &&\n> +\t\tgrep \"1 tasks\" trace.out &&\n> +\t\t>trace.out &&\n> +\n> +\t\tgit config submodule.diffJobs 8 &&\n> +\t\tGIT_TRACE=$(pwd)/trace.out git diff &&\n> +\t\tgrep \"8 tasks\" trace.out &&\n> +\t\t>trace.out &&\n> +\n> +\t\tGIT_TRACE=$(pwd)/trace.out git -c submodule.diffJobs=0 diff &&\n> +\t\tgrep \"preparing to run up to [0-9]* tasks\" trace.out &&\n> +\t\t! grep \"up to 0 tasks\" trace.out &&\n> +\t\t>trace.out\n> +\t)\n> +'\n> +\n> test_expect_success 'git diff --raw HEAD' '\n> \thexsz=$(test_oid hexsz) &&\n> \tgit diff --raw --abbrev=$hexsz HEAD >actual &&\n> @@ -70,6 +89,18 @@ test_expect_success 'git diff HEAD with dirty submodule (work tree)' '\n> \ttest_cmp expect.body actual.body\n> '\n> \n> +test_expect_success 'git diff HEAD with dirty submodule (work tree, parallel)' '\n> +\t(\n> +\t\tcd sub &&\n> +\t\tgit reset --hard &&\n> +\t\techo >>world\n> +\t) &&\n> +\tgit -c submodule.diffJobs=8 diff HEAD >actual &&\n> +\tsed -e \"1,/^@@/d\" actual >actual.body &&\n> +\texpect_from_to >expect.body $subtip $subprev-dirty &&\n> +\ttest_cmp expect.body actual.body\n> +'\n> +\n> test_expect_success 'git diff HEAD with dirty submodule (index)' '\n> \t(\n> \t\tcd sub &&\n> diff --git a/t/t7506-status-submodule.sh b/t/t7506-status-submodule.sh\n> index d050091345..7da64e4c4c 100755\n> --- a/t/t7506-status-submodule.sh\n> +++ b/t/t7506-status-submodule.sh\n> @@ -412,4 +412,29 @@ test_expect_success 'status with added file in nested submodule (short)' '\n> \tEOF\n> '\n> \n> +test_expect_success 'status in superproject with submodules respects parallel settings' '\n> +\ttest_when_finished \"rm -f trace.out\" &&\n> +\t(\n> +\t\tGIT_TRACE=$(pwd)/trace.out git status &&\n> +\t\tgrep \"1 tasks\" trace.out &&\n> +\t\t>trace.out &&\n> +\n> +\t\tgit config submodule.diffJobs 8 &&\n> +\t\tGIT_TRACE=$(pwd)/trace.out git status &&\n> +\t\tgrep \"8 tasks\" trace.out &&\n> +\t\t>trace.out &&\n> +\n> +\t\tGIT_TRACE=$(pwd)/trace.out git -c submodule.diffJobs=0 status &&\n> +\t\tgrep \"preparing to run up to [0-9]* tasks\" trace.out &&\n> +\t\t! grep \"up to 0 tasks\" trace.out &&\n> +\t\t>trace.out\n> +\t)\n> +'\n> +\n> +test_expect_success 'status in superproject with submodules (parallel)' '\n> +\tgit -C super status --porcelain >output &&\n> +\tgit -C super -c submodule.diffJobs=8 status --porcelain >output_parallel &&\n> +\tdiff output output_parallel\n\nShouldn't this be a \"test_cmp\" instead of \"diff\", and use \"actual\" and\n\"expect\" instead of \"output\" and \"output_parallel\"?\n\nI'd also rename the test to something like \"output with\nsubmodule.diffJobs=N equals submodule.diffJobs=1\".\n\nExcept is that even correct? Don't we need to set submodule.diffJobs=1\nexplicitly so it doesn't default to online_cpus() here? Maybe I missed\nan earlier config setup...\n"},{"id":"473116","messageId":"230307.865ybc273g.gmgdl@evledraar.gmail.com","threadId":"59034","inReplyTo":"20230302220251.1474923-6-calvinwan@google.com","subject":"Re: [PATCH v9 6/6] diff-lib: parallelize run_diff_files for submodules","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2023-03-07T10:21:10Z","receivedAt":"2023-03-07T10:23:35Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, Mar 02 2023, Calvin Wan wrote:\n\n> +\t\tif (git_config_get_ulong(\"submodule.diffjobs\", ¶llel_jobs))\n> +\t\t\tparallel_jobs = 1;\n\nSomething I missed when eyeballing this in my just-sent review, here we\nhave a \"revs->repo\" already, so let's not fall back on \"the_repository\",\nbut use it. I think you want this as a fix-up:\n\t\n\tdiff --git a/diff-lib.c b/diff-lib.c\n\tindex 925d64ff58c..ec8a0f98085 100644\n\t--- a/diff-lib.c\n\t+++ b/diff-lib.c\n\t@@ -312,7 +312,8 @@ int run_diff_files(struct rev_info *revs, unsigned int option)\n\t \t\tunsigned long parallel_jobs;\n\t \t\tstruct string_list_item *item;\n\t \n\t-\t\tif (git_config_get_ulong(\"submodule.diffjobs\", ¶llel_jobs))\n\t+\t\tif (repo_config_get_ulong(revs->repo, \"submodule.diffjobs\",\n\t+\t\t\t\t\t ¶llel_jobs))\n\t \t\t\tparallel_jobs = 1;\n\t \t\telse if (!parallel_jobs)\n\t \t\t\tparallel_jobs = online_cpus();\n"},{"id":"473131","messageId":"xmqqv8jczbs0.fsf@gitster.g","threadId":"59034","inReplyTo":"230307.865ybc273g.gmgdl@evledraar.gmail.com","subject":"Re: [PATCH v9 6/6] diff-lib: parallelize run_diff_files for submodules","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-03-07T17:55:43Z","receivedAt":"2023-03-07T18:03:07Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ævar Arnfjörð Bjarmason writes:\n\n> On Thu, Mar 02 2023, Calvin Wan wrote:\n>\n>> +\t\tif (git_config_get_ulong(\"submodule.diffjobs\", ¶llel_jobs))\n>> +\t\t\tparallel_jobs = 1;\n>\n> Something I missed when eyeballing this in my just-sent review, here we\n> have a \"revs->repo\" already, so let's not fall back on \"the_repository\",\n> but use it. I think you want this as a fix-up:\n> \t\n> \tdiff --git a/diff-lib.c b/diff-lib.c\n> \tindex 925d64ff58c..ec8a0f98085 100644\n> \t--- a/diff-lib.c\n> \t+++ b/diff-lib.c\n> \t@@ -312,7 +312,8 @@ int run_diff_files(struct rev_info *revs, unsigned int option)\n> \t \t\tunsigned long parallel_jobs;\n> \t \t\tstruct string_list_item *item;\n> \t \n> \t-\t\tif (git_config_get_ulong(\"submodule.diffjobs\", ¶llel_jobs))\n> \t+\t\tif (repo_config_get_ulong(revs->repo, \"submodule.diffjobs\",\n> \t+\t\t\t\t\t ¶llel_jobs))\n> \t \t\t\tparallel_jobs = 1;\n> \t \t\telse if (!parallel_jobs)\n> \t \t\t\tparallel_jobs = online_cpus();\n\nGood eyes. Thanks for a careful review.\n"},{"id":"473620","messageId":"kl6ljzzguqss.fsf@chooglen-macbookpro.roam.corp.google.com","threadId":"59034","inReplyTo":"20230302220251.1474923-6-calvinwan@google.com","subject":"Re: [PATCH v9 6/6] diff-lib: parallelize run_diff_files for submodules","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2023-03-17T01:09:39Z","receivedAt":"2023-03-17T01:09:45Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"\nI haven't verified if the code in this version is correct or not, as I\nfound it a bit difficult to follow through the churn. After reading this\nseries again, I've established a better mental model of the code, and I\nthink there are some renames and documentation changes we can make to\nmake this clearer.\n\nUnfortunately, I think the biggest clarification would be _yet_ another\nrefactor, and I'm not sure if we actually want to bear so much churn. I\nmight do this refactor locally to see if it really is _much_ cleaner or\nnot.\n\nIf anyone has thoughts on the refactor, do chime in.\n\nCalvin Wan writes:\n\n> diff --git a/diff-lib.c b/diff-lib.c\n> index 744ae98a69..7fe6ced950 100644\n> --- a/diff-lib.c\n> +++ b/diff-lib.c\n> @@ -65,26 +66,41 @@ static int check_removed(const struct index_state *istate, const struct cache_en\n> * Return 1 when changes are detected, 0 otherwise. If the DIRTY_SUBMODULES\n> * option is set, the caller does not only want to know if a submodule is\n> * modified at all but wants to know all the conditions that are met (new\n> - * commits, untracked content and/or modified content).\n> + * commits, untracked content and/or modified content). If\n> + * defer_submodule_status bit is set, dirty_submodule will be left to the\n> + * caller to set. defer_submodule_status can also be set to 0 in this\n> + * function if there is no need to check if the submodule is modified.\n> */\n> static int match_stat_with_submodule(struct diff_options *diffopt,\n> \t\t\t\t const struct cache_entry *ce,\n> \t\t\t\t struct stat *st, unsigned ce_option,\n> -\t\t\t\t unsigned *dirty_submodule)\n> +\t\t\t\t unsigned *dirty_submodule, int *defer_submodule_status,\n> +\t\t\t\t unsigned *ignore_untracked)\n> {\n> \tint changed = ie_match_stat(diffopt->repo->index, ce, st, ce_option);\n> +\tint defer = 0;\n> +\n> \tif (S_ISGITLINK(ce->ce_mode)) {\n> \t\tstruct diff_flags orig_flags = diffopt->flags;\n> \t\tif (!diffopt->flags.override_submodule_config)\n> \t\t\tset_diffopt_flags_from_submodule_config(diffopt, ce->name);\n> -\t\tif (diffopt->flags.ignore_submodules)\n> +\t\tif (diffopt->flags.ignore_submodules) {\n> \t\t\tchanged = 0;\n> -\t\telse if (!diffopt->flags.ignore_dirty_submodules &&\n> -\t\t\t (!changed || diffopt->flags.dirty_submodules))\n> -\t\t\t*dirty_submodule = is_submodule_modified(ce->name,\n> -\t\t\t\t\t\t\t\t diffopt->flags.ignore_untracked_in_submodules);\n> +\t\t} else if (!diffopt->flags.ignore_dirty_submodules &&\n> +\t\t\t (!changed || diffopt->flags.dirty_submodules)) {\n> +\t\t\tif (defer_submodule_status && *defer_submodule_status) {\n> +\t\t\t\tdefer = 1;\n> +\t\t\t\t*ignore_untracked = diffopt->flags.ignore_untracked_in_submodules;\n> +\t\t\t} else {\n> +\t\t\t\t*dirty_submodule = is_submodule_modified(ce->name,\n> +\t\t\t\t\t diffopt->flags.ignore_untracked_in_submodules);\n> +\t\t\t}\n> +\t\t}\n> \t\tdiffopt->flags = orig_flags;\n> \t}\n> +\n> +\tif (defer_submodule_status)\n> +\t\t*defer_submodule_status = defer;\n\nThe crux of this patch is that we are replacing some serial operation\nwith a parallel operation. The replacement happens here, where we are\nreplacing is_submodule_modified() by 'deferring' it.\n\nSo to verify if the parallel implementation is correct, we should\ncompare the \"setup\" and \"finish\" steps in is_submodule_modified() and\nget_submodules_status(). Eyeballing it, it looks correct, especially\nbecause we made sure to refactor out the shared logic in previous\npatches.\n\nTo reflect this, I think it would be clearer to rename\nget_submodules_status() to something similar (e.g.\nare_submodules_modified_parallel()), with an explicit comment saying\nthat it is meant to be a parallel implementation of\nis_submodule_modified().\n\nExcept, I told a little white lie in the previous paragraph, because\nget_submodules_status() isn't _just_ a parallel implementation of\nis_submodule_modified()...\n\n> @@ -268,13 +286,52 @@ int run_diff_files(struct rev_info *revs, unsigned int option)\n> \t\t\t}\n> \n> \t\t\tchanged = match_stat_with_submodule(&revs->diffopt, ce, &st,\n> -\t\t\t\t\t\t\t ce_option, &dirty_submodule);\n> +\t\t\t\t\t\t\t ce_option, NULL,\n> +\t\t\t\t\t\t\t &defer_submodule_status,\n> +\t\t\t\t\t\t\t &ignore_untracked);\n> \t\t\tnewmode = ce_mode_from_stat(ce, st.st_mode);\n> +\t\t\tif (defer_submodule_status) {\n> +\t\t\t\tstruct submodule_status_util tmp = {\n> +\t\t\t\t\t.changed = changed,\n> +\t\t\t\t\t.dirty_submodule = 0,\n> +\t\t\t\t\t.ignore_untracked = ignore_untracked,\n> +\t\t\t\t\t.newmode = newmode,\n> +\t\t\t\t\t.ce = ce,\n> +\t\t\t\t\t.path = ce->name,\n> +\t\t\t\t};\n> +\t\t\t\tstruct string_list_item *item;\n> +\n> +\t\t\t\titem = string_list_append(&submodules, ce->name);\n> +\t\t\t\titem->util = xmalloc(sizeof(tmp));\n> +\t\t\t\tmemcpy(item->util, &tmp, sizeof(tmp));\n> +\t\t\t\tcontinue;\n> +\t\t\t}\n\nbecause get_submodules_status() doesn't just contain the results of\nthe parallel processes, it is _also_ shuttling \"changed\" and\n\"ignore_untracked\" from match_stat_with_submodule(), as well as\n.newmode, .ce and .path from run_diff_files() (basically everything\nexcept .dirty_submodule)...\n\n> \t\t}\n> \n> -\t\trecord_file_diff(&revs->diffopt, newmode, dirty_submodule,\n> -\t\t\t\t changed, istate, ce);\n> +\t\tif (!defer_submodule_status)\n> +\t\t\trecord_file_diff(&revs->diffopt, newmode, 0,\n> +\t\t\t\t\t changed,istate, ce);\n> +\t}\n> +\tif (submodules.nr) {\n> +\t\tunsigned long parallel_jobs;\n> +\t\tstruct string_list_item *item;\n> +\n> +\t\tif (git_config_get_ulong(\"submodule.diffjobs\", ¶llel_jobs))\n> +\t\t\tparallel_jobs = 1;\n> +\t\telse if (!parallel_jobs)\n> +\t\t\tparallel_jobs = online_cpus();\n> +\n> +\t\tif (get_submodules_status(&submodules, parallel_jobs))\n> +\t\t\tdie(_(\"submodule status failed\"));\n> +\t\tfor_each_string_list_item(item, &submodules) {\n> +\t\t\tstruct submodule_status_util *util = item->util;\n> +\n> +\t\t\trecord_file_diff(&revs->diffopt, util->newmode,\n> +\t\t\t\t\t util->dirty_submodule, util->changed,\n> +\t\t\t\t\t istate, util->ce);\n> +\t\t}\n\nso that we can pass all of this back into record_file_diff(). The only\nmember that is changed by the parallel process is .dirty_submodule,\nwhich is exactly what we would expect from a parallel version of\nis_submodule_modified().\n\nIf we don't want to do a bigger refactor, I think we should also add\ncomments to members of \"struct submodule_status_util\" to document where\nthey come from and what they are used for.\n\nThe rest of the comments are refactor-related.\n\nIt would be good if we could avoid mixing unrelated information sources\nin \"struct submodule_status_util\", since a) this makes it very tightly\ncoupled to run_diff_files() and b) it causes us to repeat ourselves in\nthe same function (.changed = changed, record_file_diff()).\n\nThe only reason why the code looks this way right now is that\nmatch_stat_with_submodule() sets defer_submodule_status based on whether\nor not we should ignore the submodule, and this eventually tells\nget_submodule_status() what submodules it needs to care about. But,\ndeciding whether to spawn a subprocess for which submodule is exactly\nwhat the .get_next_task member is for.\n\n> diff --git a/submodule.c b/submodule.c\n> index 426074cebb..6f6e150a3f 100644\n> --- a/submodule.c\n> +++ b/submodule.c\n> @@ -1981,6 +1994,121 @@ unsigned is_submodule_modified(const char *path, int ignore_untracked)\n> \treturn dirty_submodule;\n> }\n> \n> +static struct status_task *\n> +get_status_task_from_index(struct submodule_parallel_status *sps,\n> +\t\t\t struct strbuf *err)\n> +{\n> +\tfor (; sps->index_count < sps->submodule_names->nr; sps->index_count++) {\n> +\t\tstruct submodule_status_util *util = sps->submodule_names->items[sps->index_count].util;\n> +\t\tstruct status_task *task;\n> +\n> +\t\tif (!verify_submodule_git_directory(util->path))\n> +\t\t\tcontinue;\n\nSo right here, we could use the \"check if this submodule should be\nignored\" logic form match_stat_with_submodule() to decide whether or not\nto spawn the subprocess. IOW, I am advocating for\nget_submodules_status() to be a parallel version of\nmatch_stat_with_submodule() (not a parallel version of\nis_submodule_modified() that shuttles extra information).\n\nAnother sign that this refactor is a good idea is that it lets us\nsimplify _existing_ submodule logic in run_diff_files(). Prior to this\npatch, we have:\n\n unsigned dirty_submodule = 0;\n ...\n\t\t\tchanged = match_stat_with_submodule(&revs->diffopt, ce, &st,\n\t\t\t\t\t\t\t ce_option, NULL,\n\t\t\t\t\t\t\t &defer_submodule_status,\n\t\t\t\t\t\t\t &ignore_untracked);\n // If submodule was deferred, shuttle a bunch of information\n // If not, call record_file_diff()\n\nbut the body of match_stat_with_submodule() is just ie_match_stat() +\nsome additional submodule logic. Post refactor, this would look\nsomething like:\n\n struct string_list submodules;\n ...\n // For any submodule, just append it to a list and let the\n // parallel thing take care of it.\n if (S_ISGITLINK(ce->ce_mode) {\n // Probably pass .newmode and .ce to the util too...\n string_list_append(submodules, ce->name);\n } else {\n changed = ie_match_stat(foo, bar, baz);\n record_file_diff();\n }\n ...\n if (submodules.nr) {\n parallel_match_stat_with_submodule_wip_name(&submodules);\n for_each_string_list_item(item, &submodules) {\n record_file_diff(&item);\n }\n }\n\nWhich I think is easier to follow, since we won't need\ndefer_submodule_status any more, and we don't shuttle information from\nmatch_stat_with_submodule(). Though I'm a bit unhappy that it's still\npretty coupled to run_diff_files() (it still has to shuttle .newmode,\n.ce). Also, I don't think this refactor lets us avoid the refactors we\ndid in the previous patches.\n\n> +\n> +\t\ttask = xmalloc(sizeof(*task));\n> +\t\ttask->path = util->path;\n> +\t\ttask->ignore_untracked = util->ignore_untracked;\n> +\t\tstrbuf_init(&task->out, 0);\n> +\t\tsps->index_count++;\n> +\t\treturn task;\n> +\t}\n> +\treturn NULL;\n> +}\n> +\n> +static int get_next_submodule_status(struct child_process *cp,\n> +\t\t\t\t struct strbuf *err, void *data,\n> +\t\t\t\t void **task_cb)\n> +{\n> +\tstruct submodule_parallel_status *sps = data;\n> +\tstruct status_task *task = get_status_task_from_index(sps, err);\n\nAs an aside, I think we can inline get_status_task_from_index(). I\nsuspect this pattern was copied from get_next_submodule(), which\ngets fetch tasks from two different places (hence _from_index and\n_from_changed), but here I don't think we will ever get status tasks\nfrom more than one place.\n"},{"id":"473621","messageId":"kl6lh6ukum29.fsf@chooglen-macbookpro.roam.corp.google.com","threadId":"59034","inReplyTo":"kl6ljzzguqss.fsf@chooglen-macbookpro.roam.corp.google.com","subject":"Re: [PATCH v9 6/6] diff-lib: parallelize run_diff_files for submodules","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2023-03-17T02:51:58Z","receivedAt":"2023-03-17T02:52:12Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"Glen Choo writes:\n\n> It would be good if we could avoid mixing unrelated information sources\n> in \"struct submodule_status_util\", since a) this makes it very tightly\n> coupled to run_diff_files() and b) it causes us to repeat ourselves in\n> the same function (.changed = changed, record_file_diff()).\n>\n> The only reason why the code looks this way right now is that\n> match_stat_with_submodule() sets defer_submodule_status based on whether\n> or not we should ignore the submodule, and this eventually tells\n> get_submodule_status() what submodules it needs to care about. But,\n> deciding whether to spawn a subprocess for which submodule is exactly\n> what the .get_next_task member is for.\n>\n>> diff --git a/submodule.c b/submodule.c\n>> index 426074cebb..6f6e150a3f 100644\n>> --- a/submodule.c\n>> +++ b/submodule.c\n>> @@ -1981,6 +1994,121 @@ unsigned is_submodule_modified(const char *path, int ignore_untracked)\n>> \treturn dirty_submodule;\n>> }\n>> \n>> +static struct status_task *\n>> +get_status_task_from_index(struct submodule_parallel_status *sps,\n>> +\t\t\t struct strbuf *err)\n>> +{\n>> +\tfor (; sps->index_count < sps->submodule_names->nr; sps->index_count++) {\n>> +\t\tstruct submodule_status_util *util = sps->submodule_names->items[sps->index_count].util;\n>> +\t\tstruct status_task *task;\n>> +\n>> +\t\tif (!verify_submodule_git_directory(util->path))\n>> +\t\t\tcontinue;\n>\n> So right here, we could use the \"check if this submodule should be\n> ignored\" logic form match_stat_with_submodule() to decide whether or not\n> to spawn the subprocess. IOW, I am advocating for\n> get_submodules_status() to be a parallel version of\n> match_stat_with_submodule() (not a parallel version of\n> is_submodule_modified() that shuttles extra information).\n\nIt turns out to be quite difficult to implement a parallel\nmatch_stat_with_submodule():\n\n a) we can't remove it because it still has another caller\n b) its internals are quite hard to refactor: one conditional arm depends\n on \"changed\", which is set by calling ie_match_stat(), which in turn\n requires the \"struct stat\" to have already been lstat()-ed...\n\nSo even though this series adds a lot, it is just about as minimally\ninvasive as possible.\n\nI suspect that there are some possible cleanups down the line, e.g.\nis_submodule_modified() is rightfully only called by diff-lib.c , so I\nthink it should be a static function there. And once we move that, we\ncan make our parallel function static, and then we don't have to worry\nabout tight coupling to run_diff_files(). To keep the range-diff\nmanageable, that can be left for a future cleanup though.\n"},{"id":"473671","messageId":"kl6l7cvfun3b.fsf@chooglen-macbookpro.roam.corp.google.com","threadId":"59034","inReplyTo":"20230302220251.1474923-3-calvinwan@google.com","subject":"Re: [PATCH v9 3/6] submodule: move status parsing into function","fromName":"Glen Choo","fromEmail":"chooglen@google.com","sentAt":"2023-03-17T20:42:00Z","receivedAt":"2023-03-17T20:42:11Z","isPatch":true,"sender":{"key":"glencbz@gmail.com","avatar":"https://avatars.githubusercontent.com/u/58092771?v=4"},"body":"Calvin Wan writes:\n\n> A future patch requires the ability to parse the output of git\n> status --porcelain=2. Move parsing code from is_submodule_modified to\n> parse_status_porcelain.\n\nIf my mental model is correct [1], i.e. that we are implementing a\nparallel version of is_submodule_modified(). I think we should be more\nexplicit in this patch and the next, e.g.:\n\n In a later patch, we will implement a parallel version of\n is_submodule_modified(). Refactor its \"git status --porcelain=2\"\n parsing code so that we can reuse it both the parallel and\n non-parallel versions.\n\nIf so, then this is pretty much doing the same thing as the next patch,\nso if the --color-moved diff isn't too bad, I think we can squash them,\nwhich will make the commit message easier to write too:\n\n In a later patch, we will implement a parallel version of\n is_submodule_modified(). Refactor its setup and parsing code so that\n we can reuse it both the parallel and non-parallel versions.\n\n - Setting up the subprocess is moved to prepare_status_porcelain()\n - XYZ is moved to verify_submodule_git_directory()\n - ABC is moved to parse_foobarbaz()\n\nJust an idea. I don't think squashing is necessarily better, but being\nexplciit that we want a parallel version of is_submodule_modified() will\nmake this easier to follow.\n\n[1] https://lore.kernel.org/git/kl6ljzzguqss.fsf@chooglen-macbookpro.roam.corp.google.com\n"}]}