{"thread":{"id":"26021","subject":"[WIP/RFC PATCH] Teach fetch and pull the on-demand mode for submodules","startedAt":"2010-12-10T18:45:27Z","lastAt":"2010-12-10T18:45:27Z","messageCount":1,"participants":["Jens Lehmann"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"157835","messageId":"4D027547.2050709@web.de","threadId":"26021","inReplyTo":null,"subject":"[WIP/RFC PATCH] Teach fetch and pull the on-demand mode for submodules","fromName":"Jens Lehmann","fromEmail":"jens.lehmann@web.de","sentAt":"2010-12-10T18:45:27Z","receivedAt":"2010-12-10T18:45:27Z","isPatch":true,"sender":{"key":"jens.lehmann@web.de","avatar":"https://avatars.githubusercontent.com/u/135220?v=4"},"body":"To be able to recursively checkout submodules it is sufficient to only\nthen fetch a submodule recursively when the new commits fetched in the\nsuperproject record new commits for it.\n\nBefore a new or changed ref from upstream is updated in update_local_ref()\n\"git rev-list <new-sha1> --not --branches --remotes\" is used to determine\nall newly fetched commits. These are then walked and diffed against their\nparent(s) to see if a submodule has been changed. If that is the case, its\npath is stored to fetch it after the superproject fetch is completed. This\nis the new default behavior for fetch and pull.\n\nUsing the \"--recurse-submodules\" or the \"--no-recurse-submodules\" option\ndisables the examination of the fetched refs because the result will be\nignored anyway.\n\nBecause of the currently lacking infrastructure for storing deleted and\nnew submodules in the .git directory of the superproject fetch and pull\ncan currently only fetch submodules that are already checked out.\n\nSigned-off-by: Jens Lehmann <Jens.Lehmann@web.de>\n---\n\n\nSo here is the first usable iteration of on-demand fetching for submodules\nfor people to test and comment on.\n\nStill missing:\n\n1) A new \"on-demand\" (other suggestions are welcome) value for the\n   recurse[-]submodules command line and config options.\n\n2) Tests. This patch currently breaks 5 tests and needs some of its own.\n\n3) Optimizations:\n\n   a) Check if the submodule commit is already present locally and don't\n      add it to the to-be-fetched-list then.\n\n   b) It hurts me to do the same computations again locally that the remote\n      side already did to decide what commits to send. But not being very\n      familiar with revision walking and transport I couldn't come up with\n      a better solution for now. Ideas welcome.\n\nAnd when we start implementing deletion and creation of submodules where\nthe .git directories have to live in the superprojects .git this code can\nbe extended to fetch into those directories when the submodule isn't even\nchecked out.\n\nThis patch applies to next.\n\n\n builtin/fetch.c |   11 ++++++++-\n submodule.c     |   66 ++++++++++++++++++++++++++++++++++++++++++++++++++++++-\n submodule.h     |    1 +\n 3 files changed, 76 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex 357f3cd..417fe77 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -284,6 +284,9 @@ static int update_local_ref(struct ref *ref,\n \t\telse {\n \t\t\tmsg = \"storing head\";\n \t\t\twhat = \"[new branch]\";\n+\t\t\tif ((recurse_submodules != RECURSE_SUBMODULES_OFF) &&\n+\t\t\t    (recurse_submodules != RECURSE_SUBMODULES_ON))\n+\t\t\t\tcheck_for_new_submodule_commits(ref->new_sha1);\n \t\t}\n\n \t\tr = s_update_ref(msg, ref, 0);\n@@ -299,6 +302,9 @@ static int update_local_ref(struct ref *ref,\n \t\tstrcpy(quickref, find_unique_abbrev(current->object.sha1, DEFAULT_ABBREV));\n \t\tstrcat(quickref, \"..\");\n \t\tstrcat(quickref, find_unique_abbrev(ref->new_sha1, DEFAULT_ABBREV));\n+\t\tif ((recurse_submodules != RECURSE_SUBMODULES_OFF) &&\n+\t\t    (recurse_submodules != RECURSE_SUBMODULES_ON))\n+\t\t\tcheck_for_new_submodule_commits(ref->new_sha1);\n \t\tr = s_update_ref(\"fast-forward\", ref, 1);\n \t\tsprintf(display, \"%c %-*s %-*s -> %s%s\", r ? '!' : ' ',\n \t\t\tTRANSPORT_SUMMARY_WIDTH, quickref, REFCOL_WIDTH, remote,\n@@ -310,6 +316,9 @@ static int update_local_ref(struct ref *ref,\n \t\tstrcpy(quickref, find_unique_abbrev(current->object.sha1, DEFAULT_ABBREV));\n \t\tstrcat(quickref, \"...\");\n \t\tstrcat(quickref, find_unique_abbrev(ref->new_sha1, DEFAULT_ABBREV));\n+\t\tif ((recurse_submodules != RECURSE_SUBMODULES_OFF) &&\n+\t\t    (recurse_submodules != RECURSE_SUBMODULES_ON))\n+\t\t\tcheck_for_new_submodule_commits(ref->new_sha1);\n \t\tr = s_update_ref(\"forced-update\", ref, 1);\n \t\tsprintf(display, \"%c %-*s %-*s -> %s  (%s)\", r ? '!' : '+',\n \t\t\tTRANSPORT_SUMMARY_WIDTH, quickref, REFCOL_WIDTH, remote,\n@@ -951,7 +960,7 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)\n \t\tint num_options = 0;\n \t\t/* Set recursion as default when we already are recursing */\n \t\tif (submodule_prefix[0])\n-\t\t\tset_config_fetch_recurse_submodules(1);\n+\t\t\tset_config_fetch_recurse_submodules(recurse_submodules == RECURSE_SUBMODULES_ON ? 1 : -1);\n \t\tgitmodules_config();\n \t\tgit_config(submodule_config, NULL);\n \t\tadd_options_to_argv(&num_options, options);\ndiff --git a/submodule.c b/submodule.c\nindex 4e62900..54eda96 100644\n--- a/submodule.c\n+++ b/submodule.c\n@@ -12,7 +12,8 @@\n struct string_list config_name_for_path;\n struct string_list config_fetch_recurse_submodules_for_name;\n struct string_list config_ignore_for_name;\n-static int config_fetch_recurse_submodules;\n+static int config_fetch_recurse_submodules = -1;\n+struct string_list changed_submodule_paths;\n\n static int add_submodule_odb(const char *path)\n {\n@@ -248,6 +249,66 @@ void set_config_fetch_recurse_submodules(int value)\n \tconfig_fetch_recurse_submodules = value;\n }\n\n+static void submodule_collect_changed_cb(struct diff_queue_struct *q,\n+\t\t\t\t\t struct diff_options *options,\n+\t\t\t\t\t void *data)\n+{\n+\tint i;\n+\tfor (i = 0; i < q->nr; i++) {\n+\t\tstruct diff_filepair *p = q->queue[i];\n+\t\tif (S_ISGITLINK(p->two->mode)) {\n+\t\t\tif (S_ISGITLINK(p->one->mode)) {\n+\t\t\t\t/* FIXME: We should be using the name configured in the\n+\t\t\t\t * .gitmodules file of the commit we are examining here to\n+\t\t\t\t * be able to follow moved submodules. */\n+\t\t\t\tstruct string_list_item *path;\n+\t\t\t\tpath = unsorted_string_list_lookup(&changed_submodule_paths, p->two->path);\n+\t\t\t\tif (!path)\n+\t\t\t\t\tstring_list_append(&changed_submodule_paths, xstrdup(p->two->path));\n+\t\t\t} else {\n+\t\t\t\t/* Submodule is new or was moved here */\n+\n+\t\t\t\t/* FIXME: When the .git directories of submodules live inside\n+\t\t\t\t * the superprojects .git directory some day we should fetch\n+\t\t\t\t * new submodules into that location too so they can be\n+\t\t\t\t * checked out from there when needed. */\n+\t\t\t\treturn;\n+\t\t\t}\n+\t\t}\n+\t}\n+}\n+\n+void check_for_new_submodule_commits(unsigned char new_sha1[20])\n+{\n+\tstruct rev_info rev;\n+\tstruct commit *commit;\n+\tint argc = 5;\n+\tconst char *argv[] = {NULL, NULL, \"--not\", \"--branches\", \"--remotes\", NULL};\n+\n+\tinit_revisions(&rev, NULL);\n+\targv[1] = xstrdup(sha1_to_hex(new_sha1));\n+\tsetup_revisions(argc, argv, &rev, NULL);\n+\tif (prepare_revision_walk(&rev))\n+\t\tdie(\"revision walk setup failed\");\n+\n+\twhile ((commit = get_revision(&rev))) {\n+\t\tstruct commit_list *parent = commit->parents;\n+\t\twhile (parent) {\n+\t\t\tstruct diff_options diff_opts;\n+\t\t\tdiff_setup(&diff_opts);\n+\t\t\tdiff_opts.output_format |= DIFF_FORMAT_CALLBACK;\n+\t\t\tdiff_opts.format_callback = submodule_collect_changed_cb;\n+\t\t\tif (diff_setup_done(&diff_opts) < 0)\n+\t\t\t\tdie(\"diff-setup\");\n+\t\t\tdiff_tree_sha1(parent->item->object.sha1, commit->object.sha1, \"\", &diff_opts);\n+\t\t\tdiffcore_std(&diff_opts);\n+\t\t\tdiff_flush(&diff_opts);\n+\t\t\tparent = parent->next;\n+\t\t}\n+\t}\n+\tfree((char *)argv[1]);\n+}\n+\n int fetch_populated_submodules(int num_options, const char **options,\n \t\t\t       const char *prefix, int ignore_config,\n \t\t\t       int quiet)\n@@ -300,6 +361,9 @@ int fetch_populated_submodules(int num_options, const char **options,\n \t\t\t} else {\n \t\t\t\tif (!config_fetch_recurse_submodules)\n \t\t\t\t\tcontinue;\n+\t\t\t\tif ((config_fetch_recurse_submodules < 0) &&\n+\t\t\t\t    !unsorted_string_list_lookup(&changed_submodule_paths, ce->name))\n+\t\t\t\t\tcontinue;\n \t\t\t}\n \t\t}\n\ndiff --git a/submodule.h b/submodule.h\nindex 4729023..a574f11 100644\n--- a/submodule.h\n+++ b/submodule.h\n@@ -14,6 +14,7 @@ void show_submodule_summary(FILE *f, const char *path,\n \t\tunsigned dirty_submodule,\n \t\tconst char *del, const char *add, const char *reset);\n void set_config_fetch_recurse_submodules(int value);\n+void check_for_new_submodule_commits(unsigned char new_sha1[20]);\n int fetch_populated_submodules(int num_options, const char **options,\n \t\t\t       const char *prefix, int ignore_config,\n \t\t\t       int quiet);\n-- \n1.7.3.3.711.g95750\n"}]}