{"thread":{"id":"48819","subject":"[PATCH] fetch-pack: write shallow, then check connectivity","startedAt":"2018-07-02T22:08:56Z","lastAt":"2018-07-02T22:08:56Z","messageCount":1,"participants":["Jonathan Tan"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"351570","messageId":"20180702220843.260595-1-jonathantanmy@google.com","threadId":"48819","inReplyTo":null,"subject":"[PATCH] fetch-pack: write shallow, then check connectivity","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2018-07-02T22:08:43Z","receivedAt":"2018-07-02T22:08:56Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"When fetching, connectivity is checked after the shallow file is\nupdated. There are 2 issues with this: (1) the connectivity check is\nonly performed up to ancestors of existing refs (which is not thorough\nenough if we were deepening an existing ref in the first place), and (2)\nthere is no rollback of the shallow file if the connectivity check\nfails.\n\nTo solve (1), update the connectivity check to check the ancestry chain\ncompletely in the case of a deepening fetch by refraining from passing\n\"--not --all\" when invoking rev-list in connected.c.\n\nTo solve (2), have fetch_pack() perform its own connectivity check\nbefore updating the shallow file. To support existing use cases in which\n\"git fetch-pack\" is used to download objects without much regard as to\nthe connectivity of the resulting objects with respect to the existing\nrepository, the connectivity check is only done if necessary (that is,\nthe fetch is not a clone, and the fetch involves shallow/deepen\nfunctionality). \"git fetch\" still performs its own connectivity check,\npreserving correctness but sometimes performing redundant work. This\nredundancy is mitigated by the fact that fetch_pack() reports if it has\nperformed a connectivity check itself, and if the transport supports\nconnect or stateless-connect, it will bubble up that report so that \"git\nfetch\" knows not to perform the connectivity check in such a case.\n\nThis was noticed when a user tried to deepen an existing repository by\nfetching with --no-shallow from a server that did not send all necessary\nobjects - the connectivity check as run by \"git fetch\" succeeded, but a\nsubsequent \"git fsck\" failed.\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\n---\nThis is on bw/ref-in-want because I wanted to use its one-time-sed\nfunctionality.\n\nThis supersedes the version in [1], which handled issue (1) but not\nissue (2).\n\n[1] https://public-inbox.org/git/20180627173237.25620-1-jonathantanmy@google.com/\n---\n builtin/fetch.c          | 15 +++++++++-----\n connected.c              |  6 ++++--\n connected.h              |  7 +++++++\n fetch-pack.c             | 31 +++++++++++++++++++++++++++++\n fetch-pack.h             | 15 ++++++++++++++\n t/t5537-fetch-shallow.sh | 43 ++++++++++++++++++++++++++++++++++++++++\n transport.c              |  1 +\n transport.h              | 11 ++++++++++\n 8 files changed, 122 insertions(+), 7 deletions(-)\n\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex 8362a97a2e..f8c68d7844 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -769,7 +769,7 @@ static int iterate_ref_map(void *cb_data, struct object_id *oid)\n }\n \n static int store_updated_refs(const char *raw_url, const char *remote_name,\n-\t\tstruct ref *ref_map)\n+\t\t\t      int connectivity_checked, struct ref *ref_map)\n {\n \tFILE *fp;\n \tstruct commit *commit;\n@@ -791,10 +791,12 @@ static int store_updated_refs(const char *raw_url, const char *remote_name,\n \telse\n \t\turl = xstrdup(\"foreign\");\n \n-\trm = ref_map;\n-\tif (check_connected(iterate_ref_map, &rm, NULL)) {\n-\t\trc = error(_(\"%s did not send all necessary objects\\n\"), url);\n-\t\tgoto abort;\n+\tif (!connectivity_checked) {\n+\t\trm = ref_map;\n+\t\tif (check_connected(iterate_ref_map, &rm, NULL)) {\n+\t\t\trc = error(_(\"%s did not send all necessary objects\\n\"), url);\n+\t\t\tgoto abort;\n+\t\t}\n \t}\n \n \tprepare_format_display(ref_map);\n@@ -960,8 +962,11 @@ static int fetch_refs(struct transport *transport, struct ref *ref_map,\n \n static int consume_refs(struct transport *transport, struct ref *ref_map)\n {\n+\tint connectivity_checked = transport->smart_options\n+\t\t? transport->smart_options->connectivity_checked : 0;\n \tint ret = store_updated_refs(transport->url,\n \t\t\t\t     transport->remote->name,\n+\t\t\t\t     connectivity_checked,\n \t\t\t\t     ref_map);\n \ttransport_unlock_pack(transport);\n \treturn ret;\ndiff --git a/connected.c b/connected.c\nindex 91feb78815..1bba888eff 100644\n--- a/connected.c\n+++ b/connected.c\n@@ -58,8 +58,10 @@ int check_connected(oid_iterate_fn fn, void *cb_data,\n \targv_array_push(&rev_list.args, \"--stdin\");\n \tif (repository_format_partial_clone)\n \t\targv_array_push(&rev_list.args, \"--exclude-promisor-objects\");\n-\targv_array_push(&rev_list.args, \"--not\");\n-\targv_array_push(&rev_list.args, \"--all\");\n+\tif (!opt->is_deepening_fetch) {\n+\t\targv_array_push(&rev_list.args, \"--not\");\n+\t\targv_array_push(&rev_list.args, \"--all\");\n+\t}\n \targv_array_push(&rev_list.args, \"--quiet\");\n \tif (opt->progress)\n \t\targv_array_pushf(&rev_list.args, \"--progress=%s\",\ndiff --git a/connected.h b/connected.h\nindex a53f03a61a..322dc76372 100644\n--- a/connected.h\n+++ b/connected.h\n@@ -38,6 +38,13 @@ struct check_connected_options {\n \t * Insert these variables into the environment of the child process.\n \t */\n \tconst char **env;\n+\n+\t/*\n+\t * If non-zero, check the ancestry chain completely, not stopping at\n+\t * any existing ref. This is necessary when deepening existing refs\n+\t * during a fetch.\n+\t */\n+\tunsigned is_deepening_fetch : 1;\n };\n \n #define CHECK_CONNECTED_INIT { 0 }\ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex 681b440615..fb8d27f614 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -19,6 +19,7 @@\n #include \"sha1-array.h\"\n #include \"oidset.h\"\n #include \"packfile.h\"\n+#include \"connected.h\"\n \n static int transfer_unpack_limit = -1;\n static int fetch_unpack_limit = -1;\n@@ -1593,6 +1594,18 @@ static void update_shallow(struct fetch_pack_args *args,\n \toid_array_clear(&ref);\n }\n \n+static int iterate_ref_map(void *cb_data, struct object_id *oid)\n+{\n+\tstruct ref **rm = cb_data;\n+\tstruct ref *ref = *rm;\n+\n+\tif (!ref)\n+\t\treturn -1; /* end of the list */\n+\t*rm = ref->next;\n+\toidcpy(oid, &ref->old_oid);\n+\treturn 0;\n+}\n+\n struct ref *fetch_pack(struct fetch_pack_args *args,\n \t\t       int fd[], struct child_process *conn,\n \t\t       const struct ref *ref,\n@@ -1621,7 +1634,25 @@ struct ref *fetch_pack(struct fetch_pack_args *args,\n \t\tref_cpy = do_fetch_pack(args, fd, ref, sought, nr_sought,\n \t\t\t\t\t&si, pack_lockfile);\n \treprepare_packed_git(the_repository);\n+\n+\tif (!args->cloning && args->deepen) {\n+\t\tstruct check_connected_options opt = CHECK_CONNECTED_INIT;\n+\t\tstruct ref *iterator = ref_cpy;\n+\t\topt.shallow_file = alternate_shallow_file;\n+\t\tif (args->deepen)\n+\t\t\topt.is_deepening_fetch = 1;\n+\t\tif (check_connected(iterate_ref_map, &iterator, &opt)) {\n+\t\t\terror(_(\"remote did not send all necessary objects\"));\n+\t\t\tfree_refs(ref_cpy);\n+\t\t\tref_cpy = NULL;\n+\t\t\trollback_lock_file(&shallow_lock);\n+\t\t\tgoto cleanup;\n+\t\t}\n+\t\targs->connectivity_checked = 1;\n+\t}\n+\n \tupdate_shallow(args, ref_cpy, &si);\n+cleanup:\n \tclear_shallow_info(&si);\n \treturn ref_cpy;\n }\ndiff --git a/fetch-pack.h b/fetch-pack.h\nindex bb45a366a8..2160be9164 100644\n--- a/fetch-pack.h\n+++ b/fetch-pack.h\n@@ -41,6 +41,21 @@ struct fetch_pack_args {\n \t * regardless of which object flags it uses (if any).\n \t */\n \tunsigned no_dependents:1;\n+\n+\t/*\n+\t * Because fetch_pack() overwrites the shallow file upon a\n+\t * successful deepening non-clone fetch, if this struct\n+\t * specifies such a fetch, fetch_pack() needs to perform a\n+\t * connectivity check before deciding if a fetch is successful\n+\t * (and overwriting the shallow file). fetch_pack() sets this\n+\t * field to 1 if such a connectivity check was performed.\n+\t *\n+\t * This is different from check_self_contained_and_connected\n+\t * in that the former allows existing objects in the\n+\t * repository to satisfy connectivity needs, whereas the\n+\t * latter doesn't.\n+\t */\n+\tunsigned connectivity_checked:1;\n };\n \n /*\ndiff --git a/t/t5537-fetch-shallow.sh b/t/t5537-fetch-shallow.sh\nindex df8d2f095a..a7afb66049 100755\n--- a/t/t5537-fetch-shallow.sh\n+++ b/t/t5537-fetch-shallow.sh\n@@ -186,4 +186,47 @@ EOF\n \ttest_cmp expect actual\n '\n \n+. \"$TEST_DIRECTORY\"/lib-httpd.sh\n+start_httpd\n+\n+REPO=\"$HTTPD_DOCUMENT_ROOT_PATH/repo\"\n+\n+test_expect_success 'shallow fetches check connectivity before writing shallow file' '\n+\trm -rf \"$REPO\" client &&\n+\n+\tgit init \"$REPO\" &&\n+\ttest_commit -C \"$REPO\" one &&\n+\ttest_commit -C \"$REPO\" two &&\n+\ttest_commit -C \"$REPO\" three &&\n+\n+\tgit init client &&\n+\n+\t# Use protocol v2 to ensure that shallow information is sent exactly\n+\t# once by the server, since we are planning to manipulate it.\n+\tgit -C \"$REPO\" config protocol.version 2 &&\n+\tgit -C client config protocol.version 2 &&\n+\n+\tgit -C client fetch --depth=2 \"$HTTPD_URL/one_time_sed/repo\" master:a_branch &&\n+\n+\t# Craft a situation in which the server sends back an unshallow request\n+\t# with an empty packfile. This is done by refetching with a shorter\n+\t# depth (to ensure that the packfile is empty), and overwriting the\n+\t# shallow line in the response with the unshallow line we want.\n+\tprintf \"s/0034shallow %s/0036unshallow %s/\" \\\n+\t       \"$(git -C \"$REPO\" rev-parse HEAD)\" \\\n+\t       \"$(git -C \"$REPO\" rev-parse HEAD^)\" \\\n+\t       >\"$HTTPD_ROOT_PATH/one-time-sed\" &&\n+\ttest_must_fail git -C client fetch --depth=1 \"$HTTPD_URL/one_time_sed/repo\" \\\n+\t\tmaster:a_branch &&\n+\n+\t# Ensure that the one-time-sed script was used.\n+\t! test -e \"$HTTPD_ROOT_PATH/one-time-sed\" &&\n+\n+\t# Ensure that the resulting repo is consistent, despite our failure to\n+\t# fetch.\n+\tgit -C client fsck\n+'\n+\n+stop_httpd\n+\n test_done\ndiff --git a/transport.c b/transport.c\nindex 39d8c2fa55..fdd813f684 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -350,6 +350,7 @@ static int fetch_refs_via_pack(struct transport *transport,\n \tdata->got_remote_heads = 0;\n \tdata->options.self_contained_and_connected =\n \t\targs.self_contained_and_connected;\n+\tdata->options.connectivity_checked = args.connectivity_checked;\n \n \tif (refs == NULL)\n \t\tret = -1;\ndiff --git a/transport.h b/transport.h\nindex 3dff767a87..7a9a7fcaf3 100644\n--- a/transport.h\n+++ b/transport.h\n@@ -18,6 +18,17 @@ struct git_transport_options {\n \tunsigned deepen_relative : 1;\n \tunsigned from_promisor : 1;\n \tunsigned no_dependents : 1;\n+\n+\t/*\n+\t * If this transport supports connect or stateless-connect,\n+\t * the corresponding field in struct fetch_pack_args is copied\n+\t * here after fetching.\n+\t *\n+\t * See the definition of connectivity_checked in struct\n+\t * fetch_pack_args for more information.\n+\t */\n+\tunsigned connectivity_checked:1;\n+\n \tint depth;\n \tconst char *deepen_since;\n \tconst struct string_list *deepen_not;\n-- \n2.18.0.399.gad0ab374a1-goog\n\n"}]}