{"thread":{"id":"33935","subject":"[PATCH v4 0/4] nd/clone-connectivity-shortcut updates","startedAt":"2013-05-26T01:16:13Z","lastAt":"2013-05-26T01:16:17Z","messageCount":5,"participants":["Nguyễn Thái Ngọc Duy"],"isPatch":true,"patchVersion":4,"patchTotal":4},"messages":[{"id":"218551","messageId":"1369530977-12746-1-git-send-email-pclouds@gmail.com","threadId":"33935","inReplyTo":null,"subject":"[PATCH v4 0/4] nd/clone-connectivity-shortcut updates","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-05-26T01:16:13Z","receivedAt":"2013-05-26T01:16:13Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"This addresses the comments from Junio and Eric in v3 [1].\n\n[1] http://thread.gmane.org/gmane.comp.version-control.git/219611/focus=223584\n\nNguyễn Thái Ngọc Duy (4):\n  clone: let the user know when check_everything_connected is run\n  fetch-pack: prepare updated shallow file before fetching the pack\n  index-pack: remove dead code (it should never happen)\n  clone: open a shortcut for connectivity check\n\n Documentation/git-index-pack.txt |  3 ++\n builtin/clone.c                  | 15 +++++--\n builtin/index-pack.c             | 38 ++++++++++++------\n commit.h                         |  2 +\n connected.c                      | 34 +++++++++++++++-\n connected.h                      |  5 +++\n fetch-pack.c                     | 84 ++++++++++++++++++++++------------------\n fetch-pack.h                     |  4 +-\n git.c                            |  7 ++++\n shallow.c                        | 42 +++++++++++++++++++-\n t/t5500-fetch-pack.sh            |  7 ++++\n transport.c                      |  4 ++\n transport.h                      |  2 +\n 13 files changed, 192 insertions(+), 55 deletions(-)\n\n-- \n1.8.2.83.gc99314b\n"},{"id":"218552","messageId":"1369530977-12746-2-git-send-email-pclouds@gmail.com","threadId":"33935","inReplyTo":"1369530977-12746-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v4 1/4] clone: let the user know when check_everything_connected is run","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-05-26T01:16:14Z","receivedAt":"2013-05-26T01:16:14Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"check_everything_connected could take a long time, especially in the\nclone case where the whole DAG is traversed. The user deserves to know\nwhat's going on.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>\nSigned-off-by: Junio C Hamano <gitster@pobox.com>\n---\n builtin/clone.c | 4 ++++\n 1 file changed, 4 insertions(+)\n\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex 035ab64..dad4265 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -546,8 +546,12 @@ static void update_remote_refs(const struct ref *refs,\n {\n \tconst struct ref *rm = mapped_refs;\n \n+\tif (0 <= option_verbosity)\n+\t\tprintf(_(\"Checking connectivity... \"));\n \tif (check_everything_connected(iterate_ref_map, 0, &rm))\n \t\tdie(_(\"remote did not send all necessary objects\"));\n+\tif (0 <= option_verbosity)\n+\t\tprintf(_(\"done\\n\"));\n \n \tif (refs) {\n \t\twrite_remote_refs(mapped_refs);\n-- \n1.8.2.83.gc99314b\n"},{"id":"218553","messageId":"1369530977-12746-3-git-send-email-pclouds@gmail.com","threadId":"33935","inReplyTo":"1369530977-12746-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v4 2/4] fetch-pack: prepare updated shallow file before fetching the pack","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-05-26T01:16:15Z","receivedAt":"2013-05-26T01:16:15Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"index-pack --strict looks up and follows parent commits. If shallow\ninformation is not ready by the time index-pack is run, index-pack may\nbe led to non-existent objects. Make fetch-pack save shallow file to\ndisk before invoking index-pack.\n\ngit learns new global option --shallow-file to pass on the alternate\nshallow file path. Undocumented (and not even support --shallow-file=\nsyntax) because it's unlikely to be used again elsewhere.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>\n---\n commit.h              |  2 ++\n fetch-pack.c          | 73 ++++++++++++++++++++++++++-------------------------\n git.c                 |  7 +++++\n shallow.c             | 42 +++++++++++++++++++++++++++--\n t/t5500-fetch-pack.sh |  7 +++++\n 5 files changed, 93 insertions(+), 38 deletions(-)\n\ndiff --git a/commit.h b/commit.h\nindex 67bd509..6e9c7cd 100644\n--- a/commit.h\n+++ b/commit.h\n@@ -176,6 +176,8 @@ extern int for_each_commit_graft(each_commit_graft_fn, void *);\n extern int is_repository_shallow(void);\n extern struct commit_list *get_shallow_commits(struct object_array *heads,\n \t\tint depth, int shallow_flag, int not_shallow_flag);\n+extern void check_shallow_file_for_update(void);\n+extern void set_alternate_shallow_file(const char *path);\n \n int is_descendant_of(struct commit *, struct commit_list *);\n int in_merge_bases(struct commit *, struct commit *);\ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex f156dd4..6b5467c 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -20,6 +20,8 @@ static int no_done;\n static int fetch_fsck_objects = -1;\n static int transfer_fsck_objects = -1;\n static int agent_supported;\n+static struct lock_file shallow_lock;\n+static const char *alternate_shallow_file;\n \n #define COMPLETE\t(1U << 0)\n #define COMMON\t\t(1U << 1)\n@@ -683,7 +685,7 @@ static int get_pack(struct fetch_pack_args *args,\n \t\t    int xd[2], char **pack_lockfile)\n {\n \tstruct async demux;\n-\tconst char *argv[20];\n+\tconst char *argv[22];\n \tchar keep_arg[256];\n \tchar hdr_arg[256];\n \tconst char **av;\n@@ -724,6 +726,11 @@ static int get_pack(struct fetch_pack_args *args,\n \t\t\tdo_keep = 1;\n \t}\n \n+\tif (alternate_shallow_file) {\n+\t\t*av++ = \"--shallow-file\";\n+\t\t*av++ = alternate_shallow_file;\n+\t}\n+\n \tif (do_keep) {\n \t\tif (pack_lockfile)\n \t\t\tcmd.out = -1;\n@@ -779,6 +786,27 @@ static int cmp_ref_by_name(const void *a_, const void *b_)\n \treturn strcmp(a->name, b->name);\n }\n \n+static void setup_alternate_shallow(void)\n+{\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tint fd;\n+\n+\tcheck_shallow_file_for_update();\n+\tfd = hold_lock_file_for_update(&shallow_lock, git_path(\"shallow\"),\n+\t\t\t\t       LOCK_DIE_ON_ERROR);\n+\tif (write_shallow_commits(&sb, 0)) {\n+\t\tif (write_in_full(fd, sb.buf, sb.len) != sb.len)\n+\t\t\tdie_errno(\"failed to write to %s\", shallow_lock.filename);\n+\t\talternate_shallow_file = shallow_lock.filename;\n+\t} else\n+\t\t/*\n+\t\t * is_repository_shallow() sees empty string as \"no\n+\t\t * shallow file\".\n+\t\t */\n+\t\talternate_shallow_file = \"\";\n+\tstrbuf_release(&sb);\n+}\n+\n static struct ref *do_fetch_pack(struct fetch_pack_args *args,\n \t\t\t\t int fd[2],\n \t\t\t\t const struct ref *orig_ref,\n@@ -858,6 +886,8 @@ static struct ref *do_fetch_pack(struct fetch_pack_args *args,\n \n \tif (args->stateless_rpc)\n \t\tpacket_flush(fd[1]);\n+\tif (args->depth > 0)\n+\t\tsetup_alternate_shallow();\n \tif (get_pack(args, fd, pack_lockfile))\n \t\tdie(\"git fetch-pack: fetch failed.\");\n \n@@ -936,15 +966,9 @@ struct ref *fetch_pack(struct fetch_pack_args *args,\n \t\t       struct ref **sought, int nr_sought,\n \t\t       char **pack_lockfile)\n {\n-\tstruct stat st;\n \tstruct ref *ref_cpy;\n \n \tfetch_pack_setup();\n-\tif (args->depth > 0) {\n-\t\tif (stat(git_path(\"shallow\"), &st))\n-\t\t\tst.st_mtime = 0;\n-\t}\n-\n \tif (nr_sought)\n \t\tnr_sought = remove_duplicates_in_refs(sought, nr_sought);\n \n@@ -954,35 +978,12 @@ struct ref *fetch_pack(struct fetch_pack_args *args,\n \t}\n \tref_cpy = do_fetch_pack(args, fd, ref, sought, nr_sought, pack_lockfile);\n \n-\tif (args->depth > 0) {\n-\t\tstatic struct lock_file lock;\n-\t\tstruct cache_time mtime;\n-\t\tstruct strbuf sb = STRBUF_INIT;\n-\t\tchar *shallow = git_path(\"shallow\");\n-\t\tint fd;\n-\n-\t\tmtime.sec = st.st_mtime;\n-\t\tmtime.nsec = ST_MTIME_NSEC(st);\n-\t\tif (stat(shallow, &st)) {\n-\t\t\tif (mtime.sec)\n-\t\t\t\tdie(\"shallow file was removed during fetch\");\n-\t\t} else if (st.st_mtime != mtime.sec\n-#ifdef USE_NSEC\n-\t\t\t\t|| ST_MTIME_NSEC(st) != mtime.nsec\n-#endif\n-\t\t\t  )\n-\t\t\tdie(\"shallow file was changed during fetch\");\n-\n-\t\tfd = hold_lock_file_for_update(&lock, shallow,\n-\t\t\t\t\t       LOCK_DIE_ON_ERROR);\n-\t\tif (!write_shallow_commits(&sb, 0)\n-\t\t || write_in_full(fd, sb.buf, sb.len) != sb.len) {\n-\t\t\tunlink_or_warn(shallow);\n-\t\t\trollback_lock_file(&lock);\n-\t\t} else {\n-\t\t\tcommit_lock_file(&lock);\n-\t\t}\n-\t\tstrbuf_release(&sb);\n+\tif (alternate_shallow_file) {\n+\t\tif (*alternate_shallow_file == '\\0') { /* --unshallow */\n+\t\t\tunlink_or_warn(git_path(\"shallow\"));\n+\t\t\trollback_lock_file(&shallow_lock);\n+\t\t} else\n+\t\t\tcommit_lock_file(&shallow_lock);\n \t}\n \n \treprepare_packed_git();\ndiff --git a/git.c b/git.c\nindex 1ada169..88eef5a 100644\n--- a/git.c\n+++ b/git.c\n@@ -4,6 +4,7 @@\n #include \"help.h\"\n #include \"quote.h\"\n #include \"run-command.h\"\n+#include \"commit.h\"\n \n const char git_usage_string[] =\n \t\"git [--version] [--help] [-c name=value]\\n\"\n@@ -146,6 +147,12 @@ static int handle_options(const char ***argv, int *argc, int *envchanged)\n \t\t\tsetenv(GIT_LITERAL_PATHSPECS_ENVIRONMENT, \"0\", 1);\n \t\t\tif (envchanged)\n \t\t\t\t*envchanged = 1;\n+\t\t} else if (!strcmp(cmd, \"--shallow-file\")) {\n+\t\t\t(*argv)++;\n+\t\t\t(*argc)--;\n+\t\t\tset_alternate_shallow_file((*argv)[0]);\n+\t\t\tif (envchanged)\n+\t\t\t\t*envchanged = 1;\n \t\t} else {\n \t\t\tfprintf(stderr, \"Unknown option: %s\\n\", cmd);\n \t\t\tusage(git_usage_string);\ndiff --git a/shallow.c b/shallow.c\nindex 6be915f..cbe2526 100644\n--- a/shallow.c\n+++ b/shallow.c\n@@ -3,6 +3,16 @@\n #include \"tag.h\"\n \n static int is_shallow = -1;\n+static struct stat shallow_stat;\n+static char *alternate_shallow_file;\n+\n+void set_alternate_shallow_file(const char *path)\n+{\n+\tif (is_shallow != -1)\n+\t\tdie(\"BUG: is_repository_shallow must not be called before set_alternate_shallow_file\");\n+\tfree(alternate_shallow_file);\n+\talternate_shallow_file = path ? xstrdup(path) : NULL;\n+}\n \n int register_shallow(const unsigned char *sha1)\n {\n@@ -21,12 +31,21 @@ int is_repository_shallow(void)\n {\n \tFILE *fp;\n \tchar buf[1024];\n+\tconst char *path = alternate_shallow_file;\n \n \tif (is_shallow >= 0)\n \t\treturn is_shallow;\n \n-\tfp = fopen(git_path(\"shallow\"), \"r\");\n-\tif (!fp) {\n+\tif (!path)\n+\t\tpath = git_path(\"shallow\");\n+\t/*\n+\t * fetch-pack sets '--shallow-file \"\"' as an indicator that no\n+\t * shallow file should be used. We could just open it and it\n+\t * will likely fail. But let's do an explicit check instead.\n+\t */\n+\tif (!*path ||\n+\t    stat(path, &shallow_stat) ||\n+\t    (fp = fopen(path, \"r\")) == NULL) {\n \t\tis_shallow = 0;\n \t\treturn is_shallow;\n \t}\n@@ -108,3 +127,22 @@ struct commit_list *get_shallow_commits(struct object_array *heads, int depth,\n \n \treturn result;\n }\n+\n+void check_shallow_file_for_update(void)\n+{\n+\tstruct stat st;\n+\n+\tif (!is_shallow)\n+\t\treturn;\n+\telse if (is_shallow == -1)\n+\t\tdie(\"BUG: shallow must be initialized by now\");\n+\n+\tif (stat(git_path(\"shallow\"), &st))\n+\t\tdie(\"shallow file was removed during fetch\");\n+\telse if (st.st_mtime != shallow_stat.st_mtime\n+#ifdef USE_NSEC\n+\t\t || ST_MTIME_NSEC(st) != ST_MTIME_NSEC(shallow_stat)\n+#endif\n+\t\t   )\n+\t\tdie(\"shallow file was changed during fetch\");\n+}\ndiff --git a/t/t5500-fetch-pack.sh b/t/t5500-fetch-pack.sh\nindex d574085..557b073 100755\n--- a/t/t5500-fetch-pack.sh\n+++ b/t/t5500-fetch-pack.sh\n@@ -135,6 +135,13 @@ test_expect_success 'clone shallow depth 1' '\n \ttest \"`git --git-dir=shallow0/.git rev-list --count HEAD`\" = 1\n '\n \n+test_expect_success 'clone shallow depth 1 with fsck' '\n+\tgit config --global fetch.fsckobjects true &&\n+\tgit clone --no-single-branch --depth 1 \"file://$(pwd)/.\" shallow0fsck &&\n+\ttest \"`git --git-dir=shallow0fsck/.git rev-list --count HEAD`\" = 1 &&\n+\tgit config --global --unset fetch.fsckobjects\n+'\n+\n test_expect_success 'clone shallow' '\n \tgit clone --no-single-branch --depth 2 \"file://$(pwd)/.\" shallow\n '\n-- \n1.8.2.83.gc99314b\n"},{"id":"218554","messageId":"1369530977-12746-4-git-send-email-pclouds@gmail.com","threadId":"33935","inReplyTo":"1369530977-12746-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v4 3/4] index-pack: remove dead code (it should never happen)","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-05-26T01:16:16Z","receivedAt":"2013-05-26T01:16:16Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Signed-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>\nSigned-off-by: Junio C Hamano <gitster@pobox.com>\n---\n builtin/index-pack.c | 3 +--\n 1 file changed, 1 insertion(+), 2 deletions(-)\n\ndiff --git a/builtin/index-pack.c b/builtin/index-pack.c\nindex 79dfe47..f52a04f 100644\n--- a/builtin/index-pack.c\n+++ b/builtin/index-pack.c\n@@ -747,8 +747,7 @@ static void sha1_object(const void *data, struct object_entry *obj_entry,\n \t\t\tint eaten;\n \t\t\tvoid *buf = (void *) data;\n \n-\t\t\tif (!buf)\n-\t\t\t\tbuf = new_data = get_data_from_pack(obj_entry);\n+\t\t\tassert(data && \"data can only be NULL for large _blobs_\");\n \n \t\t\t/*\n \t\t\t * we do not need to free the memory here, as the\n-- \n1.8.2.83.gc99314b\n"},{"id":"218555","messageId":"1369530977-12746-5-git-send-email-pclouds@gmail.com","threadId":"33935","inReplyTo":"1369530977-12746-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v4 4/4] clone: open a shortcut for connectivity check","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-05-26T01:16:17Z","receivedAt":"2013-05-26T01:16:17Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"In order to make sure the cloned repository is good, we run \"rev-list\n--objects --not --all $new_refs\" on the repository. This is expensive\non large repositories. This patch attempts to mitigate the impact in\nthis special case.\n\nIn the \"good\" clone case, we only have one pack. If all of the\nfollowing are met, we can be sure that all objects reachable from the\nnew refs exist, which is the intention of running \"rev-list ...\":\n\n - all refs point to an object in the pack\n - there are no dangling pointers in any object in the pack\n - no objects in the pack point to objects outside the pack\n\nThe second and third checks can be done with the help of index-pack as\na slight variation of --strict check (which introduces a new condition\nfor the shortcut: pack transfer must be used and the number of objects\nlarge enough to call index-pack). The first is checked in\ncheck_everything_connected after we get an \"ok\" from index-pack.\n\n\"index-pack + new checks\" is still faster than the current \"index-pack\n+ rev-list\", which is the whole point of this patch. If any of the\nconditions fail, we fall back to the good old but expensive \"rev-list\n..\". In that case it's even more expensive because we have to pay for\nthe new checks in index-pack. But that should only happen when the\nother side is either buggy or malicious.\n\nCloning linux-2.6 over file://\n\n        before         after\nreal    3m25.693s      2m53.050s\nuser    5m2.037s       4m42.396s\nsys     0m13.750s      0m16.574s\n\nA more realistic test with ssh:// over wireless\n\n        before         after\nreal    11m26.629s     10m4.213s\nuser    5m43.196s      5m19.444s\nsys     0m35.812s      0m37.630s\n\nThis shortcut is not applied to shallow clones, partly because shallow\nclones should have no more objects than a usual fetch and the cost of\nrev-list is acceptable, partly to avoid dealing with corner cases when\ngrafting is involved.\n\nThis shortcut does not apply to unpack-objects code path either\nbecause the number of objects must be small in order to trigger that\ncode path.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy <pclouds@gmail.com>\n---\n Documentation/git-index-pack.txt |  3 +++\n builtin/clone.c                  | 11 ++++++++---\n builtin/index-pack.c             | 35 ++++++++++++++++++++++++++---------\n connected.c                      | 34 +++++++++++++++++++++++++++++++++-\n connected.h                      |  5 +++++\n fetch-pack.c                     | 11 ++++++++++-\n fetch-pack.h                     |  4 +++-\n transport.c                      |  4 ++++\n transport.h                      |  2 ++\n 9 files changed, 94 insertions(+), 15 deletions(-)\n\ndiff --git a/Documentation/git-index-pack.txt b/Documentation/git-index-pack.txt\nindex bde8eec..7a4e055 100644\n--- a/Documentation/git-index-pack.txt\n+++ b/Documentation/git-index-pack.txt\n@@ -74,6 +74,9 @@ OPTIONS\n --strict::\n \tDie, if the pack contains broken objects or links.\n \n+--check-self-contained-and-connected::\n+\tDie if the pack contains broken links. For internal use only.\n+\n --threads=<n>::\n \tSpecifies the number of threads to spawn when resolving\n \tdeltas. This requires that index-pack be compiled with\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex dad4265..069e81e 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -542,13 +542,15 @@ static void update_remote_refs(const struct ref *refs,\n \t\t\t       const struct ref *mapped_refs,\n \t\t\t       const struct ref *remote_head_points_at,\n \t\t\t       const char *branch_top,\n-\t\t\t       const char *msg)\n+\t\t\t       const char *msg,\n+\t\t\t       struct transport *transport)\n {\n \tconst struct ref *rm = mapped_refs;\n \n \tif (0 <= option_verbosity)\n \t\tprintf(_(\"Checking connectivity... \"));\n-\tif (check_everything_connected(iterate_ref_map, 0, &rm))\n+\tif (check_everything_connected_with_transport(iterate_ref_map,\n+\t\t\t\t\t\t      0, &rm, transport))\n \t\tdie(_(\"remote did not send all necessary objects\"));\n \tif (0 <= option_verbosity)\n \t\tprintf(_(\"done\\n\"));\n@@ -893,6 +895,9 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \t\tif (option_upload_pack)\n \t\t\ttransport_set_option(transport, TRANS_OPT_UPLOADPACK,\n \t\t\t\t\t     option_upload_pack);\n+\n+\t\tif (transport->smart_options && !option_depth)\n+\t\t\ttransport->smart_options->check_self_contained_and_connected = 1;\n \t}\n \n \trefs = transport_get_remote_refs(transport);\n@@ -954,7 +959,7 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \t\ttransport_fetch_refs(transport, mapped_refs);\n \n \tupdate_remote_refs(refs, mapped_refs, remote_head_points_at,\n-\t\t\t   branch_top.buf, reflog_msg.buf);\n+\t\t\t   branch_top.buf, reflog_msg.buf, transport);\n \n \tupdate_head(our_head_points_at, remote_head, reflog_msg.buf);\n \ndiff --git a/builtin/index-pack.c b/builtin/index-pack.c\nindex f52a04f..9c1cfac 100644\n--- a/builtin/index-pack.c\n+++ b/builtin/index-pack.c\n@@ -77,8 +77,10 @@ static int nr_threads;\n \n static int from_stdin;\n static int strict;\n+static int do_fsck_object;\n static int verbose;\n static int show_stat;\n+static int check_self_contained_and_connected;\n \n static struct progress *progress;\n \n@@ -187,13 +189,13 @@ static int mark_link(struct object *obj, int type, void *data)\n \n /* The content of each linked object must have been checked\n    or it must be already present in the object database */\n-static void check_object(struct object *obj)\n+static unsigned check_object(struct object *obj)\n {\n \tif (!obj)\n-\t\treturn;\n+\t\treturn 0;\n \n \tif (!(obj->flags & FLAG_LINK))\n-\t\treturn;\n+\t\treturn 0;\n \n \tif (!(obj->flags & FLAG_CHECKED)) {\n \t\tunsigned long size;\n@@ -201,17 +203,20 @@ static void check_object(struct object *obj)\n \t\tif (type != obj->type || type <= 0)\n \t\t\tdie(_(\"object of unexpected type\"));\n \t\tobj->flags |= FLAG_CHECKED;\n-\t\treturn;\n+\t\treturn 1;\n \t}\n+\n+\treturn 0;\n }\n \n-static void check_objects(void)\n+static unsigned check_objects(void)\n {\n-\tunsigned i, max;\n+\tunsigned i, max, foreign_nr = 0;\n \n \tmax = get_max_object_index();\n \tfor (i = 0; i < max; i++)\n-\t\tcheck_object(get_indexed_object(i));\n+\t\tforeign_nr += check_object(get_indexed_object(i));\n+\treturn foreign_nr;\n }\n \n \n@@ -756,7 +761,8 @@ static void sha1_object(const void *data, struct object_entry *obj_entry,\n \t\t\tobj = parse_object_buffer(sha1, type, size, buf, &eaten);\n \t\t\tif (!obj)\n \t\t\t\tdie(_(\"invalid %s\"), typename(type));\n-\t\t\tif (fsck_object(obj, 1, fsck_error_function))\n+\t\t\tif (do_fsck_object &&\n+\t\t\t    fsck_object(obj, 1, fsck_error_function))\n \t\t\t\tdie(_(\"Error in object\"));\n \t\t\tif (fsck_walk(obj, mark_link, NULL))\n \t\t\t\tdie(_(\"Not all child objects of %s are reachable\"), sha1_to_hex(obj->sha1));\n@@ -1490,6 +1496,7 @@ int cmd_index_pack(int argc, const char **argv, const char *prefix)\n \tstruct pack_idx_entry **idx_objects;\n \tstruct pack_idx_option opts;\n \tunsigned char pack_sha1[20];\n+\tunsigned foreign_nr = 1;\t/* zero is a \"good\" value, assume bad */\n \n \tif (argc == 2 && !strcmp(argv[1], \"-h\"))\n \t\tusage(index_pack_usage);\n@@ -1511,6 +1518,10 @@ int cmd_index_pack(int argc, const char **argv, const char *prefix)\n \t\t\t\tfix_thin_pack = 1;\n \t\t\t} else if (!strcmp(arg, \"--strict\")) {\n \t\t\t\tstrict = 1;\n+\t\t\t\tdo_fsck_object = 1;\n+\t\t\t} else if (!strcmp(arg, \"--check-self-contained-and-connected\")) {\n+\t\t\t\tstrict = 1;\n+\t\t\t\tcheck_self_contained_and_connected = 1;\n \t\t\t} else if (!strcmp(arg, \"--verify\")) {\n \t\t\t\tverify = 1;\n \t\t\t} else if (!strcmp(arg, \"--verify-stat\")) {\n@@ -1624,7 +1635,7 @@ int cmd_index_pack(int argc, const char **argv, const char *prefix)\n \tconclude_pack(fix_thin_pack, curr_pack, pack_sha1);\n \tfree(deltas);\n \tif (strict)\n-\t\tcheck_objects();\n+\t\tforeign_nr = check_objects();\n \n \tif (show_stat)\n \t\tshow_pack_info(stat_only);\n@@ -1650,5 +1661,11 @@ int cmd_index_pack(int argc, const char **argv, const char *prefix)\n \tif (index_name == NULL)\n \t\tfree((void *) curr_index);\n \n+\t/*\n+\t * Let the caller know this pack is not self contained\n+\t */\n+\tif (check_self_contained_and_connected && foreign_nr)\n+\t\treturn 1;\n+\n \treturn 0;\n }\ndiff --git a/connected.c b/connected.c\nindex 1e89c1c..fae8d64 100644\n--- a/connected.c\n+++ b/connected.c\n@@ -2,7 +2,12 @@\n #include \"run-command.h\"\n #include \"sigchain.h\"\n #include \"connected.h\"\n+#include \"transport.h\"\n \n+int check_everything_connected(sha1_iterate_fn fn, int quiet, void *cb_data)\n+{\n+\treturn check_everything_connected_with_transport(fn, quiet, cb_data, NULL);\n+}\n /*\n  * If we feed all the commits we want to verify to this command\n  *\n@@ -14,7 +19,10 @@\n  *\n  * Returns 0 if everything is connected, non-zero otherwise.\n  */\n-int check_everything_connected(sha1_iterate_fn fn, int quiet, void *cb_data)\n+int check_everything_connected_with_transport(sha1_iterate_fn fn,\n+\t\t\t\t\t      int quiet,\n+\t\t\t\t\t      void *cb_data,\n+\t\t\t\t\t      struct transport *transport)\n {\n \tstruct child_process rev_list;\n \tconst char *argv[] = {\"rev-list\", \"--objects\",\n@@ -22,10 +30,23 @@ int check_everything_connected(sha1_iterate_fn fn, int quiet, void *cb_data)\n \tchar commit[41];\n \tunsigned char sha1[20];\n \tint err = 0;\n+\tstruct packed_git *new_pack = NULL;\n \n \tif (fn(cb_data, sha1))\n \t\treturn err;\n \n+\tif (transport && transport->smart_options &&\n+\t    transport->smart_options->self_contained_and_connected &&\n+\t    transport->pack_lockfile &&\n+\t    !suffixcmp(transport->pack_lockfile, \".keep\")) {\n+\t\tstruct strbuf idx_file = STRBUF_INIT;\n+\t\tstrbuf_addstr(&idx_file, transport->pack_lockfile);\n+\t\tstrbuf_setlen(&idx_file, idx_file.len - 5); /* \".keep\" */\n+\t\tstrbuf_addstr(&idx_file, \".idx\");\n+\t\tnew_pack = add_packed_git(idx_file.buf, idx_file.len, 1);\n+\t\tstrbuf_release(&idx_file);\n+\t}\n+\n \tif (quiet)\n \t\targv[5] = \"--quiet\";\n \n@@ -42,6 +63,17 @@ int check_everything_connected(sha1_iterate_fn fn, int quiet, void *cb_data)\n \n \tcommit[40] = '\\n';\n \tdo {\n+\t\t/*\n+\t\t * If index-pack already checked that:\n+\t\t * - there are no dangling pointers in the new pack\n+\t\t * - the pack is self contained\n+\t\t * Then if the updated ref is in the new pack, then we\n+\t\t * are sure the ref is good and not sending it to\n+\t\t * rev-list for verification.\n+\t\t */\n+\t\tif (new_pack && find_pack_entry_one(sha1, new_pack))\n+\t\t\tcontinue;\n+\n \t\tmemcpy(commit, sha1_to_hex(sha1), 40);\n \t\tif (write_in_full(rev_list.in, commit, 41) < 0) {\n \t\t\tif (errno != EPIPE && errno != EINVAL)\ndiff --git a/connected.h b/connected.h\nindex 7e4585a..0b060b7 100644\n--- a/connected.h\n+++ b/connected.h\n@@ -1,6 +1,8 @@\n #ifndef CONNECTED_H\n #define CONNECTED_H\n \n+struct transport;\n+\n /*\n  * Take callback data, and return next object name in the buffer.\n  * When called after returning the name for the last object, return -1\n@@ -16,5 +18,8 @@ typedef int (*sha1_iterate_fn)(void *, unsigned char [20]);\n  * Return 0 if Ok, non zero otherwise (i.e. some missing objects)\n  */\n extern int check_everything_connected(sha1_iterate_fn, int quiet, void *cb_data);\n+extern int check_everything_connected_with_transport(sha1_iterate_fn, int quiet,\n+\t\t\t\t\t\t     void *cb_data,\n+\t\t\t\t\t\t     struct transport *transport);\n \n #endif /* CONNECTED_H */\ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex 6b5467c..abe5ffb 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -691,6 +691,7 @@ static int get_pack(struct fetch_pack_args *args,\n \tconst char **av;\n \tint do_keep = args->keep_pack;\n \tstruct child_process cmd;\n+\tint ret;\n \n \tmemset(&demux, 0, sizeof(demux));\n \tif (use_sideband) {\n@@ -747,11 +748,14 @@ static int get_pack(struct fetch_pack_args *args,\n \t\t\t\tstrcpy(keep_arg + s, \"localhost\");\n \t\t\t*av++ = keep_arg;\n \t\t}\n+\t\tif (args->check_self_contained_and_connected)\n+\t\t\t*av++ = \"--check-self-contained-and-connected\";\n \t}\n \telse {\n \t\t*av++ = \"unpack-objects\";\n \t\tif (args->quiet || args->no_progress)\n \t\t\t*av++ = \"-q\";\n+\t\targs->check_self_contained_and_connected = 0;\n \t}\n \tif (*hdr_arg)\n \t\t*av++ = hdr_arg;\n@@ -772,7 +776,12 @@ static int get_pack(struct fetch_pack_args *args,\n \t\tclose(cmd.out);\n \t}\n \n-\tif (finish_command(&cmd))\n+\tret = finish_command(&cmd);\n+\tif (!ret || (args->check_self_contained_and_connected && ret == 1))\n+\t\targs->self_contained_and_connected =\n+\t\t\targs->check_self_contained_and_connected &&\n+\t\t\tret == 0;\n+\telse\n \t\tdie(\"%s failed\", argv[0]);\n \tif (use_sideband && finish_async(&demux))\n \t\tdie(\"error in sideband demultiplexer\");\ndiff --git a/fetch-pack.h b/fetch-pack.h\nindex dc5266c..40f08ba 100644\n--- a/fetch-pack.h\n+++ b/fetch-pack.h\n@@ -16,7 +16,9 @@ struct fetch_pack_args {\n \t\tverbose:1,\n \t\tno_progress:1,\n \t\tinclude_tag:1,\n-\t\tstateless_rpc:1;\n+\t\tstateless_rpc:1,\n+\t\tcheck_self_contained_and_connected:1,\n+\t\tself_contained_and_connected:1;\n };\n \n /*\ndiff --git a/transport.c b/transport.c\nindex ba5d8af..359a671 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -534,6 +534,8 @@ static int fetch_refs_via_pack(struct transport *transport,\n \targs.quiet = (transport->verbose < 0);\n \targs.no_progress = !transport->progress;\n \targs.depth = data->options.depth;\n+\targs.check_self_contained_and_connected =\n+\t\tdata->options.check_self_contained_and_connected;\n \n \tif (!data->got_remote_heads) {\n \t\tconnect_setup(transport, 0, 0);\n@@ -551,6 +553,8 @@ static int fetch_refs_via_pack(struct transport *transport,\n \t\trefs = NULL;\n \tdata->conn = NULL;\n \tdata->got_remote_heads = 0;\n+\tdata->options.self_contained_and_connected =\n+\t\targs.self_contained_and_connected;\n \n \tfree_refs(refs_tmp);\n \ndiff --git a/transport.h b/transport.h\nindex fcb1d25..4edebc5 100644\n--- a/transport.h\n+++ b/transport.h\n@@ -8,6 +8,8 @@ struct git_transport_options {\n \tunsigned thin : 1;\n \tunsigned keep : 1;\n \tunsigned followtags : 1;\n+\tunsigned check_self_contained_and_connected : 1;\n+\tunsigned self_contained_and_connected : 1;\n \tint depth;\n \tconst char *uploadpack;\n \tconst char *receivepack;\n-- \n1.8.2.83.gc99314b\n"}]}