{"thread":{"id":"35400","subject":"[PATCH v3 00/28] First class shallow clone","startedAt":"2013-11-25T03:55:26Z","lastAt":"2014-01-09T21:57:47Z","messageCount":79,"participants":["Nguyễn Thái Ngọc Duy","Junio C Hamano","Eric Sunshine","Duy Nguyen","Jeff King"],"isPatch":true,"patchVersion":3,"patchTotal":28},"messages":[{"id":"231027","messageId":"1385351754-9954-1-git-send-email-pclouds@gmail.com","threadId":"35400","inReplyTo":null,"subject":"[PATCH v3 00/28] First class shallow clone","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-11-25T03:55:26Z","receivedAt":"2013-11-25T03:55:26Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Compared to v2 [1], v3 grows a bit. The biggest difference is\n.git/shallow is not updated by default (except when you clone from\na shallow repository). When you send something, the \"safe\" refs that\ndo not need new shallow roots are accepted at the receiver end, the\nothers rejected.\n\nTo accept those other refs, either use \"fetch --update-shallow\" or\nenable receive.shallowupdate on the receiver side of a push.\n\nFiltering \"safe\" refs requires walking through some commits, so\nit'll be more expensive than normal full clones. This is especially\ntrue for the receiver of a push (see 07/28 and 17/28). I envision\nshallow repos are used as upstream to archive old history, so this is\nnot a good news. Commit cache (or pack v4) might help. We might even\nbe able to move some work from receive-pack to send-pack to reduce\nserver load..\n\n[1] http://mid.gmane.org/1374314290-5976-1-git-send-email-pclouds@gmail.com\n\nNguyễn Thái Ngọc Duy (28):\n transport.h: remove send_pack prototype, already defined in send-pack.h\n send-pack: forbid pushing from a shallow repository\n clone: prevent --reference to a shallow repository\n update-server-info: do not publish shallow clones\n\n This part is just cleanup.\n\n Advertise shallow graft information on the server end\n connect.c: teach get_remote_heads to parse \"shallow\" lines\n shallow.c: add remove_reachable_shallow_points()\n shallow.c: add mark_new_shallow_refs()\n shallow.c: extend setup_*_shallow() to accept extra shallow points\n fetch-pack.c: move shallow update code out of fetch_pack()\n fetch-pack.h: one statement per bitfield declaration\n clone: support remote shallow repository\n fetch: support fetching from a shallow repository\n upload-pack: make sure deepening preserves shallow roots\n fetch: add --update-shallow to get refs that require updating .git/shallow\n\n Basic shallow fetch/clone support on git protocol\n\n receive-pack: reorder some code in unpack()\n Support pushing from a shallow clone\n New var GIT_SHALLOW_FILE to propagate --shallow-file to subprocesses\n connected.c: add new variant that runs with --shallow-file\n receive-pack: allow pushing with new shallow roots\n send-pack: support pushing to a shallow clone\n remote-curl: pass ref SHA-1 to fetch-pack as well\n \n Push support\n\n Support fetch/clone over http\n receive-pack: support pushing to a shallow clone via http\n send-pack: support pushing from a shallow clone via http\n\n smart-http support\n\n git-clone.txt: remove shallow clone limitations\n clone: use git protocol for cloning shallow repo locally\n prune: clean .git/shallow after pruning objects\n\n miscellaneous\n\n Documentation/config.txt | 4 +\n Documentation/fetch-options.txt | 14 +-\n Documentation/git-clone.txt | 7 +-\n Documentation/gitremote-helpers.txt | 7 +\n Documentation/technical/pack-protocol.txt | 7 +-\n builtin/clone.c | 18 +-\n builtin/fetch-pack.c | 23 +-\n builtin/fetch.c | 15 +-\n builtin/gc.c | 1 +\n builtin/prune.c | 4 +\n builtin/receive-pack.c | 248 +++++++++++++++++----\n builtin/send-pack.c | 5 +-\n cache.h | 1 +\n commit.h | 19 +-\n connect.c | 14 +-\n connected.c | 42 +++-\n connected.h | 2 +\n environment.c | 6 +\n fetch-pack.c | 132 ++++++++++--\n fetch-pack.h | 29 +--\n git.c | 2 +-\n remote-curl.c | 33 ++-\n remote.h | 5 +-\n send-pack.c | 20 +-\n server-info.c | 9 +\n shallow.c | 348 +++++++++++++++++++++++++++++-\n t/t5536-fetch-shallow.sh (new +x) | 193 +++++++++++++++++\n t/t5537-push-shallow.sh (new +x) | 184 ++++++++++++++++\n t/t5601-clone.sh | 7 +\n transport-helper.c | 6 +\n transport.c | 22 +-\n transport.h | 16 +-\n upload-pack.c | 8 +-\n 33 files changed, 1323 insertions(+), 128 deletions(-)\n create mode 100755 t/t5536-fetch-shallow.sh\n create mode 100755 t/t5537-push-shallow.sh\n\n-- \n1.8.2.83.gc99314b\n"},{"id":"231031","messageId":"1385351754-9954-2-git-send-email-pclouds@gmail.com","threadId":"35400","inReplyTo":"1385351754-9954-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v3 01/28] transport.h: remove send_pack prototype, already defined in send-pack.h","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-11-25T03:55:27Z","receivedAt":"2013-11-25T03:55:27Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n transport.h | 6 ------\n 1 file changed, 6 deletions(-)\n\ndiff --git a/transport.h b/transport.h\nindex 8f96bed..b3679bb 100644\n--- a/transport.h\n+++ b/transport.h\n@@ -193,10 +193,4 @@ void transport_print_push_status(const char *dest, struct ref *refs,\n \n typedef void alternate_ref_fn(const struct ref *, void *);\n extern void for_each_alternate_ref(alternate_ref_fn, void *);\n-\n-struct send_pack_args;\n-extern int send_pack(struct send_pack_args *args,\n-\t\t int fd[], struct child_process *conn,\n-\t\t struct ref *remote_refs,\n-\t\t struct extra_have_objects *extra_have);\n #endif\n-- \n1.8.2.83.gc99314b\n"},{"id":"231029","messageId":"1385351754-9954-3-git-send-email-pclouds@gmail.com","threadId":"35400","inReplyTo":"1385351754-9954-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v3 02/28] send-pack: forbid pushing from a shallow repository","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-11-25T03:55:28Z","receivedAt":"2013-11-25T03:55:28Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"send-pack can send a pack with loose ends to the server. receive-pack\nbefore 6d4bb38 (fetch: verify we have everything we need before\nupdating our ref - 2011-09-01) does not detect this and keeps the pack\nanyway, which corrupts the repository, at least from fsck point of\nview.\n\nsend-pack will learn to safely push from a shallow repository later.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/send-pack.c | 3 +++\n 1 file changed, 3 insertions(+)\n\ndiff --git a/builtin/send-pack.c b/builtin/send-pack.c\nindex 4482f16..51121f2 100644\n--- a/builtin/send-pack.c\n+++ b/builtin/send-pack.c\n@@ -207,6 +207,9 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)\n \t (send_all && args.send_mirror))\n \t\tusage(send_pack_usage);\n \n+\tif (is_repository_shallow())\n+\t\tdie(\"attempt to push from a shallow repository\");\n+\n \tif (remote_name) {\n \t\tremote = remote_get(remote_name);\n \t\tif (!remote_has_url(remote, dest)) {\n-- \n1.8.2.83.gc99314b\n"},{"id":"231030","messageId":"1385351754-9954-4-git-send-email-pclouds@gmail.com","threadId":"35400","inReplyTo":"1385351754-9954-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v3 03/28] clone: prevent --reference to a shallow repository","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-11-25T03:55:29Z","receivedAt":"2013-11-25T03:55:29Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"If we borrow objects from another repository, we should also pay\nattention to their $GIT_DIR/shallow (and even info/grafts). But\ncurrent alternates code does not.\n\nReject alternate repos that are shallow because we do not do it\nright. In future we alternate code may be updated to check\n$GIT_DIR/shallow properly so that this restriction could be lifted.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/clone.c | 6 ++++++\n 1 file changed, 6 insertions(+)\n\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex 874e0fd..900f564 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -252,6 +252,12 @@ static int add_one_reference(struct string_list_item *item, void *cb_data)\n \t\tdie(_(\"reference repository '%s' is not a local repository.\"),\n \t\t item->string);\n \n+\tif (!access(mkpath(\"%s/shallow\", ref_git), F_OK))\n+\t\tdie(_(\"reference repository '%s' is shallow\"), item->string);\n+\n+\tif (!access(mkpath(\"%s/info/grafts\", ref_git), F_OK))\n+\t\tdie(_(\"reference repository '%s' is grafted\"), item->string);\n+\n \tstrbuf_addf(&alternate, \"%s/objects\", ref_git);\n \tadd_to_alternates_file(alternate.buf);\n \tstrbuf_release(&alternate);\n-- \n1.8.2.83.gc99314b\n"},{"id":"231028","messageId":"1385351754-9954-5-git-send-email-pclouds@gmail.com","threadId":"35400","inReplyTo":"1385351754-9954-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v3 04/28] update-server-info: do not publish shallow clones","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-11-25T03:55:30Z","receivedAt":"2013-11-25T03:55:30Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Dumb commit walker does not care about .git/shallow and until someone\nsteps up to make it happen, let's not publish shallow clones using\ndumb protocols.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n server-info.c | 9 +++++++++\n 1 file changed, 9 insertions(+)\n\ndiff --git a/server-info.c b/server-info.c\nindex 9ec744e..a8df6a5 100644\n--- a/server-info.c\n+++ b/server-info.c\n@@ -33,6 +33,10 @@ static int update_info_refs(int force)\n \tstrcpy(path1 + len, \"+\");\n \n \tsafe_create_leading_directories(path0);\n+\tif (is_repository_shallow()) {\n+\t\tunlink(path0);\n+\t\treturn error(\"info/refs not updated for shallow clone\");\n+\t}\n \tinfo_ref_fp = fopen(path1, \"w\");\n \tif (!info_ref_fp)\n \t\treturn error(\"unable to update %s\", path1);\n@@ -217,6 +221,11 @@ static int update_info_packs(int force)\n \tstrcpy(name, infofile);\n \tstrcpy(name + namelen, \"+\");\n \n+\tif (is_repository_shallow()) {\n+\t\tunlink(infofile);\n+\t\treturn error(\"info/packs not updated for shallow clone\");\n+\t}\n+\n \tinit_pack_info(infofile, force);\n \n \tsafe_create_leading_directories(name);\n-- \n1.8.2.83.gc99314b\n"},{"id":"231033","messageId":"1385351754-9954-6-git-send-email-pclouds@gmail.com","threadId":"35400","inReplyTo":"1385351754-9954-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v3 05/28] Advertise shallow graft information on the server end","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-11-25T03:55:31Z","receivedAt":"2013-11-25T03:55:31Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"If either receive-pack or upload-pack is called on a shallow\nrepository, shallow graft points will be sent after the ref\nadvertisement (but before the packet flush), so that the client has\nthe full \"shape\" of the server's commit graph.\n\nThis breaks the protocol for all clients trying to push to a shallow\nrepo, or fetch from one. Which is basically the same end result as\ntoday's \"is_repository_shallow() && die()\" in receive-pack and\nupload-pack. New clients will be made aware of shallow upstream and\ncan make use of this information.\n\nSmart HTTP is not affected by this patch. Shallow support on\nsmart-http comes later separately.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n Documentation/technical/pack-protocol.txt | 3 +++\n builtin/receive-pack.c | 4 +++-\n commit.h | 1 +\n shallow.c | 15 +++++++++++++++\n upload-pack.c | 6 ++++--\n 5 files changed, 26 insertions(+), 3 deletions(-)\n\ndiff --git a/Documentation/technical/pack-protocol.txt b/Documentation/technical/pack-protocol.txt\nindex b898e97..eb8edd1 100644\n--- a/Documentation/technical/pack-protocol.txt\n+++ b/Documentation/technical/pack-protocol.txt\n@@ -161,6 +161,7 @@ MUST peel the ref if it's an annotated tag.\n \n ----\n advertised-refs = (no-refs / list-of-refs)\n+\t\t *shallow\n \t\t flush-pkt\n \n no-refs = PKT-LINE(zero-id SP \"capabilities^{}\"\n@@ -174,6 +175,8 @@ MUST peel the ref if it's an annotated tag.\n other-tip = obj-id SP refname LF\n other-peeled = obj-id SP refname \"^{}\" LF\n \n+ shallow = PKT-LINE(\"shallow\" SP obj-id)\n+\n capability-list = capability *(SP capability)\n capability = 1*(LC_ALPHA / DIGIT / \"-\" / \"_\")\n LC_ALPHA = %x61-7A\ndiff --git a/builtin/receive-pack.c b/builtin/receive-pack.c\nindex 67ce1ef..cc8c34f 100644\n--- a/builtin/receive-pack.c\n+++ b/builtin/receive-pack.c\n@@ -178,6 +178,8 @@ static void write_head_info(void)\n \tif (!sent_capabilities)\n \t\tshow_ref(\"capabilities^{}\", null_sha1);\n \n+\tadvertise_shallow_grafts(1);\n+\n \t/* EOF */\n \tpacket_flush(1);\n }\n@@ -998,7 +1000,7 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix)\n \tif (!enter_repo(dir, 0))\n \t\tdie(\"'%s' does not appear to be a git repository\", dir);\n \n-\tif (is_repository_shallow())\n+\tif (is_repository_shallow() && stateless_rpc)\n \t\tdie(\"attempt to push into a shallow repository\");\n \n \tgit_config(receive_pack_config, NULL);\ndiff --git a/commit.h b/commit.h\nindex bd841f4..a879526 100644\n--- a/commit.h\n+++ b/commit.h\n@@ -205,6 +205,7 @@ extern int write_shallow_commits(struct strbuf *out, int use_pack_protocol);\n extern void setup_alternate_shallow(struct lock_file *shallow_lock,\n \t\t\t\t const char **alternate_shallow_file);\n extern char *setup_temporary_shallow(void);\n+extern void advertise_shallow_grafts(int);\n \n int is_descendant_of(struct commit *, struct commit_list *);\n int in_merge_bases(struct commit *, struct commit *);\ndiff --git a/shallow.c b/shallow.c\nindex cdf37d6..9552512 100644\n--- a/shallow.c\n+++ b/shallow.c\n@@ -220,3 +220,18 @@ void setup_alternate_shallow(struct lock_file *shallow_lock,\n \t\t*alternate_shallow_file = \"\";\n \tstrbuf_release(&sb);\n }\n+\n+static int advertise_shallow_grafts_cb(const struct commit_graft *graft, void *cb)\n+{\n+\tint fd = *(int*)cb;\n+\tif (graft->nr_parent == -1)\n+\t\tpacket_write(fd, \"shallow %s\\n\", sha1_to_hex(graft->sha1));\n+\treturn 0;\n+}\n+\n+void advertise_shallow_grafts(int fd)\n+{\n+\tif (!is_repository_shallow())\n+\t\treturn;\n+\tfor_each_commit_graft(advertise_shallow_grafts_cb, &fd);\n+}\ndiff --git a/upload-pack.c b/upload-pack.c\nindex c989a73..38b2a29 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -758,6 +758,7 @@ static void upload_pack(void)\n \t\treset_timeout();\n \t\thead_ref_namespaced(send_ref, &symref);\n \t\tfor_each_namespaced_ref(send_ref, &symref);\n+\t\tadvertise_shallow_grafts(1);\n \t\tpacket_flush(1);\n \t} else {\n \t\thead_ref_namespaced(mark_our_ref, NULL);\n@@ -835,8 +836,9 @@ int main(int argc, char **argv)\n \n \tif (!enter_repo(dir, strict))\n \t\tdie(\"'%s' does not appear to be a git repository\", dir);\n-\tif (is_repository_shallow())\n-\t\tdie(\"attempt to fetch/clone from a shallow repository\");\n+\tif (is_repository_shallow() && stateless_rpc)\n+\t\tdie(\"attempt to push into a shallow repository\");\n+\n \tgit_config(upload_pack_config, NULL);\n \tupload_pack();\n \treturn 0;\n-- \n1.8.2.83.gc99314b\n"},{"id":"231032","messageId":"1385351754-9954-7-git-send-email-pclouds@gmail.com","threadId":"35400","inReplyTo":"1385351754-9954-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v3 06/28] connect.c: teach get_remote_heads to parse \"shallow\" lines","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-11-25T03:55:32Z","receivedAt":"2013-11-25T03:55:32Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"No callers pass a non-empty pointer as shallow_points at this\nstage. As a result, all clients still refuse to talk to shallow\nrepository on the other end.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/fetch-pack.c | 2 +-\n builtin/send-pack.c | 2 +-\n connect.c | 12 +++++++++++-\n remote-curl.c | 2 +-\n remote.h | 3 ++-\n transport.c | 7 ++++---\n 6 files changed, 20 insertions(+), 8 deletions(-)\n\ndiff --git a/builtin/fetch-pack.c b/builtin/fetch-pack.c\nindex c8e8582..c1d918f 100644\n--- a/builtin/fetch-pack.c\n+++ b/builtin/fetch-pack.c\n@@ -150,7 +150,7 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)\n \t\t\t\t args.verbose ? CONNECT_VERBOSE : 0);\n \t}\n \n-\tget_remote_heads(fd[0], NULL, 0, &ref, 0, NULL);\n+\tget_remote_heads(fd[0], NULL, 0, &ref, 0, NULL, NULL);\n \n \tref = fetch_pack(&args, fd, conn, ref, dest,\n \t\t\t sought, nr_sought, pack_lockfile_ptr);\ndiff --git a/builtin/send-pack.c b/builtin/send-pack.c\nindex 51121f2..bfa9253 100644\n--- a/builtin/send-pack.c\n+++ b/builtin/send-pack.c\n@@ -233,7 +233,7 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)\n \n \tmemset(&extra_have, 0, sizeof(extra_have));\n \n-\tget_remote_heads(fd[0], NULL, 0, &remote_refs, REF_NORMAL, &extra_have);\n+\tget_remote_heads(fd[0], NULL, 0, &remote_refs, REF_NORMAL, &extra_have, NULL);\n \n \ttransport_verify_remote_names(nr_refspecs, refspecs);\n \ndiff --git a/connect.c b/connect.c\nindex 06e88b0..d0602b0 100644\n--- a/connect.c\n+++ b/connect.c\n@@ -122,7 +122,8 @@ static void annotate_refs_with_symref_info(struct ref *ref)\n */\n struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,\n \t\t\t struct ref **list, unsigned int flags,\n-\t\t\t struct extra_have_objects *extra_have)\n+\t\t\t struct extra_have_objects *extra_have,\n+\t\t\t struct extra_have_objects *shallow_points)\n {\n \tstruct ref **orig_list = list;\n \tint got_at_least_one_head = 0;\n@@ -148,6 +149,15 @@ struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,\n \t\tif (len > 4 && !prefixcmp(buffer, \"ERR \"))\n \t\t\tdie(\"remote error: %s\", buffer + 4);\n \n+\t\tif (len == 48 && !prefixcmp(buffer, \"shallow \")) {\n+\t\t\tif (get_sha1_hex(buffer + 8, old_sha1))\n+\t\t\t\tdie(\"protocol error: expected shallow sha, got '%s'\", buffer + 8);\n+\t\t\tif (!shallow_points)\n+\t\t\t\tdie(\"repository on the other end cannot be shallow\");\n+\t\t\tadd_extra_have(shallow_points, old_sha1);\n+\t\t\tcontinue;\n+\t\t}\n+\n \t\tif (len < 42 || get_sha1_hex(buffer, old_sha1) || buffer[40] != ' ')\n \t\t\tdie(\"protocol error: expected sha/ref, got '%s'\", buffer);\n \t\tname = buffer + 41;\ndiff --git a/remote-curl.c b/remote-curl.c\nindex c9b891a..222210f 100644\n--- a/remote-curl.c\n+++ b/remote-curl.c\n@@ -107,7 +107,7 @@ static struct ref *parse_git_refs(struct discovery *heads, int for_push)\n {\n \tstruct ref *list = NULL;\n \tget_remote_heads(-1, heads->buf, heads->len, &list,\n-\t\t\t for_push ? REF_NORMAL : 0, NULL);\n+\t\t\t for_push ? REF_NORMAL : 0, NULL, NULL);\n \treturn list;\n }\n \ndiff --git a/remote.h b/remote.h\nindex 131130a..773faa9 100644\n--- a/remote.h\n+++ b/remote.h\n@@ -143,7 +143,8 @@ struct extra_have_objects {\n };\n extern struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,\n \t\t\t\t struct ref **list, unsigned int flags,\n-\t\t\t\t struct extra_have_objects *);\n+\t\t\t\t struct extra_have_objects *have,\n+\t\t\t\t struct extra_have_objects *shallow);\n \n int resolve_remote_symref(struct ref *ref, struct ref *list);\n int ref_newer(const unsigned char *new_sha1, const unsigned char *old_sha1);\ndiff --git a/transport.c b/transport.c\nindex 7202b77..9c51767 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -511,7 +511,7 @@ static struct ref *get_refs_via_connect(struct transport *transport, int for_pus\n \n \tconnect_setup(transport, for_push, 0);\n \tget_remote_heads(data->fd[0], NULL, 0, &refs,\n-\t\t\t for_push ? REF_NORMAL : 0, &data->extra_have);\n+\t\t\t for_push ? REF_NORMAL : 0, &data->extra_have, NULL);\n \tdata->got_remote_heads = 1;\n \n \treturn refs;\n@@ -541,7 +541,8 @@ static int fetch_refs_via_pack(struct transport *transport,\n \n \tif (!data->got_remote_heads) {\n \t\tconnect_setup(transport, 0, 0);\n-\t\tget_remote_heads(data->fd[0], NULL, 0, &refs_tmp, 0, NULL);\n+\t\tget_remote_heads(data->fd[0], NULL, 0, &refs_tmp, 0,\n+\t\t\t\t NULL, NULL);\n \t\tdata->got_remote_heads = 1;\n \t}\n \n@@ -805,7 +806,7 @@ static int git_transport_push(struct transport *transport, struct ref *remote_re\n \t\tstruct ref *tmp_refs;\n \t\tconnect_setup(transport, 1, 0);\n \n-\t\tget_remote_heads(data->fd[0], NULL, 0, &tmp_refs, REF_NORMAL, NULL);\n+\t\tget_remote_heads(data->fd[0], NULL, 0, &tmp_refs, REF_NORMAL, NULL, NULL);\n \t\tdata->got_remote_heads = 1;\n \t}\n \n-- \n1.8.2.83.gc99314b\n"},{"id":"231034","messageId":"1385351754-9954-8-git-send-email-pclouds@gmail.com","threadId":"35400","inReplyTo":"1385351754-9954-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v3 07/28] shallow.c: add remove_reachable_shallow_points()","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-11-25T03:55:33Z","receivedAt":"2013-11-25T03:55:33Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"When we receive a pack and the shallow points from another repository,\nwe may want to add more shallow points to current repo to make sure no\ncommits point to nowhere. However we do not want to add unnecessary\nshallow points and cut our history short because the other end is a\nshallow version of this repo. The output shallow points may or may not\nbe added to .git/shallow, depending on whether they are actually\nreachable in the new pack.\n\nThis function filters such shallow points out, leaving ones that might\npotentially be added. A simple has_sha1_file won't do because we may\nhave incomplete object islands (i.e. not connected to any refs) and\nthe shallow points are on these islands. In that case we want to keep\nthe shallow points as candidates until we figure out if the new pack\nconnects to such object islands.\n\nTypical cases that use remove_reachable_shallow_points() are:\n\n - fetch from a repo that has longer history: in this case all remote\n shallow roots do not exist in the client repo, this function will\n be cheap as it just does a few lookup_commit_graft and\n has_sha1_file.\n\n - fetch from a repo that has exactly the same shallow root set\n (e.g. a clone from a shallow repo): this case may trigger\n in_merge_bases_many all the way to roots. An exception is made to\n avoid such costly path with a faith that .git/shallow does not\n usually points to unreachable commit islands.\n\n - push from a shallow repo that has shorter history than the\n server's: in_merge_bases_many() is unavoidable, so the longer\n history the client has the higher the server cost is. The cost may\n be reduced with the help of pack bitmaps, or commit cache, though.\n\n - push from a shallow clone that has exactly the same shallow root\n set: the same as the second fetch case above, .i.e. cheap by\n exception.\n\nThe function must be called before the new pack is installed, or we\nwon't know which objects are ours, which theirs.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n commit.h | 3 +++\n connect.c | 2 +-\n remote.h | 1 +\n shallow.c | 45 +++++++++++++++++++++++++++++++++++++++++++++\n 4 files changed, 50 insertions(+), 1 deletion(-)\n\ndiff --git a/commit.h b/commit.h\nindex a879526..98044e6 100644\n--- a/commit.h\n+++ b/commit.h\n@@ -193,6 +193,7 @@ extern struct commit_list *get_octopus_merge_bases(struct commit_list *in);\n /* largest positive number a signed 32-bit integer can contain */\n #define INFINITE_DEPTH 0x7fffffff\n \n+struct extra_have_objects;\n extern int register_shallow(const unsigned char *sha1);\n extern int unregister_shallow(const unsigned char *sha1);\n extern int for_each_commit_graft(each_commit_graft_fn, void *);\n@@ -206,6 +207,8 @@ extern void setup_alternate_shallow(struct lock_file *shallow_lock,\n \t\t\t\t const char **alternate_shallow_file);\n extern char *setup_temporary_shallow(void);\n extern void advertise_shallow_grafts(int);\n+extern void remove_reachable_shallow_points(struct extra_have_objects *out,\n+\t\t\t\t\t const struct extra_have_objects *in);\n \n int is_descendant_of(struct commit *, struct commit_list *);\n int in_merge_bases(struct commit *, struct commit *);\ndiff --git a/connect.c b/connect.c\nindex d0602b0..80e4360 100644\n--- a/connect.c\n+++ b/connect.c\n@@ -45,7 +45,7 @@ int check_ref_type(const struct ref *ref, int flags)\n \treturn check_ref(ref->name, strlen(ref->name), flags);\n }\n \n-static void add_extra_have(struct extra_have_objects *extra, unsigned char *sha1)\n+void add_extra_have(struct extra_have_objects *extra, unsigned char *sha1)\n {\n \tALLOC_GROW(extra->array, extra->nr + 1, extra->alloc);\n \thashcpy(&(extra->array[extra->nr][0]), sha1);\ndiff --git a/remote.h b/remote.h\nindex 773faa9..ff604ff 100644\n--- a/remote.h\n+++ b/remote.h\n@@ -145,6 +145,7 @@ extern struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,\n \t\t\t\t struct ref **list, unsigned int flags,\n \t\t\t\t struct extra_have_objects *have,\n \t\t\t\t struct extra_have_objects *shallow);\n+extern void add_extra_have(struct extra_have_objects *extra, unsigned char *sha1);\n \n int resolve_remote_symref(struct ref *ref, struct ref *list);\n int ref_newer(const unsigned char *new_sha1, const unsigned char *old_sha1);\ndiff --git a/shallow.c b/shallow.c\nindex 9552512..a974d2d 100644\n--- a/shallow.c\n+++ b/shallow.c\n@@ -2,6 +2,8 @@\n #include \"commit.h\"\n #include \"tag.h\"\n #include \"pkt-line.h\"\n+#include \"remote.h\"\n+#include \"refs.h\"\n \n static int is_shallow = -1;\n static struct stat shallow_stat;\n@@ -235,3 +237,46 @@ void advertise_shallow_grafts(int fd)\n \t\treturn;\n \tfor_each_commit_graft(advertise_shallow_grafts_cb, &fd);\n }\n+\n+struct commit_array {\n+\tstruct commit **commits;\n+\tint nr, alloc;\n+};\n+\n+static int add_ref(const char *refname,\n+\t\t const unsigned char *sha1, int flags, void *cb_data)\n+{\n+\tstruct commit_array *ca = cb_data;\n+\tALLOC_GROW(ca->commits, ca->nr + 1, ca->alloc);\n+\tca->commits[ca->nr++] = lookup_commit(sha1);\n+\treturn 0;\n+}\n+\n+void remove_reachable_shallow_points(struct extra_have_objects *out,\n+\t\t\t\t const struct extra_have_objects *in)\n+{\n+\tstruct commit_array ca;\n+\tint i;\n+\n+\tmemset(&ca, 0, sizeof(ca));\n+\thead_ref(add_ref, &ca);\n+\tfor_each_ref(add_ref, &ca);\n+\tfor (i = 0; i < in->nr; i++) {\n+\t\tstruct commit_graft *graft = lookup_commit_graft(in->array[i]);\n+\t\t/*\n+\t\t * For a clone from a shallow upstream, the clone has\n+\t\t * the same shallow roots as upstream and it will\n+\t\t * trigger in_merge_bases_many() all the way to roots.\n+\t\t * Avoid that costly path and assume .git/shallow is\n+\t\t * good most of the time.\n+\t\t */\n+\t\tif (graft && graft->nr_parent < 0)\n+\t\t\tcontinue;\n+\t\tif (has_sha1_file(in->array[i]) &&\n+\t\t in_merge_bases_many(lookup_commit(in->array[i]),\n+\t\t\t\t\tca.nr, ca.commits))\n+\t\t\tcontinue;\n+\t\tadd_extra_have(out, in->array[i]);\n+\t}\n+\tfree(ca.commits);\n+}\n-- \n1.8.2.83.gc99314b\n"},{"id":"231035","messageId":"1385351754-9954-9-git-send-email-pclouds@gmail.com","threadId":"35400","inReplyTo":"1385351754-9954-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v3 08/28] shallow.c: add mark_new_shallow_refs()","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-11-25T03:55:34Z","receivedAt":"2013-11-25T03:55:34Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"When we receive a pack and the shallow points from another repository,\nwe may need to add more shallow points to current repo to make sure no\ncommits point to nowhere. But usually we don't want to do so because\n(in future) new shallow points invalidate pack bitmaps and we need to\nrebuild them again, which is not cheap.\n\nSo the default way is we allow ref updates that do not introduce new\nshallow points and mark the others. If the user is fine with new\nshallow point addition, we accept the marked refs.\n\nBut even so we do not blindly accept all shallow points provided. Some\nof them might not point to any commits in the new pack. Some might\neven do, but those might be unreachable object islands. Only shallow\npoints that are reachable from old and new refs can stay.\n\nThe way it's implemented is paint down from each ref, attach a bitmap\nto each commit where one bit represents one ref. In order to avoid\nallocating new bitmap for every commit, we try to reuse the same\nbitmap for parent commits if possible. This reduces allocation and\nleaks deliberately because it's hard to keep/time consuming track how\nmany pointers to the same buffer.\n\nIn a typical push or fetch, the new pack should not carry new shallow\nroots. If the current repo does not have any commit islands refered by\nthe sender's shallow roots either, this function is just a few\nhas_sha1_file(). So quite cheap.\n\nOnce the sender diverts from that path (or the receiver detects\nshallow roots attached to commit islands from remove_reachable_shallow_points),\nit'll be a lot more expensive. Pack bitmaps won't help this kind of\ncommit traversal, but commit cache might.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n commit.h | 4 ++\n shallow.c | 219 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 223 insertions(+)\n\ndiff --git a/commit.h b/commit.h\nindex 98044e6..e1fd587 100644\n--- a/commit.h\n+++ b/commit.h\n@@ -194,6 +194,7 @@ extern struct commit_list *get_octopus_merge_bases(struct commit_list *in);\n #define INFINITE_DEPTH 0x7fffffff\n \n struct extra_have_objects;\n+struct ref;\n extern int register_shallow(const unsigned char *sha1);\n extern int unregister_shallow(const unsigned char *sha1);\n extern int for_each_commit_graft(each_commit_graft_fn, void *);\n@@ -209,6 +210,9 @@ extern char *setup_temporary_shallow(void);\n extern void advertise_shallow_grafts(int);\n extern void remove_reachable_shallow_points(struct extra_have_objects *out,\n \t\t\t\t\t const struct extra_have_objects *in);\n+extern int mark_new_shallow_refs(const struct extra_have_objects *ref,\n+\t\t\t\t int *ref_status, uint32_t **used,\n+\t\t\t\t const struct extra_have_objects *shallow);\n \n int is_descendant_of(struct commit *, struct commit_list *);\n int in_merge_bases(struct commit *, struct commit *);\ndiff --git a/shallow.c b/shallow.c\nindex a974d2d..c92a1dc 100644\n--- a/shallow.c\n+++ b/shallow.c\n@@ -4,6 +4,8 @@\n #include \"pkt-line.h\"\n #include \"remote.h\"\n #include \"refs.h\"\n+#include \"diff.h\"\n+#include \"revision.h\"\n \n static int is_shallow = -1;\n static struct stat shallow_stat;\n@@ -280,3 +282,220 @@ void remove_reachable_shallow_points(struct extra_have_objects *out,\n \t}\n \tfree(ca.commits);\n }\n+\n+static int paint_down(const unsigned char *sha1, int id, int nr_bits, int quick)\n+{\n+\tint hit_bottom = 0;\n+\tunsigned int i, nr;\n+\tstruct commit_list *head = NULL;\n+\tint bitmap_nr = (nr_bits + 31) / 32;\n+\tint bitmap_size = bitmap_nr * sizeof(uint32_t);\n+\tuint32_t *tmp = xmalloc(bitmap_size);\n+\tuint32_t *bitmap = xcalloc(bitmap_size, sizeof(uint32_t));\n+\tbitmap[id / 32] |= (1 << (id % 32));\n+\tcommit_list_insert(lookup_commit(sha1), &head);\n+\twhile (head) {\n+\t\tstruct commit_list *p;\n+\t\tstruct commit *c = head->item;\n+\t\tuint32_t *c_util = c->util;\n+\n+\t\tp = head;\n+\t\thead = head->next;\n+\t\tfree(p);\n+\n+\t\tif (c->object.flags & (SEEN | UNINTERESTING))\n+\t\t\tcontinue;\n+\t\telse\n+\t\t\tc->object.flags |= SEEN;\n+\n+\t\tif (c->util == NULL)\n+\t\t\tc->util = bitmap;\n+\t\telse {\n+\t\t\t/*\n+\t\t\t * Deliberately leak a lot in commit->util\n+\t\t\t * because there can be many pointers to the\n+\t\t\t * same bitmap. Probably should allocate in a\n+\t\t\t * pool and free the whole pool at the end.\n+\t\t\t */\n+\t\t\tmemcpy(tmp, c_util, bitmap_size);\n+\t\t\tfor (i = 0; i < bitmap_nr; i++)\n+\t\t\t\ttmp[i] |= bitmap[i];\n+\t\t\tif (memcmp(tmp, c_util, bitmap_size)) {\n+\t\t\t\tc->util = xmalloc(bitmap_size);\n+\t\t\t\tmemcpy(c->util, tmp, bitmap_size);\n+\t\t\t}\n+\t\t}\n+\n+\t\tif (c->object.flags & BOTTOM) {\n+\t\t\thit_bottom = 1;\n+\t\t\tif (quick) {\n+\t\t\t\tfree_commit_list(head);\n+\t\t\t\tbreak;\n+\t\t\t} else\n+\t\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (parse_commit(c))\n+\t\t\tdie(\"unable to parse commit %s\",\n+\t\t\t sha1_to_hex(c->object.sha1));\n+\n+\t\tfor (p = c->parents; p; p = p->next) {\n+\t\t\tif (p->item->object.flags & SEEN)\n+\t\t\t\tcontinue;\n+\t\t\tif (p->item->util == NULL || p->item->util == c_util)\n+\t\t\t\tp->item->util = c->util;\n+\t\t\tcommit_list_insert(p->item, &head);\n+\t\t}\n+\t}\n+\n+\tnr = get_max_object_index();\n+\tfor (i = 0; i < nr; i++) {\n+\t\tstruct object *o = get_indexed_object(i);\n+\t\tif (o && o->type == OBJ_COMMIT) {\n+\t\t\to->flags &= ~SEEN;\n+\t\t}\n+\t}\n+\n+\tfree(tmp);\n+\treturn hit_bottom;\n+}\n+\n+static int mark_uninteresting(const char *refname,\n+\t\t\t const unsigned char *sha1,\n+\t\t\t int flags, void *cb_data)\n+{\n+\tstruct commit *commit = lookup_commit(sha1);\n+\tcommit->object.flags |= UNINTERESTING;\n+\tmark_parents_uninteresting(commit);\n+\treturn 0;\n+}\n+\n+struct saved_util {\n+\tunsigned char sha1[20];\n+\tvoid *util;\n+};\n+\n+/*\n+ * Given a set of refs and shallow roots, find out what ref can reach\n+ * any of the given roots. If so mark that ref \"reject_flag\". If\n+ * \"used\" is not NULL, mark all reachable roots. Return how many refs\n+ * that need new shallow points.\n+ */\n+int mark_new_shallow_refs(const struct extra_have_objects *ref,\n+\t\t\t int *ref_status, uint32_t **used,\n+\t\t\t const struct extra_have_objects *shallow)\n+{\n+\tstruct saved_util *util = NULL;\n+\tunsigned int i, nr, ret = 0, nr_util = 0, alloc_util = 0;\n+\n+\t/*\n+\t * Quick check to see if we may need to add new shallow\n+\t * roots. Go through the list of root candidates and check if\n+\t * they exist (either in current repo, or in the new pack, we\n+\t * can't distinguish).\n+\t *\n+\t * 1) If none of the new roots exist, the pack must connect to\n+\t * the main object graph, which is already guarded by\n+\t * current repo's shallow roots and we will not need to\n+\t * consider adding new shallow roots, so we can exit early.\n+\t *\n+\t * 2) The pack may connect to some existing object islands in\n+\t * current repo then add shallow roots to plug loose ends\n+\t * from those islands. In that case, new shallow roots must\n+\t * also exist in the repo as this stage (old objects plus\n+\t * the new pack).\n+\t *\n+\t * 3) The last, easiest case, is the pack contains some\n+\t * shallow roots, which may be used to tie up loose ends of\n+\t * some new refs, or redundanty (tying up loose ends of new\n+\t * object islands)\n+\t */\n+\tfor (i = 0;i < shallow->nr; i++)\n+\t\tif (has_sha1_file(shallow->array[i]))\n+\t\t\tbreak;\n+\tif (i == shallow->nr)\n+\t\t/*\n+\t\t * this is the first and also the common case, where\n+\t\t * the new pack does not carry any new shallow\n+\t\t * points. No need to to the expensive commit traverse\n+\t\t * dance below.\n+\t\t */\n+\t\treturn 0;\n+\n+\t/*\n+\t * Prepare the commit graph to track what refs can reach what\n+\t * (new) shallow points.\n+\t */\n+\tnr = get_max_object_index();\n+\tfor (i = 0; i < nr; i++) {\n+\t\tstruct object *o = get_indexed_object(i);\n+\t\tstruct commit *c = (struct commit *)o;\n+\t\tif (!o || o->type != OBJ_COMMIT)\n+\t\t\tcontinue;\n+\n+\t\to->flags &= ~(UNINTERESTING | BOTTOM | SEEN);\n+\t\t/*\n+\t\t * git-fetch makes use of \"util\" field. Save it and\n+\t\t * restore later. For fetch/clone/push, \"nr\" should be\n+\t\t * small because rev-list is delayed to pack-objects.\n+\t\t */\n+\t\tif (c->util) {\n+\t\t\tALLOC_GROW(util, nr_util+1, alloc_util);\n+\t\t\thashcpy(util[nr_util].sha1, o->sha1);\n+\t\t\tutil[nr_util].util = c->util;\n+\t\t\tnr_util++;\n+\t\t\tc->util = NULL;\n+\t\t}\n+\t}\n+\n+\t/*\n+\t * \"--not --all\" to cut short the traversal if new refs\n+\t * connect to old refs. If not (e.g. force ref updates) it'll\n+\t * have to go down to the current shallow roots.\n+\t *\n+\t * We could detect that a new commit is connected to an\n+\t * existing commit by keeping new objects in a pack (i.e. the\n+\t * index-pack code path) then check commit origin. If so stop\n+\t * short, so we don't need to get to the bottom. But then it\n+\t * will not work for case #2 because we need to go through\n+\t * some of our commits before reaching new shallow roots.\n+\t */\n+\thead_ref(mark_uninteresting, NULL);\n+\tfor_each_ref(mark_uninteresting, NULL);\n+\n+\tfor (i = 0; i < shallow->nr; i++)\n+\t\tif (has_sha1_file(shallow->array[i])) {\n+\t\t\tstruct commit *c = lookup_commit(shallow->array[i]);\n+\t\t\tc->object.flags |= BOTTOM;\n+\t\t}\n+\n+\tfor (i = 0; i < ref->nr; i++)\n+\t\tif (paint_down(ref->array[i], i, ref->nr, used == NULL)) {\n+\t\t\tif (ref_status)\n+\t\t\t\tref_status[i] = 1;\n+\t\t\tret++;\n+\t\t}\n+\n+\tif (used) {\n+\t\tfor (i = 0; i < shallow->nr; i++) {\n+\t\t\tstruct commit *c = lookup_commit(shallow->array[i]);\n+\t\t\tused[i] = c->util;\n+\t\t}\n+\t}\n+\n+\tif (nr_util) {\n+\t\tnr = get_max_object_index();\n+\t\tfor (i = 0; i < nr; i++) {\n+\t\t\tstruct object *o = get_indexed_object(i);\n+\t\t\tif (o && o->type == OBJ_COMMIT)\n+\t\t\t\t((struct commit *)o)->util = NULL;\n+\t\t}\n+\t\tfor (i = 0; i < nr_util; i++) {\n+\t\t\tstruct commit *c = lookup_commit(util[i].sha1);\n+\t\t\tc->util = util[i].util;\n+\t\t}\n+\t\tfree(util);\n+\t}\n+\n+\treturn ret;\n+}\n-- \n1.8.2.83.gc99314b\n"},{"id":"231036","messageId":"1385351754-9954-10-git-send-email-pclouds@gmail.com","threadId":"35400","inReplyTo":"1385351754-9954-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v3 09/28] shallow.c: extend setup_*_shallow() to accept extra shallow points","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-11-25T03:55:35Z","receivedAt":"2013-11-25T03:55:35Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n commit.h | 8 +++++---\n fetch-pack.c | 5 +++--\n shallow.c | 20 +++++++++++++++-----\n upload-pack.c | 2 +-\n 4 files changed, 24 insertions(+), 11 deletions(-)\n\ndiff --git a/commit.h b/commit.h\nindex e1fd587..3af4699 100644\n--- a/commit.h\n+++ b/commit.h\n@@ -203,10 +203,12 @@ extern struct commit_list *get_shallow_commits(struct object_array *heads,\n \t\tint depth, int shallow_flag, int not_shallow_flag);\n extern void check_shallow_file_for_update(void);\n extern void set_alternate_shallow_file(const char *path);\n-extern int write_shallow_commits(struct strbuf *out, int use_pack_protocol);\n+extern int write_shallow_commits(struct strbuf *out, int use_pack_protocol,\n+\t\t\t\t const struct extra_have_objects *extra);\n extern void setup_alternate_shallow(struct lock_file *shallow_lock,\n-\t\t\t\t const char **alternate_shallow_file);\n-extern char *setup_temporary_shallow(void);\n+\t\t\t\t const char **alternate_shallow_file,\n+\t\t\t\t const struct extra_have_objects *extra);\n+extern char *setup_temporary_shallow(const struct extra_have_objects *extra);\n extern void advertise_shallow_grafts(int);\n extern void remove_reachable_shallow_points(struct extra_have_objects *out,\n \t\t\t\t\t const struct extra_have_objects *in);\ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex 1042448..0e7483e 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -311,7 +311,7 @@ static int find_common(struct fetch_pack_args *args,\n \t}\n \n \tif (is_repository_shallow())\n-\t\twrite_shallow_commits(&req_buf, 1);\n+\t\twrite_shallow_commits(&req_buf, 1, NULL);\n \tif (args->depth > 0)\n \t\tpacket_buf_write(&req_buf, \"deepen %d\", args->depth);\n \tpacket_buf_flush(&req_buf);\n@@ -850,7 +850,8 @@ static struct ref *do_fetch_pack(struct fetch_pack_args *args,\n \tif (args->stateless_rpc)\n \t\tpacket_flush(fd[1]);\n \tif (args->depth > 0)\n-\t\tsetup_alternate_shallow(&shallow_lock, &alternate_shallow_file);\n+\t\tsetup_alternate_shallow(&shallow_lock, &alternate_shallow_file,\n+\t\t\t\t\tNULL);\n \telse\n \t\talternate_shallow_file = NULL;\n \tif (get_pack(args, fd, pack_lockfile))\ndiff --git a/shallow.c b/shallow.c\nindex c92a1dc..2ec0c8c 100644\n--- a/shallow.c\n+++ b/shallow.c\n@@ -169,22 +169,31 @@ static int write_one_shallow(const struct commit_graft *graft, void *cb_data)\n \treturn 0;\n }\n \n-int write_shallow_commits(struct strbuf *out, int use_pack_protocol)\n+int write_shallow_commits(struct strbuf *out, int use_pack_protocol,\n+\t\t\t const struct extra_have_objects *extra)\n {\n \tstruct write_shallow_data data;\n+\tint i;\n \tdata.out = out;\n \tdata.use_pack_protocol = use_pack_protocol;\n \tdata.count = 0;\n \tfor_each_commit_graft(write_one_shallow, &data);\n+\tif (!extra)\n+\t\treturn data.count;\n+\tfor (i = 0; i < extra->nr; i++) {\n+\t\tstrbuf_addstr(out, sha1_to_hex(extra->array[i]));\n+\t\tstrbuf_addch(out, '\\n');\n+\t\tdata.count++;\n+\t}\n \treturn data.count;\n }\n \n-char *setup_temporary_shallow(void)\n+char *setup_temporary_shallow(const struct extra_have_objects *extra)\n {\n \tstruct strbuf sb = STRBUF_INIT;\n \tint fd;\n \n-\tif (write_shallow_commits(&sb, 0)) {\n+\tif (write_shallow_commits(&sb, 0, extra)) {\n \t\tstruct strbuf path = STRBUF_INIT;\n \t\tstrbuf_addstr(&path, git_path(\"shallow_XXXXXX\"));\n \t\tfd = xmkstemp(path.buf);\n@@ -203,7 +212,8 @@ char *setup_temporary_shallow(void)\n }\n \n void setup_alternate_shallow(struct lock_file *shallow_lock,\n-\t\t\t const char **alternate_shallow_file)\n+\t\t\t const char **alternate_shallow_file,\n+\t\t\t const struct extra_have_objects *extra)\n {\n \tstruct strbuf sb = STRBUF_INIT;\n \tint fd;\n@@ -211,7 +221,7 @@ void setup_alternate_shallow(struct lock_file *shallow_lock,\n \tcheck_shallow_file_for_update();\n \tfd = hold_lock_file_for_update(shallow_lock, git_path(\"shallow\"),\n \t\t\t\t LOCK_DIE_ON_ERROR);\n-\tif (write_shallow_commits(&sb, 0)) {\n+\tif (write_shallow_commits(&sb, 0, extra)) {\n \t\tif (write_in_full(fd, sb.buf, sb.len) != sb.len)\n \t\t\tdie_errno(\"failed to write to %s\",\n \t\t\t\t shallow_lock->filename);\ndiff --git a/upload-pack.c b/upload-pack.c\nindex 38b2a29..f082f06 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -84,7 +84,7 @@ static void create_pack_file(void)\n \tchar *shallow_file = NULL;\n \n \tif (shallow_nr) {\n-\t\tshallow_file = setup_temporary_shallow();\n+\t\tshallow_file = setup_temporary_shallow(NULL);\n \t\targv[arg++] = \"--shallow-file\";\n \t\targv[arg++] = shallow_file;\n \t}\n-- \n1.8.2.83.gc99314b\n"},{"id":"231037","messageId":"1385351754-9954-11-git-send-email-pclouds@gmail.com","threadId":"35400","inReplyTo":"1385351754-9954-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v3 10/28] fetch-pack.c: move shallow update code out of fetch_pack()","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-11-25T03:55:36Z","receivedAt":"2013-11-25T03:55:36Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n fetch-pack.c | 22 +++++++++++++---------\n 1 file changed, 13 insertions(+), 9 deletions(-)\n\ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex 0e7483e..35d097e 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -925,6 +925,18 @@ static int remove_duplicates_in_refs(struct ref **ref, int nr)\n \treturn dst;\n }\n \n+static void update_shallow(struct fetch_pack_args *args)\n+{\n+\tif (args->depth > 0 && alternate_shallow_file) {\n+\t\tif (*alternate_shallow_file == '\\0') { /* --unshallow */\n+\t\t\tunlink_or_warn(git_path(\"shallow\"));\n+\t\t\trollback_lock_file(&shallow_lock);\n+\t\t} else\n+\t\t\tcommit_lock_file(&shallow_lock);\n+\t\treturn;\n+\t}\n+}\n+\n struct ref *fetch_pack(struct fetch_pack_args *args,\n \t\t int fd[], struct child_process *conn,\n \t\t const struct ref *ref,\n@@ -943,15 +955,7 @@ struct ref *fetch_pack(struct fetch_pack_args *args,\n \t\tdie(\"no matching remote head\");\n \t}\n \tref_cpy = do_fetch_pack(args, fd, ref, sought, nr_sought, pack_lockfile);\n-\n-\tif (args->depth > 0 && alternate_shallow_file) {\n-\t\tif (*alternate_shallow_file == '\\0') { /* --unshallow */\n-\t\t\tunlink_or_warn(git_path(\"shallow\"));\n-\t\t\trollback_lock_file(&shallow_lock);\n-\t\t} else\n-\t\t\tcommit_lock_file(&shallow_lock);\n-\t}\n-\n+\tupdate_shallow(args);\n \treprepare_packed_git();\n \treturn ref_cpy;\n }\n-- \n1.8.2.83.gc99314b\n"},{"id":"231038","messageId":"1385351754-9954-12-git-send-email-pclouds@gmail.com","threadId":"35400","inReplyTo":"1385351754-9954-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v3 11/28] fetch-pack.h: one statement per bitfield declaration","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-11-25T03:55:37Z","receivedAt":"2013-11-25T03:55:37Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n fetch-pack.h | 24 ++++++++++++------------\n 1 file changed, 12 insertions(+), 12 deletions(-)\n\ndiff --git a/fetch-pack.h b/fetch-pack.h\nindex 461cbf3..9b08388 100644\n--- a/fetch-pack.h\n+++ b/fetch-pack.h\n@@ -8,18 +8,18 @@ struct fetch_pack_args {\n \tconst char *uploadpack;\n \tint unpacklimit;\n \tint depth;\n-\tunsigned quiet:1,\n-\t\tkeep_pack:1,\n-\t\tlock_pack:1,\n-\t\tuse_thin_pack:1,\n-\t\tfetch_all:1,\n-\t\tstdin_refs:1,\n-\t\tverbose:1,\n-\t\tno_progress:1,\n-\t\tinclude_tag:1,\n-\t\tstateless_rpc:1,\n-\t\tcheck_self_contained_and_connected:1,\n-\t\tself_contained_and_connected:1;\n+\tunsigned quiet:1;\n+\tunsigned keep_pack:1;\n+\tunsigned lock_pack:1;\n+\tunsigned use_thin_pack:1;\n+\tunsigned fetch_all:1;\n+\tunsigned stdin_refs:1;\n+\tunsigned verbose:1;\n+\tunsigned no_progress:1;\n+\tunsigned include_tag:1;\n+\tunsigned stateless_rpc:1;\n+\tunsigned check_self_contained_and_connected:1;\n+\tunsigned self_contained_and_connected:1;\n };\n \n /*\n-- \n1.8.2.83.gc99314b\n"},{"id":"231039","messageId":"1385351754-9954-13-git-send-email-pclouds@gmail.com","threadId":"35400","inReplyTo":"1385351754-9954-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v3 12/28] clone: support remote shallow repository","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-11-25T03:55:38Z","receivedAt":"2013-11-25T03:55:38Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"The gist of cloning a remote shallow repository is simple. The remote\ngives you a pack and all its shallow roots. All you need to do is set\nyour .git/shallow correctly.\n\nThe rule to create this .git/shallow is simple and more importantly,\ncheap: if a shallow root is found in the pack, it's probably used\n(i.e. reachable from some refs), so we add it. Others are dropped.\n\nOne may notice this method seems flawed by the word \"probably\". A\nshallow point may not be reachable from any refs at all if it's\nattached to an object island (a group of objects that are not\nreachable by any refs).\n\nIf that object island is not complete, a new fetch request may send\nmore objects to connect it to some ref. At that time, because we\nincorrectly installed the shallow root in this island, the user will\nnot see anything after that root. This is not desired.\n\nThere is a stricter method to rule out unwanted shallow roots above:\nmark_new_shallow_refs(). But it's expensive and even more so for a\nclone because it's proportional to the (potentially huge) number of\ncommits.\n\nGiven that object islands are rare (C Git never sends such islands), a\ntradeoff is made to surprise the user occasionally but work faster\neveryday. --strict option could be added later to enforce\nmark_new_shallow_refs().\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/clone.c | 1 +\n builtin/fetch-pack.c | 2 +-\n fetch-pack.c | 50 +++++++++++++++++++++++++++++++++++++++++++++++---\n fetch-pack.h | 4 ++++\n transport.c | 11 ++++++++---\n transport.h | 6 ++++++\n 6 files changed, 67 insertions(+), 7 deletions(-)\n\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex 900f564..0b182ce 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -889,6 +889,7 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \n \tremote = remote_get(option_origin);\n \ttransport = transport_get(remote, remote->url[0]);\n+\ttransport->cloning = 1;\n \n \tif (!transport->get_refs_list || (!is_local && !transport->fetch))\n \t\tdie(_(\"Don't know how to clone %s\"), transport->url);\ndiff --git a/builtin/fetch-pack.c b/builtin/fetch-pack.c\nindex c1d918f..927424b 100644\n--- a/builtin/fetch-pack.c\n+++ b/builtin/fetch-pack.c\n@@ -153,7 +153,7 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)\n \tget_remote_heads(fd[0], NULL, 0, &ref, 0, NULL, NULL);\n \n \tref = fetch_pack(&args, fd, conn, ref, dest,\n-\t\t\t sought, nr_sought, pack_lockfile_ptr);\n+\t\t\t sought, nr_sought, NULL, pack_lockfile_ptr);\n \tif (pack_lockfile) {\n \t\tprintf(\"lock %s\\n\", pack_lockfile);\n \t\tfflush(stdout);\ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex 35d097e..b76581a 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -774,6 +774,7 @@ static struct ref *do_fetch_pack(struct fetch_pack_args *args,\n \t\t\t\t int fd[2],\n \t\t\t\t const struct ref *orig_ref,\n \t\t\t\t struct ref **sought, int nr_sought,\n+\t\t\t\t struct extra_have_objects *shallow,\n \t\t\t\t char **pack_lockfile)\n {\n \tstruct ref *ref = copy_ref_list(orig_ref);\n@@ -852,6 +853,8 @@ static struct ref *do_fetch_pack(struct fetch_pack_args *args,\n \tif (args->depth > 0)\n \t\tsetup_alternate_shallow(&shallow_lock, &alternate_shallow_file,\n \t\t\t\t\tNULL);\n+\telse if (args->cloning && shallow && shallow->nr)\n+\t\talternate_shallow_file = setup_temporary_shallow(shallow);\n \telse\n \t\talternate_shallow_file = NULL;\n \tif (get_pack(args, fd, pack_lockfile))\n@@ -925,8 +928,11 @@ static int remove_duplicates_in_refs(struct ref **ref, int nr)\n \treturn dst;\n }\n \n-static void update_shallow(struct fetch_pack_args *args)\n+static void update_shallow(struct fetch_pack_args *args,\n+\t\t\t struct extra_have_objects *shallow)\n {\n+\tint i;\n+\n \tif (args->depth > 0 && alternate_shallow_file) {\n \t\tif (*alternate_shallow_file == '\\0') { /* --unshallow */\n \t\t\tunlink_or_warn(git_path(\"shallow\"));\n@@ -935,6 +941,42 @@ static void update_shallow(struct fetch_pack_args *args)\n \t\t\tcommit_lock_file(&shallow_lock);\n \t\treturn;\n \t}\n+\n+\tif (!shallow || !shallow->nr)\n+\t\treturn;\n+\n+\tif (alternate_shallow_file) {\n+\t\t/*\n+\t\t * The temporary shallow file is only useful for\n+\t\t * index-pack and unpack-objects because it may\n+\t\t * contain more roots than we want. Delete it.\n+\t\t */\n+\t\tif (*alternate_shallow_file)\n+\t\t\tunlink(alternate_shallow_file);\n+\t\tfree((char*)alternate_shallow_file);\n+\t}\n+\n+\tif (args->cloning) {\n+\t\t/*\n+\t\t * remote is shallow, but this is a clone, there are\n+\t\t * no objects in repo to worry about. Accept any\n+\t\t * shallow points that exist in the pack (iow in repo\n+\t\t * after get_pack() and reprepare_packed_git())\n+\t\t */\n+\t\tstruct extra_have_objects extra;\n+\t\tmemset(&extra, 0, sizeof(extra));\n+\t\tfor (i = 0; i < shallow->nr; i++)\n+\t\t\tif (has_sha1_file(shallow->array[i]))\n+\t\t\t\tadd_extra_have(&extra, shallow->array[i]);\n+\t\tif (extra.nr) {\n+\t\t\tsetup_alternate_shallow(&shallow_lock,\n+\t\t\t\t\t\t&alternate_shallow_file,\n+\t\t\t\t\t\t&extra);\n+\t\t\tcommit_lock_file(&shallow_lock);\n+\t\t}\n+\t\tfree(extra.array);\n+\t\treturn;\n+\t}\n }\n \n struct ref *fetch_pack(struct fetch_pack_args *args,\n@@ -942,6 +984,7 @@ struct ref *fetch_pack(struct fetch_pack_args *args,\n \t\t const struct ref *ref,\n \t\t const char *dest,\n \t\t struct ref **sought, int nr_sought,\n+\t\t struct extra_have_objects *shallow,\n \t\t char **pack_lockfile)\n {\n \tstruct ref *ref_cpy;\n@@ -954,8 +997,9 @@ struct ref *fetch_pack(struct fetch_pack_args *args,\n \t\tpacket_flush(fd[1]);\n \t\tdie(\"no matching remote head\");\n \t}\n-\tref_cpy = do_fetch_pack(args, fd, ref, sought, nr_sought, pack_lockfile);\n-\tupdate_shallow(args);\n+\tref_cpy = do_fetch_pack(args, fd, ref, sought, nr_sought,\n+\t\t\t\tshallow, pack_lockfile);\n \treprepare_packed_git();\n+\tupdate_shallow(args, shallow);\n \treturn ref_cpy;\n }\ndiff --git a/fetch-pack.h b/fetch-pack.h\nindex 9b08388..cabfb60 100644\n--- a/fetch-pack.h\n+++ b/fetch-pack.h\n@@ -4,6 +4,8 @@\n #include \"string-list.h\"\n #include \"run-command.h\"\n \n+struct extra_have_objects;\n+\n struct fetch_pack_args {\n \tconst char *uploadpack;\n \tint unpacklimit;\n@@ -20,6 +22,7 @@ struct fetch_pack_args {\n \tunsigned stateless_rpc:1;\n \tunsigned check_self_contained_and_connected:1;\n \tunsigned self_contained_and_connected:1;\n+\tunsigned cloning:1;\n };\n \n /*\n@@ -33,6 +36,7 @@ struct ref *fetch_pack(struct fetch_pack_args *args,\n \t\t const char *dest,\n \t\t struct ref **sought,\n \t\t int nr_sought,\n+\t\t struct extra_have_objects *shallow,\n \t\t char **pack_lockfile);\n \n #endif\ndiff --git a/transport.c b/transport.c\nindex 9c51767..fa3dc16 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -455,6 +455,7 @@ struct git_transport_data {\n \tint fd[2];\n \tunsigned got_remote_heads : 1;\n \tstruct extra_have_objects extra_have;\n+\tstruct extra_have_objects shallow;\n };\n \n static int set_git_option(struct git_transport_options *opts,\n@@ -511,7 +512,9 @@ static struct ref *get_refs_via_connect(struct transport *transport, int for_pus\n \n \tconnect_setup(transport, for_push, 0);\n \tget_remote_heads(data->fd[0], NULL, 0, &refs,\n-\t\t\t for_push ? REF_NORMAL : 0, &data->extra_have, NULL);\n+\t\t\t for_push ? REF_NORMAL : 0,\n+\t\t\t &data->extra_have,\n+\t\t\t transport->cloning ? &data->shallow : NULL);\n \tdata->got_remote_heads = 1;\n \n \treturn refs;\n@@ -538,17 +541,19 @@ static int fetch_refs_via_pack(struct transport *transport,\n \targs.depth = data->options.depth;\n \targs.check_self_contained_and_connected =\n \t\tdata->options.check_self_contained_and_connected;\n+\targs.cloning = transport->cloning;\n \n \tif (!data->got_remote_heads) {\n \t\tconnect_setup(transport, 0, 0);\n \t\tget_remote_heads(data->fd[0], NULL, 0, &refs_tmp, 0,\n-\t\t\t\t NULL, NULL);\n+\t\t\t\t NULL,\n+\t\t\t\t transport->cloning ? &data->shallow : NULL);\n \t\tdata->got_remote_heads = 1;\n \t}\n \n \trefs = fetch_pack(&args, data->fd, data->conn,\n \t\t\t refs_tmp ? refs_tmp : transport->remote_refs,\n-\t\t\t dest, to_fetch, nr_heads,\n+\t\t\t dest, to_fetch, nr_heads, &data->shallow,\n \t\t\t &transport->pack_lockfile);\n \tclose(data->fd[0]);\n \tclose(data->fd[1]);\ndiff --git a/transport.h b/transport.h\nindex b3679bb..59842d4 100644\n--- a/transport.h\n+++ b/transport.h\n@@ -35,6 +35,12 @@ struct transport {\n \t */\n \tunsigned cannot_reuse : 1;\n \n+\t/*\n+\t * A hint from caller that it will be performing a clone, not\n+\t * normal fetch. IOW the repository is guaranteed empty.\n+\t */\n+\tunsigned cloning : 1;\n+\n \t/**\n \t * Returns 0 if successful, positive if the option is not\n \t * recognized or is inapplicable, and negative if the option\n-- \n1.8.2.83.gc99314b\n"},{"id":"231040","messageId":"1385351754-9954-14-git-send-email-pclouds@gmail.com","threadId":"35400","inReplyTo":"1385351754-9954-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v3 13/28] fetch: support fetching from a shallow repository","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-11-25T03:55:39Z","receivedAt":"2013-11-25T03:55:39Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"This patch just put together pieces from the previous patches:\n\n - Before getting the new pack, we need to remove all new reachable\n shallow roots. The remaining roots may or may not be added to\n .git/shallow.\n\n - After getting the pack, walk all new refs until they connect to\n current refs, or hit the bottom of current repo, or hit new\n shallow roots.\n\nThose refs that hit new shallow roots are rejected because by default\nwe do not allow to update .git/shallow (the only exception so far is\ncloning from a shallow repo, which is more like creating .git/shallow\nthan updating it)\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/fetch.c | 9 +++\n fetch-pack.c | 35 ++++++++++-\n remote.h | 1 +\n t/t5536-fetch-shallow.sh (new +x) | 128 ++++++++++++++++++++++++++++++++++++++\n transport.c | 11 +++-\n 5 files changed, 179 insertions(+), 5 deletions(-)\n create mode 100755 t/t5536-fetch-shallow.sh\n\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex bd7a101..7b41a7e 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -405,6 +405,8 @@ static int iterate_ref_map(void *cb_data, unsigned char sha1[20])\n \tstruct ref **rm = cb_data;\n \tstruct ref *ref = *rm;\n \n+\twhile (ref && ref->status == REF_STATUS_REJECT_SHALLOW)\n+\t\tref = ref->next;\n \tif (!ref)\n \t\treturn -1; /* end of the list */\n \t*rm = ref->next;\n@@ -451,6 +453,13 @@ static int store_updated_refs(const char *raw_url, const char *remote_name,\n \t\t\tstruct ref *ref = NULL;\n \t\t\tconst char *merge_status_marker = \"\";\n \n+\t\t\tif (rm->status == REF_STATUS_REJECT_SHALLOW) {\n+\t\t\t\tif (want_status == FETCH_HEAD_MERGE)\n+\t\t\t\t\twarning(_(\"reject %s because shallow roots are not allowed to be updated\"),\n+\t\t\t\t\t\trm->peer_ref ? rm->peer_ref->name : rm->name);\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\n \t\t\tcommit = lookup_commit_reference_gently(rm->old_sha1, 1);\n \t\t\tif (!commit)\n \t\t\t\trm->fetch_head_status = FETCH_HEAD_NOT_FOR_MERGE;\ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex b76581a..64fa5d2 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -855,7 +855,17 @@ static struct ref *do_fetch_pack(struct fetch_pack_args *args,\n \t\t\t\t\tNULL);\n \telse if (args->cloning && shallow && shallow->nr)\n \t\talternate_shallow_file = setup_temporary_shallow(shallow);\n-\telse\n+\telse if (!args->cloning && shallow && shallow->nr) {\n+\t\tstruct extra_have_objects extra;\n+\t\tmemset(&extra, 0, sizeof(extra));\n+\t\tremove_reachable_shallow_points(&extra, shallow);\n+\t\tif (extra.nr) {\n+\t\t\talternate_shallow_file = setup_temporary_shallow(&extra);\n+\t\t\tfree(shallow->array);\n+\t\t\t*shallow = extra;\n+\t\t} else\n+\t\t\talternate_shallow_file = NULL;\n+\t} else\n \t\talternate_shallow_file = NULL;\n \tif (get_pack(args, fd, pack_lockfile))\n \t\tdie(\"git fetch-pack: fetch failed.\");\n@@ -929,8 +939,11 @@ static int remove_duplicates_in_refs(struct ref **ref, int nr)\n }\n \n static void update_shallow(struct fetch_pack_args *args,\n+\t\t\t struct ref **sought, int nr_sought,\n \t\t\t struct extra_have_objects *shallow)\n {\n+\tstruct extra_have_objects ref;\n+\tint *status;\n \tint i;\n \n \tif (args->depth > 0 && alternate_shallow_file) {\n@@ -977,6 +990,24 @@ static void update_shallow(struct fetch_pack_args *args,\n \t\tfree(extra.array);\n \t\treturn;\n \t}\n+\n+\tmemset(&ref, 0, sizeof(ref));\n+\tfor (i = 0; i < nr_sought; i++)\n+\t\tadd_extra_have(&ref, sought[i]->old_sha1);\n+\n+\tstatus = xcalloc(nr_sought, sizeof(*status));\n+\n+\t/*\n+\t * remote is also shallow, check what ref is safe to update\n+\t * without updating .git/shallow\n+\t */\n+\tif (mark_new_shallow_refs(&ref, status, NULL, shallow)) {\n+\t\tfor (i = 0; i < nr_sought; i++)\n+\t\t\tif (status[i])\n+\t\t\t\tsought[i]->status = REF_STATUS_REJECT_SHALLOW;\n+\t}\n+\tfree(status);\n+\tfree(ref.array);\n }\n \n struct ref *fetch_pack(struct fetch_pack_args *args,\n@@ -1000,6 +1031,6 @@ struct ref *fetch_pack(struct fetch_pack_args *args,\n \tref_cpy = do_fetch_pack(args, fd, ref, sought, nr_sought,\n \t\t\t\tshallow, pack_lockfile);\n \treprepare_packed_git();\n-\tupdate_shallow(args, shallow);\n+\tupdate_shallow(args, sought, nr_sought, shallow);\n \treturn ref_cpy;\n }\ndiff --git a/remote.h b/remote.h\nindex ff604ff..e519c26 100644\n--- a/remote.h\n+++ b/remote.h\n@@ -109,6 +109,7 @@ struct ref {\n \t\tREF_STATUS_REJECT_FETCH_FIRST,\n \t\tREF_STATUS_REJECT_NEEDS_FORCE,\n \t\tREF_STATUS_REJECT_STALE,\n+\t\tREF_STATUS_REJECT_SHALLOW,\n \t\tREF_STATUS_UPTODATE,\n \t\tREF_STATUS_REMOTE_REJECT,\n \t\tREF_STATUS_EXPECTING_REPORT\ndiff --git a/t/t5536-fetch-shallow.sh b/t/t5536-fetch-shallow.sh\nnew file mode 100755\nindex 0000000..41de74d\n--- /dev/null\n+++ b/t/t5536-fetch-shallow.sh\n@@ -0,0 +1,128 @@\n+#!/bin/sh\n+\n+test_description='fetch/clone from a shallow clone'\n+\n+. ./test-lib.sh\n+\n+commit() {\n+\techo \"$1\" >tracked &&\n+\tgit add tracked &&\n+\tgit commit -m \"$1\"\n+}\n+\n+test_expect_success 'setup' '\n+\tcommit 1 &&\n+\tcommit 2 &&\n+\tcommit 3 &&\n+\tcommit 4 &&\n+\tgit config --global transfer.fsckObjects true\n+'\n+\n+test_expect_success 'setup shallow clone' '\n+\tgit clone --no-local --depth=2 .git shallow &&\n+\tgit --git-dir=shallow/.git log --format=%s >actual &&\n+\tcat <expect &&\n+4\n+3\n+EOF\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'clone from shallow clone' '\n+\tgit clone --no-local shallow shallow2 &&\n+\t(\n+\tcd shallow2 &&\n+\tgit fsck &&\n+\tgit log --format=%s >actual &&\n+\tcat <expect &&\n+4\n+3\n+EOF\n+\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'fetch from shallow clone' '\n+\t(\n+\tcd shallow &&\n+\tcommit 5\n+\t) &&\n+\t(\n+\tcd shallow2 &&\n+\tgit fetch &&\n+\tgit fsck &&\n+\tgit log --format=%s origin/master >actual &&\n+\tcat <expect &&\n+5\n+4\n+3\n+EOF\n+\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'fetch --depth from shallow clone' '\n+\t(\n+\tcd shallow &&\n+\tcommit 6\n+\t) &&\n+\t(\n+\tcd shallow2 &&\n+\tgit fetch --depth=2 &&\n+\tgit fsck &&\n+\tgit log --format=%s origin/master >actual &&\n+\tcat <expect &&\n+6\n+5\n+EOF\n+\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'fetch something upstream has but hidden by clients shallow boundaries' '\n+\t# the blob \"1\" is available in .git but hidden by the\n+\t# shallow2/.git/shallow and it should be resent\n+\t! git --git-dir=shallow2/.git cat-file blob `echo 1|git hash-object --stdin` >/dev/null &&\n+\techo 1 > 1.t &&\n+\tgit add 1.t &&\n+\tgit commit -m add-1-back &&\n+\t(\n+\tcd shallow2 &&\n+\tgit fetch ../.git +refs/heads/master:refs/remotes/top/master &&\n+\tgit fsck &&\n+\tgit log --format=%s top/master >actual &&\n+\tcat <expect &&\n+add-1-back\n+4\n+3\n+EOF\n+\ttest_cmp expect actual\n+\t) &&\n+\tgit --git-dir=shallow2/.git cat-file blob `echo 1|git hash-object --stdin` >/dev/null\n+\n+'\n+\n+test_expect_success 'fetch that requires changes in .git/shallow is filtered' '\n+\t(\n+\tcd shallow &&\n+\tgit checkout --orphan no-shallow &&\n+\tcommit no-shallow\n+\t) &&\n+\tgit init notshallow &&\n+\t(\n+\tcd notshallow &&\n+\tgit fetch ../shallow/.git refs/heads/*:refs/remotes/shallow/*&&\n+\tgit for-each-ref --format=\"%(refname)\" >actual.refs &&\n+\tcat <expect.refs &&\n+refs/remotes/shallow/no-shallow\n+EOF\n+\ttest_cmp expect.refs actual.refs &&\n+\tgit log --format=%s shallow/no-shallow >actual &&\n+\tcat <expect &&\n+no-shallow\n+EOF\n+\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_done\ndiff --git a/transport.c b/transport.c\nindex fa3dc16..d6d14eb 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -514,7 +514,7 @@ static struct ref *get_refs_via_connect(struct transport *transport, int for_pus\n \tget_remote_heads(data->fd[0], NULL, 0, &refs,\n \t\t\t for_push ? REF_NORMAL : 0,\n \t\t\t &data->extra_have,\n-\t\t\t transport->cloning ? &data->shallow : NULL);\n+\t\t\t &data->shallow);\n \tdata->got_remote_heads = 1;\n \n \treturn refs;\n@@ -546,8 +546,7 @@ static int fetch_refs_via_pack(struct transport *transport,\n \tif (!data->got_remote_heads) {\n \t\tconnect_setup(transport, 0, 0);\n \t\tget_remote_heads(data->fd[0], NULL, 0, &refs_tmp, 0,\n-\t\t\t\t NULL,\n-\t\t\t\t transport->cloning ? &data->shallow : NULL);\n+\t\t\t\t NULL, &data->shallow);\n \t\tdata->got_remote_heads = 1;\n \t}\n \n@@ -719,6 +718,10 @@ static int print_one_push_status(struct ref *ref, const char *dest, int count, i\n \t\tprint_ref_status('!', \"[rejected]\", ref, ref->peer_ref,\n \t\t\t\t\t\t \"stale info\", porcelain);\n \t\tbreak;\n+\tcase REF_STATUS_REJECT_SHALLOW:\n+\t\tprint_ref_status('!', \"[rejected]\", ref, ref->peer_ref,\n+\t\t\t\t\t\t \"new shallow roots not allowed\", porcelain);\n+\t\tbreak;\n \tcase REF_STATUS_REMOTE_REJECT:\n \t\tprint_ref_status('!', \"[remote rejected]\", ref,\n \t\t\t\t\t\t ref->deletion ? NULL : ref->peer_ref,\n@@ -814,6 +817,8 @@ static int git_transport_push(struct transport *transport, struct ref *remote_re\n \t\tget_remote_heads(data->fd[0], NULL, 0, &tmp_refs, REF_NORMAL, NULL, NULL);\n \t\tdata->got_remote_heads = 1;\n \t}\n+\tif (data->shallow.nr)\n+\t\tdie(\"pushing to a shallow repository is not supported\");\n \n \tmemset(&args, 0, sizeof(args));\n \targs.send_mirror = !!(flags & TRANSPORT_PUSH_MIRROR);\n-- \n1.8.2.83.gc99314b\n"},{"id":"231041","messageId":"1385351754-9954-15-git-send-email-pclouds@gmail.com","threadId":"35400","inReplyTo":"1385351754-9954-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v3 14/28] upload-pack: make sure deepening preserves shallow roots","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-11-25T03:55:40Z","receivedAt":"2013-11-25T03:55:40Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"When \"fetch --depth=N\" where N exceeds the longest chain of history in\nthe source repo, usually we just send an \"unshallow\" line to the\nclient so full history is obtained.\n\nWhen the source repo is shallow we need to make sure to \"unshallow\"\nthe current shallow point _and_ \"shallow\" again when the commit\nreaches its shallow bottom in the source repo.\n\nThis should fix both cases: large and --unshallow.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n Documentation/fetch-options.txt | 8 ++++++--\n shallow.c | 6 +++++-\n t/t5536-fetch-shallow.sh | 16 ++++++++++++++++\n upload-pack.c | 2 +-\n 4 files changed, 28 insertions(+), 4 deletions(-)\n\ndiff --git a/Documentation/fetch-options.txt b/Documentation/fetch-options.txt\nindex ba1fe49..a83d2b4 100644\n--- a/Documentation/fetch-options.txt\n+++ b/Documentation/fetch-options.txt\n@@ -14,8 +14,12 @@\n \tbranch history. Tags for the deepened commits are not fetched.\n \n --unshallow::\n-\tConvert a shallow repository to a complete one, removing all\n-\tthe limitations imposed by shallow repositories.\n+\tIf the source repository is complete, convert a shallow\n+\trepository to a complete one, removing all the limitations\n+\timposed by shallow repositories.\n++\n+If the source repository is shallow, fetch as much as possible so that\n+the current repository has the same history as the source repository.\n \n ifndef::git-pull[]\n --dry-run::\ndiff --git a/shallow.c b/shallow.c\nindex 2ec0c8c..1baf6b5 100644\n--- a/shallow.c\n+++ b/shallow.c\n@@ -73,6 +73,7 @@ struct commit_list *get_shallow_commits(struct object_array *heads, int depth,\n \tstruct commit_list *result = NULL;\n \tstruct object_array stack = OBJECT_ARRAY_INIT;\n \tstruct commit *commit = NULL;\n+\tstruct commit_graft *graft;\n \n \twhile (commit || i < heads->nr || stack.nr) {\n \t\tstruct commit_list *p;\n@@ -97,7 +98,10 @@ struct commit_list *get_shallow_commits(struct object_array *heads, int depth,\n \t\tif (parse_commit(commit))\n \t\t\tdie(\"invalid commit\");\n \t\tcur_depth++;\n-\t\tif (cur_depth >= depth) {\n+\t\tif ((depth != INFINITE_DEPTH && cur_depth >= depth) ||\n+\t\t (is_repository_shallow() && !commit->parents &&\n+\t\t (graft = lookup_commit_graft(commit->object.sha1)) != NULL &&\n+\t\t graft->nr_parent < 0)) {\n \t\t\tcommit_list_insert(commit, &result);\n \t\t\tcommit->object.flags |= shallow_flag;\n \t\t\tcommit = NULL;\ndiff --git a/t/t5536-fetch-shallow.sh b/t/t5536-fetch-shallow.sh\nindex 41de74d..e011ead 100755\n--- a/t/t5536-fetch-shallow.sh\n+++ b/t/t5536-fetch-shallow.sh\n@@ -79,6 +79,22 @@ EOF\n \t)\n '\n \n+test_expect_success 'fetch --unshallow from shallow clone' '\n+\t(\n+\tcd shallow2 &&\n+\tgit fetch --unshallow &&\n+\tgit fsck &&\n+\tgit log --format=%s origin/master >actual &&\n+\tcat <expect &&\n+6\n+5\n+4\n+3\n+EOF\n+\ttest_cmp expect actual\n+\t)\n+'\n+\n test_expect_success 'fetch something upstream has but hidden by clients shallow boundaries' '\n \t# the blob \"1\" is available in .git but hidden by the\n \t# shallow2/.git/shallow and it should be resent\ndiff --git a/upload-pack.c b/upload-pack.c\nindex f082f06..28269c7 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -619,7 +619,7 @@ static void receive_needs(void)\n \tif (depth > 0) {\n \t\tstruct commit_list *result = NULL, *backup = NULL;\n \t\tint i;\n-\t\tif (depth == INFINITE_DEPTH)\n+\t\tif (depth == INFINITE_DEPTH && !is_repository_shallow())\n \t\t\tfor (i = 0; i < shallows.nr; i++) {\n \t\t\t\tstruct object *object = shallows.objects[i].item;\n \t\t\t\tobject->flags |= NOT_SHALLOW;\n-- \n1.8.2.83.gc99314b\n"},{"id":"231048","messageId":"1385351754-9954-16-git-send-email-pclouds@gmail.com","threadId":"35400","inReplyTo":"1385351754-9954-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v3 15/28] fetch: add --update-shallow to get refs that require updating .git/shallow","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-11-25T03:55:41Z","receivedAt":"2013-11-25T03:55:41Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n Documentation/fetch-options.txt | 6 ++++++\n builtin/fetch.c | 6 +++++-\n fetch-pack.c | 26 ++++++++++++++++++++++++++\n fetch-pack.h | 1 +\n t/t5536-fetch-shallow.sh | 22 ++++++++++++++++++++++\n transport.c | 4 ++++\n transport.h | 4 ++++\n 7 files changed, 68 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/fetch-options.txt b/Documentation/fetch-options.txt\nindex a83d2b4..54043e3 100644\n--- a/Documentation/fetch-options.txt\n+++ b/Documentation/fetch-options.txt\n@@ -21,6 +21,12 @@\n If the source repository is shallow, fetch as much as possible so that\n the current repository has the same history as the source repository.\n \n+--update-shallow::\n+\tBy default when fetching from a shallow repository,\n+\t`git fetch` refuses refs that require updating\n+\t.git/shallow. This option updates .git/shallow and accept such\n+\trefs.\n+\n ifndef::git-pull[]\n --dry-run::\n \tShow what would be done, without making any changes.\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex 7b41a7e..d2e4fc0 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -36,7 +36,7 @@ static int prune = -1; /* unspecified */\n \n static int all, append, dry_run, force, keep, multiple, update_head_ok, verbosity;\n static int progress = -1, recurse_submodules = RECURSE_SUBMODULES_DEFAULT;\n-static int tags = TAGS_DEFAULT, unshallow;\n+static int tags = TAGS_DEFAULT, unshallow, update_shallow;\n static const char *depth;\n static const char *upload_pack;\n static struct strbuf default_rla = STRBUF_INIT;\n@@ -104,6 +104,8 @@ static struct option builtin_fetch_options[] = {\n \t{ OPTION_STRING, 0, \"recurse-submodules-default\",\n \t\t &recurse_submodules_default, NULL,\n \t\t N_(\"default mode for recursion\"), PARSE_OPT_HIDDEN },\n+\tOPT_BOOL(0, \"update-shallow\", &update_shallow,\n+\t\t N_(\"accept refs that update .git/shallow\")),\n \tOPT_END()\n };\n \n@@ -768,6 +770,8 @@ static struct transport *prepare_transport(struct remote *remote)\n \t\tset_option(transport, TRANS_OPT_KEEP, \"yes\");\n \tif (depth)\n \t\tset_option(transport, TRANS_OPT_DEPTH, depth);\n+\tif (update_shallow)\n+\t\tset_option(transport, TRANS_OPT_UPDATE_SHALLOW, \"yes\");\n \treturn transport;\n }\n \ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex 64fa5d2..82aa5db 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -995,6 +995,32 @@ static void update_shallow(struct fetch_pack_args *args,\n \tfor (i = 0; i < nr_sought; i++)\n \t\tadd_extra_have(&ref, sought[i]->old_sha1);\n \n+\tif (args->update_shallow) {\n+\t\t/*\n+\t\t * remote is also shallow, .git/shallow may be updated\n+\t\t * so all refs can be accepted. Make sure we only add\n+\t\t * shallow roots that are actually reachable from new\n+\t\t * refs.\n+\t\t */\n+\t\tuint32_t **used = xmalloc(sizeof(*used) * shallow->nr);\n+\t\tif (mark_new_shallow_refs(&ref, NULL, used, shallow)) {\n+\t\t\tstruct extra_have_objects extra;\n+\t\t\tmemset(&extra, 0, sizeof(extra));\n+\t\t\tfor (i = 0; i < shallow->nr; i++)\n+\t\t\t\tif (used[i])\n+\t\t\t\t\tadd_extra_have(&extra,\n+\t\t\t\t\t\t shallow->array[i]);\n+\t\t\tsetup_alternate_shallow(&shallow_lock,\n+\t\t\t\t\t\t&alternate_shallow_file,\n+\t\t\t\t\t\t&extra);\n+\t\t\tcommit_lock_file(&shallow_lock);\n+\t\t\tfree(extra.array);\n+\t\t}\n+\t\tfree(used);\n+\t\tfree(ref.array);\n+\t\treturn;\n+\t}\n+\n \tstatus = xcalloc(nr_sought, sizeof(*status));\n \n \t/*\ndiff --git a/fetch-pack.h b/fetch-pack.h\nindex cabfb60..5cfb77b 100644\n--- a/fetch-pack.h\n+++ b/fetch-pack.h\n@@ -23,6 +23,7 @@ struct fetch_pack_args {\n \tunsigned check_self_contained_and_connected:1;\n \tunsigned self_contained_and_connected:1;\n \tunsigned cloning:1;\n+\tunsigned update_shallow:1;\n };\n \n /*\ndiff --git a/t/t5536-fetch-shallow.sh b/t/t5536-fetch-shallow.sh\nindex e011ead..95b6313 100755\n--- a/t/t5536-fetch-shallow.sh\n+++ b/t/t5536-fetch-shallow.sh\n@@ -141,4 +141,26 @@ EOF\n \t)\n '\n \n+test_expect_success 'fetch --update-shallow' '\n+\t(\n+\tcd notshallow &&\n+\tgit fetch --update-shallow ../shallow/.git refs/heads/*:refs/remotes/shallow/* &&\n+\tgit fsck &&\n+\tgit for-each-ref --format=\"%(refname)\" |sort >actual.refs &&\n+\tcat <expect.refs &&\n+refs/remotes/shallow/master\n+refs/remotes/shallow/no-shallow\n+EOF\n+\ttest_cmp expect.refs actual.refs &&\n+\tgit log --format=%s shallow/master >actual &&\n+\tcat <expect &&\n+6\n+5\n+4\n+3\n+EOF\n+\ttest_cmp expect actual\n+\t)\n+'\n+\n test_done\ndiff --git a/transport.c b/transport.c\nindex d6d14eb..c0be6b1 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -476,6 +476,9 @@ static int set_git_option(struct git_transport_options *opts,\n \t} else if (!strcmp(name, TRANS_OPT_KEEP)) {\n \t\topts->keep = !!value;\n \t\treturn 0;\n+\t} else if (!strcmp(name, TRANS_OPT_UPDATE_SHALLOW)) {\n+\t\topts->update_shallow = !!value;\n+\t\treturn 0;\n \t} else if (!strcmp(name, TRANS_OPT_DEPTH)) {\n \t\tif (!value)\n \t\t\topts->depth = 0;\n@@ -542,6 +545,7 @@ static int fetch_refs_via_pack(struct transport *transport,\n \targs.check_self_contained_and_connected =\n \t\tdata->options.check_self_contained_and_connected;\n \targs.cloning = transport->cloning;\n+\targs.update_shallow = data->options.update_shallow;\n \n \tif (!data->got_remote_heads) {\n \t\tconnect_setup(transport, 0, 0);\ndiff --git a/transport.h b/transport.h\nindex 59842d4..02ea248 100644\n--- a/transport.h\n+++ b/transport.h\n@@ -11,6 +11,7 @@ struct git_transport_options {\n \tunsigned followtags : 1;\n \tunsigned check_self_contained_and_connected : 1;\n \tunsigned self_contained_and_connected : 1;\n+\tunsigned update_shallow : 1;\n \tint depth;\n \tconst char *uploadpack;\n \tconst char *receivepack;\n@@ -152,6 +153,9 @@ struct transport *transport_get(struct remote *, const char *);\n /* Aggressively fetch annotated tags if possible */\n #define TRANS_OPT_FOLLOWTAGS \"followtags\"\n \n+/* Accept refs that may update .git/shallow without --depth */\n+#define TRANS_OPT_UPDATE_SHALLOW \"updateshallow\"\n+\n /**\n * Returns 0 if the option was used, non-zero otherwise. Prints a\n * message to stderr if the option is not used.\n-- \n1.8.2.83.gc99314b\n"},{"id":"231043","messageId":"1385351754-9954-17-git-send-email-pclouds@gmail.com","threadId":"35400","inReplyTo":"1385351754-9954-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v3 16/28] receive-pack: reorder some code in unpack()","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-11-25T03:55:42Z","receivedAt":"2013-11-25T03:55:42Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"This is the preparation for adding --shallow-file to both\nunpack-objects and index-pack. To sum up:\n\n - status/code, ip/child, unpacker/keeper and i (now ac) are moved out\n to function top level\n\n - successful flow now ends at the end of the function\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/receive-pack.c | 74 +++++++++++++++++++++++---------------------------\n 1 file changed, 34 insertions(+), 40 deletions(-)\n\ndiff --git a/builtin/receive-pack.c b/builtin/receive-pack.c\nindex cc8c34f..22e162d 100644\n--- a/builtin/receive-pack.c\n+++ b/builtin/receive-pack.c\n@@ -822,8 +822,11 @@ static const char *pack_lockfile;\n static const char *unpack(int err_fd)\n {\n \tstruct pack_header hdr;\n+\tconst char *argv[7];\n \tconst char *hdr_err;\n+\tint status, ac = 0;\n \tchar hdr_arg[38];\n+\tstruct child_process child;\n \tint fsck_objects = (receive_fsck_objects >= 0\n \t\t\t ? receive_fsck_objects\n \t\t\t : transfer_fsck_objects >= 0\n@@ -840,63 +843,54 @@ static const char *unpack(int err_fd)\n \t\t\t\"--pack_header=%\"PRIu32\",%\"PRIu32,\n \t\t\tntohl(hdr.hdr_version), ntohl(hdr.hdr_entries));\n \n+\tmemset(&child, 0, sizeof(child));\n \tif (ntohl(hdr.hdr_entries) < unpack_limit) {\n-\t\tint code, i = 0;\n-\t\tstruct child_process child;\n-\t\tconst char *unpacker[5];\n-\t\tunpacker[i++] = \"unpack-objects\";\n+\t\targv[ac++] = \"unpack-objects\";\n \t\tif (quiet)\n-\t\t\tunpacker[i++] = \"-q\";\n+\t\t\targv[ac++] = \"-q\";\n \t\tif (fsck_objects)\n-\t\t\tunpacker[i++] = \"--strict\";\n-\t\tunpacker[i++] = hdr_arg;\n-\t\tunpacker[i++] = NULL;\n-\t\tmemset(&child, 0, sizeof(child));\n-\t\tchild.argv = unpacker;\n+\t\t\targv[ac++] = \"--strict\";\n+\t\targv[ac++] = hdr_arg;\n+\t\targv[ac++] = NULL;\n+\t\tchild.argv = argv;\n \t\tchild.no_stdout = 1;\n \t\tchild.err = err_fd;\n \t\tchild.git_cmd = 1;\n-\t\tcode = run_command(&child);\n-\t\tif (!code)\n-\t\t\treturn NULL;\n-\t\treturn \"unpack-objects abnormal exit\";\n+\t\tstatus = run_command(&child);\n+\t\tif (status)\n+\t\t\treturn \"unpack-objects abnormal exit\";\n \t} else {\n-\t\tconst char *keeper[7];\n-\t\tint s, status, i = 0;\n+\t\tint s;\n \t\tchar keep_arg[256];\n-\t\tstruct child_process ip;\n \n \t\ts = sprintf(keep_arg, \"--keep=receive-pack %\"PRIuMAX\" on \", (uintmax_t) getpid());\n \t\tif (gethostname(keep_arg + s, sizeof(keep_arg) - s))\n \t\t\tstrcpy(keep_arg + s, \"localhost\");\n \n-\t\tkeeper[i++] = \"index-pack\";\n-\t\tkeeper[i++] = \"--stdin\";\n+\t\targv[ac++] = \"index-pack\";\n+\t\targv[ac++] = \"--stdin\";\n \t\tif (fsck_objects)\n-\t\t\tkeeper[i++] = \"--strict\";\n+\t\t\targv[ac++] = \"--strict\";\n \t\tif (fix_thin)\n-\t\t\tkeeper[i++] = \"--fix-thin\";\n-\t\tkeeper[i++] = hdr_arg;\n-\t\tkeeper[i++] = keep_arg;\n-\t\tkeeper[i++] = NULL;\n-\t\tmemset(&ip, 0, sizeof(ip));\n-\t\tip.argv = keeper;\n-\t\tip.out = -1;\n-\t\tip.err = err_fd;\n-\t\tip.git_cmd = 1;\n-\t\tstatus = start_command(&ip);\n-\t\tif (status) {\n+\t\t\targv[ac++] = \"--fix-thin\";\n+\t\targv[ac++] = hdr_arg;\n+\t\targv[ac++] = keep_arg;\n+\t\targv[ac++] = NULL;\n+\t\tchild.argv = argv;\n+\t\tchild.out = -1;\n+\t\tchild.err = err_fd;\n+\t\tchild.git_cmd = 1;\n+\t\tstatus = start_command(&child);\n+\t\tif (status)\n \t\t\treturn \"index-pack fork failed\";\n-\t\t}\n-\t\tpack_lockfile = index_pack_lockfile(ip.out);\n-\t\tclose(ip.out);\n-\t\tstatus = finish_command(&ip);\n-\t\tif (!status) {\n-\t\t\treprepare_packed_git();\n-\t\t\treturn NULL;\n-\t\t}\n-\t\treturn \"index-pack abnormal exit\";\n+\t\tpack_lockfile = index_pack_lockfile(child.out);\n+\t\tclose(child.out);\n+\t\tstatus = finish_command(&child);\n+\t\tif (status)\n+\t\t\treturn \"index-pack abnormal exit\";\n+\t\treprepare_packed_git();\n \t}\n+\treturn NULL;\n }\n \n static const char *unpack_with_sideband(void)\n-- \n1.8.2.83.gc99314b\n"},{"id":"231042","messageId":"1385351754-9954-18-git-send-email-pclouds@gmail.com","threadId":"35400","inReplyTo":"1385351754-9954-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v3 17/28] Support pushing from a shallow clone","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-11-25T03:55:43Z","receivedAt":"2013-11-25T03:55:43Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Pushing from a shallow clone using today's send-pack and receive-pack\nmay work, if the transferred pack does not end up at any graft\npoints. If it does, recent receive-pack that does connectivity check\nwill reject the push. If receive-pack is old and does not have the\nconnectivity check, the upstream repo becomes corrupt.\n\nThe pack protocol is updated and send-pack now sends all shallow\ngrafts before it sends the commands, if the repo is shallow. This\nprotocol extension will break current receive-pack, which is intended,\nmostly to stop corrupting the upstream repo.\n\nChanges on the receiver are similar to what has been done in\nfetch-pack, i.e. filter out refs that require new shallow roots then\ngo along as usual.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n Documentation/technical/pack-protocol.txt | 4 +-\n builtin/receive-pack.c | 68 +++++++++++++++++++++++++++++-\n builtin/send-pack.c | 2 +-\n send-pack.c | 3 ++\n t/t5537-push-shallow.sh (new +x) | 70 +++++++++++++++++++++++++++++++\n 5 files changed, 143 insertions(+), 4 deletions(-)\n create mode 100755 t/t5537-push-shallow.sh\n\ndiff --git a/Documentation/technical/pack-protocol.txt b/Documentation/technical/pack-protocol.txt\nindex eb8edd1..c73b62f 100644\n--- a/Documentation/technical/pack-protocol.txt\n+++ b/Documentation/technical/pack-protocol.txt\n@@ -464,7 +464,9 @@ contain all the objects that the server will need to complete the new\n references.\n \n ----\n- update-request = command-list [pack-file]\n+ update-request = *shallow command-list [pack-file]\n+\n+ shallow = PKT-LINE(\"shallow\" SP obj-id)\n \n command-list = PKT-LINE(command NUL capability-list LF)\n \t\t *PKT-LINE(command LF)\ndiff --git a/builtin/receive-pack.c b/builtin/receive-pack.c\nindex 22e162d..254feff 100644\n--- a/builtin/receive-pack.c\n+++ b/builtin/receive-pack.c\n@@ -43,6 +43,9 @@ static int fix_thin = 1;\n static const char *head_name;\n static void *head_name_to_free;\n static int sent_capabilities;\n+static int shallow_push;\n+static const char* alternate_shallow_file;\n+static struct extra_have_objects shallow;\n \n static enum deny_action parse_deny_action(const char *var, const char *value)\n {\n@@ -189,6 +192,7 @@ struct command {\n \tconst char *error_string;\n \tunsigned int skip_update:1,\n \t\t did_not_exist:1;\n+\tint index;\n \tunsigned char old_sha1[20];\n \tunsigned char new_sha1[20];\n \tchar ref_name[FLEX_ARRAY]; /* more */\n@@ -687,7 +691,7 @@ static int iterate_receive_command_list(void *cb_data, unsigned char sha1[20])\n \tstruct command *cmd = *cmd_list;\n \n \twhile (cmd) {\n-\t\tif (!is_null_sha1(cmd->new_sha1)) {\n+\t\tif (!is_null_sha1(cmd->new_sha1) && !cmd->skip_update) {\n \t\t\thashcpy(sha1, cmd->new_sha1);\n \t\t\t*cmd_list = cmd->next;\n \t\t\treturn 0;\n@@ -712,8 +716,25 @@ static void reject_updates_to_hidden(struct command *commands)\n \t}\n }\n \n+static void filter_shallow_refs(struct command *commands,\n+\t\t\t\tconst int *ref_status, int nr_ref)\n+{\n+\tstruct command *cmd;\n+\tfor (cmd = commands; cmd; cmd = cmd->next)\n+\t\tif (!is_null_sha1(cmd->new_sha1) && ref_status[cmd->index]) {\n+\t\t\tcmd->error_string = \"shallow update not allowed\";\n+\t\t\tcmd->skip_update = 1;\n+\t\t}\n+\tif (*alternate_shallow_file) {\n+\t\tunlink(alternate_shallow_file);\n+\t\talternate_shallow_file = NULL;\n+\t}\n+}\n+\n static void execute_commands(struct command *commands, const char *unpacker_error)\n {\n+\tint *ref_status = NULL;\n+\tstruct extra_have_objects ref;\n \tstruct command *cmd;\n \tunsigned char sha1[20];\n \n@@ -723,6 +744,20 @@ static void execute_commands(struct command *commands, const char *unpacker_erro\n \t\treturn;\n \t}\n \n+\tmemset(&ref, 0, sizeof(ref));\n+\tif (shallow_push) {\n+\t\tint i;\n+\t\tfor (i = 0, cmd = commands; cmd; i++, cmd = cmd->next)\n+\t\t\tif (!is_null_sha1(cmd->new_sha1)) {\n+\t\t\t\tadd_extra_have(&ref, cmd->new_sha1);\n+\t\t\t\tcmd->index = i;\n+\t\t\t}\n+\n+\t\tref_status = xmalloc(sizeof(*ref_status) * ref.nr);\n+\t\tif (mark_new_shallow_refs(&ref, ref_status, NULL, &shallow))\n+\t\t\tfilter_shallow_refs(commands, ref_status, ref.nr);\n+\t}\n+\n \tcmd = commands;\n \tif (check_everything_connected(iterate_receive_command_list,\n \t\t\t\t 0, &cmd))\n@@ -752,6 +787,8 @@ static void execute_commands(struct command *commands, const char *unpacker_erro\n \n \t\tcmd->error_string = update(cmd);\n \t}\n+\tfree(ref.array);\n+\tfree(ref_status);\n }\n \n static struct command *read_head_info(void)\n@@ -768,6 +805,15 @@ static struct command *read_head_info(void)\n \t\tline = packet_read_line(0, &len);\n \t\tif (!line)\n \t\t\tbreak;\n+\n+\t\tif (len == 48 && !prefixcmp(line, \"shallow \")) {\n+\t\t\tif (get_sha1_hex(line + 8, old_sha1))\n+\t\t\t\tdie(\"protocol error: expected shallow sha, got '%s'\", line + 8);\n+\t\t\tadd_extra_have(&shallow, old_sha1);\n+\t\t\tshallow_push = 1;\n+\t\t\tcontinue;\n+\t\t}\n+\n \t\tif (len < 83 ||\n \t\t line[40] != ' ' ||\n \t\t line[81] != ' ' ||\n@@ -822,7 +868,7 @@ static const char *pack_lockfile;\n static const char *unpack(int err_fd)\n {\n \tstruct pack_header hdr;\n-\tconst char *argv[7];\n+\tconst char *argv[9];\n \tconst char *hdr_err;\n \tint status, ac = 0;\n \tchar hdr_arg[38];\n@@ -843,6 +889,24 @@ static const char *unpack(int err_fd)\n \t\t\t\"--pack_header=%\"PRIu32\",%\"PRIu32,\n \t\t\tntohl(hdr.hdr_version), ntohl(hdr.hdr_entries));\n \n+\tif (shallow_push) {\n+\t\tstruct extra_have_objects extra;\n+\t\tmemset(&extra, 0, sizeof(extra));\n+\t\tremove_reachable_shallow_points(&extra, &shallow);\n+\t\tif (extra.nr) {\n+\t\t\talternate_shallow_file = setup_temporary_shallow(&extra);\n+\t\t\tfree(shallow.array);\n+\t\t\tshallow = extra;\n+\t\t} else {\n+\t\t\talternate_shallow_file = NULL;\n+\t\t\tshallow_push = 0;\n+\t\t}\n+\t\tif (alternate_shallow_file && *alternate_shallow_file) {\n+\t\t\targv[ac++] = \"--shallow-file\";\n+\t\t\targv[ac++] = alternate_shallow_file;\n+\t\t}\n+\t}\n+\n \tmemset(&child, 0, sizeof(child));\n \tif (ntohl(hdr.hdr_entries) < unpack_limit) {\n \t\targv[ac++] = \"unpack-objects\";\ndiff --git a/builtin/send-pack.c b/builtin/send-pack.c\nindex bfa9253..5db1311 100644\n--- a/builtin/send-pack.c\n+++ b/builtin/send-pack.c\n@@ -207,7 +207,7 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)\n \t (send_all && args.send_mirror))\n \t\tusage(send_pack_usage);\n \n-\tif (is_repository_shallow())\n+\tif (is_repository_shallow() && args.stateless_rpc)\n \t\tdie(\"attempt to push from a shallow repository\");\n \n \tif (remote_name) {\ndiff --git a/send-pack.c b/send-pack.c\nindex fab62e3..8b5571c 100644\n--- a/send-pack.c\n+++ b/send-pack.c\n@@ -213,6 +213,9 @@ int send_pack(struct send_pack_args *args,\n \t\treturn 0;\n \t}\n \n+\tif (!args->dry_run)\n+\t\tadvertise_shallow_grafts(out);\n+\n \t/*\n \t * Finally, tell the other end!\n \t */\ndiff --git a/t/t5537-push-shallow.sh b/t/t5537-push-shallow.sh\nnew file mode 100755\nindex 0000000..650c31a\n--- /dev/null\n+++ b/t/t5537-push-shallow.sh\n@@ -0,0 +1,70 @@\n+#!/bin/sh\n+\n+test_description='push from/to a shallow clone'\n+\n+. ./test-lib.sh\n+\n+commit() {\n+\techo \"$1\" >tracked &&\n+\tgit add tracked &&\n+\tgit commit -m \"$1\"\n+}\n+\n+test_expect_success 'setup' '\n+\tgit config --global transfer.fsckObjects true &&\n+\tcommit 1 &&\n+\tcommit 2 &&\n+\tcommit 3 &&\n+\tcommit 4 &&\n+\t(\n+\tgit init full-abc &&\n+\tcd full-abc &&\n+\tcommit a &&\n+\tcommit b &&\n+\tcommit c\n+\t) &&\n+\tgit clone --no-local --depth=2 .git shallow &&\n+\tgit --git-dir=shallow/.git log --format=%s >actual &&\n+\tcat <expect &&\n+4\n+3\n+EOF\n+\ttest_cmp expect actual &&\n+\tgit clone --no-local --depth=2 full-abc/.git shallow2 &&\n+\tgit --git-dir=shallow2/.git log --format=%s >actual &&\n+\tcat <expect &&\n+c\n+b\n+EOF\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'push from shallow clone' '\n+\t(\n+\tcd shallow &&\n+\tcommit 5 &&\n+\tgit push ../.git +master:refs/remotes/shallow/master\n+\t) &&\n+\tgit log --format=%s shallow/master >actual &&\n+\tgit fsck &&\n+\tcat <expect &&\n+5\n+4\n+3\n+2\n+1\n+EOF\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'push from shallow clone, with grafted roots' '\n+\t(\n+\tcd shallow2 &&\n+\ttest_must_fail git push ../.git +master:refs/remotes/shallow2/master 2>err &&\n+\tgrep \"shallow2/master.*shallow update not allowed\" err\n+\t) &&\n+\ttest_must_fail git rev-parse shallow2/master &&\n+\tgit fsck\n+'\n+\n+test_done\n-- \n1.8.2.83.gc99314b\n"},{"id":"231044","messageId":"1385351754-9954-19-git-send-email-pclouds@gmail.com","threadId":"35400","inReplyTo":"1385351754-9954-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v3 18/28] New var GIT_SHALLOW_FILE to propagate --shallow-file to subprocesses","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-11-25T03:55:44Z","receivedAt":"2013-11-25T03:55:44Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"This may be needed when a hook is run after a new shallow pack is\nreceived, but .git/shallow is not settled yet. A temporary shallow\nfile to plug all loose ends should be used instead. GIT_SHALLOW_FILE\nis overriden by --shallow-file.\n\n--shallow-file does not work in this case because the hook may spawn\nmany git subprocesses and the launch commands do not have\n--shallow-file as it's a recent addition.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n cache.h | 1 +\n commit.h | 2 +-\n environment.c | 6 ++++++\n git.c | 2 +-\n shallow.c | 4 +++-\n 5 files changed, 12 insertions(+), 3 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex ce377e1..185d692 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -354,6 +354,7 @@ static inline enum object_type object_type(unsigned int mode)\n #define DB_ENVIRONMENT \"GIT_OBJECT_DIRECTORY\"\n #define INDEX_ENVIRONMENT \"GIT_INDEX_FILE\"\n #define GRAFT_ENVIRONMENT \"GIT_GRAFT_FILE\"\n+#define GIT_SHALLOW_FILE_ENVIRONMENT \"GIT_SHALLOW_FILE\"\n #define TEMPLATE_DIR_ENVIRONMENT \"GIT_TEMPLATE_DIR\"\n #define CONFIG_ENVIRONMENT \"GIT_CONFIG\"\n #define CONFIG_DATA_ENVIRONMENT \"GIT_CONFIG_PARAMETERS\"\ndiff --git a/commit.h b/commit.h\nindex 3af4699..0ff70fa 100644\n--- a/commit.h\n+++ b/commit.h\n@@ -202,7 +202,7 @@ extern int is_repository_shallow(void);\n extern struct commit_list *get_shallow_commits(struct object_array *heads,\n \t\tint depth, int shallow_flag, int not_shallow_flag);\n extern void check_shallow_file_for_update(void);\n-extern void set_alternate_shallow_file(const char *path);\n+extern void set_alternate_shallow_file(const char *path, int override);\n extern int write_shallow_commits(struct strbuf *out, int use_pack_protocol,\n \t\t\t\t const struct extra_have_objects *extra);\n extern void setup_alternate_shallow(struct lock_file *shallow_lock,\ndiff --git a/environment.c b/environment.c\nindex 0a15349..b73b39d 100644\n--- a/environment.c\n+++ b/environment.c\n@@ -10,6 +10,7 @@\n #include \"cache.h\"\n #include \"refs.h\"\n #include \"fmt-merge-msg.h\"\n+#include \"commit.h\"\n \n int trust_executable_bit = 1;\n int trust_ctime = 1;\n@@ -97,6 +98,7 @@ const char * const local_repo_env[] = {\n \tINDEX_ENVIRONMENT,\n \tNO_REPLACE_OBJECTS_ENVIRONMENT,\n \tGIT_PREFIX_ENVIRONMENT,\n+\tGIT_SHALLOW_FILE_ENVIRONMENT,\n \tNULL\n };\n \n@@ -124,6 +126,7 @@ static char *expand_namespace(const char *raw_namespace)\n static void setup_git_env(void)\n {\n \tconst char *gitfile;\n+\tconst char *shallow_file;\n \n \tgit_dir = getenv(GIT_DIR_ENVIRONMENT);\n \tif (!git_dir)\n@@ -147,6 +150,9 @@ static void setup_git_env(void)\n \t\tread_replace_refs = 0;\n \tnamespace = expand_namespace(getenv(GIT_NAMESPACE_ENVIRONMENT));\n \tnamespace_len = strlen(namespace);\n+\tshallow_file = getenv(GIT_SHALLOW_FILE_ENVIRONMENT);\n+\tif (shallow_file)\n+\t\tset_alternate_shallow_file(shallow_file, 0);\n }\n \n int is_bare_repository(void)\ndiff --git a/git.c b/git.c\nindex cb5208d..179c4f6 100644\n--- a/git.c\n+++ b/git.c\n@@ -162,7 +162,7 @@ static int handle_options(const char ***argv, int *argc, int *envchanged)\n \t\t} else if (!strcmp(cmd, \"--shallow-file\")) {\n \t\t\t(*argv)++;\n \t\t\t(*argc)--;\n-\t\t\tset_alternate_shallow_file((*argv)[0]);\n+\t\t\tset_alternate_shallow_file((*argv)[0], 1);\n \t\t\tif (envchanged)\n \t\t\t\t*envchanged = 1;\n \t\t} else if (!strcmp(cmd, \"-C\")) {\ndiff --git a/shallow.c b/shallow.c\nindex 1baf6b5..abb04db 100644\n--- a/shallow.c\n+++ b/shallow.c\n@@ -11,10 +11,12 @@ static int is_shallow = -1;\n static struct stat shallow_stat;\n static char *alternate_shallow_file;\n \n-void set_alternate_shallow_file(const char *path)\n+void set_alternate_shallow_file(const char *path, int override)\n {\n \tif (is_shallow != -1)\n \t\tdie(\"BUG: is_repository_shallow must not be called before set_alternate_shallow_file\");\n+\tif (alternate_shallow_file && !override)\n+\t\treturn;\n \tfree(alternate_shallow_file);\n \talternate_shallow_file = path ? xstrdup(path) : NULL;\n }\n-- \n1.8.2.83.gc99314b\n"},{"id":"231045","messageId":"1385351754-9954-20-git-send-email-pclouds@gmail.com","threadId":"35400","inReplyTo":"1385351754-9954-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v3 19/28] connected.c: add new variant that runs with --shallow-file","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-11-25T03:55:45Z","receivedAt":"2013-11-25T03:55:45Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n connected.c | 42 ++++++++++++++++++++++++++++++++++--------\n connected.h | 2 ++\n 2 files changed, 36 insertions(+), 8 deletions(-)\n\ndiff --git a/connected.c b/connected.c\nindex fae8d64..427389d 100644\n--- a/connected.c\n+++ b/connected.c\n@@ -19,17 +19,17 @@ int check_everything_connected(sha1_iterate_fn fn, int quiet, void *cb_data)\n *\n * Returns 0 if everything is connected, non-zero otherwise.\n */\n-int check_everything_connected_with_transport(sha1_iterate_fn fn,\n-\t\t\t\t\t int quiet,\n-\t\t\t\t\t void *cb_data,\n-\t\t\t\t\t struct transport *transport)\n+static int check_everything_connected_real(sha1_iterate_fn fn,\n+\t\t\t\t\t int quiet,\n+\t\t\t\t\t void *cb_data,\n+\t\t\t\t\t struct transport *transport,\n+\t\t\t\t\t const char *shallow_file)\n {\n \tstruct child_process rev_list;\n-\tconst char *argv[] = {\"rev-list\", \"--objects\",\n-\t\t\t \"--stdin\", \"--not\", \"--all\", NULL, NULL};\n+\tconst char *argv[9];\n \tchar commit[41];\n \tunsigned char sha1[20];\n-\tint err = 0;\n+\tint err = 0, ac = 0;\n \tstruct packed_git *new_pack = NULL;\n \n \tif (fn(cb_data, sha1))\n@@ -47,8 +47,18 @@ int check_everything_connected_with_transport(sha1_iterate_fn fn,\n \t\tstrbuf_release(&idx_file);\n \t}\n \n+\tif (shallow_file) {\n+\t\targv[ac++] = \"--shallow-file\";\n+\t\targv[ac++] = shallow_file;\n+\t}\n+\targv[ac++] = \"rev-list\";\n+\targv[ac++] = \"--objects\";\n+\targv[ac++] = \"--stdin\";\n+\targv[ac++] = \"--not\";\n+\targv[ac++] = \"--all\";\n \tif (quiet)\n-\t\targv[5] = \"--quiet\";\n+\t\targv[ac++] = \"--quiet\";\n+\targv[ac] = NULL;\n \n \tmemset(&rev_list, 0, sizeof(rev_list));\n \trev_list.argv = argv;\n@@ -92,3 +102,19 @@ int check_everything_connected_with_transport(sha1_iterate_fn fn,\n \tsigchain_pop(SIGPIPE);\n \treturn finish_command(&rev_list) || err;\n }\n+\n+int check_everything_connected_with_transport(sha1_iterate_fn fn,\n+\t\t\t\t\t int quiet,\n+\t\t\t\t\t void *cb_data,\n+\t\t\t\t\t struct transport *transport)\n+{\n+\treturn check_everything_connected_real(fn, quiet, cb_data,\n+\t\t\t\t\t transport, NULL);\n+}\n+\n+int check_shallow_connected(sha1_iterate_fn fn, int quiet, void *cb_data,\n+\t\t\t const char *shallow_file)\n+{\n+\treturn check_everything_connected_real(fn, quiet, cb_data,\n+\t\t\t\t\t NULL, shallow_file);\n+}\ndiff --git a/connected.h b/connected.h\nindex 0b060b7..071d408 100644\n--- a/connected.h\n+++ b/connected.h\n@@ -18,6 +18,8 @@ typedef int (*sha1_iterate_fn)(void *, unsigned char [20]);\n * Return 0 if Ok, non zero otherwise (i.e. some missing objects)\n */\n extern int check_everything_connected(sha1_iterate_fn, int quiet, void *cb_data);\n+extern int check_shallow_connected(sha1_iterate_fn, int quiet, void *cb_data,\n+\t\t\t\t const char *shallow_file);\n extern int check_everything_connected_with_transport(sha1_iterate_fn, int quiet,\n \t\t\t\t\t\t void *cb_data,\n \t\t\t\t\t\t struct transport *transport);\n-- \n1.8.2.83.gc99314b\n"},{"id":"231046","messageId":"1385351754-9954-21-git-send-email-pclouds@gmail.com","threadId":"35400","inReplyTo":"1385351754-9954-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v3 20/28] receive-pack: allow pushing with new shallow roots","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-11-25T03:55:46Z","receivedAt":"2013-11-25T03:55:46Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"This is more expensive than the current mode and potentially\ninvalidates caches like pack bitmaps. It's only enabled if\nreceive.shallowupdate is on.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n Documentation/config.txt | 4 ++\n builtin/receive-pack.c | 111 ++++++++++++++++++++++++++++++++++++++++++++---\n t/t5537-push-shallow.sh | 16 +++++++\n 3 files changed, 125 insertions(+), 6 deletions(-)\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex ab26963..1a0bd0d 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -2026,6 +2026,10 @@ receive.updateserverinfo::\n \tIf set to true, git-receive-pack will run git-update-server-info\n \tafter receiving data from git-push and updating refs.\n \n+receive.shallowupdate::\n+\tIf set to true, .git/shallow can be updated when new refs\n+\trequire new shallow roots. Otherwise those refs are rejected.\n+\n remote.pushdefault::\n \tThe remote to push to by default. Overrides\n \t`branch..remote` for all branches, and is overridden by\ndiff --git a/builtin/receive-pack.c b/builtin/receive-pack.c\nindex 254feff..366ecde 100644\n--- a/builtin/receive-pack.c\n+++ b/builtin/receive-pack.c\n@@ -43,7 +43,7 @@ static int fix_thin = 1;\n static const char *head_name;\n static void *head_name_to_free;\n static int sent_capabilities;\n-static int shallow_push;\n+static int shallow_push, shallow_update;\n static const char* alternate_shallow_file;\n static struct extra_have_objects shallow;\n \n@@ -124,6 +124,11 @@ static int receive_pack_config(const char *var, const char *value, void *cb)\n \t\treturn 0;\n \t}\n \n+\tif (strcmp(var, \"receive.shallowupdate\") == 0) {\n+\t\tshallow_update = git_config_bool(var, value);\n+\t\treturn 0;\n+\t}\n+\n \treturn git_default_config(var, value, cb);\n }\n \n@@ -191,6 +196,7 @@ struct command {\n \tstruct command *next;\n \tconst char *error_string;\n \tunsigned int skip_update:1,\n+\t\t checked_connectivity:1,\n \t\t did_not_exist:1;\n \tint index;\n \tunsigned char old_sha1[20];\n@@ -424,7 +430,44 @@ static void refuse_unconfigured_deny_delete_current(void)\n \t\trp_error(\"%s\", refuse_unconfigured_deny_delete_current_msg[i]);\n }\n \n-static const char *update(struct command *cmd)\n+static int command_singleton_iterator(void *cb_data, unsigned char sha1[20]);\n+static int update_ref_shallow(struct command *cmd, uint32_t **used_shallow)\n+{\n+\tstatic struct lock_file shallow_lock;\n+\tstruct extra_have_objects extra;\n+\tconst char *alt_file;\n+\tuint32_t mask = 1 << (cmd->index % 32);\n+\tint i;\n+\n+\tmemset(&extra, 0, sizeof(extra));\n+\tfor (i = 0; i < shallow.nr; i++)\n+\t\tif (used_shallow[i] &&\n+\t\t used_shallow[i][cmd->index / 32] & mask)\n+\t\t\tadd_extra_have(&extra, shallow.array[i]);\n+\n+\tsetup_alternate_shallow(&shallow_lock, &alt_file, &extra);\n+\tif (check_shallow_connected(command_singleton_iterator,\n+\t\t\t\t 0, cmd, alt_file)) {\n+\t\trollback_lock_file(&shallow_lock);\n+\t\tfree(extra.array);\n+\t\treturn -1;\n+\t}\n+\n+\tcommit_lock_file(&shallow_lock);\n+\n+\t/*\n+\t * Make sure setup_alternate_shallow() for the next ref does\n+\t * not lose these new roots..\n+\t */\n+\tfor (i = 0; i < extra.nr; i++)\n+\t\tregister_shallow(extra.array[i]);\n+\n+\tcmd->checked_connectivity = 1;\n+\tfree(extra.array);\n+\treturn 0;\n+}\n+\n+static const char *update(struct command *cmd, uint32_t **used_shallow)\n {\n \tconst char *name = cmd->ref_name;\n \tstruct strbuf namespaced_name_buf = STRBUF_INIT;\n@@ -532,6 +575,10 @@ static const char *update(struct command *cmd)\n \t\treturn NULL; /* good */\n \t}\n \telse {\n+\t\tif (shallow_push && !cmd->checked_connectivity &&\n+\t\t update_ref_shallow(cmd, used_shallow))\n+\t\t\treturn \"shallow error\";\n+\n \t\tlock = lock_any_ref_for_update(namespaced_name, old_sha1,\n \t\t\t\t\t 0, NULL);\n \t\tif (!lock) {\n@@ -678,6 +725,8 @@ static void set_connectivity_errors(struct command *commands)\n \n \tfor (cmd = commands; cmd; cmd = cmd->next) {\n \t\tstruct command *singleton = cmd;\n+\t\tif (shallow_push && !cmd->checked_connectivity)\n+\t\t\tcontinue;\n \t\tif (!check_everything_connected(command_singleton_iterator,\n \t\t\t\t\t\t0, &singleton))\n \t\t\tcontinue;\n@@ -691,7 +740,8 @@ static int iterate_receive_command_list(void *cb_data, unsigned char sha1[20])\n \tstruct command *cmd = *cmd_list;\n \n \twhile (cmd) {\n-\t\tif (!is_null_sha1(cmd->new_sha1) && !cmd->skip_update) {\n+\t\tif (!is_null_sha1(cmd->new_sha1) && !cmd->skip_update &&\n+\t\t (!shallow_push || cmd->checked_connectivity)) {\n \t\t\thashcpy(sha1, cmd->new_sha1);\n \t\t\t*cmd_list = cmd->next;\n \t\t\treturn 0;\n@@ -733,9 +783,10 @@ static void filter_shallow_refs(struct command *commands,\n \n static void execute_commands(struct command *commands, const char *unpacker_error)\n {\n-\tint *ref_status = NULL;\n+\tint *ref_status = NULL, checked_connectivity;\n \tstruct extra_have_objects ref;\n \tstruct command *cmd;\n+\tuint32_t **used_shallow = NULL;\n \tunsigned char sha1[20];\n \n \tif (unpacker_error) {\n@@ -754,8 +805,39 @@ static void execute_commands(struct command *commands, const char *unpacker_erro\n \t\t\t}\n \n \t\tref_status = xmalloc(sizeof(*ref_status) * ref.nr);\n-\t\tif (mark_new_shallow_refs(&ref, ref_status, NULL, &shallow))\n+\t\tif (shallow_update)\n+\t\t\tused_shallow = xmalloc(sizeof(*used_shallow) * shallow.nr);\n+\t\tif (!mark_new_shallow_refs(&ref, ref_status, used_shallow,\n+\t\t\t\t\t &shallow))\n+\t\t\tshallow_push = 0;\n+\t\telse if (!shallow_update) {\n \t\t\tfilter_shallow_refs(commands, ref_status, ref.nr);\n+\t\t\tshallow_push = 0;\n+\t\t}\n+\t}\n+\n+\t/*\n+\t * If shallow_push is still on at this point, we know some of\n+\t * the new refs may require extra shallow roots. But we don't\n+\t * know yet which ref will be accepted because hooks may\n+\t * reject some of them. So we can't add all new shallow roots\n+\t * in. Go through ref by ref and only add relevant shallow\n+\t * roots right before write_ref_sha1.\n+\t */\n+\tif (shallow_push) {\n+\t\tint i;\n+\t\t/* keep hooks happy */\n+\t\tsetenv(GIT_SHALLOW_FILE_ENVIRONMENT, alternate_shallow_file, 1);\n+\t\tfor (i = 0, cmd = commands; i < ref.nr; i++, cmd = cmd->next) {\n+\t\t\t/*\n+\t\t\t * marker for iterate_receive_command_list.\n+\t\t\t * All safe refs are run through the next\n+\t\t\t * check_everything_connected() the rest one\n+\t\t\t * by one in update()\n+\t\t\t */\n+\t\t\tcmd->checked_connectivity = !ref_status[i];\n+\t\t\tcmd->index = i;\n+\t\t}\n \t}\n \n \tcmd = commands;\n@@ -778,6 +860,7 @@ static void execute_commands(struct command *commands, const char *unpacker_erro\n \tfree(head_name_to_free);\n \thead_name = head_name_to_free = resolve_refdup(\"HEAD\", sha1, 0, NULL);\n \n+\tchecked_connectivity = 1;\n \tfor (cmd = commands; cmd; cmd = cmd->next) {\n \t\tif (cmd->error_string)\n \t\t\tcontinue;\n@@ -785,10 +868,26 @@ static void execute_commands(struct command *commands, const char *unpacker_erro\n \t\tif (cmd->skip_update)\n \t\t\tcontinue;\n \n-\t\tcmd->error_string = update(cmd);\n+\t\tcmd->error_string = update(cmd, used_shallow);\n+\t\tif (shallow_push && !cmd->error_string &&\n+\t\t !cmd->checked_connectivity) {\n+\t\t\terror(\"BUG: connectivity check has not been run on ref %s\",\n+\t\t\t cmd->ref_name);\n+\t\t\tchecked_connectivity = 0;\n+\t\t}\n+\t}\n+\n+\tif (shallow_push) {\n+\t\tif (!checked_connectivity)\n+\t\t\terror(\"BUG: run 'git fsck' for safety.\\n\"\n+\t\t\t \"If there are errors, try to remove \"\n+\t\t\t \"the reported refs above\");\n+\t\tif (*alternate_shallow_file)\n+\t\t\tunlink(alternate_shallow_file);\n \t}\n \tfree(ref.array);\n \tfree(ref_status);\n+\tfree(used_shallow);\n }\n \n static struct command *read_head_info(void)\ndiff --git a/t/t5537-push-shallow.sh b/t/t5537-push-shallow.sh\nindex 650c31a..0084a31 100755\n--- a/t/t5537-push-shallow.sh\n+++ b/t/t5537-push-shallow.sh\n@@ -67,4 +67,20 @@ test_expect_success 'push from shallow clone, with grafted roots' '\n \tgit fsck\n '\n \n+test_expect_success 'add new shallow root with receive.updateshallow on' '\n+\tgit config receive.shallowupdate true &&\n+\t(\n+\tcd shallow2 &&\n+\tGIT_TRACE=2 git push ../.git +master:refs/remotes/shallow2/master\n+\t) &&\n+\tgit log --format=%s shallow2/master >actual &&\n+\tgit fsck &&\n+\tcat <expect &&\n+c\n+b\n+EOF\n+\ttest_cmp expect actual &&\n+\tgit config receive.shallowupdate false\n+'\n+\n test_done\n-- \n1.8.2.83.gc99314b\n"},{"id":"231047","messageId":"1385351754-9954-22-git-send-email-pclouds@gmail.com","threadId":"35400","inReplyTo":"1385351754-9954-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v3 21/28] send-pack: support pushing to a shallow clone","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-11-25T03:55:47Z","receivedAt":"2013-11-25T03:55:47Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/send-pack.c | 5 ++++-\n t/t5537-push-shallow.sh | 38 ++++++++++++++++++++++++++++++++++++++\n transport.c | 5 ++---\n 3 files changed, 44 insertions(+), 4 deletions(-)\n\ndiff --git a/builtin/send-pack.c b/builtin/send-pack.c\nindex 5db1311..0031165 100644\n--- a/builtin/send-pack.c\n+++ b/builtin/send-pack.c\n@@ -100,6 +100,7 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)\n \tint fd[2];\n \tstruct child_process *conn;\n \tstruct extra_have_objects extra_have;\n+\tstruct extra_have_objects shallow;\n \tstruct ref *remote_refs, *local_refs;\n \tint ret;\n \tint helper_status = 0;\n@@ -232,8 +233,10 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)\n \t}\n \n \tmemset(&extra_have, 0, sizeof(extra_have));\n+\tmemset(&shallow, 0, sizeof(shallow));\n \n-\tget_remote_heads(fd[0], NULL, 0, &remote_refs, REF_NORMAL, &extra_have, NULL);\n+\tget_remote_heads(fd[0], NULL, 0, &remote_refs, REF_NORMAL,\n+\t\t\t &extra_have, &shallow);\n \n \ttransport_verify_remote_names(nr_refspecs, refspecs);\n \ndiff --git a/t/t5537-push-shallow.sh b/t/t5537-push-shallow.sh\nindex 0084a31..ccb41b6 100755\n--- a/t/t5537-push-shallow.sh\n+++ b/t/t5537-push-shallow.sh\n@@ -83,4 +83,42 @@ EOF\n \tgit config receive.shallowupdate false\n '\n \n+test_expect_success 'push from shallow to shallow' '\n+\t(\n+\tcd shallow &&\n+\tgit --git-dir=../shallow2/.git config receive.shallowupdate true &&\n+\tgit push ../shallow2/.git +master:refs/remotes/shallow/master &&\n+\tgit --git-dir=../shallow2/.git config receive.shallowupdate false\n+\t) &&\n+\t(\n+\tcd shallow2 &&\n+\tgit log --format=%s shallow/master >actual &&\n+\tgit fsck &&\n+\tcat <expect &&\n+5\n+4\n+3\n+EOF\n+\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'push from full to shallow' '\n+\t! git --git-dir=shallow2/.git cat-file blob `echo 1|git hash-object --stdin` &&\n+\tcommit 1 &&\n+\tgit push shallow2/.git +master:refs/remotes/top/master &&\n+\t(\n+\tcd shallow2 &&\n+\tgit log --format=%s top/master >actual &&\n+\tgit fsck &&\n+\tcat <expect &&\n+1\n+4\n+3\n+EOF\n+\ttest_cmp expect actual &&\n+\tgit cat-file blob `echo 1|git hash-object --stdin` >/dev/null\n+\t)\n+'\n+\n test_done\ndiff --git a/transport.c b/transport.c\nindex c0be6b1..6a3fe9b 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -818,11 +818,10 @@ static int git_transport_push(struct transport *transport, struct ref *remote_re\n \t\tstruct ref *tmp_refs;\n \t\tconnect_setup(transport, 1, 0);\n \n-\t\tget_remote_heads(data->fd[0], NULL, 0, &tmp_refs, REF_NORMAL, NULL, NULL);\n+\t\tget_remote_heads(data->fd[0], NULL, 0, &tmp_refs, REF_NORMAL,\n+\t\t\t\t NULL, &data->shallow);\n \t\tdata->got_remote_heads = 1;\n \t}\n-\tif (data->shallow.nr)\n-\t\tdie(\"pushing to a shallow repository is not supported\");\n \n \tmemset(&args, 0, sizeof(args));\n \targs.send_mirror = !!(flags & TRANSPORT_PUSH_MIRROR);\n-- \n1.8.2.83.gc99314b\n"},{"id":"231049","messageId":"1385351754-9954-23-git-send-email-pclouds@gmail.com","threadId":"35400","inReplyTo":"1385351754-9954-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v3 22/28] remote-curl: pass ref SHA-1 to fetch-pack as well","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-11-25T03:55:48Z","receivedAt":"2013-11-25T03:55:48Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/fetch-pack.c | 7 +++++++\n remote-curl.c | 3 ++-\n 2 files changed, 9 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/fetch-pack.c b/builtin/fetch-pack.c\nindex 927424b..aa6e596 100644\n--- a/builtin/fetch-pack.c\n+++ b/builtin/fetch-pack.c\n@@ -13,6 +13,13 @@ static void add_sought_entry_mem(struct ref ***sought, int *nr, int *alloc,\n \t\t\t\t const char *name, int namelen)\n {\n \tstruct ref *ref = xcalloc(1, sizeof(*ref) + namelen + 1);\n+\tunsigned char sha1[20];\n+\n+\tif (namelen > 41 && name[40] == ' ' && !get_sha1_hex(name, sha1)) {\n+\t\thashcpy(ref->old_sha1, sha1);\n+\t\tname += 41;\n+\t\tnamelen -= 41;\n+\t}\n \n \tmemcpy(ref->name, name, namelen);\n \tref->name[namelen] = '\\0';\ndiff --git a/remote-curl.c b/remote-curl.c\nindex 222210f..25d6730 100644\n--- a/remote-curl.c\n+++ b/remote-curl.c\n@@ -719,7 +719,8 @@ static int fetch_git(struct discovery *heads,\n \t\tstruct ref *ref = to_fetch[i];\n \t\tif (!ref->name || !*ref->name)\n \t\t\tdie(\"cannot fetch by sha1 over smart http\");\n-\t\tpacket_buf_write(&preamble, \"%s\\n\", ref->name);\n+\t\tpacket_buf_write(&preamble, \"%s %s\\n\",\n+\t\t\t\t sha1_to_hex(ref->old_sha1), ref->name);\n \t}\n \tpacket_buf_flush(&preamble);\n \n-- \n1.8.2.83.gc99314b\n"},{"id":"231050","messageId":"1385351754-9954-24-git-send-email-pclouds@gmail.com","threadId":"35400","inReplyTo":"1385351754-9954-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v3 23/28] Support fetch/clone over http","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-11-25T03:55:49Z","receivedAt":"2013-11-25T03:55:49Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n Documentation/gitremote-helpers.txt | 7 +++++++\n builtin/fetch-pack.c | 16 +++++++++++++---\n remote-curl.c | 30 ++++++++++++++++++++++++++++--\n t/t5536-fetch-shallow.sh | 27 +++++++++++++++++++++++++++\n transport-helper.c | 6 ++++++\n upload-pack.c | 2 --\n 6 files changed, 81 insertions(+), 7 deletions(-)\n\ndiff --git a/Documentation/gitremote-helpers.txt b/Documentation/gitremote-helpers.txt\nindex f1f4ca9..c2908db 100644\n--- a/Documentation/gitremote-helpers.txt\n+++ b/Documentation/gitremote-helpers.txt\n@@ -437,6 +437,13 @@ set by Git if the remote helper has the 'option' capability.\n 'option check-connectivity' \\{'true'|'false'\\}::\n \tRequest the helper to check connectivity of a clone.\n \n+'option cloning \\{'true'|'false'\\}::\n+\tNotify the helper this is a clone request (i.e. the current\n+\trepository is guaranteed empty).\n+\n+'option update-shallow \\{'true'|'false'\\}::\n+\tAllow to extend .git/shallow if the new refs require it.\n+\n SEE ALSO\n --------\n linkgit:git-remote[1]\ndiff --git a/builtin/fetch-pack.c b/builtin/fetch-pack.c\nindex aa6e596..64d1c0e 100644\n--- a/builtin/fetch-pack.c\n+++ b/builtin/fetch-pack.c\n@@ -46,6 +46,7 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)\n \tchar **pack_lockfile_ptr = NULL;\n \tstruct child_process *conn;\n \tstruct fetch_pack_args args;\n+\tstruct extra_have_objects shallow;\n \n \tpacket_trace_identity(\"fetch-pack\");\n \n@@ -113,6 +114,14 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)\n \t\t\targs.check_self_contained_and_connected = 1;\n \t\t\tcontinue;\n \t\t}\n+\t\tif (!strcmp(\"--cloning\", arg)) {\n+\t\t\targs.cloning = 1;\n+\t\t\tcontinue;\n+\t\t}\n+\t\tif (!strcmp(\"--update-shallow\", arg)) {\n+\t\t\targs.update_shallow = 1;\n+\t\t\tcontinue;\n+\t\t}\n \t\tusage(fetch_pack_usage);\n \t}\n \n@@ -157,10 +166,11 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)\n \t\t\t\t args.verbose ? CONNECT_VERBOSE : 0);\n \t}\n \n-\tget_remote_heads(fd[0], NULL, 0, &ref, 0, NULL, NULL);\n+\tmemset(&shallow, 0, sizeof(shallow));\n+\tget_remote_heads(fd[0], NULL, 0, &ref, 0, NULL, &shallow);\n \n-\tref = fetch_pack(&args, fd, conn, ref, dest,\n-\t\t\t sought, nr_sought, NULL, pack_lockfile_ptr);\n+\tref = fetch_pack(&args, fd, conn, ref, dest, sought, nr_sought,\n+\t\t\t &shallow, pack_lockfile_ptr);\n \tif (pack_lockfile) {\n \t\tprintf(\"lock %s\\n\", pack_lockfile);\n \t\tfflush(stdout);\ndiff --git a/remote-curl.c b/remote-curl.c\nindex 25d6730..0ae06b3 100644\n--- a/remote-curl.c\n+++ b/remote-curl.c\n@@ -20,6 +20,8 @@ struct options {\n \tunsigned long depth;\n \tunsigned progress : 1,\n \t\tcheck_self_contained_and_connected : 1,\n+\t\tcloning : 1,\n+\t\tupdate_shallow : 1,\n \t\tfollowtags : 1,\n \t\tdry_run : 1,\n \t\tthin : 1;\n@@ -88,6 +90,24 @@ static int set_option(const char *name, const char *value)\n \t\tstrbuf_release(&val);\n \t\treturn 0;\n \t}\n+\telse if (!strcmp(name, \"cloning\")) {\n+\t\tif (!strcmp(value, \"true\"))\n+\t\t\toptions.cloning = 1;\n+\t\telse if (!strcmp(value, \"false\"))\n+\t\t\toptions.cloning = 0;\n+\t\telse\n+\t\t\treturn -1;\n+\t\treturn 0;\n+\t}\n+\telse if (!strcmp(name, \"update-shallow\")) {\n+\t\tif (!strcmp(value, \"true\"))\n+\t\t\toptions.update_shallow = 1;\n+\t\telse if (!strcmp(value, \"false\"))\n+\t\t\toptions.update_shallow = 0;\n+\t\telse\n+\t\t\treturn -1;\n+\t\treturn 0;\n+\t}\n \telse {\n \t\treturn 1 /* unsupported */;\n \t}\n@@ -99,6 +119,7 @@ struct discovery {\n \tchar *buf;\n \tsize_t len;\n \tstruct ref *refs;\n+\tstruct extra_have_objects shallow;\n \tunsigned proto_git : 1;\n };\n static struct discovery *last_discovery;\n@@ -107,7 +128,7 @@ static struct ref *parse_git_refs(struct discovery *heads, int for_push)\n {\n \tstruct ref *list = NULL;\n \tget_remote_heads(-1, heads->buf, heads->len, &list,\n-\t\t\t for_push ? REF_NORMAL : 0, NULL, NULL);\n+\t\t\t for_push ? REF_NORMAL : 0, NULL, &heads->shallow);\n \treturn list;\n }\n \n@@ -168,6 +189,7 @@ static void free_discovery(struct discovery *d)\n \tif (d) {\n \t\tif (d == last_discovery)\n \t\t\tlast_discovery = NULL;\n+\t\tfree(d->shallow.array);\n \t\tfree(d->buf_alloc);\n \t\tfree_refs(d->refs);\n \t\tfree(d);\n@@ -688,7 +710,7 @@ static int fetch_git(struct discovery *heads,\n \tstruct strbuf preamble = STRBUF_INIT;\n \tchar *depth_arg = NULL;\n \tint argc = 0, i, err;\n-\tconst char *argv[16];\n+\tconst char *argv[17];\n \n \targv[argc++] = \"fetch-pack\";\n \targv[argc++] = \"--stateless-rpc\";\n@@ -704,6 +726,10 @@ static int fetch_git(struct discovery *heads,\n \t}\n \tif (options.check_self_contained_and_connected)\n \t\targv[argc++] = \"--check-self-contained-and-connected\";\n+\tif (options.cloning)\n+\t\targv[argc++] = \"--cloning\";\n+\tif (options.update_shallow)\n+\t\targv[argc++] = \"--update-shallow\";\n \tif (!options.progress)\n \t\targv[argc++] = \"--no-progress\";\n \tif (options.depth) {\ndiff --git a/t/t5536-fetch-shallow.sh b/t/t5536-fetch-shallow.sh\nindex 95b6313..bc96436 100755\n--- a/t/t5536-fetch-shallow.sh\n+++ b/t/t5536-fetch-shallow.sh\n@@ -163,4 +163,31 @@ EOF\n \t)\n '\n \n+if test -n \"$NO_CURL\" -o -z \"$GIT_TEST_HTTPD\"; then\n+\tsay 'skipping remaining tests, git built without http support'\n+\ttest_done\n+fi\n+\n+LIB_HTTPD_PORT=${LIB_HTTPD_PORT-'5536'}\n+. \"$TEST_DIRECTORY\"/lib-httpd.sh\n+start_httpd\n+\n+test_expect_success 'clone http repository' '\n+\tgit clone --bare --no-local shallow \"$HTTPD_DOCUMENT_ROOT_PATH/repo.git\" &&\n+\tgit clone $HTTPD_URL/smart/repo.git clone &&\n+\t(\n+\tcd clone &&\n+\tgit fsck &&\n+\tgit log --format=%s origin/master >actual &&\n+\tcat <expect &&\n+6\n+5\n+4\n+3\n+EOF\n+\ttest_cmp expect actual\n+\t)\n+'\n+\n+stop_httpd\n test_done\ndiff --git a/transport-helper.c b/transport-helper.c\nindex 673b7c2..e2b4203 100644\n--- a/transport-helper.c\n+++ b/transport-helper.c\n@@ -360,6 +360,12 @@ static int fetch_with_fetch(struct transport *transport,\n \t data->transport_options.check_self_contained_and_connected)\n \t\tset_helper_option(transport, \"check-connectivity\", \"true\");\n \n+\tif (transport->cloning)\n+\t\tset_helper_option(transport, \"cloning\", \"true\");\n+\n+\tif (data->transport_options.update_shallow)\n+\t\tset_helper_option(transport, \"update-shallow\", \"true\");\n+\n \tfor (i = 0; i < nr_heads; i++) {\n \t\tconst struct ref *posn = to_fetch[i];\n \t\tif (posn->status & REF_STATUS_UPTODATE)\ndiff --git a/upload-pack.c b/upload-pack.c\nindex 28269c7..2d02297 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -836,8 +836,6 @@ int main(int argc, char **argv)\n \n \tif (!enter_repo(dir, strict))\n \t\tdie(\"'%s' does not appear to be a git repository\", dir);\n-\tif (is_repository_shallow() && stateless_rpc)\n-\t\tdie(\"attempt to push into a shallow repository\");\n \n \tgit_config(upload_pack_config, NULL);\n \tupload_pack();\n-- \n1.8.2.83.gc99314b\n"},{"id":"231055","messageId":"1385351754-9954-25-git-send-email-pclouds@gmail.com","threadId":"35400","inReplyTo":"1385351754-9954-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v3 24/28] receive-pack: support pushing to a shallow clone via http","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-11-25T03:55:50Z","receivedAt":"2013-11-25T03:55:50Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/receive-pack.c | 3 ---\n t/t5537-push-shallow.sh | 35 +++++++++++++++++++++++++++++++++++\n 2 files changed, 35 insertions(+), 3 deletions(-)\n\ndiff --git a/builtin/receive-pack.c b/builtin/receive-pack.c\nindex 366ecde..ec681ba 100644\n--- a/builtin/receive-pack.c\n+++ b/builtin/receive-pack.c\n@@ -1157,9 +1157,6 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix)\n \tif (!enter_repo(dir, 0))\n \t\tdie(\"'%s' does not appear to be a git repository\", dir);\n \n-\tif (is_repository_shallow() && stateless_rpc)\n-\t\tdie(\"attempt to push into a shallow repository\");\n-\n \tgit_config(receive_pack_config, NULL);\n \n \tif (0 <= transfer_unpack_limit)\ndiff --git a/t/t5537-push-shallow.sh b/t/t5537-push-shallow.sh\nindex ccb41b6..d252a78 100755\n--- a/t/t5537-push-shallow.sh\n+++ b/t/t5537-push-shallow.sh\n@@ -16,6 +16,7 @@ test_expect_success 'setup' '\n \tcommit 2 &&\n \tcommit 3 &&\n \tcommit 4 &&\n+\tgit clone . full &&\n \t(\n \tgit init full-abc &&\n \tcd full-abc &&\n@@ -121,4 +122,38 @@ EOF\n \t)\n '\n \n+if test -n \"$NO_CURL\" -o -z \"$GIT_TEST_HTTPD\"; then\n+\tsay 'skipping remaining tests, git built without http support'\n+\ttest_done\n+fi\n+\n+LIB_HTTPD_PORT=${LIB_HTTPD_PORT-'5537'}\n+. \"$TEST_DIRECTORY\"/lib-httpd.sh\n+start_httpd\n+\n+test_expect_success 'push to shallow repo via http' '\n+\tgit clone --bare --no-local shallow \"$HTTPD_DOCUMENT_ROOT_PATH/repo.git\" &&\n+\t(\n+\tcd \"$HTTPD_DOCUMENT_ROOT_PATH/repo.git\" &&\n+\tgit config http.receivepack true\n+\t) &&\n+\t(\n+\tcd full &&\n+\tcommit 9 &&\n+\tgit push $HTTPD_URL/smart/repo.git +master:refs/remotes/top/master\n+\t) &&\n+\t(\n+\tcd \"$HTTPD_DOCUMENT_ROOT_PATH/repo.git\" &&\n+\tgit fsck &&\n+\tgit log --format=%s top/master >actual &&\n+\tcat <expect &&\n+9\n+4\n+3\n+EOF\n+\ttest_cmp expect actual\n+\t)\n+'\n+\n+stop_httpd\n test_done\n-- \n1.8.2.83.gc99314b\n"},{"id":"231052","messageId":"1385351754-9954-26-git-send-email-pclouds@gmail.com","threadId":"35400","inReplyTo":"1385351754-9954-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v3 25/28] send-pack: support pushing from a shallow clone via http","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-11-25T03:55:51Z","receivedAt":"2013-11-25T03:55:51Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/send-pack.c | 3 ---\n send-pack.c | 19 +++++++++++++++++--\n t/t5537-push-shallow.sh | 25 +++++++++++++++++++++++++\n 3 files changed, 42 insertions(+), 5 deletions(-)\n\ndiff --git a/builtin/send-pack.c b/builtin/send-pack.c\nindex 0031165..966b45e 100644\n--- a/builtin/send-pack.c\n+++ b/builtin/send-pack.c\n@@ -208,9 +208,6 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)\n \t (send_all && args.send_mirror))\n \t\tusage(send_pack_usage);\n \n-\tif (is_repository_shallow() && args.stateless_rpc)\n-\t\tdie(\"attempt to push from a shallow repository\");\n-\n \tif (remote_name) {\n \t\tremote = remote_get(remote_name);\n \t\tif (!remote_has_url(remote, dest)) {\ndiff --git a/send-pack.c b/send-pack.c\nindex 8b5571c..13167ea 100644\n--- a/send-pack.c\n+++ b/send-pack.c\n@@ -174,6 +174,21 @@ static int sideband_demux(int in, int out, void *data)\n \treturn ret;\n }\n \n+static int advertise_shallow_grafts_cb(const struct commit_graft *graft, void *cb)\n+{\n+\tstruct strbuf *sb = cb;\n+\tif (graft->nr_parent == -1)\n+\t\tpacket_buf_write(sb, \"shallow %s\\n\", sha1_to_hex(graft->sha1));\n+\treturn 0;\n+}\n+\n+void advertise_shallow_grafts_buf(struct strbuf *sb)\n+{\n+\tif (!is_repository_shallow())\n+\t\treturn;\n+\tfor_each_commit_graft(advertise_shallow_grafts_cb, sb);\n+}\n+\n int send_pack(struct send_pack_args *args,\n \t int fd[], struct child_process *conn,\n \t struct ref *remote_refs,\n@@ -214,7 +229,7 @@ int send_pack(struct send_pack_args *args,\n \t}\n \n \tif (!args->dry_run)\n-\t\tadvertise_shallow_grafts(out);\n+\t\tadvertise_shallow_grafts_buf(&req_buf);\n \n \t/*\n \t * Finally, tell the other end!\n@@ -275,7 +290,7 @@ int send_pack(struct send_pack_args *args,\n \t}\n \n \tif (args->stateless_rpc) {\n-\t\tif (!args->dry_run && cmds_sent) {\n+\t\tif (!args->dry_run && (cmds_sent || is_repository_shallow())) {\n \t\t\tpacket_buf_flush(&req_buf);\n \t\t\tsend_sideband(out, -1, req_buf.buf, req_buf.len, LARGE_PACKET_MAX);\n \t\t}\ndiff --git a/t/t5537-push-shallow.sh b/t/t5537-push-shallow.sh\nindex d252a78..f8200f7 100755\n--- a/t/t5537-push-shallow.sh\n+++ b/t/t5537-push-shallow.sh\n@@ -155,5 +155,30 @@ EOF\n \t)\n '\n \n+test_expect_success 'push from shallow repo via http' '\n+\tmv \"$HTTPD_DOCUMENT_ROOT_PATH/repo.git\" shallow-upstream.git &&\n+\tgit clone --bare --no-local full \"$HTTPD_DOCUMENT_ROOT_PATH/repo.git\" &&\n+\t(\n+\tcd \"$HTTPD_DOCUMENT_ROOT_PATH/repo.git\" &&\n+\tgit config http.receivepack true\n+\t) &&\n+\tcommit 10 &&\n+\tgit push $HTTPD_URL/smart/repo.git +master:refs/remotes/top/master &&\n+\t(\n+\tcd \"$HTTPD_DOCUMENT_ROOT_PATH/repo.git\" &&\n+\tgit fsck &&\n+\tgit log --format=%s top/master >actual &&\n+\tcat <expect &&\n+10\n+1\n+4\n+3\n+2\n+1\n+EOF\n+\ttest_cmp expect actual\n+\t)\n+'\n+\n stop_httpd\n test_done\n-- \n1.8.2.83.gc99314b\n"},{"id":"231053","messageId":"1385351754-9954-27-git-send-email-pclouds@gmail.com","threadId":"35400","inReplyTo":"1385351754-9954-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v3 26/28] git-clone.txt: remove shallow clone limitations","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-11-25T03:55:52Z","receivedAt":"2013-11-25T03:55:52Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Now that git supports push/pull from/to a shallow clone, these\nlimitations are not true anymore.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n Documentation/git-clone.txt | 7 +------\n 1 file changed, 1 insertion(+), 6 deletions(-)\n\ndiff --git a/Documentation/git-clone.txt b/Documentation/git-clone.txt\nindex 450f158..4987857 100644\n--- a/Documentation/git-clone.txt\n+++ b/Documentation/git-clone.txt\n@@ -181,12 +181,7 @@ objects from the source repository into a pack in the cloned repository.\n \n --depth ::\n \tCreate a 'shallow' clone with a history truncated to the\n-\tspecified number of revisions. A shallow repository has a\n-\tnumber of limitations (you cannot clone or fetch from\n-\tit, nor push from nor into it), but is adequate if you\n-\tare only interested in the recent history of a large project\n-\twith a long history, and would want to send in fixes\n-\tas patches.\n+\tspecified number of revisions.\n \n --[no-]single-branch::\n \tClone only the history leading to the tip of a single branch,\n-- \n1.8.2.83.gc99314b\n"},{"id":"231051","messageId":"1385351754-9954-28-git-send-email-pclouds@gmail.com","threadId":"35400","inReplyTo":"1385351754-9954-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v3 27/28] clone: use git protocol for cloning shallow repo locally","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-11-25T03:55:53Z","receivedAt":"2013-11-25T03:55:53Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"clone_local() does not handle $SRC/shallow. It could be made so, but\nit's simpler to use fetch-pack/upload-pack instead.\n\nThis used by be caught by the check in upload-pack, which is triggered\nby transport_get_remote_refs(), even in local clone case. The check is\nnow gone and check_everything_connected() should catch the result\nincomplete repo. But check_everything_connected() will soon be skipped\nin local clone case, opening a door to corrupt repo. This patch should\nclose that door.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/clone.c | 11 +++++++++--\n t/t5601-clone.sh | 7 +++++++\n 2 files changed, 16 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex 0b182ce..71ee68b 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -797,8 +797,15 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \telse\n \t\trepo = repo_name;\n \tis_local = option_local != 0 && path && !is_bundle;\n-\tif (is_local && option_depth)\n-\t\twarning(_(\"--depth is ignored in local clones; use file:// instead.\"));\n+\tif (is_local) {\n+\t\tif (option_depth)\n+\t\t\twarning(_(\"--depth is ignored in local clones; use file:// instead.\"));\n+\t\tif (!access(mkpath(\"%s/shallow\", path), F_OK)) {\n+\t\t\tif (option_local > 0)\n+\t\t\t\twarning(_(\"source repository is shallow, ignoring --local\"));\n+\t\t\tis_local = 0;\n+\t\t}\n+\t}\n \tif (option_local > 0 && !is_local)\n \t\twarning(_(\"--local is ignored\"));\n \ndiff --git a/t/t5601-clone.sh b/t/t5601-clone.sh\nindex 1d1c875..c226cff 100755\n--- a/t/t5601-clone.sh\n+++ b/t/t5601-clone.sh\n@@ -340,4 +340,11 @@ test_expect_success 'clone from a repository with two identical branches' '\n \n '\n \n+test_expect_success 'shallow clone locally' '\n+\tgit clone --depth=1 --no-local src ssrrcc &&\n+\tgit clone ssrrcc ddsstt &&\n+\ttest_cmp ssrrcc/.git/shallow ddsstt/.git/shallow &&\n+\t( cd ddsstt && git fsck )\n+'\n+\n test_done\n-- \n1.8.2.83.gc99314b\n"},{"id":"231054","messageId":"1385351754-9954-29-git-send-email-pclouds@gmail.com","threadId":"35400","inReplyTo":"1385351754-9954-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v3 28/28] prune: clean .git/shallow after pruning objects","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-11-25T03:55:54Z","receivedAt":"2013-11-25T03:55:54Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"This patch teaches \"prune\" to remove shallow roots that are no longer\nreachable from any refs (e.g. when the relevant refs are removed).\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/gc.c | 1 +\n builtin/prune.c | 4 ++++\n commit.h | 1 +\n shallow.c | 43 +++++++++++++++++++++++++++++++++++++++++--\n 4 files changed, 47 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex c14190f..cec8ecd 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -16,6 +16,7 @@\n #include \"run-command.h\"\n #include \"sigchain.h\"\n #include \"argv-array.h\"\n+#include \"commit.h\"\n \n #define FAILED_RUN \"failed to run %s\"\n \ndiff --git a/builtin/prune.c b/builtin/prune.c\nindex 6366917..023aea8 100644\n--- a/builtin/prune.c\n+++ b/builtin/prune.c\n@@ -170,5 +170,9 @@ int cmd_prune(int argc, const char **argv, const char *prefix)\n \ts = mkpathdup(\"%s/pack\", get_object_directory());\n \tremove_temporary_files(s);\n \tfree(s);\n+\n+\tif (!show_only && is_repository_shallow())\n+\t\tprune_shallow();\n+\n \treturn 0;\n }\ndiff --git a/commit.h b/commit.h\nindex 0ff70fa..ff93a59 100644\n--- a/commit.h\n+++ b/commit.h\n@@ -215,6 +215,7 @@ extern void remove_reachable_shallow_points(struct extra_have_objects *out,\n extern int mark_new_shallow_refs(const struct extra_have_objects *ref,\n \t\t\t\t int *ref_status, uint32_t **used,\n \t\t\t\t const struct extra_have_objects *shallow);\n+extern void prune_shallow(void);\n \n int is_descendant_of(struct commit *, struct commit_list *);\n int in_merge_bases(struct commit *, struct commit *);\ndiff --git a/shallow.c b/shallow.c\nindex abb04db..0eda40f 100644\n--- a/shallow.c\n+++ b/shallow.c\n@@ -157,6 +157,7 @@ struct write_shallow_data {\n \tstruct strbuf *out;\n \tint use_pack_protocol;\n \tint count;\n+\tint seen_only;\n };\n \n static int write_one_shallow(const struct commit_graft *graft, void *cb_data)\n@@ -165,6 +166,11 @@ static int write_one_shallow(const struct commit_graft *graft, void *cb_data)\n \tconst char *hex = sha1_to_hex(graft->sha1);\n \tif (graft->nr_parent != -1)\n \t\treturn 0;\n+\tif (data->seen_only) {\n+\t\tstruct commit *c = lookup_commit(graft->sha1);\n+\t\tif (!c || !(c->object.flags & SEEN))\n+\t\t\treturn 0;\n+\t}\n \tdata->count++;\n \tif (data->use_pack_protocol)\n \t\tpacket_buf_write(data->out, \"shallow %s\", hex);\n@@ -175,14 +181,16 @@ static int write_one_shallow(const struct commit_graft *graft, void *cb_data)\n \treturn 0;\n }\n \n-int write_shallow_commits(struct strbuf *out, int use_pack_protocol,\n-\t\t\t const struct extra_have_objects *extra)\n+static int write_shallow_commits_1(struct strbuf *out, int use_pack_protocol,\n+\t\t\t\t const struct extra_have_objects *extra,\n+\t\t\t\t int seen_only)\n {\n \tstruct write_shallow_data data;\n \tint i;\n \tdata.out = out;\n \tdata.use_pack_protocol = use_pack_protocol;\n \tdata.count = 0;\n+\tdata.seen_only = seen_only;\n \tfor_each_commit_graft(write_one_shallow, &data);\n \tif (!extra)\n \t\treturn data.count;\n@@ -194,6 +202,12 @@ int write_shallow_commits(struct strbuf *out, int use_pack_protocol,\n \treturn data.count;\n }\n \n+int write_shallow_commits(struct strbuf *out, int use_pack_protocol,\n+\t\t\t const struct extra_have_objects *extra)\n+{\n+\treturn write_shallow_commits_1(out, use_pack_protocol, extra, 0);\n+}\n+\n char *setup_temporary_shallow(const struct extra_have_objects *extra)\n {\n \tstruct strbuf sb = STRBUF_INIT;\n@@ -515,3 +529,28 @@ int mark_new_shallow_refs(const struct extra_have_objects *ref,\n \n \treturn ret;\n }\n+\n+/*\n+ * mark_reachable_objects() should have been run prior to this and all\n+ * reachable commits marked as \"SEEN\".\n+ */\n+void prune_shallow(void)\n+{\n+\tstatic struct lock_file shallow_lock;\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tint fd;\n+\n+\tcheck_shallow_file_for_update();\n+\tfd = hold_lock_file_for_update(&shallow_lock, git_path(\"shallow\"),\n+\t\t\t\t LOCK_DIE_ON_ERROR);\n+\tif (write_shallow_commits_1(&sb, 0, NULL, 1)) {\n+\t\tif (write_in_full(fd, sb.buf, sb.len) != sb.len)\n+\t\t\tdie_errno(\"failed to write to %s\",\n+\t\t\t\t shallow_lock.filename);\n+\t\tcommit_lock_file(&shallow_lock);\n+\t} else {\n+\t\tunlink(git_path(\"shallow\"));\n+\t\trollback_lock_file(&shallow_lock);\n+\t}\n+\tstrbuf_release(&sb);\n+}\n-- \n1.8.2.83.gc99314b\n"},{"id":"231090","messageId":"xmqqr4a4xn1a.fsf@gitster.dls.corp.google.com","threadId":"35400","inReplyTo":"1385351754-9954-5-git-send-email-pclouds@gmail.com","subject":"Re: [PATCH v3 04/28] update-server-info: do not publish shallow clones","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-11-25T20:08:01Z","receivedAt":"2013-11-25T20:08:01Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Nguyễn Thái Ngọc Duy writes:\n\n> Dumb commit walker does not care about .git/shallow and until someone\n> steps up to make it happen, let's not publish shallow clones using\n> dumb protocols.\n>\n> Signed-off-by: Nguyễn Thái Ngọc Duy \n> ---\n> server-info.c | 9 +++++++++\n> 1 file changed, 9 insertions(+)\n\nI doubt that pros-and-cons is in this patch's favor. Without this\npatch, if a fetch requests commits just on the surface in this\nshallow repository, the walker would happily get the necessary\nobjects just fine. If the request has to dig deeper and cross the\nshallow boundary, the walker will get a failure and error out.\n\nWith this patch, both will error out. So overall, the patch did not\nmake anything safer (e.g. preventing from introducing new corruption\non the recipient's end) while breaking a case where it worked just\nfine, no?\n\nOr am I missing something? Not that dumb walkers would matter very\nmuch these days, ...\n\n> diff --git a/server-info.c b/server-info.c\n> index 9ec744e..a8df6a5 100644\n> --- a/server-info.c\n> +++ b/server-info.c\n> @@ -33,6 +33,10 @@ static int update_info_refs(int force)\n> \tstrcpy(path1 + len, \"+\");\n> \n> \tsafe_create_leading_directories(path0);\n> +\tif (is_repository_shallow()) {\n> +\t\tunlink(path0);\n> +\t\treturn error(\"info/refs not updated for shallow clone\");\n> +\t}\n> \tinfo_ref_fp = fopen(path1, \"w\");\n> \tif (!info_ref_fp)\n> \t\treturn error(\"unable to update %s\", path1);\n> @@ -217,6 +221,11 @@ static int update_info_packs(int force)\n> \tstrcpy(name, infofile);\n> \tstrcpy(name + namelen, \"+\");\n> \n> +\tif (is_repository_shallow()) {\n> +\t\tunlink(infofile);\n> +\t\treturn error(\"info/packs not updated for shallow clone\");\n> +\t}\n> +\n> \tinit_pack_info(infofile, force);\n> \n> \tsafe_create_leading_directories(name);\n"},{"id":"231112","messageId":"xmqqtxf0w440.fsf@gitster.dls.corp.google.com","threadId":"35400","inReplyTo":"1385351754-9954-7-git-send-email-pclouds@gmail.com","subject":"Re: [PATCH v3 06/28] connect.c: teach get_remote_heads to parse \"shallow\" lines","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-11-25T21:42:07Z","receivedAt":"2013-11-25T21:42:07Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Nguyễn Thái Ngọc Duy writes:\n\n> No callers pass a non-empty pointer as shallow_points at this\n> stage. As a result, all clients still refuse to talk to shallow\n> repository on the other end.\n>\n> Signed-off-by: Nguyễn Thái Ngọc Duy \n> ---\n> builtin/fetch-pack.c | 2 +-\n> builtin/send-pack.c | 2 +-\n> connect.c | 12 +++++++++++-\n> remote-curl.c | 2 +-\n> remote.h | 3 ++-\n> transport.c | 7 ++++---\n> 6 files changed, 20 insertions(+), 8 deletions(-)\n>\n> diff --git a/builtin/fetch-pack.c b/builtin/fetch-pack.c\n> index c8e8582..c1d918f 100644\n> --- a/builtin/fetch-pack.c\n> +++ b/builtin/fetch-pack.c\n> @@ -150,7 +150,7 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)\n> \t\t\t\t args.verbose ? CONNECT_VERBOSE : 0);\n> \t}\n> \n> -\tget_remote_heads(fd[0], NULL, 0, &ref, 0, NULL);\n> +\tget_remote_heads(fd[0], NULL, 0, &ref, 0, NULL, NULL);\n> \n> \tref = fetch_pack(&args, fd, conn, ref, dest,\n> \t\t\t sought, nr_sought, pack_lockfile_ptr);\n> diff --git a/builtin/send-pack.c b/builtin/send-pack.c\n> index 51121f2..bfa9253 100644\n> --- a/builtin/send-pack.c\n> +++ b/builtin/send-pack.c\n> @@ -233,7 +233,7 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)\n> \n> \tmemset(&extra_have, 0, sizeof(extra_have));\n> \n> -\tget_remote_heads(fd[0], NULL, 0, &remote_refs, REF_NORMAL, &extra_have);\n> +\tget_remote_heads(fd[0], NULL, 0, &remote_refs, REF_NORMAL, &extra_have, NULL);\n> \n> \ttransport_verify_remote_names(nr_refspecs, refspecs);\n> \n> diff --git a/connect.c b/connect.c\n> index 06e88b0..d0602b0 100644\n> --- a/connect.c\n> +++ b/connect.c\n> @@ -122,7 +122,8 @@ static void annotate_refs_with_symref_info(struct ref *ref)\n> */\n> struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,\n> \t\t\t struct ref **list, unsigned int flags,\n> -\t\t\t struct extra_have_objects *extra_have)\n> +\t\t\t struct extra_have_objects *extra_have,\n> +\t\t\t struct extra_have_objects *shallow_points)\n\nThe _shape_ of the information you would want to keep track of for\nthe shallow cut-off points may exactly be the same as that is for\nextra-have endspoints, but it still feels wrong to throw these\nshallow cut-off points into a structure called \"extra have\". After\nall, these are objects the other end does not have, which is the\ndirect opposite of extra-have.\n\nPerhaps a preparatory patch needs to rename the structure type to\nobject_name_list or something. And then we can make the variable\nnames, not typenames, responsible for signalling what they mean,\ni.e.\n\n\tget_remote_heads(...\n \tstruct list_of_objects *extra_have,\n struct list_of_objects *shallow_points);\n\nwhen we introduce the new parameter.\n"},{"id":"231113","messageId":"xmqqpppow3kv.fsf@gitster.dls.corp.google.com","threadId":"35400","inReplyTo":"1385351754-9954-8-git-send-email-pclouds@gmail.com","subject":"Re: [PATCH v3 07/28] shallow.c: add remove_reachable_shallow_points()","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-11-25T21:53:36Z","receivedAt":"2013-11-25T21:53:36Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Nguyễn Thái Ngọc Duy writes:\n\n> When we receive a pack and the shallow points from another repository,\n> we may want to add more shallow points to current repo to make sure no\n> commits point to nowhere. However we do not want to add unnecessary\n> shallow points and cut our history short because the other end is a\n> shallow version of this repo. The output shallow points may or may not\n> be added to .git/shallow, depending on whether they are actually\n> reachable in the new pack.\n>\n> This function filters such shallow points out, leaving ones that might\n> potentially be added. A simple has_sha1_file won't do because we may\n> have incomplete object islands (i.e. not connected to any refs) and\n> the shallow points are on these islands. In that case we want to keep\n> the shallow points as candidates until we figure out if the new pack\n> connects to such object islands.\n>\n> Typical cases that use remove_reachable_shallow_points() are:\n>\n> - fetch from a repo that has longer history: in this case all remote\n> shallow roots do not exist in the client repo, this function will\n> be cheap as it just does a few lookup_commit_graft and\n> has_sha1_file.\n\nIt is unclear why. If you fetched from a repository more complete\nthan you, you may deepen your history which may allow you to unplug\nthe shallow points you originally had, and remote \"shallow root\" (by\nthe way, lets find a single good phrase to represent this thing and\nstick to it) may want to replace them, no?\n\n> - fetch from a repo that has exactly the same shallow root set\n> (e.g. a clone from a shallow repo): this case may trigger\n> in_merge_bases_many all the way to roots. An exception is made to\n> avoid such costly path with a faith that .git/shallow does not\n> usually points to unreachable commit islands.\n\n... and when the faith is broken, you will end up with a broken\nrepository???\n\n> - push from a shallow repo that has shorter history than the\n> server's: in_merge_bases_many() is unavoidable, so the longer\n> history the client has the higher the server cost is. The cost may\n> be reduced with the help of pack bitmaps, or commit cache, though.\n>\n> - push from a shallow clone that has exactly the same shallow root\n> set: the same as the second fetch case above, .i.e. cheap by\n> exception.\n>\n> The function must be called before the new pack is installed, or we\n> won't know which objects are ours, which theirs.\n>\n> Signed-off-by: Nguyễn Thái Ngọc Duy \n> ---\n> commit.h | 3 +++\n> connect.c | 2 +-\n> remote.h | 1 +\n> shallow.c | 45 +++++++++++++++++++++++++++++++++++++++++++++\n> 4 files changed, 50 insertions(+), 1 deletion(-)\n>\n> diff --git a/commit.h b/commit.h\n> index a879526..98044e6 100644\n> --- a/commit.h\n> +++ b/commit.h\n> @@ -193,6 +193,7 @@ extern struct commit_list *get_octopus_merge_bases(struct commit_list *in);\n> /* largest positive number a signed 32-bit integer can contain */\n> #define INFINITE_DEPTH 0x7fffffff\n> \n> +struct extra_have_objects;\n> extern int register_shallow(const unsigned char *sha1);\n> extern int unregister_shallow(const unsigned char *sha1);\n> extern int for_each_commit_graft(each_commit_graft_fn, void *);\n> @@ -206,6 +207,8 @@ extern void setup_alternate_shallow(struct lock_file *shallow_lock,\n> \t\t\t\t const char **alternate_shallow_file);\n> extern char *setup_temporary_shallow(void);\n> extern void advertise_shallow_grafts(int);\n> +extern void remove_reachable_shallow_points(struct extra_have_objects *out,\n> +\t\t\t\t\t const struct extra_have_objects *in);\n> \n> int is_descendant_of(struct commit *, struct commit_list *);\n> int in_merge_bases(struct commit *, struct commit *);\n> diff --git a/connect.c b/connect.c\n> index d0602b0..80e4360 100644\n> --- a/connect.c\n> +++ b/connect.c\n> @@ -45,7 +45,7 @@ int check_ref_type(const struct ref *ref, int flags)\n> \treturn check_ref(ref->name, strlen(ref->name), flags);\n> }\n> \n> -static void add_extra_have(struct extra_have_objects *extra, unsigned char *sha1)\n> +void add_extra_have(struct extra_have_objects *extra, unsigned char *sha1)\n> {\n> \tALLOC_GROW(extra->array, extra->nr + 1, extra->alloc);\n> \thashcpy(&(extra->array[extra->nr][0]), sha1);\n> diff --git a/remote.h b/remote.h\n> index 773faa9..ff604ff 100644\n> --- a/remote.h\n> +++ b/remote.h\n> @@ -145,6 +145,7 @@ extern struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,\n> \t\t\t\t struct ref **list, unsigned int flags,\n> \t\t\t\t struct extra_have_objects *have,\n> \t\t\t\t struct extra_have_objects *shallow);\n> +extern void add_extra_have(struct extra_have_objects *extra, unsigned char *sha1);\n> \n> int resolve_remote_symref(struct ref *ref, struct ref *list);\n> int ref_newer(const unsigned char *new_sha1, const unsigned char *old_sha1);\n> diff --git a/shallow.c b/shallow.c\n> index 9552512..a974d2d 100644\n> --- a/shallow.c\n> +++ b/shallow.c\n> @@ -2,6 +2,8 @@\n> #include \"commit.h\"\n> #include \"tag.h\"\n> #include \"pkt-line.h\"\n> +#include \"remote.h\"\n> +#include \"refs.h\"\n> \n> static int is_shallow = -1;\n> static struct stat shallow_stat;\n> @@ -235,3 +237,46 @@ void advertise_shallow_grafts(int fd)\n> \t\treturn;\n> \tfor_each_commit_graft(advertise_shallow_grafts_cb, &fd);\n> }\n> +\n> +struct commit_array {\n> +\tstruct commit **commits;\n> +\tint nr, alloc;\n> +};\n> +\n> +static int add_ref(const char *refname,\n> +\t\t const unsigned char *sha1, int flags, void *cb_data)\n> +{\n> +\tstruct commit_array *ca = cb_data;\n> +\tALLOC_GROW(ca->commits, ca->nr + 1, ca->alloc);\n> +\tca->commits[ca->nr++] = lookup_commit(sha1);\n> +\treturn 0;\n> +}\n\nCan't a ref point at a non-commit-ish? Is the code prepared to deal\nwith such an entry (possibly a NULL pointer) in the commit_array\nstruct?\n\n> +void remove_reachable_shallow_points(struct extra_have_objects *out,\n> +\t\t\t\t const struct extra_have_objects *in)\n> +{\n> +\tstruct commit_array ca;\n> +\tint i;\n> +\n> +\tmemset(&ca, 0, sizeof(ca));\n> +\thead_ref(add_ref, &ca);\n> +\tfor_each_ref(add_ref, &ca);\n> +\tfor (i = 0; i < in->nr; i++) {\n> +\t\tstruct commit_graft *graft = lookup_commit_graft(in->array[i]);\n> +\t\t/*\n> +\t\t * For a clone from a shallow upstream, the clone has\n> +\t\t * the same shallow roots as upstream and it will\n> +\t\t * trigger in_merge_bases_many() all the way to roots.\n> +\t\t * Avoid that costly path and assume .git/shallow is\n> +\t\t * good most of the time.\n> +\t\t */\n> +\t\tif (graft && graft->nr_parent < 0)\n> +\t\t\tcontinue;\n> +\t\tif (has_sha1_file(in->array[i]) &&\n> +\t\t in_merge_bases_many(lookup_commit(in->array[i]),\n> +\t\t\t\t\tca.nr, ca.commits))\n> +\t\t\tcontinue;\n> +\t\tadd_extra_have(out, in->array[i]);\n> +\t}\n> +\tfree(ca.commits);\n> +}\n"},{"id":"231114","messageId":"xmqqli0cw2b9.fsf@gitster.dls.corp.google.com","threadId":"35400","inReplyTo":"1385351754-9954-9-git-send-email-pclouds@gmail.com","subject":"Re: [PATCH v3 08/28] shallow.c: add mark_new_shallow_refs()","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-11-25T22:20:58Z","receivedAt":"2013-11-25T22:20:58Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Nguyễn Thái Ngọc Duy writes:\n\n> When we receive a pack and the shallow points from another repository,\n> we may need to add more shallow points to current repo to make sure no\n> commits point to nowhere. But usually we don't want to do so because\n> (in future) new shallow points invalidate pack bitmaps and we need to\n> rebuild them again, which is not cheap.\n>\n> So the default way is we allow ref updates that do not introduce new\n> shallow points and mark the others. If the user is fine with new\n> shallow point addition, we accept the marked refs.\n>\n> But even so we do not blindly accept all shallow points provided. Some\n> of them might not point to any commits in the new pack. Some might\n> even do, but those might be unreachable object islands. Only shallow\n> points that are reachable from old and new refs can stay.\n>\n> The way it's implemented is paint down from each ref, attach a bitmap\n> to each commit where one bit represents one ref. In order to avoid\n> allocating new bitmap for every commit, we try to reuse the same\n> bitmap for parent commits if possible. This reduces allocation and\n> leaks deliberately because it's hard to keep/time consuming track how\n> many pointers to the same buffer.\n>\n> In a typical push or fetch, the new pack should not carry new shallow\n> roots. If the current repo does not have any commit islands refered by\n> the sender's shallow roots either, this function is just a few\n> has_sha1_file(). So quite cheap.\n>\n> Once the sender diverts from that path (or the receiver detects\n> shallow roots attached to commit islands from remove_reachable_shallow_points),\n> it'll be a lot more expensive. Pack bitmaps won't help this kind of\n> commit traversal, but commit cache might.\n>\n> Signed-off-by: Nguyễn Thái Ngọc Duy \n> ---\n> commit.h | 4 ++\n> shallow.c | 219 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n> 2 files changed, 223 insertions(+)\n\nHmph. the use of ->util field in this patch feels that it was\nsomething commit-slab data structure was invented to solve.\n\n\n> diff --git a/shallow.c b/shallow.c\n> index a974d2d..c92a1dc 100644\n> --- a/shallow.c\n> +++ b/shallow.c\n> @@ -4,6 +4,8 @@\n> #include \"pkt-line.h\"\n> #include \"remote.h\"\n> #include \"refs.h\"\n> +#include \"diff.h\"\n> +#include \"revision.h\"\n> \n> static int is_shallow = -1;\n> static struct stat shallow_stat;\n> @@ -280,3 +282,220 @@ void remove_reachable_shallow_points(struct extra_have_objects *out,\n> \t}\n> \tfree(ca.commits);\n> }\n> +\n> +static int paint_down(const unsigned char *sha1, int id, int nr_bits, int quick)\n> +{\n> +\tint hit_bottom = 0;\n> +\tunsigned int i, nr;\n> +\tstruct commit_list *head = NULL;\n> +\tint bitmap_nr = (nr_bits + 31) / 32;\n> +\tint bitmap_size = bitmap_nr * sizeof(uint32_t);\n> +\tuint32_t *tmp = xmalloc(bitmap_size);\n> +\tuint32_t *bitmap = xcalloc(bitmap_size, sizeof(uint32_t));\n> +\tbitmap[id / 32] |= (1 << (id % 32));\n> +\tcommit_list_insert(lookup_commit(sha1), &head);\n> +\twhile (head) {\n> +\t\tstruct commit_list *p;\n> +\t\tstruct commit *c = head->item;\n> +\t\tuint32_t *c_util = c->util;\n> +\n> +\t\tp = head;\n> +\t\thead = head->next;\n> +\t\tfree(p);\n> +\n> +\t\tif (c->object.flags & (SEEN | UNINTERESTING))\n> +\t\t\tcontinue;\n> +\t\telse\n> +\t\t\tc->object.flags |= SEEN;\n> +\n> +\t\tif (c->util == NULL)\n> +\t\t\tc->util = bitmap;\n> +\t\telse {\n> +\t\t\t/*\n> +\t\t\t * Deliberately leak a lot in commit->util\n> +\t\t\t * because there can be many pointers to the\n> +\t\t\t * same bitmap. Probably should allocate in a\n> +\t\t\t * pool and free the whole pool at the end.\n> +\t\t\t */\n\n... or perhaps make the bitmap into\n\n\tstruct {\n\t int refcnt;\n \tuint32_t bits[FLEX_ARRAY];\n\t}\n\nand refcnt them?\n\n\n> +\t\t\tmemcpy(tmp, c_util, bitmap_size);\n> +\t\t\tfor (i = 0; i < bitmap_nr; i++)\n> +\t\t\t\ttmp[i] |= bitmap[i];\n> +\t\t\tif (memcmp(tmp, c_util, bitmap_size)) {\n> +\t\t\t\tc->util = xmalloc(bitmap_size);\n> +\t\t\t\tmemcpy(c->util, tmp, bitmap_size);\n> +\t\t\t}\n> +\t\t}\n> +\n> +\t\tif (c->object.flags & BOTTOM) {\n> +\t\t\thit_bottom = 1;\n> +\t\t\tif (quick) {\n> +\t\t\t\tfree_commit_list(head);\n> +\t\t\t\tbreak;\n> +\t\t\t} else\n> +\t\t\t\tcontinue;\n> +\t\t}\n> +\n> +\t\tif (parse_commit(c))\n> +\t\t\tdie(\"unable to parse commit %s\",\n> +\t\t\t sha1_to_hex(c->object.sha1));\n> +\n> +\t\tfor (p = c->parents; p; p = p->next) {\n> +\t\t\tif (p->item->object.flags & SEEN)\n> +\t\t\t\tcontinue;\n> +\t\t\tif (p->item->util == NULL || p->item->util == c_util)\n> +\t\t\t\tp->item->util = c->util;\n> +\t\t\tcommit_list_insert(p->item, &head);\n> +\t\t}\n> +\t}\n> +\n> +\tnr = get_max_object_index();\n> +\tfor (i = 0; i < nr; i++) {\n> +\t\tstruct object *o = get_indexed_object(i);\n> +\t\tif (o && o->type == OBJ_COMMIT) {\n> +\t\t\to->flags &= ~SEEN;\n> +\t\t}\n> +\t}\n> +\n> +\tfree(tmp);\n> +\treturn hit_bottom;\n> +}\n> +\n> +static int mark_uninteresting(const char *refname,\n> +\t\t\t const unsigned char *sha1,\n> +\t\t\t int flags, void *cb_data)\n> +{\n> +\tstruct commit *commit = lookup_commit(sha1);\n> +\tcommit->object.flags |= UNINTERESTING;\n> +\tmark_parents_uninteresting(commit);\n> +\treturn 0;\n> +}\n> +\n> +struct saved_util {\n> +\tunsigned char sha1[20];\n> +\tvoid *util;\n> +};\n> +\n> +/*\n> + * Given a set of refs and shallow roots, find out what ref can reach\n> + * any of the given roots. If so mark that ref \"reject_flag\". If\n> + * \"used\" is not NULL, mark all reachable roots. Return how many refs\n> + * that need new shallow points.\n> + */\n> +int mark_new_shallow_refs(const struct extra_have_objects *ref,\n> +\t\t\t int *ref_status, uint32_t **used,\n> +\t\t\t const struct extra_have_objects *shallow)\n> +{\n> +\tstruct saved_util *util = NULL;\n> +\tunsigned int i, nr, ret = 0, nr_util = 0, alloc_util = 0;\n> +\n> +\t/*\n> +\t * Quick check to see if we may need to add new shallow\n> +\t * roots. Go through the list of root candidates and check if\n> +\t * they exist (either in current repo, or in the new pack, we\n> +\t * can't distinguish).\n> +\t *\n> +\t * 1) If none of the new roots exist, the pack must connect to\n> +\t * the main object graph, which is already guarded by\n> +\t * current repo's shallow roots and we will not need to\n> +\t * consider adding new shallow roots, so we can exit early.\n> +\t *\n> +\t * 2) The pack may connect to some existing object islands in\n> +\t * current repo then add shallow roots to plug loose ends\n> +\t * from those islands. In that case, new shallow roots must\n> +\t * also exist in the repo as this stage (old objects plus\n> +\t * the new pack).\n> +\t *\n> +\t * 3) The last, easiest case, is the pack contains some\n> +\t * shallow roots, which may be used to tie up loose ends of\n> +\t * some new refs, or redundanty (tying up loose ends of new\n> +\t * object islands)\n> +\t */\n> +\tfor (i = 0;i < shallow->nr; i++)\n> +\t\tif (has_sha1_file(shallow->array[i]))\n> +\t\t\tbreak;\n> +\tif (i == shallow->nr)\n> +\t\t/*\n> +\t\t * this is the first and also the common case, where\n> +\t\t * the new pack does not carry any new shallow\n> +\t\t * points. No need to to the expensive commit traverse\n> +\t\t * dance below.\n> +\t\t */\n> +\t\treturn 0;\n\nI am Confused.\n\nThe loop only made sure that all the elements in the array[] is\nstill missing (or, ... is this function supposed to be called before\ninstalling a new pack??? It is unclear. But if new objects were\nunpacked while receiving, then there is no \"not install the new\nobjects and check\" possible, so I'd assume this is called after\nreceiving and registering a new pack to the object store).\n\nBut then, can it be that you had N-1 \"shallow points\" originally,\nthe pack has a reference to a new missing commit, and the array has\nN \"shallow points\" in total? Or is the caller expected to call this\nfunction with shallow pointing at a pre-transfer shallow points?\n\n> +\t/*\n> +\t * Prepare the commit graph to track what refs can reach what\n> +\t * (new) shallow points.\n> +\t */\n> +\tnr = get_max_object_index();\n\nHmph. At this point (again, there is no description on where in the\noverall sequence of events this function is designed to be called,\nso it is impossible to review the logic), is it expected that we\nhave seen all the objects under the sun and marked them in a\nspecific way?\n\n> +\tfor (i = 0; i < nr; i++) {\n> +\t\tstruct object *o = get_indexed_object(i);\n> +\t\tstruct commit *c = (struct commit *)o;\n> +\t\tif (!o || o->type != OBJ_COMMIT)\n> +\t\t\tcontinue;\n> +\n> +\t\to->flags &= ~(UNINTERESTING | BOTTOM | SEEN);\n> +\t\t/*\n> +\t\t * git-fetch makes use of \"util\" field. Save it and\n> +\t\t * restore later. For fetch/clone/push, \"nr\" should be\n> +\t\t * small because rev-list is delayed to pack-objects.\n> +\t\t */\n> +\t\tif (c->util) {\n> +\t\t\tALLOC_GROW(util, nr_util+1, alloc_util);\n> +\t\t\thashcpy(util[nr_util].sha1, o->sha1);\n> +\t\t\tutil[nr_util].util = c->util;\n> +\t\t\tnr_util++;\n> +\t\t\tc->util = NULL;\n> +\t\t}\n> +\t}\n> +\n> +\t/*\n> +\t * \"--not --all\" to cut short the traversal if new refs\n> +\t * connect to old refs. If not (e.g. force ref updates) it'll\n> +\t * have to go down to the current shallow roots.\n> +\t *\n> +\t * We could detect that a new commit is connected to an\n> +\t * existing commit by keeping new objects in a pack (i.e. the\n> +\t * index-pack code path) then check commit origin. If so stop\n> +\t * short, so we don't need to get to the bottom. But then it\n> +\t * will not work for case #2 because we need to go through\n> +\t * some of our commits before reaching new shallow roots.\n> +\t */\n> +\thead_ref(mark_uninteresting, NULL);\n> +\tfor_each_ref(mark_uninteresting, NULL);\n> +\n> +\tfor (i = 0; i < shallow->nr; i++)\n> +\t\tif (has_sha1_file(shallow->array[i])) {\n> +\t\t\tstruct commit *c = lookup_commit(shallow->array[i]);\n> +\t\t\tc->object.flags |= BOTTOM;\n> +\t\t}\n> +\n> +\tfor (i = 0; i < ref->nr; i++)\n> +\t\tif (paint_down(ref->array[i], i, ref->nr, used == NULL)) {\n> +\t\t\tif (ref_status)\n> +\t\t\t\tref_status[i] = 1;\n> +\t\t\tret++;\n> +\t\t}\n> +\n> +\tif (used) {\n> +\t\tfor (i = 0; i < shallow->nr; i++) {\n> +\t\t\tstruct commit *c = lookup_commit(shallow->array[i]);\n> +\t\t\tused[i] = c->util;\n> +\t\t}\n> +\t}\n> +\n> +\tif (nr_util) {\n> +\t\tnr = get_max_object_index();\n> +\t\tfor (i = 0; i < nr; i++) {\n> +\t\t\tstruct object *o = get_indexed_object(i);\n> +\t\t\tif (o && o->type == OBJ_COMMIT)\n> +\t\t\t\t((struct commit *)o)->util = NULL;\n> +\t\t}\n> +\t\tfor (i = 0; i < nr_util; i++) {\n> +\t\t\tstruct commit *c = lookup_commit(util[i].sha1);\n> +\t\t\tc->util = util[i].util;\n> +\t\t}\n> +\t\tfree(util);\n> +\t}\n> +\n> +\treturn ret;\n> +}\n"},{"id":"231115","messageId":"xmqqhab0w244.fsf@gitster.dls.corp.google.com","threadId":"35400","inReplyTo":"1385351754-9954-10-git-send-email-pclouds@gmail.com","subject":"Re: [PATCH v3 09/28] shallow.c: extend setup_*_shallow() to accept extra shallow points","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-11-25T22:25:15Z","receivedAt":"2013-11-25T22:25:15Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Nguyễn Thái Ngọc Duy writes:\n\n> Signed-off-by: Nguyễn Thái Ngọc Duy \n> ---\n> commit.h | 8 +++++---\n> fetch-pack.c | 5 +++--\n> shallow.c | 20 +++++++++++++++-----\n> upload-pack.c | 2 +-\n> 4 files changed, 24 insertions(+), 11 deletions(-)\n>\n> diff --git a/commit.h b/commit.h\n> index e1fd587..3af4699 100644\n> --- a/commit.h\n> +++ b/commit.h\n> @@ -203,10 +203,12 @@ extern struct commit_list *get_shallow_commits(struct object_array *heads,\n> \t\tint depth, int shallow_flag, int not_shallow_flag);\n> extern void check_shallow_file_for_update(void);\n> extern void set_alternate_shallow_file(const char *path);\n> -extern int write_shallow_commits(struct strbuf *out, int use_pack_protocol);\n> +extern int write_shallow_commits(struct strbuf *out, int use_pack_protocol,\n> +\t\t\t\t const struct extra_have_objects *extra);\n\nConfusing.\n\nSounds as if you got the extra \".have\" and storing them in the\n.git/shallow file (which of course would not make much sense), which\nis not what is going on.\n\nAlso it is unclear how the sanity check the previous step seems to\nmake and the new list of shallow commit names this patch adds to\nexisting functions are designed to interact. I think the whole\ndesign needs a better explanation of the flow at the higher level.\n"},{"id":"231116","messageId":"xmqqd2low1b5.fsf@gitster.dls.corp.google.com","threadId":"35400","inReplyTo":"xmqqtxf0w440.fsf@gitster.dls.corp.google.com","subject":"Re: [PATCH v3 06/28] connect.c: teach get_remote_heads to parse \"shallow\" lines","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-11-25T22:42:38Z","receivedAt":"2013-11-25T22:42:38Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n> Perhaps a preparatory patch needs to rename the structure type to\n> object_name_list or something. And then we can make the variable\n> names, not typenames, responsible for signalling what they mean,\n> i.e.\n>\n> \tget_remote_heads(...\n> \tstruct list_of_objects *extra_have,\n> struct list_of_objects *shallow_points);\n>\n> when we introduce the new parameter.\n\nYuck, and these are not \"list-of-objects\", either. They are\n\"list-of-object-names\".\n"},{"id":"231125","messageId":"CAPig+cRVXvBwWDC2Rp8M9hDCKexCJCEGeKLOa__aMX3q9X8S1w@mail.gmail.com","threadId":"35400","inReplyTo":"1385351754-9954-4-git-send-email-pclouds@gmail.com","subject":"Re: [PATCH v3 03/28] clone: prevent --reference to a shallow repository","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2013-11-26T05:52:14Z","receivedAt":"2013-11-26T05:52:14Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Sun, Nov 24, 2013 at 10:55 PM, Nguyễn Thái Ngọc Duy\n wrote:\n> If we borrow objects from another repository, we should also pay\n> attention to their $GIT_DIR/shallow (and even info/grafts). But\n> current alternates code does not.\n>\n> Reject alternate repos that are shallow because we do not do it\n> right. In future we alternate code may be updated to check\n\ns/we/the/\n\n> $GIT_DIR/shallow properly so that this restriction could be lifted.\n>\n> Signed-off-by: Nguyễn Thái Ngọc Duy \n> ---\n> builtin/clone.c | 6 ++++++\n> 1 file changed, 6 insertions(+)\n>\n> diff --git a/builtin/clone.c b/builtin/clone.c\n> index 874e0fd..900f564 100644\n> --- a/builtin/clone.c\n> +++ b/builtin/clone.c\n> @@ -252,6 +252,12 @@ static int add_one_reference(struct string_list_item *item, void *cb_data)\n> die(_(\"reference repository '%s' is not a local repository.\"),\n> item->string);\n>\n> + if (!access(mkpath(\"%s/shallow\", ref_git), F_OK))\n> + die(_(\"reference repository '%s' is shallow\"), item->string);\n> +\n> + if (!access(mkpath(\"%s/info/grafts\", ref_git), F_OK))\n> + die(_(\"reference repository '%s' is grafted\"), item->string);\n> +\n> strbuf_addf(&alternate, \"%s/objects\", ref_git);\n> add_to_alternates_file(alternate.buf);\n> strbuf_release(&alternate);\n> --\n> 1.8.2.83.gc99314b\n"},{"id":"231134","messageId":"CACsJy8Duzkj4WVvXY8JEx4yNBjmY43+GCCp7oeQk21WT+5CJsg@mail.gmail.com","threadId":"35400","inReplyTo":"xmqqr4a4xn1a.fsf@gitster.dls.corp.google.com","subject":"Re: [PATCH v3 04/28] update-server-info: do not publish shallow clones","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2013-11-26T12:41:26Z","receivedAt":"2013-11-26T12:41:26Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Tue, Nov 26, 2013 at 3:08 AM, Junio C Hamano wrote:\n> Nguyễn Thái Ngọc Duy writes:\n>\n>> Dumb commit walker does not care about .git/shallow and until someone\n>> steps up to make it happen, let's not publish shallow clones using\n>> dumb protocols.\n>>\n>> Signed-off-by: Nguyễn Thái Ngọc Duy \n>> ---\n>> server-info.c | 9 +++++++++\n>> 1 file changed, 9 insertions(+)\n>\n> I doubt that pros-and-cons is in this patch's favor. Without this\n> patch, if a fetch requests commits just on the surface in this\n> shallow repository, the walker would happily get the necessary\n> objects just fine. If the request has to dig deeper and cross the\n> shallow boundary, the walker will get a failure and error out.\n>\n> With this patch, both will error out. So overall, the patch did not\n> make anything safer (e.g. preventing from introducing new corruption\n> on the recipient's end) while breaking a case where it worked just\n> fine, no?\n>\n> Or am I missing something? Not that dumb walkers would matter very\n> much these days, ...\n\nNo you're not. If it may fail, in my opinion it should fail early than\nwalk all the way and fail. But yeah dropping the patch is fine too. I\ndon't care too much about dumb walkers.\n-- \nDuy\n"},{"id":"231136","messageId":"CACsJy8DgaqtQUohRwek2+cPxTEssvtDGb812Uhjy+R7pib2O8Q@mail.gmail.com","threadId":"35400","inReplyTo":"xmqqli0cw2b9.fsf@gitster.dls.corp.google.com","subject":"Re: [PATCH v3 08/28] shallow.c: add mark_new_shallow_refs()","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2013-11-26T13:18:30Z","receivedAt":"2013-11-26T13:18:30Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Tue, Nov 26, 2013 at 5:20 AM, Junio C Hamano wrote:\n> Hmph. the use of ->util field in this patch feels that it was\n> something commit-slab data structure was invented to solve.\n\nGood stuff! Thanks.\n\n>> + if (c->util == NULL)\n>> + c->util = bitmap;\n>> + else {\n>> + /*\n>> + * Deliberately leak a lot in commit->util\n>> + * because there can be many pointers to the\n>> + * same bitmap. Probably should allocate in a\n>> + * pool and free the whole pool at the end.\n>> + */\n>\n> ... or perhaps make the bitmap into\n>\n> struct {\n> int refcnt;\n> uint32_t bits[FLEX_ARRAY];\n> }\n>\n> and refcnt them?\n\nI still prefer memory pools so I just need to do a few free() than\nwalking through all the commits again and refcnt-- or free() them.\n\n>> + /*\n>> + * Quick check to see if we may need to add new shallow\n>> + * roots. Go through the list of root candidates and check if\n>> + * they exist (either in current repo, or in the new pack, we\n>> + * can't distinguish).\n>> + *\n>> + * 1) If none of the new roots exist, the pack must connect to\n>> + * the main object graph, which is already guarded by\n>> + * current repo's shallow roots and we will not need to\n>> + * consider adding new shallow roots, so we can exit early.\n>> + *\n>> + * 2) The pack may connect to some existing object islands in\n>> + * current repo then add shallow roots to plug loose ends\n>> + * from those islands. In that case, new shallow roots must\n>> + * also exist in the repo as this stage (old objects plus\n>> + * the new pack).\n>> + *\n>> + * 3) The last, easiest case, is the pack contains some\n>> + * shallow roots, which may be used to tie up loose ends of\n>> + * some new refs, or redundanty (tying up loose ends of new\n>> + * object islands)\n>> + */\n>> + for (i = 0;i < shallow->nr; i++)\n>> + if (has_sha1_file(shallow->array[i]))\n>> + break;\n>> + if (i == shallow->nr)\n>> + /*\n>> + * this is the first and also the common case, where\n>> + * the new pack does not carry any new shallow\n>> + * points. No need to to the expensive commit traverse\n>> + * dance below.\n>> + */\n>> + return 0;\n>\n> I am Confused.\n>\n> The loop only made sure that all the elements in the array[] is\n> still missing (or, ... is this function supposed to be called before\n> installing a new pack??? It is unclear. But if new objects were\n> unpacked while receiving, then there is no \"not install the new\n> objects and check\" possible, so I'd assume this is called after\n> receiving and registering a new pack to the object store).\n\nYes it's called after installing the new packs (or after unpacking).\nI'll mention about this.\n\n> But then, can it be that you had N-1 \"shallow points\" originally,\n> the pack has a reference to a new missing commit, and the array has\n> N \"shallow points\" in total? Or is the caller expected to call this\n> function with shallow pointing at a pre-transfer shallow points?\n\nAnother thing I did not mention is all shallow commits we have are\nalready filtered out by remove_reachable_shallow_points. By the time\nyou get here, the array only contains the shallow commits we don't\nhave that may or may not be referenced by a something (oh yeah I did\nnot handle tags!) in the new pack. So if all of them are missing (i.e.\nthe new pack does not reference to any of them), they're useless and\ncan be thrown away.\n\nSorry to break the patches this way and lose the overall call flow.\nIt's just too big to put all into one patch. 13/28 is the one that put\nthe pieces together but basically\n\n 1. receive the remote's .git/shallow\n 2. call remote_reachable_shallow_points() to exclude our shallow commits\n 3. get the pack and install it (or unpack it)\n 4. call this function to determine what new ref needs new shallow\ncommits from the result of #2\n\n>> + /*\n>> + * Prepare the commit graph to track what refs can reach what\n>> + * (new) shallow points.\n>> + */\n>> + nr = get_max_object_index();\n>\n> Hmph. At this point (again, there is no description on where in the\n> overall sequence of events this function is designed to be called,\n> so it is impossible to review the logic), is it expected that we\n> have seen all the objects under the sun and marked them in a\n> specific way?\n\nNo. At this point we just make sure to have clean flags in commit\nobjects. paint_down() is the one that walks through and marks them as\nseen.\n-- \nDuy\n"},{"id":"231145","messageId":"CAPig+cSnq8qWusYHYoWRf_n_GaLL_tXTQk5LkLoXuh_WgB00Dg@mail.gmail.com","threadId":"35400","inReplyTo":"1385351754-9954-18-git-send-email-pclouds@gmail.com","subject":"Re: [PATCH v3 17/28] Support pushing from a shallow clone","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2013-11-26T20:38:35Z","receivedAt":"2013-11-26T20:38:35Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Sun, Nov 24, 2013 at 10:55 PM, Nguyễn Thái Ngọc Duy\n wrote:\n> Pushing from a shallow clone using today's send-pack and receive-pack\n> may work, if the transferred pack does not end up at any graft\n> points. If it does, recent receive-pack that does connectivity check\n> will reject the push. If receive-pack is old and does not have the\n> connectivity check, the upstream repo becomes corrupt.\n>\n> The pack protocol is updated and send-pack now sends all shallow\n> grafts before it sends the commands, if the repo is shallow. This\n> protocol extension will break current receive-pack, which is intended,\n> mostly to stop corrupting the upstream repo.\n>\n> Changes on the receiver are similar to what has been done in\n> fetch-pack, i.e. filter out refs that require new shallow roots then\n> go along as usual.\n>\n> Signed-off-by: Nguyễn Thái Ngọc Duy \n> ---\n> diff --git a/Documentation/technical/pack-protocol.txt b/Documentation/technical/pack-protocol.txt\n> index eb8edd1..c73b62f 100644\n> --- a/Documentation/technical/pack-protocol.txt\n> +++ b/Documentation/technical/pack-protocol.txt\n> @@ -43,6 +43,9 @@ static int fix_thin = 1;\n> static const char *head_name;\n> static void *head_name_to_free;\n> static int sent_capabilities;\n> +static int shallow_push;\n> +static const char* alternate_shallow_file;\n\ns/char\\* /char */\n\n> +static struct extra_have_objects shallow;\n>\n> static enum deny_action parse_deny_action(const char *var, const char *value)\n> {\n"},{"id":"231151","messageId":"xmqq7gbuvm8q.fsf@gitster.dls.corp.google.com","threadId":"35400","inReplyTo":"CACsJy8DgaqtQUohRwek2+cPxTEssvtDGb812Uhjy+R7pib2O8Q@mail.gmail.com","subject":"Re: [PATCH v3 08/28] shallow.c: add mark_new_shallow_refs()","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-11-26T22:20:21Z","receivedAt":"2013-11-26T22:20:21Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Duy Nguyen writes:\n\n> On Tue, Nov 26, 2013 at 5:20 AM, Junio C Hamano wrote:\n>> Hmph. the use of ->util field in this patch feels that it was\n>> something commit-slab data structure was invented to solve.\n>\n> Good stuff! Thanks.\n>\n>>> + if (c->util == NULL)\n>>> + c->util = bitmap;\n>>> + else {\n>>> + /*\n>>> + * Deliberately leak a lot in commit->util\n>>> + * because there can be many pointers to the\n>>> + * same bitmap. Probably should allocate in a\n>>> + * pool and free the whole pool at the end.\n>>> + */\n>>\n>> ... or perhaps make the bitmap into\n>>\n>> struct {\n>> int refcnt;\n>> uint32_t bits[FLEX_ARRAY];\n>> }\n>>\n>> and refcnt them?\n>\n> I still prefer memory pools so I just need to do a few free() than\n> walking through all the commits again and refcnt-- or free() them.\n\nFair enough.\n\n> Sorry to break the patches this way and lose the overall call flow.\n> It's just too big to put all into one patch. 13/28 is the one that put\n> the pieces together but basically\n>\n> 1. receive the remote's .git/shallow\n> 2. call remote_reachable_shallow_points() to exclude our shallow commits\n> 3. get the pack and install it (or unpack it)\n> 4. call this function to determine what new ref needs new shallow\n> commits from the result of #2\n\nThanks for a roadmap. Will find time to re-read the thing with it.\n"},{"id":"231154","messageId":"CAPig+cS1rbX4DZSv-V5_fOY8bfT5jUvXZCtAWpSfhcHUNhV8LQ@mail.gmail.com","threadId":"35400","inReplyTo":"1385351754-9954-28-git-send-email-pclouds@gmail.com","subject":"Re: [PATCH v3 27/28] clone: use git protocol for cloning shallow repo locally","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2013-11-27T01:36:42Z","receivedAt":"2013-11-27T01:36:42Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Sun, Nov 24, 2013 at 10:55 PM, Nguyễn Thái Ngọc Duy\n wrote:\n> clone_local() does not handle $SRC/shallow. It could be made so, but\n> it's simpler to use fetch-pack/upload-pack instead.\n>\n> This used by be caught by the check in upload-pack, which is triggered\n\ns/used by/used to/\n\n> by transport_get_remote_refs(), even in local clone case. The check is\n> now gone and check_everything_connected() should catch the result\n> incomplete repo. But check_everything_connected() will soon be skipped\n> in local clone case, opening a door to corrupt repo. This patch should\n> close that door.\n>\n> Signed-off-by: Nguyễn Thái Ngọc Duy \n"},{"id":"231155","messageId":"CAPig+cRMj_RY53--OLDmonZLzhzf0Cu+JrDc8LAicSn5QGKNyQ@mail.gmail.com","threadId":"35400","inReplyTo":"1385351754-9954-16-git-send-email-pclouds@gmail.com","subject":"Re: [PATCH v3 15/28] fetch: add --update-shallow to get refs that require updating .git/shallow","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2013-11-27T01:53:52Z","receivedAt":"2013-11-27T01:53:52Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Sun, Nov 24, 2013 at 10:55 PM, Nguyễn Thái Ngọc Duy\n wrote:\n> diff --git a/t/t5536-fetch-shallow.sh b/t/t5536-fetch-shallow.sh\n> index e011ead..95b6313 100755\n> --- a/t/t5536-fetch-shallow.sh\n> +++ b/t/t5536-fetch-shallow.sh\n> @@ -141,4 +141,26 @@ EOF\n> )\n> '\n>\n> +test_expect_success 'fetch --update-shallow' '\n> + (\n> + cd notshallow &&\n> + git fetch --update-shallow ../shallow/.git refs/heads/*:refs/remotes/shallow/* &&\n> + git fsck &&\n> + git for-each-ref --format=\"%(refname)\" |sort >actual.refs &&\n\nExit status of git-for-each-ref could be lost down the pipe.\n\n> + cat <expect.refs &&\n> +refs/remotes/shallow/master\n> +refs/remotes/shallow/no-shallow\n> +EOF\n> + test_cmp expect.refs actual.refs &&\n> + git log --format=%s shallow/master >actual &&\n> + cat <expect &&\n> +6\n> +5\n> +4\n> +3\n> +EOF\n> + test_cmp expect actual\n> + )\n> +'\n> +\n> test_done\n"},{"id":"231162","messageId":"CAPig+cTaNuq+f7=27wnZgNi4niMt0xQyQxKq3VdpuH8ZvXxdYQ@mail.gmail.com","threadId":"35400","inReplyTo":"1385351754-9954-14-git-send-email-pclouds@gmail.com","subject":"Re: [PATCH v3 13/28] fetch: support fetching from a shallow repository","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2013-11-27T09:47:26Z","receivedAt":"2013-11-27T09:47:26Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Sun, Nov 24, 2013 at 10:55 PM, Nguyễn Thái Ngọc Duy\n wrote:\n> diff --git a/t/t5536-fetch-shallow.sh b/t/t5536-fetch-shallow.sh\n> new file mode 100755\n> index 0000000..41de74d\n> --- /dev/null\n> +++ b/t/t5536-fetch-shallow.sh\n> @@ -0,0 +1,128 @@\n> +#!/bin/sh\n> +\n> +test_description='fetch/clone from a shallow clone'\n> +\n> +. ./test-lib.sh\n> +\n> +commit() {\n> + echo \"$1\" >tracked &&\n> + git add tracked &&\n> + git commit -m \"$1\"\n> +}\n> +\n> +test_expect_success 'fetch something upstream has but hidden by clients shallow boundaries' '\n> + # the blob \"1\" is available in .git but hidden by the\n> + # shallow2/.git/shallow and it should be resent\n> + ! git --git-dir=shallow2/.git cat-file blob `echo 1|git hash-object --stdin` >/dev/null &&\n> + echo 1 > 1.t &&\n\ns/> />/\n\n> + git add 1.t &&\n> + git commit -m add-1-back &&\n> + (\n> + cd shallow2 &&\n> + git fetch ../.git +refs/heads/master:refs/remotes/top/master &&\n> + git fsck &&\n> + git log --format=%s top/master >actual &&\n> + cat <expect &&\n> +add-1-back\n> +4\n> +3\n> +EOF\n> + test_cmp expect actual\n> + ) &&\n> + git --git-dir=shallow2/.git cat-file blob `echo 1|git hash-object --stdin` >/dev/null\n> +\n> +'\n"},{"id":"231189","messageId":"CACsJy8CJRzCVzJ7LfEY_2Gxzq=96BB44UQUeMD0R-a6buYNu4A@mail.gmail.com","threadId":"35400","inReplyTo":"CAPig+cRMj_RY53--OLDmonZLzhzf0Cu+JrDc8LAicSn5QGKNyQ@mail.gmail.com","subject":"Re: [PATCH v3 15/28] fetch: add --update-shallow to get refs that require updating .git/shallow","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2013-11-27T12:54:14Z","receivedAt":"2013-11-27T12:54:14Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Wed, Nov 27, 2013 at 8:53 AM, Eric Sunshine wrote:\n> On Sun, Nov 24, 2013 at 10:55 PM, Nguyễn Thái Ngọc Duy\n> wrote:\n>> diff --git a/t/t5536-fetch-shallow.sh b/t/t5536-fetch-shallow.sh\n>> index e011ead..95b6313 100755\n>> --- a/t/t5536-fetch-shallow.sh\n>> +++ b/t/t5536-fetch-shallow.sh\n>> @@ -141,4 +141,26 @@ EOF\n>> )\n>> '\n>>\n>> +test_expect_success 'fetch --update-shallow' '\n>> + (\n>> + cd notshallow &&\n>> + git fetch --update-shallow ../shallow/.git refs/heads/*:refs/remotes/shallow/* &&\n>> + git fsck &&\n>> + git for-each-ref --format=\"%(refname)\" |sort >actual.refs &&\n>\n> Exit status of git-for-each-ref could be lost down the pipe.\n\nI think it's ok. If for-each-ref is broken or something the final\noutput in actual.refs will be different anyway.\n-- \nDuy\n"},{"id":"231190","messageId":"CACsJy8CwbNuOVuQ7nBjXuosX-5MCo_u1pFUHqLXzBa9hd7M8Vg@mail.gmail.com","threadId":"35400","inReplyTo":"xmqqd2low1b5.fsf@gitster.dls.corp.google.com","subject":"Re: [PATCH v3 06/28] connect.c: teach get_remote_heads to parse \"shallow\" lines","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2013-11-27T13:02:14Z","receivedAt":"2013-11-27T13:02:14Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Tue, Nov 26, 2013 at 5:42 AM, Junio C Hamano wrote:\n> Junio C Hamano writes:\n>\n>> Perhaps a preparatory patch needs to rename the structure type to\n>> object_name_list or something. And then we can make the variable\n>> names, not typenames, responsible for signalling what they mean,\n>> i.e.\n>>\n>> get_remote_heads(...\n>> struct list_of_objects *extra_have,\n>> struct list_of_objects *shallow_points);\n>>\n>> when we introduce the new parameter.\n>\n> Yuck, and these are not \"list-of-objects\", either. They are\n> \"list-of-object-names\".\n\nI was thinking of renaming it to sha1_array and what do you know, we\ndo have 'struct sha1_array' with exact same (or better) functionality\nthat extra_have_objects provides. Will replace extra_have_objects with\ncurrent sha1_array.\n-- \nDuy\n"},{"id":"231203","messageId":"xmqqa9gpu0n0.fsf@gitster.dls.corp.google.com","threadId":"35400","inReplyTo":"CACsJy8CJRzCVzJ7LfEY_2Gxzq=96BB44UQUeMD0R-a6buYNu4A@mail.gmail.com","subject":"Re: [PATCH v3 15/28] fetch: add --update-shallow to get refs that require updating .git/shallow","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-11-27T19:04:35Z","receivedAt":"2013-11-27T19:04:35Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Duy Nguyen writes:\n\n> On Wed, Nov 27, 2013 at 8:53 AM, Eric Sunshine wrote:\n>> On Sun, Nov 24, 2013 at 10:55 PM, Nguyễn Thái Ngọc Duy\n>> wrote:\n>>> diff --git a/t/t5536-fetch-shallow.sh b/t/t5536-fetch-shallow.sh\n>>> index e011ead..95b6313 100755\n>>> --- a/t/t5536-fetch-shallow.sh\n>>> +++ b/t/t5536-fetch-shallow.sh\n>>> @@ -141,4 +141,26 @@ EOF\n>>> )\n>>> '\n>>>\n>>> +test_expect_success 'fetch --update-shallow' '\n>>> + (\n>>> + cd notshallow &&\n>>> + git fetch --update-shallow ../shallow/.git refs/heads/*:refs/remotes/shallow/* &&\n>>> + git fsck &&\n>>> + git for-each-ref --format=\"%(refname)\" |sort >actual.refs &&\n>>\n>> Exit status of git-for-each-ref could be lost down the pipe.\n>\n> I think it's ok. If for-each-ref is broken or something the final\n> output in actual.refs will be different anyway.\n\nYou can tell for-each-ref to sort its output in a way you specify,\nno?\n"},{"id":"231395","messageId":"xmqqy542q4aj.fsf@gitster.dls.corp.google.com","threadId":"35400","inReplyTo":"1385351754-9954-17-git-send-email-pclouds@gmail.com","subject":"Re: [PATCH v3 16/28] receive-pack: reorder some code in unpack()","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-12-02T22:25:08Z","receivedAt":"2013-12-02T22:25:08Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Nguyễn Thái Ngọc Duy writes:\n\n> This is the preparation for adding --shallow-file to both\n> unpack-objects and index-pack. To sum up:\n>\n> - status/code, ip/child, unpacker/keeper and i (now ac) are moved out\n> to function top level\n>\n> - successful flow now ends at the end of the function\n>\n> Signed-off-by: Nguyễn Thái Ngọc Duy \n\nSounds sensible; for a new piece of code, we probably should see if\nargv_array API is more appropriate, and I think it is for this case.\n\nThanks.\n\n> ---\n> builtin/receive-pack.c | 74 +++++++++++++++++++++++---------------------------\n> 1 file changed, 34 insertions(+), 40 deletions(-)\n>\n> diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c\n> index cc8c34f..22e162d 100644\n> --- a/builtin/receive-pack.c\n> +++ b/builtin/receive-pack.c\n> @@ -822,8 +822,11 @@ static const char *pack_lockfile;\n> static const char *unpack(int err_fd)\n> {\n> \tstruct pack_header hdr;\n> +\tconst char *argv[7];\n> \tconst char *hdr_err;\n> +\tint status, ac = 0;\n> \tchar hdr_arg[38];\n> +\tstruct child_process child;\n> \tint fsck_objects = (receive_fsck_objects >= 0\n> \t\t\t ? receive_fsck_objects\n> \t\t\t : transfer_fsck_objects >= 0\n> @@ -840,63 +843,54 @@ static const char *unpack(int err_fd)\n> \t\t\t\"--pack_header=%\"PRIu32\",%\"PRIu32,\n> \t\t\tntohl(hdr.hdr_version), ntohl(hdr.hdr_entries));\n> \n> +\tmemset(&child, 0, sizeof(child));\n> \tif (ntohl(hdr.hdr_entries) < unpack_limit) {\n> -\t\tint code, i = 0;\n> -\t\tstruct child_process child;\n> -\t\tconst char *unpacker[5];\n> -\t\tunpacker[i++] = \"unpack-objects\";\n> +\t\targv[ac++] = \"unpack-objects\";\n> \t\tif (quiet)\n> -\t\t\tunpacker[i++] = \"-q\";\n> +\t\t\targv[ac++] = \"-q\";\n> \t\tif (fsck_objects)\n> -\t\t\tunpacker[i++] = \"--strict\";\n> -\t\tunpacker[i++] = hdr_arg;\n> -\t\tunpacker[i++] = NULL;\n> -\t\tmemset(&child, 0, sizeof(child));\n> -\t\tchild.argv = unpacker;\n> +\t\t\targv[ac++] = \"--strict\";\n> +\t\targv[ac++] = hdr_arg;\n> +\t\targv[ac++] = NULL;\n> +\t\tchild.argv = argv;\n> \t\tchild.no_stdout = 1;\n> \t\tchild.err = err_fd;\n> \t\tchild.git_cmd = 1;\n> -\t\tcode = run_command(&child);\n> -\t\tif (!code)\n> -\t\t\treturn NULL;\n> -\t\treturn \"unpack-objects abnormal exit\";\n> +\t\tstatus = run_command(&child);\n> +\t\tif (status)\n> +\t\t\treturn \"unpack-objects abnormal exit\";\n> \t} else {\n> -\t\tconst char *keeper[7];\n> -\t\tint s, status, i = 0;\n> +\t\tint s;\n> \t\tchar keep_arg[256];\n> -\t\tstruct child_process ip;\n> \n> \t\ts = sprintf(keep_arg, \"--keep=receive-pack %\"PRIuMAX\" on \", (uintmax_t) getpid());\n> \t\tif (gethostname(keep_arg + s, sizeof(keep_arg) - s))\n> \t\t\tstrcpy(keep_arg + s, \"localhost\");\n> \n> -\t\tkeeper[i++] = \"index-pack\";\n> -\t\tkeeper[i++] = \"--stdin\";\n> +\t\targv[ac++] = \"index-pack\";\n> +\t\targv[ac++] = \"--stdin\";\n> \t\tif (fsck_objects)\n> -\t\t\tkeeper[i++] = \"--strict\";\n> +\t\t\targv[ac++] = \"--strict\";\n> \t\tif (fix_thin)\n> -\t\t\tkeeper[i++] = \"--fix-thin\";\n> -\t\tkeeper[i++] = hdr_arg;\n> -\t\tkeeper[i++] = keep_arg;\n> -\t\tkeeper[i++] = NULL;\n> -\t\tmemset(&ip, 0, sizeof(ip));\n> -\t\tip.argv = keeper;\n> -\t\tip.out = -1;\n> -\t\tip.err = err_fd;\n> -\t\tip.git_cmd = 1;\n> -\t\tstatus = start_command(&ip);\n> -\t\tif (status) {\n> +\t\t\targv[ac++] = \"--fix-thin\";\n> +\t\targv[ac++] = hdr_arg;\n> +\t\targv[ac++] = keep_arg;\n> +\t\targv[ac++] = NULL;\n> +\t\tchild.argv = argv;\n> +\t\tchild.out = -1;\n> +\t\tchild.err = err_fd;\n> +\t\tchild.git_cmd = 1;\n> +\t\tstatus = start_command(&child);\n> +\t\tif (status)\n> \t\t\treturn \"index-pack fork failed\";\n> -\t\t}\n> -\t\tpack_lockfile = index_pack_lockfile(ip.out);\n> -\t\tclose(ip.out);\n> -\t\tstatus = finish_command(&ip);\n> -\t\tif (!status) {\n> -\t\t\treprepare_packed_git();\n> -\t\t\treturn NULL;\n> -\t\t}\n> -\t\treturn \"index-pack abnormal exit\";\n> +\t\tpack_lockfile = index_pack_lockfile(child.out);\n> +\t\tclose(child.out);\n> +\t\tstatus = finish_command(&child);\n> +\t\tif (status)\n> +\t\t\treturn \"index-pack abnormal exit\";\n> +\t\treprepare_packed_git();\n> \t}\n> +\treturn NULL;\n> }\n> \n> static const char *unpack_with_sideband(void)\n"},{"id":"231584","messageId":"1386248575-10206-1-git-send-email-pclouds@gmail.com","threadId":"35400","inReplyTo":"1385351754-9954-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v4 00/28] First class shallow clone","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-12-05T13:02:27Z","receivedAt":"2013-12-05T13:02:27Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"This reroll should fix all comments I have received in v3. I reordered\nthe shallow checks a bit so in common case it should be as cheap as\na normal fetch or push. See 08/28 and 20/28 for the big picture. I'm not\nentirely happy with the hook issue in 20/28, but it looks good enough\nfor me. There are a few XXXes for further improvement, but I'll keep\nthem until this lands.\n\nTo recap, this series allows you to clone from a shallow repo, push to\nor fetch from any shallow repo. Normally it will reject new refs\nthat introduce new shallow boundaries to your repository, so if you're\nin a full clone, it will always stay a full clone. Use \"fetch\n--update-shallow\" or set receive.shallowupdate to accept those refs.\n\nNguyễn Thái Ngọc Duy (28):\n transport.h: remove send_pack prototype, already defined in send-pack.h\n Replace struct extra_have_objects with struct sha1_array\n send-pack: forbid pushing from a shallow repository\n clone: prevent --reference to a shallow repository\n Make the sender advertise shallow commits to the receiver\n connect.c: teach get_remote_heads to parse \"shallow\" lines\n shallow.c: extend setup_*_shallow() to accept extra shallow commits\n shallow.c: the 8 steps to select new commits for .git/shallow\n shallow.c: steps 6 and 7 to select new commits for .git/shallow\n fetch-pack.c: move shallow update code out of fetch_pack()\n fetch-pack.h: one statement per bitfield declaration\n clone: support remote shallow repository\n fetch: support fetching from a shallow repository\n upload-pack: make sure deepening preserves shallow roots\n fetch: add --update-shallow to accept refs that update .git/shallow\n receive-pack: reorder some code in unpack()\n receive/send-pack: support pushing from a shallow clone\n New var GIT_SHALLOW_FILE to propagate --shallow-file to subprocesses\n connected.c: add new variant that runs with --shallow-file\n receive-pack: allow pushes that update .git/shallow\n send-pack: support pushing to a shallow clone\n remote-curl: pass ref SHA-1 to fetch-pack as well\n Support shallow fetch/clone over smart-http\n receive-pack: support pushing to a shallow clone via http\n send-pack: support pushing from a shallow clone via http\n clone: use git protocol for cloning shallow repo locally\n prune: clean .git/shallow after pruning objects\n git-clone.txt: remove shallow clone limitations\n\n Documentation/config.txt | 4 +\n Documentation/fetch-options.txt | 14 +-\n Documentation/git-clone.txt | 7 +-\n Documentation/git-prune.txt | 2 +\n Documentation/gitremote-helpers.txt | 7 +\n Documentation/technical/pack-protocol.txt | 7 +-\n builtin/clone.c | 18 +-\n builtin/fetch-pack.c | 23 +-\n builtin/fetch.c | 15 +-\n builtin/gc.c | 1 +\n builtin/prune.c | 4 +\n builtin/receive-pack.c | 314 +++++++++++++++----\n builtin/send-pack.c | 9 +-\n cache.h | 3 +\n commit.h | 39 ++-\n connect.c | 22 +-\n connected.c | 42 ++-\n connected.h | 2 +\n environment.c | 6 +\n fetch-pack.c | 132 +++++++-\n fetch-pack.h | 29 +-\n git.c | 2 +-\n remote-curl.c | 34 ++-\n remote.h | 9 +-\n send-pack.c | 27 +-\n send-pack.h | 2 +-\n shallow.c | 486 +++++++++++++++++++++++++++++-\n t/t5304-prune.sh | 10 +\n t/t5536-fetch-shallow.sh (new +x) | 203 +++++++++++++\n t/t5537-push-shallow.sh (new +x) | 183 +++++++++++\n t/t5601-clone.sh | 7 +\n trace.c | 2 +-\n transport-helper.c | 6 +\n transport.c | 25 +-\n transport.h | 16 +-\n upload-pack.c | 8 +-\n 36 files changed, 1555 insertions(+), 165 deletions(-)\n create mode 100755 t/t5536-fetch-shallow.sh\n create mode 100755 t/t5537-push-shallow.sh\n\n-- \n1.8.5.1.25.g8667982\n"},{"id":"231586","messageId":"1386248575-10206-2-git-send-email-pclouds@gmail.com","threadId":"35400","inReplyTo":"1386248575-10206-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v4 01/28] transport.h: remove send_pack prototype, already defined in send-pack.h","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-12-05T13:02:28Z","receivedAt":"2013-12-05T13:02:28Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Signed-off-by: Nguyễn Thái Ngọc Duy \n---\n transport.h | 6 ------\n 1 file changed, 6 deletions(-)\n\ndiff --git a/transport.h b/transport.h\nindex 8f96bed..b3679bb 100644\n--- a/transport.h\n+++ b/transport.h\n@@ -193,10 +193,4 @@ void transport_print_push_status(const char *dest, struct ref *refs,\n \n typedef void alternate_ref_fn(const struct ref *, void *);\n extern void for_each_alternate_ref(alternate_ref_fn, void *);\n-\n-struct send_pack_args;\n-extern int send_pack(struct send_pack_args *args,\n-\t\t int fd[], struct child_process *conn,\n-\t\t struct ref *remote_refs,\n-\t\t struct extra_have_objects *extra_have);\n #endif\n-- \n1.8.5.1.25.g8667982\n"},{"id":"231585","messageId":"1386248575-10206-3-git-send-email-pclouds@gmail.com","threadId":"35400","inReplyTo":"1386248575-10206-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v4 02/28] Replace struct extra_have_objects with struct sha1_array","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-12-05T13:02:29Z","receivedAt":"2013-12-05T13:02:29Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"The latter can do everything the former can and is used in many more\nplaces.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/send-pack.c | 5 ++---\n connect.c | 12 +++---------\n remote.h | 7 ++-----\n send-pack.c | 7 ++++---\n send-pack.h | 2 +-\n transport.c | 3 ++-\n 6 files changed, 14 insertions(+), 22 deletions(-)\n\ndiff --git a/builtin/send-pack.c b/builtin/send-pack.c\nindex 4482f16..faaa603 100644\n--- a/builtin/send-pack.c\n+++ b/builtin/send-pack.c\n@@ -10,6 +10,7 @@\n #include \"quote.h\"\n #include \"transport.h\"\n #include \"version.h\"\n+#include \"sha1-array.h\"\n \n static const char send_pack_usage[] =\n \"git send-pack [--all | --mirror] [--dry-run] [--force] [--receive-pack=] [--verbose] [--thin] [:] [...]\\n\"\n@@ -99,7 +100,7 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)\n \tconst char *dest = NULL;\n \tint fd[2];\n \tstruct child_process *conn;\n-\tstruct extra_have_objects extra_have;\n+\tstruct sha1_array extra_have = SHA1_ARRAY_INIT;\n \tstruct ref *remote_refs, *local_refs;\n \tint ret;\n \tint helper_status = 0;\n@@ -228,8 +229,6 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)\n \t\t\targs.verbose ? CONNECT_VERBOSE : 0);\n \t}\n \n-\tmemset(&extra_have, 0, sizeof(extra_have));\n-\n \tget_remote_heads(fd[0], NULL, 0, &remote_refs, REF_NORMAL, &extra_have);\n \n \ttransport_verify_remote_names(nr_refspecs, refspecs);\ndiff --git a/connect.c b/connect.c\nindex 06e88b0..48eec41 100644\n--- a/connect.c\n+++ b/connect.c\n@@ -8,6 +8,7 @@\n #include \"connect.h\"\n #include \"url.h\"\n #include \"string-list.h\"\n+#include \"sha1-array.h\"\n \n static char *server_capabilities;\n static const char *parse_feature_value(const char *, const char *, int *);\n@@ -45,13 +46,6 @@ int check_ref_type(const struct ref *ref, int flags)\n \treturn check_ref(ref->name, strlen(ref->name), flags);\n }\n \n-static void add_extra_have(struct extra_have_objects *extra, unsigned char *sha1)\n-{\n-\tALLOC_GROW(extra->array, extra->nr + 1, extra->alloc);\n-\thashcpy(&(extra->array[extra->nr][0]), sha1);\n-\textra->nr++;\n-}\n-\n static void die_initial_contact(int got_at_least_one_head)\n {\n \tif (got_at_least_one_head)\n@@ -122,7 +116,7 @@ static void annotate_refs_with_symref_info(struct ref *ref)\n */\n struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,\n \t\t\t struct ref **list, unsigned int flags,\n-\t\t\t struct extra_have_objects *extra_have)\n+\t\t\t struct sha1_array *extra_have)\n {\n \tstruct ref **orig_list = list;\n \tint got_at_least_one_head = 0;\n@@ -160,7 +154,7 @@ struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,\n \n \t\tif (extra_have &&\n \t\t name_len == 5 && !memcmp(\".have\", name, 5)) {\n-\t\t\tadd_extra_have(extra_have, old_sha1);\n+\t\t\tsha1_array_append(extra_have, old_sha1);\n \t\t\tcontinue;\n \t\t}\n \ndiff --git a/remote.h b/remote.h\nindex 131130a..984519b 100644\n--- a/remote.h\n+++ b/remote.h\n@@ -137,13 +137,10 @@ int check_ref_type(const struct ref *ref, int flags);\n */\n void free_refs(struct ref *ref);\n \n-struct extra_have_objects {\n-\tint nr, alloc;\n-\tunsigned char (*array)[20];\n-};\n+struct sha1_array;\n extern struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,\n \t\t\t\t struct ref **list, unsigned int flags,\n-\t\t\t\t struct extra_have_objects *);\n+\t\t\t\t struct sha1_array *extra_have);\n \n int resolve_remote_symref(struct ref *ref, struct ref *list);\n int ref_newer(const unsigned char *new_sha1, const unsigned char *old_sha1);\ndiff --git a/send-pack.c b/send-pack.c\nindex fab62e3..14005fa 100644\n--- a/send-pack.c\n+++ b/send-pack.c\n@@ -10,6 +10,7 @@\n #include \"quote.h\"\n #include \"transport.h\"\n #include \"version.h\"\n+#include \"sha1-array.h\"\n \n static int feed_object(const unsigned char *sha1, int fd, int negative)\n {\n@@ -28,7 +29,7 @@ static int feed_object(const unsigned char *sha1, int fd, int negative)\n /*\n * Make a pack stream and spit it out into file descriptor fd\n */\n-static int pack_objects(int fd, struct ref *refs, struct extra_have_objects *extra, struct send_pack_args *args)\n+static int pack_objects(int fd, struct ref *refs, struct sha1_array *extra, struct send_pack_args *args)\n {\n \t/*\n \t * The child becomes pack-objects --revs; we feed\n@@ -71,7 +72,7 @@ static int pack_objects(int fd, struct ref *refs, struct extra_have_objects *ext\n \t * parameters by writing to the pipe.\n \t */\n \tfor (i = 0; i < extra->nr; i++)\n-\t\tif (!feed_object(extra->array[i], po.in, 1))\n+\t\tif (!feed_object(extra->sha1[i], po.in, 1))\n \t\t\tbreak;\n \n \twhile (refs) {\n@@ -177,7 +178,7 @@ static int sideband_demux(int in, int out, void *data)\n int send_pack(struct send_pack_args *args,\n \t int fd[], struct child_process *conn,\n \t struct ref *remote_refs,\n-\t struct extra_have_objects *extra_have)\n+\t struct sha1_array *extra_have)\n {\n \tint in = fd[0];\n \tint out = fd[1];\ndiff --git a/send-pack.h b/send-pack.h\nindex 05d7ab1..8e84392 100644\n--- a/send-pack.h\n+++ b/send-pack.h\n@@ -16,6 +16,6 @@ struct send_pack_args {\n \n int send_pack(struct send_pack_args *args,\n \t int fd[], struct child_process *conn,\n-\t struct ref *remote_refs, struct extra_have_objects *extra_have);\n+\t struct ref *remote_refs, struct sha1_array *extra_have);\n \n #endif\ndiff --git a/transport.c b/transport.c\nindex 7202b77..12e46ad 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -14,6 +14,7 @@\n #include \"url.h\"\n #include \"submodule.h\"\n #include \"string-list.h\"\n+#include \"sha1-array.h\"\n \n /* rsync support */\n \n@@ -454,7 +455,7 @@ struct git_transport_data {\n \tstruct child_process *conn;\n \tint fd[2];\n \tunsigned got_remote_heads : 1;\n-\tstruct extra_have_objects extra_have;\n+\tstruct sha1_array extra_have;\n };\n \n static int set_git_option(struct git_transport_options *opts,\n-- \n1.8.5.1.25.g8667982\n"},{"id":"231587","messageId":"1386248575-10206-4-git-send-email-pclouds@gmail.com","threadId":"35400","inReplyTo":"1386248575-10206-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v4 03/28] send-pack: forbid pushing from a shallow repository","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-12-05T13:02:30Z","receivedAt":"2013-12-05T13:02:30Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"send-pack can send a pack with loose ends to the server. receive-pack\nbefore 6d4bb38 (fetch: verify we have everything we need before\nupdating our ref - 2011-09-01) does not detect this and keeps the pack\nanyway, which corrupts the repository, at least from fsck point of\nview.\n\nsend-pack will learn to safely push from a shallow repository later.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/send-pack.c | 3 +++\n 1 file changed, 3 insertions(+)\n\ndiff --git a/builtin/send-pack.c b/builtin/send-pack.c\nindex faaa603..961df04 100644\n--- a/builtin/send-pack.c\n+++ b/builtin/send-pack.c\n@@ -208,6 +208,9 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)\n \t (send_all && args.send_mirror))\n \t\tusage(send_pack_usage);\n \n+\tif (is_repository_shallow())\n+\t\tdie(\"attempt to push from a shallow repository\");\n+\n \tif (remote_name) {\n \t\tremote = remote_get(remote_name);\n \t\tif (!remote_has_url(remote, dest)) {\n-- \n1.8.5.1.25.g8667982\n"},{"id":"231588","messageId":"1386248575-10206-5-git-send-email-pclouds@gmail.com","threadId":"35400","inReplyTo":"1386248575-10206-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v4 04/28] clone: prevent --reference to a shallow repository","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-12-05T13:02:31Z","receivedAt":"2013-12-05T13:02:31Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"If we borrow objects from another repository, we should also pay\nattention to their $GIT_DIR/shallow (and even info/grafts). But\ncurrent alternates code does not.\n\nReject alternate repos that are shallow because we do not do it\nright. In future the alternate code may be updated to check\n$GIT_DIR/shallow properly so that this restriction could be lifted.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/clone.c | 6 ++++++\n 1 file changed, 6 insertions(+)\n\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex 874e0fd..900f564 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -252,6 +252,12 @@ static int add_one_reference(struct string_list_item *item, void *cb_data)\n \t\tdie(_(\"reference repository '%s' is not a local repository.\"),\n \t\t item->string);\n \n+\tif (!access(mkpath(\"%s/shallow\", ref_git), F_OK))\n+\t\tdie(_(\"reference repository '%s' is shallow\"), item->string);\n+\n+\tif (!access(mkpath(\"%s/info/grafts\", ref_git), F_OK))\n+\t\tdie(_(\"reference repository '%s' is grafted\"), item->string);\n+\n \tstrbuf_addf(&alternate, \"%s/objects\", ref_git);\n \tadd_to_alternates_file(alternate.buf);\n \tstrbuf_release(&alternate);\n-- \n1.8.5.1.25.g8667982\n"},{"id":"231589","messageId":"1386248575-10206-6-git-send-email-pclouds@gmail.com","threadId":"35400","inReplyTo":"1386248575-10206-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v4 05/28] Make the sender advertise shallow commits to the receiver","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-12-05T13:02:32Z","receivedAt":"2013-12-05T13:02:32Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"If either receive-pack or upload-pack is called on a shallow\nrepository, shallow commits (*) will be sent after the ref\nadvertisement (but before the packet flush), so that the receiver has\nthe full \"shape\" of the sender's commit graph. This will be needed for\nthe receiver to update its .git/shallow if necessary.\n\nThis breaks the protocol for all clients trying to push to a shallow\nrepo, or fetch from one. Which is basically the same end result as\ntoday's \"is_repository_shallow() && die()\" in receive-pack and\nupload-pack. New clients will be made aware of shallow upstream and\ncan make use of this information.\n\nThe sender must send all shallow commits that are sent in the\nfollowing pack. It may send more shallow commits than necessary.\n\nupload-pack for example may choose to advertise no shallow commits if\nit knows in advance that the pack it's going to send contains no\nshallow commits. But upload-pack is the server, so we choose the\ncheaper way, send full .git/shallow and let the client deal with it.\n\nSmart HTTP is not affected by this patch. Shallow support on\nsmart-http comes later separately.\n\n(*) A shallow commit is a commit that terminates the revision\n walker. It is usually put in .git/shallow in order to keep the\n revision walker from going out of bound because there is no\n guarantee that objects behind this commit is available.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n Documentation/technical/pack-protocol.txt | 3 +++\n builtin/receive-pack.c | 4 +++-\n commit.h | 1 +\n shallow.c | 15 +++++++++++++++\n upload-pack.c | 6 ++++--\n 5 files changed, 26 insertions(+), 3 deletions(-)\n\ndiff --git a/Documentation/technical/pack-protocol.txt b/Documentation/technical/pack-protocol.txt\nindex b898e97..eb8edd1 100644\n--- a/Documentation/technical/pack-protocol.txt\n+++ b/Documentation/technical/pack-protocol.txt\n@@ -161,6 +161,7 @@ MUST peel the ref if it's an annotated tag.\n \n ----\n advertised-refs = (no-refs / list-of-refs)\n+\t\t *shallow\n \t\t flush-pkt\n \n no-refs = PKT-LINE(zero-id SP \"capabilities^{}\"\n@@ -174,6 +175,8 @@ MUST peel the ref if it's an annotated tag.\n other-tip = obj-id SP refname LF\n other-peeled = obj-id SP refname \"^{}\" LF\n \n+ shallow = PKT-LINE(\"shallow\" SP obj-id)\n+\n capability-list = capability *(SP capability)\n capability = 1*(LC_ALPHA / DIGIT / \"-\" / \"_\")\n LC_ALPHA = %x61-7A\ndiff --git a/builtin/receive-pack.c b/builtin/receive-pack.c\nindex 67ce1ef..cc8c34f 100644\n--- a/builtin/receive-pack.c\n+++ b/builtin/receive-pack.c\n@@ -178,6 +178,8 @@ static void write_head_info(void)\n \tif (!sent_capabilities)\n \t\tshow_ref(\"capabilities^{}\", null_sha1);\n \n+\tadvertise_shallow_grafts(1);\n+\n \t/* EOF */\n \tpacket_flush(1);\n }\n@@ -998,7 +1000,7 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix)\n \tif (!enter_repo(dir, 0))\n \t\tdie(\"'%s' does not appear to be a git repository\", dir);\n \n-\tif (is_repository_shallow())\n+\tif (is_repository_shallow() && stateless_rpc)\n \t\tdie(\"attempt to push into a shallow repository\");\n \n \tgit_config(receive_pack_config, NULL);\ndiff --git a/commit.h b/commit.h\nindex bd841f4..a879526 100644\n--- a/commit.h\n+++ b/commit.h\n@@ -205,6 +205,7 @@ extern int write_shallow_commits(struct strbuf *out, int use_pack_protocol);\n extern void setup_alternate_shallow(struct lock_file *shallow_lock,\n \t\t\t\t const char **alternate_shallow_file);\n extern char *setup_temporary_shallow(void);\n+extern void advertise_shallow_grafts(int);\n \n int is_descendant_of(struct commit *, struct commit_list *);\n int in_merge_bases(struct commit *, struct commit *);\ndiff --git a/shallow.c b/shallow.c\nindex cdf37d6..9552512 100644\n--- a/shallow.c\n+++ b/shallow.c\n@@ -220,3 +220,18 @@ void setup_alternate_shallow(struct lock_file *shallow_lock,\n \t\t*alternate_shallow_file = \"\";\n \tstrbuf_release(&sb);\n }\n+\n+static int advertise_shallow_grafts_cb(const struct commit_graft *graft, void *cb)\n+{\n+\tint fd = *(int*)cb;\n+\tif (graft->nr_parent == -1)\n+\t\tpacket_write(fd, \"shallow %s\\n\", sha1_to_hex(graft->sha1));\n+\treturn 0;\n+}\n+\n+void advertise_shallow_grafts(int fd)\n+{\n+\tif (!is_repository_shallow())\n+\t\treturn;\n+\tfor_each_commit_graft(advertise_shallow_grafts_cb, &fd);\n+}\ndiff --git a/upload-pack.c b/upload-pack.c\nindex c989a73..38b2a29 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -758,6 +758,7 @@ static void upload_pack(void)\n \t\treset_timeout();\n \t\thead_ref_namespaced(send_ref, &symref);\n \t\tfor_each_namespaced_ref(send_ref, &symref);\n+\t\tadvertise_shallow_grafts(1);\n \t\tpacket_flush(1);\n \t} else {\n \t\thead_ref_namespaced(mark_our_ref, NULL);\n@@ -835,8 +836,9 @@ int main(int argc, char **argv)\n \n \tif (!enter_repo(dir, strict))\n \t\tdie(\"'%s' does not appear to be a git repository\", dir);\n-\tif (is_repository_shallow())\n-\t\tdie(\"attempt to fetch/clone from a shallow repository\");\n+\tif (is_repository_shallow() && stateless_rpc)\n+\t\tdie(\"attempt to push into a shallow repository\");\n+\n \tgit_config(upload_pack_config, NULL);\n \tupload_pack();\n \treturn 0;\n-- \n1.8.5.1.25.g8667982\n"},{"id":"231590","messageId":"1386248575-10206-7-git-send-email-pclouds@gmail.com","threadId":"35400","inReplyTo":"1386248575-10206-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v4 06/28] connect.c: teach get_remote_heads to parse \"shallow\" lines","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-12-05T13:02:33Z","receivedAt":"2013-12-05T13:02:33Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"No callers pass a non-empty pointer as shallow_points at this\nstage. As a result, all clients still refuse to talk to shallow\nrepository on the other end.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/fetch-pack.c | 2 +-\n builtin/send-pack.c | 2 +-\n connect.c | 12 +++++++++++-\n remote-curl.c | 2 +-\n remote.h | 3 ++-\n transport.c | 7 ++++---\n 6 files changed, 20 insertions(+), 8 deletions(-)\n\ndiff --git a/builtin/fetch-pack.c b/builtin/fetch-pack.c\nindex c8e8582..c1d918f 100644\n--- a/builtin/fetch-pack.c\n+++ b/builtin/fetch-pack.c\n@@ -150,7 +150,7 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)\n \t\t\t\t args.verbose ? CONNECT_VERBOSE : 0);\n \t}\n \n-\tget_remote_heads(fd[0], NULL, 0, &ref, 0, NULL);\n+\tget_remote_heads(fd[0], NULL, 0, &ref, 0, NULL, NULL);\n \n \tref = fetch_pack(&args, fd, conn, ref, dest,\n \t\t\t sought, nr_sought, pack_lockfile_ptr);\ndiff --git a/builtin/send-pack.c b/builtin/send-pack.c\nindex 961df04..62cc4d3 100644\n--- a/builtin/send-pack.c\n+++ b/builtin/send-pack.c\n@@ -232,7 +232,7 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)\n \t\t\targs.verbose ? CONNECT_VERBOSE : 0);\n \t}\n \n-\tget_remote_heads(fd[0], NULL, 0, &remote_refs, REF_NORMAL, &extra_have);\n+\tget_remote_heads(fd[0], NULL, 0, &remote_refs, REF_NORMAL, &extra_have, NULL);\n \n \ttransport_verify_remote_names(nr_refspecs, refspecs);\n \ndiff --git a/connect.c b/connect.c\nindex 48eec41..efadd3c 100644\n--- a/connect.c\n+++ b/connect.c\n@@ -116,7 +116,8 @@ static void annotate_refs_with_symref_info(struct ref *ref)\n */\n struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,\n \t\t\t struct ref **list, unsigned int flags,\n-\t\t\t struct sha1_array *extra_have)\n+\t\t\t struct sha1_array *extra_have,\n+\t\t\t struct sha1_array *shallow_points)\n {\n \tstruct ref **orig_list = list;\n \tint got_at_least_one_head = 0;\n@@ -142,6 +143,15 @@ struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,\n \t\tif (len > 4 && !prefixcmp(buffer, \"ERR \"))\n \t\t\tdie(\"remote error: %s\", buffer + 4);\n \n+\t\tif (len == 48 && !prefixcmp(buffer, \"shallow \")) {\n+\t\t\tif (get_sha1_hex(buffer + 8, old_sha1))\n+\t\t\t\tdie(\"protocol error: expected shallow sha-1, got '%s'\", buffer + 8);\n+\t\t\tif (!shallow_points)\n+\t\t\t\tdie(\"repository on the other end cannot be shallow\");\n+\t\t\tsha1_array_append(shallow_points, old_sha1);\n+\t\t\tcontinue;\n+\t\t}\n+\n \t\tif (len < 42 || get_sha1_hex(buffer, old_sha1) || buffer[40] != ' ')\n \t\t\tdie(\"protocol error: expected sha/ref, got '%s'\", buffer);\n \t\tname = buffer + 41;\ndiff --git a/remote-curl.c b/remote-curl.c\nindex c9b891a..222210f 100644\n--- a/remote-curl.c\n+++ b/remote-curl.c\n@@ -107,7 +107,7 @@ static struct ref *parse_git_refs(struct discovery *heads, int for_push)\n {\n \tstruct ref *list = NULL;\n \tget_remote_heads(-1, heads->buf, heads->len, &list,\n-\t\t\t for_push ? REF_NORMAL : 0, NULL);\n+\t\t\t for_push ? REF_NORMAL : 0, NULL, NULL);\n \treturn list;\n }\n \ndiff --git a/remote.h b/remote.h\nindex 984519b..5d217d5 100644\n--- a/remote.h\n+++ b/remote.h\n@@ -140,7 +140,8 @@ void free_refs(struct ref *ref);\n struct sha1_array;\n extern struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,\n \t\t\t\t struct ref **list, unsigned int flags,\n-\t\t\t\t struct sha1_array *extra_have);\n+\t\t\t\t struct sha1_array *extra_have,\n+\t\t\t\t struct sha1_array *shallow);\n \n int resolve_remote_symref(struct ref *ref, struct ref *list);\n int ref_newer(const unsigned char *new_sha1, const unsigned char *old_sha1);\ndiff --git a/transport.c b/transport.c\nindex 12e46ad..90453df 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -512,7 +512,7 @@ static struct ref *get_refs_via_connect(struct transport *transport, int for_pus\n \n \tconnect_setup(transport, for_push, 0);\n \tget_remote_heads(data->fd[0], NULL, 0, &refs,\n-\t\t\t for_push ? REF_NORMAL : 0, &data->extra_have);\n+\t\t\t for_push ? REF_NORMAL : 0, &data->extra_have, NULL);\n \tdata->got_remote_heads = 1;\n \n \treturn refs;\n@@ -542,7 +542,8 @@ static int fetch_refs_via_pack(struct transport *transport,\n \n \tif (!data->got_remote_heads) {\n \t\tconnect_setup(transport, 0, 0);\n-\t\tget_remote_heads(data->fd[0], NULL, 0, &refs_tmp, 0, NULL);\n+\t\tget_remote_heads(data->fd[0], NULL, 0, &refs_tmp, 0,\n+\t\t\t\t NULL, NULL);\n \t\tdata->got_remote_heads = 1;\n \t}\n \n@@ -806,7 +807,7 @@ static int git_transport_push(struct transport *transport, struct ref *remote_re\n \t\tstruct ref *tmp_refs;\n \t\tconnect_setup(transport, 1, 0);\n \n-\t\tget_remote_heads(data->fd[0], NULL, 0, &tmp_refs, REF_NORMAL, NULL);\n+\t\tget_remote_heads(data->fd[0], NULL, 0, &tmp_refs, REF_NORMAL, NULL, NULL);\n \t\tdata->got_remote_heads = 1;\n \t}\n \n-- \n1.8.5.1.25.g8667982\n"},{"id":"231591","messageId":"1386248575-10206-8-git-send-email-pclouds@gmail.com","threadId":"35400","inReplyTo":"1386248575-10206-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v4 07/28] shallow.c: extend setup_*_shallow() to accept extra shallow commits","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-12-05T13:02:34Z","receivedAt":"2013-12-05T13:02:34Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Signed-off-by: Nguyễn Thái Ngọc Duy \n---\n commit.h | 8 +++++---\n fetch-pack.c | 5 +++--\n shallow.c | 20 +++++++++++++++-----\n upload-pack.c | 2 +-\n 4 files changed, 24 insertions(+), 11 deletions(-)\n\ndiff --git a/commit.h b/commit.h\nindex a879526..1faf717 100644\n--- a/commit.h\n+++ b/commit.h\n@@ -201,10 +201,12 @@ extern struct commit_list *get_shallow_commits(struct object_array *heads,\n \t\tint depth, int shallow_flag, int not_shallow_flag);\n extern void check_shallow_file_for_update(void);\n extern void set_alternate_shallow_file(const char *path);\n-extern int write_shallow_commits(struct strbuf *out, int use_pack_protocol);\n+extern int write_shallow_commits(struct strbuf *out, int use_pack_protocol,\n+\t\t\t\t const struct sha1_array *extra);\n extern void setup_alternate_shallow(struct lock_file *shallow_lock,\n-\t\t\t\t const char **alternate_shallow_file);\n-extern char *setup_temporary_shallow(void);\n+\t\t\t\t const char **alternate_shallow_file,\n+\t\t\t\t const struct sha1_array *extra);\n+extern char *setup_temporary_shallow(const struct sha1_array *extra);\n extern void advertise_shallow_grafts(int);\n \n int is_descendant_of(struct commit *, struct commit_list *);\ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex 1042448..0e7483e 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -311,7 +311,7 @@ static int find_common(struct fetch_pack_args *args,\n \t}\n \n \tif (is_repository_shallow())\n-\t\twrite_shallow_commits(&req_buf, 1);\n+\t\twrite_shallow_commits(&req_buf, 1, NULL);\n \tif (args->depth > 0)\n \t\tpacket_buf_write(&req_buf, \"deepen %d\", args->depth);\n \tpacket_buf_flush(&req_buf);\n@@ -850,7 +850,8 @@ static struct ref *do_fetch_pack(struct fetch_pack_args *args,\n \tif (args->stateless_rpc)\n \t\tpacket_flush(fd[1]);\n \tif (args->depth > 0)\n-\t\tsetup_alternate_shallow(&shallow_lock, &alternate_shallow_file);\n+\t\tsetup_alternate_shallow(&shallow_lock, &alternate_shallow_file,\n+\t\t\t\t\tNULL);\n \telse\n \t\talternate_shallow_file = NULL;\n \tif (get_pack(args, fd, pack_lockfile))\ndiff --git a/shallow.c b/shallow.c\nindex 9552512..f9d1633 100644\n--- a/shallow.c\n+++ b/shallow.c\n@@ -165,22 +165,31 @@ static int write_one_shallow(const struct commit_graft *graft, void *cb_data)\n \treturn 0;\n }\n \n-int write_shallow_commits(struct strbuf *out, int use_pack_protocol)\n+int write_shallow_commits(struct strbuf *out, int use_pack_protocol,\n+\t\t\t const struct sha1_array *extra)\n {\n \tstruct write_shallow_data data;\n+\tint i;\n \tdata.out = out;\n \tdata.use_pack_protocol = use_pack_protocol;\n \tdata.count = 0;\n \tfor_each_commit_graft(write_one_shallow, &data);\n+\tif (!extra)\n+\t\treturn data.count;\n+\tfor (i = 0; i < extra->nr; i++) {\n+\t\tstrbuf_addstr(out, sha1_to_hex(extra->sha1[i]));\n+\t\tstrbuf_addch(out, '\\n');\n+\t\tdata.count++;\n+\t}\n \treturn data.count;\n }\n \n-char *setup_temporary_shallow(void)\n+char *setup_temporary_shallow(const struct sha1_array *extra)\n {\n \tstruct strbuf sb = STRBUF_INIT;\n \tint fd;\n \n-\tif (write_shallow_commits(&sb, 0)) {\n+\tif (write_shallow_commits(&sb, 0, extra)) {\n \t\tstruct strbuf path = STRBUF_INIT;\n \t\tstrbuf_addstr(&path, git_path(\"shallow_XXXXXX\"));\n \t\tfd = xmkstemp(path.buf);\n@@ -199,7 +208,8 @@ char *setup_temporary_shallow(void)\n }\n \n void setup_alternate_shallow(struct lock_file *shallow_lock,\n-\t\t\t const char **alternate_shallow_file)\n+\t\t\t const char **alternate_shallow_file,\n+\t\t\t const struct sha1_array *extra)\n {\n \tstruct strbuf sb = STRBUF_INIT;\n \tint fd;\n@@ -207,7 +217,7 @@ void setup_alternate_shallow(struct lock_file *shallow_lock,\n \tcheck_shallow_file_for_update();\n \tfd = hold_lock_file_for_update(shallow_lock, git_path(\"shallow\"),\n \t\t\t\t LOCK_DIE_ON_ERROR);\n-\tif (write_shallow_commits(&sb, 0)) {\n+\tif (write_shallow_commits(&sb, 0, extra)) {\n \t\tif (write_in_full(fd, sb.buf, sb.len) != sb.len)\n \t\t\tdie_errno(\"failed to write to %s\",\n \t\t\t\t shallow_lock->filename);\ndiff --git a/upload-pack.c b/upload-pack.c\nindex 38b2a29..f082f06 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -84,7 +84,7 @@ static void create_pack_file(void)\n \tchar *shallow_file = NULL;\n \n \tif (shallow_nr) {\n-\t\tshallow_file = setup_temporary_shallow();\n+\t\tshallow_file = setup_temporary_shallow(NULL);\n \t\targv[arg++] = \"--shallow-file\";\n \t\targv[arg++] = shallow_file;\n \t}\n-- \n1.8.5.1.25.g8667982\n"},{"id":"231592","messageId":"1386248575-10206-9-git-send-email-pclouds@gmail.com","threadId":"35400","inReplyTo":"1386248575-10206-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v4 08/28] shallow.c: the 8 steps to select new commits for .git/shallow","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-12-05T13:02:35Z","receivedAt":"2013-12-05T13:02:35Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Suppose a fetch or push is requested between two shallow repositories\n(with no history deepening or shortening). A pack that contains\nnecessary objects is transferred over together with .git/shallow of\nthe sender. The receiver has to determine whether it needs to update\n.git/shallow if new refs needs new shallow comits.\n\nThe rule here is avoid updating .git/shallow by default. But we don't\nwant to waste the received pack. If the pack contains two refs, one\nneeds new shallow commits installed in .git/shallow and one does not,\nwe keep the latter and reject/warn about the former.\n\nEven if .git/shallow update is allowed, we only add shallow commits\nstrictly necessary for the former ref (remember the sender can send\nmore shallow commits than necessary) and pay attention not to\naccidentally cut the receiver history short (no history shortening is\nasked for)\n\nSo the steps to figure out what ref need what new shallow commits are:\n\n1. Split the sender shallow commit list into \"ours\" and \"theirs\" list\n by has_sha1_file. Those that exist in current repo in \"ours\", the\n remaining in \"theirs\".\n\n2. Check the receiver .git/shallow, remove from \"ours\" the ones that\n also exist in .git/shallow.\n\n3. Fetch the new pack. Either install or unpack it.\n\n4. Do has_sha1_file on \"theirs\" list again. Drop the ones that fail\n has_sha1_file. Obviously the new pack does not need them.\n\n5. If the pack is kept, remove from \"ours\" the ones that do not exist\n in the new pack.\n\n6. Walk the new refs to answer the question \"what shallow commits,\n both ours and theirs, are required in .git/shallow in order to add\n this ref?\". Shallow commits not associated to any refs are removed\n from their respective list.\n\n7. (*) Check reachability (from the current refs) of all remaining\n commits in \"ours\". Those reachable are removed. We do not want to\n cut any part of our (reachable) history. We only check up\n commits. True reachability test is done by\n check_everything_connected() at the end as usual.\n\n8. Combine the final \"ours\" and \"theirs\" and add them all to\n .git/shallow. Install new refs. The case where some hook rejects\n some refs on a push is explained in more detail in the push\n patches.\n\nOf these steps, #6 and #7 are expensive. Both require walking through\nsome commits, or in the worst case all commits. And we rather avoid\nthem in at least common case, where the transferred pack does not\ncontain any shallow commits that the sender advertises. Let's look at\neach scenario:\n\n1) the sender has longer history than the receiver\n\n All shallow commits from the sender will be put into \"theirs\" list\n at step 1 because none of them exists in current repo. In the\n common case, \"theirs\" becomes empty at step 4 and exit early.\n\n2) the sender has shorter history than the receiver\n\n All shallow commits from the sender are likely in \"ours\" list at\n step 1. In the common case, if the new pack is kept, we could empty\n \"ours\" and exit early at step 5.\n\n If the pack is not kept, we hit the expensive step 6 then exit\n after \"ours\" is emptied. There'll be only a handful of objects to\n walk in fast-forward case. If it's forced update, we may need to\n walk to the bottom.\n\n3) the sender has same .git/shallow as the receiver\n\n This is similar to case 2 except that \"ours\" should be emptied at\n step 2 and exit early.\n\nA fetch after \"clone --depth=X\" is case 1. A fetch after \"clone\" (from\na shallow repo) is case 3. Luckily they're cheap for the common case.\n\nA push from \"clone --depth=X\" falls into case 2, which is expensive.\nSome more work may be done at the sender/client side to avoid more\nwork on the server side: if the transferred pack does not contain any\nshallow commits, send-pack should not send any shallow commits to the\nreceive-pack, effectively turning it into a normal push and avoid all\nsteps.\n\nThis patch implements all steps except #3, already handled by\nfetch-pack and receive-pack, #6 and #7, which has their own patch due\nto their size.\n\n(*) in previous versions step 7 was put before step 3. I reorder it so\n that the common case that keeps the pack does not need to walk\n commits at all. In future if we implement faster commit\n reachability check (maybe with the help of pack bitmaps or commit\n cache), step 7 could become cheap and be moved up before 6 again.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n cache.h | 2 ++\n commit.h | 15 +++++++++++++\n shallow.c | 72 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n trace.c | 2 +-\n 4 files changed, 90 insertions(+), 1 deletion(-)\n\ndiff --git a/cache.h b/cache.h\nindex ce377e1..55dd4e3 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1236,6 +1236,8 @@ __attribute__((format (printf, 2, 3)))\n extern void trace_argv_printf(const char **argv, const char *format, ...);\n extern void trace_repo_setup(const char *prefix);\n extern int trace_want(const char *key);\n+__attribute__((format (printf, 2, 3)))\n+extern void trace_printf_key(const char *key, const char *fmt, ...);\n extern void trace_strbuf(const char *key, const struct strbuf *buf);\n \n void packet_trace_identity(const char *prog);\ndiff --git a/commit.h b/commit.h\nindex 1faf717..9ead93b 100644\n--- a/commit.h\n+++ b/commit.h\n@@ -193,6 +193,8 @@ extern struct commit_list *get_octopus_merge_bases(struct commit_list *in);\n /* largest positive number a signed 32-bit integer can contain */\n #define INFINITE_DEPTH 0x7fffffff\n \n+struct sha1_array;\n+struct ref;\n extern int register_shallow(const unsigned char *sha1);\n extern int unregister_shallow(const unsigned char *sha1);\n extern int for_each_commit_graft(each_commit_graft_fn, void *);\n@@ -209,6 +211,19 @@ extern void setup_alternate_shallow(struct lock_file *shallow_lock,\n extern char *setup_temporary_shallow(const struct sha1_array *extra);\n extern void advertise_shallow_grafts(int);\n \n+struct shallow_info {\n+\tstruct sha1_array *shallow;\n+\tint *ours, nr_ours;\n+\tint *theirs, nr_theirs;\n+\tstruct sha1_array *ref;\n+};\n+\n+extern void prepare_shallow_info(struct shallow_info *, struct sha1_array *);\n+extern void clear_shallow_info(struct shallow_info *);\n+extern void remove_nonexistent_theirs_shallow(struct shallow_info *);\n+extern void remove_nonexistent_ours_in_pack(struct shallow_info *,\n+\t\t\t\t\t struct packed_git *);\n+\n int is_descendant_of(struct commit *, struct commit_list *);\n int in_merge_bases(struct commit *, struct commit *);\n int in_merge_bases_many(struct commit *, int, struct commit **);\ndiff --git a/shallow.c b/shallow.c\nindex f9d1633..a6547ca 100644\n--- a/shallow.c\n+++ b/shallow.c\n@@ -2,6 +2,12 @@\n #include \"commit.h\"\n #include \"tag.h\"\n #include \"pkt-line.h\"\n+#include \"remote.h\"\n+#include \"refs.h\"\n+#include \"sha1-array.h\"\n+#include \"diff.h\"\n+#include \"revision.h\"\n+#include \"commit-slab.h\"\n \n static int is_shallow = -1;\n static struct stat shallow_stat;\n@@ -245,3 +251,69 @@ void advertise_shallow_grafts(int fd)\n \t\treturn;\n \tfor_each_commit_graft(advertise_shallow_grafts_cb, &fd);\n }\n+\n+#define TRACE_KEY \"GIT_TRACE_SHALLOW\"\n+\n+/*\n+ * Step 1, split sender shallow commits into \"ours\" and \"theirs\"\n+ * Step 2, clean \"ours\" based on .git/shallow\n+ */\n+void prepare_shallow_info(struct shallow_info *info, struct sha1_array *sa)\n+{\n+\tint i;\n+\ttrace_printf_key(TRACE_KEY, \"shallow: prepare_shallow_info\\n\");\n+\tmemset(info, 0, sizeof(*info));\n+\tinfo->shallow = sa;\n+\tif (!sa)\n+\t\treturn;\n+\tinfo->ours = xmalloc(sizeof(*info->ours) * sa->nr);\n+\tinfo->theirs = xmalloc(sizeof(*info->theirs) * sa->nr);\n+\tfor (i = 0; i < sa->nr; i++) {\n+\t\tif (has_sha1_file(sa->sha1[i])) {\n+\t\t\tstruct commit_graft *graft;\n+\t\t\tgraft = lookup_commit_graft(sa->sha1[i]);\n+\t\t\tif (graft && graft->nr_parent < 0)\n+\t\t\t\tcontinue;\n+\t\t\tinfo->ours[info->nr_ours++] = i;\n+\t\t} else\n+\t\t\tinfo->theirs[info->nr_theirs++] = i;\n+\t}\n+}\n+\n+void clear_shallow_info(struct shallow_info *info)\n+{\n+\tfree(info->ours);\n+\tfree(info->theirs);\n+}\n+\n+/* Step 4, remove non-existent ones in \"theirs\" after getting the pack */\n+\n+void remove_nonexistent_theirs_shallow(struct shallow_info *info)\n+{\n+\tunsigned char (*sha1)[20] = info->shallow->sha1;\n+\tint i, dst;\n+\ttrace_printf_key(TRACE_KEY, \"shallow: remove_nonexistent_theirs_shallow\\n\");\n+\tfor (i = dst = 0; i < info->nr_theirs; i++) {\n+\t\tif (i != dst)\n+\t\t\tinfo->theirs[dst] = info->theirs[i];\n+\t\tif (has_sha1_file(sha1[info->theirs[i]]))\n+\t\t\tdst++;\n+\t}\n+\tinfo->nr_theirs = dst;\n+}\n+\n+/* Step 5, remove non-existent ones in \"ours\" in the pack */\n+void remove_nonexistent_ours_in_pack(struct shallow_info *info,\n+\t\t\t\t struct packed_git *p)\n+{\n+\tunsigned char (*sha1)[20] = info->shallow->sha1;\n+\tint i, dst;\n+\ttrace_printf_key(TRACE_KEY, \"shallow: remove_nonexistent_ours_in_pack\\n\");\n+\tfor (i = dst = 0; i < info->nr_ours; i++) {\n+\t\tif (i != dst)\n+\t\t\tinfo->ours[dst] = info->ours[i];\n+\t\tif (find_pack_entry_one(sha1[info->ours[i]], p))\n+\t\t\tdst++;\n+\t}\n+\tinfo->nr_ours = dst;\n+}\ndiff --git a/trace.c b/trace.c\nindex 3d744d1..08180a9 100644\n--- a/trace.c\n+++ b/trace.c\n@@ -76,7 +76,7 @@ static void trace_vprintf(const char *key, const char *fmt, va_list ap)\n }\n \n __attribute__((format (printf, 2, 3)))\n-static void trace_printf_key(const char *key, const char *fmt, ...)\n+void trace_printf_key(const char *key, const char *fmt, ...)\n {\n \tva_list ap;\n \tva_start(ap, fmt);\n-- \n1.8.5.1.25.g8667982\n"},{"id":"231593","messageId":"1386248575-10206-10-git-send-email-pclouds@gmail.com","threadId":"35400","inReplyTo":"1386248575-10206-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v4 09/28] shallow.c: steps 6 and 7 to select new commits for .git/shallow","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-12-05T13:02:36Z","receivedAt":"2013-12-05T13:02:36Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Signed-off-by: Nguyễn Thái Ngọc Duy \n---\n commit.h | 3 +\n shallow.c | 295 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 298 insertions(+)\n\ndiff --git a/commit.h b/commit.h\nindex 9ead93b..69bca3e 100644\n--- a/commit.h\n+++ b/commit.h\n@@ -223,6 +223,9 @@ extern void clear_shallow_info(struct shallow_info *);\n extern void remove_nonexistent_theirs_shallow(struct shallow_info *);\n extern void remove_nonexistent_ours_in_pack(struct shallow_info *,\n \t\t\t\t\t struct packed_git *);\n+extern void assign_shallow_commits_to_refs(struct shallow_info *info,\n+\t\t\t\t\t uint32_t **used,\n+\t\t\t\t\t int *ref_status);\n \n int is_descendant_of(struct commit *, struct commit_list *);\n int in_merge_bases(struct commit *, struct commit *);\ndiff --git a/shallow.c b/shallow.c\nindex a6547ca..975d699 100644\n--- a/shallow.c\n+++ b/shallow.c\n@@ -317,3 +317,298 @@ void remove_nonexistent_ours_in_pack(struct shallow_info *info,\n \t}\n \tinfo->nr_ours = dst;\n }\n+\n+define_commit_slab(ref_bitmap, uint32_t *);\n+\n+struct paint_info {\n+\tstruct ref_bitmap ref_bitmap;\n+\tunsigned nr_bits;\n+\tchar **slab;\n+\tchar *free, *end;\n+\tunsigned slab_count;\n+};\n+\n+static uint32_t *paint_alloc(struct paint_info *info)\n+{\n+\tunsigned nr = (info->nr_bits + 31) / 32;\n+\tunsigned size = nr * sizeof(uint32_t);\n+\tvoid *p;\n+\tif (!info->slab_count || info->free + size > info->end) {\n+\t\tinfo->slab_count++;\n+\t\tinfo->slab = xrealloc(info->slab,\n+\t\t\t\t info->slab_count * sizeof(*info->slab));\n+\t\tinfo->free = xmalloc(COMMIT_SLAB_SIZE);\n+\t\tinfo->slab[info->slab_count - 1] = info->free;\n+\t\tinfo->end = info->free + COMMIT_SLAB_SIZE;\n+\t}\n+\tp = info->free;\n+\tinfo->free += size;\n+\treturn p;\n+}\n+\n+/*\n+ * Given a commit SHA-1, walk down to parents until either SEEN,\n+ * UNINTERESTING or BOTTOM is hit. Set the id-th bit in ref_bitmap for\n+ * all walked commits.\n+ */\n+static void paint_down(struct paint_info *info, const unsigned char *sha1,\n+\t\t int id)\n+{\n+\tunsigned int i, nr;\n+\tstruct commit_list *head = NULL;\n+\tint bitmap_nr = (info->nr_bits + 31) / 32;\n+\tint bitmap_size = bitmap_nr * sizeof(uint32_t);\n+\tuint32_t *tmp = xmalloc(bitmap_size); /* to be freed before return */\n+\tuint32_t *bitmap = paint_alloc(info);\n+\tstruct commit *c = lookup_commit_reference_gently(sha1, 1);\n+\tif (!c)\n+\t\treturn;\n+\tmemset(bitmap, 0, bitmap_size);\n+\tbitmap[id / 32] |= (1 << (id % 32));\n+\tcommit_list_insert(c, &head);\n+\twhile (head) {\n+\t\tstruct commit_list *p;\n+\t\tstruct commit *c = head->item;\n+\t\tuint32_t **refs = ref_bitmap_at(&info->ref_bitmap, c);\n+\n+\t\tp = head;\n+\t\thead = head->next;\n+\t\tfree(p);\n+\n+\t\t/* XXX check \"UNINTERESTING\" from pack bitmaps if available */\n+\t\tif (c->object.flags & (SEEN | UNINTERESTING))\n+\t\t\tcontinue;\n+\t\telse\n+\t\t\tc->object.flags |= SEEN;\n+\n+\t\tif (*refs == NULL)\n+\t\t\t*refs = bitmap;\n+\t\telse {\n+\t\t\tmemcpy(tmp, *refs, bitmap_size);\n+\t\t\tfor (i = 0; i < bitmap_nr; i++)\n+\t\t\t\ttmp[i] |= bitmap[i];\n+\t\t\tif (memcmp(tmp, *refs, bitmap_size)) {\n+\t\t\t\t*refs = paint_alloc(info);\n+\t\t\t\tmemcpy(*refs, tmp, bitmap_size);\n+\t\t\t}\n+\t\t}\n+\n+\t\tif (c->object.flags & BOTTOM)\n+\t\t\tcontinue;\n+\n+\t\tif (parse_commit(c))\n+\t\t\tdie(\"unable to parse commit %s\",\n+\t\t\t sha1_to_hex(c->object.sha1));\n+\n+\t\tfor (p = c->parents; p; p = p->next) {\n+\t\t\tuint32_t **p_refs = ref_bitmap_at(&info->ref_bitmap,\n+\t\t\t\t\t\t\t p->item);\n+\t\t\tif (p->item->object.flags & SEEN)\n+\t\t\t\tcontinue;\n+\t\t\tif (*p_refs == NULL || *p_refs == *refs)\n+\t\t\t\t*p_refs = *refs;\n+\t\t\tcommit_list_insert(p->item, &head);\n+\t\t}\n+\t}\n+\n+\tnr = get_max_object_index();\n+\tfor (i = 0; i < nr; i++) {\n+\t\tstruct object *o = get_indexed_object(i);\n+\t\tif (o && o->type == OBJ_COMMIT) {\n+\t\t\to->flags &= ~SEEN;\n+\t\t}\n+\t}\n+\n+\tfree(tmp);\n+}\n+\n+static int mark_uninteresting(const char *refname,\n+\t\t\t const unsigned char *sha1,\n+\t\t\t int flags, void *cb_data)\n+{\n+\tstruct commit *commit = lookup_commit_reference_gently(sha1, 1);\n+\tif (!commit)\n+\t\treturn 0;\n+\tcommit->object.flags |= UNINTERESTING;\n+\tmark_parents_uninteresting(commit);\n+\treturn 0;\n+}\n+\n+static void post_assign_shallow(struct shallow_info *info,\n+\t\t\t\tstruct ref_bitmap *ref_bitmap,\n+\t\t\t\tint *ref_status);\n+/*\n+ * Step 6(+7), associate shallow commits with new refs\n+ *\n+ * info->ref must be initialized before calling this function.\n+ *\n+ * If used is not NULL, it's an array of info->shallow->nr\n+ * bitmaps. The n-th bit set in the m-th bitmap if ref[n] needs the\n+ * m-th shallow commit from info->shallow.\n+ *\n+ * If used is NULL, \"ours\" and \"theirs\" are updated. And if ref_status\n+ * is not NULL it's an array of ref->nr ints. ref_status[i] is true if\n+ * the ref needs some shallow commits from either info->ours or\n+ * info->theirs.\n+ */\n+void assign_shallow_commits_to_refs(struct shallow_info *info,\n+\t\t\t\t uint32_t **used, int *ref_status)\n+{\n+\tunsigned char (*sha1)[20] = info->shallow->sha1;\n+\tstruct sha1_array *ref = info->ref;\n+\tunsigned int i, nr;\n+\tint *shallow, nr_shallow = 0;\n+\tstruct paint_info pi;\n+\n+\ttrace_printf_key(TRACE_KEY, \"shallow: assign_shallow_commits_to_refs\\n\");\n+\tshallow = xmalloc(sizeof(*shallow) * (info->nr_ours + info->nr_theirs));\n+\tfor (i = 0; i < info->nr_ours; i++)\n+\t\tshallow[nr_shallow++] = info->ours[i];\n+\tfor (i = 0; i < info->nr_theirs; i++)\n+\t\tshallow[nr_shallow++] = info->theirs[i];\n+\n+\t/*\n+\t * Prepare the commit graph to track what refs can reach what\n+\t * (new) shallow commits.\n+\t */\n+\tnr = get_max_object_index();\n+\tfor (i = 0; i < nr; i++) {\n+\t\tstruct object *o = get_indexed_object(i);\n+\t\tif (!o || o->type != OBJ_COMMIT)\n+\t\t\tcontinue;\n+\n+\t\to->flags &= ~(UNINTERESTING | BOTTOM | SEEN);\n+\t}\n+\n+\tmemset(&pi, 0, sizeof(pi));\n+\tinit_ref_bitmap(&pi.ref_bitmap);\n+\tpi.nr_bits = ref->nr;\n+\n+\t/*\n+\t * \"--not --all\" to cut short the traversal if new refs\n+\t * connect to old refs. If not (e.g. force ref updates) it'll\n+\t * have to go down to the current shallow commits.\n+\t */\n+\thead_ref(mark_uninteresting, NULL);\n+\tfor_each_ref(mark_uninteresting, NULL);\n+\n+\t/* Mark potential bottoms so we won't go out of bound */\n+\tfor (i = 0; i < nr_shallow; i++) {\n+\t\tstruct commit *c = lookup_commit(sha1[shallow[i]]);\n+\t\tc->object.flags |= BOTTOM;\n+\t}\n+\n+\tfor (i = 0; i < ref->nr; i++)\n+\t\tpaint_down(&pi, ref->sha1[i], i);\n+\n+\tif (used) {\n+\t\tint bitmap_size = ((pi.nr_bits + 31) / 32) * sizeof(uint32_t);\n+\t\tmemset(used, 0, sizeof(*used) * info->shallow->nr);\n+\t\tfor (i = 0; i < nr_shallow; i++) {\n+\t\t\tconst struct commit *c = lookup_commit(sha1[shallow[i]]);\n+\t\t\tuint32_t **map = ref_bitmap_at(&pi.ref_bitmap, c);\n+\t\t\tif (*map)\n+\t\t\t\tused[shallow[i]] = xmemdupz(*map, bitmap_size);\n+\t\t}\n+\t\t/*\n+\t\t * unreachable shallow commits are not removed from\n+\t\t * \"ours\" and \"theirs\". The user is supposed to run\n+\t\t * step 7 on every ref separately and not trust \"ours\"\n+\t\t * and \"theirs\" any more.\n+\t\t */\n+\t} else\n+\t\tpost_assign_shallow(info, &pi.ref_bitmap, ref_status);\n+\n+\tclear_ref_bitmap(&pi.ref_bitmap);\n+\tfor (i = 0; i < pi.slab_count; i++)\n+\t\tfree(pi.slab[i]);\n+\tfree(pi.slab);\n+\tfree(shallow);\n+}\n+\n+struct commit_array {\n+\tstruct commit **commits;\n+\tint nr, alloc;\n+};\n+\n+static int add_ref(const char *refname,\n+\t\t const unsigned char *sha1, int flags, void *cb_data)\n+{\n+\tstruct commit_array *ca = cb_data;\n+\tALLOC_GROW(ca->commits, ca->nr + 1, ca->alloc);\n+\tca->commits[ca->nr] = lookup_commit_reference_gently(sha1, 1);\n+\tif (ca->commits[ca->nr])\n+\t\tca->nr++;\n+\treturn 0;\n+}\n+\n+static void update_refstatus(int *ref_status, int nr, uint32_t *bitmap)\n+{\n+\tint i;\n+\tif (!ref_status)\n+\t\treturn;\n+\tfor (i = 0; i < nr; i++)\n+\t\tif (bitmap[i / 32] & (1 << (i % 32)))\n+\t\t\tref_status[i]++;\n+}\n+\n+/*\n+ * Step 7, reachability test on \"ours\" at commit level\n+ */\n+static void post_assign_shallow(struct shallow_info *info,\n+\t\t\t\tstruct ref_bitmap *ref_bitmap,\n+\t\t\t\tint *ref_status)\n+{\n+\tunsigned char (*sha1)[20] = info->shallow->sha1;\n+\tstruct commit *c;\n+\tuint32_t **bitmap;\n+\tint dst, i, j;\n+\tint bitmap_nr = (info->ref->nr + 31) / 32;\n+\tstruct commit_array ca;\n+\n+\ttrace_printf_key(TRACE_KEY, \"shallow: post_assign_shallow\\n\");\n+\tif (ref_status)\n+\t\tmemset(ref_status, 0, sizeof(*ref_status) * info->ref->nr);\n+\n+\t/* Remove unreachable shallow commits from \"theirs\" */\n+\tfor (i = dst = 0; i < info->nr_theirs; i++) {\n+\t\tif (i != dst)\n+\t\t\tinfo->theirs[dst] = info->theirs[i];\n+\t\tc = lookup_commit(sha1[info->theirs[i]]);\n+\t\tbitmap = ref_bitmap_at(ref_bitmap, c);\n+\t\tif (!*bitmap)\n+\t\t\tcontinue;\n+\t\tfor (j = 0; j < bitmap_nr; j++)\n+\t\t\tif (bitmap[0][j]) {\n+\t\t\t\tupdate_refstatus(ref_status, info->ref->nr, *bitmap);\n+\t\t\t\tdst++;\n+\t\t\t\tbreak;\n+\t\t\t}\n+\t}\n+\tinfo->nr_theirs = dst;\n+\n+\tmemset(&ca, 0, sizeof(ca));\n+\thead_ref(add_ref, &ca);\n+\tfor_each_ref(add_ref, &ca);\n+\n+\t/* Remove unreachable shallow commits from \"ours\" */\n+\tfor (i = dst = 0; i < info->nr_ours; i++) {\n+\t\tif (i != dst)\n+\t\t\tinfo->ours[dst] = info->ours[i];\n+\t\tc = lookup_commit(sha1[info->ours[i]]);\n+\t\tbitmap = ref_bitmap_at(ref_bitmap, c);\n+\t\tif (!*bitmap)\n+\t\t\tcontinue;\n+\t\tfor (j = 0; j < bitmap_nr; j++)\n+\t\t\tif (bitmap[0][j] &&\n+\t\t\t /* Step 7, reachability test at commit level */\n+\t\t\t !in_merge_bases_many(c, ca.nr, ca.commits)) {\n+\t\t\t\tupdate_refstatus(ref_status, info->ref->nr, *bitmap);\n+\t\t\t\tdst++;\n+\t\t\t\tbreak;\n+\t\t\t}\n+\t}\n+\tinfo->nr_ours = dst;\n+\n+\tfree(ca.commits);\n+}\n-- \n1.8.5.1.25.g8667982\n"},{"id":"231594","messageId":"1386248575-10206-11-git-send-email-pclouds@gmail.com","threadId":"35400","inReplyTo":"1386248575-10206-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v4 10/28] fetch-pack.c: move shallow update code out of fetch_pack()","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-12-05T13:02:37Z","receivedAt":"2013-12-05T13:02:37Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Signed-off-by: Nguyễn Thái Ngọc Duy \n---\n fetch-pack.c | 22 +++++++++++++---------\n 1 file changed, 13 insertions(+), 9 deletions(-)\n\ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex 0e7483e..35d097e 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -925,6 +925,18 @@ static int remove_duplicates_in_refs(struct ref **ref, int nr)\n \treturn dst;\n }\n \n+static void update_shallow(struct fetch_pack_args *args)\n+{\n+\tif (args->depth > 0 && alternate_shallow_file) {\n+\t\tif (*alternate_shallow_file == '\\0') { /* --unshallow */\n+\t\t\tunlink_or_warn(git_path(\"shallow\"));\n+\t\t\trollback_lock_file(&shallow_lock);\n+\t\t} else\n+\t\t\tcommit_lock_file(&shallow_lock);\n+\t\treturn;\n+\t}\n+}\n+\n struct ref *fetch_pack(struct fetch_pack_args *args,\n \t\t int fd[], struct child_process *conn,\n \t\t const struct ref *ref,\n@@ -943,15 +955,7 @@ struct ref *fetch_pack(struct fetch_pack_args *args,\n \t\tdie(\"no matching remote head\");\n \t}\n \tref_cpy = do_fetch_pack(args, fd, ref, sought, nr_sought, pack_lockfile);\n-\n-\tif (args->depth > 0 && alternate_shallow_file) {\n-\t\tif (*alternate_shallow_file == '\\0') { /* --unshallow */\n-\t\t\tunlink_or_warn(git_path(\"shallow\"));\n-\t\t\trollback_lock_file(&shallow_lock);\n-\t\t} else\n-\t\t\tcommit_lock_file(&shallow_lock);\n-\t}\n-\n+\tupdate_shallow(args);\n \treprepare_packed_git();\n \treturn ref_cpy;\n }\n-- \n1.8.5.1.25.g8667982\n"},{"id":"231595","messageId":"1386248575-10206-12-git-send-email-pclouds@gmail.com","threadId":"35400","inReplyTo":"1386248575-10206-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v4 11/28] fetch-pack.h: one statement per bitfield declaration","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-12-05T13:02:38Z","receivedAt":"2013-12-05T13:02:38Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Signed-off-by: Nguyễn Thái Ngọc Duy \n---\n fetch-pack.h | 24 ++++++++++++------------\n 1 file changed, 12 insertions(+), 12 deletions(-)\n\ndiff --git a/fetch-pack.h b/fetch-pack.h\nindex 461cbf3..9b08388 100644\n--- a/fetch-pack.h\n+++ b/fetch-pack.h\n@@ -8,18 +8,18 @@ struct fetch_pack_args {\n \tconst char *uploadpack;\n \tint unpacklimit;\n \tint depth;\n-\tunsigned quiet:1,\n-\t\tkeep_pack:1,\n-\t\tlock_pack:1,\n-\t\tuse_thin_pack:1,\n-\t\tfetch_all:1,\n-\t\tstdin_refs:1,\n-\t\tverbose:1,\n-\t\tno_progress:1,\n-\t\tinclude_tag:1,\n-\t\tstateless_rpc:1,\n-\t\tcheck_self_contained_and_connected:1,\n-\t\tself_contained_and_connected:1;\n+\tunsigned quiet:1;\n+\tunsigned keep_pack:1;\n+\tunsigned lock_pack:1;\n+\tunsigned use_thin_pack:1;\n+\tunsigned fetch_all:1;\n+\tunsigned stdin_refs:1;\n+\tunsigned verbose:1;\n+\tunsigned no_progress:1;\n+\tunsigned include_tag:1;\n+\tunsigned stateless_rpc:1;\n+\tunsigned check_self_contained_and_connected:1;\n+\tunsigned self_contained_and_connected:1;\n };\n \n /*\n-- \n1.8.5.1.25.g8667982\n"},{"id":"231596","messageId":"1386248575-10206-13-git-send-email-pclouds@gmail.com","threadId":"35400","inReplyTo":"1386248575-10206-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v4 12/28] clone: support remote shallow repository","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-12-05T13:02:39Z","receivedAt":"2013-12-05T13:02:39Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Cloning from a shallow repository does not follow the \"8 steps for new\n.git/shallow\" because if it does we need to get through step 6 for all\nrefs. That means commit walking down to the bottom.\n\nInstead the rule to create .git/shallow is simpler and, more\nimportantly, cheap: if a shallow commit is found in the pack, it's\nprobably used (i.e. reachable from some refs), so we add it. Others\nare dropped.\n\nOne may notice this method seems flawed by the word \"probably\". A\nshallow commit may not be reachable from any refs at all if it's\nattached to an object island (a group of objects that are not\nreachable by any refs).\n\nIf that object island is not complete, a new fetch request may send\nmore objects to connect it to some ref. At that time, because we\nincorrectly installed the shallow commit in this island, the user will\nnot see anything after that commit (fsck is still ok). This is not\ndesired.\n\nGiven that object islands are rare (C Git never sends such islands for\nsecurity reasons) and do not really harm the repository integrity, a\ntradeoff is made to surprise the user occasionally but work faster\neveryday.\n\nA new option --strict could be added later that follows exactly the 8\nsteps. \"git prune\" can also learn to remove dangling objects _and_ the\nshallow commits that are attached to them from .git/shallow.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/clone.c | 1 +\n builtin/fetch-pack.c | 2 +-\n fetch-pack.c | 54 +++++++++++++++++++++++++++++++++++++++++++++++++---\n fetch-pack.h | 4 ++++\n transport.c | 11 ++++++++---\n transport.h | 6 ++++++\n 6 files changed, 71 insertions(+), 7 deletions(-)\n\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex 900f564..0b182ce 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -889,6 +889,7 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \n \tremote = remote_get(option_origin);\n \ttransport = transport_get(remote, remote->url[0]);\n+\ttransport->cloning = 1;\n \n \tif (!transport->get_refs_list || (!is_local && !transport->fetch))\n \t\tdie(_(\"Don't know how to clone %s\"), transport->url);\ndiff --git a/builtin/fetch-pack.c b/builtin/fetch-pack.c\nindex c1d918f..927424b 100644\n--- a/builtin/fetch-pack.c\n+++ b/builtin/fetch-pack.c\n@@ -153,7 +153,7 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)\n \tget_remote_heads(fd[0], NULL, 0, &ref, 0, NULL, NULL);\n \n \tref = fetch_pack(&args, fd, conn, ref, dest,\n-\t\t\t sought, nr_sought, pack_lockfile_ptr);\n+\t\t\t sought, nr_sought, NULL, pack_lockfile_ptr);\n \tif (pack_lockfile) {\n \t\tprintf(\"lock %s\\n\", pack_lockfile);\n \t\tfflush(stdout);\ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex 35d097e..e18a9fd 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -13,6 +13,7 @@\n #include \"transport.h\"\n #include \"version.h\"\n #include \"prio-queue.h\"\n+#include \"sha1-array.h\"\n \n static int transfer_unpack_limit = -1;\n static int fetch_unpack_limit = -1;\n@@ -774,6 +775,7 @@ static struct ref *do_fetch_pack(struct fetch_pack_args *args,\n \t\t\t\t int fd[2],\n \t\t\t\t const struct ref *orig_ref,\n \t\t\t\t struct ref **sought, int nr_sought,\n+\t\t\t\t struct shallow_info *si,\n \t\t\t\t char **pack_lockfile)\n {\n \tstruct ref *ref = copy_ref_list(orig_ref);\n@@ -852,6 +854,8 @@ static struct ref *do_fetch_pack(struct fetch_pack_args *args,\n \tif (args->depth > 0)\n \t\tsetup_alternate_shallow(&shallow_lock, &alternate_shallow_file,\n \t\t\t\t\tNULL);\n+\telse if (args->cloning && si->shallow && si->shallow->nr)\n+\t\talternate_shallow_file = setup_temporary_shallow(si->shallow);\n \telse\n \t\talternate_shallow_file = NULL;\n \tif (get_pack(args, fd, pack_lockfile))\n@@ -925,8 +929,11 @@ static int remove_duplicates_in_refs(struct ref **ref, int nr)\n \treturn dst;\n }\n \n-static void update_shallow(struct fetch_pack_args *args)\n+static void update_shallow(struct fetch_pack_args *args,\n+\t\t\t struct shallow_info *si)\n {\n+\tint i;\n+\n \tif (args->depth > 0 && alternate_shallow_file) {\n \t\tif (*alternate_shallow_file == '\\0') { /* --unshallow */\n \t\t\tunlink_or_warn(git_path(\"shallow\"));\n@@ -935,6 +942,42 @@ static void update_shallow(struct fetch_pack_args *args)\n \t\t\tcommit_lock_file(&shallow_lock);\n \t\treturn;\n \t}\n+\n+\tif (!si->shallow || !si->shallow->nr)\n+\t\treturn;\n+\n+\tif (alternate_shallow_file) {\n+\t\t/*\n+\t\t * The temporary shallow file is only useful for\n+\t\t * index-pack and unpack-objects because it may\n+\t\t * contain more roots than we want. Delete it.\n+\t\t */\n+\t\tif (*alternate_shallow_file)\n+\t\t\tunlink(alternate_shallow_file);\n+\t\tfree((char*)alternate_shallow_file);\n+\t}\n+\n+\tif (args->cloning) {\n+\t\t/*\n+\t\t * remote is shallow, but this is a clone, there are\n+\t\t * no objects in repo to worry about. Accept any\n+\t\t * shallow points that exist in the pack (iow in repo\n+\t\t * after get_pack() and reprepare_packed_git())\n+\t\t */\n+\t\tstruct sha1_array extra = SHA1_ARRAY_INIT;\n+\t\tunsigned char (*sha1)[20] = si->shallow->sha1;\n+\t\tfor (i = 0; i < si->shallow->nr; i++)\n+\t\t\tif (has_sha1_file(sha1[i]))\n+\t\t\t\tsha1_array_append(&extra, sha1[i]);\n+\t\tif (extra.nr) {\n+\t\t\tsetup_alternate_shallow(&shallow_lock,\n+\t\t\t\t\t\t&alternate_shallow_file,\n+\t\t\t\t\t\t&extra);\n+\t\t\tcommit_lock_file(&shallow_lock);\n+\t\t}\n+\t\tsha1_array_clear(&extra);\n+\t\treturn;\n+\t}\n }\n \n struct ref *fetch_pack(struct fetch_pack_args *args,\n@@ -942,9 +985,11 @@ struct ref *fetch_pack(struct fetch_pack_args *args,\n \t\t const struct ref *ref,\n \t\t const char *dest,\n \t\t struct ref **sought, int nr_sought,\n+\t\t struct sha1_array *shallow,\n \t\t char **pack_lockfile)\n {\n \tstruct ref *ref_cpy;\n+\tstruct shallow_info si;\n \n \tfetch_pack_setup();\n \tif (nr_sought)\n@@ -954,8 +999,11 @@ struct ref *fetch_pack(struct fetch_pack_args *args,\n \t\tpacket_flush(fd[1]);\n \t\tdie(\"no matching remote head\");\n \t}\n-\tref_cpy = do_fetch_pack(args, fd, ref, sought, nr_sought, pack_lockfile);\n-\tupdate_shallow(args);\n+\tprepare_shallow_info(&si, shallow);\n+\tref_cpy = do_fetch_pack(args, fd, ref, sought, nr_sought,\n+\t\t\t\t&si, pack_lockfile);\n \treprepare_packed_git();\n+\tupdate_shallow(args, &si);\n+\tclear_shallow_info(&si);\n \treturn ref_cpy;\n }\ndiff --git a/fetch-pack.h b/fetch-pack.h\nindex 9b08388..ce59537 100644\n--- a/fetch-pack.h\n+++ b/fetch-pack.h\n@@ -4,6 +4,8 @@\n #include \"string-list.h\"\n #include \"run-command.h\"\n \n+struct sha1_array;\n+\n struct fetch_pack_args {\n \tconst char *uploadpack;\n \tint unpacklimit;\n@@ -20,6 +22,7 @@ struct fetch_pack_args {\n \tunsigned stateless_rpc:1;\n \tunsigned check_self_contained_and_connected:1;\n \tunsigned self_contained_and_connected:1;\n+\tunsigned cloning:1;\n };\n \n /*\n@@ -33,6 +36,7 @@ struct ref *fetch_pack(struct fetch_pack_args *args,\n \t\t const char *dest,\n \t\t struct ref **sought,\n \t\t int nr_sought,\n+\t\t struct sha1_array *shallow,\n \t\t char **pack_lockfile);\n \n #endif\ndiff --git a/transport.c b/transport.c\nindex 90453df..91c4667 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -456,6 +456,7 @@ struct git_transport_data {\n \tint fd[2];\n \tunsigned got_remote_heads : 1;\n \tstruct sha1_array extra_have;\n+\tstruct sha1_array shallow;\n };\n \n static int set_git_option(struct git_transport_options *opts,\n@@ -512,7 +513,9 @@ static struct ref *get_refs_via_connect(struct transport *transport, int for_pus\n \n \tconnect_setup(transport, for_push, 0);\n \tget_remote_heads(data->fd[0], NULL, 0, &refs,\n-\t\t\t for_push ? REF_NORMAL : 0, &data->extra_have, NULL);\n+\t\t\t for_push ? REF_NORMAL : 0,\n+\t\t\t &data->extra_have,\n+\t\t\t transport->cloning ? &data->shallow : NULL);\n \tdata->got_remote_heads = 1;\n \n \treturn refs;\n@@ -539,17 +542,19 @@ static int fetch_refs_via_pack(struct transport *transport,\n \targs.depth = data->options.depth;\n \targs.check_self_contained_and_connected =\n \t\tdata->options.check_self_contained_and_connected;\n+\targs.cloning = transport->cloning;\n \n \tif (!data->got_remote_heads) {\n \t\tconnect_setup(transport, 0, 0);\n \t\tget_remote_heads(data->fd[0], NULL, 0, &refs_tmp, 0,\n-\t\t\t\t NULL, NULL);\n+\t\t\t\t NULL,\n+\t\t\t\t transport->cloning ? &data->shallow : NULL);\n \t\tdata->got_remote_heads = 1;\n \t}\n \n \trefs = fetch_pack(&args, data->fd, data->conn,\n \t\t\t refs_tmp ? refs_tmp : transport->remote_refs,\n-\t\t\t dest, to_fetch, nr_heads,\n+\t\t\t dest, to_fetch, nr_heads, &data->shallow,\n \t\t\t &transport->pack_lockfile);\n \tclose(data->fd[0]);\n \tclose(data->fd[1]);\ndiff --git a/transport.h b/transport.h\nindex b3679bb..59842d4 100644\n--- a/transport.h\n+++ b/transport.h\n@@ -35,6 +35,12 @@ struct transport {\n \t */\n \tunsigned cannot_reuse : 1;\n \n+\t/*\n+\t * A hint from caller that it will be performing a clone, not\n+\t * normal fetch. IOW the repository is guaranteed empty.\n+\t */\n+\tunsigned cloning : 1;\n+\n \t/**\n \t * Returns 0 if successful, positive if the option is not\n \t * recognized or is inapplicable, and negative if the option\n-- \n1.8.5.1.25.g8667982\n"},{"id":"231597","messageId":"1386248575-10206-14-git-send-email-pclouds@gmail.com","threadId":"35400","inReplyTo":"1386248575-10206-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v4 13/28] fetch: support fetching from a shallow repository","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-12-05T13:02:40Z","receivedAt":"2013-12-05T13:02:40Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"This patch just put together pieces from the 8 steps patch. We stop at\nstep 7 and reject refs that require new shallow commits.\n\nNote that, by rejecting refs that require new shallow commits, we\nleave dangling objects in the repo, which become \"object islands\" by\nthe next \"git fetch\" of the same source.\n\nIf the first fetch our \"ours\" set is zero and we do practically\nnothing at step 7, \"ours\" is full at the next fetch and we may need to\nwalk through commits for reachability test. Room for improvement.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/fetch.c | 9 +++\n fetch-pack.c | 32 +++++++++-\n remote.h | 1 +\n t/t5536-fetch-shallow.sh (new +x) | 128 ++++++++++++++++++++++++++++++++++++++\n transport.c | 11 +++-\n 5 files changed, 176 insertions(+), 5 deletions(-)\n create mode 100755 t/t5536-fetch-shallow.sh\n\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex bd7a101..7b41a7e 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -405,6 +405,8 @@ static int iterate_ref_map(void *cb_data, unsigned char sha1[20])\n \tstruct ref **rm = cb_data;\n \tstruct ref *ref = *rm;\n \n+\twhile (ref && ref->status == REF_STATUS_REJECT_SHALLOW)\n+\t\tref = ref->next;\n \tif (!ref)\n \t\treturn -1; /* end of the list */\n \t*rm = ref->next;\n@@ -451,6 +453,13 @@ static int store_updated_refs(const char *raw_url, const char *remote_name,\n \t\t\tstruct ref *ref = NULL;\n \t\t\tconst char *merge_status_marker = \"\";\n \n+\t\t\tif (rm->status == REF_STATUS_REJECT_SHALLOW) {\n+\t\t\t\tif (want_status == FETCH_HEAD_MERGE)\n+\t\t\t\t\twarning(_(\"reject %s because shallow roots are not allowed to be updated\"),\n+\t\t\t\t\t\trm->peer_ref ? rm->peer_ref->name : rm->name);\n+\t\t\t\tcontinue;\n+\t\t\t}\n+\n \t\t\tcommit = lookup_commit_reference_gently(rm->old_sha1, 1);\n \t\t\tif (!commit)\n \t\t\t\trm->fetch_head_status = FETCH_HEAD_NOT_FOR_MERGE;\ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex e18a9fd..7c11aff 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -854,7 +854,7 @@ static struct ref *do_fetch_pack(struct fetch_pack_args *args,\n \tif (args->depth > 0)\n \t\tsetup_alternate_shallow(&shallow_lock, &alternate_shallow_file,\n \t\t\t\t\tNULL);\n-\telse if (args->cloning && si->shallow && si->shallow->nr)\n+\telse if (si->nr_ours || si->nr_theirs)\n \t\talternate_shallow_file = setup_temporary_shallow(si->shallow);\n \telse\n \t\talternate_shallow_file = NULL;\n@@ -930,8 +930,11 @@ static int remove_duplicates_in_refs(struct ref **ref, int nr)\n }\n \n static void update_shallow(struct fetch_pack_args *args,\n+\t\t\t struct ref **sought, int nr_sought,\n \t\t\t struct shallow_info *si)\n {\n+\tstruct sha1_array ref = SHA1_ARRAY_INIT;\n+\tint *status;\n \tint i;\n \n \tif (args->depth > 0 && alternate_shallow_file) {\n@@ -978,6 +981,31 @@ static void update_shallow(struct fetch_pack_args *args,\n \t\tsha1_array_clear(&extra);\n \t\treturn;\n \t}\n+\n+\tif (!si->nr_ours && !si->nr_theirs)\n+\t\treturn;\n+\n+\tremove_nonexistent_theirs_shallow(si);\n+\t/* XXX remove_nonexistent_ours_in_pack() */\n+\tif (!si->nr_ours && !si->nr_theirs)\n+\t\treturn;\n+\tfor (i = 0; i < nr_sought; i++)\n+\t\tsha1_array_append(&ref, sought[i]->old_sha1);\n+\tsi->ref = &ref;\n+\n+\t/*\n+\t * remote is also shallow, check what ref is safe to update\n+\t * without updating .git/shallow\n+\t */\n+\tstatus = xcalloc(nr_sought, sizeof(*status));\n+\tassign_shallow_commits_to_refs(si, NULL, status);\n+\tif (si->nr_ours || si->nr_theirs) {\n+\t\tfor (i = 0; i < nr_sought; i++)\n+\t\t\tif (status[i])\n+\t\t\t\tsought[i]->status = REF_STATUS_REJECT_SHALLOW;\n+\t}\n+\tfree(status);\n+\tsha1_array_clear(&ref);\n }\n \n struct ref *fetch_pack(struct fetch_pack_args *args,\n@@ -1003,7 +1031,7 @@ struct ref *fetch_pack(struct fetch_pack_args *args,\n \tref_cpy = do_fetch_pack(args, fd, ref, sought, nr_sought,\n \t\t\t\t&si, pack_lockfile);\n \treprepare_packed_git();\n-\tupdate_shallow(args, &si);\n+\tupdate_shallow(args, sought, nr_sought, &si);\n \tclear_shallow_info(&si);\n \treturn ref_cpy;\n }\ndiff --git a/remote.h b/remote.h\nindex 5d217d5..3498091 100644\n--- a/remote.h\n+++ b/remote.h\n@@ -109,6 +109,7 @@ struct ref {\n \t\tREF_STATUS_REJECT_FETCH_FIRST,\n \t\tREF_STATUS_REJECT_NEEDS_FORCE,\n \t\tREF_STATUS_REJECT_STALE,\n+\t\tREF_STATUS_REJECT_SHALLOW,\n \t\tREF_STATUS_UPTODATE,\n \t\tREF_STATUS_REMOTE_REJECT,\n \t\tREF_STATUS_EXPECTING_REPORT\ndiff --git a/t/t5536-fetch-shallow.sh b/t/t5536-fetch-shallow.sh\nnew file mode 100755\nindex 0000000..d211052\n--- /dev/null\n+++ b/t/t5536-fetch-shallow.sh\n@@ -0,0 +1,128 @@\n+#!/bin/sh\n+\n+test_description='fetch/clone from a shallow clone'\n+\n+. ./test-lib.sh\n+\n+commit() {\n+\techo \"$1\" >tracked &&\n+\tgit add tracked &&\n+\tgit commit -m \"$1\"\n+}\n+\n+test_expect_success 'setup' '\n+\tcommit 1 &&\n+\tcommit 2 &&\n+\tcommit 3 &&\n+\tcommit 4 &&\n+\tgit config --global transfer.fsckObjects true\n+'\n+\n+test_expect_success 'setup shallow clone' '\n+\tgit clone --no-local --depth=2 .git shallow &&\n+\tgit --git-dir=shallow/.git log --format=%s >actual &&\n+\tcat <expect &&\n+4\n+3\n+EOF\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'clone from shallow clone' '\n+\tgit clone --no-local shallow shallow2 &&\n+\t(\n+\tcd shallow2 &&\n+\tgit fsck &&\n+\tgit log --format=%s >actual &&\n+\tcat <expect &&\n+4\n+3\n+EOF\n+\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'fetch from shallow clone' '\n+\t(\n+\tcd shallow &&\n+\tcommit 5\n+\t) &&\n+\t(\n+\tcd shallow2 &&\n+\tgit fetch &&\n+\tgit fsck &&\n+\tgit log --format=%s origin/master >actual &&\n+\tcat <expect &&\n+5\n+4\n+3\n+EOF\n+\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'fetch --depth from shallow clone' '\n+\t(\n+\tcd shallow &&\n+\tcommit 6\n+\t) &&\n+\t(\n+\tcd shallow2 &&\n+\tgit fetch --depth=2 &&\n+\tgit fsck &&\n+\tgit log --format=%s origin/master >actual &&\n+\tcat <expect &&\n+6\n+5\n+EOF\n+\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'fetch something upstream has but hidden by clients shallow boundaries' '\n+\t# the blob \"1\" is available in .git but hidden by the\n+\t# shallow2/.git/shallow and it should be resent\n+\t! git --git-dir=shallow2/.git cat-file blob `echo 1|git hash-object --stdin` >/dev/null &&\n+\techo 1 >1.t &&\n+\tgit add 1.t &&\n+\tgit commit -m add-1-back &&\n+\t(\n+\tcd shallow2 &&\n+\tgit fetch ../.git +refs/heads/master:refs/remotes/top/master &&\n+\tgit fsck &&\n+\tgit log --format=%s top/master >actual &&\n+\tcat <expect &&\n+add-1-back\n+4\n+3\n+EOF\n+\ttest_cmp expect actual\n+\t) &&\n+\tgit --git-dir=shallow2/.git cat-file blob `echo 1|git hash-object --stdin` >/dev/null\n+\n+'\n+\n+test_expect_success 'fetch that requires changes in .git/shallow is filtered' '\n+\t(\n+\tcd shallow &&\n+\tgit checkout --orphan no-shallow &&\n+\tcommit no-shallow\n+\t) &&\n+\tgit init notshallow &&\n+\t(\n+\tcd notshallow &&\n+\tgit fetch ../shallow/.git refs/heads/*:refs/remotes/shallow/*&&\n+\tgit for-each-ref --format=\"%(refname)\" >actual.refs &&\n+\tcat <expect.refs &&\n+refs/remotes/shallow/no-shallow\n+EOF\n+\ttest_cmp expect.refs actual.refs &&\n+\tgit log --format=%s shallow/no-shallow >actual &&\n+\tcat <expect &&\n+no-shallow\n+EOF\n+\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_done\ndiff --git a/transport.c b/transport.c\nindex 91c4667..491360b 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -515,7 +515,7 @@ static struct ref *get_refs_via_connect(struct transport *transport, int for_pus\n \tget_remote_heads(data->fd[0], NULL, 0, &refs,\n \t\t\t for_push ? REF_NORMAL : 0,\n \t\t\t &data->extra_have,\n-\t\t\t transport->cloning ? &data->shallow : NULL);\n+\t\t\t &data->shallow);\n \tdata->got_remote_heads = 1;\n \n \treturn refs;\n@@ -547,8 +547,7 @@ static int fetch_refs_via_pack(struct transport *transport,\n \tif (!data->got_remote_heads) {\n \t\tconnect_setup(transport, 0, 0);\n \t\tget_remote_heads(data->fd[0], NULL, 0, &refs_tmp, 0,\n-\t\t\t\t NULL,\n-\t\t\t\t transport->cloning ? &data->shallow : NULL);\n+\t\t\t\t NULL, &data->shallow);\n \t\tdata->got_remote_heads = 1;\n \t}\n \n@@ -720,6 +719,10 @@ static int print_one_push_status(struct ref *ref, const char *dest, int count, i\n \t\tprint_ref_status('!', \"[rejected]\", ref, ref->peer_ref,\n \t\t\t\t\t\t \"stale info\", porcelain);\n \t\tbreak;\n+\tcase REF_STATUS_REJECT_SHALLOW:\n+\t\tprint_ref_status('!', \"[rejected]\", ref, ref->peer_ref,\n+\t\t\t\t\t\t \"new shallow roots not allowed\", porcelain);\n+\t\tbreak;\n \tcase REF_STATUS_REMOTE_REJECT:\n \t\tprint_ref_status('!', \"[remote rejected]\", ref,\n \t\t\t\t\t\t ref->deletion ? NULL : ref->peer_ref,\n@@ -815,6 +818,8 @@ static int git_transport_push(struct transport *transport, struct ref *remote_re\n \t\tget_remote_heads(data->fd[0], NULL, 0, &tmp_refs, REF_NORMAL, NULL, NULL);\n \t\tdata->got_remote_heads = 1;\n \t}\n+\tif (data->shallow.nr)\n+\t\tdie(\"pushing to a shallow repository is not supported\");\n \n \tmemset(&args, 0, sizeof(args));\n \targs.send_mirror = !!(flags & TRANSPORT_PUSH_MIRROR);\n-- \n1.8.5.1.25.g8667982\n"},{"id":"231598","messageId":"1386248575-10206-15-git-send-email-pclouds@gmail.com","threadId":"35400","inReplyTo":"1386248575-10206-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v4 14/28] upload-pack: make sure deepening preserves shallow roots","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-12-05T13:02:41Z","receivedAt":"2013-12-05T13:02:41Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"When \"fetch --depth=N\" where N exceeds the longest chain of history in\nthe source repo, usually we just send an \"unshallow\" line to the\nclient so full history is obtained.\n\nWhen the source repo is shallow we need to make sure to \"unshallow\"\nthe current shallow point _and_ \"shallow\" again when the commit\nreaches its shallow bottom in the source repo.\n\nThis should fix both cases: large and --unshallow.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n Documentation/fetch-options.txt | 8 ++++++--\n shallow.c | 6 +++++-\n t/t5536-fetch-shallow.sh | 16 ++++++++++++++++\n upload-pack.c | 2 +-\n 4 files changed, 28 insertions(+), 4 deletions(-)\n\ndiff --git a/Documentation/fetch-options.txt b/Documentation/fetch-options.txt\nindex ba1fe49..a83d2b4 100644\n--- a/Documentation/fetch-options.txt\n+++ b/Documentation/fetch-options.txt\n@@ -14,8 +14,12 @@\n \tbranch history. Tags for the deepened commits are not fetched.\n \n --unshallow::\n-\tConvert a shallow repository to a complete one, removing all\n-\tthe limitations imposed by shallow repositories.\n+\tIf the source repository is complete, convert a shallow\n+\trepository to a complete one, removing all the limitations\n+\timposed by shallow repositories.\n++\n+If the source repository is shallow, fetch as much as possible so that\n+the current repository has the same history as the source repository.\n \n ifndef::git-pull[]\n --dry-run::\ndiff --git a/shallow.c b/shallow.c\nindex 975d699..ae46b10 100644\n--- a/shallow.c\n+++ b/shallow.c\n@@ -75,6 +75,7 @@ struct commit_list *get_shallow_commits(struct object_array *heads, int depth,\n \tstruct commit_list *result = NULL;\n \tstruct object_array stack = OBJECT_ARRAY_INIT;\n \tstruct commit *commit = NULL;\n+\tstruct commit_graft *graft;\n \n \twhile (commit || i < heads->nr || stack.nr) {\n \t\tstruct commit_list *p;\n@@ -99,7 +100,10 @@ struct commit_list *get_shallow_commits(struct object_array *heads, int depth,\n \t\tif (parse_commit(commit))\n \t\t\tdie(\"invalid commit\");\n \t\tcur_depth++;\n-\t\tif (cur_depth >= depth) {\n+\t\tif ((depth != INFINITE_DEPTH && cur_depth >= depth) ||\n+\t\t (is_repository_shallow() && !commit->parents &&\n+\t\t (graft = lookup_commit_graft(commit->object.sha1)) != NULL &&\n+\t\t graft->nr_parent < 0)) {\n \t\t\tcommit_list_insert(commit, &result);\n \t\t\tcommit->object.flags |= shallow_flag;\n \t\t\tcommit = NULL;\ndiff --git a/t/t5536-fetch-shallow.sh b/t/t5536-fetch-shallow.sh\nindex d211052..022cb2c 100755\n--- a/t/t5536-fetch-shallow.sh\n+++ b/t/t5536-fetch-shallow.sh\n@@ -79,6 +79,22 @@ EOF\n \t)\n '\n \n+test_expect_success 'fetch --unshallow from shallow clone' '\n+\t(\n+\tcd shallow2 &&\n+\tgit fetch --unshallow &&\n+\tgit fsck &&\n+\tgit log --format=%s origin/master >actual &&\n+\tcat <expect &&\n+6\n+5\n+4\n+3\n+EOF\n+\ttest_cmp expect actual\n+\t)\n+'\n+\n test_expect_success 'fetch something upstream has but hidden by clients shallow boundaries' '\n \t# the blob \"1\" is available in .git but hidden by the\n \t# shallow2/.git/shallow and it should be resent\ndiff --git a/upload-pack.c b/upload-pack.c\nindex f082f06..28269c7 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -619,7 +619,7 @@ static void receive_needs(void)\n \tif (depth > 0) {\n \t\tstruct commit_list *result = NULL, *backup = NULL;\n \t\tint i;\n-\t\tif (depth == INFINITE_DEPTH)\n+\t\tif (depth == INFINITE_DEPTH && !is_repository_shallow())\n \t\t\tfor (i = 0; i < shallows.nr; i++) {\n \t\t\t\tstruct object *object = shallows.objects[i].item;\n \t\t\t\tobject->flags |= NOT_SHALLOW;\n-- \n1.8.5.1.25.g8667982\n"},{"id":"231599","messageId":"1386248575-10206-16-git-send-email-pclouds@gmail.com","threadId":"35400","inReplyTo":"1386248575-10206-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v4 15/28] fetch: add --update-shallow to accept refs that update .git/shallow","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-12-05T13:02:42Z","receivedAt":"2013-12-05T13:02:42Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"The same steps are done as in when --update-shallow is not given. The\nonly difference is we now add all shallow commits in \"ours\" and\n\"theirs\" to .git/shallow (aka \"step 8\").\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n Documentation/fetch-options.txt | 6 ++++++\n builtin/fetch.c | 6 +++++-\n fetch-pack.c | 27 +++++++++++++++++++++++++++\n fetch-pack.h | 1 +\n t/t5536-fetch-shallow.sh | 32 ++++++++++++++++++++++++++++++++\n transport.c | 4 ++++\n transport.h | 4 ++++\n 7 files changed, 79 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/fetch-options.txt b/Documentation/fetch-options.txt\nindex a83d2b4..54043e3 100644\n--- a/Documentation/fetch-options.txt\n+++ b/Documentation/fetch-options.txt\n@@ -21,6 +21,12 @@\n If the source repository is shallow, fetch as much as possible so that\n the current repository has the same history as the source repository.\n \n+--update-shallow::\n+\tBy default when fetching from a shallow repository,\n+\t`git fetch` refuses refs that require updating\n+\t.git/shallow. This option updates .git/shallow and accept such\n+\trefs.\n+\n ifndef::git-pull[]\n --dry-run::\n \tShow what would be done, without making any changes.\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex 7b41a7e..d2e4fc0 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -36,7 +36,7 @@ static int prune = -1; /* unspecified */\n \n static int all, append, dry_run, force, keep, multiple, update_head_ok, verbosity;\n static int progress = -1, recurse_submodules = RECURSE_SUBMODULES_DEFAULT;\n-static int tags = TAGS_DEFAULT, unshallow;\n+static int tags = TAGS_DEFAULT, unshallow, update_shallow;\n static const char *depth;\n static const char *upload_pack;\n static struct strbuf default_rla = STRBUF_INIT;\n@@ -104,6 +104,8 @@ static struct option builtin_fetch_options[] = {\n \t{ OPTION_STRING, 0, \"recurse-submodules-default\",\n \t\t &recurse_submodules_default, NULL,\n \t\t N_(\"default mode for recursion\"), PARSE_OPT_HIDDEN },\n+\tOPT_BOOL(0, \"update-shallow\", &update_shallow,\n+\t\t N_(\"accept refs that update .git/shallow\")),\n \tOPT_END()\n };\n \n@@ -768,6 +770,8 @@ static struct transport *prepare_transport(struct remote *remote)\n \t\tset_option(transport, TRANS_OPT_KEEP, \"yes\");\n \tif (depth)\n \t\tset_option(transport, TRANS_OPT_DEPTH, depth);\n+\tif (update_shallow)\n+\t\tset_option(transport, TRANS_OPT_UPDATE_SHALLOW, \"yes\");\n \treturn transport;\n }\n \ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex 7c11aff..cd219b8 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -993,6 +993,33 @@ static void update_shallow(struct fetch_pack_args *args,\n \t\tsha1_array_append(&ref, sought[i]->old_sha1);\n \tsi->ref = &ref;\n \n+\tif (args->update_shallow) {\n+\t\t/*\n+\t\t * remote is also shallow, .git/shallow may be updated\n+\t\t * so all refs can be accepted. Make sure we only add\n+\t\t * shallow roots that are actually reachable from new\n+\t\t * refs.\n+\t\t */\n+\t\tstruct sha1_array extra = SHA1_ARRAY_INIT;\n+\t\tunsigned char (*sha1)[20] = si->shallow->sha1;\n+\t\tassign_shallow_commits_to_refs(si, NULL, NULL);\n+\t\tif (!si->nr_ours && !si->nr_theirs) {\n+\t\t\tsha1_array_clear(&ref);\n+\t\t\treturn;\n+\t\t}\n+\t\tfor (i = 0; i < si->nr_ours; i++)\n+\t\t\tsha1_array_append(&extra, sha1[si->ours[i]]);\n+\t\tfor (i = 0; i < si->nr_theirs; i++)\n+\t\t\tsha1_array_append(&extra, sha1[si->theirs[i]]);\n+\t\tsetup_alternate_shallow(&shallow_lock,\n+\t\t\t\t\t&alternate_shallow_file,\n+\t\t\t\t\t&extra);\n+\t\tcommit_lock_file(&shallow_lock);\n+\t\tsha1_array_clear(&extra);\n+\t\tsha1_array_clear(&ref);\n+\t\treturn;\n+\t}\n+\n \t/*\n \t * remote is also shallow, check what ref is safe to update\n \t * without updating .git/shallow\ndiff --git a/fetch-pack.h b/fetch-pack.h\nindex ce59537..ada02f5 100644\n--- a/fetch-pack.h\n+++ b/fetch-pack.h\n@@ -23,6 +23,7 @@ struct fetch_pack_args {\n \tunsigned check_self_contained_and_connected:1;\n \tunsigned self_contained_and_connected:1;\n \tunsigned cloning:1;\n+\tunsigned update_shallow:1;\n };\n \n /*\ndiff --git a/t/t5536-fetch-shallow.sh b/t/t5536-fetch-shallow.sh\nindex 022cb2c..3ae9092 100755\n--- a/t/t5536-fetch-shallow.sh\n+++ b/t/t5536-fetch-shallow.sh\n@@ -141,4 +141,36 @@ EOF\n \t)\n '\n \n+test_expect_success 'fetch --update-shallow' '\n+\t(\n+\tcd shallow &&\n+\tgit checkout master &&\n+\tcommit 7 &&\n+\tgit tag -m foo heavy-tag HEAD^ &&\n+\tgit tag light-tag HEAD^:tracked\n+\t) &&\n+\t(\n+\tcd notshallow &&\n+\tgit fetch --update-shallow ../shallow/.git refs/heads/*:refs/remotes/shallow/* &&\n+\tgit fsck &&\n+\tgit for-each-ref --sort=refname --format=\"%(refname)\" >actual.refs &&\n+\tcat <expect.refs &&\n+refs/remotes/shallow/master\n+refs/remotes/shallow/no-shallow\n+refs/tags/heavy-tag\n+refs/tags/light-tag\n+EOF\n+\ttest_cmp expect.refs actual.refs &&\n+\tgit log --format=%s shallow/master >actual &&\n+\tcat <expect &&\n+7\n+6\n+5\n+4\n+3\n+EOF\n+\ttest_cmp expect actual\n+\t)\n+'\n+\n test_done\ndiff --git a/transport.c b/transport.c\nindex 491360b..a09fdb6 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -477,6 +477,9 @@ static int set_git_option(struct git_transport_options *opts,\n \t} else if (!strcmp(name, TRANS_OPT_KEEP)) {\n \t\topts->keep = !!value;\n \t\treturn 0;\n+\t} else if (!strcmp(name, TRANS_OPT_UPDATE_SHALLOW)) {\n+\t\topts->update_shallow = !!value;\n+\t\treturn 0;\n \t} else if (!strcmp(name, TRANS_OPT_DEPTH)) {\n \t\tif (!value)\n \t\t\topts->depth = 0;\n@@ -543,6 +546,7 @@ static int fetch_refs_via_pack(struct transport *transport,\n \targs.check_self_contained_and_connected =\n \t\tdata->options.check_self_contained_and_connected;\n \targs.cloning = transport->cloning;\n+\targs.update_shallow = data->options.update_shallow;\n \n \tif (!data->got_remote_heads) {\n \t\tconnect_setup(transport, 0, 0);\ndiff --git a/transport.h b/transport.h\nindex 59842d4..02ea248 100644\n--- a/transport.h\n+++ b/transport.h\n@@ -11,6 +11,7 @@ struct git_transport_options {\n \tunsigned followtags : 1;\n \tunsigned check_self_contained_and_connected : 1;\n \tunsigned self_contained_and_connected : 1;\n+\tunsigned update_shallow : 1;\n \tint depth;\n \tconst char *uploadpack;\n \tconst char *receivepack;\n@@ -152,6 +153,9 @@ struct transport *transport_get(struct remote *, const char *);\n /* Aggressively fetch annotated tags if possible */\n #define TRANS_OPT_FOLLOWTAGS \"followtags\"\n \n+/* Accept refs that may update .git/shallow without --depth */\n+#define TRANS_OPT_UPDATE_SHALLOW \"updateshallow\"\n+\n /**\n * Returns 0 if the option was used, non-zero otherwise. Prints a\n * message to stderr if the option is not used.\n-- \n1.8.5.1.25.g8667982\n"},{"id":"231600","messageId":"1386248575-10206-17-git-send-email-pclouds@gmail.com","threadId":"35400","inReplyTo":"1386248575-10206-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v4 16/28] receive-pack: reorder some code in unpack()","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-12-05T13:02:43Z","receivedAt":"2013-12-05T13:02:43Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"This is the preparation for adding --shallow-file to both\nunpack-objects and index-pack. To sum up:\n\n - struct argv_array used instead of const char **\n\n - status/code, ip/child, unpacker/keeper are moved out to function\n top level\n\n - successful flow now ends at the end of the function\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/receive-pack.c | 70 ++++++++++++++++++++++----------------------------\n 1 file changed, 30 insertions(+), 40 deletions(-)\n\ndiff --git a/builtin/receive-pack.c b/builtin/receive-pack.c\nindex cc8c34f..8927ddf 100644\n--- a/builtin/receive-pack.c\n+++ b/builtin/receive-pack.c\n@@ -13,6 +13,7 @@\n #include \"string-list.h\"\n #include \"sha1-array.h\"\n #include \"connected.h\"\n+#include \"argv-array.h\"\n #include \"version.h\"\n \n static const char receive_pack_usage[] = \"git receive-pack \";\n@@ -822,8 +823,11 @@ static const char *pack_lockfile;\n static const char *unpack(int err_fd)\n {\n \tstruct pack_header hdr;\n+\tstruct argv_array av = ARGV_ARRAY_INIT;\n \tconst char *hdr_err;\n+\tint status;\n \tchar hdr_arg[38];\n+\tstruct child_process child;\n \tint fsck_objects = (receive_fsck_objects >= 0\n \t\t\t ? receive_fsck_objects\n \t\t\t : transfer_fsck_objects >= 0\n@@ -840,63 +844,49 @@ static const char *unpack(int err_fd)\n \t\t\t\"--pack_header=%\"PRIu32\",%\"PRIu32,\n \t\t\tntohl(hdr.hdr_version), ntohl(hdr.hdr_entries));\n \n+\tmemset(&child, 0, sizeof(child));\n \tif (ntohl(hdr.hdr_entries) < unpack_limit) {\n-\t\tint code, i = 0;\n-\t\tstruct child_process child;\n-\t\tconst char *unpacker[5];\n-\t\tunpacker[i++] = \"unpack-objects\";\n+\t\targv_array_pushl(&av, \"unpack-objects\", hdr_arg, NULL);\n \t\tif (quiet)\n-\t\t\tunpacker[i++] = \"-q\";\n+\t\t\targv_array_push(&av, \"-q\");\n \t\tif (fsck_objects)\n-\t\t\tunpacker[i++] = \"--strict\";\n-\t\tunpacker[i++] = hdr_arg;\n-\t\tunpacker[i++] = NULL;\n-\t\tmemset(&child, 0, sizeof(child));\n-\t\tchild.argv = unpacker;\n+\t\t\targv_array_push(&av, \"--strict\");\n+\t\tchild.argv = av.argv;\n \t\tchild.no_stdout = 1;\n \t\tchild.err = err_fd;\n \t\tchild.git_cmd = 1;\n-\t\tcode = run_command(&child);\n-\t\tif (!code)\n-\t\t\treturn NULL;\n-\t\treturn \"unpack-objects abnormal exit\";\n+\t\tstatus = run_command(&child);\n+\t\tif (status)\n+\t\t\treturn \"unpack-objects abnormal exit\";\n \t} else {\n-\t\tconst char *keeper[7];\n-\t\tint s, status, i = 0;\n+\t\tint s;\n \t\tchar keep_arg[256];\n-\t\tstruct child_process ip;\n \n \t\ts = sprintf(keep_arg, \"--keep=receive-pack %\"PRIuMAX\" on \", (uintmax_t) getpid());\n \t\tif (gethostname(keep_arg + s, sizeof(keep_arg) - s))\n \t\t\tstrcpy(keep_arg + s, \"localhost\");\n \n-\t\tkeeper[i++] = \"index-pack\";\n-\t\tkeeper[i++] = \"--stdin\";\n+\t\targv_array_pushl(&av, \"index-pack\",\n+\t\t\t\t \"--stdin\", hdr_arg, keep_arg, NULL);\n \t\tif (fsck_objects)\n-\t\t\tkeeper[i++] = \"--strict\";\n+\t\t\targv_array_push(&av, \"--strict\");\n \t\tif (fix_thin)\n-\t\t\tkeeper[i++] = \"--fix-thin\";\n-\t\tkeeper[i++] = hdr_arg;\n-\t\tkeeper[i++] = keep_arg;\n-\t\tkeeper[i++] = NULL;\n-\t\tmemset(&ip, 0, sizeof(ip));\n-\t\tip.argv = keeper;\n-\t\tip.out = -1;\n-\t\tip.err = err_fd;\n-\t\tip.git_cmd = 1;\n-\t\tstatus = start_command(&ip);\n-\t\tif (status) {\n+\t\t\targv_array_push(&av, \"--fix-thin\");\n+\t\tchild.argv = av.argv;\n+\t\tchild.out = -1;\n+\t\tchild.err = err_fd;\n+\t\tchild.git_cmd = 1;\n+\t\tstatus = start_command(&child);\n+\t\tif (status)\n \t\t\treturn \"index-pack fork failed\";\n-\t\t}\n-\t\tpack_lockfile = index_pack_lockfile(ip.out);\n-\t\tclose(ip.out);\n-\t\tstatus = finish_command(&ip);\n-\t\tif (!status) {\n-\t\t\treprepare_packed_git();\n-\t\t\treturn NULL;\n-\t\t}\n-\t\treturn \"index-pack abnormal exit\";\n+\t\tpack_lockfile = index_pack_lockfile(child.out);\n+\t\tclose(child.out);\n+\t\tstatus = finish_command(&child);\n+\t\tif (status)\n+\t\t\treturn \"index-pack abnormal exit\";\n+\t\treprepare_packed_git();\n \t}\n+\treturn NULL;\n }\n \n static const char *unpack_with_sideband(void)\n-- \n1.8.5.1.25.g8667982\n"},{"id":"231601","messageId":"1386248575-10206-18-git-send-email-pclouds@gmail.com","threadId":"35400","inReplyTo":"1386248575-10206-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v4 17/28] receive/send-pack: support pushing from a shallow clone","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-12-05T13:02:44Z","receivedAt":"2013-12-05T13:02:44Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Signed-off-by: Nguyễn Thái Ngọc Duy \n---\n Documentation/technical/pack-protocol.txt | 4 +-\n builtin/receive-pack.c | 78 +++++++++++++++++++++++++++----\n builtin/send-pack.c | 2 +-\n send-pack.c | 3 ++\n t/t5537-push-shallow.sh (new +x) | 70 +++++++++++++++++++++++++++\n 5 files changed, 146 insertions(+), 11 deletions(-)\n create mode 100755 t/t5537-push-shallow.sh\n\ndiff --git a/Documentation/technical/pack-protocol.txt b/Documentation/technical/pack-protocol.txt\nindex eb8edd1..c73b62f 100644\n--- a/Documentation/technical/pack-protocol.txt\n+++ b/Documentation/technical/pack-protocol.txt\n@@ -464,7 +464,9 @@ contain all the objects that the server will need to complete the new\n references.\n \n ----\n- update-request = command-list [pack-file]\n+ update-request = *shallow command-list [pack-file]\n+\n+ shallow = PKT-LINE(\"shallow\" SP obj-id)\n \n command-list = PKT-LINE(command NUL capability-list LF)\n \t\t *PKT-LINE(command LF)\ndiff --git a/builtin/receive-pack.c b/builtin/receive-pack.c\nindex 8927ddf..b9de2e8 100644\n--- a/builtin/receive-pack.c\n+++ b/builtin/receive-pack.c\n@@ -44,6 +44,7 @@ static int fix_thin = 1;\n static const char *head_name;\n static void *head_name_to_free;\n static int sent_capabilities;\n+static const char *alt_shallow_file;\n \n static enum deny_action parse_deny_action(const char *var, const char *value)\n {\n@@ -190,6 +191,7 @@ struct command {\n \tconst char *error_string;\n \tunsigned int skip_update:1,\n \t\t did_not_exist:1;\n+\tint index;\n \tunsigned char old_sha1[20];\n \tunsigned char new_sha1[20];\n \tchar ref_name[FLEX_ARRAY]; /* more */\n@@ -688,7 +690,7 @@ static int iterate_receive_command_list(void *cb_data, unsigned char sha1[20])\n \tstruct command *cmd = *cmd_list;\n \n \twhile (cmd) {\n-\t\tif (!is_null_sha1(cmd->new_sha1)) {\n+\t\tif (!is_null_sha1(cmd->new_sha1) && !cmd->skip_update) {\n \t\t\thashcpy(sha1, cmd->new_sha1);\n \t\t\t*cmd_list = cmd->next;\n \t\t\treturn 0;\n@@ -755,7 +757,7 @@ static void execute_commands(struct command *commands, const char *unpacker_erro\n \t}\n }\n \n-static struct command *read_head_info(void)\n+static struct command *read_head_info(struct sha1_array *shallow)\n {\n \tstruct command *commands = NULL;\n \tstruct command **p = &commands;\n@@ -769,6 +771,14 @@ static struct command *read_head_info(void)\n \t\tline = packet_read_line(0, &len);\n \t\tif (!line)\n \t\t\tbreak;\n+\n+\t\tif (len == 48 && !prefixcmp(line, \"shallow \")) {\n+\t\t\tif (get_sha1_hex(line + 8, old_sha1))\n+\t\t\t\tdie(\"protocol error: expected shallow sha, got '%s'\", line + 8);\n+\t\t\tsha1_array_append(shallow, old_sha1);\n+\t\t\tcontinue;\n+\t\t}\n+\n \t\tif (len < 83 ||\n \t\t line[40] != ' ' ||\n \t\t line[81] != ' ' ||\n@@ -820,7 +830,7 @@ static const char *parse_pack_header(struct pack_header *hdr)\n \n static const char *pack_lockfile;\n \n-static const char *unpack(int err_fd)\n+static const char *unpack(int err_fd, struct shallow_info *si)\n {\n \tstruct pack_header hdr;\n \tstruct argv_array av = ARGV_ARRAY_INIT;\n@@ -844,6 +854,11 @@ static const char *unpack(int err_fd)\n \t\t\t\"--pack_header=%\"PRIu32\",%\"PRIu32,\n \t\t\tntohl(hdr.hdr_version), ntohl(hdr.hdr_entries));\n \n+\tif (si->nr_ours || si->nr_theirs) {\n+\t\talt_shallow_file = setup_temporary_shallow(si->shallow);\n+\t\targv_array_pushl(&av, \"--shallow-file\", alt_shallow_file, NULL);\n+\t}\n+\n \tmemset(&child, 0, sizeof(child));\n \tif (ntohl(hdr.hdr_entries) < unpack_limit) {\n \t\targv_array_pushl(&av, \"unpack-objects\", hdr_arg, NULL);\n@@ -889,13 +904,13 @@ static const char *unpack(int err_fd)\n \treturn NULL;\n }\n \n-static const char *unpack_with_sideband(void)\n+static const char *unpack_with_sideband(struct shallow_info *si)\n {\n \tstruct async muxer;\n \tconst char *ret;\n \n \tif (!use_sideband)\n-\t\treturn unpack(0);\n+\t\treturn unpack(0, si);\n \n \tmemset(&muxer, 0, sizeof(muxer));\n \tmuxer.proc = copy_to_sideband;\n@@ -903,12 +918,48 @@ static const char *unpack_with_sideband(void)\n \tif (start_async(&muxer))\n \t\treturn NULL;\n \n-\tret = unpack(muxer.in);\n+\tret = unpack(muxer.in, si);\n \n \tfinish_async(&muxer);\n \treturn ret;\n }\n \n+static void update_shallow_info(struct command *commands,\n+\t\t\t\tstruct shallow_info *si,\n+\t\t\t\tstruct sha1_array *ref)\n+{\n+\tstruct command *cmd;\n+\tint *ref_status;\n+\tremove_nonexistent_theirs_shallow(si);\n+\t/* XXX remove_nonexistent_ours_in_pack() */\n+\tif (!si->nr_ours && !si->nr_theirs)\n+\t\treturn;\n+\n+\tfor (cmd = commands; cmd; cmd = cmd->next) {\n+\t\tif (is_null_sha1(cmd->new_sha1))\n+\t\t\tcontinue;\n+\t\tsha1_array_append(ref, cmd->new_sha1);\n+\t\tcmd->index = ref->nr - 1;\n+\t}\n+\tsi->ref = ref;\n+\n+\tref_status = xmalloc(sizeof(*ref_status) * ref->nr);\n+\tassign_shallow_commits_to_refs(si, NULL, ref_status);\n+\tfor (cmd = commands; cmd; cmd = cmd->next) {\n+\t\tif (is_null_sha1(cmd->new_sha1))\n+\t\t\tcontinue;\n+\t\tif (ref_status[cmd->index]) {\n+\t\t\tcmd->error_string = \"shallow update not allowed\";\n+\t\t\tcmd->skip_update = 1;\n+\t\t}\n+\t}\n+\tif (alt_shallow_file && *alt_shallow_file) {\n+\t\tunlink(alt_shallow_file);\n+\t\talt_shallow_file = NULL;\n+\t}\n+\tfree(ref_status);\n+}\n+\n static void report(struct command *commands, const char *unpack_status)\n {\n \tstruct command *cmd;\n@@ -950,6 +1001,9 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix)\n \tint i;\n \tchar *dir = NULL;\n \tstruct command *commands;\n+\tstruct sha1_array shallow = SHA1_ARRAY_INIT;\n+\tstruct sha1_array ref = SHA1_ARRAY_INIT;\n+\tstruct shallow_info si;\n \n \tpacket_trace_identity(\"receive-pack\");\n \n@@ -1006,11 +1060,14 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix)\n \tif (advertise_refs)\n \t\treturn 0;\n \n-\tif ((commands = read_head_info()) != NULL) {\n+\tif ((commands = read_head_info(&shallow)) != NULL) {\n \t\tconst char *unpack_status = NULL;\n \n-\t\tif (!delete_only(commands))\n-\t\t\tunpack_status = unpack_with_sideband();\n+\t\tprepare_shallow_info(&si, &shallow);\n+\t\tif (!delete_only(commands)) {\n+\t\t\tunpack_status = unpack_with_sideband(&si);\n+\t\t\tupdate_shallow_info(commands, &si, &ref);\n+\t\t}\n \t\texecute_commands(commands, unpack_status);\n \t\tif (pack_lockfile)\n \t\t\tunlink_or_warn(pack_lockfile);\n@@ -1027,8 +1084,11 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix)\n \t\t}\n \t\tif (auto_update_server_info)\n \t\t\tupdate_server_info(0);\n+\t\tclear_shallow_info(&si);\n \t}\n \tif (use_sideband)\n \t\tpacket_flush(1);\n+\tsha1_array_clear(&shallow);\n+\tsha1_array_clear(&ref);\n \treturn 0;\n }\ndiff --git a/builtin/send-pack.c b/builtin/send-pack.c\nindex 62cc4d3..ea2ab28 100644\n--- a/builtin/send-pack.c\n+++ b/builtin/send-pack.c\n@@ -208,7 +208,7 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)\n \t (send_all && args.send_mirror))\n \t\tusage(send_pack_usage);\n \n-\tif (is_repository_shallow())\n+\tif (is_repository_shallow() && args.stateless_rpc)\n \t\tdie(\"attempt to push from a shallow repository\");\n \n \tif (remote_name) {\ndiff --git a/send-pack.c b/send-pack.c\nindex 14005fa..cd536b4 100644\n--- a/send-pack.c\n+++ b/send-pack.c\n@@ -214,6 +214,9 @@ int send_pack(struct send_pack_args *args,\n \t\treturn 0;\n \t}\n \n+\tif (!args->dry_run)\n+\t\tadvertise_shallow_grafts(out);\n+\n \t/*\n \t * Finally, tell the other end!\n \t */\ndiff --git a/t/t5537-push-shallow.sh b/t/t5537-push-shallow.sh\nnew file mode 100755\nindex 0000000..650c31a\n--- /dev/null\n+++ b/t/t5537-push-shallow.sh\n@@ -0,0 +1,70 @@\n+#!/bin/sh\n+\n+test_description='push from/to a shallow clone'\n+\n+. ./test-lib.sh\n+\n+commit() {\n+\techo \"$1\" >tracked &&\n+\tgit add tracked &&\n+\tgit commit -m \"$1\"\n+}\n+\n+test_expect_success 'setup' '\n+\tgit config --global transfer.fsckObjects true &&\n+\tcommit 1 &&\n+\tcommit 2 &&\n+\tcommit 3 &&\n+\tcommit 4 &&\n+\t(\n+\tgit init full-abc &&\n+\tcd full-abc &&\n+\tcommit a &&\n+\tcommit b &&\n+\tcommit c\n+\t) &&\n+\tgit clone --no-local --depth=2 .git shallow &&\n+\tgit --git-dir=shallow/.git log --format=%s >actual &&\n+\tcat <expect &&\n+4\n+3\n+EOF\n+\ttest_cmp expect actual &&\n+\tgit clone --no-local --depth=2 full-abc/.git shallow2 &&\n+\tgit --git-dir=shallow2/.git log --format=%s >actual &&\n+\tcat <expect &&\n+c\n+b\n+EOF\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'push from shallow clone' '\n+\t(\n+\tcd shallow &&\n+\tcommit 5 &&\n+\tgit push ../.git +master:refs/remotes/shallow/master\n+\t) &&\n+\tgit log --format=%s shallow/master >actual &&\n+\tgit fsck &&\n+\tcat <expect &&\n+5\n+4\n+3\n+2\n+1\n+EOF\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'push from shallow clone, with grafted roots' '\n+\t(\n+\tcd shallow2 &&\n+\ttest_must_fail git push ../.git +master:refs/remotes/shallow2/master 2>err &&\n+\tgrep \"shallow2/master.*shallow update not allowed\" err\n+\t) &&\n+\ttest_must_fail git rev-parse shallow2/master &&\n+\tgit fsck\n+'\n+\n+test_done\n-- \n1.8.5.1.25.g8667982\n"},{"id":"231602","messageId":"1386248575-10206-19-git-send-email-pclouds@gmail.com","threadId":"35400","inReplyTo":"1386248575-10206-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v4 18/28] New var GIT_SHALLOW_FILE to propagate --shallow-file to subprocesses","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-12-05T13:02:45Z","receivedAt":"2013-12-05T13:02:45Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"This may be needed when a hook is run after a new shallow pack is\nreceived, but .git/shallow is not settled yet. A temporary shallow\nfile to plug all loose ends should be used instead. GIT_SHALLOW_FILE\nis overriden by --shallow-file.\n\n--shallow-file does not work in this case because the hook may spawn\nmany git subprocesses and the launch commands do not have\n--shallow-file as it's a recent addition.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n cache.h | 1 +\n commit.h | 2 +-\n environment.c | 6 ++++++\n git.c | 2 +-\n shallow.c | 4 +++-\n 5 files changed, 12 insertions(+), 3 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 55dd4e3..8b13287 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -354,6 +354,7 @@ static inline enum object_type object_type(unsigned int mode)\n #define DB_ENVIRONMENT \"GIT_OBJECT_DIRECTORY\"\n #define INDEX_ENVIRONMENT \"GIT_INDEX_FILE\"\n #define GRAFT_ENVIRONMENT \"GIT_GRAFT_FILE\"\n+#define GIT_SHALLOW_FILE_ENVIRONMENT \"GIT_SHALLOW_FILE\"\n #define TEMPLATE_DIR_ENVIRONMENT \"GIT_TEMPLATE_DIR\"\n #define CONFIG_ENVIRONMENT \"GIT_CONFIG\"\n #define CONFIG_DATA_ENVIRONMENT \"GIT_CONFIG_PARAMETERS\"\ndiff --git a/commit.h b/commit.h\nindex 69bca3e..79649ef 100644\n--- a/commit.h\n+++ b/commit.h\n@@ -202,7 +202,7 @@ extern int is_repository_shallow(void);\n extern struct commit_list *get_shallow_commits(struct object_array *heads,\n \t\tint depth, int shallow_flag, int not_shallow_flag);\n extern void check_shallow_file_for_update(void);\n-extern void set_alternate_shallow_file(const char *path);\n+extern void set_alternate_shallow_file(const char *path, int override);\n extern int write_shallow_commits(struct strbuf *out, int use_pack_protocol,\n \t\t\t\t const struct sha1_array *extra);\n extern void setup_alternate_shallow(struct lock_file *shallow_lock,\ndiff --git a/environment.c b/environment.c\nindex 0a15349..b73b39d 100644\n--- a/environment.c\n+++ b/environment.c\n@@ -10,6 +10,7 @@\n #include \"cache.h\"\n #include \"refs.h\"\n #include \"fmt-merge-msg.h\"\n+#include \"commit.h\"\n \n int trust_executable_bit = 1;\n int trust_ctime = 1;\n@@ -97,6 +98,7 @@ const char * const local_repo_env[] = {\n \tINDEX_ENVIRONMENT,\n \tNO_REPLACE_OBJECTS_ENVIRONMENT,\n \tGIT_PREFIX_ENVIRONMENT,\n+\tGIT_SHALLOW_FILE_ENVIRONMENT,\n \tNULL\n };\n \n@@ -124,6 +126,7 @@ static char *expand_namespace(const char *raw_namespace)\n static void setup_git_env(void)\n {\n \tconst char *gitfile;\n+\tconst char *shallow_file;\n \n \tgit_dir = getenv(GIT_DIR_ENVIRONMENT);\n \tif (!git_dir)\n@@ -147,6 +150,9 @@ static void setup_git_env(void)\n \t\tread_replace_refs = 0;\n \tnamespace = expand_namespace(getenv(GIT_NAMESPACE_ENVIRONMENT));\n \tnamespace_len = strlen(namespace);\n+\tshallow_file = getenv(GIT_SHALLOW_FILE_ENVIRONMENT);\n+\tif (shallow_file)\n+\t\tset_alternate_shallow_file(shallow_file, 0);\n }\n \n int is_bare_repository(void)\ndiff --git a/git.c b/git.c\nindex cb5208d..179c4f6 100644\n--- a/git.c\n+++ b/git.c\n@@ -162,7 +162,7 @@ static int handle_options(const char ***argv, int *argc, int *envchanged)\n \t\t} else if (!strcmp(cmd, \"--shallow-file\")) {\n \t\t\t(*argv)++;\n \t\t\t(*argc)--;\n-\t\t\tset_alternate_shallow_file((*argv)[0]);\n+\t\t\tset_alternate_shallow_file((*argv)[0], 1);\n \t\t\tif (envchanged)\n \t\t\t\t*envchanged = 1;\n \t\t} else if (!strcmp(cmd, \"-C\")) {\ndiff --git a/shallow.c b/shallow.c\nindex ae46b10..5710690 100644\n--- a/shallow.c\n+++ b/shallow.c\n@@ -13,10 +13,12 @@ static int is_shallow = -1;\n static struct stat shallow_stat;\n static char *alternate_shallow_file;\n \n-void set_alternate_shallow_file(const char *path)\n+void set_alternate_shallow_file(const char *path, int override)\n {\n \tif (is_shallow != -1)\n \t\tdie(\"BUG: is_repository_shallow must not be called before set_alternate_shallow_file\");\n+\tif (alternate_shallow_file && !override)\n+\t\treturn;\n \tfree(alternate_shallow_file);\n \talternate_shallow_file = path ? xstrdup(path) : NULL;\n }\n-- \n1.8.5.1.25.g8667982\n"},{"id":"231603","messageId":"1386248575-10206-20-git-send-email-pclouds@gmail.com","threadId":"35400","inReplyTo":"1386248575-10206-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v4 19/28] connected.c: add new variant that runs with --shallow-file","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-12-05T13:02:46Z","receivedAt":"2013-12-05T13:02:46Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Signed-off-by: Nguyễn Thái Ngọc Duy \n---\n connected.c | 42 ++++++++++++++++++++++++++++++++++--------\n connected.h | 2 ++\n 2 files changed, 36 insertions(+), 8 deletions(-)\n\ndiff --git a/connected.c b/connected.c\nindex fae8d64..427389d 100644\n--- a/connected.c\n+++ b/connected.c\n@@ -19,17 +19,17 @@ int check_everything_connected(sha1_iterate_fn fn, int quiet, void *cb_data)\n *\n * Returns 0 if everything is connected, non-zero otherwise.\n */\n-int check_everything_connected_with_transport(sha1_iterate_fn fn,\n-\t\t\t\t\t int quiet,\n-\t\t\t\t\t void *cb_data,\n-\t\t\t\t\t struct transport *transport)\n+static int check_everything_connected_real(sha1_iterate_fn fn,\n+\t\t\t\t\t int quiet,\n+\t\t\t\t\t void *cb_data,\n+\t\t\t\t\t struct transport *transport,\n+\t\t\t\t\t const char *shallow_file)\n {\n \tstruct child_process rev_list;\n-\tconst char *argv[] = {\"rev-list\", \"--objects\",\n-\t\t\t \"--stdin\", \"--not\", \"--all\", NULL, NULL};\n+\tconst char *argv[9];\n \tchar commit[41];\n \tunsigned char sha1[20];\n-\tint err = 0;\n+\tint err = 0, ac = 0;\n \tstruct packed_git *new_pack = NULL;\n \n \tif (fn(cb_data, sha1))\n@@ -47,8 +47,18 @@ int check_everything_connected_with_transport(sha1_iterate_fn fn,\n \t\tstrbuf_release(&idx_file);\n \t}\n \n+\tif (shallow_file) {\n+\t\targv[ac++] = \"--shallow-file\";\n+\t\targv[ac++] = shallow_file;\n+\t}\n+\targv[ac++] = \"rev-list\";\n+\targv[ac++] = \"--objects\";\n+\targv[ac++] = \"--stdin\";\n+\targv[ac++] = \"--not\";\n+\targv[ac++] = \"--all\";\n \tif (quiet)\n-\t\targv[5] = \"--quiet\";\n+\t\targv[ac++] = \"--quiet\";\n+\targv[ac] = NULL;\n \n \tmemset(&rev_list, 0, sizeof(rev_list));\n \trev_list.argv = argv;\n@@ -92,3 +102,19 @@ int check_everything_connected_with_transport(sha1_iterate_fn fn,\n \tsigchain_pop(SIGPIPE);\n \treturn finish_command(&rev_list) || err;\n }\n+\n+int check_everything_connected_with_transport(sha1_iterate_fn fn,\n+\t\t\t\t\t int quiet,\n+\t\t\t\t\t void *cb_data,\n+\t\t\t\t\t struct transport *transport)\n+{\n+\treturn check_everything_connected_real(fn, quiet, cb_data,\n+\t\t\t\t\t transport, NULL);\n+}\n+\n+int check_shallow_connected(sha1_iterate_fn fn, int quiet, void *cb_data,\n+\t\t\t const char *shallow_file)\n+{\n+\treturn check_everything_connected_real(fn, quiet, cb_data,\n+\t\t\t\t\t NULL, shallow_file);\n+}\ndiff --git a/connected.h b/connected.h\nindex 0b060b7..071d408 100644\n--- a/connected.h\n+++ b/connected.h\n@@ -18,6 +18,8 @@ typedef int (*sha1_iterate_fn)(void *, unsigned char [20]);\n * Return 0 if Ok, non zero otherwise (i.e. some missing objects)\n */\n extern int check_everything_connected(sha1_iterate_fn, int quiet, void *cb_data);\n+extern int check_shallow_connected(sha1_iterate_fn, int quiet, void *cb_data,\n+\t\t\t\t const char *shallow_file);\n extern int check_everything_connected_with_transport(sha1_iterate_fn, int quiet,\n \t\t\t\t\t\t void *cb_data,\n \t\t\t\t\t\t struct transport *transport);\n-- \n1.8.5.1.25.g8667982\n"},{"id":"231604","messageId":"1386248575-10206-21-git-send-email-pclouds@gmail.com","threadId":"35400","inReplyTo":"1386248575-10206-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v4 20/28] receive-pack: allow pushes that update .git/shallow","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-12-05T13:02:47Z","receivedAt":"2013-12-05T13:02:47Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"The basic 8 steps to update .git/shallow does not fully apply here\nbecause the user may choose to accept just a few refs (while fetch\nalways accepts all refs). The steps are modified a bit.\n\n1-6. same as before. After calling assign_shallow_commits_to_refs at\n step 6, each shallow commit has a bitmap that marks all refs that\n require it.\n\n7. mark all \"ours\" shallow commits that are reachable from any\n refs. We will need to do the original step 7 on them later.\n\n8. go over all shallow commit bitmaps, mark refs that require new\n shallow commits.\n\n9. setup a strict temporary shallow file to plug all the holes, even\n if it may cut some of our history short. This file is used by all\n hooks. The hooks could use --shallow-file=$GIT_DIR/shallow to\n overcome this and reach everything in current repo.\n\n10. go over the new refs one by one. For each ref, do the reachability\n test if it needs a shallow commit on the list from step 7. Remove\n it if it's reachable from our refs. Gather all required shallow\n commits, run check_everything_connected() with the new ref, then\n install them to .git/shallow.\n\nThis mode is disabled by default and can be turned on with\nreceive.shallowupdate\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n Documentation/config.txt | 4 ++\n builtin/receive-pack.c | 163 +++++++++++++++++++++++++++++++++++++++++++----\n commit.h | 9 +++\n shallow.c | 23 +++++++\n t/t5537-push-shallow.sh | 15 +++++\n 5 files changed, 201 insertions(+), 13 deletions(-)\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex ab26963..1a0bd0d 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -2026,6 +2026,10 @@ receive.updateserverinfo::\n \tIf set to true, git-receive-pack will run git-update-server-info\n \tafter receiving data from git-push and updating refs.\n \n+receive.shallowupdate::\n+\tIf set to true, .git/shallow can be updated when new refs\n+\trequire new shallow roots. Otherwise those refs are rejected.\n+\n remote.pushdefault::\n \tThe remote to push to by default. Overrides\n \t`branch..remote` for all branches, and is overridden by\ndiff --git a/builtin/receive-pack.c b/builtin/receive-pack.c\nindex b9de2e8..5c85bb4 100644\n--- a/builtin/receive-pack.c\n+++ b/builtin/receive-pack.c\n@@ -44,6 +44,7 @@ static int fix_thin = 1;\n static const char *head_name;\n static void *head_name_to_free;\n static int sent_capabilities;\n+static int shallow_update;\n static const char *alt_shallow_file;\n \n static enum deny_action parse_deny_action(const char *var, const char *value)\n@@ -123,6 +124,11 @@ static int receive_pack_config(const char *var, const char *value, void *cb)\n \t\treturn 0;\n \t}\n \n+\tif (strcmp(var, \"receive.shallowupdate\") == 0) {\n+\t\tshallow_update = git_config_bool(var, value);\n+\t\treturn 0;\n+\t}\n+\n \treturn git_default_config(var, value, cb);\n }\n \n@@ -423,7 +429,46 @@ static void refuse_unconfigured_deny_delete_current(void)\n \t\trp_error(\"%s\", refuse_unconfigured_deny_delete_current_msg[i]);\n }\n \n-static const char *update(struct command *cmd)\n+static int command_singleton_iterator(void *cb_data, unsigned char sha1[20]);\n+static int update_shallow_ref(struct command *cmd, struct shallow_info *si)\n+{\n+\tstatic struct lock_file shallow_lock;\n+\tstruct sha1_array extra = SHA1_ARRAY_INIT;\n+\tconst char *alt_file;\n+\tuint32_t mask = 1 << (cmd->index % 32);\n+\tint i;\n+\n+\ttrace_printf_key(\"GIT_TRACE_SHALLOW\",\n+\t\t\t \"shallow: update_shallow_ref %s\\n\", cmd->ref_name);\n+\tfor (i = 0; i < si->shallow->nr; i++)\n+\t\tif (si->used_shallow[i] &&\n+\t\t (si->used_shallow[i][cmd->index / 32] & mask) &&\n+\t\t !delayed_reachability_test(si, i))\n+\t\t\tsha1_array_append(&extra, si->shallow->sha1[i]);\n+\n+\tsetup_alternate_shallow(&shallow_lock, &alt_file, &extra);\n+\tif (check_shallow_connected(command_singleton_iterator,\n+\t\t\t\t 0, cmd, alt_file)) {\n+\t\trollback_lock_file(&shallow_lock);\n+\t\tsha1_array_clear(&extra);\n+\t\treturn -1;\n+\t}\n+\n+\tcommit_lock_file(&shallow_lock);\n+\n+\t/*\n+\t * Make sure setup_alternate_shallow() for the next ref does\n+\t * not lose these new roots..\n+\t */\n+\tfor (i = 0; i < extra.nr; i++)\n+\t\tregister_shallow(extra.sha1[i]);\n+\n+\tsi->shallow_ref[cmd->index] = 0;\n+\tsha1_array_clear(&extra);\n+\treturn 0;\n+}\n+\n+static const char *update(struct command *cmd, struct shallow_info *si)\n {\n \tconst char *name = cmd->ref_name;\n \tstruct strbuf namespaced_name_buf = STRBUF_INIT;\n@@ -531,6 +576,10 @@ static const char *update(struct command *cmd)\n \t\treturn NULL; /* good */\n \t}\n \telse {\n+\t\tif (shallow_update && si->shallow_ref[cmd->index] &&\n+\t\t update_shallow_ref(cmd, si))\n+\t\t\treturn \"shallow error\";\n+\n \t\tlock = lock_any_ref_for_update(namespaced_name, old_sha1,\n \t\t\t\t\t 0, NULL);\n \t\tif (!lock) {\n@@ -671,12 +720,16 @@ static int command_singleton_iterator(void *cb_data, unsigned char sha1[20])\n \treturn 0;\n }\n \n-static void set_connectivity_errors(struct command *commands)\n+static void set_connectivity_errors(struct command *commands,\n+\t\t\t\t struct shallow_info *si)\n {\n \tstruct command *cmd;\n \n \tfor (cmd = commands; cmd; cmd = cmd->next) {\n \t\tstruct command *singleton = cmd;\n+\t\tif (shallow_update && si->shallow_ref[cmd->index])\n+\t\t\t/* to be checked in update_shallow_ref() */\n+\t\t\tcontinue;\n \t\tif (!check_everything_connected(command_singleton_iterator,\n \t\t\t\t\t\t0, &singleton))\n \t\t\tcontinue;\n@@ -684,18 +737,26 @@ static void set_connectivity_errors(struct command *commands)\n \t}\n }\n \n+struct iterate_data {\n+\tstruct command *cmds;\n+\tstruct shallow_info *si;\n+};\n+\n static int iterate_receive_command_list(void *cb_data, unsigned char sha1[20])\n {\n-\tstruct command **cmd_list = cb_data;\n+\tstruct iterate_data *data = cb_data;\n+\tstruct command **cmd_list = &data->cmds;\n \tstruct command *cmd = *cmd_list;\n \n-\twhile (cmd) {\n+\tfor (; cmd; cmd = cmd->next) {\n+\t\tif (shallow_update && data->si->shallow_ref[cmd->index])\n+\t\t\t/* to be checked in update_shallow_ref() */\n+\t\t\tcontinue;\n \t\tif (!is_null_sha1(cmd->new_sha1) && !cmd->skip_update) {\n \t\t\thashcpy(sha1, cmd->new_sha1);\n \t\t\t*cmd_list = cmd->next;\n \t\t\treturn 0;\n \t\t}\n-\t\tcmd = cmd->next;\n \t}\n \t*cmd_list = NULL;\n \treturn -1; /* end of list */\n@@ -715,10 +776,14 @@ static void reject_updates_to_hidden(struct command *commands)\n \t}\n }\n \n-static void execute_commands(struct command *commands, const char *unpacker_error)\n+static void execute_commands(struct command *commands,\n+\t\t\t const char *unpacker_error,\n+\t\t\t struct shallow_info *si)\n {\n+\tint checked_connectivity;\n \tstruct command *cmd;\n \tunsigned char sha1[20];\n+\tstruct iterate_data data;\n \n \tif (unpacker_error) {\n \t\tfor (cmd = commands; cmd; cmd = cmd->next)\n@@ -726,10 +791,10 @@ static void execute_commands(struct command *commands, const char *unpacker_erro\n \t\treturn;\n \t}\n \n-\tcmd = commands;\n-\tif (check_everything_connected(iterate_receive_command_list,\n-\t\t\t\t 0, &cmd))\n-\t\tset_connectivity_errors(commands);\n+\tdata.cmds = commands;\n+\tdata.si = si;\n+\tif (check_everything_connected(iterate_receive_command_list, 0, &data))\n+\t\tset_connectivity_errors(commands, si);\n \n \treject_updates_to_hidden(commands);\n \n@@ -746,6 +811,7 @@ static void execute_commands(struct command *commands, const char *unpacker_erro\n \tfree(head_name_to_free);\n \thead_name = head_name_to_free = resolve_refdup(\"HEAD\", sha1, 0, NULL);\n \n+\tchecked_connectivity = 1;\n \tfor (cmd = commands; cmd; cmd = cmd->next) {\n \t\tif (cmd->error_string)\n \t\t\tcontinue;\n@@ -753,7 +819,22 @@ static void execute_commands(struct command *commands, const char *unpacker_erro\n \t\tif (cmd->skip_update)\n \t\t\tcontinue;\n \n-\t\tcmd->error_string = update(cmd);\n+\t\tcmd->error_string = update(cmd, si);\n+\t\tif (shallow_update && !cmd->error_string &&\n+\t\t si->shallow_ref[cmd->index]) {\n+\t\t\terror(\"BUG: connectivity check has not been run on ref %s\",\n+\t\t\t cmd->ref_name);\n+\t\t\tchecked_connectivity = 0;\n+\t\t}\n+\t}\n+\n+\tif (shallow_update) {\n+\t\tif (!checked_connectivity)\n+\t\t\terror(\"BUG: run 'git fsck' for safety.\\n\"\n+\t\t\t \"If there are errors, try to remove \"\n+\t\t\t \"the reported refs above\");\n+\t\tif (alt_shallow_file && *alt_shallow_file)\n+\t\t\tunlink(alt_shallow_file);\n \t}\n }\n \n@@ -924,6 +1005,53 @@ static const char *unpack_with_sideband(struct shallow_info *si)\n \treturn ret;\n }\n \n+static void prepare_shallow_update(struct command *commands,\n+\t\t\t\t struct shallow_info *si)\n+{\n+\tint i, j, k, bitmap_size = (si->ref->nr + 31) / 32;\n+\n+\tsi->used_shallow = xmalloc(sizeof(*si->used_shallow) *\n+\t\t\t\t si->shallow->nr);\n+\tassign_shallow_commits_to_refs(si, si->used_shallow, NULL);\n+\n+\tsi->need_reachability_test =\n+\t\txcalloc(si->shallow->nr, sizeof(*si->need_reachability_test));\n+\tsi->reachable =\n+\t\txcalloc(si->shallow->nr, sizeof(*si->reachable));\n+\tsi->shallow_ref = xcalloc(si->ref->nr, sizeof(*si->shallow_ref));\n+\n+\tfor (i = 0; i < si->nr_ours; i++)\n+\t\tsi->need_reachability_test[si->ours[i]] = 1;\n+\n+\tfor (i = 0; i < si->shallow->nr; i++) {\n+\t\tif (!si->used_shallow[i])\n+\t\t\tcontinue;\n+\t\tfor (j = 0; j < bitmap_size; j++) {\n+\t\t\tif (!si->used_shallow[i][j])\n+\t\t\t\tcontinue;\n+\t\t\tsi->need_reachability_test[i]++;\n+\t\t\tfor (k = 0; k < 32; k++)\n+\t\t\t\tif (si->used_shallow[i][j] & (1 << k))\n+\t\t\t\t\tsi->shallow_ref[j * 32 + k]++;\n+\t\t}\n+\n+\t\t/*\n+\t\t * true for those associated with some refs and belong\n+\t\t * in \"ours\" list aka \"step 7 not done yet\"\n+\t\t */\n+\t\tsi->need_reachability_test[i] =\n+\t\t\tsi->need_reachability_test[i] > 1;\n+\t}\n+\n+\t/*\n+\t * keep hooks happy by forcing a temporary shallow file via\n+\t * env variable because we can't add --shallow-file to every\n+\t * command. check_everything_connected() will be done with\n+\t * true .git/shallow though.\n+\t */\n+\tsetenv(GIT_SHALLOW_FILE_ENVIRONMENT, alt_shallow_file, 1);\n+}\n+\n static void update_shallow_info(struct command *commands,\n \t\t\t\tstruct shallow_info *si,\n \t\t\t\tstruct sha1_array *ref)\n@@ -932,8 +1060,10 @@ static void update_shallow_info(struct command *commands,\n \tint *ref_status;\n \tremove_nonexistent_theirs_shallow(si);\n \t/* XXX remove_nonexistent_ours_in_pack() */\n-\tif (!si->nr_ours && !si->nr_theirs)\n+\tif (!si->nr_ours && !si->nr_theirs) {\n+\t\tshallow_update = 0;\n \t\treturn;\n+\t}\n \n \tfor (cmd = commands; cmd; cmd = cmd->next) {\n \t\tif (is_null_sha1(cmd->new_sha1))\n@@ -943,6 +1073,11 @@ static void update_shallow_info(struct command *commands,\n \t}\n \tsi->ref = ref;\n \n+\tif (shallow_update) {\n+\t\tprepare_shallow_update(commands, si);\n+\t\treturn;\n+\t}\n+\n \tref_status = xmalloc(sizeof(*ref_status) * ref->nr);\n \tassign_shallow_commits_to_refs(si, NULL, ref_status);\n \tfor (cmd = commands; cmd; cmd = cmd->next) {\n@@ -1064,11 +1199,13 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix)\n \t\tconst char *unpack_status = NULL;\n \n \t\tprepare_shallow_info(&si, &shallow);\n+\t\tif (!si.nr_ours && !si.nr_theirs)\n+\t\t\tshallow_update = 0;\n \t\tif (!delete_only(commands)) {\n \t\t\tunpack_status = unpack_with_sideband(&si);\n \t\t\tupdate_shallow_info(commands, &si, &ref);\n \t\t}\n-\t\texecute_commands(commands, unpack_status);\n+\t\texecute_commands(commands, unpack_status, &si);\n \t\tif (pack_lockfile)\n \t\t\tunlink_or_warn(pack_lockfile);\n \t\tif (report_status)\ndiff --git a/commit.h b/commit.h\nindex 79649ef..a1f2d49 100644\n--- a/commit.h\n+++ b/commit.h\n@@ -216,6 +216,14 @@ struct shallow_info {\n \tint *ours, nr_ours;\n \tint *theirs, nr_theirs;\n \tstruct sha1_array *ref;\n+\n+\t/* for receive-pack */\n+\tuint32_t **used_shallow;\n+\tint *need_reachability_test;\n+\tint *reachable;\n+\tint *shallow_ref;\n+\tstruct commit **commits;\n+\tint nr_commits;\n };\n \n extern void prepare_shallow_info(struct shallow_info *, struct sha1_array *);\n@@ -226,6 +234,7 @@ extern void remove_nonexistent_ours_in_pack(struct shallow_info *,\n extern void assign_shallow_commits_to_refs(struct shallow_info *info,\n \t\t\t\t\t uint32_t **used,\n \t\t\t\t\t int *ref_status);\n+extern int delayed_reachability_test(struct shallow_info *si, int c);\n \n int is_descendant_of(struct commit *, struct commit_list *);\n int in_merge_bases(struct commit *, struct commit *);\ndiff --git a/shallow.c b/shallow.c\nindex 5710690..e611639 100644\n--- a/shallow.c\n+++ b/shallow.c\n@@ -618,3 +618,26 @@ static void post_assign_shallow(struct shallow_info *info,\n \n \tfree(ca.commits);\n }\n+\n+/* (Delayed) step 7, reachability test at commit level */\n+int delayed_reachability_test(struct shallow_info *si, int c)\n+{\n+\tif (si->need_reachability_test[c]) {\n+\t\tstruct commit *commit = lookup_commit(si->shallow->sha1[c]);\n+\n+\t\tif (!si->commits) {\n+\t\t\tstruct commit_array ca;\n+\t\t\tmemset(&ca, 0, sizeof(ca));\n+\t\t\thead_ref(add_ref, &ca);\n+\t\t\tfor_each_ref(add_ref, &ca);\n+\t\t\tsi->commits = ca.commits;\n+\t\t\tsi->nr_commits = ca.nr;\n+\t\t}\n+\n+\t\tsi->reachable[c] =in_merge_bases_many(commit,\n+\t\t\t\t\t\t si->nr_commits,\n+\t\t\t\t\t\t si->commits);\n+\t\tsi->need_reachability_test[c] = 0;\n+\t}\n+\treturn si->reachable[c];\n+}\ndiff --git a/t/t5537-push-shallow.sh b/t/t5537-push-shallow.sh\nindex 650c31a..ff5eb5b 100755\n--- a/t/t5537-push-shallow.sh\n+++ b/t/t5537-push-shallow.sh\n@@ -67,4 +67,19 @@ test_expect_success 'push from shallow clone, with grafted roots' '\n \tgit fsck\n '\n \n+test_expect_success 'add new shallow root with receive.updateshallow on' '\n+\ttest_config receive.shallowupdate true &&\n+\t(\n+\tcd shallow2 &&\n+\tgit push ../.git +master:refs/remotes/shallow2/master\n+\t) &&\n+\tgit log --format=%s shallow2/master >actual &&\n+\tgit fsck &&\n+\tcat <expect &&\n+c\n+b\n+EOF\n+\ttest_cmp expect actual\n+'\n+\n test_done\n-- \n1.8.5.1.25.g8667982\n"},{"id":"231605","messageId":"1386248575-10206-22-git-send-email-pclouds@gmail.com","threadId":"35400","inReplyTo":"1386248575-10206-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v4 21/28] send-pack: support pushing to a shallow clone","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-12-05T13:02:48Z","receivedAt":"2013-12-05T13:02:48Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Signed-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/send-pack.c | 4 +++-\n t/t5537-push-shallow.sh | 38 ++++++++++++++++++++++++++++++++++++++\n transport.c | 5 ++---\n 3 files changed, 43 insertions(+), 4 deletions(-)\n\ndiff --git a/builtin/send-pack.c b/builtin/send-pack.c\nindex ea2ab28..664dd20 100644\n--- a/builtin/send-pack.c\n+++ b/builtin/send-pack.c\n@@ -101,6 +101,7 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)\n \tint fd[2];\n \tstruct child_process *conn;\n \tstruct sha1_array extra_have = SHA1_ARRAY_INIT;\n+\tstruct sha1_array shallow = SHA1_ARRAY_INIT;\n \tstruct ref *remote_refs, *local_refs;\n \tint ret;\n \tint helper_status = 0;\n@@ -232,7 +233,8 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)\n \t\t\targs.verbose ? CONNECT_VERBOSE : 0);\n \t}\n \n-\tget_remote_heads(fd[0], NULL, 0, &remote_refs, REF_NORMAL, &extra_have, NULL);\n+\tget_remote_heads(fd[0], NULL, 0, &remote_refs, REF_NORMAL,\n+\t\t\t &extra_have, &shallow);\n \n \ttransport_verify_remote_names(nr_refspecs, refspecs);\n \ndiff --git a/t/t5537-push-shallow.sh b/t/t5537-push-shallow.sh\nindex ff5eb5b..f5c74e6 100755\n--- a/t/t5537-push-shallow.sh\n+++ b/t/t5537-push-shallow.sh\n@@ -82,4 +82,42 @@ EOF\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'push from shallow to shallow' '\n+\t(\n+\tcd shallow &&\n+\tgit --git-dir=../shallow2/.git config receive.shallowupdate true &&\n+\tgit push ../shallow2/.git +master:refs/remotes/shallow/master &&\n+\tgit --git-dir=../shallow2/.git config receive.shallowupdate false\n+\t) &&\n+\t(\n+\tcd shallow2 &&\n+\tgit log --format=%s shallow/master >actual &&\n+\tgit fsck &&\n+\tcat <expect &&\n+5\n+4\n+3\n+EOF\n+\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'push from full to shallow' '\n+\t! git --git-dir=shallow2/.git cat-file blob `echo 1|git hash-object --stdin` &&\n+\tcommit 1 &&\n+\tgit push shallow2/.git +master:refs/remotes/top/master &&\n+\t(\n+\tcd shallow2 &&\n+\tgit log --format=%s top/master >actual &&\n+\tgit fsck &&\n+\tcat <expect &&\n+1\n+4\n+3\n+EOF\n+\ttest_cmp expect actual &&\n+\tgit cat-file blob `echo 1|git hash-object --stdin` >/dev/null\n+\t)\n+'\n+\n test_done\ndiff --git a/transport.c b/transport.c\nindex a09fdb6..d596abb 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -819,11 +819,10 @@ static int git_transport_push(struct transport *transport, struct ref *remote_re\n \t\tstruct ref *tmp_refs;\n \t\tconnect_setup(transport, 1, 0);\n \n-\t\tget_remote_heads(data->fd[0], NULL, 0, &tmp_refs, REF_NORMAL, NULL, NULL);\n+\t\tget_remote_heads(data->fd[0], NULL, 0, &tmp_refs, REF_NORMAL,\n+\t\t\t\t NULL, &data->shallow);\n \t\tdata->got_remote_heads = 1;\n \t}\n-\tif (data->shallow.nr)\n-\t\tdie(\"pushing to a shallow repository is not supported\");\n \n \tmemset(&args, 0, sizeof(args));\n \targs.send_mirror = !!(flags & TRANSPORT_PUSH_MIRROR);\n-- \n1.8.5.1.25.g8667982\n"},{"id":"231606","messageId":"1386248575-10206-23-git-send-email-pclouds@gmail.com","threadId":"35400","inReplyTo":"1386248575-10206-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v4 22/28] remote-curl: pass ref SHA-1 to fetch-pack as well","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-12-05T13:02:49Z","receivedAt":"2013-12-05T13:02:49Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Signed-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/fetch-pack.c | 7 +++++++\n remote-curl.c | 3 ++-\n 2 files changed, 9 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/fetch-pack.c b/builtin/fetch-pack.c\nindex 927424b..aa6e596 100644\n--- a/builtin/fetch-pack.c\n+++ b/builtin/fetch-pack.c\n@@ -13,6 +13,13 @@ static void add_sought_entry_mem(struct ref ***sought, int *nr, int *alloc,\n \t\t\t\t const char *name, int namelen)\n {\n \tstruct ref *ref = xcalloc(1, sizeof(*ref) + namelen + 1);\n+\tunsigned char sha1[20];\n+\n+\tif (namelen > 41 && name[40] == ' ' && !get_sha1_hex(name, sha1)) {\n+\t\thashcpy(ref->old_sha1, sha1);\n+\t\tname += 41;\n+\t\tnamelen -= 41;\n+\t}\n \n \tmemcpy(ref->name, name, namelen);\n \tref->name[namelen] = '\\0';\ndiff --git a/remote-curl.c b/remote-curl.c\nindex 222210f..25d6730 100644\n--- a/remote-curl.c\n+++ b/remote-curl.c\n@@ -719,7 +719,8 @@ static int fetch_git(struct discovery *heads,\n \t\tstruct ref *ref = to_fetch[i];\n \t\tif (!ref->name || !*ref->name)\n \t\t\tdie(\"cannot fetch by sha1 over smart http\");\n-\t\tpacket_buf_write(&preamble, \"%s\\n\", ref->name);\n+\t\tpacket_buf_write(&preamble, \"%s %s\\n\",\n+\t\t\t\t sha1_to_hex(ref->old_sha1), ref->name);\n \t}\n \tpacket_buf_flush(&preamble);\n \n-- \n1.8.5.1.25.g8667982\n"},{"id":"231607","messageId":"1386248575-10206-24-git-send-email-pclouds@gmail.com","threadId":"35400","inReplyTo":"1386248575-10206-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v4 23/28] Support shallow fetch/clone over smart-http","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-12-05T13:02:50Z","receivedAt":"2013-12-05T13:02:50Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Signed-off-by: Nguyễn Thái Ngọc Duy \n---\n Documentation/gitremote-helpers.txt | 7 +++++++\n builtin/fetch-pack.c | 16 +++++++++++++---\n remote-curl.c | 31 +++++++++++++++++++++++++++++--\n t/t5536-fetch-shallow.sh | 27 +++++++++++++++++++++++++++\n transport-helper.c | 6 ++++++\n upload-pack.c | 2 --\n 6 files changed, 82 insertions(+), 7 deletions(-)\n\ndiff --git a/Documentation/gitremote-helpers.txt b/Documentation/gitremote-helpers.txt\nindex f1f4ca9..c2908db 100644\n--- a/Documentation/gitremote-helpers.txt\n+++ b/Documentation/gitremote-helpers.txt\n@@ -437,6 +437,13 @@ set by Git if the remote helper has the 'option' capability.\n 'option check-connectivity' \\{'true'|'false'\\}::\n \tRequest the helper to check connectivity of a clone.\n \n+'option cloning \\{'true'|'false'\\}::\n+\tNotify the helper this is a clone request (i.e. the current\n+\trepository is guaranteed empty).\n+\n+'option update-shallow \\{'true'|'false'\\}::\n+\tAllow to extend .git/shallow if the new refs require it.\n+\n SEE ALSO\n --------\n linkgit:git-remote[1]\ndiff --git a/builtin/fetch-pack.c b/builtin/fetch-pack.c\nindex aa6e596..81fae38 100644\n--- a/builtin/fetch-pack.c\n+++ b/builtin/fetch-pack.c\n@@ -3,6 +3,7 @@\n #include \"fetch-pack.h\"\n #include \"remote.h\"\n #include \"connect.h\"\n+#include \"sha1-array.h\"\n \n static const char fetch_pack_usage[] =\n \"git fetch-pack [--all] [--stdin] [--quiet|-q] [--keep|-k] [--thin] \"\n@@ -46,6 +47,7 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)\n \tchar **pack_lockfile_ptr = NULL;\n \tstruct child_process *conn;\n \tstruct fetch_pack_args args;\n+\tstruct sha1_array shallow = SHA1_ARRAY_INIT;\n \n \tpacket_trace_identity(\"fetch-pack\");\n \n@@ -113,6 +115,14 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)\n \t\t\targs.check_self_contained_and_connected = 1;\n \t\t\tcontinue;\n \t\t}\n+\t\tif (!strcmp(\"--cloning\", arg)) {\n+\t\t\targs.cloning = 1;\n+\t\t\tcontinue;\n+\t\t}\n+\t\tif (!strcmp(\"--update-shallow\", arg)) {\n+\t\t\targs.update_shallow = 1;\n+\t\t\tcontinue;\n+\t\t}\n \t\tusage(fetch_pack_usage);\n \t}\n \n@@ -157,10 +167,10 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)\n \t\t\t\t args.verbose ? CONNECT_VERBOSE : 0);\n \t}\n \n-\tget_remote_heads(fd[0], NULL, 0, &ref, 0, NULL, NULL);\n+\tget_remote_heads(fd[0], NULL, 0, &ref, 0, NULL, &shallow);\n \n-\tref = fetch_pack(&args, fd, conn, ref, dest,\n-\t\t\t sought, nr_sought, NULL, pack_lockfile_ptr);\n+\tref = fetch_pack(&args, fd, conn, ref, dest, sought, nr_sought,\n+\t\t\t &shallow, pack_lockfile_ptr);\n \tif (pack_lockfile) {\n \t\tprintf(\"lock %s\\n\", pack_lockfile);\n \t\tfflush(stdout);\ndiff --git a/remote-curl.c b/remote-curl.c\nindex 25d6730..3d97e3d 100644\n--- a/remote-curl.c\n+++ b/remote-curl.c\n@@ -10,6 +10,7 @@\n #include \"sideband.h\"\n #include \"argv-array.h\"\n #include \"credential.h\"\n+#include \"sha1-array.h\"\n \n static struct remote *remote;\n /* always ends with a trailing slash */\n@@ -20,6 +21,8 @@ struct options {\n \tunsigned long depth;\n \tunsigned progress : 1,\n \t\tcheck_self_contained_and_connected : 1,\n+\t\tcloning : 1,\n+\t\tupdate_shallow : 1,\n \t\tfollowtags : 1,\n \t\tdry_run : 1,\n \t\tthin : 1;\n@@ -88,6 +91,24 @@ static int set_option(const char *name, const char *value)\n \t\tstrbuf_release(&val);\n \t\treturn 0;\n \t}\n+\telse if (!strcmp(name, \"cloning\")) {\n+\t\tif (!strcmp(value, \"true\"))\n+\t\t\toptions.cloning = 1;\n+\t\telse if (!strcmp(value, \"false\"))\n+\t\t\toptions.cloning = 0;\n+\t\telse\n+\t\t\treturn -1;\n+\t\treturn 0;\n+\t}\n+\telse if (!strcmp(name, \"update-shallow\")) {\n+\t\tif (!strcmp(value, \"true\"))\n+\t\t\toptions.update_shallow = 1;\n+\t\telse if (!strcmp(value, \"false\"))\n+\t\t\toptions.update_shallow = 0;\n+\t\telse\n+\t\t\treturn -1;\n+\t\treturn 0;\n+\t}\n \telse {\n \t\treturn 1 /* unsupported */;\n \t}\n@@ -99,6 +120,7 @@ struct discovery {\n \tchar *buf;\n \tsize_t len;\n \tstruct ref *refs;\n+\tstruct sha1_array shallow;\n \tunsigned proto_git : 1;\n };\n static struct discovery *last_discovery;\n@@ -107,7 +129,7 @@ static struct ref *parse_git_refs(struct discovery *heads, int for_push)\n {\n \tstruct ref *list = NULL;\n \tget_remote_heads(-1, heads->buf, heads->len, &list,\n-\t\t\t for_push ? REF_NORMAL : 0, NULL, NULL);\n+\t\t\t for_push ? REF_NORMAL : 0, NULL, &heads->shallow);\n \treturn list;\n }\n \n@@ -168,6 +190,7 @@ static void free_discovery(struct discovery *d)\n \tif (d) {\n \t\tif (d == last_discovery)\n \t\t\tlast_discovery = NULL;\n+\t\tfree(d->shallow.sha1);\n \t\tfree(d->buf_alloc);\n \t\tfree_refs(d->refs);\n \t\tfree(d);\n@@ -688,7 +711,7 @@ static int fetch_git(struct discovery *heads,\n \tstruct strbuf preamble = STRBUF_INIT;\n \tchar *depth_arg = NULL;\n \tint argc = 0, i, err;\n-\tconst char *argv[16];\n+\tconst char *argv[17];\n \n \targv[argc++] = \"fetch-pack\";\n \targv[argc++] = \"--stateless-rpc\";\n@@ -704,6 +727,10 @@ static int fetch_git(struct discovery *heads,\n \t}\n \tif (options.check_self_contained_and_connected)\n \t\targv[argc++] = \"--check-self-contained-and-connected\";\n+\tif (options.cloning)\n+\t\targv[argc++] = \"--cloning\";\n+\tif (options.update_shallow)\n+\t\targv[argc++] = \"--update-shallow\";\n \tif (!options.progress)\n \t\targv[argc++] = \"--no-progress\";\n \tif (options.depth) {\ndiff --git a/t/t5536-fetch-shallow.sh b/t/t5536-fetch-shallow.sh\nindex 3ae9092..79ce472 100755\n--- a/t/t5536-fetch-shallow.sh\n+++ b/t/t5536-fetch-shallow.sh\n@@ -173,4 +173,31 @@ EOF\n \t)\n '\n \n+if test -n \"$NO_CURL\" -o -z \"$GIT_TEST_HTTPD\"; then\n+\tsay 'skipping remaining tests, git built without http support'\n+\ttest_done\n+fi\n+\n+LIB_HTTPD_PORT=${LIB_HTTPD_PORT-'5536'}\n+. \"$TEST_DIRECTORY\"/lib-httpd.sh\n+start_httpd\n+\n+test_expect_success 'clone http repository' '\n+\tgit clone --bare --no-local shallow \"$HTTPD_DOCUMENT_ROOT_PATH/repo.git\" &&\n+\tgit clone $HTTPD_URL/smart/repo.git clone &&\n+\t(\n+\tcd clone &&\n+\tgit fsck &&\n+\tgit log --format=%s origin/master >actual &&\n+\tcat <expect &&\n+6\n+5\n+4\n+3\n+EOF\n+\ttest_cmp expect actual\n+\t)\n+'\n+\n+stop_httpd\n test_done\ndiff --git a/transport-helper.c b/transport-helper.c\nindex 673b7c2..e2b4203 100644\n--- a/transport-helper.c\n+++ b/transport-helper.c\n@@ -360,6 +360,12 @@ static int fetch_with_fetch(struct transport *transport,\n \t data->transport_options.check_self_contained_and_connected)\n \t\tset_helper_option(transport, \"check-connectivity\", \"true\");\n \n+\tif (transport->cloning)\n+\t\tset_helper_option(transport, \"cloning\", \"true\");\n+\n+\tif (data->transport_options.update_shallow)\n+\t\tset_helper_option(transport, \"update-shallow\", \"true\");\n+\n \tfor (i = 0; i < nr_heads; i++) {\n \t\tconst struct ref *posn = to_fetch[i];\n \t\tif (posn->status & REF_STATUS_UPTODATE)\ndiff --git a/upload-pack.c b/upload-pack.c\nindex 28269c7..2d02297 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -836,8 +836,6 @@ int main(int argc, char **argv)\n \n \tif (!enter_repo(dir, strict))\n \t\tdie(\"'%s' does not appear to be a git repository\", dir);\n-\tif (is_repository_shallow() && stateless_rpc)\n-\t\tdie(\"attempt to push into a shallow repository\");\n \n \tgit_config(upload_pack_config, NULL);\n \tupload_pack();\n-- \n1.8.5.1.25.g8667982\n"},{"id":"231608","messageId":"1386248575-10206-25-git-send-email-pclouds@gmail.com","threadId":"35400","inReplyTo":"1386248575-10206-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v4 24/28] receive-pack: support pushing to a shallow clone via http","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-12-05T13:02:51Z","receivedAt":"2013-12-05T13:02:51Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Signed-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/receive-pack.c | 3 ---\n t/t5537-push-shallow.sh | 35 +++++++++++++++++++++++++++++++++++\n 2 files changed, 35 insertions(+), 3 deletions(-)\n\ndiff --git a/builtin/receive-pack.c b/builtin/receive-pack.c\nindex 5c85bb4..78fe8ee 100644\n--- a/builtin/receive-pack.c\n+++ b/builtin/receive-pack.c\n@@ -1179,9 +1179,6 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix)\n \tif (!enter_repo(dir, 0))\n \t\tdie(\"'%s' does not appear to be a git repository\", dir);\n \n-\tif (is_repository_shallow() && stateless_rpc)\n-\t\tdie(\"attempt to push into a shallow repository\");\n-\n \tgit_config(receive_pack_config, NULL);\n \n \tif (0 <= transfer_unpack_limit)\ndiff --git a/t/t5537-push-shallow.sh b/t/t5537-push-shallow.sh\nindex f5c74e6..866621a 100755\n--- a/t/t5537-push-shallow.sh\n+++ b/t/t5537-push-shallow.sh\n@@ -16,6 +16,7 @@ test_expect_success 'setup' '\n \tcommit 2 &&\n \tcommit 3 &&\n \tcommit 4 &&\n+\tgit clone . full &&\n \t(\n \tgit init full-abc &&\n \tcd full-abc &&\n@@ -120,4 +121,38 @@ EOF\n \t)\n '\n \n+if test -n \"$NO_CURL\" -o -z \"$GIT_TEST_HTTPD\"; then\n+\tsay 'skipping remaining tests, git built without http support'\n+\ttest_done\n+fi\n+\n+LIB_HTTPD_PORT=${LIB_HTTPD_PORT-'5537'}\n+. \"$TEST_DIRECTORY\"/lib-httpd.sh\n+start_httpd\n+\n+test_expect_success 'push to shallow repo via http' '\n+\tgit clone --bare --no-local shallow \"$HTTPD_DOCUMENT_ROOT_PATH/repo.git\" &&\n+\t(\n+\tcd \"$HTTPD_DOCUMENT_ROOT_PATH/repo.git\" &&\n+\tgit config http.receivepack true\n+\t) &&\n+\t(\n+\tcd full &&\n+\tcommit 9 &&\n+\tgit push $HTTPD_URL/smart/repo.git +master:refs/remotes/top/master\n+\t) &&\n+\t(\n+\tcd \"$HTTPD_DOCUMENT_ROOT_PATH/repo.git\" &&\n+\tgit fsck &&\n+\tgit log --format=%s top/master >actual &&\n+\tcat <expect &&\n+9\n+4\n+3\n+EOF\n+\ttest_cmp expect actual\n+\t)\n+'\n+\n+stop_httpd\n test_done\n-- \n1.8.5.1.25.g8667982\n"},{"id":"231609","messageId":"1386248575-10206-26-git-send-email-pclouds@gmail.com","threadId":"35400","inReplyTo":"1386248575-10206-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v4 25/28] send-pack: support pushing from a shallow clone via http","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-12-05T13:02:52Z","receivedAt":"2013-12-05T13:02:52Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Signed-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/send-pack.c | 3 ---\n send-pack.c | 19 +++++++++++++++++--\n t/t5537-push-shallow.sh | 25 +++++++++++++++++++++++++\n 3 files changed, 42 insertions(+), 5 deletions(-)\n\ndiff --git a/builtin/send-pack.c b/builtin/send-pack.c\nindex 664dd20..cc25744 100644\n--- a/builtin/send-pack.c\n+++ b/builtin/send-pack.c\n@@ -209,9 +209,6 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)\n \t (send_all && args.send_mirror))\n \t\tusage(send_pack_usage);\n \n-\tif (is_repository_shallow() && args.stateless_rpc)\n-\t\tdie(\"attempt to push from a shallow repository\");\n-\n \tif (remote_name) {\n \t\tremote = remote_get(remote_name);\n \t\tif (!remote_has_url(remote, dest)) {\ndiff --git a/send-pack.c b/send-pack.c\nindex cd536b4..848d15e 100644\n--- a/send-pack.c\n+++ b/send-pack.c\n@@ -175,6 +175,21 @@ static int sideband_demux(int in, int out, void *data)\n \treturn ret;\n }\n \n+static int advertise_shallow_grafts_cb(const struct commit_graft *graft, void *cb)\n+{\n+\tstruct strbuf *sb = cb;\n+\tif (graft->nr_parent == -1)\n+\t\tpacket_buf_write(sb, \"shallow %s\\n\", sha1_to_hex(graft->sha1));\n+\treturn 0;\n+}\n+\n+void advertise_shallow_grafts_buf(struct strbuf *sb)\n+{\n+\tif (!is_repository_shallow())\n+\t\treturn;\n+\tfor_each_commit_graft(advertise_shallow_grafts_cb, sb);\n+}\n+\n int send_pack(struct send_pack_args *args,\n \t int fd[], struct child_process *conn,\n \t struct ref *remote_refs,\n@@ -215,7 +230,7 @@ int send_pack(struct send_pack_args *args,\n \t}\n \n \tif (!args->dry_run)\n-\t\tadvertise_shallow_grafts(out);\n+\t\tadvertise_shallow_grafts_buf(&req_buf);\n \n \t/*\n \t * Finally, tell the other end!\n@@ -276,7 +291,7 @@ int send_pack(struct send_pack_args *args,\n \t}\n \n \tif (args->stateless_rpc) {\n-\t\tif (!args->dry_run && cmds_sent) {\n+\t\tif (!args->dry_run && (cmds_sent || is_repository_shallow())) {\n \t\t\tpacket_buf_flush(&req_buf);\n \t\t\tsend_sideband(out, -1, req_buf.buf, req_buf.len, LARGE_PACKET_MAX);\n \t\t}\ndiff --git a/t/t5537-push-shallow.sh b/t/t5537-push-shallow.sh\nindex 866621a..0a6e40f 100755\n--- a/t/t5537-push-shallow.sh\n+++ b/t/t5537-push-shallow.sh\n@@ -154,5 +154,30 @@ EOF\n \t)\n '\n \n+test_expect_success 'push from shallow repo via http' '\n+\tmv \"$HTTPD_DOCUMENT_ROOT_PATH/repo.git\" shallow-upstream.git &&\n+\tgit clone --bare --no-local full \"$HTTPD_DOCUMENT_ROOT_PATH/repo.git\" &&\n+\t(\n+\tcd \"$HTTPD_DOCUMENT_ROOT_PATH/repo.git\" &&\n+\tgit config http.receivepack true\n+\t) &&\n+\tcommit 10 &&\n+\tgit push $HTTPD_URL/smart/repo.git +master:refs/remotes/top/master &&\n+\t(\n+\tcd \"$HTTPD_DOCUMENT_ROOT_PATH/repo.git\" &&\n+\tgit fsck &&\n+\tgit log --format=%s top/master >actual &&\n+\tcat <expect &&\n+10\n+1\n+4\n+3\n+2\n+1\n+EOF\n+\ttest_cmp expect actual\n+\t)\n+'\n+\n stop_httpd\n test_done\n-- \n1.8.5.1.25.g8667982\n"},{"id":"231610","messageId":"1386248575-10206-27-git-send-email-pclouds@gmail.com","threadId":"35400","inReplyTo":"1386248575-10206-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v4 26/28] clone: use git protocol for cloning shallow repo locally","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-12-05T13:02:53Z","receivedAt":"2013-12-05T13:02:53Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"clone_local() does not handle $SRC/shallow. It could be made so, but\nit's simpler to use fetch-pack/upload-pack instead.\n\nThis used to be caught by the check in upload-pack, which is triggered\nby transport_get_remote_refs(), even in local clone case. The check is\nnow gone and check_everything_connected() should catch the result\nincomplete repo. But check_everything_connected() will soon be skipped\nin local clone case, opening a door to corrupt repo. This patch should\nclose that door.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/clone.c | 11 +++++++++--\n t/t5601-clone.sh | 7 +++++++\n 2 files changed, 16 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex 0b182ce..71ee68b 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -797,8 +797,15 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \telse\n \t\trepo = repo_name;\n \tis_local = option_local != 0 && path && !is_bundle;\n-\tif (is_local && option_depth)\n-\t\twarning(_(\"--depth is ignored in local clones; use file:// instead.\"));\n+\tif (is_local) {\n+\t\tif (option_depth)\n+\t\t\twarning(_(\"--depth is ignored in local clones; use file:// instead.\"));\n+\t\tif (!access(mkpath(\"%s/shallow\", path), F_OK)) {\n+\t\t\tif (option_local > 0)\n+\t\t\t\twarning(_(\"source repository is shallow, ignoring --local\"));\n+\t\t\tis_local = 0;\n+\t\t}\n+\t}\n \tif (option_local > 0 && !is_local)\n \t\twarning(_(\"--local is ignored\"));\n \ndiff --git a/t/t5601-clone.sh b/t/t5601-clone.sh\nindex 1d1c875..c226cff 100755\n--- a/t/t5601-clone.sh\n+++ b/t/t5601-clone.sh\n@@ -340,4 +340,11 @@ test_expect_success 'clone from a repository with two identical branches' '\n \n '\n \n+test_expect_success 'shallow clone locally' '\n+\tgit clone --depth=1 --no-local src ssrrcc &&\n+\tgit clone ssrrcc ddsstt &&\n+\ttest_cmp ssrrcc/.git/shallow ddsstt/.git/shallow &&\n+\t( cd ddsstt && git fsck )\n+'\n+\n test_done\n-- \n1.8.5.1.25.g8667982\n"},{"id":"231611","messageId":"1386248575-10206-28-git-send-email-pclouds@gmail.com","threadId":"35400","inReplyTo":"1386248575-10206-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v4 27/28] prune: clean .git/shallow after pruning objects","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-12-05T13:02:54Z","receivedAt":"2013-12-05T13:02:54Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"This patch teaches \"prune\" to remove shallow roots that are no longer\nreachable from any refs (e.g. when the relevant refs are removed).\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n Documentation/git-prune.txt | 2 ++\n builtin/gc.c | 1 +\n builtin/prune.c | 4 ++++\n commit.h | 1 +\n shallow.c | 55 +++++++++++++++++++++++++++++++++++++++++++--\n t/t5304-prune.sh | 10 +++++++++\n 6 files changed, 71 insertions(+), 2 deletions(-)\n\ndiff --git a/Documentation/git-prune.txt b/Documentation/git-prune.txt\nindex bf82410..058ac0d 100644\n--- a/Documentation/git-prune.txt\n+++ b/Documentation/git-prune.txt\n@@ -24,6 +24,8 @@ objects unreachable from any of these head objects from the object database.\n In addition, it\n prunes the unpacked objects that are also found in packs by\n running 'git prune-packed'.\n+It also removes entries from .git/shallow that are not reachable by\n+any ref.\n \n Note that unreachable, packed objects will remain. If this is\n not desired, see linkgit:git-repack[1].\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex c14190f..cec8ecd 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -16,6 +16,7 @@\n #include \"run-command.h\"\n #include \"sigchain.h\"\n #include \"argv-array.h\"\n+#include \"commit.h\"\n \n #define FAILED_RUN \"failed to run %s\"\n \ndiff --git a/builtin/prune.c b/builtin/prune.c\nindex 6366917..2214040 100644\n--- a/builtin/prune.c\n+++ b/builtin/prune.c\n@@ -170,5 +170,9 @@ int cmd_prune(int argc, const char **argv, const char *prefix)\n \ts = mkpathdup(\"%s/pack\", get_object_directory());\n \tremove_temporary_files(s);\n \tfree(s);\n+\n+\tif (is_repository_shallow())\n+\t\tprune_shallow(show_only);\n+\n \treturn 0;\n }\ndiff --git a/commit.h b/commit.h\nindex a1f2d49..affe210 100644\n--- a/commit.h\n+++ b/commit.h\n@@ -235,6 +235,7 @@ extern void assign_shallow_commits_to_refs(struct shallow_info *info,\n \t\t\t\t\t uint32_t **used,\n \t\t\t\t\t int *ref_status);\n extern int delayed_reachability_test(struct shallow_info *si, int c);\n+extern void prune_shallow(int show_only);\n \n int is_descendant_of(struct commit *, struct commit_list *);\n int in_merge_bases(struct commit *, struct commit *);\ndiff --git a/shallow.c b/shallow.c\nindex e611639..f2ab72c 100644\n--- a/shallow.c\n+++ b/shallow.c\n@@ -155,10 +155,14 @@ void check_shallow_file_for_update(void)\n \t\tdie(\"shallow file was changed during fetch\");\n }\n \n+#define SEEN_ONLY 1\n+#define VERBOSE 2\n+\n struct write_shallow_data {\n \tstruct strbuf *out;\n \tint use_pack_protocol;\n \tint count;\n+\tunsigned flags;\n };\n \n static int write_one_shallow(const struct commit_graft *graft, void *cb_data)\n@@ -167,6 +171,15 @@ static int write_one_shallow(const struct commit_graft *graft, void *cb_data)\n \tconst char *hex = sha1_to_hex(graft->sha1);\n \tif (graft->nr_parent != -1)\n \t\treturn 0;\n+\tif (data->flags & SEEN_ONLY) {\n+\t\tstruct commit *c = lookup_commit(graft->sha1);\n+\t\tif (!c || !(c->object.flags & SEEN)) {\n+\t\t\tif (data->flags & VERBOSE)\n+\t\t\t\tprintf(\"Removing %s from .git/shallow\\n\",\n+\t\t\t\t sha1_to_hex(c->object.sha1));\n+\t\t\treturn 0;\n+\t\t}\n+\t}\n \tdata->count++;\n \tif (data->use_pack_protocol)\n \t\tpacket_buf_write(data->out, \"shallow %s\", hex);\n@@ -177,14 +190,16 @@ static int write_one_shallow(const struct commit_graft *graft, void *cb_data)\n \treturn 0;\n }\n \n-int write_shallow_commits(struct strbuf *out, int use_pack_protocol,\n-\t\t\t const struct sha1_array *extra)\n+static int write_shallow_commits_1(struct strbuf *out, int use_pack_protocol,\n+\t\t\t\t const struct sha1_array *extra,\n+\t\t\t\t unsigned flags)\n {\n \tstruct write_shallow_data data;\n \tint i;\n \tdata.out = out;\n \tdata.use_pack_protocol = use_pack_protocol;\n \tdata.count = 0;\n+\tdata.flags = flags;\n \tfor_each_commit_graft(write_one_shallow, &data);\n \tif (!extra)\n \t\treturn data.count;\n@@ -196,6 +211,12 @@ int write_shallow_commits(struct strbuf *out, int use_pack_protocol,\n \treturn data.count;\n }\n \n+int write_shallow_commits(struct strbuf *out, int use_pack_protocol,\n+\t\t\t const struct sha1_array *extra)\n+{\n+\treturn write_shallow_commits_1(out, use_pack_protocol, extra, 0);\n+}\n+\n char *setup_temporary_shallow(const struct sha1_array *extra)\n {\n \tstruct strbuf sb = STRBUF_INIT;\n@@ -258,6 +279,36 @@ void advertise_shallow_grafts(int fd)\n \tfor_each_commit_graft(advertise_shallow_grafts_cb, &fd);\n }\n \n+/*\n+ * mark_reachable_objects() should have been run prior to this and all\n+ * reachable commits marked as \"SEEN\".\n+ */\n+void prune_shallow(int show_only)\n+{\n+\tstatic struct lock_file shallow_lock;\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tint fd;\n+\n+\tif (show_only) {\n+\t\twrite_shallow_commits_1(&sb, 0, NULL, SEEN_ONLY | VERBOSE);\n+\t\tstrbuf_release(&sb);\n+\t\treturn;\n+\t}\n+\tcheck_shallow_file_for_update();\n+\tfd = hold_lock_file_for_update(&shallow_lock, git_path(\"shallow\"),\n+\t\t\t\t LOCK_DIE_ON_ERROR);\n+\tif (write_shallow_commits_1(&sb, 0, NULL, SEEN_ONLY)) {\n+\t\tif (write_in_full(fd, sb.buf, sb.len) != sb.len)\n+\t\t\tdie_errno(\"failed to write to %s\",\n+\t\t\t\t shallow_lock.filename);\n+\t\tcommit_lock_file(&shallow_lock);\n+\t} else {\n+\t\tunlink(git_path(\"shallow\"));\n+\t\trollback_lock_file(&shallow_lock);\n+\t}\n+\tstrbuf_release(&sb);\n+}\n+\n #define TRACE_KEY \"GIT_TRACE_SHALLOW\"\n \n /*\ndiff --git a/t/t5304-prune.sh b/t/t5304-prune.sh\nindex e4bb3a1..66c9a41 100755\n--- a/t/t5304-prune.sh\n+++ b/t/t5304-prune.sh\n@@ -221,4 +221,14 @@ EOF\n \ttest_cmp expected actual\n '\n \n+test_expect_success 'prune .git/shallow' '\n+\tSHA1=`echo hi|git commit-tree HEAD^{tree}` &&\n+\techo $SHA1 >.git/shallow &&\n+\tgit prune --dry-run >out &&\n+\tgrep $SHA1 .git/shallow &&\n+\tgrep $SHA1 out &&\n+\tgit prune &&\n+\t! test -f .git/shallow\n+'\n+\n test_done\n-- \n1.8.5.1.25.g8667982\n"},{"id":"231612","messageId":"1386248575-10206-29-git-send-email-pclouds@gmail.com","threadId":"35400","inReplyTo":"1386248575-10206-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v4 28/28] git-clone.txt: remove shallow clone limitations","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-12-05T13:02:55Z","receivedAt":"2013-12-05T13:02:55Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Now that git supports data transfer from or to a shallow clone, these\nlimitations are not true anymore.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n Documentation/git-clone.txt | 7 +------\n 1 file changed, 1 insertion(+), 6 deletions(-)\n\ndiff --git a/Documentation/git-clone.txt b/Documentation/git-clone.txt\nindex 450f158..4987857 100644\n--- a/Documentation/git-clone.txt\n+++ b/Documentation/git-clone.txt\n@@ -181,12 +181,7 @@ objects from the source repository into a pack in the cloned repository.\n \n --depth ::\n \tCreate a 'shallow' clone with a history truncated to the\n-\tspecified number of revisions. A shallow repository has a\n-\tnumber of limitations (you cannot clone or fetch from\n-\tit, nor push from nor into it), but is adequate if you\n-\tare only interested in the recent history of a large project\n-\twith a long history, and would want to send in fixes\n-\tas patches.\n+\tspecified number of revisions.\n \n --[no-]single-branch::\n \tClone only the history leading to the tip of a single branch,\n-- \n1.8.5.1.25.g8667982\n"},{"id":"232908","messageId":"20140108112543.GA4419@sigill.intra.peff.net","threadId":"35400","inReplyTo":"1386248575-10206-24-git-send-email-pclouds@gmail.com","subject":"Re: [PATCH v4 23/28] Support shallow fetch/clone over smart-http","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2014-01-08T11:25:43Z","receivedAt":"2014-01-08T11:25:43Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Dec 05, 2013 at 08:02:50PM +0700, Nguyễn Thái Ngọc Duy wrote:\n\n> Signed-off-by: Nguyễn Thái Ngọc Duy \n> ---\n> Documentation/gitremote-helpers.txt | 7 +++++++\n> builtin/fetch-pack.c | 16 +++++++++++++---\n> remote-curl.c | 31 +++++++++++++++++++++++++++++--\n> t/t5536-fetch-shallow.sh | 27 +++++++++++++++++++++++++++\n\nI'm getting test failures in 'next' with GIT_TEST_HTTPD set, and they\nare bisectable to this patch (actually, the moral equivalent of it, as\nit looks like the commit message was tweaked along with the test number\nwhen it was applied). The failures look like this:\n\n $ GIT_TEST_HTTPD=1 ./t5537-fetch-shallow.sh -v -i\n [...]\n ok 9 - fetch --update-shallow\n \n expecting success: \n git clone --bare --no-local shallow \"$HTTPD_DOCUMENT_ROOT_PATH/repo.git\" &&\n git clone $HTTPD_URL/smart/repo.git clone &&\n (\n cd clone &&\n git fsck &&\n git log --format=%s origin/master >actual &&\n cat <expect &&\n 6\n 5\n 4\n 3\n EOF\n test_cmp expect actual\n )\n \n Cloning into bare repository '/home/peff/compile/git/t/trash directory.t5537-fetch-shallow/httpd/www/repo.git'...\n remote: Counting objects: 19, done. \n remote: Compressing objects: 100% (7/7), done. \n remote: Total 19 (delta 0), reused 6 (delta 0) \n Receiving objects: 100% (19/19), done.\n Checking connectivity... done.\n Cloning into 'clone'...\n remote: Counting objects: 19, done. \n remote: Compressing objects: 100% (7/7), done. \n remote: Total 19 (delta 0), reused 19 (delta 0) \n Unpacking objects: 100% (19/19), done.\n Checking connectivity... done.\n Checking object directories: 100% (256/256), done.\n --- expect 2014-01-08 11:20:20.178546452 +0000\n +++ actual 2014-01-08 11:20:20.178546452 +0000\n @@ -1,3 +1,4 @@\n +7\n 6\n 5\n 4\n not ok 10 - clone http repository\n\n\nIf you do end up tweaking the test, you may also want to fix:\n\n> +LIB_HTTPD_PORT=${LIB_HTTPD_PORT-'5536'}\n> +. \"$TEST_DIRECTORY\"/lib-httpd.sh\n\nSince the test number got switched, it would be nice to tweak the port\nnumber to match it, in case the real t5536 ever starts using lib-httpd.\n\n-Peff\n"},{"id":"232910","messageId":"1389183199-6496-1-git-send-email-pclouds@gmail.com","threadId":"35400","inReplyTo":"20140108112543.GA4419@sigill.intra.peff.net","subject":"[PATCH] t5537: fix incorrect expectation in test case 10","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2014-01-08T12:13:19Z","receivedAt":"2014-01-08T12:13:19Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Commit 48d25ca adds a new commit \"7\" to the repo that the next test case\nin commit 1609488 clones from. But the next test case does not expect\nthis commit. For these tests, it's the bottom that's important, not\nthe top. Fix the expected commit list.\n\nWhile at it, fix the default http port number to 5537. Otherwise when\nt5536 learns to test httpd, running test in parallel may fail.\n\nReferences:\n\n48d25ca fetch: add --update-shallow to accept... - 2013-12-05\n1609488 smart-http: support shallow fetch/clone - 2013-12-05\n\nNoticed-by: Jeff King \nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n I obviously made a mistake with patch reordering or something. And\n embarassing because I did not run tests with GIT_TEST_HTTPD again\n before submission.\n\n t/t5537-fetch-shallow.sh | 3 ++-\n 1 file changed, 2 insertions(+), 1 deletion(-)\n\ndiff --git a/t/t5537-fetch-shallow.sh b/t/t5537-fetch-shallow.sh\nindex 79ce472..b0fa738 100755\n--- a/t/t5537-fetch-shallow.sh\n+++ b/t/t5537-fetch-shallow.sh\n@@ -178,7 +178,7 @@ if test -n \"$NO_CURL\" -o -z \"$GIT_TEST_HTTPD\"; then\n \ttest_done\n fi\n \n-LIB_HTTPD_PORT=${LIB_HTTPD_PORT-'5536'}\n+LIB_HTTPD_PORT=${LIB_HTTPD_PORT-'5537'}\n . \"$TEST_DIRECTORY\"/lib-httpd.sh\n start_httpd\n \n@@ -190,6 +190,7 @@ test_expect_success 'clone http repository' '\n \tgit fsck &&\n \tgit log --format=%s origin/master >actual &&\n \tcat <expect &&\n+7\n 6\n 5\n 4\n-- \n1.8.5.2.240.g8478abd\n"},{"id":"232971","messageId":"20140109215746.GC32069@sigill.intra.peff.net","threadId":"35400","inReplyTo":"1389183199-6496-1-git-send-email-pclouds@gmail.com","subject":"Re: [PATCH] t5537: fix incorrect expectation in test case 10","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2014-01-09T21:57:47Z","receivedAt":"2014-01-09T21:57:47Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Jan 08, 2014 at 07:13:19PM +0700, Nguyễn Thái Ngọc Duy wrote:\n\n> Commit 48d25ca adds a new commit \"7\" to the repo that the next test case\n> in commit 1609488 clones from. But the next test case does not expect\n> this commit. For these tests, it's the bottom that's important, not\n> the top. Fix the expected commit list.\n\nGiven the test output, I had a feeling it was something like this but\ndidn't dive in (and figured it would be obvious to you).\n\nPatch looks sane. Thanks for a quick turnaround.\n\n-Peff\n"}]}