{"thread":{"id":"34452","subject":"[PATCH 0/7] First class shallow clone","startedAt":"2013-07-17T12:47:07Z","lastAt":"2013-07-24T16:50:49Z","messageCount":44,"participants":["Nguyễn Thái Ngọc Duy","Junio C Hamano","Philip Oakley","Duy Nguyen","Piotr Krukowiecki"],"isPatch":true,"patchVersion":1,"patchTotal":7},"messages":[{"id":"223572","messageId":"1374065234-870-1-git-send-email-pclouds@gmail.com","threadId":"34452","inReplyTo":null,"subject":"[PATCH 0/7] First class shallow clone","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-07-17T12:47:07Z","receivedAt":"2013-07-17T12:47:07Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"This is probably the first attempt to treat shallow clones just like\nordinary ones. Which means you can push or fetch/clone between any two\nrepos, regardless of their shallow status. There are two purposes\nbehind this:\n\n - to make local/shallow clone <-> (complete) upstream repo workflow\n smoother, the note about shallow clone limitation in git-clone.txt\n can be removed\n\n - to make it possible for upstream to provide a lightweight repo that\n others can use. For example, big repos with lots of activities can\n be split into new base repo that only contains the work of maximum one\n year and a complete repo mostly for archive.\n\nThis is a naive approach. I might overlook something again, which is\nwhy I publish it early to get more eyes on it.\n\nThe idea is simple: in shallow case, we provide the pack _and_\n.git/shallow file to the other end. The other end will setup extra\ngrafting to make sure the updated repo is \"complete\". More in\nindividual patch messages.\n\nThere might be issues with generating optimum pack for transfer when\nboth ends are shallow.. There's also an interesting issue, whether we\ncan take advantage of commit bitmaps in shallow clones if they are\nmore widely used.. We also might need a config key to protect a repo\nfrom becoming shallow by a fetch or push, if the repo is to be a\nbackup one..\n\nNguyễn Thái Ngọc Duy (7):\n transport.h: remove send_pack prototype, already defined in send-pack.h\n {receive,upload}-pack: advertise shallow graft information\n connect.c: teach get_remote_heads to parse \"shallow\" lines\n Move setup_alternate_shallow and write_shallow_commits to shallow.c\n fetch-pack: support fetching from a shallow repository\n {send,receive}-pack: support pushing from a shallow clone\n send-pack: support pushing to a shallow clone\n\n Documentation/technical/pack-protocol.txt | 7 +-\n builtin/fetch-pack.c | 6 +-\n builtin/receive-pack.c | 54 ++++++++++++----\n builtin/send-pack.c | 7 +-\n cache.h | 1 +\n commit.h | 8 +++\n connect.c | 12 +++-\n fetch-pack.c | 78 +++++++---------------\n fetch-pack.h | 1 +\n remote-curl.c | 2 +-\n send-pack.c | 41 ++++++++++--\n send-pack.h | 4 +-\n shallow.c | 103 ++++++++++++++++++++++++++++++\n t/t5536-fetch-shallow.sh (new +x) | 75 ++++++++++++++++++++++\n t/t5537-push-shallow.sh (new +x) | 85 ++++++++++++++++++++++++\n transport.c | 14 ++--\n transport.h | 6 --\n upload-pack.c | 3 +-\n 18 files changed, 414 insertions(+), 93 deletions(-)\n create mode 100755 t/t5536-fetch-shallow.sh\n create mode 100755 t/t5537-push-shallow.sh\n\n-- \n1.8.2.83.gc99314b\n"},{"id":"223573","messageId":"1374065234-870-2-git-send-email-pclouds@gmail.com","threadId":"34452","inReplyTo":"1374065234-870-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 1/7] transport.h: remove send_pack prototype, already defined in send-pack.h","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-07-17T12:47:08Z","receivedAt":"2013-07-17T12:47:08Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n transport.h | 6 ------\n 1 file changed, 6 deletions(-)\n\ndiff --git a/transport.h b/transport.h\nindex ea70ea7..3178dc9 100644\n--- a/transport.h\n+++ b/transport.h\n@@ -182,10 +182,4 @@ void transport_print_push_status(const char *dest, struct ref *refs,\n \n typedef void alternate_ref_fn(const struct ref *, void *);\n extern void for_each_alternate_ref(alternate_ref_fn, void *);\n-\n-struct send_pack_args;\n-extern int send_pack(struct send_pack_args *args,\n-\t\t int fd[], struct child_process *conn,\n-\t\t struct ref *remote_refs,\n-\t\t struct extra_have_objects *extra_have);\n #endif\n-- \n1.8.2.83.gc99314b\n"},{"id":"223574","messageId":"1374065234-870-3-git-send-email-pclouds@gmail.com","threadId":"34452","inReplyTo":"1374065234-870-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 2/7] {receive,upload}-pack: advertise shallow graft information","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-07-17T12:47:09Z","receivedAt":"2013-07-17T12:47:09Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"If either receive-pack or upload-pack is called on a shallow\nrepository, shallow graft points will be sent after the ref\nadvertisement (but before the packet flush).\n\nThis breaks the protocol for all clients trying to push to a shallow\nrepo, or fetch from one. Which is basically the same end result as\ntoday's \"is_repository_shallow() && die()\" in receive-pack and\nupload-pack. New clients will be made aware of shallow upstream and\ncan make use of this information.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n Documentation/technical/pack-protocol.txt | 3 +++\n builtin/receive-pack.c | 5 ++---\n commit.h | 1 +\n shallow.c | 16 ++++++++++++++++\n upload-pack.c | 3 +--\n 5 files changed, 23 insertions(+), 5 deletions(-)\n\ndiff --git a/Documentation/technical/pack-protocol.txt b/Documentation/technical/pack-protocol.txt\nindex b898e97..eb8edd1 100644\n--- a/Documentation/technical/pack-protocol.txt\n+++ b/Documentation/technical/pack-protocol.txt\n@@ -161,6 +161,7 @@ MUST peel the ref if it's an annotated tag.\n \n ----\n advertised-refs = (no-refs / list-of-refs)\n+\t\t *shallow\n \t\t flush-pkt\n \n no-refs = PKT-LINE(zero-id SP \"capabilities^{}\"\n@@ -174,6 +175,8 @@ MUST peel the ref if it's an annotated tag.\n other-tip = obj-id SP refname LF\n other-peeled = obj-id SP refname \"^{}\" LF\n \n+ shallow = PKT-LINE(\"shallow\" SP obj-id)\n+\n capability-list = capability *(SP capability)\n capability = 1*(LC_ALPHA / DIGIT / \"-\" / \"_\")\n LC_ALPHA = %x61-7A\ndiff --git a/builtin/receive-pack.c b/builtin/receive-pack.c\nindex e3eb5fc..2d8e19b 100644\n--- a/builtin/receive-pack.c\n+++ b/builtin/receive-pack.c\n@@ -176,6 +176,8 @@ static void write_head_info(void)\n \tif (!sent_capabilities)\n \t\tshow_ref(\"capabilities^{}\", null_sha1);\n \n+\tadvertise_shallow_grafts(1);\n+\n \t/* EOF */\n \tpacket_flush(1);\n }\n@@ -990,9 +992,6 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix)\n \tif (!enter_repo(dir, 0))\n \t\tdie(\"'%s' does not appear to be a git repository\", dir);\n \n-\tif (is_repository_shallow())\n-\t\tdie(\"attempt to push into a shallow repository\");\n-\n \tgit_config(receive_pack_config, NULL);\n \n \tif (0 <= transfer_unpack_limit)\ndiff --git a/commit.h b/commit.h\nindex 4d452dc..e0688c3 100644\n--- a/commit.h\n+++ b/commit.h\n@@ -187,6 +187,7 @@ extern struct commit_list *get_shallow_commits(struct object_array *heads,\n \t\tint depth, int shallow_flag, int not_shallow_flag);\n extern void check_shallow_file_for_update(void);\n extern void set_alternate_shallow_file(const char *path);\n+extern void advertise_shallow_grafts(int);\n \n int is_descendant_of(struct commit *, struct commit_list *);\n int in_merge_bases(struct commit *, struct commit *);\ndiff --git a/shallow.c b/shallow.c\nindex cbe2526..ccdfefc 100644\n--- a/shallow.c\n+++ b/shallow.c\n@@ -1,6 +1,7 @@\n #include \"cache.h\"\n #include \"commit.h\"\n #include \"tag.h\"\n+#include \"pkt-line.h\"\n \n static int is_shallow = -1;\n static struct stat shallow_stat;\n@@ -146,3 +147,18 @@ void check_shallow_file_for_update(void)\n \t\t )\n \t\tdie(\"shallow file was changed during fetch\");\n }\n+\n+static int advertise_shallow_grafts_cb(const struct commit_graft *graft, void *cb)\n+{\n+\tint fd = *(int*)cb;\n+\tif (graft->nr_parent == -1)\n+\t\tpacket_write(fd, \"shallow %s\\n\", sha1_to_hex(graft->sha1));\n+\treturn 0;\n+}\n+\n+void advertise_shallow_grafts(int fd)\n+{\n+\tif (!is_repository_shallow())\n+\t\treturn;\n+\tfor_each_commit_graft(advertise_shallow_grafts_cb, &fd);\n+}\ndiff --git a/upload-pack.c b/upload-pack.c\nindex 127e59a..6cefe43 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -766,6 +766,7 @@ static void upload_pack(void)\n \t\treset_timeout();\n \t\thead_ref_namespaced(send_ref, NULL);\n \t\tfor_each_namespaced_ref(send_ref, NULL);\n+\t\tadvertise_shallow_grafts(1);\n \t\tpacket_flush(1);\n \t} else {\n \t\thead_ref_namespaced(mark_our_ref, NULL);\n@@ -837,8 +838,6 @@ int main(int argc, char **argv)\n \n \tif (!enter_repo(dir, strict))\n \t\tdie(\"'%s' does not appear to be a git repository\", dir);\n-\tif (is_repository_shallow())\n-\t\tdie(\"attempt to fetch/clone from a shallow repository\");\n \tgit_config(upload_pack_config, NULL);\n \tupload_pack();\n \treturn 0;\n-- \n1.8.2.83.gc99314b\n"},{"id":"223575","messageId":"1374065234-870-4-git-send-email-pclouds@gmail.com","threadId":"34452","inReplyTo":"1374065234-870-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 3/7] connect.c: teach get_remote_heads to parse \"shallow\" lines","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-07-17T12:47:10Z","receivedAt":"2013-07-17T12:47:10Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"No callers pass a non-empty pointer as shallow_points at this\nstage. As a result, all clients still refuse to talk to shallow\nrepository on the other end.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/fetch-pack.c | 2 +-\n builtin/send-pack.c | 2 +-\n cache.h | 1 +\n connect.c | 12 +++++++++++-\n remote-curl.c | 2 +-\n transport.c | 7 ++++---\n 6 files changed, 19 insertions(+), 7 deletions(-)\n\ndiff --git a/builtin/fetch-pack.c b/builtin/fetch-pack.c\nindex aba4465..080e599 100644\n--- a/builtin/fetch-pack.c\n+++ b/builtin/fetch-pack.c\n@@ -144,7 +144,7 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)\n \t\t\t\t args.verbose ? CONNECT_VERBOSE : 0);\n \t}\n \n-\tget_remote_heads(fd[0], NULL, 0, &ref, 0, NULL);\n+\tget_remote_heads(fd[0], NULL, 0, &ref, 0, NULL, NULL);\n \n \tref = fetch_pack(&args, fd, conn, ref, dest,\n \t\t\t sought, nr_sought, pack_lockfile_ptr);\ndiff --git a/builtin/send-pack.c b/builtin/send-pack.c\nindex 152c4ea..f0ad0ce 100644\n--- a/builtin/send-pack.c\n+++ b/builtin/send-pack.c\n@@ -207,7 +207,7 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)\n \n \tmemset(&extra_have, 0, sizeof(extra_have));\n \n-\tget_remote_heads(fd[0], NULL, 0, &remote_refs, REF_NORMAL, &extra_have);\n+\tget_remote_heads(fd[0], NULL, 0, &remote_refs, REF_NORMAL, &extra_have, NULL);\n \n \ttransport_verify_remote_names(nr_refspecs, refspecs);\n \ndiff --git a/cache.h b/cache.h\nindex dd0fb33..7665e03 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1091,6 +1091,7 @@ struct extra_have_objects {\n };\n extern struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,\n \t\t\t\t struct ref **list, unsigned int flags,\n+\t\t\t\t struct extra_have_objects *,\n \t\t\t\t struct extra_have_objects *);\n extern int server_supports(const char *feature);\n extern int parse_feature_request(const char *features, const char *feature);\ndiff --git a/connect.c b/connect.c\nindex a0783d4..1cba424 100644\n--- a/connect.c\n+++ b/connect.c\n@@ -64,7 +64,8 @@ static void die_initial_contact(int got_at_least_one_head)\n */\n struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,\n \t\t\t struct ref **list, unsigned int flags,\n-\t\t\t struct extra_have_objects *extra_have)\n+\t\t\t struct extra_have_objects *extra_have,\n+\t\t\t struct extra_have_objects *shallow_points)\n {\n \tint got_at_least_one_head = 0;\n \n@@ -89,6 +90,15 @@ struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,\n \t\tif (len > 4 && !prefixcmp(buffer, \"ERR \"))\n \t\t\tdie(\"remote error: %s\", buffer + 4);\n \n+\t\tif (len == 48 && !prefixcmp(buffer, \"shallow \")) {\n+\t\t\tif (get_sha1_hex(buffer + 8, old_sha1))\n+\t\t\t\tdie(\"protocol error: expected shallow sha, got '%s'\", buffer + 8);\n+\t\t\tif (!shallow_points)\n+\t\t\t\tdie(\"repository on the other end cannot be shallow\");\n+\t\t\tadd_extra_have(shallow_points, old_sha1);\n+\t\t\tcontinue;\n+\t\t}\n+\n \t\tif (len < 42 || get_sha1_hex(buffer, old_sha1) || buffer[40] != ' ')\n \t\t\tdie(\"protocol error: expected sha/ref, got '%s'\", buffer);\n \t\tname = buffer + 41;\ndiff --git a/remote-curl.c b/remote-curl.c\nindex 5b3ce9e..c329bd3 100644\n--- a/remote-curl.c\n+++ b/remote-curl.c\n@@ -86,7 +86,7 @@ static struct ref *parse_git_refs(struct discovery *heads, int for_push)\n {\n \tstruct ref *list = NULL;\n \tget_remote_heads(-1, heads->buf, heads->len, &list,\n-\t\t\t for_push ? REF_NORMAL : 0, NULL);\n+\t\t\t for_push ? REF_NORMAL : 0, NULL, NULL);\n \treturn list;\n }\n \ndiff --git a/transport.c b/transport.c\nindex e15db98..10a8cb8 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -509,7 +509,7 @@ static struct ref *get_refs_via_connect(struct transport *transport, int for_pus\n \n \tconnect_setup(transport, for_push, 0);\n \tget_remote_heads(data->fd[0], NULL, 0, &refs,\n-\t\t\t for_push ? REF_NORMAL : 0, &data->extra_have);\n+\t\t\t for_push ? REF_NORMAL : 0, &data->extra_have, NULL);\n \tdata->got_remote_heads = 1;\n \n \treturn refs;\n@@ -539,7 +539,8 @@ static int fetch_refs_via_pack(struct transport *transport,\n \n \tif (!data->got_remote_heads) {\n \t\tconnect_setup(transport, 0, 0);\n-\t\tget_remote_heads(data->fd[0], NULL, 0, &refs_tmp, 0, NULL);\n+\t\tget_remote_heads(data->fd[0], NULL, 0, &refs_tmp, 0,\n+\t\t\t\t NULL, NULL);\n \t\tdata->got_remote_heads = 1;\n \t}\n \n@@ -799,7 +800,7 @@ static int git_transport_push(struct transport *transport, struct ref *remote_re\n \t\tstruct ref *tmp_refs;\n \t\tconnect_setup(transport, 1, 0);\n \n-\t\tget_remote_heads(data->fd[0], NULL, 0, &tmp_refs, REF_NORMAL, NULL);\n+\t\tget_remote_heads(data->fd[0], NULL, 0, &tmp_refs, REF_NORMAL, NULL, NULL);\n \t\tdata->got_remote_heads = 1;\n \t}\n \n-- \n1.8.2.83.gc99314b\n"},{"id":"223576","messageId":"1374065234-870-5-git-send-email-pclouds@gmail.com","threadId":"34452","inReplyTo":"1374065234-870-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 4/7] Move setup_alternate_shallow and write_shallow_commits to shallow.c","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-07-17T12:47:11Z","receivedAt":"2013-07-17T12:47:11Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n commit.h | 3 +++\n fetch-pack.c | 53 +----------------------------------------------------\n shallow.c | 53 +++++++++++++++++++++++++++++++++++++++++++++++++++++\n 3 files changed, 57 insertions(+), 52 deletions(-)\n\ndiff --git a/commit.h b/commit.h\nindex e0688c3..678fa20 100644\n--- a/commit.h\n+++ b/commit.h\n@@ -188,6 +188,9 @@ extern struct commit_list *get_shallow_commits(struct object_array *heads,\n extern void check_shallow_file_for_update(void);\n extern void set_alternate_shallow_file(const char *path);\n extern void advertise_shallow_grafts(int);\n+extern int write_shallow_commits(struct strbuf *out, int use_pack_protocol);\n+extern void setup_alternate_shallow(struct lock_file *shallow_lock,\n+\t\t\t\t const char **alternate_shallow_file);\n \n int is_descendant_of(struct commit *, struct commit_list *);\n int in_merge_bases(struct commit *, struct commit *);\ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex abe5ffb..dc71a2b 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -185,36 +185,6 @@ static void consume_shallow_list(struct fetch_pack_args *args, int fd)\n \t}\n }\n \n-struct write_shallow_data {\n-\tstruct strbuf *out;\n-\tint use_pack_protocol;\n-\tint count;\n-};\n-\n-static int write_one_shallow(const struct commit_graft *graft, void *cb_data)\n-{\n-\tstruct write_shallow_data *data = cb_data;\n-\tconst char *hex = sha1_to_hex(graft->sha1);\n-\tdata->count++;\n-\tif (data->use_pack_protocol)\n-\t\tpacket_buf_write(data->out, \"shallow %s\", hex);\n-\telse {\n-\t\tstrbuf_addstr(data->out, hex);\n-\t\tstrbuf_addch(data->out, '\\n');\n-\t}\n-\treturn 0;\n-}\n-\n-static int write_shallow_commits(struct strbuf *out, int use_pack_protocol)\n-{\n-\tstruct write_shallow_data data;\n-\tdata.out = out;\n-\tdata.use_pack_protocol = use_pack_protocol;\n-\tdata.count = 0;\n-\tfor_each_commit_graft(write_one_shallow, &data);\n-\treturn data.count;\n-}\n-\n static enum ack_type get_ack(int fd, unsigned char *result_sha1)\n {\n \tint len;\n@@ -795,27 +765,6 @@ static int cmp_ref_by_name(const void *a_, const void *b_)\n \treturn strcmp(a->name, b->name);\n }\n \n-static void setup_alternate_shallow(void)\n-{\n-\tstruct strbuf sb = STRBUF_INIT;\n-\tint fd;\n-\n-\tcheck_shallow_file_for_update();\n-\tfd = hold_lock_file_for_update(&shallow_lock, git_path(\"shallow\"),\n-\t\t\t\t LOCK_DIE_ON_ERROR);\n-\tif (write_shallow_commits(&sb, 0)) {\n-\t\tif (write_in_full(fd, sb.buf, sb.len) != sb.len)\n-\t\t\tdie_errno(\"failed to write to %s\", shallow_lock.filename);\n-\t\talternate_shallow_file = shallow_lock.filename;\n-\t} else\n-\t\t/*\n-\t\t * is_repository_shallow() sees empty string as \"no\n-\t\t * shallow file\".\n-\t\t */\n-\t\talternate_shallow_file = \"\";\n-\tstrbuf_release(&sb);\n-}\n-\n static struct ref *do_fetch_pack(struct fetch_pack_args *args,\n \t\t\t\t int fd[2],\n \t\t\t\t const struct ref *orig_ref,\n@@ -896,7 +845,7 @@ static struct ref *do_fetch_pack(struct fetch_pack_args *args,\n \tif (args->stateless_rpc)\n \t\tpacket_flush(fd[1]);\n \tif (args->depth > 0)\n-\t\tsetup_alternate_shallow();\n+\t\tsetup_alternate_shallow(&shallow_lock, &alternate_shallow_file);\n \tif (get_pack(args, fd, pack_lockfile))\n \t\tdie(\"git fetch-pack: fetch failed.\");\n \ndiff --git a/shallow.c b/shallow.c\nindex ccdfefc..ee9edd4 100644\n--- a/shallow.c\n+++ b/shallow.c\n@@ -162,3 +162,56 @@ void advertise_shallow_grafts(int fd)\n \t\treturn;\n \tfor_each_commit_graft(advertise_shallow_grafts_cb, &fd);\n }\n+\n+struct write_shallow_data {\n+\tstruct strbuf *out;\n+\tint use_pack_protocol;\n+\tint count;\n+};\n+\n+static int write_one_shallow(const struct commit_graft *graft, void *cb_data)\n+{\n+\tstruct write_shallow_data *data = cb_data;\n+\tconst char *hex = sha1_to_hex(graft->sha1);\n+\tdata->count++;\n+\tif (data->use_pack_protocol)\n+\t\tpacket_buf_write(data->out, \"shallow %s\", hex);\n+\telse {\n+\t\tstrbuf_addstr(data->out, hex);\n+\t\tstrbuf_addch(data->out, '\\n');\n+\t}\n+\treturn 0;\n+}\n+\n+int write_shallow_commits(struct strbuf *out, int use_pack_protocol)\n+{\n+\tstruct write_shallow_data data;\n+\tdata.out = out;\n+\tdata.use_pack_protocol = use_pack_protocol;\n+\tdata.count = 0;\n+\tfor_each_commit_graft(write_one_shallow, &data);\n+\treturn data.count;\n+}\n+\n+void setup_alternate_shallow(struct lock_file *shallow_lock,\n+\t\t\t const char **alternate_shallow_file)\n+{\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tint fd;\n+\n+\tcheck_shallow_file_for_update();\n+\tfd = hold_lock_file_for_update(shallow_lock, git_path(\"shallow\"),\n+\t\t\t\t LOCK_DIE_ON_ERROR);\n+\tif (write_shallow_commits(&sb, 0)) {\n+\t\tif (write_in_full(fd, sb.buf, sb.len) != sb.len)\n+\t\t\tdie_errno(\"failed to write to %s\",\n+\t\t\t\t shallow_lock->filename);\n+\t\t*alternate_shallow_file = shallow_lock->filename;\n+\t} else\n+\t\t/*\n+\t\t * is_repository_shallow() sees empty string as \"no\n+\t\t * shallow file\".\n+\t\t */\n+\t\t*alternate_shallow_file = \"\";\n+\tstrbuf_release(&sb);\n+}\n-- \n1.8.2.83.gc99314b\n"},{"id":"223578","messageId":"1374065234-870-6-git-send-email-pclouds@gmail.com","threadId":"34452","inReplyTo":"1374065234-870-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 5/7] fetch-pack: support fetching from a shallow repository","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-07-17T12:47:12Z","receivedAt":"2013-07-17T12:47:12Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"upload-pack already advertises all shallow grafts if server repository\nis shallow. This information can be used to add more grafts to the\nclient if the server sends commit chains down to its graft points.\n\nIf the server is shallow, before we receive the pack, we setup a\ntemporary shallow file that contains both local graft points and the\nserver's. This stops index-pack from going beyond server's graft\npoints.\n\nOnly server graft points that do not have corresponding SHA-1s in\nlocal repo are added to the temp shallow file because we don't want to\naccidentally cut the client history because the server's is\nshorter. The client cutting can only happen when --depth is requested.\n\nAfter index-pack finishes successfully, we write the temporary shallow\ndown with one exception: unused graft points provided by the server\nare removed. We don't want those lying around and suddenly become\nactive.\n\nNote that in the \"shallow -> shallow\" case, the server might not have\nenough information to find common roots to create an optimum pack. It\nmight send complete commit chains down to the graft points as a\nresult. I don't think we can improve this, unless upload-pack somehow\nhas access to a full repository.\n\n\"shallow -> shallow\" case only makes sense when the upstream provides\na stable shallow repo (e.g. make a cut every year or so and ask devs\nto all move to the new base). If the cloned repos are all based on a\nstable (shallow) upstream, the above problem is unlikely to happen.\n\nA side effect of this change is we can now clone from a shallow\nrepository. And a full repository may automatically become shallow if\nyou fetch from a shallow repository.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/fetch-pack.c | 6 ++--\n commit.h | 8 +++--\n fetch-pack.c | 27 +++++++++++---\n fetch-pack.h | 1 +\n shallow.c | 46 ++++++++++++++++++++----\n t/t5536-fetch-shallow.sh (new +x) | 75 +++++++++++++++++++++++++++++++++++++++\n transport.c | 8 +++--\n 7 files changed, 153 insertions(+), 18 deletions(-)\n create mode 100755 t/t5536-fetch-shallow.sh\n\ndiff --git a/builtin/fetch-pack.c b/builtin/fetch-pack.c\nindex 080e599..b89d753 100644\n--- a/builtin/fetch-pack.c\n+++ b/builtin/fetch-pack.c\n@@ -37,6 +37,7 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)\n \tchar **pack_lockfile_ptr = NULL;\n \tstruct child_process *conn;\n \tstruct fetch_pack_args args;\n+\tstruct extra_have_objects shallow;\n \n \tpacket_trace_identity(\"fetch-pack\");\n \n@@ -144,10 +145,11 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)\n \t\t\t\t args.verbose ? CONNECT_VERBOSE : 0);\n \t}\n \n-\tget_remote_heads(fd[0], NULL, 0, &ref, 0, NULL, NULL);\n+\tmemset(&shallow, 0, sizeof(shallow));\n+\tget_remote_heads(fd[0], NULL, 0, &ref, 0, NULL, &shallow);\n \n \tref = fetch_pack(&args, fd, conn, ref, dest,\n-\t\t\t sought, nr_sought, pack_lockfile_ptr);\n+\t\t\t sought, nr_sought, &shallow, pack_lockfile_ptr);\n \tif (pack_lockfile) {\n \t\tprintf(\"lock %s\\n\", pack_lockfile);\n \t\tfflush(stdout);\ndiff --git a/commit.h b/commit.h\nindex 678fa20..7faf0e4 100644\n--- a/commit.h\n+++ b/commit.h\n@@ -188,9 +188,13 @@ extern struct commit_list *get_shallow_commits(struct object_array *heads,\n extern void check_shallow_file_for_update(void);\n extern void set_alternate_shallow_file(const char *path);\n extern void advertise_shallow_grafts(int);\n-extern int write_shallow_commits(struct strbuf *out, int use_pack_protocol);\n+extern int write_shallow_commits(struct strbuf *out, int use_pack_protocol,\n+\t\t\t\t struct extra_have_objects *extra,\n+\t\t\t\t int remove_unused_grafts);\n extern void setup_alternate_shallow(struct lock_file *shallow_lock,\n-\t\t\t\t const char **alternate_shallow_file);\n+\t\t\t\t const char **alternate_shallow_file,\n+\t\t\t\t struct extra_have_objects *extra,\n+\t\t\t\t int rewrite);\n \n int is_descendant_of(struct commit *, struct commit_list *);\n int in_merge_bases(struct commit *, struct commit *);\ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex dc71a2b..68b95a5 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -311,7 +311,7 @@ static int find_common(struct fetch_pack_args *args,\n \t}\n \n \tif (is_repository_shallow())\n-\t\twrite_shallow_commits(&req_buf, 1);\n+\t\twrite_shallow_commits(&req_buf, 1, NULL, 0);\n \tif (args->depth > 0)\n \t\tpacket_buf_write(&req_buf, \"deepen %d\", args->depth);\n \tpacket_buf_flush(&req_buf);\n@@ -769,6 +769,7 @@ static struct ref *do_fetch_pack(struct fetch_pack_args *args,\n \t\t\t\t int fd[2],\n \t\t\t\t const struct ref *orig_ref,\n \t\t\t\t struct ref **sought, int nr_sought,\n+\t\t\t\t struct extra_have_objects *shallow,\n \t\t\t\t char **pack_lockfile)\n {\n \tstruct ref *ref = copy_ref_list(orig_ref);\n@@ -844,8 +845,9 @@ static struct ref *do_fetch_pack(struct fetch_pack_args *args,\n \n \tif (args->stateless_rpc)\n \t\tpacket_flush(fd[1]);\n-\tif (args->depth > 0)\n-\t\tsetup_alternate_shallow(&shallow_lock, &alternate_shallow_file);\n+\tif (args->depth > 0 || shallow->nr)\n+\t\tsetup_alternate_shallow(&shallow_lock, &alternate_shallow_file,\n+\t\t\t\t\tshallow, 0);\n \tif (get_pack(args, fd, pack_lockfile))\n \t\tdie(\"git fetch-pack: fetch failed.\");\n \n@@ -922,6 +924,7 @@ struct ref *fetch_pack(struct fetch_pack_args *args,\n \t\t const struct ref *ref,\n \t\t const char *dest,\n \t\t struct ref **sought, int nr_sought,\n+\t\t struct extra_have_objects *shallow,\n \t\t char **pack_lockfile)\n {\n \tstruct ref *ref_cpy;\n@@ -934,14 +937,28 @@ struct ref *fetch_pack(struct fetch_pack_args *args,\n \t\tpacket_flush(fd[1]);\n \t\tdie(\"no matching remote head\");\n \t}\n-\tref_cpy = do_fetch_pack(args, fd, ref, sought, nr_sought, pack_lockfile);\n+\tref_cpy = do_fetch_pack(args, fd, ref, sought, nr_sought,\n+\t\t\t\tshallow, pack_lockfile);\n \n \tif (alternate_shallow_file) {\n \t\tif (*alternate_shallow_file == '\\0') { /* --unshallow */\n \t\t\tunlink_or_warn(git_path(\"shallow\"));\n \t\t\trollback_lock_file(&shallow_lock);\n-\t\t} else\n+\t\t} else {\n+\t\t\t/*\n+\t\t\t * The server is a shallow clone and it sends\n+\t\t\t * us all of its shallow grafts. Some may be\n+\t\t\t * needed if the server sends objects down to\n+\t\t\t * the bottom. Remove all unused grafts.\n+\t\t\t */\n+\t\t\tif (shallow->nr) {\n+\t\t\t\treprepare_packed_git();\n+\t\t\t\tsetup_alternate_shallow(&shallow_lock,\n+\t\t\t\t\t\t\t&alternate_shallow_file,\n+\t\t\t\t\t\t\tshallow, 1);\n+\t\t\t}\n \t\t\tcommit_lock_file(&shallow_lock);\n+\t\t}\n \t}\n \n \treprepare_packed_git();\ndiff --git a/fetch-pack.h b/fetch-pack.h\nindex 40f08ba..c55bf1c 100644\n--- a/fetch-pack.h\n+++ b/fetch-pack.h\n@@ -32,6 +32,7 @@ struct ref *fetch_pack(struct fetch_pack_args *args,\n \t\t const char *dest,\n \t\t struct ref **sought,\n \t\t int nr_sought,\n+\t\t struct extra_have_objects *shallow,\n \t\t char **pack_lockfile);\n \n #endif\ndiff --git a/shallow.c b/shallow.c\nindex ee9edd4..d909d95 100644\n--- a/shallow.c\n+++ b/shallow.c\n@@ -183,26 +183,60 @@ static int write_one_shallow(const struct commit_graft *graft, void *cb_data)\n \treturn 0;\n }\n \n-int write_shallow_commits(struct strbuf *out, int use_pack_protocol)\n+int write_shallow_commits(struct strbuf *out, int use_pack_protocol,\n+\t\t\t struct extra_have_objects *extra,\n+\t\t\t int remove_unused_grafts)\n {\n \tstruct write_shallow_data data;\n+\tint i;\n \tdata.out = out;\n \tdata.use_pack_protocol = use_pack_protocol;\n \tdata.count = 0;\n \tfor_each_commit_graft(write_one_shallow, &data);\n+\tif (!extra)\n+\t\treturn data.count;\n+\n+\tfor (i = 0; i < extra->nr; i++) {\n+\t\tif (!remove_unused_grafts && has_sha1_file(extra->array[i]))\n+\t\t\t/*\n+\t\t\t * The server may have even shorter history\n+\t\t\t * than the client (e.g. the client has\n+\t\t\t * A-B-C-D but the server only has C-D). We do\n+\t\t\t * NOT want to cut client's history down to\n+\t\t\t * C-D simply because the server is set up\n+\t\t\t * so. If we don't have \"C\" and the server\n+\t\t\t * sends C-D to us, then we set up a graft at\n+\t\t\t * \"C\".\n+\t\t\t */\n+\t\t\tcontinue;\n+\t\tif (remove_unused_grafts && !has_sha1_file(extra->array[i]))\n+\t\t\tcontinue;\n+\t\tstrbuf_addstr(out, sha1_to_hex(extra->array[i]));\n+\t\tstrbuf_addch(out, '\\n');\n+\t\tdata.count++;\n+\t}\n \treturn data.count;\n }\n \n void setup_alternate_shallow(struct lock_file *shallow_lock,\n-\t\t\t const char **alternate_shallow_file)\n+\t\t\t const char **alternate_shallow_file,\n+\t\t\t struct extra_have_objects *extra,\n+\t\t\t int rewrite)\n {\n \tstruct strbuf sb = STRBUF_INIT;\n \tint fd;\n \n-\tcheck_shallow_file_for_update();\n-\tfd = hold_lock_file_for_update(shallow_lock, git_path(\"shallow\"),\n-\t\t\t\t LOCK_DIE_ON_ERROR);\n-\tif (write_shallow_commits(&sb, 0)) {\n+\tif (!rewrite)\n+\t\tfd = hold_lock_file_for_update(shallow_lock,\n+\t\t\t\t\t git_path(\"shallow\"),\n+\t\t\t\t\t LOCK_DIE_ON_ERROR);\n+\telse\n+\t\tfd = shallow_lock->fd;\n+\tif (write_shallow_commits(&sb, 0, extra, rewrite)) {\n+\t\tif (rewrite && (lseek(fd, 0, SEEK_SET) == -1 ||\n+\t\t\t\tftruncate(fd, 0) == -1))\n+\t\t\tdie_errno(\"unable to truncate the new shallow file %s\",\n+\t\t\t\t shallow_lock->filename);\n \t\tif (write_in_full(fd, sb.buf, sb.len) != sb.len)\n \t\t\tdie_errno(\"failed to write to %s\",\n \t\t\t\t shallow_lock->filename);\ndiff --git a/t/t5536-fetch-shallow.sh b/t/t5536-fetch-shallow.sh\nnew file mode 100755\nindex 0000000..599928f\n--- /dev/null\n+++ b/t/t5536-fetch-shallow.sh\n@@ -0,0 +1,75 @@\n+#!/bin/sh\n+\n+test_description='fetch/clone from a shallow clone'\n+\n+. ./test-lib.sh\n+\n+test_expect_success 'setup' '\n+\ttest_commit 1 &&\n+\ttest_commit 2 &&\n+\ttest_commit 3 &&\n+\ttest_commit 4\n+'\n+\n+test_expect_success 'setup shallow clone' '\n+\tgit clone --no-local --depth=2 .git shallow &&\n+\tgit --git-dir=shallow/.git log --format=%s >actual &&\n+\tcat <expect &&\n+4\n+3\n+EOF\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'clone from shallow clone' '\n+\tgit clone --no-local shallow shallow2 &&\n+\t(\n+\tcd shallow2 &&\n+\tgit fsck &&\n+\tgit log --format=%s >actual &&\n+\tcat <expect &&\n+4\n+3\n+EOF\n+\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'fetch from shallow clone' '\n+\t(\n+\tcd shallow &&\n+\ttest_commit 5\n+\t) &&\n+\t(\n+\tcd shallow2 &&\n+\tgit fetch &&\n+\tgit fsck &&\n+\tgit log --format=%s origin/master >actual &&\n+\tcat <expect &&\n+5\n+4\n+3\n+EOF\n+\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'fetch --depth from shallow clone' '\n+\t(\n+\tcd shallow &&\n+\ttest_commit 6\n+\t) &&\n+\t(\n+\tcd shallow2 &&\n+\tgit fetch --depth=2 &&\n+\tgit fsck &&\n+\tgit log --format=%s origin/master >actual &&\n+\tcat <expect &&\n+6\n+5\n+EOF\n+\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_done\ndiff --git a/transport.c b/transport.c\nindex 10a8cb8..17fef16 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -453,6 +453,7 @@ struct git_transport_data {\n \tint fd[2];\n \tunsigned got_remote_heads : 1;\n \tstruct extra_have_objects extra_have;\n+\tstruct extra_have_objects shallow;\n };\n \n static int set_git_option(struct git_transport_options *opts,\n@@ -509,7 +510,8 @@ static struct ref *get_refs_via_connect(struct transport *transport, int for_pus\n \n \tconnect_setup(transport, for_push, 0);\n \tget_remote_heads(data->fd[0], NULL, 0, &refs,\n-\t\t\t for_push ? REF_NORMAL : 0, &data->extra_have, NULL);\n+\t\t\t for_push ? REF_NORMAL : 0,\n+\t\t\t &data->extra_have, &data->shallow);\n \tdata->got_remote_heads = 1;\n \n \treturn refs;\n@@ -540,13 +542,13 @@ static int fetch_refs_via_pack(struct transport *transport,\n \tif (!data->got_remote_heads) {\n \t\tconnect_setup(transport, 0, 0);\n \t\tget_remote_heads(data->fd[0], NULL, 0, &refs_tmp, 0,\n-\t\t\t\t NULL, NULL);\n+\t\t\t\t NULL, &data->shallow);\n \t\tdata->got_remote_heads = 1;\n \t}\n \n \trefs = fetch_pack(&args, data->fd, data->conn,\n \t\t\t refs_tmp ? refs_tmp : transport->remote_refs,\n-\t\t\t dest, to_fetch, nr_heads,\n+\t\t\t dest, to_fetch, nr_heads, &data->shallow,\n \t\t\t &transport->pack_lockfile);\n \tclose(data->fd[0]);\n \tclose(data->fd[1]);\n-- \n1.8.2.83.gc99314b\n"},{"id":"223577","messageId":"1374065234-870-7-git-send-email-pclouds@gmail.com","threadId":"34452","inReplyTo":"1374065234-870-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 6/7] {send,receive}-pack: support pushing from a shallow clone","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-07-17T12:47:13Z","receivedAt":"2013-07-17T12:47:13Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Pushing from a shallow clone using today's send-pack and receive-pack\nmay work, if the transferred pack does not ends up at any graft\npoints. If it does, recent receive-pack that does connectivity check\nwill reject the push. If receive-pack is old, the upstream repo\nbecomes corrupt.\n\nThe pack protocol is updated and send-pack now sends all shallow\ngrafts before it sends the commands, if the repo is shallow. This\nprotocol extension will break current receive-pack, which is intended,\nmostly to stop corrupting the upstream repo.\n\nThe receiver end, the newreceive-pack, does something similar to\nfetch-pack: it creates a temporary shallow file with grafts from\nsend-pack, then receives the pack, and finally writes the refined\nshallow file down.\n\nshadow file is not cleaned up after deleting (or force updating) a ref\nif that ref is the only way to reach the graft points. The reason is\nonce we delete graft points, we can't recover. That may make reflog\nentries on server useless. Leave that for the administrators to decide\nwhen to clean up shadow file (maybe at repack/gc time).\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n Documentation/technical/pack-protocol.txt | 4 +-\n builtin/receive-pack.c | 49 ++++++++++++++++++----\n send-pack.c | 3 ++\n t/t5537-push-shallow.sh (new +x) | 67 +++++++++++++++++++++++++++++++\n 4 files changed, 114 insertions(+), 9 deletions(-)\n create mode 100755 t/t5537-push-shallow.sh\n\ndiff --git a/Documentation/technical/pack-protocol.txt b/Documentation/technical/pack-protocol.txt\nindex eb8edd1..c73b62f 100644\n--- a/Documentation/technical/pack-protocol.txt\n+++ b/Documentation/technical/pack-protocol.txt\n@@ -464,7 +464,9 @@ contain all the objects that the server will need to complete the new\n references.\n \n ----\n- update-request = command-list [pack-file]\n+ update-request = *shallow command-list [pack-file]\n+\n+ shallow = PKT-LINE(\"shallow\" SP obj-id)\n \n command-list = PKT-LINE(command NUL capability-list LF)\n \t\t *PKT-LINE(command LF)\ndiff --git a/builtin/receive-pack.c b/builtin/receive-pack.c\nindex 2d8e19b..0537e26 100644\n--- a/builtin/receive-pack.c\n+++ b/builtin/receive-pack.c\n@@ -41,6 +41,10 @@ static int auto_gc = 1;\n static const char *head_name;\n static void *head_name_to_free;\n static int sent_capabilities;\n+static int shallow_changed;\n+static const char* alternate_shallow_file;\n+static struct lock_file shallow_lock;\n+static struct extra_have_objects shallow;\n \n static enum deny_action parse_deny_action(const char *var, const char *value)\n {\n@@ -751,6 +755,13 @@ static void execute_commands(struct command *commands, const char *unpacker_erro\n \t}\n }\n \n+static void add_extra_have(struct extra_have_objects *extra, unsigned char *sha1)\n+{\n+\tALLOC_GROW(extra->array, extra->nr + 1, extra->alloc);\n+\thashcpy(&(extra->array[extra->nr][0]), sha1);\n+\textra->nr++;\n+}\n+\n static struct command *read_head_info(void)\n {\n \tstruct command *commands = NULL;\n@@ -765,6 +776,17 @@ static struct command *read_head_info(void)\n \t\tline = packet_read_line(0, &len);\n \t\tif (!line)\n \t\t\tbreak;\n+\n+\t\tif (len == 48 && !prefixcmp(line, \"shallow \")) {\n+\t\t\tif (get_sha1_hex(line + 8, old_sha1))\n+\t\t\t\tdie(\"protocol error: expected shallow sha, got '%s'\", line + 8);\n+\t\t\tif (!has_sha1_file(old_sha1)) {\n+\t\t\t\tadd_extra_have(&shallow, old_sha1);\n+\t\t\t\tshallow_changed = 1;\n+\t\t\t}\n+\t\t\tcontinue;\n+\t\t}\n+\n \t\tif (len < 83 ||\n \t\t line[40] != ' ' ||\n \t\t line[81] != ' ' ||\n@@ -827,6 +849,12 @@ static const char *unpack(int err_fd)\n \t\t\t ? transfer_fsck_objects\n \t\t\t : 0);\n \n+\tif (shallow_changed)\n+\t\tsetup_alternate_shallow(&shallow_lock,\n+\t\t\t\t\t&alternate_shallow_file,\n+\t\t\t\t\t&shallow, 0);\n+\n+\n \thdr_err = parse_pack_header(&hdr);\n \tif (hdr_err) {\n \t\tif (err_fd > 0)\n@@ -854,9 +882,8 @@ static const char *unpack(int err_fd)\n \t\tchild.err = err_fd;\n \t\tchild.git_cmd = 1;\n \t\tcode = run_command(&child);\n-\t\tif (!code)\n-\t\t\treturn NULL;\n-\t\treturn \"unpack-objects abnormal exit\";\n+\t\tif (code)\n+\t\t\treturn \"unpack-objects abnormal exit\";\n \t} else {\n \t\tconst char *keeper[7];\n \t\tint s, status, i = 0;\n@@ -887,12 +914,18 @@ static const char *unpack(int err_fd)\n \t\tpack_lockfile = index_pack_lockfile(ip.out);\n \t\tclose(ip.out);\n \t\tstatus = finish_command(&ip);\n-\t\tif (!status) {\n-\t\t\treprepare_packed_git();\n-\t\t\treturn NULL;\n-\t\t}\n-\t\treturn \"index-pack abnormal exit\";\n+\t\tif (status)\n+\t\t\treturn \"index-pack abnormal exit\";\n+\t\treprepare_packed_git();\n+\t}\n+\n+\tif (shallow_changed) {\n+\t\tsetup_alternate_shallow(&shallow_lock,\n+\t\t\t\t\t&alternate_shallow_file,\n+\t\t\t\t\t&shallow, 1);\n+\t\tcommit_lock_file(&shallow_lock);\n \t}\n+\treturn NULL;\n }\n \n static const char *unpack_with_sideband(void)\ndiff --git a/send-pack.c b/send-pack.c\nindex 7d172ef..81d4b1c 100644\n--- a/send-pack.c\n+++ b/send-pack.c\n@@ -212,6 +212,9 @@ int send_pack(struct send_pack_args *args,\n \t\treturn 0;\n \t}\n \n+\tif (!args->dry_run && !args->stateless_rpc)\n+\t\tadvertise_shallow_grafts(out);\n+\n \t/*\n \t * Finally, tell the other end!\n \t */\ndiff --git a/t/t5537-push-shallow.sh b/t/t5537-push-shallow.sh\nnew file mode 100755\nindex 0000000..30fdbc8\n--- /dev/null\n+++ b/t/t5537-push-shallow.sh\n@@ -0,0 +1,67 @@\n+#!/bin/sh\n+\n+test_description='push from/to a shallow clone'\n+\n+. ./test-lib.sh\n+\n+test_expect_success 'setup' '\n+\ttest_commit 1 &&\n+\ttest_commit 2 &&\n+\ttest_commit 3 &&\n+\ttest_commit 4\n+'\n+\n+test_expect_success 'setup shallow clone' '\n+\tgit clone --no-local --depth=2 .git shallow &&\n+\tgit --git-dir=shallow/.git log --format=%s >actual &&\n+\tcat <expect &&\n+4\n+3\n+EOF\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'push from shallow clone' '\n+\t(\n+\tcd shallow &&\n+\ttest_commit 5 &&\n+\tgit push ../.git +master:refs/remotes/shallow/master\n+\t) &&\n+\tgit log --format=%s shallow/master >actual &&\n+\tgit fsck &&\n+\tcat <expect &&\n+5\n+4\n+3\n+2\n+1\n+EOF\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'push from shallow clone, with grafted roots' '\n+\tgit init shallow2 &&\n+\t(\n+\tcd shallow2 &&\n+\ttest_commit a &&\n+\ttest_commit b &&\n+\ttest_commit c &&\n+\tgit rev-parse b > .git/shallow &&\n+\tgit log --format=%s >actual &&\n+\tcat <expect &&\n+c\n+b\n+EOF\n+\ttest_cmp expect actual &&\n+\tgit push ../.git +master:refs/remotes/shallow2/master\n+\t) &&\n+\tgit log --format=%s shallow2/master >actual &&\n+\tgit fsck &&\n+\tcat <expect &&\n+c\n+b\n+EOF\n+\ttest_cmp expect actual\n+'\n+\n+test_done\n-- \n1.8.2.83.gc99314b\n"},{"id":"223579","messageId":"1374065234-870-8-git-send-email-pclouds@gmail.com","threadId":"34452","inReplyTo":"1374065234-870-1-git-send-email-pclouds@gmail.com","subject":"[PATCH 7/7] send-pack: support pushing to a shallow clone","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-07-17T12:47:14Z","receivedAt":"2013-07-17T12:47:14Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"When send-pack receives \"shallow\" lines from receive-pack, it knows\nthe other end does not have a complete commit chains. It restrict\nitself to the commits that are not cut out by either end to make sure\nthe result pack is usuable by receive-pack.\n\nThe same technique here, using setup_alternate_shallow() and\n--shallow-file, might simplify similar code in upload-pack.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/send-pack.c | 7 +++++--\n send-pack.c | 38 +++++++++++++++++++++++++++++++++-----\n send-pack.h | 4 +++-\n t/t5537-push-shallow.sh | 18 ++++++++++++++++++\n transport.c | 5 +++--\n 5 files changed, 62 insertions(+), 10 deletions(-)\n\ndiff --git a/builtin/send-pack.c b/builtin/send-pack.c\nindex f0ad0ce..b177120 100644\n--- a/builtin/send-pack.c\n+++ b/builtin/send-pack.c\n@@ -94,6 +94,7 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)\n \tint fd[2];\n \tstruct child_process *conn;\n \tstruct extra_have_objects extra_have;\n+\tstruct extra_have_objects shallow;\n \tstruct ref *remote_refs, *local_refs;\n \tint ret;\n \tint helper_status = 0;\n@@ -206,8 +207,10 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)\n \t}\n \n \tmemset(&extra_have, 0, sizeof(extra_have));\n+\tmemset(&shallow, 0, sizeof(shallow));\n \n-\tget_remote_heads(fd[0], NULL, 0, &remote_refs, REF_NORMAL, &extra_have, NULL);\n+\tget_remote_heads(fd[0], NULL, 0, &remote_refs, REF_NORMAL,\n+\t\t\t &extra_have, &shallow);\n \n \ttransport_verify_remote_names(nr_refspecs, refspecs);\n \n@@ -227,7 +230,7 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)\n \tset_ref_status_for_push(remote_refs, args.send_mirror,\n \t\targs.force_update);\n \n-\tret = send_pack(&args, fd, conn, remote_refs, &extra_have);\n+\tret = send_pack(&args, fd, conn, remote_refs, &extra_have, &shallow);\n \n \tif (helper_status)\n \t\tprint_helper_status(remote_refs);\ndiff --git a/send-pack.c b/send-pack.c\nindex 81d4b1c..08de681 100644\n--- a/send-pack.c\n+++ b/send-pack.c\n@@ -27,14 +27,19 @@ static int feed_object(const unsigned char *sha1, int fd, int negative)\n /*\n * Make a pack stream and spit it out into file descriptor fd\n */\n-static int pack_objects(int fd, struct ref *refs, struct extra_have_objects *extra, struct send_pack_args *args)\n+static int pack_objects(int fd, struct ref *refs,\n+\t\t\tstruct extra_have_objects *extra,\n+\t\t\tstruct extra_have_objects *extra_shallow,\n+\t\t\tstruct send_pack_args *args)\n {\n \t/*\n \t * The child becomes pack-objects --revs; we feed\n \t * the revision parameters to it via its stdin and\n \t * let its stdout go back to the other end.\n \t */\n-\tconst char *argv[] = {\n+\tconst char *av[] = {\n+\t\t\"--shallow-file\",\n+\t\tNULL,\n \t\t\"pack-objects\",\n \t\t\"--all-progress-implied\",\n \t\t\"--revs\",\n@@ -45,10 +50,27 @@ static int pack_objects(int fd, struct ref *refs, struct extra_have_objects *ext\n \t\tNULL,\n \t\tNULL,\n \t};\n+\tconst char **argv;\n \tstruct child_process po;\n+\tstatic struct lock_file shallow_lock;\n+\tconst char *alternate_shallow_file = NULL;\n \tint i;\n \n-\ti = 4;\n+\tif (extra_shallow->nr) {\n+\t\tmemset(&shallow_lock, 0, sizeof(shallow_lock));\n+\t\t/* just to load up .git/shallow if exists */\n+\t\tis_repository_shallow();\n+\t\tsetup_alternate_shallow(&shallow_lock,\n+\t\t\t\t\t&alternate_shallow_file,\n+\t\t\t\t\textra_shallow,\n+\t\t\t\t\t0);\n+\t\tav[1] = alternate_shallow_file;\n+\t\targv = av;\n+\t\ti = 6;\n+\t} else {\n+\t\targv = &av[2];\n+\t\ti = 4;\n+\t}\n \tif (args->use_thin_pack)\n \t\targv[i++] = \"--thin\";\n \tif (args->use_ofs_delta)\n@@ -100,6 +122,10 @@ static int pack_objects(int fd, struct ref *refs, struct extra_have_objects *ext\n \n \tif (finish_command(&po))\n \t\treturn -1;\n+\n+\tif (extra_shallow->nr)\n+\t\trollback_lock_file(&shallow_lock);\n+\n \treturn 0;\n }\n \n@@ -176,7 +202,8 @@ static int sideband_demux(int in, int out, void *data)\n int send_pack(struct send_pack_args *args,\n \t int fd[], struct child_process *conn,\n \t struct ref *remote_refs,\n-\t struct extra_have_objects *extra_have)\n+\t struct extra_have_objects *extra_have,\n+\t struct extra_have_objects *extra_shallow)\n {\n \tint in = fd[0];\n \tint out = fd[1];\n@@ -294,7 +321,8 @@ int send_pack(struct send_pack_args *args,\n \t}\n \n \tif (new_refs && cmds_sent) {\n-\t\tif (pack_objects(out, remote_refs, extra_have, args) < 0) {\n+\t\tif (pack_objects(out, remote_refs, extra_have, extra_shallow,\n+\t\t\t\t args) < 0) {\n \t\t\tfor (ref = remote_refs; ref; ref = ref->next)\n \t\t\t\tref->status = REF_STATUS_NONE;\n \t\t\tif (args->stateless_rpc)\ndiff --git a/send-pack.h b/send-pack.h\nindex 05d7ab1..1c8a343 100644\n--- a/send-pack.h\n+++ b/send-pack.h\n@@ -16,6 +16,8 @@ struct send_pack_args {\n \n int send_pack(struct send_pack_args *args,\n \t int fd[], struct child_process *conn,\n-\t struct ref *remote_refs, struct extra_have_objects *extra_have);\n+\t struct ref *remote_refs,\n+\t struct extra_have_objects *extra_have,\n+\t struct extra_have_objects *extra_shallow);\n \n #endif\ndiff --git a/t/t5537-push-shallow.sh b/t/t5537-push-shallow.sh\nindex 30fdbc8..76d3d51 100755\n--- a/t/t5537-push-shallow.sh\n+++ b/t/t5537-push-shallow.sh\n@@ -64,4 +64,22 @@ EOF\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'push from shallow to shallow' '\n+\t(\n+\tcd shallow &&\n+\tgit push ../shallow2/.git +master:refs/remotes/shallow/master\n+\t) &&\n+\t(\n+\tcd shallow2 &&\n+\tgit log --format=%s shallow/master >actual &&\n+\tgit fsck &&\n+\tcat <expect &&\n+5\n+4\n+3\n+EOF\n+\ttest_cmp expect actual\n+\t)\n+'\n+\n test_done\ndiff --git a/transport.c b/transport.c\nindex 17fef16..d592e8d 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -802,7 +802,8 @@ static int git_transport_push(struct transport *transport, struct ref *remote_re\n \t\tstruct ref *tmp_refs;\n \t\tconnect_setup(transport, 1, 0);\n \n-\t\tget_remote_heads(data->fd[0], NULL, 0, &tmp_refs, REF_NORMAL, NULL, NULL);\n+\t\tget_remote_heads(data->fd[0], NULL, 0, &tmp_refs, REF_NORMAL,\n+\t\t\t\t NULL, &data->shallow);\n \t\tdata->got_remote_heads = 1;\n \t}\n \n@@ -817,7 +818,7 @@ static int git_transport_push(struct transport *transport, struct ref *remote_re\n \targs.porcelain = !!(flags & TRANSPORT_PUSH_PORCELAIN);\n \n \tret = send_pack(&args, data->fd, data->conn, remote_refs,\n-\t\t\t&data->extra_have);\n+\t\t\t&data->extra_have, &data->shallow);\n \n \tclose(data->fd[1]);\n \tclose(data->fd[0]);\n-- \n1.8.2.83.gc99314b\n"},{"id":"223823","messageId":"7v61w5udhs.fsf@alter.siamese.dyndns.org","threadId":"34452","inReplyTo":"1374065234-870-6-git-send-email-pclouds@gmail.com","subject":"Re: [PATCH 5/7] fetch-pack: support fetching from a shallow repository","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-07-20T03:17:03Z","receivedAt":"2013-07-20T03:17:03Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Nguyễn Thái Ngọc Duy writes:\n\n> Only server graft points that do not have corresponding SHA-1s in\n> local repo are added to the temp shallow file because we don't want to\n> accidentally cut the client history because the server's is\n> shorter. The client cutting can only happen when --depth is requested.\n\nThis \"have corresponding SHA-1\" is iffy from connectivity point of view,\nisn't it? That is,...\n\n> +\tfor (i = 0; i < extra->nr; i++) {\n> +\t\tif (!remove_unused_grafts && has_sha1_file(extra->array[i]))\n\n...shouldn't this require much more than \"has_sha1_file()\", like\n\"has it, and we have everything behind it with respect to the\nshallow graft points we started with\" or something?\n\nOtherwise an isolated island that was left behind by an earlier\naborted commit walker fetch may mislead us to believe that we have\nthe complete history behind this commit object.\n"},{"id":"223824","messageId":"7v1u6tud12.fsf@alter.siamese.dyndns.org","threadId":"34452","inReplyTo":"1374065234-870-4-git-send-email-pclouds@gmail.com","subject":"Re: [PATCH 3/7] connect.c: teach get_remote_heads to parse \"shallow\" lines","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-07-20T03:27:05Z","receivedAt":"2013-07-20T03:27:05Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Nguyễn Thái Ngọc Duy writes:\n\n> diff --git a/cache.h b/cache.h\n> index dd0fb33..7665e03 100644\n> --- a/cache.h\n> +++ b/cache.h\n> @@ -1091,6 +1091,7 @@ struct extra_have_objects {\n> };\n> extern struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,\n> \t\t\t\t struct ref **list, unsigned int flags,\n> +\t\t\t\t struct extra_have_objects *,\n> \t\t\t\t struct extra_have_objects *);\n\nWhen it is the sole parameter of a specific type (i.e. not \"char *\",\nint, size_t, etc. but an application specific structure like \"struct\nextra_have_objects *\"), it is perfectly fine (and even preferrable)\nto omit the parameter name from the declaration, as it is clear what\nthe parameter is and means.\n\nBut when you add another of the same type, you need to give both of\nthem a descriptive name (e.g. the ones you use in the definition).\n\nOtherwise, somebody who wants to write a caller cannot tell which\n\"struct extra_have_objects *\" parameter is the extra and which one\nis the shallow graft points list.\n"},{"id":"223829","messageId":"1374314290-5976-1-git-send-email-pclouds@gmail.com","threadId":"34452","inReplyTo":"1374065234-870-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v2 00/16] First class shallow clone","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-07-20T09:57:54Z","receivedAt":"2013-07-20T09:57:54Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"v2 includes:\n\n - fix Junio comments, especially the one that may lead to incomplete\n commit islands.\n - fix send-pack setting up temporary shallow file, but never passes\n it to index-pack/unpack-objects (also fix the tests to catch this)\n - support smart http\n - add core.noshallow for repos that wish to be always complete\n - fix locally cloning a shallow repository\n - make upload-pack pass --shallow-file to pack-objects in order to\n remove duplicate object counting code just for shallow case.\n\nNguyễn Thái Ngọc Duy (16):\n send-pack: forbid pushing from a shallow repository\n {receive,upload}-pack: advertise shallow graft information\n connect.c: teach get_remote_heads to parse \"shallow\" lines\n Move setup_alternate_shallow and write_shallow_commits to shallow.c\n fetch-pack: support fetching from a shallow repository\n {send,receive}-pack: support pushing from a shallow clone\n send-pack: support pushing to a shallow clone\n upload-pack: let pack-objects do the object counting in shallow case\n pack-protocol.txt: a bit about smart http\n Add document for command arguments for supporting smart http\n {fetch,upload}-pack: support fetching from a shallow clone via smart http\n receive-pack: support pushing to a shallow clone via http\n send-pack: support pushing from a shallow clone via http\n git-clone.txt: remove shallow clone limitations\n config: add core.noshallow to prevent turning a repo into a shallow one\n clone: use git protocol for cloning shallow repo locally\n\n Documentation/config.txt | 5 +\n Documentation/git-clone.txt | 7 +-\n Documentation/git-fetch-pack.txt | 11 +-\n Documentation/git-receive-pack.txt | 16 ++-\n Documentation/git-send-pack.txt | 9 +-\n Documentation/git-upload-pack.txt | 13 ++-\n Documentation/technical/pack-protocol.txt | 76 ++++++++++++-\n builtin/clone.c | 14 ++-\n builtin/fetch-pack.c | 6 +-\n builtin/receive-pack.c | 76 +++++++++++--\n builtin/send-pack.c | 7 +-\n cache.h | 4 +-\n commit.h | 27 +++++\n config.c | 5 +\n connect.c | 12 +-\n environment.c | 1 +\n fetch-pack.c | 90 ++++++---------\n fetch-pack.h | 1 +\n remote-curl.c | 4 +-\n send-pack.c | 57 +++++++++-\n send-pack.h | 4 +-\n shallow.c | 147 +++++++++++++++++++++++++\n t/t5530-upload-pack-error.sh | 3 -\n t/t5536-fetch-shallow.sh (new +x) | 141 ++++++++++++++++++++++++\n t/t5537-push-shallow.sh (new +x) | 176 ++++++++++++++++++++++++++++++\n t/t5601-clone.sh | 7 ++\n transport.c | 14 ++-\n upload-pack.c | 132 ++++++----------------\n 28 files changed, 858 insertions(+), 207 deletions(-)\n create mode 100755 t/t5536-fetch-shallow.sh\n create mode 100755 t/t5537-push-shallow.sh\n\n-- \n1.8.2.83.gc99314b\n"},{"id":"223833","messageId":"1374314290-5976-2-git-send-email-pclouds@gmail.com","threadId":"34452","inReplyTo":"1374314290-5976-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v2 01/16] send-pack: forbid pushing from a shallow repository","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-07-20T09:57:55Z","receivedAt":"2013-07-20T09:57:55Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"send-pack can send a pack with loose ends to the server. receive-pack\nbefore 6d4bb38 (fetch: verify we have everything we need before\nupdating our ref - 2011-09-01) does not detect this and keeps the pack\nanyway, which corrupts the repository, at least from fsck point of\nview.\n\nsend-pack will learn to safely push from a shallow repository later.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/send-pack.c | 3 +++\n 1 file changed, 3 insertions(+)\n\ndiff --git a/builtin/send-pack.c b/builtin/send-pack.c\nindex 152c4ea..d4cbf64 100644\n--- a/builtin/send-pack.c\n+++ b/builtin/send-pack.c\n@@ -184,6 +184,9 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)\n \t (send_all && args.send_mirror))\n \t\tusage(send_pack_usage);\n \n+\tif (is_repository_shallow())\n+\t\tdie(\"attempt to push from a shallow repository\");\n+\n \tif (remote_name) {\n \t\tremote = remote_get(remote_name);\n \t\tif (!remote_has_url(remote, dest)) {\n-- \n1.8.2.83.gc99314b\n"},{"id":"223832","messageId":"1374314290-5976-3-git-send-email-pclouds@gmail.com","threadId":"34452","inReplyTo":"1374314290-5976-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v2 02/16] {receive,upload}-pack: advertise shallow graft information","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-07-20T09:57:56Z","receivedAt":"2013-07-20T09:57:56Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"If either receive-pack or upload-pack is called on a shallow\nrepository, shallow graft points will be sent after the ref\nadvertisement (but before the packet flush), so that the client has\nthe full \"shape\" of the server's commit graph.\n\nThis breaks the protocol for all clients trying to push to a shallow\nrepo, or fetch from one. Which is basically the same end result as\ntoday's \"is_repository_shallow() && die()\" in receive-pack and\nupload-pack. New clients will be made aware of shallow upstream and\ncan make use of this information.\n\nSmart HTTP is not affected by this patch. Shallow support on\nsmart-http comes later separately.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n Documentation/technical/pack-protocol.txt | 3 +++\n builtin/receive-pack.c | 4 +++-\n commit.h | 1 +\n shallow.c | 16 ++++++++++++++++\n upload-pack.c | 6 ++++--\n 5 files changed, 27 insertions(+), 3 deletions(-)\n\ndiff --git a/Documentation/technical/pack-protocol.txt b/Documentation/technical/pack-protocol.txt\nindex b898e97..eb8edd1 100644\n--- a/Documentation/technical/pack-protocol.txt\n+++ b/Documentation/technical/pack-protocol.txt\n@@ -161,6 +161,7 @@ MUST peel the ref if it's an annotated tag.\n \n ----\n advertised-refs = (no-refs / list-of-refs)\n+\t\t *shallow\n \t\t flush-pkt\n \n no-refs = PKT-LINE(zero-id SP \"capabilities^{}\"\n@@ -174,6 +175,8 @@ MUST peel the ref if it's an annotated tag.\n other-tip = obj-id SP refname LF\n other-peeled = obj-id SP refname \"^{}\" LF\n \n+ shallow = PKT-LINE(\"shallow\" SP obj-id)\n+\n capability-list = capability *(SP capability)\n capability = 1*(LC_ALPHA / DIGIT / \"-\" / \"_\")\n LC_ALPHA = %x61-7A\ndiff --git a/builtin/receive-pack.c b/builtin/receive-pack.c\nindex e3eb5fc..6ffe526 100644\n--- a/builtin/receive-pack.c\n+++ b/builtin/receive-pack.c\n@@ -176,6 +176,8 @@ static void write_head_info(void)\n \tif (!sent_capabilities)\n \t\tshow_ref(\"capabilities^{}\", null_sha1);\n \n+\tadvertise_shallow_grafts(1);\n+\n \t/* EOF */\n \tpacket_flush(1);\n }\n@@ -990,7 +992,7 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix)\n \tif (!enter_repo(dir, 0))\n \t\tdie(\"'%s' does not appear to be a git repository\", dir);\n \n-\tif (is_repository_shallow())\n+\tif (is_repository_shallow() && stateless_rpc)\n \t\tdie(\"attempt to push into a shallow repository\");\n \n \tgit_config(receive_pack_config, NULL);\ndiff --git a/commit.h b/commit.h\nindex 4d452dc..e0688c3 100644\n--- a/commit.h\n+++ b/commit.h\n@@ -187,6 +187,7 @@ extern struct commit_list *get_shallow_commits(struct object_array *heads,\n \t\tint depth, int shallow_flag, int not_shallow_flag);\n extern void check_shallow_file_for_update(void);\n extern void set_alternate_shallow_file(const char *path);\n+extern void advertise_shallow_grafts(int);\n \n int is_descendant_of(struct commit *, struct commit_list *);\n int in_merge_bases(struct commit *, struct commit *);\ndiff --git a/shallow.c b/shallow.c\nindex cbe2526..ccdfefc 100644\n--- a/shallow.c\n+++ b/shallow.c\n@@ -1,6 +1,7 @@\n #include \"cache.h\"\n #include \"commit.h\"\n #include \"tag.h\"\n+#include \"pkt-line.h\"\n \n static int is_shallow = -1;\n static struct stat shallow_stat;\n@@ -146,3 +147,18 @@ void check_shallow_file_for_update(void)\n \t\t )\n \t\tdie(\"shallow file was changed during fetch\");\n }\n+\n+static int advertise_shallow_grafts_cb(const struct commit_graft *graft, void *cb)\n+{\n+\tint fd = *(int*)cb;\n+\tif (graft->nr_parent == -1)\n+\t\tpacket_write(fd, \"shallow %s\\n\", sha1_to_hex(graft->sha1));\n+\treturn 0;\n+}\n+\n+void advertise_shallow_grafts(int fd)\n+{\n+\tif (!is_repository_shallow())\n+\t\treturn;\n+\tfor_each_commit_graft(advertise_shallow_grafts_cb, &fd);\n+}\ndiff --git a/upload-pack.c b/upload-pack.c\nindex 127e59a..c377a3e 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -766,6 +766,7 @@ static void upload_pack(void)\n \t\treset_timeout();\n \t\thead_ref_namespaced(send_ref, NULL);\n \t\tfor_each_namespaced_ref(send_ref, NULL);\n+\t\tadvertise_shallow_grafts(1);\n \t\tpacket_flush(1);\n \t} else {\n \t\thead_ref_namespaced(mark_our_ref, NULL);\n@@ -837,8 +838,9 @@ int main(int argc, char **argv)\n \n \tif (!enter_repo(dir, strict))\n \t\tdie(\"'%s' does not appear to be a git repository\", dir);\n-\tif (is_repository_shallow())\n-\t\tdie(\"attempt to fetch/clone from a shallow repository\");\n+\tif (is_repository_shallow() && stateless_rpc)\n+\t\tdie(\"attempt to push into a shallow repository\");\n+\n \tgit_config(upload_pack_config, NULL);\n \tupload_pack();\n \treturn 0;\n-- \n1.8.2.83.gc99314b\n"},{"id":"223830","messageId":"1374314290-5976-4-git-send-email-pclouds@gmail.com","threadId":"34452","inReplyTo":"1374314290-5976-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v2 03/16] connect.c: teach get_remote_heads to parse \"shallow\" lines","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-07-20T09:57:57Z","receivedAt":"2013-07-20T09:57:57Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"No callers pass a non-empty pointer as shallow_points at this\nstage. As a result, all clients still refuse to talk to shallow\nrepository on the other end.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/fetch-pack.c | 2 +-\n builtin/send-pack.c | 2 +-\n cache.h | 3 ++-\n connect.c | 12 +++++++++++-\n remote-curl.c | 2 +-\n transport.c | 7 ++++---\n 6 files changed, 20 insertions(+), 8 deletions(-)\n\ndiff --git a/builtin/fetch-pack.c b/builtin/fetch-pack.c\nindex aba4465..080e599 100644\n--- a/builtin/fetch-pack.c\n+++ b/builtin/fetch-pack.c\n@@ -144,7 +144,7 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)\n \t\t\t\t args.verbose ? CONNECT_VERBOSE : 0);\n \t}\n \n-\tget_remote_heads(fd[0], NULL, 0, &ref, 0, NULL);\n+\tget_remote_heads(fd[0], NULL, 0, &ref, 0, NULL, NULL);\n \n \tref = fetch_pack(&args, fd, conn, ref, dest,\n \t\t\t sought, nr_sought, pack_lockfile_ptr);\ndiff --git a/builtin/send-pack.c b/builtin/send-pack.c\nindex d4cbf64..4be5931 100644\n--- a/builtin/send-pack.c\n+++ b/builtin/send-pack.c\n@@ -210,7 +210,7 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)\n \n \tmemset(&extra_have, 0, sizeof(extra_have));\n \n-\tget_remote_heads(fd[0], NULL, 0, &remote_refs, REF_NORMAL, &extra_have);\n+\tget_remote_heads(fd[0], NULL, 0, &remote_refs, REF_NORMAL, &extra_have, NULL);\n \n \ttransport_verify_remote_names(nr_refspecs, refspecs);\n \ndiff --git a/cache.h b/cache.h\nindex dd0fb33..7f17228 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1091,7 +1091,8 @@ struct extra_have_objects {\n };\n extern struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,\n \t\t\t\t struct ref **list, unsigned int flags,\n-\t\t\t\t struct extra_have_objects *);\n+\t\t\t\t struct extra_have_objects *have,\n+\t\t\t\t struct extra_have_objects *shallow);\n extern int server_supports(const char *feature);\n extern int parse_feature_request(const char *features, const char *feature);\n extern const char *server_feature_value(const char *feature, int *len_ret);\ndiff --git a/connect.c b/connect.c\nindex a0783d4..1cba424 100644\n--- a/connect.c\n+++ b/connect.c\n@@ -64,7 +64,8 @@ static void die_initial_contact(int got_at_least_one_head)\n */\n struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,\n \t\t\t struct ref **list, unsigned int flags,\n-\t\t\t struct extra_have_objects *extra_have)\n+\t\t\t struct extra_have_objects *extra_have,\n+\t\t\t struct extra_have_objects *shallow_points)\n {\n \tint got_at_least_one_head = 0;\n \n@@ -89,6 +90,15 @@ struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,\n \t\tif (len > 4 && !prefixcmp(buffer, \"ERR \"))\n \t\t\tdie(\"remote error: %s\", buffer + 4);\n \n+\t\tif (len == 48 && !prefixcmp(buffer, \"shallow \")) {\n+\t\t\tif (get_sha1_hex(buffer + 8, old_sha1))\n+\t\t\t\tdie(\"protocol error: expected shallow sha, got '%s'\", buffer + 8);\n+\t\t\tif (!shallow_points)\n+\t\t\t\tdie(\"repository on the other end cannot be shallow\");\n+\t\t\tadd_extra_have(shallow_points, old_sha1);\n+\t\t\tcontinue;\n+\t\t}\n+\n \t\tif (len < 42 || get_sha1_hex(buffer, old_sha1) || buffer[40] != ' ')\n \t\t\tdie(\"protocol error: expected sha/ref, got '%s'\", buffer);\n \t\tname = buffer + 41;\ndiff --git a/remote-curl.c b/remote-curl.c\nindex 5b3ce9e..c329bd3 100644\n--- a/remote-curl.c\n+++ b/remote-curl.c\n@@ -86,7 +86,7 @@ static struct ref *parse_git_refs(struct discovery *heads, int for_push)\n {\n \tstruct ref *list = NULL;\n \tget_remote_heads(-1, heads->buf, heads->len, &list,\n-\t\t\t for_push ? REF_NORMAL : 0, NULL);\n+\t\t\t for_push ? REF_NORMAL : 0, NULL, NULL);\n \treturn list;\n }\n \ndiff --git a/transport.c b/transport.c\nindex e15db98..10a8cb8 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -509,7 +509,7 @@ static struct ref *get_refs_via_connect(struct transport *transport, int for_pus\n \n \tconnect_setup(transport, for_push, 0);\n \tget_remote_heads(data->fd[0], NULL, 0, &refs,\n-\t\t\t for_push ? REF_NORMAL : 0, &data->extra_have);\n+\t\t\t for_push ? REF_NORMAL : 0, &data->extra_have, NULL);\n \tdata->got_remote_heads = 1;\n \n \treturn refs;\n@@ -539,7 +539,8 @@ static int fetch_refs_via_pack(struct transport *transport,\n \n \tif (!data->got_remote_heads) {\n \t\tconnect_setup(transport, 0, 0);\n-\t\tget_remote_heads(data->fd[0], NULL, 0, &refs_tmp, 0, NULL);\n+\t\tget_remote_heads(data->fd[0], NULL, 0, &refs_tmp, 0,\n+\t\t\t\t NULL, NULL);\n \t\tdata->got_remote_heads = 1;\n \t}\n \n@@ -799,7 +800,7 @@ static int git_transport_push(struct transport *transport, struct ref *remote_re\n \t\tstruct ref *tmp_refs;\n \t\tconnect_setup(transport, 1, 0);\n \n-\t\tget_remote_heads(data->fd[0], NULL, 0, &tmp_refs, REF_NORMAL, NULL);\n+\t\tget_remote_heads(data->fd[0], NULL, 0, &tmp_refs, REF_NORMAL, NULL, NULL);\n \t\tdata->got_remote_heads = 1;\n \t}\n \n-- \n1.8.2.83.gc99314b\n"},{"id":"223831","messageId":"1374314290-5976-5-git-send-email-pclouds@gmail.com","threadId":"34452","inReplyTo":"1374314290-5976-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v2 04/16] Move setup_alternate_shallow and write_shallow_commits to shallow.c","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-07-20T09:57:58Z","receivedAt":"2013-07-20T09:57:58Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n commit.h | 3 +++\n fetch-pack.c | 53 +----------------------------------------------------\n shallow.c | 53 +++++++++++++++++++++++++++++++++++++++++++++++++++++\n 3 files changed, 57 insertions(+), 52 deletions(-)\n\ndiff --git a/commit.h b/commit.h\nindex e0688c3..678fa20 100644\n--- a/commit.h\n+++ b/commit.h\n@@ -188,6 +188,9 @@ extern struct commit_list *get_shallow_commits(struct object_array *heads,\n extern void check_shallow_file_for_update(void);\n extern void set_alternate_shallow_file(const char *path);\n extern void advertise_shallow_grafts(int);\n+extern int write_shallow_commits(struct strbuf *out, int use_pack_protocol);\n+extern void setup_alternate_shallow(struct lock_file *shallow_lock,\n+\t\t\t\t const char **alternate_shallow_file);\n \n int is_descendant_of(struct commit *, struct commit_list *);\n int in_merge_bases(struct commit *, struct commit *);\ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex abe5ffb..dc71a2b 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -185,36 +185,6 @@ static void consume_shallow_list(struct fetch_pack_args *args, int fd)\n \t}\n }\n \n-struct write_shallow_data {\n-\tstruct strbuf *out;\n-\tint use_pack_protocol;\n-\tint count;\n-};\n-\n-static int write_one_shallow(const struct commit_graft *graft, void *cb_data)\n-{\n-\tstruct write_shallow_data *data = cb_data;\n-\tconst char *hex = sha1_to_hex(graft->sha1);\n-\tdata->count++;\n-\tif (data->use_pack_protocol)\n-\t\tpacket_buf_write(data->out, \"shallow %s\", hex);\n-\telse {\n-\t\tstrbuf_addstr(data->out, hex);\n-\t\tstrbuf_addch(data->out, '\\n');\n-\t}\n-\treturn 0;\n-}\n-\n-static int write_shallow_commits(struct strbuf *out, int use_pack_protocol)\n-{\n-\tstruct write_shallow_data data;\n-\tdata.out = out;\n-\tdata.use_pack_protocol = use_pack_protocol;\n-\tdata.count = 0;\n-\tfor_each_commit_graft(write_one_shallow, &data);\n-\treturn data.count;\n-}\n-\n static enum ack_type get_ack(int fd, unsigned char *result_sha1)\n {\n \tint len;\n@@ -795,27 +765,6 @@ static int cmp_ref_by_name(const void *a_, const void *b_)\n \treturn strcmp(a->name, b->name);\n }\n \n-static void setup_alternate_shallow(void)\n-{\n-\tstruct strbuf sb = STRBUF_INIT;\n-\tint fd;\n-\n-\tcheck_shallow_file_for_update();\n-\tfd = hold_lock_file_for_update(&shallow_lock, git_path(\"shallow\"),\n-\t\t\t\t LOCK_DIE_ON_ERROR);\n-\tif (write_shallow_commits(&sb, 0)) {\n-\t\tif (write_in_full(fd, sb.buf, sb.len) != sb.len)\n-\t\t\tdie_errno(\"failed to write to %s\", shallow_lock.filename);\n-\t\talternate_shallow_file = shallow_lock.filename;\n-\t} else\n-\t\t/*\n-\t\t * is_repository_shallow() sees empty string as \"no\n-\t\t * shallow file\".\n-\t\t */\n-\t\talternate_shallow_file = \"\";\n-\tstrbuf_release(&sb);\n-}\n-\n static struct ref *do_fetch_pack(struct fetch_pack_args *args,\n \t\t\t\t int fd[2],\n \t\t\t\t const struct ref *orig_ref,\n@@ -896,7 +845,7 @@ static struct ref *do_fetch_pack(struct fetch_pack_args *args,\n \tif (args->stateless_rpc)\n \t\tpacket_flush(fd[1]);\n \tif (args->depth > 0)\n-\t\tsetup_alternate_shallow();\n+\t\tsetup_alternate_shallow(&shallow_lock, &alternate_shallow_file);\n \tif (get_pack(args, fd, pack_lockfile))\n \t\tdie(\"git fetch-pack: fetch failed.\");\n \ndiff --git a/shallow.c b/shallow.c\nindex ccdfefc..ee9edd4 100644\n--- a/shallow.c\n+++ b/shallow.c\n@@ -162,3 +162,56 @@ void advertise_shallow_grafts(int fd)\n \t\treturn;\n \tfor_each_commit_graft(advertise_shallow_grafts_cb, &fd);\n }\n+\n+struct write_shallow_data {\n+\tstruct strbuf *out;\n+\tint use_pack_protocol;\n+\tint count;\n+};\n+\n+static int write_one_shallow(const struct commit_graft *graft, void *cb_data)\n+{\n+\tstruct write_shallow_data *data = cb_data;\n+\tconst char *hex = sha1_to_hex(graft->sha1);\n+\tdata->count++;\n+\tif (data->use_pack_protocol)\n+\t\tpacket_buf_write(data->out, \"shallow %s\", hex);\n+\telse {\n+\t\tstrbuf_addstr(data->out, hex);\n+\t\tstrbuf_addch(data->out, '\\n');\n+\t}\n+\treturn 0;\n+}\n+\n+int write_shallow_commits(struct strbuf *out, int use_pack_protocol)\n+{\n+\tstruct write_shallow_data data;\n+\tdata.out = out;\n+\tdata.use_pack_protocol = use_pack_protocol;\n+\tdata.count = 0;\n+\tfor_each_commit_graft(write_one_shallow, &data);\n+\treturn data.count;\n+}\n+\n+void setup_alternate_shallow(struct lock_file *shallow_lock,\n+\t\t\t const char **alternate_shallow_file)\n+{\n+\tstruct strbuf sb = STRBUF_INIT;\n+\tint fd;\n+\n+\tcheck_shallow_file_for_update();\n+\tfd = hold_lock_file_for_update(shallow_lock, git_path(\"shallow\"),\n+\t\t\t\t LOCK_DIE_ON_ERROR);\n+\tif (write_shallow_commits(&sb, 0)) {\n+\t\tif (write_in_full(fd, sb.buf, sb.len) != sb.len)\n+\t\t\tdie_errno(\"failed to write to %s\",\n+\t\t\t\t shallow_lock->filename);\n+\t\t*alternate_shallow_file = shallow_lock->filename;\n+\t} else\n+\t\t/*\n+\t\t * is_repository_shallow() sees empty string as \"no\n+\t\t * shallow file\".\n+\t\t */\n+\t\t*alternate_shallow_file = \"\";\n+\tstrbuf_release(&sb);\n+}\n-- \n1.8.2.83.gc99314b\n"},{"id":"223834","messageId":"1374314290-5976-6-git-send-email-pclouds@gmail.com","threadId":"34452","inReplyTo":"1374314290-5976-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v2 05/16] fetch-pack: support fetching from a shallow repository","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-07-20T09:57:59Z","receivedAt":"2013-07-20T09:57:59Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"upload-pack already advertises all shallow grafts if server repository\nis shallow. This information can be used to add more grafts to the\nclient if the server sends commit chains down to its graft points.\n\nIf the server is shallow, before we receive the pack, we setup a\ntemporary shallow file that contains both local graft points and the\nserver's. This stops index-pack from going beyond server's graft\npoints.\n\nOnly server graft points that do not have corresponding SHA-1s in\nlocal repo are added to the temp shallow file because we don't want to\naccidentally cut the client history because the server's is\nshorter. The client cutting can only happen when --depth is requested.\n\nAfter index-pack finishes successfully, we write the temporary shallow\ndown with one exception: unused graft points provided by the server\nare removed. We don't want those lying around and suddenly become\nactive.\n\nNote that in the \"shallow -> shallow\" case, the server might not have\nenough information to find common roots to create an optimum pack. It\nmight send complete commit chains down to the graft points as a\nresult. I don't think we can improve this, unless upload-pack somehow\nhas access to a full repository.\n\n\"shallow -> shallow\" case only makes sense when the upstream provides\na stable shallow repo (e.g. make a cut every year or so and ask devs\nto all move to the new base). If the cloned repos are all based on a\nstable (shallow) upstream, the above problem is unlikely to happen.\n\nA side effect of this change is we can now clone from a shallow\nrepository. And a full repository may automatically become shallow if\nyou fetch from a shallow repository.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/fetch-pack.c | 7 ++-\n commit.h | 27 +++++++++-\n fetch-pack.c | 30 +++++++++--\n fetch-pack.h | 1 +\n shallow.c | 90 +++++++++++++++++++++++++++++---\n t/t5536-fetch-shallow.sh (new +x) | 105 ++++++++++++++++++++++++++++++++++++++\n transport.c | 10 ++--\n 7 files changed, 252 insertions(+), 18 deletions(-)\n create mode 100755 t/t5536-fetch-shallow.sh\n\ndiff --git a/builtin/fetch-pack.c b/builtin/fetch-pack.c\nindex 080e599..f6a6d76 100644\n--- a/builtin/fetch-pack.c\n+++ b/builtin/fetch-pack.c\n@@ -37,6 +37,7 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)\n \tchar **pack_lockfile_ptr = NULL;\n \tstruct child_process *conn;\n \tstruct fetch_pack_args args;\n+\tstruct extra_have_objects shallow;\n \n \tpacket_trace_identity(\"fetch-pack\");\n \n@@ -144,10 +145,12 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)\n \t\t\t\t args.verbose ? CONNECT_VERBOSE : 0);\n \t}\n \n-\tget_remote_heads(fd[0], NULL, 0, &ref, 0, NULL, NULL);\n+\tmemset(&shallow, 0, sizeof(shallow));\n+\tget_remote_heads(fd[0], NULL, 0, &ref, 0, NULL,\n+\t\t\t args.stateless_rpc ? NULL : &shallow);\n \n \tref = fetch_pack(&args, fd, conn, ref, dest,\n-\t\t\t sought, nr_sought, pack_lockfile_ptr);\n+\t\t\t sought, nr_sought, &shallow, pack_lockfile_ptr);\n \tif (pack_lockfile) {\n \t\tprintf(\"lock %s\\n\", pack_lockfile);\n \t\tfflush(stdout);\ndiff --git a/commit.h b/commit.h\nindex 678fa20..97fd54e 100644\n--- a/commit.h\n+++ b/commit.h\n@@ -188,9 +188,32 @@ extern struct commit_list *get_shallow_commits(struct object_array *heads,\n extern void check_shallow_file_for_update(void);\n extern void set_alternate_shallow_file(const char *path);\n extern void advertise_shallow_grafts(int);\n-extern int write_shallow_commits(struct strbuf *out, int use_pack_protocol);\n+\n+/*\n+ * Ignore graft points that cut out existing \"good\" parts of our\n+ * history. Graft points pointing to incomplete commit islands are\n+ * _not_ ignore.\n+ */\n+#define WRITE_SHALLOW_NO_CUT 1\n+/*\n+ * Remove graft points that are not attached to any SHA-1 in current\n+ * history.\n+ */\n+#define WRITE_SHALLOW_NO_UNUSED 2\n+/*\n+ * Not really write_shallow_commits() flag, this lets\n+ * setup_alternate_shallow know this is the \"final\" setup before\n+ * committing to .git/shallow\n+ */\n+#define WRITE_SHALLOW_REWRITE 4\n+\n+extern int write_shallow_commits(struct strbuf *out, int use_pack_protocol,\n+\t\t\t\t struct extra_have_objects *extra,\n+\t\t\t\t unsigned int flags);\n extern void setup_alternate_shallow(struct lock_file *shallow_lock,\n-\t\t\t\t const char **alternate_shallow_file);\n+\t\t\t\t const char **alternate_shallow_file,\n+\t\t\t\t struct extra_have_objects *extra,\n+\t\t\t\t unsigned int flags);\n \n int is_descendant_of(struct commit *, struct commit_list *);\n int in_merge_bases(struct commit *, struct commit *);\ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex dc71a2b..f337526 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -311,7 +311,7 @@ static int find_common(struct fetch_pack_args *args,\n \t}\n \n \tif (is_repository_shallow())\n-\t\twrite_shallow_commits(&req_buf, 1);\n+\t\twrite_shallow_commits(&req_buf, 1, NULL, WRITE_SHALLOW_NO_CUT);\n \tif (args->depth > 0)\n \t\tpacket_buf_write(&req_buf, \"deepen %d\", args->depth);\n \tpacket_buf_flush(&req_buf);\n@@ -769,6 +769,7 @@ static struct ref *do_fetch_pack(struct fetch_pack_args *args,\n \t\t\t\t int fd[2],\n \t\t\t\t const struct ref *orig_ref,\n \t\t\t\t struct ref **sought, int nr_sought,\n+\t\t\t\t struct extra_have_objects *shallow,\n \t\t\t\t char **pack_lockfile)\n {\n \tstruct ref *ref = copy_ref_list(orig_ref);\n@@ -844,8 +845,10 @@ static struct ref *do_fetch_pack(struct fetch_pack_args *args,\n \n \tif (args->stateless_rpc)\n \t\tpacket_flush(fd[1]);\n-\tif (args->depth > 0)\n-\t\tsetup_alternate_shallow(&shallow_lock, &alternate_shallow_file);\n+\tif (args->depth > 0 || shallow->nr)\n+\t\tsetup_alternate_shallow(&shallow_lock, &alternate_shallow_file,\n+\t\t\t\t\tshallow,\n+\t\t\t\t\tWRITE_SHALLOW_NO_CUT);\n \tif (get_pack(args, fd, pack_lockfile))\n \t\tdie(\"git fetch-pack: fetch failed.\");\n \n@@ -922,6 +925,7 @@ struct ref *fetch_pack(struct fetch_pack_args *args,\n \t\t const struct ref *ref,\n \t\t const char *dest,\n \t\t struct ref **sought, int nr_sought,\n+\t\t struct extra_have_objects *shallow,\n \t\t char **pack_lockfile)\n {\n \tstruct ref *ref_cpy;\n@@ -934,14 +938,30 @@ struct ref *fetch_pack(struct fetch_pack_args *args,\n \t\tpacket_flush(fd[1]);\n \t\tdie(\"no matching remote head\");\n \t}\n-\tref_cpy = do_fetch_pack(args, fd, ref, sought, nr_sought, pack_lockfile);\n+\tref_cpy = do_fetch_pack(args, fd, ref, sought, nr_sought,\n+\t\t\t\tshallow, pack_lockfile);\n \n \tif (alternate_shallow_file) {\n \t\tif (*alternate_shallow_file == '\\0') { /* --unshallow */\n \t\t\tunlink_or_warn(git_path(\"shallow\"));\n \t\t\trollback_lock_file(&shallow_lock);\n-\t\t} else\n+\t\t} else {\n+\t\t\t/*\n+\t\t\t * The server is a shallow clone and it sends\n+\t\t\t * us all of its shallow grafts. Some may be\n+\t\t\t * needed if the server sends objects down to\n+\t\t\t * the bottom. Remove all unused grafts.\n+\t\t\t */\n+\t\t\tif (shallow->nr) {\n+\t\t\t\treprepare_packed_git();\n+\t\t\t\tsetup_alternate_shallow(&shallow_lock,\n+\t\t\t\t\t\t\t&alternate_shallow_file,\n+\t\t\t\t\t\t\tshallow,\n+\t\t\t\t\t\t\tWRITE_SHALLOW_NO_CUT |\n+\t\t\t\t\t\t\tWRITE_SHALLOW_REWRITE);\n+\t\t\t}\n \t\t\tcommit_lock_file(&shallow_lock);\n+\t\t}\n \t}\n \n \treprepare_packed_git();\ndiff --git a/fetch-pack.h b/fetch-pack.h\nindex 40f08ba..c55bf1c 100644\n--- a/fetch-pack.h\n+++ b/fetch-pack.h\n@@ -32,6 +32,7 @@ struct ref *fetch_pack(struct fetch_pack_args *args,\n \t\t const char *dest,\n \t\t struct ref **sought,\n \t\t int nr_sought,\n+\t\t struct extra_have_objects *shallow,\n \t\t char **pack_lockfile);\n \n #endif\ndiff --git a/shallow.c b/shallow.c\nindex ee9edd4..337f676 100644\n--- a/shallow.c\n+++ b/shallow.c\n@@ -2,6 +2,7 @@\n #include \"commit.h\"\n #include \"tag.h\"\n #include \"pkt-line.h\"\n+#include \"connected.h\"\n \n static int is_shallow = -1;\n static struct stat shallow_stat;\n@@ -183,26 +184,103 @@ static int write_one_shallow(const struct commit_graft *graft, void *cb_data)\n \treturn 0;\n }\n \n-int write_shallow_commits(struct strbuf *out, int use_pack_protocol)\n+static int sha1_iterate(void *cb, unsigned char *sha1)\n+{\n+\tif (is_null_sha1(cb))\n+\t\treturn 1;\n+\thashcpy(sha1, cb);\n+\thashclr(cb);\n+\treturn 0;\n+}\n+\n+struct connectivity_cache_item {\n+\tunsigned char sha1[20];\n+\tint good;\n+};\n+\n+static int good_connectivity_from(const unsigned char *sha1)\n+{\n+\tstatic struct connectivity_cache_item *cache;\n+\tstatic int nr, alloc;\n+\tint i;\n+\tunsigned char sha1_cb[20];\n+\n+\tfor (i = 0; i < nr; i++)\n+\t\tif (!hashcmp(cache[i].sha1, sha1))\n+\t\t\treturn cache[i].good;\n+\n+\tALLOC_GROW(cache, nr + 1, alloc);\n+\thashcpy(sha1_cb, sha1);\n+\thashcpy(cache[nr].sha1, sha1);\n+\tcache[nr].good = !check_everything_connected(sha1_iterate, 1, sha1_cb);\n+\tnr++;\n+\treturn cache[nr - 1].good;\n+}\n+\n+int write_shallow_commits(struct strbuf *out, int use_pack_protocol,\n+\t\t\t struct extra_have_objects *extra,\n+\t\t\t unsigned int flags)\n {\n \tstruct write_shallow_data data;\n+\tint i;\n \tdata.out = out;\n \tdata.use_pack_protocol = use_pack_protocol;\n \tdata.count = 0;\n \tfor_each_commit_graft(write_one_shallow, &data);\n+\tif (!extra)\n+\t\treturn data.count;\n+\n+\tfor (i = 0; i < extra->nr; i++) {\n+\t\tif ((flags & WRITE_SHALLOW_NO_CUT) &&\n+\t\t good_connectivity_from(extra->array[i]))\n+\t\t\t/*\n+\t\t\t * The server may have even shorter history\n+\t\t\t * than the client (e.g. the client has\n+\t\t\t * A-B-C-D but the server only has C-D). We do\n+\t\t\t * NOT want to cut client's history down to\n+\t\t\t * C-D simply because the server is set up\n+\t\t\t * so. If we don't have \"C\" and the server\n+\t\t\t * sends C-D to us, then we set up a graft at\n+\t\t\t * \"C\".\n+\t\t\t *\n+\t\t\t * If the client has (broken)-B-C-D, then \"C\"\n+\t\t\t * _is_ accepted. The -B-C-D is an incomplete\n+\t\t\t * island that normally has no connectivity to\n+\t\t\t * the rest of the repo.\n+\t\t\t */\n+\t\t\tcontinue;\n+\t\tif ((flags & WRITE_SHALLOW_NO_UNUSED) &&\n+\t\t !has_sha1_file(extra->array[i]))\n+\t\t\tcontinue;\n+\t\tstrbuf_addstr(out, sha1_to_hex(extra->array[i]));\n+\t\tstrbuf_addch(out, '\\n');\n+\t\tdata.count++;\n+\t}\n \treturn data.count;\n }\n \n void setup_alternate_shallow(struct lock_file *shallow_lock,\n-\t\t\t const char **alternate_shallow_file)\n+\t\t\t const char **alternate_shallow_file,\n+\t\t\t struct extra_have_objects *extra,\n+\t\t\t unsigned int flags)\n {\n \tstruct strbuf sb = STRBUF_INIT;\n+\tint rewrite = flags & WRITE_SHALLOW_REWRITE;\n \tint fd;\n \n-\tcheck_shallow_file_for_update();\n-\tfd = hold_lock_file_for_update(shallow_lock, git_path(\"shallow\"),\n-\t\t\t\t LOCK_DIE_ON_ERROR);\n-\tif (write_shallow_commits(&sb, 0)) {\n+\tif (!rewrite)\n+\t\tfd = hold_lock_file_for_update(shallow_lock,\n+\t\t\t\t\t git_path(\"shallow\"),\n+\t\t\t\t\t LOCK_DIE_ON_ERROR);\n+\telse {\n+\t\tflags |= WRITE_SHALLOW_NO_UNUSED;\n+\t\tfd = shallow_lock->fd;\n+\t}\n+\tif (write_shallow_commits(&sb, 0, extra, flags)) {\n+\t\tif (rewrite && (lseek(fd, 0, SEEK_SET) == -1 ||\n+\t\t\t\tftruncate(fd, 0) == -1))\n+\t\t\tdie_errno(\"unable to truncate the new shallow file %s\",\n+\t\t\t\t shallow_lock->filename);\n \t\tif (write_in_full(fd, sb.buf, sb.len) != sb.len)\n \t\t\tdie_errno(\"failed to write to %s\",\n \t\t\t\t shallow_lock->filename);\ndiff --git a/t/t5536-fetch-shallow.sh b/t/t5536-fetch-shallow.sh\nnew file mode 100755\nindex 0000000..15a8208\n--- /dev/null\n+++ b/t/t5536-fetch-shallow.sh\n@@ -0,0 +1,105 @@\n+#!/bin/sh\n+\n+test_description='fetch/clone from a shallow clone'\n+\n+. ./test-lib.sh\n+\n+commit() {\n+\techo \"$1\" >tracked &&\n+\tgit add tracked &&\n+\tgit commit -m \"$1\"\n+}\n+\n+test_expect_success 'setup' '\n+\tcommit 1 &&\n+\tcommit 2 &&\n+\tcommit 3 &&\n+\tcommit 4 &&\n+\tgit config --global transfer.fsckObjects true\n+'\n+\n+test_expect_success 'setup shallow clone' '\n+\tgit clone --no-local --depth=2 .git shallow &&\n+\tgit --git-dir=shallow/.git log --format=%s >actual &&\n+\tcat <expect &&\n+4\n+3\n+EOF\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'clone from shallow clone' '\n+\tgit clone --no-local shallow shallow2 &&\n+\t(\n+\tcd shallow2 &&\n+\tgit fsck &&\n+\tgit log --format=%s >actual &&\n+\tcat <expect &&\n+4\n+3\n+EOF\n+\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'fetch from shallow clone' '\n+\t(\n+\tcd shallow &&\n+\tcommit 5\n+\t) &&\n+\t(\n+\tcd shallow2 &&\n+\tgit fetch &&\n+\tgit fsck &&\n+\tgit log --format=%s origin/master >actual &&\n+\tcat <expect &&\n+5\n+4\n+3\n+EOF\n+\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'fetch --depth from shallow clone' '\n+\t(\n+\tcd shallow &&\n+\tcommit 6\n+\t) &&\n+\t(\n+\tcd shallow2 &&\n+\tgit fetch --depth=2 &&\n+\tgit fsck &&\n+\tgit log --format=%s origin/master >actual &&\n+\tcat <expect &&\n+6\n+5\n+EOF\n+\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'fetch something upstream has but hidden by clients shallow boundaries' '\n+\t# the blob \"1\" is available in .git but hidden by the\n+\t# shallow2/.git/shallow and it should be resent\n+\t! git --git-dir=shallow2/.git cat-file blob `echo 1|git hash-object --stdin` >/dev/null &&\n+\techo 1 > 1.t &&\n+\tgit add 1.t &&\n+\tgit commit -m add-1-back &&\n+\t(\n+\tcd shallow2 &&\n+\tgit fetch ../.git +refs/heads/master:refs/remotes/top/master &&\n+\tgit fsck &&\n+\tgit log --format=%s top/master >actual &&\n+\tcat <expect &&\n+add-1-back\n+4\n+3\n+EOF\n+\ttest_cmp expect actual\n+\t) &&\n+\tgit --git-dir=shallow2/.git cat-file blob `echo 1|git hash-object --stdin` >/dev/null\n+\n+'\n+\n+test_done\ndiff --git a/transport.c b/transport.c\nindex 10a8cb8..f167ca5 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -453,6 +453,7 @@ struct git_transport_data {\n \tint fd[2];\n \tunsigned got_remote_heads : 1;\n \tstruct extra_have_objects extra_have;\n+\tstruct extra_have_objects shallow;\n };\n \n static int set_git_option(struct git_transport_options *opts,\n@@ -509,7 +510,8 @@ static struct ref *get_refs_via_connect(struct transport *transport, int for_pus\n \n \tconnect_setup(transport, for_push, 0);\n \tget_remote_heads(data->fd[0], NULL, 0, &refs,\n-\t\t\t for_push ? REF_NORMAL : 0, &data->extra_have, NULL);\n+\t\t\t for_push ? REF_NORMAL : 0,\n+\t\t\t &data->extra_have, &data->shallow);\n \tdata->got_remote_heads = 1;\n \n \treturn refs;\n@@ -540,13 +542,13 @@ static int fetch_refs_via_pack(struct transport *transport,\n \tif (!data->got_remote_heads) {\n \t\tconnect_setup(transport, 0, 0);\n \t\tget_remote_heads(data->fd[0], NULL, 0, &refs_tmp, 0,\n-\t\t\t\t NULL, NULL);\n+\t\t\t\t NULL, &data->shallow);\n \t\tdata->got_remote_heads = 1;\n \t}\n \n \trefs = fetch_pack(&args, data->fd, data->conn,\n \t\t\t refs_tmp ? refs_tmp : transport->remote_refs,\n-\t\t\t dest, to_fetch, nr_heads,\n+\t\t\t dest, to_fetch, nr_heads, &data->shallow,\n \t\t\t &transport->pack_lockfile);\n \tclose(data->fd[0]);\n \tclose(data->fd[1]);\n@@ -803,6 +805,8 @@ static int git_transport_push(struct transport *transport, struct ref *remote_re\n \t\tget_remote_heads(data->fd[0], NULL, 0, &tmp_refs, REF_NORMAL, NULL, NULL);\n \t\tdata->got_remote_heads = 1;\n \t}\n+\tif (data->shallow.nr)\n+\t\tdie(\"pushing to a shallow repository is not supported\");\n \n \tmemset(&args, 0, sizeof(args));\n \targs.send_mirror = !!(flags & TRANSPORT_PUSH_MIRROR);\n-- \n1.8.2.83.gc99314b\n"},{"id":"223836","messageId":"1374314290-5976-7-git-send-email-pclouds@gmail.com","threadId":"34452","inReplyTo":"1374314290-5976-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v2 06/16] {send,receive}-pack: support pushing from a shallow clone","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-07-20T09:58:00Z","receivedAt":"2013-07-20T09:58:00Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Pushing from a shallow clone using today's send-pack and receive-pack\nmay work, if the transferred pack does not ends up at any graft\npoints. If it does, recent receive-pack that does connectivity check\nwill reject the push. If receive-pack is old, the upstream repo\nbecomes corrupt.\n\nThe pack protocol is updated and send-pack now sends all shallow\ngrafts before it sends the commands, if the repo is shallow. This\nprotocol extension will break current receive-pack, which is intended,\nmostly to stop corrupting the upstream repo.\n\nThe receiver end, the newreceive-pack, does something similar to\nfetch-pack: it creates a temporary shallow file with grafts from\nsend-pack, then receives the pack, and finally writes the refined\nshallow file down.\n\nshadow file is not cleaned up after deleting (or force updating) a ref\nif that ref is the only way to reach the graft points. The reason is\nonce we delete graft points, we can't recover. That may make reflog\nentries on server useless. Leave that for the administrators to decide\nwhen to clean up shadow file (maybe at repack/gc time).\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n Documentation/technical/pack-protocol.txt | 4 +-\n builtin/receive-pack.c | 66 ++++++++++++++++++++++-----\n builtin/send-pack.c | 2 +-\n send-pack.c | 3 ++\n t/t5537-push-shallow.sh (new +x) | 74 +++++++++++++++++++++++++++++++\n 5 files changed, 137 insertions(+), 12 deletions(-)\n create mode 100755 t/t5537-push-shallow.sh\n\ndiff --git a/Documentation/technical/pack-protocol.txt b/Documentation/technical/pack-protocol.txt\nindex eb8edd1..c73b62f 100644\n--- a/Documentation/technical/pack-protocol.txt\n+++ b/Documentation/technical/pack-protocol.txt\n@@ -464,7 +464,9 @@ contain all the objects that the server will need to complete the new\n references.\n \n ----\n- update-request = command-list [pack-file]\n+ update-request = *shallow command-list [pack-file]\n+\n+ shallow = PKT-LINE(\"shallow\" SP obj-id)\n \n command-list = PKT-LINE(command NUL capability-list LF)\n \t\t *PKT-LINE(command LF)\ndiff --git a/builtin/receive-pack.c b/builtin/receive-pack.c\nindex 6ffe526..cbb2025 100644\n--- a/builtin/receive-pack.c\n+++ b/builtin/receive-pack.c\n@@ -41,6 +41,10 @@ static int auto_gc = 1;\n static const char *head_name;\n static void *head_name_to_free;\n static int sent_capabilities;\n+static int shallow_changed;\n+static const char* alternate_shallow_file;\n+static struct lock_file shallow_lock;\n+static struct extra_have_objects shallow;\n \n static enum deny_action parse_deny_action(const char *var, const char *value)\n {\n@@ -751,6 +755,13 @@ static void execute_commands(struct command *commands, const char *unpacker_erro\n \t}\n }\n \n+static void add_extra_have(struct extra_have_objects *extra, unsigned char *sha1)\n+{\n+\tALLOC_GROW(extra->array, extra->nr + 1, extra->alloc);\n+\thashcpy(&(extra->array[extra->nr][0]), sha1);\n+\textra->nr++;\n+}\n+\n static struct command *read_head_info(void)\n {\n \tstruct command *commands = NULL;\n@@ -765,6 +776,17 @@ static struct command *read_head_info(void)\n \t\tline = packet_read_line(0, &len);\n \t\tif (!line)\n \t\t\tbreak;\n+\n+\t\tif (len == 48 && !prefixcmp(line, \"shallow \")) {\n+\t\t\tif (get_sha1_hex(line + 8, old_sha1))\n+\t\t\t\tdie(\"protocol error: expected shallow sha, got '%s'\", line + 8);\n+\t\t\tif (!has_sha1_file(old_sha1)) {\n+\t\t\t\tadd_extra_have(&shallow, old_sha1);\n+\t\t\t\tshallow_changed = 1;\n+\t\t\t}\n+\t\t\tcontinue;\n+\t\t}\n+\n \t\tif (len < 83 ||\n \t\t line[40] != ' ' ||\n \t\t line[81] != ' ' ||\n@@ -827,6 +849,13 @@ static const char *unpack(int err_fd)\n \t\t\t ? transfer_fsck_objects\n \t\t\t : 0);\n \n+\tif (shallow_changed)\n+\t\tsetup_alternate_shallow(&shallow_lock,\n+\t\t\t\t\t&alternate_shallow_file,\n+\t\t\t\t\t&shallow,\n+\t\t\t\t\tWRITE_SHALLOW_NO_CUT);\n+\n+\n \thdr_err = parse_pack_header(&hdr);\n \tif (hdr_err) {\n \t\tif (err_fd > 0)\n@@ -840,7 +869,12 @@ static const char *unpack(int err_fd)\n \tif (ntohl(hdr.hdr_entries) < unpack_limit) {\n \t\tint code, i = 0;\n \t\tstruct child_process child;\n-\t\tconst char *unpacker[5];\n+\t\tconst char *unpacker[7];\n+\t\tif (alternate_shallow_file &&\n+\t\t *alternate_shallow_file) {\n+\t\t\tunpacker[i++] = \"--shallow-file\";\n+\t\t\tunpacker[i++] = alternate_shallow_file;\n+\t\t}\n \t\tunpacker[i++] = \"unpack-objects\";\n \t\tif (quiet)\n \t\t\tunpacker[i++] = \"-q\";\n@@ -854,15 +888,19 @@ static const char *unpack(int err_fd)\n \t\tchild.err = err_fd;\n \t\tchild.git_cmd = 1;\n \t\tcode = run_command(&child);\n-\t\tif (!code)\n-\t\t\treturn NULL;\n-\t\treturn \"unpack-objects abnormal exit\";\n+\t\tif (code)\n+\t\t\treturn \"unpack-objects abnormal exit\";\n \t} else {\n-\t\tconst char *keeper[7];\n+\t\tconst char *keeper[9];\n \t\tint s, status, i = 0;\n \t\tchar keep_arg[256];\n \t\tstruct child_process ip;\n \n+\t\tif (alternate_shallow_file &&\n+\t\t *alternate_shallow_file) {\n+\t\t\tkeeper[i++] = \"--shallow-file\";\n+\t\t\tkeeper[i++] = alternate_shallow_file;\n+\t\t}\n \t\ts = sprintf(keep_arg, \"--keep=receive-pack %\"PRIuMAX\" on \", (uintmax_t) getpid());\n \t\tif (gethostname(keep_arg + s, sizeof(keep_arg) - s))\n \t\t\tstrcpy(keep_arg + s, \"localhost\");\n@@ -887,12 +925,20 @@ static const char *unpack(int err_fd)\n \t\tpack_lockfile = index_pack_lockfile(ip.out);\n \t\tclose(ip.out);\n \t\tstatus = finish_command(&ip);\n-\t\tif (!status) {\n-\t\t\treprepare_packed_git();\n-\t\t\treturn NULL;\n-\t\t}\n-\t\treturn \"index-pack abnormal exit\";\n+\t\tif (status)\n+\t\t\treturn \"index-pack abnormal exit\";\n+\t\treprepare_packed_git();\n+\t}\n+\n+\tif (shallow_changed) {\n+\t\tsetup_alternate_shallow(&shallow_lock,\n+\t\t\t\t\t&alternate_shallow_file,\n+\t\t\t\t\t&shallow,\n+\t\t\t\t\tWRITE_SHALLOW_NO_CUT |\n+\t\t\t\t\tWRITE_SHALLOW_REWRITE);\n+\t\tcommit_lock_file(&shallow_lock);\n \t}\n+\treturn NULL;\n }\n \n static const char *unpack_with_sideband(void)\ndiff --git a/builtin/send-pack.c b/builtin/send-pack.c\nindex 4be5931..7dd254f 100644\n--- a/builtin/send-pack.c\n+++ b/builtin/send-pack.c\n@@ -184,7 +184,7 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)\n \t (send_all && args.send_mirror))\n \t\tusage(send_pack_usage);\n \n-\tif (is_repository_shallow())\n+\tif (is_repository_shallow() && args.stateless_rpc)\n \t\tdie(\"attempt to push from a shallow repository\");\n \n \tif (remote_name) {\ndiff --git a/send-pack.c b/send-pack.c\nindex 7d172ef..dcd3345 100644\n--- a/send-pack.c\n+++ b/send-pack.c\n@@ -212,6 +212,9 @@ int send_pack(struct send_pack_args *args,\n \t\treturn 0;\n \t}\n \n+\tif (!args->dry_run)\n+\t\tadvertise_shallow_grafts(out);\n+\n \t/*\n \t * Finally, tell the other end!\n \t */\ndiff --git a/t/t5537-push-shallow.sh b/t/t5537-push-shallow.sh\nnew file mode 100755\nindex 0000000..37ad69b\n--- /dev/null\n+++ b/t/t5537-push-shallow.sh\n@@ -0,0 +1,74 @@\n+#!/bin/sh\n+\n+test_description='push from/to a shallow clone'\n+\n+. ./test-lib.sh\n+\n+commit() {\n+\techo \"$1\" >tracked &&\n+\tgit add tracked &&\n+\tgit commit -m \"$1\"\n+}\n+\n+test_expect_success 'setup' '\n+\tgit config --global transfer.fsckObjects true &&\n+\tcommit 1 &&\n+\tcommit 2 &&\n+\tcommit 3 &&\n+\tcommit 4 &&\n+\t(\n+\tgit init full-abc &&\n+\tcd full-abc &&\n+\tcommit a &&\n+\tcommit b &&\n+\tcommit c\n+\t) &&\n+\tgit clone --no-local --depth=2 .git shallow &&\n+\tgit --git-dir=shallow/.git log --format=%s >actual &&\n+\tcat <expect &&\n+4\n+3\n+EOF\n+\ttest_cmp expect actual &&\n+\tgit clone --no-local --depth=2 full-abc/.git shallow2 &&\n+\tgit --git-dir=shallow2/.git log --format=%s >actual &&\n+\tcat <expect &&\n+c\n+b\n+EOF\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'push from shallow clone' '\n+\t(\n+\tcd shallow &&\n+\tcommit 5 &&\n+\tgit push ../.git +master:refs/remotes/shallow/master\n+\t) &&\n+\tgit log --format=%s shallow/master >actual &&\n+\tgit fsck &&\n+\tcat <expect &&\n+5\n+4\n+3\n+2\n+1\n+EOF\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'push from shallow clone, with grafted roots' '\n+\t(\n+\tcd shallow2 &&\n+\tgit push ../.git +master:refs/remotes/shallow2/master\n+\t) &&\n+\tgit log --format=%s shallow2/master >actual &&\n+\tgit fsck &&\n+\tcat <expect &&\n+c\n+b\n+EOF\n+\ttest_cmp expect actual\n+'\n+\n+test_done\n-- \n1.8.2.83.gc99314b\n"},{"id":"223835","messageId":"1374314290-5976-8-git-send-email-pclouds@gmail.com","threadId":"34452","inReplyTo":"1374314290-5976-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v2 07/16] send-pack: support pushing to a shallow clone","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-07-20T09:58:01Z","receivedAt":"2013-07-20T09:58:01Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"When send-pack receives \"shallow\" lines from receive-pack, it knows\nthe other end does not have a complete commit chains. It restrict\nitself to the commits that are not cut out by either end to make sure\nthe result pack is usuable by receive-pack.\n\nThe same technique here, using setup_alternate_shallow() and\n--shallow-file, might simplify similar code in upload-pack.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/send-pack.c | 7 +++++--\n send-pack.c | 37 ++++++++++++++++++++++++++++++++-----\n send-pack.h | 4 +++-\n t/t5537-push-shallow.sh | 36 ++++++++++++++++++++++++++++++++++++\n transport.c | 7 +++----\n 5 files changed, 79 insertions(+), 12 deletions(-)\n\ndiff --git a/builtin/send-pack.c b/builtin/send-pack.c\nindex 7dd254f..14a73ba 100644\n--- a/builtin/send-pack.c\n+++ b/builtin/send-pack.c\n@@ -94,6 +94,7 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)\n \tint fd[2];\n \tstruct child_process *conn;\n \tstruct extra_have_objects extra_have;\n+\tstruct extra_have_objects shallow;\n \tstruct ref *remote_refs, *local_refs;\n \tint ret;\n \tint helper_status = 0;\n@@ -209,8 +210,10 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)\n \t}\n \n \tmemset(&extra_have, 0, sizeof(extra_have));\n+\tmemset(&shallow, 0, sizeof(shallow));\n \n-\tget_remote_heads(fd[0], NULL, 0, &remote_refs, REF_NORMAL, &extra_have, NULL);\n+\tget_remote_heads(fd[0], NULL, 0, &remote_refs, REF_NORMAL,\n+\t\t\t &extra_have, &shallow);\n \n \ttransport_verify_remote_names(nr_refspecs, refspecs);\n \n@@ -230,7 +233,7 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)\n \tset_ref_status_for_push(remote_refs, args.send_mirror,\n \t\targs.force_update);\n \n-\tret = send_pack(&args, fd, conn, remote_refs, &extra_have);\n+\tret = send_pack(&args, fd, conn, remote_refs, &extra_have, &shallow);\n \n \tif (helper_status)\n \t\tprint_helper_status(remote_refs);\ndiff --git a/send-pack.c b/send-pack.c\nindex dcd3345..1b828b0 100644\n--- a/send-pack.c\n+++ b/send-pack.c\n@@ -27,14 +27,19 @@ static int feed_object(const unsigned char *sha1, int fd, int negative)\n /*\n * Make a pack stream and spit it out into file descriptor fd\n */\n-static int pack_objects(int fd, struct ref *refs, struct extra_have_objects *extra, struct send_pack_args *args)\n+static int pack_objects(int fd, struct ref *refs,\n+\t\t\tstruct extra_have_objects *extra,\n+\t\t\tstruct extra_have_objects *extra_shallow,\n+\t\t\tstruct send_pack_args *args)\n {\n \t/*\n \t * The child becomes pack-objects --revs; we feed\n \t * the revision parameters to it via its stdin and\n \t * let its stdout go back to the other end.\n \t */\n-\tconst char *argv[] = {\n+\tconst char *av[] = {\n+\t\t\"--shallow-file\",\n+\t\tNULL,\n \t\t\"pack-objects\",\n \t\t\"--all-progress-implied\",\n \t\t\"--revs\",\n@@ -45,10 +50,26 @@ static int pack_objects(int fd, struct ref *refs, struct extra_have_objects *ext\n \t\tNULL,\n \t\tNULL,\n \t};\n+\tconst char **argv;\n \tstruct child_process po;\n+\tstatic struct lock_file shallow_lock;\n+\tconst char *alternate_shallow_file = NULL;\n \tint i;\n \n-\ti = 4;\n+\tif (extra_shallow->nr) {\n+\t\tmemset(&shallow_lock, 0, sizeof(shallow_lock));\n+\t\t/* just to load up .git/shallow if exists */\n+\t\tis_repository_shallow();\n+\t\tsetup_alternate_shallow(&shallow_lock,\n+\t\t\t\t\t&alternate_shallow_file,\n+\t\t\t\t\textra_shallow, 0);\n+\t\tav[1] = alternate_shallow_file;\n+\t\targv = av;\n+\t\ti = 6;\n+\t} else {\n+\t\targv = &av[2];\n+\t\ti = 4;\n+\t}\n \tif (args->use_thin_pack)\n \t\targv[i++] = \"--thin\";\n \tif (args->use_ofs_delta)\n@@ -100,6 +121,10 @@ static int pack_objects(int fd, struct ref *refs, struct extra_have_objects *ext\n \n \tif (finish_command(&po))\n \t\treturn -1;\n+\n+\tif (extra_shallow->nr)\n+\t\trollback_lock_file(&shallow_lock);\n+\n \treturn 0;\n }\n \n@@ -176,7 +201,8 @@ static int sideband_demux(int in, int out, void *data)\n int send_pack(struct send_pack_args *args,\n \t int fd[], struct child_process *conn,\n \t struct ref *remote_refs,\n-\t struct extra_have_objects *extra_have)\n+\t struct extra_have_objects *extra_have,\n+\t struct extra_have_objects *extra_shallow)\n {\n \tint in = fd[0];\n \tint out = fd[1];\n@@ -294,7 +320,8 @@ int send_pack(struct send_pack_args *args,\n \t}\n \n \tif (new_refs && cmds_sent) {\n-\t\tif (pack_objects(out, remote_refs, extra_have, args) < 0) {\n+\t\tif (pack_objects(out, remote_refs, extra_have, extra_shallow,\n+\t\t\t\t args) < 0) {\n \t\t\tfor (ref = remote_refs; ref; ref = ref->next)\n \t\t\t\tref->status = REF_STATUS_NONE;\n \t\t\tif (args->stateless_rpc)\ndiff --git a/send-pack.h b/send-pack.h\nindex 05d7ab1..1c8a343 100644\n--- a/send-pack.h\n+++ b/send-pack.h\n@@ -16,6 +16,8 @@ struct send_pack_args {\n \n int send_pack(struct send_pack_args *args,\n \t int fd[], struct child_process *conn,\n-\t struct ref *remote_refs, struct extra_have_objects *extra_have);\n+\t struct ref *remote_refs,\n+\t struct extra_have_objects *extra_have,\n+\t struct extra_have_objects *extra_shallow);\n \n #endif\ndiff --git a/t/t5537-push-shallow.sh b/t/t5537-push-shallow.sh\nindex 37ad69b..beb85ce 100755\n--- a/t/t5537-push-shallow.sh\n+++ b/t/t5537-push-shallow.sh\n@@ -71,4 +71,40 @@ EOF\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'push from shallow to shallow' '\n+\t(\n+\tcd shallow &&\n+\tgit push ../shallow2/.git +master:refs/remotes/shallow/master\n+\t) &&\n+\t(\n+\tcd shallow2 &&\n+\tgit log --format=%s shallow/master >actual &&\n+\tgit fsck &&\n+\tcat <expect &&\n+5\n+4\n+3\n+EOF\n+\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'push from full to shallow' '\n+\t! git --git-dir=shallow2/.git cat-file blob `echo 1|git hash-object --stdin` &&\n+\tcommit 1 &&\n+\tgit push shallow2/.git +master:refs/remotes/top/master &&\n+\t(\n+\tcd shallow2 &&\n+\tgit log --format=%s top/master >actual &&\n+\tgit fsck &&\n+\tcat <expect &&\n+1\n+4\n+3\n+EOF\n+\ttest_cmp expect actual &&\n+\tgit cat-file blob `echo 1|git hash-object --stdin` >/dev/null\n+\t)\n+'\n+\n test_done\ndiff --git a/transport.c b/transport.c\nindex f167ca5..d592e8d 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -802,11 +802,10 @@ static int git_transport_push(struct transport *transport, struct ref *remote_re\n \t\tstruct ref *tmp_refs;\n \t\tconnect_setup(transport, 1, 0);\n \n-\t\tget_remote_heads(data->fd[0], NULL, 0, &tmp_refs, REF_NORMAL, NULL, NULL);\n+\t\tget_remote_heads(data->fd[0], NULL, 0, &tmp_refs, REF_NORMAL,\n+\t\t\t\t NULL, &data->shallow);\n \t\tdata->got_remote_heads = 1;\n \t}\n-\tif (data->shallow.nr)\n-\t\tdie(\"pushing to a shallow repository is not supported\");\n \n \tmemset(&args, 0, sizeof(args));\n \targs.send_mirror = !!(flags & TRANSPORT_PUSH_MIRROR);\n@@ -819,7 +818,7 @@ static int git_transport_push(struct transport *transport, struct ref *remote_re\n \targs.porcelain = !!(flags & TRANSPORT_PUSH_PORCELAIN);\n \n \tret = send_pack(&args, data->fd, data->conn, remote_refs,\n-\t\t\t&data->extra_have);\n+\t\t\t&data->extra_have, &data->shallow);\n \n \tclose(data->fd[1]);\n \tclose(data->fd[0]);\n-- \n1.8.2.83.gc99314b\n"},{"id":"223837","messageId":"1374314290-5976-9-git-send-email-pclouds@gmail.com","threadId":"34452","inReplyTo":"1374314290-5976-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v2 08/16] upload-pack: let pack-objects do the object counting in shallow case","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-07-20T09:58:02Z","receivedAt":"2013-07-20T09:58:02Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Remove the duplicate object counting code in upload-pack, dump out all\nregister_shallow()'d SHA-1 into a temporary shallow file and feed it\nto pack-objects. The end result is the same, except with less code,\nand fewer bytes sending over pipe to pack-objects.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n t/t5530-upload-pack-error.sh | 3 -\n upload-pack.c | 128 ++++++++++---------------------------------\n 2 files changed, 30 insertions(+), 101 deletions(-)\n\ndiff --git a/t/t5530-upload-pack-error.sh b/t/t5530-upload-pack-error.sh\nindex c983d36..3932e79 100755\n--- a/t/t5530-upload-pack-error.sh\n+++ b/t/t5530-upload-pack-error.sh\n@@ -54,9 +54,6 @@ test_expect_success 'upload-pack fails due to error in rev-list' '\n \tprintf \"0032want %s\\n0034shallow %s00000009done\\n0000\" \\\n \t\t$(git rev-parse HEAD) $(git rev-parse HEAD^) >input &&\n \ttest_must_fail git upload-pack . /dev/null 2>output.err &&\n-\t# pack-objects survived\n-\tgrep \"Total.*, reused\" output.err &&\n-\t# but there was an error, which must have been in rev-list\n \tgrep \"bad tree object\" output.err\n '\n \ndiff --git a/upload-pack.c b/upload-pack.c\nindex c377a3e..c3e68ae 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -35,7 +35,6 @@ static int no_done;\n static int use_thin_pack, use_ofs_delta, use_include_tag;\n static int no_progress, daemon_mode;\n static int allow_tip_sha1_in_want;\n-static int shallow_nr;\n static struct object_array have_obj;\n static struct object_array want_obj;\n static struct object_array extra_edge_obj;\n@@ -68,87 +67,32 @@ static ssize_t send_client_data(int fd, const char *data, ssize_t sz)\n \treturn sz;\n }\n \n-static FILE *pack_pipe = NULL;\n-static void show_commit(struct commit *commit, void *data)\n-{\n-\tif (commit->object.flags & BOUNDARY)\n-\t\tfputc('-', pack_pipe);\n-\tif (fputs(sha1_to_hex(commit->object.sha1), pack_pipe) < 0)\n-\t\tdie(\"broken output pipe\");\n-\tfputc('\\n', pack_pipe);\n-\tfflush(pack_pipe);\n-\tfree(commit->buffer);\n-\tcommit->buffer = NULL;\n-}\n-\n-static void show_object(struct object *obj,\n-\t\t\tconst struct name_path *path, const char *component,\n-\t\t\tvoid *cb_data)\n-{\n-\tshow_object_with_name(pack_pipe, obj, path, component);\n-}\n-\n-static void show_edge(struct commit *commit)\n-{\n-\tfprintf(pack_pipe, \"-%s\\n\", sha1_to_hex(commit->object.sha1));\n-}\n-\n-static int do_rev_list(int in, int out, void *user_data)\n-{\n-\tint i;\n-\tstruct rev_info revs;\n-\n-\tpack_pipe = xfdopen(out, \"w\");\n-\tinit_revisions(&revs, NULL);\n-\trevs.tag_objects = 1;\n-\trevs.tree_objects = 1;\n-\trevs.blob_objects = 1;\n-\tif (use_thin_pack)\n-\t\trevs.edge_hint = 1;\n-\n-\tfor (i = 0; i < want_obj.nr; i++) {\n-\t\tstruct object *o = want_obj.objects[i].item;\n-\t\t/* why??? */\n-\t\to->flags &= ~UNINTERESTING;\n-\t\tadd_pending_object(&revs, o, NULL);\n-\t}\n-\tfor (i = 0; i < have_obj.nr; i++) {\n-\t\tstruct object *o = have_obj.objects[i].item;\n-\t\to->flags |= UNINTERESTING;\n-\t\tadd_pending_object(&revs, o, NULL);\n-\t}\n-\tsetup_revisions(0, NULL, &revs, NULL);\n-\tif (prepare_revision_walk(&revs))\n-\t\tdie(\"revision walk setup failed\");\n-\tmark_edges_uninteresting(revs.commits, &revs, show_edge);\n-\tif (use_thin_pack)\n-\t\tfor (i = 0; i < extra_edge_obj.nr; i++)\n-\t\t\tfprintf(pack_pipe, \"-%s\\n\", sha1_to_hex(\n-\t\t\t\t\textra_edge_obj.objects[i].item->sha1));\n-\ttraverse_commit_list(&revs, show_commit, show_object, NULL);\n-\tfflush(pack_pipe);\n-\tfclose(pack_pipe);\n-\treturn 0;\n-}\n-\n static void create_pack_file(void)\n {\n-\tstruct async rev_list;\n \tstruct child_process pack_objects;\n \tchar data[8193], progress[128];\n \tchar abort_msg[] = \"aborting due to possible repository \"\n \t\t\"corruption on the remote side.\";\n \tint buffered = -1;\n \tssize_t sz;\n-\tconst char *argv[10];\n-\tint arg = 0;\n+\tconst char *argv[12];\n+\tint i, arg = 0;\n+\tFILE *pipe_fd;\n+\tstatic struct lock_file shallow_lock;\n+\tconst char *alternate_shallow_file = NULL;\n+\n+\tsetup_alternate_shallow(&shallow_lock,\n+\t\t\t\t&alternate_shallow_file,\n+\t\t\t\tNULL, 0);\n+\tif (*alternate_shallow_file) {\n+\t\targv[arg++] = \"--shallow-file\";\n+\t\targv[arg++] = alternate_shallow_file;\n+\t}\n \n \targv[arg++] = \"pack-objects\";\n-\tif (!shallow_nr) {\n-\t\targv[arg++] = \"--revs\";\n-\t\tif (use_thin_pack)\n-\t\t\targv[arg++] = \"--thin\";\n-\t}\n+\targv[arg++] = \"--revs\";\n+\tif (use_thin_pack)\n+\t\targv[arg++] = \"--thin\";\n \n \targv[arg++] = \"--stdout\";\n \tif (!no_progress)\n@@ -169,28 +113,18 @@ static void create_pack_file(void)\n \tif (start_command(&pack_objects))\n \t\tdie(\"git upload-pack: unable to fork git-pack-objects\");\n \n-\tif (shallow_nr) {\n-\t\tmemset(&rev_list, 0, sizeof(rev_list));\n-\t\trev_list.proc = do_rev_list;\n-\t\trev_list.out = pack_objects.in;\n-\t\tif (start_async(&rev_list))\n-\t\t\tdie(\"git upload-pack: unable to fork git-rev-list\");\n-\t}\n-\telse {\n-\t\tFILE *pipe_fd = xfdopen(pack_objects.in, \"w\");\n-\t\tint i;\n+\tpipe_fd = xfdopen(pack_objects.in, \"w\");\n \n-\t\tfor (i = 0; i < want_obj.nr; i++)\n-\t\t\tfprintf(pipe_fd, \"%s\\n\",\n-\t\t\t\tsha1_to_hex(want_obj.objects[i].item->sha1));\n-\t\tfprintf(pipe_fd, \"--not\\n\");\n-\t\tfor (i = 0; i < have_obj.nr; i++)\n-\t\t\tfprintf(pipe_fd, \"%s\\n\",\n-\t\t\t\tsha1_to_hex(have_obj.objects[i].item->sha1));\n-\t\tfprintf(pipe_fd, \"\\n\");\n-\t\tfflush(pipe_fd);\n-\t\tfclose(pipe_fd);\n-\t}\n+\tfor (i = 0; i < want_obj.nr; i++)\n+\t\tfprintf(pipe_fd, \"%s\\n\",\n+\t\t\tsha1_to_hex(want_obj.objects[i].item->sha1));\n+\tfprintf(pipe_fd, \"--not\\n\");\n+\tfor (i = 0; i < have_obj.nr; i++)\n+\t\tfprintf(pipe_fd, \"%s\\n\",\n+\t\t\tsha1_to_hex(have_obj.objects[i].item->sha1));\n+\tfprintf(pipe_fd, \"\\n\");\n+\tfflush(pipe_fd);\n+\tfclose(pipe_fd);\n \n \n \t/* We read from pack_objects.err to capture stderr output for\n@@ -290,8 +224,9 @@ static void create_pack_file(void)\n \t\terror(\"git upload-pack: git-pack-objects died with error.\");\n \t\tgoto fail;\n \t}\n-\tif (shallow_nr && finish_async(&rev_list))\n-\t\tgoto fail;\t/* error was already reported */\n+\n+\tif (*alternate_shallow_file)\n+\t\trollback_lock_file(&shallow_lock);\n \n \t/* flush the data */\n \tif (0 <= buffered) {\n@@ -575,7 +510,6 @@ static void receive_needs(void)\n \tint depth = 0;\n \tint has_non_tip = 0;\n \n-\tshallow_nr = 0;\n \tfor (;;) {\n \t\tstruct object *o;\n \t\tconst char *features;\n@@ -679,7 +613,6 @@ static void receive_needs(void)\n \t\t\t\tpacket_write(1, \"shallow %s\",\n \t\t\t\t\t\tsha1_to_hex(object->sha1));\n \t\t\t\tregister_shallow(object->sha1);\n-\t\t\t\tshallow_nr++;\n \t\t\t}\n \t\t\tresult = result->next;\n \t\t}\n@@ -715,7 +648,6 @@ static void receive_needs(void)\n \t\t\t\tregister_shallow(shallows.objects[i].item->sha1);\n \t\t}\n \n-\tshallow_nr += shallows.nr;\n \tfree(shallows.objects);\n }\n \n-- \n1.8.2.83.gc99314b\n"},{"id":"223838","messageId":"1374314290-5976-10-git-send-email-pclouds@gmail.com","threadId":"34452","inReplyTo":"1374314290-5976-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v2 09/16] pack-protocol.txt: a bit about smart http","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-07-20T09:58:03Z","receivedAt":"2013-07-20T09:58:03Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n Documentation/technical/pack-protocol.txt | 66 +++++++++++++++++++++++++++++++\n 1 file changed, 66 insertions(+)\n\ndiff --git a/Documentation/technical/pack-protocol.txt b/Documentation/technical/pack-protocol.txt\nindex c73b62f..a1672bc 100644\n--- a/Documentation/technical/pack-protocol.txt\n+++ b/Documentation/technical/pack-protocol.txt\n@@ -551,3 +551,69 @@ An example client/server communication might look like this:\n S: 0018ok refs/heads/debug\\n\n S: 002ang refs/heads/master non-fast-forward\\n\n ----\n+\n+Smart HTTP Transport\n+--------------------\n+\n+Smart HTTP protocol is basically \"git protocol on top of http\". The\n+base protocol is modified slightly to fit HTTP processing model: no\n+bidirectional full-duplex connections, the program may read the\n+request, write a response and must exit.\n+\n+Reference Discovery\n+-------------------\n+\n+The server end always sends the list of references in both push and\n+fetch cases. This ref list is retrieved by the client's sending HTTP\n+GET request to a smart http url ending with\n+\"/info/refs?service=\" where could be either\n+git-upload-pack or git-receive-pack for fetching or pushing\n+respectively. The output is in pkt-line format.\n+\n+----\n+ advertised-refs = service\n+\t\t flush-pkt\n+\t\t (no-refs / list-of-refs)\n+\t\t flush-pkt\n+\n+ service = PKT-LINE(\"# service=\" service-name)\n+ service-name = (\"git-upload-pack\" / \"git-receive-pack\")\n+\n+ no-refs = PKT-LINE(zero-id SP \"capabilities^{}\"\n+\t\t NUL capability-list LF)\n+\n+ list-of-refs = first-ref *other-ref\n+ first-ref = PKT-LINE(obj-id SP refname\n+\t\t NUL capability-list LF)\n+\n+ other-ref = PKT-LINE(other-tip / other-peeled)\n+ other-tip = obj-id SP refname LF\n+ other-peeled = obj-id SP refname \"^{}\" LF\n+\n+ capability-list = capability *(SP capability)\n+ capability = 1*(LC_ALPHA / DIGIT / \"-\" / \"_\")\n+ LC_ALPHA = %x61-7A\n+----\n+\n+Packfile Negotiation\n+--------------------\n+\n+For fetching, packet negotiation is via a series of HTTP POST requests\n+to an url ending with \"/git-upload-pack\" with the content in pkt-line\n+format. git-upload-pack's response consists of a \"service\" line like\n+in Reference Discovery followed by normal git-upload-pack packet\n+lines. Capability multi_ack_detailed is required by Smart HTTP.\n+\n+Common objects that are discovered are appended onto the request as\n+have lines and are sent again on the next request. This allows the\n+remote side to reinitialize its in-memory list of common objects\n+during the next request and the remote does not need to maintain the\n+negotiation state.\n+\n+Reference Update Request\n+------------------------\n+\n+For pushing, a HTTP POST request is sent to an url ending with\n+\"/git-receive-pack\" with the content in pkt-line format.\n+git-receive-pack's response consists of a \"service\" line like in\n+Reference Discovery followed by normal git-receive-pack packet lines.\n-- \n1.8.2.83.gc99314b\n"},{"id":"223839","messageId":"1374314290-5976-11-git-send-email-pclouds@gmail.com","threadId":"34452","inReplyTo":"1374314290-5976-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v2 10/16] Add document for command arguments for supporting smart http","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-07-20T09:58:04Z","receivedAt":"2013-07-20T09:58:04Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n Documentation/git-fetch-pack.txt | 11 +++++++----\n Documentation/git-receive-pack.txt | 16 +++++++++++++++-\n Documentation/git-send-pack.txt | 9 ++++++++-\n Documentation/git-upload-pack.txt | 13 ++++++++++++-\n 4 files changed, 42 insertions(+), 7 deletions(-)\n\ndiff --git a/Documentation/git-fetch-pack.txt b/Documentation/git-fetch-pack.txt\nindex 1e71754..85a9437 100644\n--- a/Documentation/git-fetch-pack.txt\n+++ b/Documentation/git-fetch-pack.txt\n@@ -9,10 +9,7 @@ git-fetch-pack - Receive missing objects from another repository\n SYNOPSIS\n --------\n [verse]\n-'git fetch-pack' [--all] [--quiet|-q] [--keep|-k] [--thin] [--include-tag]\n-\t[--upload-pack=]\n-\t[--depth=] [--no-progress]\n-\t[-v] [:] [...]\n+'git fetch-pack' [options] [:] [...]\n \n DESCRIPTION\n -----------\n@@ -90,6 +87,12 @@ be in a separate packet, and the list must end with a flush packet.\n --no-progress::\n \tDo not show the progress.\n \n+--stateless-rpc::\n+\tSmart HTTP mode.\n+\n+--lock-pack::\n+\tIssue \"lock\" command to the remote helper via stdout.\n+\n -v::\n \tRun verbosely.\n \ndiff --git a/Documentation/git-receive-pack.txt b/Documentation/git-receive-pack.txt\nindex b1f7dc6..b56d2eb 100644\n--- a/Documentation/git-receive-pack.txt\n+++ b/Documentation/git-receive-pack.txt\n@@ -9,7 +9,7 @@ git-receive-pack - Receive what is pushed into the repository\n SYNOPSIS\n --------\n [verse]\n-'git-receive-pack' \n+'git-receive-pack' [options] \n \n DESCRIPTION\n -----------\n@@ -35,6 +35,20 @@ are not fast-forwards.\n \n OPTIONS\n -------\n+--stateless-rpc::\n+\tgit-receive-pack performs only a single read-write cycle with\n+\tstdin and stdout to fit with the HTTP POST request processing\n+\tmodel where a program may read the request, write a response,\n+\tand must exit.\n+\n+--advertise-refs::\n+\tOnly the initial ref advertisement is output then exits\n+\timmediately to fit with the HTTP GET request model, where no\n+\trequest content is received but a response must be produced.\n+\n+--quiet::\n+\tMake unpack-objects at the receive-pack end quiet.\n+\n ::\n \tThe repository to sync into.\n \ndiff --git a/Documentation/git-send-pack.txt b/Documentation/git-send-pack.txt\nindex dc3a568..a88e7e0 100644\n--- a/Documentation/git-send-pack.txt\n+++ b/Documentation/git-send-pack.txt\n@@ -9,7 +9,7 @@ git-send-pack - Push objects over Git protocol to another repository\n SYNOPSIS\n --------\n [verse]\n-'git send-pack' [--all] [--dry-run] [--force] [--receive-pack=] [--verbose] [--thin] [:] [...]\n+'git send-pack' [options] [:] [...]\n \n DESCRIPTION\n -----------\n@@ -52,6 +52,13 @@ OPTIONS\n \tSend a \"thin\" pack, which records objects in deltified form based\n \ton objects not included in the pack to reduce network traffic.\n \n+--stateless-rpc::\n+\tSmart HTTP mode.\n+\n+--helper-status:\n+\tIssue status commands (e.g. \"ok\" or \"error\") to the remote\n+\thelp via stdout.\n+\n ::\n \tA remote host to house the repository. When this\n \tpart is specified, 'git-receive-pack' is invoked via\ndiff --git a/Documentation/git-upload-pack.txt b/Documentation/git-upload-pack.txt\nindex 0abc806..98d73cc 100644\n--- a/Documentation/git-upload-pack.txt\n+++ b/Documentation/git-upload-pack.txt\n@@ -9,7 +9,7 @@ git-upload-pack - Send objects packed back to git-fetch-pack\n SYNOPSIS\n --------\n [verse]\n-'git-upload-pack' [--strict] [--timeout=] \n+'git-upload-pack' [options] \n \n DESCRIPTION\n -----------\n@@ -31,6 +31,17 @@ OPTIONS\n --timeout=::\n \tInterrupt transfer after seconds of inactivity.\n \n+--stateless-rpc::\n+\tgit-upload-pack performs only a single read-write cycle with\n+\tstdin and stdout to fit with the HTTP POST request processing\n+\tmodel where a program may read the request, write a response,\n+\tand must exit.\n+\n+--advertise-refs::\n+\tOnly the initial ref advertisement is output then exits\n+\timmediately to fit with the HTTP GET request model, where no\n+\trequest content is received but a response must be produced.\n+\n ::\n \tThe repository to sync from.\n \n-- \n1.8.2.83.gc99314b\n"},{"id":"223840","messageId":"1374314290-5976-12-git-send-email-pclouds@gmail.com","threadId":"34452","inReplyTo":"1374314290-5976-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v2 11/16] {fetch,upload}-pack: support fetching from a shallow clone via smart http","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-07-20T09:58:05Z","receivedAt":"2013-07-20T09:58:05Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n Documentation/technical/pack-protocol.txt | 3 +++\n builtin/fetch-pack.c | 3 +--\n remote-curl.c | 4 +++-\n t/t5536-fetch-shallow.sh | 27 +++++++++++++++++++++++++++\n upload-pack.c | 2 --\n 5 files changed, 34 insertions(+), 5 deletions(-)\n\ndiff --git a/Documentation/technical/pack-protocol.txt b/Documentation/technical/pack-protocol.txt\nindex a1672bc..5013652 100644\n--- a/Documentation/technical/pack-protocol.txt\n+++ b/Documentation/technical/pack-protocol.txt\n@@ -574,6 +574,7 @@ respectively. The output is in pkt-line format.\n advertised-refs = service\n \t\t flush-pkt\n \t\t (no-refs / list-of-refs)\n+\t\t *shallow\n \t\t flush-pkt\n \n service = PKT-LINE(\"# service=\" service-name)\n@@ -590,6 +591,8 @@ respectively. The output is in pkt-line format.\n other-tip = obj-id SP refname LF\n other-peeled = obj-id SP refname \"^{}\" LF\n \n+ shallow = PKT-LINE(\"shallow\" SP obj-id)\n+\n capability-list = capability *(SP capability)\n capability = 1*(LC_ALPHA / DIGIT / \"-\" / \"_\")\n LC_ALPHA = %x61-7A\ndiff --git a/builtin/fetch-pack.c b/builtin/fetch-pack.c\nindex f6a6d76..b89d753 100644\n--- a/builtin/fetch-pack.c\n+++ b/builtin/fetch-pack.c\n@@ -146,8 +146,7 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)\n \t}\n \n \tmemset(&shallow, 0, sizeof(shallow));\n-\tget_remote_heads(fd[0], NULL, 0, &ref, 0, NULL,\n-\t\t\t args.stateless_rpc ? NULL : &shallow);\n+\tget_remote_heads(fd[0], NULL, 0, &ref, 0, NULL, &shallow);\n \n \tref = fetch_pack(&args, fd, conn, ref, dest,\n \t\t\t sought, nr_sought, &shallow, pack_lockfile_ptr);\ndiff --git a/remote-curl.c b/remote-curl.c\nindex c329bd3..de2cc8a 100644\n--- a/remote-curl.c\n+++ b/remote-curl.c\n@@ -78,6 +78,7 @@ struct discovery {\n \tchar *buf;\n \tsize_t len;\n \tstruct ref *refs;\n+\tstruct extra_have_objects shallow;\n \tunsigned proto_git : 1;\n };\n static struct discovery *last_discovery;\n@@ -86,7 +87,7 @@ static struct ref *parse_git_refs(struct discovery *heads, int for_push)\n {\n \tstruct ref *list = NULL;\n \tget_remote_heads(-1, heads->buf, heads->len, &list,\n-\t\t\t for_push ? REF_NORMAL : 0, NULL, NULL);\n+\t\t\t for_push ? REF_NORMAL : 0, NULL, &heads->shallow);\n \treturn list;\n }\n \n@@ -146,6 +147,7 @@ static void free_discovery(struct discovery *d)\n \tif (d) {\n \t\tif (d == last_discovery)\n \t\t\tlast_discovery = NULL;\n+\t\tfree(d->shallow.array);\n \t\tfree(d->buf_alloc);\n \t\tfree_refs(d->refs);\n \t\tfree(d);\ndiff --git a/t/t5536-fetch-shallow.sh b/t/t5536-fetch-shallow.sh\nindex 15a8208..6ea6347 100755\n--- a/t/t5536-fetch-shallow.sh\n+++ b/t/t5536-fetch-shallow.sh\n@@ -102,4 +102,31 @@ EOF\n \n '\n \n+if test -n \"$NO_CURL\" -o -z \"$GIT_TEST_HTTPD\"; then\n+\tsay 'skipping remaining tests, git built without http support'\n+\ttest_done\n+fi\n+\n+LIB_HTTPD_PORT=${LIB_HTTPD_PORT-'5536'}\n+. \"$TEST_DIRECTORY\"/lib-httpd.sh\n+start_httpd\n+\n+test_expect_success 'clone http repository' '\n+\tgit clone --bare --no-local shallow \"$HTTPD_DOCUMENT_ROOT_PATH/repo.git\" &&\n+\tgit clone --quiet $HTTPD_URL/smart/repo.git clone &&\n+\t(\n+\tcd clone &&\n+\tgit fsck &&\n+\tgit log --format=%s origin/master >actual &&\n+\tcat <expect &&\n+6\n+5\n+4\n+3\n+EOF\n+\ttest_cmp expect actual\n+\t)\n+'\n+\n+stop_httpd\n test_done\ndiff --git a/upload-pack.c b/upload-pack.c\nindex c3e68ae..263ae08 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -770,8 +770,6 @@ int main(int argc, char **argv)\n \n \tif (!enter_repo(dir, strict))\n \t\tdie(\"'%s' does not appear to be a git repository\", dir);\n-\tif (is_repository_shallow() && stateless_rpc)\n-\t\tdie(\"attempt to push into a shallow repository\");\n \n \tgit_config(upload_pack_config, NULL);\n \tupload_pack();\n-- \n1.8.2.83.gc99314b\n"},{"id":"223841","messageId":"1374314290-5976-13-git-send-email-pclouds@gmail.com","threadId":"34452","inReplyTo":"1374314290-5976-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v2 12/16] receive-pack: support pushing to a shallow clone via http","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-07-20T09:58:06Z","receivedAt":"2013-07-20T09:58:06Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/receive-pack.c | 3 ---\n t/t5537-push-shallow.sh | 35 +++++++++++++++++++++++++++++++++++\n 2 files changed, 35 insertions(+), 3 deletions(-)\n\ndiff --git a/builtin/receive-pack.c b/builtin/receive-pack.c\nindex cbb2025..54bf6b2 100644\n--- a/builtin/receive-pack.c\n+++ b/builtin/receive-pack.c\n@@ -1038,9 +1038,6 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix)\n \tif (!enter_repo(dir, 0))\n \t\tdie(\"'%s' does not appear to be a git repository\", dir);\n \n-\tif (is_repository_shallow() && stateless_rpc)\n-\t\tdie(\"attempt to push into a shallow repository\");\n-\n \tgit_config(receive_pack_config, NULL);\n \n \tif (0 <= transfer_unpack_limit)\ndiff --git a/t/t5537-push-shallow.sh b/t/t5537-push-shallow.sh\nindex beb85ce..02d9198 100755\n--- a/t/t5537-push-shallow.sh\n+++ b/t/t5537-push-shallow.sh\n@@ -16,6 +16,7 @@ test_expect_success 'setup' '\n \tcommit 2 &&\n \tcommit 3 &&\n \tcommit 4 &&\n+\tgit clone . full &&\n \t(\n \tgit init full-abc &&\n \tcd full-abc &&\n@@ -107,4 +108,38 @@ EOF\n \t)\n '\n \n+if test -n \"$NO_CURL\" -o -z \"$GIT_TEST_HTTPD\"; then\n+\tsay 'skipping remaining tests, git built without http support'\n+\ttest_done\n+fi\n+\n+LIB_HTTPD_PORT=${LIB_HTTPD_PORT-'5537'}\n+. \"$TEST_DIRECTORY\"/lib-httpd.sh\n+start_httpd\n+\n+test_expect_success 'push to shallow repo via http' '\n+\tgit clone --bare --no-local shallow \"$HTTPD_DOCUMENT_ROOT_PATH/repo.git\" &&\n+\t(\n+\tcd \"$HTTPD_DOCUMENT_ROOT_PATH/repo.git\" &&\n+\tgit config http.receivepack true\n+\t) &&\n+\t(\n+\tcd full &&\n+\tcommit 9 &&\n+\tgit push $HTTPD_URL/smart/repo.git +master:refs/remotes/top/master\n+\t) &&\n+\t(\n+\tcd \"$HTTPD_DOCUMENT_ROOT_PATH/repo.git\" &&\n+\tgit fsck &&\n+\tgit log --format=%s top/master >actual &&\n+\tcat <expect &&\n+9\n+4\n+3\n+EOF\n+\ttest_cmp expect actual\n+\t)\n+'\n+\n+stop_httpd\n test_done\n-- \n1.8.2.83.gc99314b\n"},{"id":"223842","messageId":"1374314290-5976-14-git-send-email-pclouds@gmail.com","threadId":"34452","inReplyTo":"1374314290-5976-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v2 13/16] send-pack: support pushing from a shallow clone via http","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-07-20T09:58:07Z","receivedAt":"2013-07-20T09:58:07Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/send-pack.c | 3 ---\n send-pack.c | 19 +++++++++++++++++--\n t/t5537-push-shallow.sh | 25 +++++++++++++++++++++++++\n 3 files changed, 42 insertions(+), 5 deletions(-)\n\ndiff --git a/builtin/send-pack.c b/builtin/send-pack.c\nindex 14a73ba..b177120 100644\n--- a/builtin/send-pack.c\n+++ b/builtin/send-pack.c\n@@ -185,9 +185,6 @@ int cmd_send_pack(int argc, const char **argv, const char *prefix)\n \t (send_all && args.send_mirror))\n \t\tusage(send_pack_usage);\n \n-\tif (is_repository_shallow() && args.stateless_rpc)\n-\t\tdie(\"attempt to push from a shallow repository\");\n-\n \tif (remote_name) {\n \t\tremote = remote_get(remote_name);\n \t\tif (!remote_has_url(remote, dest)) {\ndiff --git a/send-pack.c b/send-pack.c\nindex 1b828b0..520c838 100644\n--- a/send-pack.c\n+++ b/send-pack.c\n@@ -198,6 +198,21 @@ static int sideband_demux(int in, int out, void *data)\n \treturn ret;\n }\n \n+static int advertise_shallow_grafts_cb(const struct commit_graft *graft, void *cb)\n+{\n+\tstruct strbuf *sb = cb;\n+\tif (graft->nr_parent == -1)\n+\t\tpacket_buf_write(sb, \"shallow %s\\n\", sha1_to_hex(graft->sha1));\n+\treturn 0;\n+}\n+\n+void advertise_shallow_grafts_buf(struct strbuf *sb)\n+{\n+\tif (!is_repository_shallow())\n+\t\treturn;\n+\tfor_each_commit_graft(advertise_shallow_grafts_cb, sb);\n+}\n+\n int send_pack(struct send_pack_args *args,\n \t int fd[], struct child_process *conn,\n \t struct ref *remote_refs,\n@@ -239,7 +254,7 @@ int send_pack(struct send_pack_args *args,\n \t}\n \n \tif (!args->dry_run)\n-\t\tadvertise_shallow_grafts(out);\n+\t\tadvertise_shallow_grafts_buf(&req_buf);\n \n \t/*\n \t * Finally, tell the other end!\n@@ -299,7 +314,7 @@ int send_pack(struct send_pack_args *args,\n \t}\n \n \tif (args->stateless_rpc) {\n-\t\tif (!args->dry_run && cmds_sent) {\n+\t\tif (!args->dry_run && (cmds_sent || is_repository_shallow())) {\n \t\t\tpacket_buf_flush(&req_buf);\n \t\t\tsend_sideband(out, -1, req_buf.buf, req_buf.len, LARGE_PACKET_MAX);\n \t\t}\ndiff --git a/t/t5537-push-shallow.sh b/t/t5537-push-shallow.sh\nindex 02d9198..8bea496 100755\n--- a/t/t5537-push-shallow.sh\n+++ b/t/t5537-push-shallow.sh\n@@ -141,5 +141,30 @@ EOF\n \t)\n '\n \n+test_expect_success 'push from shallow repo via http' '\n+\tmv \"$HTTPD_DOCUMENT_ROOT_PATH/repo.git\" shallow-upstream.git &&\n+\tgit clone --bare --no-local full \"$HTTPD_DOCUMENT_ROOT_PATH/repo.git\" &&\n+\t(\n+\tcd \"$HTTPD_DOCUMENT_ROOT_PATH/repo.git\" &&\n+\tgit config http.receivepack true\n+\t) &&\n+\tcommit 10 &&\n+\tgit push $HTTPD_URL/smart/repo.git +master:refs/remotes/top/master &&\n+\t(\n+\tcd \"$HTTPD_DOCUMENT_ROOT_PATH/repo.git\" &&\n+\tgit fsck &&\n+\tgit log --format=%s top/master >actual &&\n+\tcat <expect &&\n+10\n+1\n+4\n+3\n+2\n+1\n+EOF\n+\ttest_cmp expect actual\n+\t)\n+'\n+\n stop_httpd\n test_done\n-- \n1.8.2.83.gc99314b\n"},{"id":"223843","messageId":"1374314290-5976-15-git-send-email-pclouds@gmail.com","threadId":"34452","inReplyTo":"1374314290-5976-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v2 14/16] git-clone.txt: remove shallow clone limitations","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-07-20T09:58:08Z","receivedAt":"2013-07-20T09:58:08Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"Now that git supports push/pull from/to a shallow clone, these\nlimitations are not true anymore.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n Documentation/git-clone.txt | 7 +------\n 1 file changed, 1 insertion(+), 6 deletions(-)\n\ndiff --git a/Documentation/git-clone.txt b/Documentation/git-clone.txt\nindex 450f158..4987857 100644\n--- a/Documentation/git-clone.txt\n+++ b/Documentation/git-clone.txt\n@@ -181,12 +181,7 @@ objects from the source repository into a pack in the cloned repository.\n \n --depth ::\n \tCreate a 'shallow' clone with a history truncated to the\n-\tspecified number of revisions. A shallow repository has a\n-\tnumber of limitations (you cannot clone or fetch from\n-\tit, nor push from nor into it), but is adequate if you\n-\tare only interested in the recent history of a large project\n-\twith a long history, and would want to send in fixes\n-\tas patches.\n+\tspecified number of revisions.\n \n --[no-]single-branch::\n \tClone only the history leading to the tip of a single branch,\n-- \n1.8.2.83.gc99314b\n"},{"id":"223844","messageId":"1374314290-5976-16-git-send-email-pclouds@gmail.com","threadId":"34452","inReplyTo":"1374314290-5976-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v2 15/16] config: add core.noshallow to prevent turning a repo into a shallow one","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-07-20T09:58:09Z","receivedAt":"2013-07-20T09:58:09Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n Documentation/config.txt | 5 +++++\n builtin/receive-pack.c | 9 ++++++++-\n cache.h | 1 +\n config.c | 5 +++++\n environment.c | 1 +\n fetch-pack.c | 9 ++++++++-\n t/t5536-fetch-shallow.sh | 9 +++++++++\n t/t5537-push-shallow.sh | 6 ++++++\n 8 files changed, 43 insertions(+), 2 deletions(-)\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex 81856dd..e811180 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -655,6 +655,11 @@ core.abbrev::\n \tfor abbreviated object names to stay unique for sufficiently long\n \ttime.\n \n+core.noshallow::\n+\tIf true, reject any pushes or fetches that may turn the\n+\trepository into a shallow one. This setting is ignored if the\n+\trepository is already shallow.\n+\n add.ignore-errors::\n add.ignoreErrors::\n \tTells 'git add' to continue adding files when some files cannot be\ndiff --git a/builtin/receive-pack.c b/builtin/receive-pack.c\nindex 54bf6b2..95ea481 100644\n--- a/builtin/receive-pack.c\n+++ b/builtin/receive-pack.c\n@@ -936,7 +936,14 @@ static const char *unpack(int err_fd)\n \t\t\t\t\t&shallow,\n \t\t\t\t\tWRITE_SHALLOW_NO_CUT |\n \t\t\t\t\tWRITE_SHALLOW_REWRITE);\n-\t\tcommit_lock_file(&shallow_lock);\n+\t\tif (*alternate_shallow_file == '\\0') {\n+\t\t\tunlink_or_warn(git_path(\"shallow\"));\n+\t\t\trollback_lock_file(&shallow_lock);\n+\t\t} else {\n+\t\t\tif (!is_repository_shallow() && cannot_be_shallow)\n+\t\t\t\tdie(\"not allowed to turn this repository into a shallow one\");\n+\t\t\tcommit_lock_file(&shallow_lock);\n+\t\t}\n \t}\n \treturn NULL;\n }\ndiff --git a/cache.h b/cache.h\nindex 7f17228..3a52b08 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -592,6 +592,7 @@ extern int fsync_object_files;\n extern int core_preload_index;\n extern int core_apply_sparse_checkout;\n extern int precomposed_unicode;\n+extern int cannot_be_shallow;\n \n /*\n * The character that begins a commented line in user-editable file\ndiff --git a/config.c b/config.c\nindex d04e815..31f5a57 100644\n--- a/config.c\n+++ b/config.c\n@@ -784,6 +784,11 @@ static int git_default_core_config(const char *var, const char *value)\n \t\treturn 0;\n \t}\n \n+\tif (!strcmp(var, \"core.noshallow\")) {\n+\t\tcannot_be_shallow = git_config_bool(var, value);\n+\t\treturn 0;\n+\t}\n+\n \t/* Add other config variables here and to Documentation/config.txt. */\n \treturn 0;\n }\ndiff --git a/environment.c b/environment.c\nindex 0cb67b2..14c8005 100644\n--- a/environment.c\n+++ b/environment.c\n@@ -61,6 +61,7 @@ int merge_log_config = -1;\n int precomposed_unicode = -1; /* see probe_utf8_pathname_composition() */\n struct startup_info *startup_info;\n unsigned long pack_size_limit_cfg;\n+int cannot_be_shallow;\n \n /*\n * The character that begins a commented line in user-editable file\ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex f337526..40e7aa2 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -960,7 +960,14 @@ struct ref *fetch_pack(struct fetch_pack_args *args,\n \t\t\t\t\t\t\tWRITE_SHALLOW_NO_CUT |\n \t\t\t\t\t\t\tWRITE_SHALLOW_REWRITE);\n \t\t\t}\n-\t\t\tcommit_lock_file(&shallow_lock);\n+\t\t\tif (*alternate_shallow_file == '\\0') {\n+\t\t\t\tunlink_or_warn(git_path(\"shallow\"));\n+\t\t\t\trollback_lock_file(&shallow_lock);\n+\t\t\t} else {\n+\t\t\t\tif (!is_repository_shallow() && cannot_be_shallow)\n+\t\t\t\t\tdie(\"not allowed to turn this repository into a shallow one\");\n+\t\t\t\tcommit_lock_file(&shallow_lock);\n+\t\t\t}\n \t\t}\n \t}\n \ndiff --git a/t/t5536-fetch-shallow.sh b/t/t5536-fetch-shallow.sh\nindex 6ea6347..b7f89b1 100755\n--- a/t/t5536-fetch-shallow.sh\n+++ b/t/t5536-fetch-shallow.sh\n@@ -102,6 +102,15 @@ EOF\n \n '\n \n+test_expect_success 'core.noshallow' '\n+\tgit init clean &&\n+\t(\n+\tcd clean &&\n+\tgit config core.noshallow true &&\n+\ttest_must_fail git fetch ../shallow/.git\n+\t)\n+'\n+\n if test -n \"$NO_CURL\" -o -z \"$GIT_TEST_HTTPD\"; then\n \tsay 'skipping remaining tests, git built without http support'\n \ttest_done\ndiff --git a/t/t5537-push-shallow.sh b/t/t5537-push-shallow.sh\nindex 8bea496..0edd51f 100755\n--- a/t/t5537-push-shallow.sh\n+++ b/t/t5537-push-shallow.sh\n@@ -108,6 +108,12 @@ EOF\n \t)\n '\n \n+test_expect_success 'core.noshallow' '\n+\tgit init clean &&\n+\tgit --git-dir=clean/.git config core.noshallow true &&\n+\ttest_must_fail git --git-dir=shallow/.git push clean master:refs/remotes/shallow/master\n+'\n+\n if test -n \"$NO_CURL\" -o -z \"$GIT_TEST_HTTPD\"; then\n \tsay 'skipping remaining tests, git built without http support'\n \ttest_done\n-- \n1.8.2.83.gc99314b\n"},{"id":"223845","messageId":"1374314290-5976-17-git-send-email-pclouds@gmail.com","threadId":"34452","inReplyTo":"1374314290-5976-1-git-send-email-pclouds@gmail.com","subject":"[PATCH v2 16/16] clone: use git protocol for cloning shallow repo locally","fromName":"Nguyễn Thái Ngọc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2013-07-20T09:58:10Z","receivedAt":"2013-07-20T09:58:10Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"clone_local() does not handle $SRC/shallow. It could be made so, but\nit's simpler to use fetch-pack/upload-pack instead.\n\nThis used by be caught by the check in upload-pack, which is triggered\nby transport_get_remote_refs(), even in local clone case. The check is\nnow gone and check_everything_connected() should catch the result\nincomplete repo. But check_everything_connected() will soon be skipped\nin local clone case, opening a door to corrupt repo. This patch should\nclose that door.\n\nSigned-off-by: Nguyễn Thái Ngọc Duy \n---\n builtin/clone.c | 14 ++++++++++++--\n t/t5601-clone.sh | 7 +++++++\n 2 files changed, 19 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex 17f57cd..09e383e 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -789,8 +789,18 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \telse\n \t\trepo = repo_name;\n \tis_local = option_local != 0 && path && !is_bundle;\n-\tif (is_local && option_depth)\n-\t\twarning(_(\"--depth is ignored in local clones; use file:// instead.\"));\n+\tif (is_local) {\n+\t\tstruct strbuf sb = STRBUF_INIT;\n+\t\tif (option_depth)\n+\t\t\twarning(_(\"--depth is ignored in local clones; use file:// instead.\"));\n+\t\tstrbuf_addf(&sb, \"%s/shallow\", path);\n+\t\tif (!access(sb.buf, F_OK)) {\n+\t\t\tif (option_local > 0)\n+\t\t\t\twarning(_(\"source repository is shallow, ignoring --local\"));\n+\t\t\tis_local = 0;\n+\t\t}\n+\t\tstrbuf_release(&sb);\n+\t}\n \tif (option_local > 0 && !is_local)\n \t\twarning(_(\"--local is ignored\"));\n \ndiff --git a/t/t5601-clone.sh b/t/t5601-clone.sh\nindex 0629149..35a2a3d 100755\n--- a/t/t5601-clone.sh\n+++ b/t/t5601-clone.sh\n@@ -285,4 +285,11 @@ test_expect_success NOT_MINGW,NOT_CYGWIN 'clone local path foo:bar' '\n \tgit clone \"./foo:bar\" foobar\n '\n \n+test_expect_success 'shallow clone locally' '\n+\tgit clone --depth=1 --no-local src ssrrcc &&\n+\tgit clone ssrrcc ddsstt &&\n+\ttest_cmp ssrrcc/.git/shallow ddsstt/.git/shallow &&\n+\t( cd ddsstt && git fsck )\n+'\n+\n test_done\n-- \n1.8.2.83.gc99314b\n"},{"id":"223930","messageId":"DADC072FCD7749C8B51D9DE594D67AB7@PhilipOakley","threadId":"34452","inReplyTo":"1374314290-5976-6-git-send-email-pclouds@gmail.com","subject":"Re: [PATCH v2 05/16] fetch-pack: support fetching from a shallow repository","fromName":"Philip Oakley","fromEmail":"philipoakley@iee.org","sentAt":"2013-07-22T19:10:41Z","receivedAt":"2013-07-22T19:10:41Z","isPatch":true,"sender":{"key":"philipoakley@iee.email","avatar":"https://avatars.githubusercontent.com/u/914343?v=4"},"body":"From: \"Nguyễn Thái Ngọc Duy\" \n> upload-pack already advertises all shallow grafts if server repository\n> is shallow. This information can be used to add more grafts to the\n> client if the server sends commit chains down to its graft points.\n>\n> If the server is shallow, before we receive the pack, we setup a\n> temporary shallow file that contains both local graft points and the\n> server's. This stops index-pack from going beyond server's graft\n> points.\n>\n> Only server graft points that do not have corresponding SHA-1s in\n> local repo are added to the temp shallow file because we don't want to\n> accidentally cut the client history because the server's is\n> shorter. The client cutting can only happen when --depth is requested.\n>\n> After index-pack finishes successfully, we write the temporary shallow\n> down with one exception: unused graft points provided by the server\n> are removed. We don't want those lying around and suddenly become\n> active.\n>\n> Note that in the \"shallow -> shallow\" case, the server might not have\n> enough information to find common roots to create an optimum pack. It\n> might send complete commit chains down to the graft points as a\n> result. I don't think we can improve this, unless upload-pack somehow\n> has access to a full repository.\n>\n> \"shallow -> shallow\" case only makes sense when the upstream provides\n> a stable shallow repo (e.g. make a cut every year or so and ask devs\n> to all move to the new base). If the cloned repos are all based on a\n> stable (shallow) upstream, the above problem is unlikely to happen.\n>\n> A side effect of this change is we can now clone from a shallow\n> repository.\n\n> And a full repository may automatically become shallow if\n> you fetch from a shallow repository.\n\nThis \"sounds\" risky: Perhaps add a note about '[PATCH v2 15/16] config: \nadd core.noshallow to prevent turning a repo into a shallow one' which \nwill become available later in the series. The later core.noshallow may \nbe a hostage to fortune.\n\n>\n> Signed-off-by: Nguyễn Thái Ngọc Duy \n> ---\n> builtin/fetch-pack.c | 7 ++-\n> commit.h | 27 +++++++++-\n> fetch-pack.c | 30 +++++++++--\n> fetch-pack.h | 1 +\n> shallow.c | 90 \n> +++++++++++++++++++++++++++++---\n> t/t5536-fetch-shallow.sh (new +x) | 105 \n> ++++++++++++++++++++++++++++++++++++++\n> transport.c | 10 ++--\n> 7 files changed, 252 insertions(+), 18 deletions(-)\n> create mode 100755 t/t5536-fetch-shallow.sh\n>\n> diff --git a/builtin/fetch-pack.c b/builtin/fetch-pack.c\n> index 080e599..f6a6d76 100644\n> --- a/builtin/fetch-pack.c\n> +++ b/builtin/fetch-pack.c\n> @@ -37,6 +37,7 @@ int cmd_fetch_pack(int argc, const char **argv, \n> const char *prefix)\n> char **pack_lockfile_ptr = NULL;\n> struct child_process *conn;\n> struct fetch_pack_args args;\n> + struct extra_have_objects shallow;\n>\n> packet_trace_identity(\"fetch-pack\");\n>\n> @@ -144,10 +145,12 @@ int cmd_fetch_pack(int argc, const char **argv, \n> const char *prefix)\n> args.verbose ? CONNECT_VERBOSE : 0);\n> }\n>\n> - get_remote_heads(fd[0], NULL, 0, &ref, 0, NULL, NULL);\n> + memset(&shallow, 0, sizeof(shallow));\n> + get_remote_heads(fd[0], NULL, 0, &ref, 0, NULL,\n> + args.stateless_rpc ? NULL : &shallow);\n>\n> ref = fetch_pack(&args, fd, conn, ref, dest,\n> - sought, nr_sought, pack_lockfile_ptr);\n> + sought, nr_sought, &shallow, pack_lockfile_ptr);\n> if (pack_lockfile) {\n> printf(\"lock %s\\n\", pack_lockfile);\n> fflush(stdout);\n> diff --git a/commit.h b/commit.h\n> index 678fa20..97fd54e 100644\n> --- a/commit.h\n> +++ b/commit.h\n> @@ -188,9 +188,32 @@ extern struct commit_list \n> *get_shallow_commits(struct object_array *heads,\n> extern void check_shallow_file_for_update(void);\n> extern void set_alternate_shallow_file(const char *path);\n> extern void advertise_shallow_grafts(int);\n> -extern int write_shallow_commits(struct strbuf *out, int \n> use_pack_protocol);\n> +\n> +/*\n> + * Ignore graft points that cut out existing \"good\" parts of our\n> + * history. Graft points pointing to incomplete commit islands are\n> + * _not_ ignore.\n> + */\n> +#define WRITE_SHALLOW_NO_CUT 1\n> +/*\n> + * Remove graft points that are not attached to any SHA-1 in current\n> + * history.\n> + */\n> +#define WRITE_SHALLOW_NO_UNUSED 2\n> +/*\n> + * Not really write_shallow_commits() flag, this lets\n> + * setup_alternate_shallow know this is the \"final\" setup before\n> + * committing to .git/shallow\n> + */\n> +#define WRITE_SHALLOW_REWRITE 4\n> +\n> +extern int write_shallow_commits(struct strbuf *out, int \n> use_pack_protocol,\n> + struct extra_have_objects *extra,\n> + unsigned int flags);\n> extern void setup_alternate_shallow(struct lock_file *shallow_lock,\n> - const char **alternate_shallow_file);\n> + const char **alternate_shallow_file,\n> + struct extra_have_objects *extra,\n> + unsigned int flags);\n>\n> int is_descendant_of(struct commit *, struct commit_list *);\n> int in_merge_bases(struct commit *, struct commit *);\n> diff --git a/fetch-pack.c b/fetch-pack.c\n> index dc71a2b..f337526 100644\n> --- a/fetch-pack.c\n> +++ b/fetch-pack.c\n> @@ -311,7 +311,7 @@ static int find_common(struct fetch_pack_args \n> *args,\n> }\n>\n> if (is_repository_shallow())\n> - write_shallow_commits(&req_buf, 1);\n> + write_shallow_commits(&req_buf, 1, NULL, WRITE_SHALLOW_NO_CUT);\n> if (args->depth > 0)\n> packet_buf_write(&req_buf, \"deepen %d\", args->depth);\n> packet_buf_flush(&req_buf);\n> @@ -769,6 +769,7 @@ static struct ref *do_fetch_pack(struct \n> fetch_pack_args *args,\n> int fd[2],\n> const struct ref *orig_ref,\n> struct ref **sought, int nr_sought,\n> + struct extra_have_objects *shallow,\n> char **pack_lockfile)\n> {\n> struct ref *ref = copy_ref_list(orig_ref);\n> @@ -844,8 +845,10 @@ static struct ref *do_fetch_pack(struct \n> fetch_pack_args *args,\n>\n> if (args->stateless_rpc)\n> packet_flush(fd[1]);\n> - if (args->depth > 0)\n> - setup_alternate_shallow(&shallow_lock, &alternate_shallow_file);\n> + if (args->depth > 0 || shallow->nr)\n> + setup_alternate_shallow(&shallow_lock, &alternate_shallow_file,\n> + shallow,\n> + WRITE_SHALLOW_NO_CUT);\n> if (get_pack(args, fd, pack_lockfile))\n> die(\"git fetch-pack: fetch failed.\");\n>\n> @@ -922,6 +925,7 @@ struct ref *fetch_pack(struct fetch_pack_args \n> *args,\n> const struct ref *ref,\n> const char *dest,\n> struct ref **sought, int nr_sought,\n> + struct extra_have_objects *shallow,\n> char **pack_lockfile)\n> {\n> struct ref *ref_cpy;\n> @@ -934,14 +938,30 @@ struct ref *fetch_pack(struct fetch_pack_args \n> *args,\n> packet_flush(fd[1]);\n> die(\"no matching remote head\");\n> }\n> - ref_cpy = do_fetch_pack(args, fd, ref, sought, nr_sought, \n> pack_lockfile);\n> + ref_cpy = do_fetch_pack(args, fd, ref, sought, nr_sought,\n> + shallow, pack_lockfile);\n>\n> if (alternate_shallow_file) {\n> if (*alternate_shallow_file == '\\0') { /* --unshallow */\n> unlink_or_warn(git_path(\"shallow\"));\n> rollback_lock_file(&shallow_lock);\n> - } else\n> + } else {\n> + /*\n> + * The server is a shallow clone and it sends\n> + * us all of its shallow grafts. Some may be\n> + * needed if the server sends objects down to\n> + * the bottom. Remove all unused grafts.\n> + */\n> + if (shallow->nr) {\n> + reprepare_packed_git();\n> + setup_alternate_shallow(&shallow_lock,\n> + &alternate_shallow_file,\n> + shallow,\n> + WRITE_SHALLOW_NO_CUT |\n> + WRITE_SHALLOW_REWRITE);\n> + }\n> commit_lock_file(&shallow_lock);\n> + }\n> }\n>\n> reprepare_packed_git();\n> diff --git a/fetch-pack.h b/fetch-pack.h\n> index 40f08ba..c55bf1c 100644\n> --- a/fetch-pack.h\n> +++ b/fetch-pack.h\n> @@ -32,6 +32,7 @@ struct ref *fetch_pack(struct fetch_pack_args *args,\n> const char *dest,\n> struct ref **sought,\n> int nr_sought,\n> + struct extra_have_objects *shallow,\n> char **pack_lockfile);\n>\n> #endif\n> diff --git a/shallow.c b/shallow.c\n> index ee9edd4..337f676 100644\n> --- a/shallow.c\n> +++ b/shallow.c\n> @@ -2,6 +2,7 @@\n> #include \"commit.h\"\n> #include \"tag.h\"\n> #include \"pkt-line.h\"\n> +#include \"connected.h\"\n>\n> static int is_shallow = -1;\n> static struct stat shallow_stat;\n> @@ -183,26 +184,103 @@ static int write_one_shallow(const struct \n> commit_graft *graft, void *cb_data)\n> return 0;\n> }\n>\n> -int write_shallow_commits(struct strbuf *out, int use_pack_protocol)\n> +static int sha1_iterate(void *cb, unsigned char *sha1)\n> +{\n> + if (is_null_sha1(cb))\n> + return 1;\n> + hashcpy(sha1, cb);\n> + hashclr(cb);\n> + return 0;\n> +}\n> +\n> +struct connectivity_cache_item {\n> + unsigned char sha1[20];\n> + int good;\n> +};\n> +\n> +static int good_connectivity_from(const unsigned char *sha1)\n> +{\n> + static struct connectivity_cache_item *cache;\n> + static int nr, alloc;\n> + int i;\n> + unsigned char sha1_cb[20];\n> +\n> + for (i = 0; i < nr; i++)\n> + if (!hashcmp(cache[i].sha1, sha1))\n> + return cache[i].good;\n> +\n> + ALLOC_GROW(cache, nr + 1, alloc);\n> + hashcpy(sha1_cb, sha1);\n> + hashcpy(cache[nr].sha1, sha1);\n> + cache[nr].good = !check_everything_connected(sha1_iterate, 1, \n> sha1_cb);\n> + nr++;\n> + return cache[nr - 1].good;\n> +}\n> +\n> +int write_shallow_commits(struct strbuf *out, int use_pack_protocol,\n> + struct extra_have_objects *extra,\n> + unsigned int flags)\n> {\n> struct write_shallow_data data;\n> + int i;\n> data.out = out;\n> data.use_pack_protocol = use_pack_protocol;\n> data.count = 0;\n> for_each_commit_graft(write_one_shallow, &data);\n> + if (!extra)\n> + return data.count;\n> +\n> + for (i = 0; i < extra->nr; i++) {\n> + if ((flags & WRITE_SHALLOW_NO_CUT) &&\n> + good_connectivity_from(extra->array[i]))\n> + /*\n> + * The server may have even shorter history\n> + * than the client (e.g. the client has\n> + * A-B-C-D but the server only has C-D). We do\n> + * NOT want to cut client's history down to\n> + * C-D simply because the server is set up\n> + * so. If we don't have \"C\" and the server\n> + * sends C-D to us, then we set up a graft at\n> + * \"C\".\n> + *\n> + * If the client has (broken)-B-C-D, then \"C\"\n> + * _is_ accepted. The -B-C-D is an incomplete\n> + * island that normally has no connectivity to\n> + * the rest of the repo.\n> + */\n> + continue;\n> + if ((flags & WRITE_SHALLOW_NO_UNUSED) &&\n> + !has_sha1_file(extra->array[i]))\n> + continue;\n> + strbuf_addstr(out, sha1_to_hex(extra->array[i]));\n> + strbuf_addch(out, '\\n');\n> + data.count++;\n> + }\n> return data.count;\n> }\n>\n> void setup_alternate_shallow(struct lock_file *shallow_lock,\n> - const char **alternate_shallow_file)\n> + const char **alternate_shallow_file,\n> + struct extra_have_objects *extra,\n> + unsigned int flags)\n> {\n> struct strbuf sb = STRBUF_INIT;\n> + int rewrite = flags & WRITE_SHALLOW_REWRITE;\n> int fd;\n>\n> - check_shallow_file_for_update();\n> - fd = hold_lock_file_for_update(shallow_lock, git_path(\"shallow\"),\n> - LOCK_DIE_ON_ERROR);\n> - if (write_shallow_commits(&sb, 0)) {\n> + if (!rewrite)\n> + fd = hold_lock_file_for_update(shallow_lock,\n> + git_path(\"shallow\"),\n> + LOCK_DIE_ON_ERROR);\n> + else {\n> + flags |= WRITE_SHALLOW_NO_UNUSED;\n> + fd = shallow_lock->fd;\n> + }\n> + if (write_shallow_commits(&sb, 0, extra, flags)) {\n> + if (rewrite && (lseek(fd, 0, SEEK_SET) == -1 ||\n> + ftruncate(fd, 0) == -1))\n> + die_errno(\"unable to truncate the new shallow file %s\",\n> + shallow_lock->filename);\n> if (write_in_full(fd, sb.buf, sb.len) != sb.len)\n> die_errno(\"failed to write to %s\",\n> shallow_lock->filename);\n> diff --git a/t/t5536-fetch-shallow.sh b/t/t5536-fetch-shallow.sh\n> new file mode 100755\n> index 0000000..15a8208\n> --- /dev/null\n> +++ b/t/t5536-fetch-shallow.sh\n> @@ -0,0 +1,105 @@\n> +#!/bin/sh\n> +\n> +test_description='fetch/clone from a shallow clone'\n> +\n> +. ./test-lib.sh\n> +\n> +commit() {\n> + echo \"$1\" >tracked &&\n> + git add tracked &&\n> + git commit -m \"$1\"\n> +}\n> +\n> +test_expect_success 'setup' '\n> + commit 1 &&\n> + commit 2 &&\n> + commit 3 &&\n> + commit 4 &&\n> + git config --global transfer.fsckObjects true\n> +'\n> +\n> +test_expect_success 'setup shallow clone' '\n> + git clone --no-local --depth=2 .git shallow &&\n> + git --git-dir=shallow/.git log --format=%s >actual &&\n> + cat <expect &&\n> +4\n> +3\n> +EOF\n> + test_cmp expect actual\n> +'\n> +\n> +test_expect_success 'clone from shallow clone' '\n> + git clone --no-local shallow shallow2 &&\n> + (\n> + cd shallow2 &&\n> + git fsck &&\n> + git log --format=%s >actual &&\n> + cat <expect &&\n> +4\n> +3\n> +EOF\n> + test_cmp expect actual\n> + )\n> +'\n> +\n> +test_expect_success 'fetch from shallow clone' '\n> + (\n> + cd shallow &&\n> + commit 5\n> + ) &&\n> + (\n> + cd shallow2 &&\n> + git fetch &&\n> + git fsck &&\n> + git log --format=%s origin/master >actual &&\n> + cat <expect &&\n> +5\n> +4\n> +3\n> +EOF\n> + test_cmp expect actual\n> + )\n> +'\n> +\n> +test_expect_success 'fetch --depth from shallow clone' '\n> + (\n> + cd shallow &&\n> + commit 6\n> + ) &&\n> + (\n> + cd shallow2 &&\n> + git fetch --depth=2 &&\n> + git fsck &&\n> + git log --format=%s origin/master >actual &&\n> + cat <expect &&\n> +6\n> +5\n> +EOF\n> + test_cmp expect actual\n> + )\n> +'\n> +\n> +test_expect_success 'fetch something upstream has but hidden by \n> clients shallow boundaries' '\n> + # the blob \"1\" is available in .git but hidden by the\n> + # shallow2/.git/shallow and it should be resent\n> + ! git --git-dir=shallow2/.git cat-file blob `echo 1|git \n> hash-object --stdin` >/dev/null &&\n> + echo 1 > 1.t &&\n> + git add 1.t &&\n> + git commit -m add-1-back &&\n> + (\n> + cd shallow2 &&\n> + git fetch ../.git +refs/heads/master:refs/remotes/top/master &&\n> + git fsck &&\n> + git log --format=%s top/master >actual &&\n> + cat <expect &&\n> +add-1-back\n> +4\n> +3\n> +EOF\n> + test_cmp expect actual\n> + ) &&\n> + git --git-dir=shallow2/.git cat-file blob `echo 1|git \n> hash-object --stdin` >/dev/null\n> +\n> +'\n> +\n> +test_done\n> diff --git a/transport.c b/transport.c\n> index 10a8cb8..f167ca5 100644\n> --- a/transport.c\n> +++ b/transport.c\n> @@ -453,6 +453,7 @@ struct git_transport_data {\n> int fd[2];\n> unsigned got_remote_heads : 1;\n> struct extra_have_objects extra_have;\n> + struct extra_have_objects shallow;\n> };\n>\n> static int set_git_option(struct git_transport_options *opts,\n> @@ -509,7 +510,8 @@ static struct ref *get_refs_via_connect(struct \n> transport *transport, int for_pus\n>\n> connect_setup(transport, for_push, 0);\n> get_remote_heads(data->fd[0], NULL, 0, &refs,\n> - for_push ? REF_NORMAL : 0, &data->extra_have, NULL);\n> + for_push ? REF_NORMAL : 0,\n> + &data->extra_have, &data->shallow);\n> data->got_remote_heads = 1;\n>\n> return refs;\n> @@ -540,13 +542,13 @@ static int fetch_refs_via_pack(struct transport \n> *transport,\n> if (!data->got_remote_heads) {\n> connect_setup(transport, 0, 0);\n> get_remote_heads(data->fd[0], NULL, 0, &refs_tmp, 0,\n> - NULL, NULL);\n> + NULL, &data->shallow);\n> data->got_remote_heads = 1;\n> }\n>\n> refs = fetch_pack(&args, data->fd, data->conn,\n> refs_tmp ? refs_tmp : transport->remote_refs,\n> - dest, to_fetch, nr_heads,\n> + dest, to_fetch, nr_heads, &data->shallow,\n> &transport->pack_lockfile);\n> close(data->fd[0]);\n> close(data->fd[1]);\n> @@ -803,6 +805,8 @@ static int git_transport_push(struct transport \n> *transport, struct ref *remote_re\n> get_remote_heads(data->fd[0], NULL, 0, &tmp_refs, REF_NORMAL, NULL, \n> NULL);\n> data->got_remote_heads = 1;\n> }\n> + if (data->shallow.nr)\n> + die(\"pushing to a shallow repository is not supported\");\n>\n> memset(&args, 0, sizeof(args));\n> args.send_mirror = !!(flags & TRANSPORT_PUSH_MIRROR);\n> -- \n> 1.8.2.83.gc99314b\n>\n> --\n> To unsubscribe from this list: send the line \"unsubscribe git\" in\n> the body of a message to majordomo@vger.kernel.org\n> More majordomo info at http://vger.kernel.org/majordomo-info.html\n>\n>\n> -----\n> No virus found in this message.\n> Checked by AVG - www.avg.com\n> Version: 2013.0.3349 / Virus Database: 3204/6504 - Release Date: \n> 07/19/13\n> \n"},{"id":"223932","messageId":"BF5D998BFA5241938531635E1D8B79DF@PhilipOakley","threadId":"34452","inReplyTo":"1374314290-5976-16-git-send-email-pclouds@gmail.com","subject":"Re: [PATCH v2 15/16] config: add core.noshallow to prevent turning a repo into a shallow one","fromName":"Philip Oakley","fromEmail":"philipoakley@iee.org","sentAt":"2013-07-22T19:23:10Z","receivedAt":"2013-07-22T19:23:10Z","isPatch":true,"sender":{"key":"philipoakley@iee.email","avatar":"https://avatars.githubusercontent.com/u/914343?v=4"},"body":"From: \"Nguyễn Thái Ngọc Duy\" \nSubject: [PATCH v2 15/16] config: add core.noshallow to prevent turning \na repo into a shallow one\n\nSurely this should be the default now that it is possible to corrupt a \ngolden repo by pushing/fetching a shallow repository to it and it then \nbecomes shallow, and all the followers become shallow as well, with \nconsequent problems (IIUC) [PATCH v2 05/16].\n\nIt would be just as easy to change the config to core.allowshallow which \nthen must be enabled by the user, and can be mentioned in the shallow \nclone option's documentation.\n\n>\n> Signed-off-by: Nguyễn Thái Ngọc Duy \n> ---\n> Documentation/config.txt | 5 +++++\n> builtin/receive-pack.c | 9 ++++++++-\n> cache.h | 1 +\n> config.c | 5 +++++\n> environment.c | 1 +\n> fetch-pack.c | 9 ++++++++-\n> t/t5536-fetch-shallow.sh | 9 +++++++++\n> t/t5537-push-shallow.sh | 6 ++++++\n> 8 files changed, 43 insertions(+), 2 deletions(-)\n>\n> diff --git a/Documentation/config.txt b/Documentation/config.txt\n> index 81856dd..e811180 100644\n> --- a/Documentation/config.txt\n> +++ b/Documentation/config.txt\n> @@ -655,6 +655,11 @@ core.abbrev::\n> for abbreviated object names to stay unique for sufficiently long\n> time.\n>\n> +core.noshallow::\n> + If true, reject any pushes or fetches that may turn the\n> + repository into a shallow one. This setting is ignored if the\n> + repository is already shallow.\n> +\n> add.ignore-errors::\n> add.ignoreErrors::\n> Tells 'git add' to continue adding files when some files cannot be\n> diff --git a/builtin/receive-pack.c b/builtin/receive-pack.c\n> index 54bf6b2..95ea481 100644\n> --- a/builtin/receive-pack.c\n> +++ b/builtin/receive-pack.c\n> @@ -936,7 +936,14 @@ static const char *unpack(int err_fd)\n> &shallow,\n> WRITE_SHALLOW_NO_CUT |\n> WRITE_SHALLOW_REWRITE);\n> - commit_lock_file(&shallow_lock);\n> + if (*alternate_shallow_file == '\\0') {\n> + unlink_or_warn(git_path(\"shallow\"));\n> + rollback_lock_file(&shallow_lock);\n> + } else {\n> + if (!is_repository_shallow() && cannot_be_shallow)\n> + die(\"not allowed to turn this repository into a shallow one\");\n> + commit_lock_file(&shallow_lock);\n> + }\n> }\n> return NULL;\n> }\n> diff --git a/cache.h b/cache.h\n> index 7f17228..3a52b08 100644\n> --- a/cache.h\n> +++ b/cache.h\n> @@ -592,6 +592,7 @@ extern int fsync_object_files;\n> extern int core_preload_index;\n> extern int core_apply_sparse_checkout;\n> extern int precomposed_unicode;\n> +extern int cannot_be_shallow;\n>\n> /*\n> * The character that begins a commented line in user-editable file\n> diff --git a/config.c b/config.c\n> index d04e815..31f5a57 100644\n> --- a/config.c\n> +++ b/config.c\n> @@ -784,6 +784,11 @@ static int git_default_core_config(const char \n> *var, const char *value)\n> return 0;\n> }\n>\n> + if (!strcmp(var, \"core.noshallow\")) {\n> + cannot_be_shallow = git_config_bool(var, value);\n> + return 0;\n> + }\n> +\n> /* Add other config variables here and to Documentation/config.txt. \n> */\n> return 0;\n> }\n> diff --git a/environment.c b/environment.c\n> index 0cb67b2..14c8005 100644\n> --- a/environment.c\n> +++ b/environment.c\n> @@ -61,6 +61,7 @@ int merge_log_config = -1;\n> int precomposed_unicode = -1; /* see probe_utf8_pathname_composition() \n> */\n> struct startup_info *startup_info;\n> unsigned long pack_size_limit_cfg;\n> +int cannot_be_shallow;\n>\n> /*\n> * The character that begins a commented line in user-editable file\n> diff --git a/fetch-pack.c b/fetch-pack.c\n> index f337526..40e7aa2 100644\n> --- a/fetch-pack.c\n> +++ b/fetch-pack.c\n> @@ -960,7 +960,14 @@ struct ref *fetch_pack(struct fetch_pack_args \n> *args,\n> WRITE_SHALLOW_NO_CUT |\n> WRITE_SHALLOW_REWRITE);\n> }\n> - commit_lock_file(&shallow_lock);\n> + if (*alternate_shallow_file == '\\0') {\n> + unlink_or_warn(git_path(\"shallow\"));\n> + rollback_lock_file(&shallow_lock);\n> + } else {\n> + if (!is_repository_shallow() && cannot_be_shallow)\n> + die(\"not allowed to turn this repository into a shallow one\");\n> + commit_lock_file(&shallow_lock);\n> + }\n> }\n> }\n>\n> diff --git a/t/t5536-fetch-shallow.sh b/t/t5536-fetch-shallow.sh\n> index 6ea6347..b7f89b1 100755\n> --- a/t/t5536-fetch-shallow.sh\n> +++ b/t/t5536-fetch-shallow.sh\n> @@ -102,6 +102,15 @@ EOF\n>\n> '\n>\n> +test_expect_success 'core.noshallow' '\n> + git init clean &&\n> + (\n> + cd clean &&\n> + git config core.noshallow true &&\n> + test_must_fail git fetch ../shallow/.git\n> + )\n> +'\n> +\n> if test -n \"$NO_CURL\" -o -z \"$GIT_TEST_HTTPD\"; then\n> say 'skipping remaining tests, git built without http support'\n> test_done\n> diff --git a/t/t5537-push-shallow.sh b/t/t5537-push-shallow.sh\n> index 8bea496..0edd51f 100755\n> --- a/t/t5537-push-shallow.sh\n> +++ b/t/t5537-push-shallow.sh\n> @@ -108,6 +108,12 @@ EOF\n> )\n> '\n>\n> +test_expect_success 'core.noshallow' '\n> + git init clean &&\n> + git --git-dir=clean/.git config core.noshallow true &&\n> + test_must_fail git --git-dir=shallow/.git push clean \n> master:refs/remotes/shallow/master\n> +'\n> +\n> if test -n \"$NO_CURL\" -o -z \"$GIT_TEST_HTTPD\"; then\n> say 'skipping remaining tests, git built without http support'\n> test_done\n> -- \n> 1.8.2.83.gc99314b\n>\n> --\n> To unsubscribe from this list: send the line \"unsubscribe git\" in\n> the body of a message to majordomo@vger.kernel.org\n> More majordomo info at http://vger.kernel.org/majordomo-info.html\n>\n>\n> -----\n> No virus found in this message.\n> Checked by AVG - www.avg.com\n> Version: 2013.0.3349 / Virus Database: 3204/6504 - Release Date: \n> 07/19/13\n> \n"},{"id":"223957","messageId":"79A9AB6FF00042D1B3E0D26B9855CCAB@PhilipOakley","threadId":"34452","inReplyTo":"1374314290-5976-1-git-send-email-pclouds@gmail.com","subject":"Re: [PATCH v2 00/16] First class shallow clone","fromName":"Philip Oakley","fromEmail":"philipoakley@iee.org","sentAt":"2013-07-22T23:41:35Z","receivedAt":"2013-07-22T23:41:35Z","isPatch":true,"sender":{"key":"philipoakley@iee.email","avatar":"https://avatars.githubusercontent.com/u/914343?v=4"},"body":"From: \"Nguyễn Thái Ngọc Duy\" \nSubject: [PATCH v2 00/16] First class shallow clone\n\nIt's nice to see that shallow can be a first class clone.\n\nThinking outside the box, does this infrastructure offer the opportunity \nto maybe add a date based depth option that would establish the shallow \nwatermark based on date rather than count. (e.g. the \"deepen\" SP depth \ncould have an alternate with a leading 'T' to indicate a time limit \nratherv than revision count - I'm expecting such a format would be an \nerror for existing servers).\n\nMy other thought was this style of cut limit list may also allow a big \nfile limit to do a similar process of listing objects (e.g. blobs) that \nare size-shallow in the repo, though it maybe a long list on some repos, \nor with a small size limit.\n\nPhilip\n\n> v2 includes:\n>\n> - fix Junio comments, especially the one that may lead to incomplete\n> commit islands.\n> - fix send-pack setting up temporary shallow file, but never passes\n> it to index-pack/unpack-objects (also fix the tests to catch this)\n> - support smart http\n> - add core.noshallow for repos that wish to be always complete\n> - fix locally cloning a shallow repository\n> - make upload-pack pass --shallow-file to pack-objects in order to\n> remove duplicate object counting code just for shallow case.\n>\n> Nguyễn Thái Ngọc Duy (16):\n> send-pack: forbid pushing from a shallow repository\n> {receive,upload}-pack: advertise shallow graft information\n> connect.c: teach get_remote_heads to parse \"shallow\" lines\n> Move setup_alternate_shallow and write_shallow_commits to shallow.c\n> fetch-pack: support fetching from a shallow repository\n> {send,receive}-pack: support pushing from a shallow clone\n> send-pack: support pushing to a shallow clone\n> upload-pack: let pack-objects do the object counting in shallow case\n> pack-protocol.txt: a bit about smart http\n> Add document for command arguments for supporting smart http\n> {fetch,upload}-pack: support fetching from a shallow clone via smart \n> http\n> receive-pack: support pushing to a shallow clone via http\n> send-pack: support pushing from a shallow clone via http\n> git-clone.txt: remove shallow clone limitations\n> config: add core.noshallow to prevent turning a repo into a shallow \n> one\n> clone: use git protocol for cloning shallow repo locally\n>\n> Documentation/config.txt | 5 +\n> Documentation/git-clone.txt | 7 +-\n> Documentation/git-fetch-pack.txt | 11 +-\n> Documentation/git-receive-pack.txt | 16 ++-\n> Documentation/git-send-pack.txt | 9 +-\n> Documentation/git-upload-pack.txt | 13 ++-\n> Documentation/technical/pack-protocol.txt | 76 ++++++++++++-\n> builtin/clone.c | 14 ++-\n> builtin/fetch-pack.c | 6 +-\n> builtin/receive-pack.c | 76 +++++++++++--\n> builtin/send-pack.c | 7 +-\n> cache.h | 4 +-\n> commit.h | 27 +++++\n> config.c | 5 +\n> connect.c | 12 +-\n> environment.c | 1 +\n> fetch-pack.c | 90 ++++++---------\n> fetch-pack.h | 1 +\n> remote-curl.c | 4 +-\n> send-pack.c | 57 +++++++++-\n> send-pack.h | 4 +-\n> shallow.c | 147 \n> +++++++++++++++++++++++++\n> t/t5530-upload-pack-error.sh | 3 -\n> t/t5536-fetch-shallow.sh (new +x) | 141 \n> ++++++++++++++++++++++++\n> t/t5537-push-shallow.sh (new +x) | 176 \n> ++++++++++++++++++++++++++++++\n> t/t5601-clone.sh | 7 ++\n> transport.c | 14 ++-\n> upload-pack.c | 132 ++++++----------------\n> 28 files changed, 858 insertions(+), 207 deletions(-)\n> create mode 100755 t/t5536-fetch-shallow.sh\n> create mode 100755 t/t5537-push-shallow.sh\n>\n> -- \n> 1.8.2.83.gc99314b\n>\n> --\n> To unsubscribe from this list: send the line \"unsubscribe git\" in\n> the body of a message to majordomo@vger.kernel.org\n> More majordomo info at http://vger.kernel.org/majordomo-info.html\n>\n>\n> -----\n> No virus found in this message.\n> Checked by AVG - www.avg.com\n> Version: 2013.0.3349 / Virus Database: 3204/6504 - Release Date: \n> 07/19/13\n> \n"},{"id":"223958","messageId":"CACsJy8AJjXMATEPrAgSYOgpZcR_khC=9S28H8LuCvuTuJk0x8w@mail.gmail.com","threadId":"34452","inReplyTo":"79A9AB6FF00042D1B3E0D26B9855CCAB@PhilipOakley","subject":"Re: [PATCH v2 00/16] First class shallow clone","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2013-07-23T01:20:54Z","receivedAt":"2013-07-23T01:20:54Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Tue, Jul 23, 2013 at 6:41 AM, Philip Oakley wrote:\n> From: \"Nguyễn Thái Ngọc Duy\" \n> Subject: [PATCH v2 00/16] First class shallow clone\n>\n> It's nice to see that shallow can be a first class clone.\n>\n> Thinking outside the box, does this infrastructure offer the opportunity to\n> maybe add a date based depth option that would establish the shallow\n> watermark based on date rather than count. (e.g. the \"deepen\" SP depth could\n\nI've been carefully avoiding the deepen issues because, as you see,\nit's complicated. But no, this series does not enable or disable new\ndeeepen mechanisms. They can always be added as protocol extensions.\nStill thinking if it's worth exposing a (restricted form of) rev-list\nto the protocol..\n\n> have an alternate with a leading 'T' to indicate a time limit ratherv than\n> revision count - I'm expecting such a format would be an error for existing\n> servers).\n>\n> My other thought was this style of cut limit list may also allow a big file\n> limit to do a similar process of listing objects (e.g. blobs) that are\n> size-shallow in the repo, though it maybe a long list on some repos, or with\n> a small size limit.\n\nThis one, on the other hand, changes the \"shape\" of the repo (now with\nholes) and might need to go through the same process we do with this\nseries. Maybe we should prepare for it now. Do you have a use case for\nsize-based filtering? What can we do with a repo with some arbitrary\nblobs missing? Another form of this is narrow clone, where we cut by\npaths, not by blob size. Narrow clone sounds more useful to me because\nit's easier to control what we leave out.\n--\nDuy\n"},{"id":"223959","messageId":"CACsJy8Ddac5r7opAgtk-CAkc=jwmBmmOs7FU6DziysCZ=992dQ@mail.gmail.com","threadId":"34452","inReplyTo":"BF5D998BFA5241938531635E1D8B79DF@PhilipOakley","subject":"Re: [PATCH v2 15/16] config: add core.noshallow to prevent turning a repo into a shallow one","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2013-07-23T01:28:00Z","receivedAt":"2013-07-23T01:28:00Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Tue, Jul 23, 2013 at 2:23 AM, Philip Oakley wrote:\n> From: \"Nguyễn Thái Ngọc Duy\" \n> Subject: [PATCH v2 15/16] config: add core.noshallow to prevent turning a\n> repo into a shallow one\n>\n> Surely this should be the default now that it is possible to corrupt a\n> golden repo by pushing/fetching a shallow repository to it and it then\n> becomes shallow, and all the followers become shallow as well, with\n> consequent problems (IIUC) [PATCH v2 05/16].\n>\n> It would be just as easy to change the config to core.allowshallow which\n> then must be enabled by the user, and can be mentioned in the shallow clone\n> option's documentation.\n\nClarification, it's not really \"corrupt\". If you have full history\nfrom a ref \"A\", fetching from another shallow clone does not touch the\nhistory of ref A at all (that is if you do _not_ specify --depth). It\nmay add a a shallow ref B, which is the reason the whole repo becomes\nshallow. The same goes for push. This is not implemented, but I'm\nthinking of adding \"clean .git/shallow\" to git repack -ad. Then if you\ndelete ref B and repack -ad, the repo could become full again.\n\nBut yeah, maybe defaulting to no shallow is better. Will do so in the\nreroll unless someone objects.\n--\nDuy\n"},{"id":"223960","messageId":"CACsJy8C3Cb+W-GSnz2VzmRBvs980yKZNskGKqjG2+89JfHYEvQ@mail.gmail.com","threadId":"34452","inReplyTo":"DADC072FCD7749C8B51D9DE594D67AB7@PhilipOakley","subject":"Re: [PATCH v2 05/16] fetch-pack: support fetching from a shallow repository","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2013-07-23T02:06:40Z","receivedAt":"2013-07-23T02:06:40Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Tue, Jul 23, 2013 at 2:10 AM, Philip Oakley wrote:\n>> And a full repository may automatically become shallow if\n>> you fetch from a shallow repository.\n>\n>\n> This \"sounds\" risky: Perhaps add a note about '[PATCH v2 15/16] config: add\n> core.noshallow to prevent turning a repo into a shallow one' which will\n> become available later in the series. The later core.noshallow may be a\n> hostage to fortune.\n\ncore.noshallow is one way. Another is reject new shallow history (i.e.\nno additions to .git/shallow) unless the user explicitly asks so\neither via --depth or a new option --shallow. This does not mean that\nfetching from a shallow clone always fails without either of those\noptions. The shallow clone may provide updates to an existing ref and\n_not_ introduce new shallow grafts and that would be successful.\ncore.noshallow (or an opposite alternative) is probably the only way\nfor push.\n--\nDuy\n"},{"id":"223961","messageId":"7v38r5apiv.fsf@alter.siamese.dyndns.org","threadId":"34452","inReplyTo":"CACsJy8Ddac5r7opAgtk-CAkc=jwmBmmOs7FU6DziysCZ=992dQ@mail.gmail.com","subject":"Re: [PATCH v2 15/16] config: add core.noshallow to prevent turning a repo into a shallow one","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-07-23T04:06:32Z","receivedAt":"2013-07-23T04:06:32Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Duy Nguyen writes:\n\n> But yeah, maybe defaulting to no shallow is better. Will do so in the\n> reroll unless someone objects.\n\nNo objections from me ;-).\nThanks.\n"},{"id":"223962","messageId":"7vy58x9aus.fsf@alter.siamese.dyndns.org","threadId":"34452","inReplyTo":"CACsJy8AJjXMATEPrAgSYOgpZcR_khC=9S28H8LuCvuTuJk0x8w@mail.gmail.com","subject":"Re: [PATCH v2 00/16] First class shallow clone","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-07-23T04:08:43Z","receivedAt":"2013-07-23T04:08:43Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Duy Nguyen writes:\n\n> This one, on the other hand, changes the \"shape\" of the repo (now with\n> holes) and might need to go through the same process we do with this\n> series. Maybe we should prepare for it now. Do you have a use case for\n> size-based filtering? What can we do with a repo with some arbitrary\n> blobs missing? Another form of this is narrow clone, where we cut by\n> paths, not by blob size. Narrow clone sounds more useful to me because\n> it's easier to control what we leave out.\n\nI was about to say \"Hear, hear\", but then stopped with a question to\nmyself: why are these \"some people do not want them\" paths in the\nsame repository in the first place?\n"},{"id":"223964","messageId":"CACsJy8A1gBh0u173FcOY6-yQ7gTHMR396HP=_1VRsyup9iO0oQ@mail.gmail.com","threadId":"34452","inReplyTo":"7vy58x9aus.fsf@alter.siamese.dyndns.org","subject":"Re: [PATCH v2 00/16] First class shallow clone","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2013-07-23T05:01:44Z","receivedAt":"2013-07-23T05:01:44Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Tue, Jul 23, 2013 at 11:08 AM, Junio C Hamano wrote:\n> Duy Nguyen writes:\n>\n>> This one, on the other hand, changes the \"shape\" of the repo (now with\n>> holes) and might need to go through the same process we do with this\n>> series. Maybe we should prepare for it now. Do you have a use case for\n>> size-based filtering? What can we do with a repo with some arbitrary\n>> blobs missing? Another form of this is narrow clone, where we cut by\n>> paths, not by blob size. Narrow clone sounds more useful to me because\n>> it's easier to control what we leave out.\n>\n> I was about to say \"Hear, hear\", but then stopped with a question to\n> myself: why are these \"some people do not want them\" paths in the\n> same repository in the first place?\n\nI think there are situations that splitting repos is not the best\nchoice but I can't think of any. There's one case though that such\n\"some people\" exist: when they migrate from another version control to\ngit and do not want to change the directory layout (because it used to\nwork ok, because of the cost of updating toolchain...)\n--\nDuy\n"},{"id":"223986","messageId":"CACsJy8A1naHQZ-DRnqBq8OWdLHa2U4dEhKw=nnOqh-yjmHpE_w@mail.gmail.com","threadId":"34452","inReplyTo":"CACsJy8C3Cb+W-GSnz2VzmRBvs980yKZNskGKqjG2+89JfHYEvQ@mail.gmail.com","subject":"Re: [PATCH v2 05/16] fetch-pack: support fetching from a shallow repository","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2013-07-23T14:39:55Z","receivedAt":"2013-07-23T14:39:55Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Tue, Jul 23, 2013 at 9:06 AM, Duy Nguyen wrote:\n> Another is reject new shallow history (i.e.\n> no additions to .git/shallow) unless the user explicitly asks so\n> either via --depth or a new option --shallow. This does not mean that\n> fetching from a shallow clone always fails without either of those\n> options.\n\nThe above was at pack level when I wrote that, although I think, from\nthe user perspective, rejecting at ref level makes more sense. That is\nif a fetch request returns one ref update with incremental updates and\none with new shallow history, instead of rejecting the whole request,\nwe reject the second ref update and accept the first one.\n--\nDuy\n"},{"id":"223998","messageId":"3791504D7BB44F5BB08DC5AC34E1CE0E@PhilipOakley","threadId":"34452","inReplyTo":"CACsJy8Ddac5r7opAgtk-CAkc=jwmBmmOs7FU6DziysCZ=992dQ@mail.gmail.com","subject":"Re: [PATCH v2 15/16] config: add core.noshallow to prevent turning a repo into a shallow one","fromName":"Philip Oakley","fromEmail":"philipoakley@iee.org","sentAt":"2013-07-23T19:44:39Z","receivedAt":"2013-07-23T19:44:39Z","isPatch":true,"sender":{"key":"philipoakley@iee.email","avatar":"https://avatars.githubusercontent.com/u/914343?v=4"},"body":"From: \"Duy Nguyen\" \nSent: Tuesday, July 23, 2013 2:28 AM\n> On Tue, Jul 23, 2013 at 2:23 AM, Philip Oakley \n> wrote:\n>> From: \"Nguyễn Thái Ngọc Duy\" \n>> Subject: [PATCH v2 15/16] config: add core.noshallow to prevent \n>> turning a\n>> repo into a shallow one\n>>\n>> Surely this should be the default now that it is possible to corrupt \n>> a\n>> golden repo by pushing/fetching a shallow repository to it and it \n>> then\n>> becomes shallow, and all the followers become shallow as well, with\n>> consequent problems (IIUC) [PATCH v2 05/16].\n>>\n>> It would be just as easy to change the config to core.allowshallow \n>> which\n>> then must be enabled by the user, and can be mentioned in the shallow \n>> clone\n>> option's documentation.\n>\n> Clarification, it's not really \"corrupt\". If you have full history\n> from a ref \"A\", fetching from another shallow clone does not touch the\n> history of ref A at all\n\n> (that is if you do _not_ specify --depth).\n\nI hadn't appreciated this conditional.\n> It\n> may add a a shallow ref B, which is the reason the whole repo becomes\n> shallow.\nI had read the initial commit message (in 05/16) as implying that it was \npossible to fool someone into pulling a shallow repo and that would make \ntheir repo just as shallow (that's without a --depth argument to the \ncommand). Had that been the case then it would have been possible to \nloose some data from deep in the DAG. Glad to hear I was mistaken. \nPerhaps if your comment above is included in the commit message to \nensure that clarification is there.\n\n>The same goes for push. This is not implemented, but I'm\n> thinking of adding \"clean .git/shallow\" to git repack -ad. Then if you\n> delete ref B and repack -ad, the repo could become full again.\n>\n> But yeah, maybe defaulting to no shallow is better. Will do so in the\n> reroll unless someone objects.\n> --\n> Duy\n> --\nPhilip \n"},{"id":"224009","messageId":"8BDFF5EEBDC8422681F1AB2C0A153CC6@PhilipOakley","threadId":"34452","inReplyTo":"CACsJy8AJjXMATEPrAgSYOgpZcR_khC=9S28H8LuCvuTuJk0x8w@mail.gmail.com","subject":"Re: [PATCH v2 00/16] First class shallow clone","fromName":"Philip Oakley","fromEmail":"philipoakley@iee.org","sentAt":"2013-07-23T22:33:22Z","receivedAt":"2013-07-23T22:33:22Z","isPatch":true,"sender":{"key":"philipoakley@iee.email","avatar":"https://avatars.githubusercontent.com/u/914343?v=4"},"body":"From: \"Duy Nguyen\" \nSent: Tuesday, July 23, 2013 2:20 AM\n> On Tue, Jul 23, 2013 at 6:41 AM, Philip Oakley \n> wrote:\n>> From: \"Nguyễn Thái Ngọc Duy\" \n>> Subject: [PATCH v2 00/16] First class shallow clone\n>>\n>> It's nice to see that shallow can be a first class clone.\n>>\n>> Thinking outside the box, does this infrastructure offer the \n>> opportunity to\n>> maybe add a date based depth option that would establish the shallow\n>> watermark based on date rather than count. (e.g. the \"deepen\" SP \n>> depth could\n>\n> I've been carefully avoiding the deepen issues because, as you see,\n> it's complicated. But no, this series does not enable or disable new\n> deeepen mechanisms. They can always be added as protocol extensions.\n> Still thinking if it's worth exposing a (restricted form of) rev-list\n> to the protocol..\n\nInteresting idea.\n>\n>> have an alternate with a leading 'T' to indicate a time limit ratherv \n>> than\n>> revision count - I'm expecting such a format would be an error for \n>> existing\n>> servers).\n>>\n>> My other thought was this style of cut limit list may also allow a \n>> big file\n>> limit to do a similar process of listing objects (e.g. blobs) that \n>> are\n>> size-shallow in the repo, though it maybe a long list on some repos, \n>> or with\n>> a small size limit.\n>\n> This one, on the other hand, changes the \"shape\" of the repo (now with\n> holes) and might need to go through the same process we do with this\n> series. Maybe we should prepare for it now. Do you have a use case for\n> size-based filtering? What can we do with a repo with some arbitrary\n> blobs missing? Another form of this is narrow clone, where we cut by\n> paths, not by blob size. Narrow clone sounds more useful to me because\n> it's easier to control what we leave out.\n\nIn some sense a project with a sub-module is a narrow clone, split at a \n'commit' object. There have been comments on the git-user list about the \nproblem of accidental adding of large files which then make the repo's \nfoot print pretty large as one use case [Git is consuming very much \nRAM]. The bigFileThreshold being one way of spotting such files as \nseparate objects, and 'trimming' them.\n\nIt doesn't feel right to 'track files and directories` as paths for \ndoing a narrow clone - it'd probably fall into the same trap as tracking \nfile renames. However if one tracks trees and blobs (as a list of sha1 \nvalues, possibly with their source path) then it should it should be \npossible to allow work on the repo with those empty directories/files in \nthe same manner as is used for sub-modules, possibly with some form of \ngit-link file as an alternate marker.\n\nThe thought process is to map sub-module working onto the other object \ntypes (blobs and trees). The user would be unable to edit the trimmed \nfiles/directories anyway, so its sha1 value can't change, allowing it to \nbe included in the next commit in the branch series.\n\nPhilip\n\n> --\n> Duy\n> --\n"},{"id":"224016","messageId":"CACsJy8AzFogspTih4mJoog6MGEWgmuae2KmFysQ0-siCvfH2yA@mail.gmail.com","threadId":"34452","inReplyTo":"8BDFF5EEBDC8422681F1AB2C0A153CC6@PhilipOakley","subject":"Re: [PATCH v2 00/16] First class shallow clone","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2013-07-24T01:57:21Z","receivedAt":"2013-07-24T01:57:21Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Wed, Jul 24, 2013 at 5:33 AM, Philip Oakley wrote:\n> In some sense a project with a sub-module is a narrow clone, split at a\n> 'commit' object.\n\nYes, except narrow clone is more flexible. You have to decide the\nsplit boundary at commit time for sub-module, while you decide the\nsame at clone time for narrow clone.\n\n> There have been comments on the git-user list about the\n> problem of accidental adding of large files which then make the repo's foot\n> print pretty large as one use case [Git is consuming very much RAM]. The\n> bigFileThreshold being one way of spotting such files as separate objects,\n> and 'trimming' them.\n\nI think rewriting history to remove those accidents is better than\nworking around it (the same for accidentally committing password). We\nmight be able to spot problems early, maybe warn user at commit time\nthat they have added an exceptionally large blob, maybe before push\ntime..\n\nThe \"Git is consuming very much RAM\" part is not right. We try to keep\nmemory usage under a limit regardless of the size of a blob. There may\nbe some cases we haven't fixed yet. Reports are welcome.\n-- \nDuy\n"},{"id":"224024","messageId":"3E185895A3E14127A858979373F76766@PhilipOakley","threadId":"34452","inReplyTo":"CACsJy8AzFogspTih4mJoog6MGEWgmuae2KmFysQ0-siCvfH2yA@mail.gmail.com","subject":"Re: [PATCH v2 00/16] First class shallow clone","fromName":"Philip Oakley","fromEmail":"philipoakley@iee.org","sentAt":"2013-07-24T07:38:55Z","receivedAt":"2013-07-24T07:38:55Z","isPatch":true,"sender":{"key":"philipoakley@iee.email","avatar":"https://avatars.githubusercontent.com/u/914343?v=4"},"body":"From: \"Duy Nguyen\" \nSent: Wednesday, July 24, 2013 2:57 AM\n> On Wed, Jul 24, 2013 at 5:33 AM, Philip Oakley \n> wrote:\n>> In some sense a project with a sub-module is a narrow clone, split at \n>> a\n>> 'commit' object.\n>\n> Yes, except narrow clone is more flexible. You have to decide the\n> split boundary at commit time for sub-module, while you decide the\n> same at clone time for narrow clone.\n>\n\nTrue. It was the thought experiment part I was referring to.\n\n>> There have been comments on the git-user list about the\n>> problem of accidental adding of large files which then make the \n>> repo's foot\n>> print pretty large as one use case [Git is consuming very much RAM]. \n>> The\n>> bigFileThreshold being one way of spotting such files as separate \n>> objects,\n>> and 'trimming' them.\n>\n> I think rewriting history to remove those accidents is better than\n> working around it (the same for accidentally committing password). We\n> might be able to spot problems early, maybe warn user at commit time\n> that they have added an exceptionally large blob, maybe before push\n> time..\n\nAgain, it was a thought experiment which related to a recent git-user \nlist comment.\nI'd expect a real use case could be a team where one member who is \npreparing documentation adds a [large] video to his branch and others \nthen get a bit concerned when they try to track it / pull it as they \nreally don't want it yet. The guy may have many versions on the central \nrepo before a final rebase has a single compressed version. Colleagues \nmay want to review the text surrounding it but not pull the video \nitself. (remembering 50 % of 'idiots' are twice as dumb as the \naverage... ;-)\n\n>\n> The \"Git is consuming very much RAM\" part is not right. We try to keep\n> memory usage under a limit regardless of the size of a blob. There may\n> be some cases we haven't fixed yet. Reports are welcome.\n\nI think this was a Windows user, but reports do pop up every now and \nagain. Some times its disc pressure, or just perceived slowness (from \nothers)\n\n\n> -- \n> Duy\n\nPhilip\n"},{"id":"224030","messageId":"CAA01CsrHaCGrt37r4bn1OBY-=skJXZBHOaPH=Gsz-8BCmiW6jQ@mail.gmail.com","threadId":"34452","inReplyTo":"CACsJy8AzFogspTih4mJoog6MGEWgmuae2KmFysQ0-siCvfH2yA@mail.gmail.com","subject":"Re: [PATCH v2 00/16] First class shallow clone","fromName":"Piotr Krukowiecki","fromEmail":"piotr.krukowiecki@gmail.com","sentAt":"2013-07-24T08:30:12Z","receivedAt":"2013-07-24T08:30:12Z","isPatch":true,"sender":{"key":"piotr.krukowiecki@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3259959?v=4"},"body":"(resending, as my phone mail client decided to send it in html, sorry\nabout that)\n\nOn Wed, Jul 24, 2013 at 3:57 AM, Duy Nguyen wrote:\n> On Wed, Jul 24, 2013 at 5:33 AM, Philip Oakley wrote:\n>> There have been comments on the git-user list about the\n>> problem of accidental adding of large files which then make the repo's foot\n>> print pretty large as one use case [Git is consuming very much RAM]. The\n>> bigFileThreshold being one way of spotting such files as separate objects,\n>> and 'trimming' them.\n>\n> I think rewriting history to remove those accidents is better than\n> working around it (the same for accidentally committing password). We\n> might be able to spot problems early, maybe warn user at commit time\n> that they have added an exceptionally large blob, maybe before push\n> time..\n\nI can imagine a situation where large files were part of the project\nat some point in history (they were required to build/use it) and\nlater were removed because build/project has changed.\n\nIt would be useful to have the history for log/blame/etc even if you\ncould not build/use old versions. A warning when checking\nout/branching such incomplete tree would be needed.\n\n-- \nPiotr Krukowiecki\n"},{"id":"224034","messageId":"CACsJy8BouRHZAmCKQ779iAUiEX-rK9_FSrxu9V-bn6MCBTBEzg@mail.gmail.com","threadId":"34452","inReplyTo":"CAA01CsrHaCGrt37r4bn1OBY-=skJXZBHOaPH=Gsz-8BCmiW6jQ@mail.gmail.com","subject":"Re: [PATCH v2 00/16] First class shallow clone","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2013-07-24T10:35:00Z","receivedAt":"2013-07-24T10:35:00Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Wed, Jul 24, 2013 at 3:30 PM, Piotr Krukowiecki\n wrote:\n> (resending, as my phone mail client decided to send it in html, sorry\n> about that)\n>\n> On Wed, Jul 24, 2013 at 3:57 AM, Duy Nguyen wrote:\n>> On Wed, Jul 24, 2013 at 5:33 AM, Philip Oakley wrote:\n>>> There have been comments on the git-user list about the\n>>> problem of accidental adding of large files which then make the repo's foot\n>>> print pretty large as one use case [Git is consuming very much RAM]. The\n>>> bigFileThreshold being one way of spotting such files as separate objects,\n>>> and 'trimming' them.\n>>\n>> I think rewriting history to remove those accidents is better than\n>> working around it (the same for accidentally committing password). We\n>> might be able to spot problems early, maybe warn user at commit time\n>> that they have added an exceptionally large blob, maybe before push\n>> time..\n>\n> I can imagine a situation where large files were part of the project\n> at some point in history (they were required to build/use it) and\n> later were removed because build/project has changed.\n>\n> It would be useful to have the history for log/blame/etc even if you\n> could not build/use old versions. A warning when checking\n> out/branching such incomplete tree would be needed.\n\nThat's what shallow clone is for. You fetch the latest (not including\nold large blobs) and work on top. For archaeology, make a full clone.\nOr do you mean log/blame/etc other paths that don't touch big blobs,\nand the clone is still incomplete?\n--\nDuy\n"},{"id":"224048","messageId":"b66fdf09-5e84-4ecc-899d-cebc644f74b7@email.android.com","threadId":"34452","inReplyTo":"CACsJy8BouRHZAmCKQ779iAUiEX-rK9_FSrxu9V-bn6MCBTBEzg@mail.gmail.com","subject":"Re: [PATCH v2 00/16] First class shallow clone","fromName":"Piotr Krukowiecki","fromEmail":"piotr.krukowiecki@gmail.com","sentAt":"2013-07-24T16:50:49Z","receivedAt":"2013-07-24T16:50:49Z","isPatch":true,"sender":{"key":"piotr.krukowiecki@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3259959?v=4"},"body":"Duy Nguyen napisał:\n>On Wed, Jul 24, 2013 at 3:30 PM, Piotr Krukowiecki\n> wrote:\n>> (resending, as my phone mail client decided to send it in html, sorry\n>> about that)\n>>\n>> On Wed, Jul 24, 2013 at 3:57 AM, Duy Nguyen \n>wrote:\n>>> On Wed, Jul 24, 2013 at 5:33 AM, Philip Oakley\n> wrote:\n>>>> There have been comments on the git-user list about the\n>>>> problem of accidental adding of large files which then make the\n>repo's foot\n>>>> print pretty large as one use case [Git is consuming very much\n>RAM]. The\n>>>> bigFileThreshold being one way of spotting such files as separate\n>objects,\n>>>> and 'trimming' them.\n>>>\n>>> I think rewriting history to remove those accidents is better than\n>>> working around it (the same for accidentally committing password).\n>We\n>>> might be able to spot problems early, maybe warn user at commit time\n>>> that they have added an exceptionally large blob, maybe before push\n>>> time..\n>>\n>> I can imagine a situation where large files were part of the project\n>> at some point in history (they were required to build/use it) and\n>> later were removed because build/project has changed.\n>>\n>> It would be useful to have the history for log/blame/etc even if you\n>> could not build/use old versions. A warning when checking\n>> out/branching such incomplete tree would be needed.\n>\n>That's what shallow clone is for. You fetch the latest (not including\n>old large blobs) and work on top. For archaeology, make a full clone.\n>Or do you mean log/blame/etc other paths that don't touch big blobs,\n>and the clone is still incomplete?\n\n\nYes, for example if large files were removed recently the last-n-commits-shallow would be useless from blame/log POV. \n"}]}