{"thread":{"id":"20836","subject":"[PATCH] make shallow repository deepening more network efficient","startedAt":"2009-09-03T23:08:33Z","lastAt":"2009-09-26T13:26:14Z","messageCount":3,"participants":["Nicolas Pitre","Junio C Hamano","Johannes Schindelin"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"122370","messageId":"alpine.LFD.2.00.0909031847520.6044@xanadu.home","threadId":"20836","inReplyTo":null,"subject":"[PATCH] make shallow repository deepening more network efficient","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2009-09-03T23:08:33Z","receivedAt":"2009-09-03T23:08:33Z","isPatch":true,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"\nFirst of all, I can't find any reason why thin pack generation is \nexplicitly disabled when dealing with a shallow repository.  The \npossible delta base objects are collected from the edge commits which \nare always obtained through history walking with the same shallow refs \nas the client, Therefore the client is always going to have those base \nobjects available. So let's remove that restriction.\n\nThen we can make shallow repository deepening much more efficient by \nusing the remote's unshallowed commits as edge commits to get preferred \nbase objects for thin pack generation.  On git.git, this makes the data \ntransfer for the deepening of a shallow repository from depth 1 to depth 2\naround 134 KB instead of 3.68 MB.\n\nSigned-off-by: Nicolas Pitre <nico@cam.org>\n\ndiff --git a/upload-pack.c b/upload-pack.c\nindex f73e3c9..c77ab71 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -32,6 +32,7 @@ static int no_progress, daemon_mode;\n static int shallow_nr;\n static struct object_array have_obj;\n static struct object_array want_obj;\n+static struct object_array extra_edge_obj;\n static unsigned int timeout;\n /* 0 for no sideband,\n  * otherwise maximum packet size (up to 65520 bytes).\n@@ -135,6 +136,10 @@ static int do_rev_list(int fd, void *create_full_pack)\n \tif (prepare_revision_walk(&revs))\n \t\tdie(\"revision walk setup failed\");\n \tmark_edges_uninteresting(revs.commits, &revs, show_edge);\n+\tif (use_thin_pack)\n+\t\tfor (i = 0; i < extra_edge_obj.nr; i++)\n+\t\t\tfprintf(pack_pipe, \"-%s\\n\", sha1_to_hex(\n+\t\t\t\t\textra_edge_obj.objects[i].item->sha1));\n \ttraverse_commit_list(&revs, show_commit, show_object, NULL);\n \tfflush(pack_pipe);\n \tfclose(pack_pipe);\n@@ -562,7 +567,6 @@ static void receive_needs(void)\n \t\tif (!prefixcmp(line, \"shallow \")) {\n \t\t\tunsigned char sha1[20];\n \t\t\tstruct object *object;\n-\t\t\tuse_thin_pack = 0;\n \t\t\tif (get_sha1(line + 8, sha1))\n \t\t\t\tdie(\"invalid shallow line: %s\", line);\n \t\t\tobject = parse_object(sha1);\n@@ -574,7 +578,6 @@ static void receive_needs(void)\n \t\t}\n \t\tif (!prefixcmp(line, \"deepen \")) {\n \t\t\tchar *end;\n-\t\t\tuse_thin_pack = 0;\n \t\t\tdepth = strtol(line + 7, &end, 0);\n \t\t\tif (end == line + 7 || depth <= 0)\n \t\t\t\tdie(\"Invalid deepen: %s\", line);\n@@ -657,6 +660,7 @@ static void receive_needs(void)\n \t\t\t\t\t\t\tNULL, &want_obj);\n \t\t\t\t\tparents = parents->next;\n \t\t\t\t}\n+\t\t\t\tadd_object_array(object, NULL, &extra_edge_obj);\n \t\t\t}\n \t\t\t/* make sure commit traversal conforms to client */\n \t\t\tregister_shallow(object->sha1);\n"},{"id":"122533","messageId":"7vmy58r72b.fsf@alter.siamese.dyndns.org","threadId":"20836","inReplyTo":"alpine.LFD.2.00.0909031847520.6044@xanadu.home","subject":"Re: [PATCH] make shallow repository deepening more network efficient","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2009-09-06T05:11:24Z","receivedAt":"2009-09-06T05:11:24Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Nicolas Pitre <nico@cam.org> writes:\n\n> First of all, I can't find any reason why thin pack generation is \n> explicitly disabled when dealing with a shallow repository.  The \n> possible delta base objects are collected from the edge commits which \n> are always obtained through history walking with the same shallow refs \n> as the client, Therefore the client is always going to have those base \n> objects available. So let's remove that restriction.\n>\n> Then we can make shallow repository deepening much more efficient by \n> using the remote's unshallowed commits as edge commits to get preferred \n> base objects for thin pack generation.  On git.git, this makes the data \n> transfer for the deepening of a shallow repository from depth 1 to depth 2\n> around 134 KB instead of 3.68 MB.\n>\n> Signed-off-by: Nicolas Pitre <nico@cam.org>\n\nDscho, this is your code from around ed09aef (support fetching into a\nshallow repository, 2006-10-30) and f53514b (allow deepening of a shallow\nrepository, 2006-10-30).  The latter disables thin pack transfer but the\nlog does not attempt to justify the change.\n\nHave any comments?\n\n> diff --git a/upload-pack.c b/upload-pack.c\n> index f73e3c9..c77ab71 100644\n> --- a/upload-pack.c\n> +++ b/upload-pack.c\n> @@ -32,6 +32,7 @@ static int no_progress, daemon_mode;\n>  static int shallow_nr;\n>  static struct object_array have_obj;\n>  static struct object_array want_obj;\n> +static struct object_array extra_edge_obj;\n>  static unsigned int timeout;\n>  /* 0 for no sideband,\n>   * otherwise maximum packet size (up to 65520 bytes).\n> @@ -135,6 +136,10 @@ static int do_rev_list(int fd, void *create_full_pack)\n>  \tif (prepare_revision_walk(&revs))\n>  \t\tdie(\"revision walk setup failed\");\n>  \tmark_edges_uninteresting(revs.commits, &revs, show_edge);\n> +\tif (use_thin_pack)\n> +\t\tfor (i = 0; i < extra_edge_obj.nr; i++)\n> +\t\t\tfprintf(pack_pipe, \"-%s\\n\", sha1_to_hex(\n> +\t\t\t\t\textra_edge_obj.objects[i].item->sha1));\n>  \ttraverse_commit_list(&revs, show_commit, show_object, NULL);\n>  \tfflush(pack_pipe);\n>  \tfclose(pack_pipe);\n> @@ -562,7 +567,6 @@ static void receive_needs(void)\n>  \t\tif (!prefixcmp(line, \"shallow \")) {\n>  \t\t\tunsigned char sha1[20];\n>  \t\t\tstruct object *object;\n> -\t\t\tuse_thin_pack = 0;\n>  \t\t\tif (get_sha1(line + 8, sha1))\n>  \t\t\t\tdie(\"invalid shallow line: %s\", line);\n>  \t\t\tobject = parse_object(sha1);\n> @@ -574,7 +578,6 @@ static void receive_needs(void)\n>  \t\t}\n>  \t\tif (!prefixcmp(line, \"deepen \")) {\n>  \t\t\tchar *end;\n> -\t\t\tuse_thin_pack = 0;\n>  \t\t\tdepth = strtol(line + 7, &end, 0);\n>  \t\t\tif (end == line + 7 || depth <= 0)\n>  \t\t\t\tdie(\"Invalid deepen: %s\", line);\n> @@ -657,6 +660,7 @@ static void receive_needs(void)\n>  \t\t\t\t\t\t\tNULL, &want_obj);\n>  \t\t\t\t\tparents = parents->next;\n>  \t\t\t\t}\n> +\t\t\t\tadd_object_array(object, NULL, &extra_edge_obj);\n>  \t\t\t}\n>  \t\t\t/* make sure commit traversal conforms to client */\n>  \t\t\tregister_shallow(object->sha1);\n"},{"id":"123841","messageId":"alpine.DEB.1.00.0909261524490.4985@pacific.mpi-cbg.de","threadId":"20836","inReplyTo":"7vmy58r72b.fsf@alter.siamese.dyndns.org","subject":"Re: [PATCH] make shallow repository deepening more network efficient","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2009-09-26T13:26:14Z","receivedAt":"2009-09-26T13:26:14Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Sat, 5 Sep 2009, Junio C Hamano wrote:\n\n> Nicolas Pitre <nico@cam.org> writes:\n> \n> > First of all, I can't find any reason why thin pack generation is \n> > explicitly disabled when dealing with a shallow repository.  The \n> > possible delta base objects are collected from the edge commits which \n> > are always obtained through history walking with the same shallow refs \n> > as the client, Therefore the client is always going to have those base \n> > objects available. So let's remove that restriction.\n> >\n> > Then we can make shallow repository deepening much more efficient by \n> > using the remote's unshallowed commits as edge commits to get preferred \n> > base objects for thin pack generation.  On git.git, this makes the data \n> > transfer for the deepening of a shallow repository from depth 1 to depth 2\n> > around 134 KB instead of 3.68 MB.\n> >\n> > Signed-off-by: Nicolas Pitre <nico@cam.org>\n> \n> Dscho, this is your code from around ed09aef (support fetching into a\n> shallow repository, 2006-10-30) and f53514b (allow deepening of a shallow\n> repository, 2006-10-30).  The latter disables thin pack transfer but the\n> log does not attempt to justify the change.\n> \n> Have any comments?\n\nUnfortunately, I do not have any.  I tried to remember, then I tried to \nfind some documentation, but stopped when I found out that I developed \nthe code on an iBook which died early Nov 2006.\n\nSo no, I do not remember why the change.\n\nSorry,\nDscho\n"}]}