{"thread":{"id":"26719","subject":"[PATCH 1/2] refs_from_alternate: helper to use refs from alternates","startedAt":"2011-03-11T20:51:10Z","lastAt":"2011-03-15T01:57:02Z","messageCount":6,"participants":["Junio C Hamano","Johannes Sixt","Shawn Pearce"],"isPatch":true,"patchVersion":1,"patchTotal":2},"messages":[{"id":"163246","messageId":"1299876671-12838-1-git-send-email-gitster@pobox.com","threadId":"26719","inReplyTo":null,"subject":"[PATCH 1/2] refs_from_alternate: helper to use refs from alternates","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-03-11T20:51:10Z","receivedAt":"2011-03-11T20:51:10Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"The receiving end of \"git push\" advertises the objects that the repository\nitself does not use, but are at the tips of refs in other repositories\nwhose object databases are used as alternates for it. This helps it avoid\nhaving to receive (and the pusher having to send) objects that are already\navailable to the receiving repository via the alternates mechanism.\n\nTweak the helper function that implements this feature, and move it to\ntransport.[ch] for future reuse by other programs.\n\nThe additional test demonstrates how this optimization is helping \"git push\",\nand \"git fetch\" is ignorant about it.\n\nSigned-off-by: Junio C Hamano <gitster@pobox.com>\n---\n builtin/receive-pack.c           |   35 ++------------------\n t/t5501-fetch-push-alternates.sh |   66 ++++++++++++++++++++++++++++++++++++++\n transport.c                      |   34 +++++++++++++++++++\n transport.h                      |    3 ++\n 4 files changed, 106 insertions(+), 32 deletions(-)\n create mode 100755 t/t5501-fetch-push-alternates.sh\n\ndiff --git a/builtin/receive-pack.c b/builtin/receive-pack.c\nindex 760817d..a5b5fc2 100644\n--- a/builtin/receive-pack.c\n+++ b/builtin/receive-pack.c\n@@ -731,43 +731,14 @@ static int delete_only(struct command *commands)\n \treturn 1;\n }\n \n-static int add_refs_from_alternate(struct alternate_object_database *e, void *unused)\n+static void add_one_alternate_ref(const struct ref *ref, void *unused)\n {\n-\tchar *other;\n-\tsize_t len;\n-\tstruct remote *remote;\n-\tstruct transport *transport;\n-\tconst struct ref *extra;\n-\n-\te->name[-1] = '\\0';\n-\tother = xstrdup(make_absolute_path(e->base));\n-\te->name[-1] = '/';\n-\tlen = strlen(other);\n-\n-\twhile (other[len-1] == '/')\n-\t\tother[--len] = '\\0';\n-\tif (len < 8 || memcmp(other + len - 8, \"/objects\", 8))\n-\t\treturn 0;\n-\t/* Is this a git repository with refs? */\n-\tmemcpy(other + len - 8, \"/refs\", 6);\n-\tif (!is_directory(other))\n-\t\treturn 0;\n-\tother[len - 8] = '\\0';\n-\tremote = remote_get(other);\n-\ttransport = transport_get(remote, other);\n-\tfor (extra = transport_get_remote_refs(transport);\n-\t     extra;\n-\t     extra = extra->next) {\n-\t\tadd_extra_ref(\".have\", extra->old_sha1, 0);\n-\t}\n-\ttransport_disconnect(transport);\n-\tfree(other);\n-\treturn 0;\n+\tadd_extra_ref(\".have\", ref->old_sha1, 0);\n }\n \n static void add_alternate_refs(void)\n {\n-\tforeach_alt_odb(add_refs_from_alternate, NULL);\n+\tforeach_alt_odb(refs_from_alternate_cb, add_one_alternate_ref);\n }\n \n int cmd_receive_pack(int argc, const char **argv, const char *prefix)\ndiff --git a/t/t5501-fetch-push-alternates.sh b/t/t5501-fetch-push-alternates.sh\nnew file mode 100755\nindex 0000000..564ef7c\n--- /dev/null\n+++ b/t/t5501-fetch-push-alternates.sh\n@@ -0,0 +1,66 @@\n+#!/bin/sh\n+\n+test_description='fetch/push involving alternates'\n+. ./test-lib.sh\n+\n+count_objects () {\n+\t(\n+\t\techo 0 &&\n+\t\tgit count-objects -v |\n+\t\tsed -n -e 's/^count: \\(.*\\)/\\1 +/p' \\\n+\t\t    -e 's/^in-pack: \\(.*\\)/\\1 +/p' &&\n+\t\techo p\n+\t) | dc\n+}\n+\n+\n+test_expect_success setup '\n+\t(\n+\t\tgit init original &&\n+\t\tcd original &&\n+\t\ti=0 &&\n+\t\twhile test $i -le 100\n+\t\tdo\n+\t\t\techo \"$i\" >count &&\n+\t\t\tgit add count &&\n+\t\t\tgit commit -m \"$i\" || exit\n+\t\t\ti=$(($i + 1))\n+\t\tdone\n+\t) &&\n+\t(\n+\t\tgit clone --reference=original \"file:///$(pwd)/original\" one &&\n+\t\tcd one &&\n+\t\techo Z >count &&\n+\t\tgit add count &&\n+\t\tgit commit -m Z &&\n+\t\tcount_objects >../one.count\n+\t) &&\n+\tA=$(pwd)/original/.git/objects &&\n+\tgit init receiver &&\n+\techo \"$A\" >receiver/.git/objects/info/alternates &&\n+\tgit init fetcher &&\n+\techo \"$A\" >fetcher/.git/objects/info/alternates\n+'\n+\n+test_expect_success 'pushing into a repository with the same alternate' '\n+\t(\n+\t\tcd one &&\n+\t\tgit push ../receiver master:refs/heads/it\n+\t) &&\n+\t(\n+\t\tcd receiver &&\n+\t\tcount_objects >../receiver.count\n+\t) &&\n+\ttest_cmp one.count receiver.count\n+'\n+\n+test_expect_failure 'fetching from a repository with the same alternate' '\n+\t(\n+\t\tcd fetcher &&\n+\t\tgit fetch ../one master:refs/heads/it &&\n+\t\tcount_objects >../fetcher.count\n+\t) &&\n+\ttest_cmp one.count fetcher.count\n+'\n+\n+test_done\ndiff --git a/transport.c b/transport.c\nindex 0078660..3dfbc6a 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -1189,3 +1189,37 @@ char *transport_anonymize_url(const char *url)\n literal_copy:\n \treturn xstrdup(url);\n }\n+\n+int refs_from_alternate_cb(struct alternate_object_database *e, void *cb)\n+{\n+\tchar *other;\n+\tsize_t len;\n+\tstruct remote *remote;\n+\tstruct transport *transport;\n+\tconst struct ref *extra;\n+\talternate_ref_fn *ref_fn = cb;\n+\n+\te->name[-1] = '\\0';\n+\tother = xstrdup(make_absolute_path(e->base));\n+\te->name[-1] = '/';\n+\tlen = strlen(other);\n+\n+\twhile (other[len-1] == '/')\n+\t\tother[--len] = '\\0';\n+\tif (len < 8 || memcmp(other + len - 8, \"/objects\", 8))\n+\t\treturn 0;\n+\t/* Is this a git repository with refs? */\n+\tmemcpy(other + len - 8, \"/refs\", 6);\n+\tif (!is_directory(other))\n+\t\treturn 0;\n+\tother[len - 8] = '\\0';\n+\tremote = remote_get(other);\n+\ttransport = transport_get(remote, other);\n+\tfor (extra = transport_get_remote_refs(transport);\n+\t     extra;\n+\t     extra = extra->next)\n+\t\tref_fn(extra, NULL);\n+\ttransport_disconnect(transport);\n+\tfree(other);\n+\treturn 0;\n+}\ndiff --git a/transport.h b/transport.h\nindex e803c0e..efb1968 100644\n--- a/transport.h\n+++ b/transport.h\n@@ -166,4 +166,7 @@ int transport_refs_pushed(struct ref *ref);\n void transport_print_push_status(const char *dest, struct ref *refs,\n \t\t  int verbose, int porcelain, int *nonfastforward);\n \n+typedef void alternate_ref_fn(const struct ref *, void *);\n+extern int refs_from_alternate_cb(struct alternate_object_database *e, void *cb);\n+\n #endif\n-- \n1.7.4.1.404.g62d316\n"},{"id":"163247","messageId":"1299876671-12838-2-git-send-email-gitster@pobox.com","threadId":"26719","inReplyTo":"1299876671-12838-1-git-send-email-gitster@pobox.com","subject":"[PATCH 2/2] fetch-pack: objects in our alternates are available to us","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-03-11T20:51:11Z","receivedAt":"2011-03-11T20:51:11Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Use the helper function split from receiving end of \"git push\" to allow\nthe same optimization on the receiving end of \"git fetch\".\n\nSigned-off-by: Junio C Hamano <gitster@pobox.com>\n---\n builtin/fetch-pack.c             |   12 ++++++++++++\n t/t5501-fetch-push-alternates.sh |    2 +-\n 2 files changed, 13 insertions(+), 1 deletions(-)\n\ndiff --git a/builtin/fetch-pack.c b/builtin/fetch-pack.c\nindex b999413..4c25968 100644\n--- a/builtin/fetch-pack.c\n+++ b/builtin/fetch-pack.c\n@@ -9,6 +9,7 @@\n #include \"fetch-pack.h\"\n #include \"remote.h\"\n #include \"run-command.h\"\n+#include \"transport.h\"\n \n static int transfer_unpack_limit = -1;\n static int fetch_unpack_limit = -1;\n@@ -217,6 +218,16 @@ static void send_request(int fd, struct strbuf *buf)\n \t\tsafe_write(fd, buf->buf, buf->len);\n }\n \n+static void insert_one_alternate_ref(const struct ref *ref, void *unused)\n+{\n+\trev_list_insert_ref(NULL, ref->old_sha1, 0, NULL);\n+}\n+\n+static void insert_alternate_refs(void)\n+{\n+\tforeach_alt_odb(refs_from_alternate_cb, insert_one_alternate_ref);\n+}\n+\n static int find_common(int fd[2], unsigned char *result_sha1,\n \t\t       struct ref *refs)\n {\n@@ -235,6 +246,7 @@ static int find_common(int fd[2], unsigned char *result_sha1,\n \tmarked = 1;\n \n \tfor_each_ref(rev_list_insert_ref, NULL);\n+\tinsert_alternate_refs();\n \n \tfetching = 0;\n \tfor ( ; refs ; refs = refs->next) {\ndiff --git a/t/t5501-fetch-push-alternates.sh b/t/t5501-fetch-push-alternates.sh\nindex 564ef7c..a0f3644 100755\n--- a/t/t5501-fetch-push-alternates.sh\n+++ b/t/t5501-fetch-push-alternates.sh\n@@ -54,7 +54,7 @@ test_expect_success 'pushing into a repository with the same alternate' '\n \ttest_cmp one.count receiver.count\n '\n \n-test_expect_failure 'fetching from a repository with the same alternate' '\n+test_expect_success 'fetching from a repository with the same alternate' '\n \t(\n \t\tcd fetcher &&\n \t\tgit fetch ../one master:refs/heads/it &&\n-- \n1.7.4.1.404.g62d316\n"},{"id":"163252","messageId":"201103120008.27973.j6t@kdbg.org","threadId":"26719","inReplyTo":"1299876671-12838-1-git-send-email-gitster@pobox.com","subject":"Re: [PATCH 1/2] refs_from_alternate: helper to use refs from alternates","fromName":"Johannes Sixt","fromEmail":"j6t@kdbg.org","sentAt":"2011-03-11T23:08:27Z","receivedAt":"2011-03-11T23:08:27Z","isPatch":true,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"On Freitag, 11. März 2011, Junio C Hamano wrote:\n> +count_objects () {\n> +\t(\n> +\t\techo 0 &&\n> +\t\tgit count-objects -v |\n> +\t\tsed -n -e 's/^count: \\(.*\\)/\\1 +/p' \\\n> +\t\t    -e 's/^in-pack: \\(.*\\)/\\1 +/p' &&\n> +\t\techo p\n> +\t) | dc\n> +}\n\nCan we have this without using dc? (We don't have it on Windows.)\n\ndiff --git a/t/t5501-fetch-push-alternates.sh \nb/t/t5501-fetch-push-alternates.sh\nindex 564ef7c..eb844cf 100755\n--- a/t/t5501-fetch-push-alternates.sh\n+++ b/t/t5501-fetch-push-alternates.sh\n@@ -4,13 +4,11 @@ test_description='fetch/push involving alternates'\n . ./test-lib.sh\n \n count_objects () {\n-\t(\n-\t\techo 0 &&\n+\techo $(( $(\n \t\tgit count-objects -v |\n \t\tsed -n -e 's/^count: \\(.*\\)/\\1 +/p' \\\n-\t\t    -e 's/^in-pack: \\(.*\\)/\\1 +/p' &&\n-\t\techo p\n-\t) | dc\n+\t\t    -e 's/^in-pack: \\(.*\\)/\\1/p'\n+\t\t) ))\n }\n \n \n"},{"id":"163253","messageId":"7vaah1fdv8.fsf@alter.siamese.dyndns.org","threadId":"26719","inReplyTo":"201103120008.27973.j6t@kdbg.org","subject":"Re: [PATCH 1/2] refs_from_alternate: helper to use refs from alternates","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-03-11T23:15:23Z","receivedAt":"2011-03-11T23:15:23Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johannes Sixt <j6t@kdbg.org> writes:\n\n> Can we have this without using dc? (We don't have it on Windows.)\n>\n> diff --git a/t/t5501-fetch-push-alternates.sh \n> b/t/t5501-fetch-push-alternates.sh\n> index 564ef7c..eb844cf 100755\n> --- a/t/t5501-fetch-push-alternates.sh\n> +++ b/t/t5501-fetch-push-alternates.sh\n> @@ -4,13 +4,11 @@ test_description='fetch/push involving alternates'\n>  . ./test-lib.sh\n>  \n>  count_objects () {\n> +\techo $(( $(\n>  \t\tgit count-objects -v |\n>  \t\tsed -n -e 's/^count: \\(.*\\)/\\1 +/p' \\\n> +\t\t    -e 's/^in-pack: \\(.*\\)/\\1/p'\n> +\t\t) ))\n>  }\n\nWhat does that do when there are only loose objects without packed ones?\n"},{"id":"163254","messageId":"7v62rpfd2l.fsf@alter.siamese.dyndns.org","threadId":"26719","inReplyTo":"7vaah1fdv8.fsf@alter.siamese.dyndns.org","subject":"Re: [PATCH 1/2] refs_from_alternate: helper to use refs from alternates","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2011-03-11T23:32:34Z","receivedAt":"2011-03-11T23:32:34Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano <gitster@pobox.com> writes:\n\n> Johannes Sixt <j6t@kdbg.org> writes:\n>\n>> Can we have this without using dc? (We don't have it on Windows.)\n>>\n>> diff --git a/t/t5501-fetch-push-alternates.sh \n>> b/t/t5501-fetch-push-alternates.sh\n>> index 564ef7c..eb844cf 100755\n>> --- a/t/t5501-fetch-push-alternates.sh\n>> +++ b/t/t5501-fetch-push-alternates.sh\n>> @@ -4,13 +4,11 @@ test_description='fetch/push involving alternates'\n>>  . ./test-lib.sh\n>>  \n>>  count_objects () {\n>> +\techo $(( $(\n>>  \t\tgit count-objects -v |\n>>  \t\tsed -n -e 's/^count: \\(.*\\)/\\1 +/p' \\\n>> +\t\t    -e 's/^in-pack: \\(.*\\)/\\1/p'\n>> +\t\t) ))\n>>  }\n>\n> What does that do when there are only loose objects without packed ones?\n\nAhh, Ok, we always say \"0\", so that is not a problem.  Will squash in.\n\nThanks.\n"},{"id":"163363","messageId":"AANLkTimYBZf72tjAr2s3TTFb3rSZb0_xLhFE4zLtEsN7@mail.gmail.com","threadId":"26719","inReplyTo":"1299876671-12838-2-git-send-email-gitster@pobox.com","subject":"Re: [PATCH 2/2] fetch-pack: objects in our alternates are available to us","fromName":"Shawn Pearce","fromEmail":"spearce@spearce.org","sentAt":"2011-03-15T01:57:02Z","receivedAt":"2011-03-15T01:57:02Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"On Fri, Mar 11, 2011 at 12:51, Junio C Hamano <gitster@pobox.com> wrote:\n> Use the helper function split from receiving end of \"git push\" to allow\n> the same optimization on the receiving end of \"git fetch\".\n>\n> Signed-off-by: Junio C Hamano <gitster@pobox.com>\n> ---\n>  builtin/fetch-pack.c             |   12 ++++++++++++\n>  t/t5501-fetch-push-alternates.sh |    2 +-\n>  2 files changed, 13 insertions(+), 1 deletions(-)\n\nI apologize for the slow response; both patches\nAcked-by: Shawn O. Pearce <spearce@spearce.org>\n\n-- \nShawn.\n"}]}