{"thread":{"id":"1803","subject":"[PATCH RFC] Add support for alternates in HTTP","startedAt":"2005-09-15T03:26:08Z","lastAt":"2005-09-16T05:49:11Z","messageCount":4,"participants":["Daniel Barkalow","Junio C Hamano","Jeff Garzik","Keith Owens"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"8575","messageId":"Pine.LNX.4.63.0509142319330.23242@iabervon.org","threadId":"1803","inReplyTo":null,"subject":"[PATCH RFC] Add support for alternates in HTTP","fromName":"Daniel Barkalow","fromEmail":"barkalow@iabervon.org","sentAt":"2005-09-15T03:26:08Z","receivedAt":"2005-09-15T03:26:08Z","isPatch":true,"sender":{"key":"barkalow@iabervon.org","avatar":"https://avatars.githubusercontent.com/u/55364219?v=4"},"body":"This tries .../objects/info/http-alternates and then\n.../objects/info/alternates, looking for a file which specifies where\nelse to download objects and packs from.\n\nIt currently only supports absolute paths, and doesn't support full URLs.\n\nSigned-off-by: Daniel Barkalow <barkalow@iabervon.org>\n\n---\nThis is probably pu material at this point; there's a bit more parsing \nwork that ought to get done once there are suitable test cases out there \nfor me to try.\n\nIt tries to find additional places it should look for things to download. \nWhat I'd like it to do is:\n\n Try .../objects/info/http-alternates, which can be a full URL, an \n absolute path (same server), or a relative path based on the objects \n directory. This should specify the objects directory, not the base.\n\n Try .../objects/info/alternates, which can be a relative path as above, \n or, just in case it works, an absolute path.\n\nCurrently implemented are the two absolute path cases, which let me fetch \nfrom the aegl repository, getting Linus's pack from his repository.\n\n http-fetch.c |  169 ++++++++++++++++++++++++++++++++++++++++++++++------------\n 1 files changed, 134 insertions(+), 35 deletions(-)\n\nc01c2fc40aafddc23a14ffdc07de3bf950e323a4\ndiff --git a/http-fetch.c b/http-fetch.c\n--- a/http-fetch.c\n+++ b/http-fetch.c\n@@ -15,7 +15,17 @@\n \n static CURL *curl;\n \n-static char *base;\n+static char *initial_base;\n+\n+struct alt_base\n+{\n+\tchar *base;\n+\tint got_indices;\n+\tstruct packed_git *packs;\n+\tstruct alt_base *next;\n+};\n+\n+struct alt_base *alt = NULL;\n \n static SHA_CTX c;\n static z_stream stream;\n@@ -71,11 +81,9 @@ void prefetch(unsigned char *sha1)\n {\n }\n \n-static int got_indices = 0;\n-\n-static struct packed_git *packs = NULL;\n+static int got_alternates = 0;\n \n-static int fetch_index(unsigned char *sha1)\n+static int fetch_index(struct alt_base *repo, unsigned char *sha1)\n {\n \tchar *filename;\n \tchar *url;\n@@ -89,9 +97,9 @@ static int fetch_index(unsigned char *sh\n \t\tfprintf(stderr, \"Getting index for pack %s\\n\",\n \t\t\tsha1_to_hex(sha1));\n \t\n-\turl = xmalloc(strlen(base) + 64);\n+\turl = xmalloc(strlen(repo->base) + 64);\n \tsprintf(url, \"%s/objects/pack/pack-%s.idx\",\n-\t\tbase, sha1_to_hex(sha1));\n+\t\trepo->base, sha1_to_hex(sha1));\n \t\n \tfilename = sha1_pack_index_name(sha1);\n \tindexfile = fopen(filename, \"w\");\n@@ -112,22 +120,92 @@ static int fetch_index(unsigned char *sh\n \treturn 0;\n }\n \n-static int setup_index(unsigned char *sha1)\n+static int setup_index(struct alt_base *repo, unsigned char *sha1)\n {\n \tstruct packed_git *new_pack;\n \tif (has_pack_file(sha1))\n \t\treturn 0; // don't list this as something we can get\n \n-\tif (fetch_index(sha1))\n+\tif (fetch_index(repo, sha1))\n \t\treturn -1;\n \n \tnew_pack = parse_pack_index(sha1);\n-\tnew_pack->next = packs;\n-\tpacks = new_pack;\n+\tnew_pack->next = repo->packs;\n+\trepo->packs = new_pack;\n \treturn 0;\n }\n \n-static int fetch_indices(void)\n+static int fetch_alternates(char *base)\n+{\n+\tint ret = 0;\n+\tstruct buffer buffer;\n+\tchar *url;\n+\tchar *data;\n+\tint i = 0;\n+\tif (got_alternates)\n+\t\treturn 0;\n+\tdata = xmalloc(4096);\n+\tbuffer.size = 4096;\n+\tbuffer.posn = 0;\n+\tbuffer.buffer = data;\n+\n+\tif (get_verbosely)\n+\t\tfprintf(stderr, \"Getting alternates list\\n\");\n+\t\n+\turl = xmalloc(strlen(base) + 31);\n+\tsprintf(url, \"%s/objects/info/http-alternates\", base);\n+\n+\tcurl_easy_setopt(curl, CURLOPT_FILE, &buffer);\n+\tcurl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, fwrite_buffer);\n+\tcurl_easy_setopt(curl, CURLOPT_URL, url);\n+\n+\tif (curl_easy_perform(curl) || !buffer.posn) {\n+\t\tsprintf(url, \"%s/objects/info/alternates\", base);\n+\t\t\n+\t\tcurl_easy_setopt(curl, CURLOPT_FILE, &buffer);\n+\t\tcurl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, fwrite_buffer);\n+\t\tcurl_easy_setopt(curl, CURLOPT_URL, url);\n+\t\t\n+\t\tif (curl_easy_perform(curl)) {\n+\t\t\treturn 0;\n+\t\t}\n+\t}\n+\n+\twhile (i < buffer.posn) {\n+\t\tint posn = i;\n+\t\twhile (posn < buffer.posn && data[posn] != '\\n')\n+\t\t\tposn++;\n+\t\tif (data[posn] == '\\n') {\n+\t\t\tif (data[i] == '/') {\n+\t\t\t\tint serverlen = strchr(base + 8, '/') - base;\n+\t\t\t\t// skip 'objects' at end\n+\t\t\t\tchar *target = \n+\t\t\t\t\txmalloc(serverlen + posn - i - 6);\n+\t\t\t\tstruct alt_base *newalt;\n+\t\t\t\tstrncpy(target, base, serverlen);\n+\t\t\t\tstrncpy(target + serverlen, data + i,\n+\t\t\t\t\tposn - i - 7);\n+\t\t\t\ttarget[serverlen + posn - i - 7] = '\\0';\n+\t\t\t\tif (get_verbosely)\n+\t\t\t\t\tfprintf(stderr, \n+\t\t\t\t\t\t\"Also look at %s\\n\", target);\n+\t\t\t\tnewalt = xmalloc(sizeof(*newalt));\n+\t\t\t\tnewalt->next = alt;\n+\t\t\t\tnewalt->base = target;\n+\t\t\t\tnewalt->got_indices = 0;\n+\t\t\t\tnewalt->packs = NULL;\n+\t\t\t\talt = newalt;\n+\t\t\t\tret++;\n+\t\t\t}\n+\t\t}\n+\t\ti = posn + 1;\n+\t}\n+\tgot_alternates = 1;\n+\t\n+\treturn ret;\n+}\n+\n+static int fetch_indices(struct alt_base *repo)\n {\n \tunsigned char sha1[20];\n \tchar *url;\n@@ -135,7 +213,7 @@ static int fetch_indices(void)\n \tchar *data;\n \tint i = 0;\n \n-\tif (got_indices)\n+\tif (repo->got_indices)\n \t\treturn 0;\n \n \tdata = xmalloc(4096);\n@@ -146,8 +224,8 @@ static int fetch_indices(void)\n \tif (get_verbosely)\n \t\tfprintf(stderr, \"Getting pack list\\n\");\n \t\n-\turl = xmalloc(strlen(base) + 21);\n-\tsprintf(url, \"%s/objects/info/packs\", base);\n+\turl = xmalloc(strlen(repo->base) + 21);\n+\tsprintf(url, \"%s/objects/info/packs\", repo->base);\n \n \tcurl_easy_setopt(curl, CURLOPT_FILE, &buffer);\n \tcurl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, fwrite_buffer);\n@@ -157,7 +235,7 @@ static int fetch_indices(void)\n \t\treturn error(\"Unable to get pack index %s\", url);\n \t}\n \n-\tdo {\n+\twhile (i < buffer.posn) {\n \t\tswitch (data[i]) {\n \t\tcase 'P':\n \t\t\ti++;\n@@ -165,7 +243,7 @@ static int fetch_indices(void)\n \t\t\t    !strncmp(data + i, \" pack-\", 6) &&\n \t\t\t    !strncmp(data + i + 46, \".pack\\n\", 6)) {\n \t\t\t\tget_sha1_hex(data + i + 6, sha1);\n-\t\t\t\tsetup_index(sha1);\n+\t\t\t\tsetup_index(repo, sha1);\n \t\t\t\ti += 51;\n \t\t\t\tbreak;\n \t\t\t}\n@@ -174,13 +252,13 @@ static int fetch_indices(void)\n \t\t\t\ti++;\n \t\t}\n \t\ti++;\n-\t} while (i < buffer.posn);\n+\t}\n \n-\tgot_indices = 1;\n+\trepo->got_indices = 1;\n \treturn 0;\n }\n \n-static int fetch_pack(unsigned char *sha1)\n+static int fetch_pack(struct alt_base *repo, unsigned char *sha1)\n {\n \tchar *url;\n \tstruct packed_git *target;\n@@ -188,12 +266,11 @@ static int fetch_pack(unsigned char *sha\n \tFILE *packfile;\n \tchar *filename;\n \n-\tif (fetch_indices())\n+\tif (fetch_indices(repo))\n \t\treturn -1;\n-\ttarget = find_sha1_pack(sha1, packs);\n+\ttarget = find_sha1_pack(sha1, repo->packs);\n \tif (!target)\n-\t\treturn error(\"Couldn't get %s: not separate or in any pack\",\n-\t\t\t     sha1_to_hex(sha1));\n+\t\treturn -1;\n \n \tif (get_verbosely) {\n \t\tfprintf(stderr, \"Getting pack %s\\n\",\n@@ -202,9 +279,9 @@ static int fetch_pack(unsigned char *sha\n \t\t\tsha1_to_hex(sha1));\n \t}\n \n-\turl = xmalloc(strlen(base) + 65);\n+\turl = xmalloc(strlen(repo->base) + 65);\n \tsprintf(url, \"%s/objects/pack/pack-%s.pack\",\n-\t\tbase, sha1_to_hex(target->sha1));\n+\t\trepo->base, sha1_to_hex(target->sha1));\n \n \tfilename = sha1_pack_name(target->sha1);\n \tpackfile = fopen(filename, \"w\");\n@@ -223,7 +300,7 @@ static int fetch_pack(unsigned char *sha\n \n \tfclose(packfile);\n \n-\tlst = &packs;\n+\tlst = &repo->packs;\n \twhile (*lst != target)\n \t\tlst = &((*lst)->next);\n \t*lst = (*lst)->next;\n@@ -233,7 +310,7 @@ static int fetch_pack(unsigned char *sha\n \treturn 0;\n }\n \n-int fetch(unsigned char *sha1)\n+int fetch_object(struct alt_base *repo, unsigned char *sha1)\n {\n \tchar *hex = sha1_to_hex(sha1);\n \tchar *filename = sha1_file_name(sha1);\n@@ -256,9 +333,9 @@ int fetch(unsigned char *sha1)\n \tcurl_easy_setopt(curl, CURLOPT_FILE, NULL);\n \tcurl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, fwrite_sha1_file);\n \n-\turl = xmalloc(strlen(base) + 50);\n-\tstrcpy(url, base);\n-\tposn = url + strlen(base);\n+\turl = xmalloc(strlen(repo->base) + 50);\n+\tstrcpy(url, repo->base);\n+\tposn = url + strlen(repo->base);\n \tstrcpy(posn, \"objects/\");\n \tposn += 8;\n \tmemcpy(posn, hex, 2);\n@@ -270,9 +347,7 @@ int fetch(unsigned char *sha1)\n \n \tif (curl_easy_perform(curl)) {\n \t\tunlink(filename);\n-\t\tif (fetch_pack(sha1))\n-\t\t\treturn error(\"Tried %s\", url);\n-\t\treturn 0;\n+\t\treturn -1;\n \t}\n \n \tclose(local);\n@@ -291,11 +366,30 @@ int fetch(unsigned char *sha1)\n \treturn 0;\n }\n \n+int fetch(unsigned char *sha1)\n+{\n+\tstruct alt_base *altbase = alt;\n+\twhile (altbase) {\n+\t\tif (!fetch_object(altbase, sha1))\n+\t\t\treturn 0;\n+\t\tif (!fetch_pack(altbase, sha1))\n+\t\t\treturn 0;\n+\t\tif (fetch_alternates(altbase->base) > 0) {\n+\t\t\taltbase = alt;\n+\t\t\tcontinue;\n+\t\t}\n+\t\taltbase = altbase->next;\n+\t}\n+\treturn error(\"Unable to find %s under %s\\n\", sha1_to_hex(sha1), \n+\t\t     initial_base);\n+}\n+\n int fetch_ref(char *ref, unsigned char *sha1)\n {\n         char *url, *posn;\n         char hex[42];\n         struct buffer buffer;\n+\tchar *base = initial_base;\n         buffer.size = 41;\n         buffer.posn = 0;\n         buffer.buffer = hex;\n@@ -361,7 +455,12 @@ int main(int argc, char **argv)\n \tcurl_easy_setopt(curl, CURLOPT_NETRC, CURL_NETRC_OPTIONAL);\n #endif\n \n-\tbase = url;\n+\talt = xmalloc(sizeof(*alt));\n+\talt->base = url;\n+\talt->got_indices = 0;\n+\talt->packs = NULL;\n+\talt->next = NULL;\n+\tinitial_base = url;\n \n \tif (pull(commit_id))\n \t\treturn 1;\n"},{"id":"8647","messageId":"7vbr2tx51n.fsf@assigned-by-dhcp.cox.net","threadId":"1803","inReplyTo":"Pine.LNX.4.63.0509142319330.23242@iabervon.org","subject":"Current state of GIT fetch/pull clients","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2005-09-15T23:52:20Z","receivedAt":"2005-09-15T23:52:20Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"After Linus suggested to use objects/info/alternates to point at\nhis linux-2.6 tree so that the maintainer trees can borrow from\nit, there may be a bit of confusion.  Here is my attempt to\nclarify the current state of affairs.\n\nIf you are a subsystem maintainer with an account $u at\nmaster.kernel.org and have your repository $tree derived from\nLinus' linux-2.6.git reopsitory, it would be at:\n\n      /pub/scm/linux/kernel/git/$u/$tree\n\nOf course, you may have more than one such $tree.  The\nsuggestion by Linus was to do (please do not do this yet -- that\nis what this message is about):\n\n    $ cd /pub/scm/linux/kernel/git/$u/$tree\n    $ cat /pub/scm/linux/kernel/git/torvalds/linux-2.6/objects \\\n        >objects/info/alternates\n    $ GIT_DIR=. git prune\n\nWhat this does is:\n\n    * A process on master.kernel.org and its mirrors running in\n      your repository (i.e. /pub/scm/linux/kernel/git/$u/$tree)\n      can read objects not found in your repository from Linus'\n      repository.  This means that $u/$tree/objects/?? would\n      contain only your changes and practically 99% of the\n      objects are coming from Linus' repository for such\n      process.  You do not have to have any objects Linus'\n      repository has in your repository.\n\nSounds very nice, doesn't it?\n\nHowever, there are currently certain issues around almost all\ngit clients, and Cogito cg-pull shares this problem.\n\nNeither http nor rsync transports know about the 'alternates'\nmechanism yet, so if a downloader does:\n\n    $ git pull http://kernel.org/pub/scm/linux/kernel/git/$u/$tree\n    $ git pull rsync://kernel.org/pub/scm/linux/kernel/git/$u/$tree\n\nunless the downloader has already fetched from Linus'\nrepository, this will not work.\n\n  * In the case of http transport, it would start from your branch\n    head, walking commits backwards, and as soon as it hits a commit\n    (or a tree/blob) that your repository is borrowing from Linus.\n\n  * In the case of rsync transport, it would slurp all objects\n    your repository has, but does not get objects from Linus'\n    repository.  Also, rsync will overwrite the\n    objects/info/alternates file the downloader has in his\n    repository with what you have in your repository, which is\n    not what we want.\n\nThe only transport that works is what Linus uses himself.  If\nthe downloader has an account on master.kernel.org:\n\n    $ git pull master.kernel.org:/pub/scm/linux/kernel/git/$u/$tree\n\nwould work, because this transport runs git in your repository\non the master.kernel.org side, and that knows how to use\nobjects/info/alternates file you set up as Linus suggested.\nAnother transport that _could_ work is git-daemon:\n\n    $ git pull git://kernel.org/pub/scm/linux/kernel/git/$u/$tree\n\nbut unfortunately kernel.org servers do not run git-daemon yet.\n\nWhat this means is that using objects/info/alternates mechanism\nin your repository is a bit premature as things currently stand,\nif you intend your repository to be used by the general public.\n\nDaniel started working on teaching http transport to be able to\nread from objects/info/alternates last night, and I am expecting\nthat would be proven stable and usable sometime next week at the\nlatest.  HPA is helping us in the discussion to whip git-daemon\ninto a shape usable on kernel.org to enable it there.  I'll be\nfixing rsync transport sometime over this weekend.\n"},{"id":"8652","messageId":"432A19F6.8050605@pobox.com","threadId":"1803","inReplyTo":"7vbr2tx51n.fsf@assigned-by-dhcp.cox.net","subject":"Re: Current state of GIT fetch/pull clients","fromName":"Jeff Garzik","fromEmail":"jgarzik@pobox.com","sentAt":"2005-09-16T01:03:50Z","receivedAt":"2005-09-16T01:03:50Z","isPatch":false,"sender":{"key":"jgarzik@pobox.com","avatar":null},"body":"Junio C Hamano wrote:\n> Neither http nor rsync transports know about the 'alternates'\n> mechanism yet, so if a downloader does:\n> \n>     $ git pull http://kernel.org/pub/scm/linux/kernel/git/$u/$tree\n>     $ git pull rsync://kernel.org/pub/scm/linux/kernel/git/$u/$tree\n> \n> unless the downloader has already fetched from Linus'\n> repository, this will not work.\n> \n>   * In the case of rsync transport, it would slurp all objects\n>     your repository has, but does not get objects from Linus'\n>     repository.  Also, rsync will overwrite the\n>     objects/info/alternates file the downloader has in his\n>     repository with what you have in your repository, which is\n>     not what we want.\n\nYes, this is why I don't bother with alternates at the present time. \nUsers of my repos, at least, have been trained to use rsync://... and \ncurrently expect to get a working tree that way.\n\n\tJeff\n"},{"id":"8656","messageId":"8953.1126849751@kao2.melbourne.sgi.com","threadId":"1803","inReplyTo":"7vbr2tx51n.fsf@assigned-by-dhcp.cox.net","subject":"Re: Current state of GIT fetch/pull clients","fromName":"Keith Owens","fromEmail":"kaos@sgi.com","sentAt":"2005-09-16T05:49:11Z","receivedAt":"2005-09-16T05:49:11Z","isPatch":false,"sender":{"key":"kaos@sgi.com","avatar":null},"body":"On Thu, 15 Sep 2005 16:52:20 -0700, \nJunio C Hamano <junkio@cox.net> wrote:\n>What this means is that using objects/info/alternates mechanism\n>in your repository is a bit premature as things currently stand,\n>if you intend your repository to be used by the general public.\n\nClients using rsync can use a workaround, although it is a bit clumsy.\nI do rsync for selected /pub/scm/linux/kernel/git/ trees, including\ntorvalds and aegl.  At one point Tony Luck (aegl) was using alternates,\nwith objects/info/alternates containing\n/pub/scm/linux/kernel/git/torvalds/linux-2.6.git/objects\n\nAs you described in your mail, rsync with alternates overwrites the\nlocal file, so the local alternates ended up pointing at a local\ndirectory that did not exist.  Creating a local symlink from\n/pub/scm/linux/kernel/git to the local directory that contains the\ntorvalds and aegl directories worked around the problem.  git checkout\non the aegl tree was quite happy to follow the symlink and pick up most\nof the files from torvalds.\n\nNo point in doing that now, Tony has reverted the aegl tree to a full\ncopy of Linus, instead of using alternates.\n"}]}