{"thread":{"id":"2023","subject":"clone: I'm only doing a max of 256 requests","startedAt":"2005-10-05T19:13:00Z","lastAt":"2005-10-06T20:16:51Z","messageCount":12,"participants":["Andy Isaacson","Junio C Hamano","Vincent Hanquez","Linus Torvalds","Alex Riesen"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"9719","messageId":"20051005191300.GC17475@hexapodia.org","threadId":"2023","inReplyTo":null,"subject":"clone: I'm only doing a max of 256 requests","fromName":"Andy Isaacson","fromEmail":"adi@hexapodia.org","sentAt":"2005-10-05T19:13:00Z","receivedAt":"2005-10-05T19:13:00Z","isPatch":false,"sender":{"key":"adi@hexapodia.org","avatar":null},"body":"Trying to do a local clone of the linux-mips.org git repo:\n\n% git clone /home/adi/linux/git/lmo/linux foo\ndefaulting to local storage area\nfatal: I'm only doing a max of 256 requests\n% git -v\ngit version 0.99.8.GIT\n\nI got git/lmo/linux from http://www.linux-mips.org/pub/scm/linux.git.\n\nAm I doing something wrong, or what? (And how should I be starting to\ndebug this? The git programs don't seem to have a useful --verbose\noption. It would be nice if \"git -v clone\" would tell me what it is\ndoing.)\n\n-andy\n"},{"id":"9721","messageId":"7virwbu4wz.fsf@assigned-by-dhcp.cox.net","threadId":"2023","inReplyTo":"20051005191300.GC17475@hexapodia.org","subject":"Re: clone: I'm only doing a max of 256 requests","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2005-10-05T19:42:20Z","receivedAt":"2005-10-05T19:42:20Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Andy Isaacson writes:\n\n> ... (And how should I be starting to\n> debug this? The git programs don't seem to have a useful --verbose\n> option. It would be nice if \"git -v clone\" would tell me what it is\n> doing.)\n\n$ git grep -n 'max of .* requests'\nupload-pack.c:141:\t\t\tdie(\"I'm only doing a max of %d requests\", MAX_NEEDS);\n\nI suspect that the repository you are cloning has too many\nbranch heads and tags under .git/refs/.\n"},{"id":"9725","messageId":"7vhdbvk6ln.fsf@assigned-by-dhcp.cox.net","threadId":"2023","inReplyTo":"7virwbu4wz.fsf@assigned-by-dhcp.cox.net","subject":"Re: clone: I'm only doing a max of 256 requests","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2005-10-05T21:16:04Z","receivedAt":"2005-10-05T21:16:04Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n> Andy Isaacson writes:\n>\n>> ... (And how should I be starting to\n>> debug this? The git programs don't seem to have a useful --verbose\n>> option. It would be nice if \"git -v clone\" would tell me what it is\n>> doing.)\n>\n> $ git grep -n 'max of .* requests'\n> upload-pack.c:141:\t\t\tdie(\"I'm only doing a max of %d requests\", MAX_NEEDS);\n>\n> I suspect that the repository you are cloning has too many\n> branch heads and tags under .git/refs/.\n\nWe can do three things, the first two being short term, the last\none a bit longer term.\n\n1. As a stop gap measure, so that your Linux kernel work can\n continue, please bump MAX_NEEDS definition in upload-pack.c\n from 256 to a bit higher. That controls the number of\n 40-letter SHA1 given to underlying rev-list via execvp(), so\n it cannot be _too_ big like 1M, lest it exceeds the exec\n argument buffer limit.\n\n2. We can add '--all' flag to git-rev-list, and have upload-pack\n use it instead, when it sees more than MAX_NEEDS refs. I\n have a patch to do this that I am currently testing.\n\n3. In addition, upload-pack should probably be taught to detect\n \"I do not have anything. Please give me objects reachable\n from all your refs\" requests, and cache the resulting pack\n somewhere (invalidate whenever any ref changes), so that next\n 'clone' request can just send it out instead of rerunning\n rev-list and pack-objects.\n"},{"id":"9727","messageId":"7vy857iqzh.fsf@assigned-by-dhcp.cox.net","threadId":"2023","inReplyTo":"7vhdbvk6ln.fsf@assigned-by-dhcp.cox.net","subject":"Re: clone: I'm only doing a max of 256 requests","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2005-10-05T21:38:42Z","receivedAt":"2005-10-05T21:38:42Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n> 1. As a stop gap measure, so that your Linux kernel work can\n> continue, please bump MAX_NEEDS definition in upload-pack.c\n> from 256 to a bit higher. That controls the number of\n> 40-letter SHA1 given to underlying rev-list via execvp(), so\n> it cannot be _too_ big like 1M, lest it exceeds the exec\n> argument buffer limit.\n\nHmph. I was reading linux-2.6/fs/exec.c::copy_strings(), but I\ndo not see any such size limit (other than exceeding the total\nmachine memory size, probably reported by alloc_page() failing)\nimposed there. Am I looking at the wrong place?\n"},{"id":"9730","messageId":"7vslvfiqe2.fsf@assigned-by-dhcp.cox.net","threadId":"2023","inReplyTo":"20051005191300.GC17475@hexapodia.org","subject":"[PATCH] upload-pack: Do not choke on too many heads request.","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2005-10-05T21:51:33Z","receivedAt":"2005-10-05T21:51:33Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Cloning from a repository with more than 256 refs (heads and tags\nincluded) will choke, because upload-pack has a built-in limit of\nfeeding not more than MAX_NEEDS (currently 256) heads to underlying\ngit-rev-list. This is a problem when cloning a repository with many\ntags, like http://www.linux-mips.org/pub/scm/linux.git, which has 290+\ntags.\n\nThis commit introduces a new flag, --all, to git-rev-list, to include\nall refs in the repository. Updated upload-pack detects requests that\nask more than MAX_NEEDS refs, and sends everything back instead.\n\nWe may probably want to tweak the definitions of MAX_NEEDS and\nMAX_HAS, but that is a separate topic.\n\nSigned-off-by: Junio C Hamano \n\n---\n\n Andy Isaacson writes:\n\n > Trying to do a local clone of the linux-mips.org git repo:\n >\n > % git clone /home/adi/linux/git/lmo/linux foo\n > defaulting to local storage area\n > fatal: I'm only doing a max of 256 requests\n > % git -v\n > git version 0.99.8.GIT\n >\n > I got git/lmo/linux from http://www.linux-mips.org/pub/scm/linux.git.\n >\n > Am I doing something wrong, or what?\n\n You are not doing anything wrong. Please try this patch.\n\n rev-list.c | 21 +++++++++++++++++++++\n rev-parse.c | 1 +\n upload-pack.c | 50 ++++++++++++++++++++++++++++++++++----------------\n 3 files changed, 56 insertions(+), 16 deletions(-)\n\napplies-to: dc721a63b8221995616e3013de11e71d94da01ef\ne091eb93258f05a58bc5d1c60f058f5f57dd92b6\ndiff --git a/rev-list.c b/rev-list.c\n--- a/rev-list.c\n+++ b/rev-list.c\n@@ -1,4 +1,5 @@\n #include \"cache.h\"\n+#include \"refs.h\"\n #include \"tag.h\"\n #include \"commit.h\"\n #include \"tree.h\"\n@@ -489,6 +490,22 @@ static void handle_one_commit(struct com\n \tcommit_list_insert(com, lst);\n }\n \n+/* for_each_ref() callback does not allow user data -- Yuck. */\n+static struct commit_list **global_lst;\n+\n+static int include_one_commit(const char *path, const unsigned char *sha1)\n+{\n+\tstruct commit *com = get_commit_reference(path, 0);\n+\thandle_one_commit(com, global_lst);\n+\treturn 0;\n+}\n+\n+static void handle_all(struct commit_list **lst)\n+{\n+\tglobal_lst = lst;\n+\tfor_each_ref(include_one_commit);\n+\tglobal_lst = NULL;\n+}\n \n int main(int argc, char **argv)\n {\n@@ -542,6 +559,10 @@ int main(int argc, char **argv)\n \t\t\tbisect_list = 1;\n \t\t\tcontinue;\n \t\t}\n+\t\tif (!strcmp(arg, \"--all\")) {\n+\t\t\thandle_all(&list);\n+\t\t\tcontinue;\n+\t\t}\n \t\tif (!strcmp(arg, \"--objects\")) {\n \t\t\ttag_objects = 1;\n \t\t\ttree_objects = 1;\ndiff --git a/rev-parse.c b/rev-parse.c\n--- a/rev-parse.c\n+++ b/rev-parse.c\n@@ -32,6 +32,7 @@ static int revs_count = 0;\n static int is_rev_argument(const char *arg)\n {\n \tstatic const char *rev_args[] = {\n+\t\t\"--all\",\n \t\t\"--bisect\",\n \t\t\"--header\",\n \t\t\"--max-age=\",\ndiff --git a/upload-pack.c b/upload-pack.c\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -30,10 +30,18 @@ static void create_pack_file(void)\n \n \tif (!pid) {\n \t\tint i;\n-\t\tint args = nr_has + nr_needs + 5;\n-\t\tchar **argv = xmalloc(args * sizeof(char *));\n-\t\tchar *buf = xmalloc(args * 45);\n-\t\tchar **p = argv;\n+\t\tint args;\n+\t\tchar **argv;\n+\t\tchar *buf;\n+\t\tchar **p;\n+\n+\t\tif (MAX_NEEDS <= nr_needs)\n+\t\t\targs = nr_has + 10;\n+\t\telse\n+\t\t\targs = nr_has + nr_needs + 5;\n+\t\targv = xmalloc(args * sizeof(char *));\n+\t\tbuf = xmalloc(args * 45);\n+\t\tp = argv;\n \n \t\tdup2(fd[1], 1);\n \t\tclose(0);\n@@ -41,10 +49,14 @@ static void create_pack_file(void)\n \t\tclose(fd[1]);\n \t\t*p++ = \"git-rev-list\";\n \t\t*p++ = \"--objects\";\n-\t\tfor (i = 0; i < nr_needs; i++) {\n-\t\t\t*p++ = buf;\n-\t\t\tmemcpy(buf, sha1_to_hex(needs_sha1[i]), 41);\n-\t\t\tbuf += 41;\n+\t\tif (MAX_NEEDS <= nr_needs)\n+\t\t\t*p++ = \"--all\";\n+\t\telse {\n+\t\t\tfor (i = 0; i < nr_needs; i++) {\n+\t\t\t\t*p++ = buf;\n+\t\t\t\tmemcpy(buf, sha1_to_hex(needs_sha1[i]), 41);\n+\t\t\t\tbuf += 41;\n+\t\t\t}\n \t\t}\n \t\tfor (i = 0; i < nr_has; i++) {\n \t\t\t*p++ = buf;\n@@ -129,18 +141,24 @@ static int receive_needs(void)\n \n \tneeds = 0;\n \tfor (;;) {\n+\t\tunsigned char dummy[20], *sha1_buf;\n \t\tlen = packet_read_line(0, line, sizeof(line));\n \t\tif (!len)\n \t\t\treturn needs;\n \n-\t\t/*\n-\t\t * This is purely theoretical right now: git-fetch-pack only\n-\t\t * ever asks for a single HEAD\n-\t\t */\n-\t\tif (needs >= MAX_NEEDS)\n-\t\t\tdie(\"I'm only doing a max of %d requests\", MAX_NEEDS);\n-\t\tif (strncmp(\"want \", line, 5) || get_sha1_hex(line+5, needs_sha1[needs]))\n-\t\t\tdie(\"git-upload-pack: protocol error, expected to get sha, not '%s'\", line);\n+\t\tsha1_buf = dummy;\n+\t\tif (needs == MAX_NEEDS) {\n+\t\t\tfprintf(stderr,\n+\t\t\t\t\"warning: supporting only a max of %d requests. \"\n+\t\t\t\t\"sending everything instead.\\n\",\n+\t\t\t\tMAX_NEEDS);\n+\t\t}\n+\t\telse if (needs < MAX_NEEDS)\n+\t\t\tsha1_buf = needs_sha1[needs];\n+\n+\t\tif (strncmp(\"want \", line, 5) || get_sha1_hex(line+5, sha1_buf))\n+\t\t\tdie(\"git-upload-pack: protocol error, \"\n+\t\t\t \"expected to get sha, not '%s'\", line);\n \t\tneeds++;\n \t}\n }\n---\n0.99.8.GIT\n"},{"id":"9734","messageId":"20051005222730.GA16593@snarc.org","threadId":"2023","inReplyTo":"7vy857iqzh.fsf@assigned-by-dhcp.cox.net","subject":"Re: clone: I'm only doing a max of 256 requests","fromName":"Vincent Hanquez","fromEmail":"vincent@snarc.org","sentAt":"2005-10-05T22:27:31Z","receivedAt":"2005-10-05T22:27:31Z","isPatch":false,"sender":{"key":"vincent@snarc.org","avatar":null},"body":"On Wed, Oct 05, 2005 at 02:38:42PM -0700, Junio C Hamano wrote:\n> Hmph. I was reading linux-2.6/fs/exec.c::copy_strings(), but I\n> do not see any such size limit (other than exceeding the total\n> machine memory size, probably reported by alloc_page() failing)\n> imposed there. Am I looking at the wrong place?\n\nwell at least the len of argv is limited by ~32K (i386) by: \n\nbprm->p = PAGE_SIZE*MAX_ARG_PAGES-sizeof(void *);\n...\nbprm->argc = count(argv, bprm->p / sizeof(void *));\n\n-- \nVincent Hanquez\n"},{"id":"9735","messageId":"Pine.LNX.4.64.0510051541300.31407@g5.osdl.org","threadId":"2023","inReplyTo":"20051005191300.GC17475@hexapodia.org","subject":"Re: clone: I'm only doing a max of 256 requests","fromName":"Linus Torvalds","fromEmail":"torvalds@osdl.org","sentAt":"2005-10-05T22:45:15Z","receivedAt":"2005-10-05T22:45:15Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Wed, 5 Oct 2005, Andy Isaacson wrote:\n>\n> Trying to do a local clone of the linux-mips.org git repo:\n> \n> % git clone /home/adi/linux/git/lmo/linux foo\n> defaulting to local storage area\n> fatal: I'm only doing a max of 256 requests\n\nThe pack upload has a totally arbitrary limit of 256 heads.\n\n> I got git/lmo/linux from http://www.linux-mips.org/pub/scm/linux.git.\n\nHeh. And:\n\n\tgit ls-remote http://www.linux-mips.org/pub/scm/linux.git | wc -l\n\nreturns \"295\". Seems to have all the old bk history in it.\n\n> Am I doing something wrong, or what?\n\nNo, just change the \"MAX_NEEDS\" define from 256 to some larger value.\n\nThere's no real reason for the limit, except that maybe we should have \nsome dynamic allocation for this.\n\n\t\tLinus\n"},{"id":"9736","messageId":"Pine.LNX.4.64.0510051547090.31407@g5.osdl.org","threadId":"2023","inReplyTo":"7vy857iqzh.fsf@assigned-by-dhcp.cox.net","subject":"Re: clone: I'm only doing a max of 256 requests","fromName":"Linus Torvalds","fromEmail":"torvalds@osdl.org","sentAt":"2005-10-05T22:48:31Z","receivedAt":"2005-10-05T22:48:31Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Wed, 5 Oct 2005, Junio C Hamano wrote:\n> \n> Hmph. I was reading linux-2.6/fs/exec.c::copy_strings(), but I\n> do not see any such size limit (other than exceeding the total\n> machine memory size, probably reported by alloc_page() failing)\n> imposed there. Am I looking at the wrong place?\n\nLook for \"MAX_ARG_PAGES\".\n\nIe the limit is about 128kB by default (32 pages). Note that that includes \nnot just arguments, but environment.\n\n\t\tLinus\n"},{"id":"9742","messageId":"7vek6zedea.fsf@assigned-by-dhcp.cox.net","threadId":"2023","inReplyTo":"Pine.LNX.4.64.0510051541300.31407@g5.osdl.org","subject":"Re: clone: I'm only doing a max of 256 requests","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2005-10-05T23:45:49Z","receivedAt":"2005-10-05T23:45:49Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Linus Torvalds writes:\n\n> No, just change the \"MAX_NEEDS\" define from 256 to some larger value.\n>\n> There's no real reason for the limit, except that maybe we should have \n> some dynamic allocation for this.\n\nIf somebody is asking for more than say 20 refs, even if the\nrepository is mature and has 1000 point releases tagged, it\nmight not make that much of a difference if we ship everything\nback instead of being selective, especially when the downloader\nsaid \"I do not have anything\", i.e. initial cloning.\n\nSo after the 'rev-list --all' patch, I was actually going to\nsuggest reducing MAX_NEEDS, to say 47 (another arbitrary\nnumber), and maybe making MAX_HAS side dynamic to hold more refs\nfor the stop list.\n\nAlso it may be worthwhile to teach upload-pack.c::got_sha1() to\nnotice when the other side says he has one object and we know\nthat object is reachable from another object he already said he\nhas, and choose not to use the older object on the has_sha1[]\nlist. The \"have\" list from fetch-pack tends to come from newer\nto older, so this would save has_sha1[] array entries from being\nconsumed by older commits when we know about the commits he has\nnear the tip of the same branch.\n"},{"id":"9762","messageId":"81b0412b0510060641g3a3c6e02m6827370dd61b5ea6@mail.gmail.com","threadId":"2023","inReplyTo":"Pine.LNX.4.64.0510051541300.31407@g5.osdl.org","subject":"Re: clone: I'm only doing a max of 256 requests","fromName":"Alex Riesen","fromEmail":"raa.lkml@gmail.com","sentAt":"2005-10-06T13:41:29Z","receivedAt":"2005-10-06T13:41:29Z","isPatch":false,"sender":{"key":"raa.lkml@gmail.com","avatar":"https://avatars.githubusercontent.com/u/324101?v=4"},"body":"On 10/6/05, Linus Torvalds wrote:\n>\n> git ls-remote http://www.linux-mips.org/pub/scm/linux.git | wc -l\n>\n\nWhich, btw, failed for me, when I tried to run it home (which has no\n.git in it yet).\nDo the scripts git-ls-remote.sh and git-parse-remote.sh really need .git/...?\n\nJust in case they don't, the attached patch removes the die(\"Not a git archive\")\n"},{"id":"9763","messageId":"Pine.LNX.4.64.0510060737160.31407@g5.osdl.org","threadId":"2023","inReplyTo":"81b0412b0510060641g3a3c6e02m6827370dd61b5ea6@mail.gmail.com","subject":"Re: clone: I'm only doing a max of 256 requests","fromName":"Linus Torvalds","fromEmail":"torvalds@osdl.org","sentAt":"2005-10-06T14:39:02Z","receivedAt":"2005-10-06T14:39:02Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Thu, 6 Oct 2005, Alex Riesen wrote:\n> On 10/6/05, Linus Torvalds wrote:\n> >\n> > git ls-remote http://www.linux-mips.org/pub/scm/linux.git | wc -l\n> >\n> \n> Which, btw, failed for me, when I tried to run it home (which has no \n> .git in it yet). Do the scripts git-ls-remote.sh and git-parse-remote.sh \n> really need .git/...?\n\nGood point. No they don't. You should be able to run \"git ls-remote\" \noutside of a local git directory.\n\n> Just in case they don't, the attached patch removes the die(\"Not a git archive\")\n\nJunio, please apply.\n\n(It still wants the \"git-sh-setup\" part if only because it uses \"die()\" in \nanother place).\n\n\t\tLinus\n"},{"id":"9774","messageId":"7v1x2y9z9o.fsf@assigned-by-dhcp.cox.net","threadId":"2023","inReplyTo":"Pine.LNX.4.64.0510060737160.31407@g5.osdl.org","subject":"Re: clone: I'm only doing a max of 256 requests","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2005-10-06T20:16:51Z","receivedAt":"2005-10-06T20:16:51Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Linus Torvalds writes:\n\n> Junio, please apply.\n\nI've been considering about this myself for quite a while, but\nhaven't done so only because I suspected the removal of the\nchecks are probably not good enough. I'll apply the patch, and\nwe will see what happens.\n"}]}