# clone: I'm only doing a max of 256 requests 12 messages from 2005-10-05 to 2005-10-06. Participants: Andy Isaacson, Junio C Hamano, Vincent Hanquez, Linus Torvalds, Alex Riesen. Thread: https://gitlist.dev/t/2023 ## Andy Isaacson, 2005-10-05 19:13 Subject: clone: I'm only doing a max of 256 requests Message-ID: <20051005191300.GC17475@hexapodia.org> URL: https://gitlist.dev/e/20051005191300.GC17475%40hexapodia.org ``` Trying to do a local clone of the linux-mips.org git repo: % git clone /home/adi/linux/git/lmo/linux foo defaulting to local storage area fatal: I'm only doing a max of 256 requests % git -v git version 0.99.8.GIT I got git/lmo/linux from http://www.linux-mips.org/pub/scm/linux.git. Am I doing something wrong, or what? (And how should I be starting to debug this? The git programs don't seem to have a useful --verbose option. It would be nice if "git -v clone" would tell me what it is doing.) -andy ``` ## Junio C Hamano, 2005-10-05 19:42 Subject: Re: clone: I'm only doing a max of 256 requests Message-ID: <7virwbu4wz.fsf@assigned-by-dhcp.cox.net> URL: https://gitlist.dev/e/7virwbu4wz.fsf%40assigned-by-dhcp.cox.net In-Reply-To: <20051005191300.GC17475@hexapodia.org> ``` Andy Isaacson writes: > ... (And how should I be starting to > debug this? The git programs don't seem to have a useful --verbose > option. It would be nice if "git -v clone" would tell me what it is > doing.) $ git grep -n 'max of .* requests' upload-pack.c:141: die("I'm only doing a max of %d requests", MAX_NEEDS); I suspect that the repository you are cloning has too many branch heads and tags under .git/refs/. ``` ## Junio C Hamano, 2005-10-05 21:16 Subject: Re: clone: I'm only doing a max of 256 requests Message-ID: <7vhdbvk6ln.fsf@assigned-by-dhcp.cox.net> URL: https://gitlist.dev/e/7vhdbvk6ln.fsf%40assigned-by-dhcp.cox.net In-Reply-To: <7virwbu4wz.fsf@assigned-by-dhcp.cox.net> ``` Junio C Hamano writes: > Andy Isaacson writes: > >> ... (And how should I be starting to >> debug this? The git programs don't seem to have a useful --verbose >> option. It would be nice if "git -v clone" would tell me what it is >> doing.) > > $ git grep -n 'max of .* requests' > upload-pack.c:141: die("I'm only doing a max of %d requests", MAX_NEEDS); > > I suspect that the repository you are cloning has too many > branch heads and tags under .git/refs/. We can do three things, the first two being short term, the last one a bit longer term. 1. As a stop gap measure, so that your Linux kernel work can continue, please bump MAX_NEEDS definition in upload-pack.c from 256 to a bit higher. That controls the number of 40-letter SHA1 given to underlying rev-list via execvp(), so it cannot be _too_ big like 1M, lest it exceeds the exec argument buffer limit. 2. We can add '--all' flag to git-rev-list, and have upload-pack use it instead, when it sees more than MAX_NEEDS refs. I have a patch to do this that I am currently testing. 3. In addition, upload-pack should probably be taught to detect "I do not have anything. Please give me objects reachable from all your refs" requests, and cache the resulting pack somewhere (invalidate whenever any ref changes), so that next 'clone' request can just send it out instead of rerunning rev-list and pack-objects. ``` ## Junio C Hamano, 2005-10-05 21:38 Subject: Re: clone: I'm only doing a max of 256 requests Message-ID: <7vy857iqzh.fsf@assigned-by-dhcp.cox.net> URL: https://gitlist.dev/e/7vy857iqzh.fsf%40assigned-by-dhcp.cox.net In-Reply-To: <7vhdbvk6ln.fsf@assigned-by-dhcp.cox.net> ``` Junio C Hamano writes: > 1. As a stop gap measure, so that your Linux kernel work can > continue, please bump MAX_NEEDS definition in upload-pack.c > from 256 to a bit higher. That controls the number of > 40-letter SHA1 given to underlying rev-list via execvp(), so > it cannot be _too_ big like 1M, lest it exceeds the exec > argument buffer limit. Hmph. I was reading linux-2.6/fs/exec.c::copy_strings(), but I do not see any such size limit (other than exceeding the total machine memory size, probably reported by alloc_page() failing) imposed there. Am I looking at the wrong place? ``` ## Junio C Hamano, 2005-10-05 21:51 Subject: [PATCH] upload-pack: Do not choke on too many heads request. Message-ID: <7vslvfiqe2.fsf@assigned-by-dhcp.cox.net> URL: https://gitlist.dev/e/7vslvfiqe2.fsf%40assigned-by-dhcp.cox.net In-Reply-To: <20051005191300.GC17475@hexapodia.org> ``` Cloning from a repository with more than 256 refs (heads and tags included) will choke, because upload-pack has a built-in limit of feeding not more than MAX_NEEDS (currently 256) heads to underlying git-rev-list. This is a problem when cloning a repository with many tags, like http://www.linux-mips.org/pub/scm/linux.git, which has 290+ tags. This commit introduces a new flag, --all, to git-rev-list, to include all refs in the repository. Updated upload-pack detects requests that ask more than MAX_NEEDS refs, and sends everything back instead. We may probably want to tweak the definitions of MAX_NEEDS and MAX_HAS, but that is a separate topic. Signed-off-by: Junio C Hamano --- Andy Isaacson writes: > Trying to do a local clone of the linux-mips.org git repo: > > % git clone /home/adi/linux/git/lmo/linux foo > defaulting to local storage area > fatal: I'm only doing a max of 256 requests > % git -v > git version 0.99.8.GIT > > I got git/lmo/linux from http://www.linux-mips.org/pub/scm/linux.git. > > Am I doing something wrong, or what? You are not doing anything wrong. Please try this patch. rev-list.c | 21 +++++++++++++++++++++ rev-parse.c | 1 + upload-pack.c | 50 ++++++++++++++++++++++++++++++++++---------------- 3 files changed, 56 insertions(+), 16 deletions(-) applies-to: dc721a63b8221995616e3013de11e71d94da01ef e091eb93258f05a58bc5d1c60f058f5f57dd92b6 diff --git a/rev-list.c b/rev-list.c --- a/rev-list.c +++ b/rev-list.c @@ -1,4 +1,5 @@ #include "cache.h" +#include "refs.h" #include "tag.h" #include "commit.h" #include "tree.h" @@ -489,6 +490,22 @@ static void handle_one_commit(struct com commit_list_insert(com, lst); } +/* for_each_ref() callback does not allow user data -- Yuck. */ +static struct commit_list **global_lst; + +static int include_one_commit(const char *path, const unsigned char *sha1) +{ + struct commit *com = get_commit_reference(path, 0); + handle_one_commit(com, global_lst); + return 0; +} + +static void handle_all(struct commit_list **lst) +{ + global_lst = lst; + for_each_ref(include_one_commit); + global_lst = NULL; +} int main(int argc, char **argv) { @@ -542,6 +559,10 @@ int main(int argc, char **argv) bisect_list = 1; continue; } + if (!strcmp(arg, "--all")) { + handle_all(&list); + continue; + } if (!strcmp(arg, "--objects")) { tag_objects = 1; tree_objects = 1; diff --git a/rev-parse.c b/rev-parse.c --- a/rev-parse.c +++ b/rev-parse.c @@ -32,6 +32,7 @@ static int revs_count = 0; static int is_rev_argument(const char *arg) { static const char *rev_args[] = { + "--all", "--bisect", "--header", "--max-age=", diff --git a/upload-pack.c b/upload-pack.c --- a/upload-pack.c +++ b/upload-pack.c @@ -30,10 +30,18 @@ static void create_pack_file(void) if (!pid) { int i; - int args = nr_has + nr_needs + 5; - char **argv = xmalloc(args * sizeof(char *)); - char *buf = xmalloc(args * 45); - char **p = argv; + int args; + char **argv; + char *buf; + char **p; + + if (MAX_NEEDS <= nr_needs) + args = nr_has + 10; + else + args = nr_has + nr_needs + 5; + argv = xmalloc(args * sizeof(char *)); + buf = xmalloc(args * 45); + p = argv; dup2(fd[1], 1); close(0); @@ -41,10 +49,14 @@ static void create_pack_file(void) close(fd[1]); *p++ = "git-rev-list"; *p++ = "--objects"; - for (i = 0; i < nr_needs; i++) { - *p++ = buf; - memcpy(buf, sha1_to_hex(needs_sha1[i]), 41); - buf += 41; + if (MAX_NEEDS <= nr_needs) + *p++ = "--all"; + else { + for (i = 0; i < nr_needs; i++) { + *p++ = buf; + memcpy(buf, sha1_to_hex(needs_sha1[i]), 41); + buf += 41; + } } for (i = 0; i < nr_has; i++) { *p++ = buf; @@ -129,18 +141,24 @@ static int receive_needs(void) needs = 0; for (;;) { + unsigned char dummy[20], *sha1_buf; len = packet_read_line(0, line, sizeof(line)); if (!len) return needs; - /* - * This is purely theoretical right now: git-fetch-pack only - * ever asks for a single HEAD - */ - if (needs >= MAX_NEEDS) - die("I'm only doing a max of %d requests", MAX_NEEDS); - if (strncmp("want ", line, 5) || get_sha1_hex(line+5, needs_sha1[needs])) - die("git-upload-pack: protocol error, expected to get sha, not '%s'", line); + sha1_buf = dummy; + if (needs == MAX_NEEDS) { + fprintf(stderr, + "warning: supporting only a max of %d requests. " + "sending everything instead.\n", + MAX_NEEDS); + } + else if (needs < MAX_NEEDS) + sha1_buf = needs_sha1[needs]; + + if (strncmp("want ", line, 5) || get_sha1_hex(line+5, sha1_buf)) + die("git-upload-pack: protocol error, " + "expected to get sha, not '%s'", line); needs++; } } --- 0.99.8.GIT ``` ## Vincent Hanquez, 2005-10-05 22:27 Subject: Re: clone: I'm only doing a max of 256 requests Message-ID: <20051005222730.GA16593@snarc.org> URL: https://gitlist.dev/e/20051005222730.GA16593%40snarc.org In-Reply-To: <7vy857iqzh.fsf@assigned-by-dhcp.cox.net> ``` On Wed, Oct 05, 2005 at 02:38:42PM -0700, Junio C Hamano wrote: > Hmph. I was reading linux-2.6/fs/exec.c::copy_strings(), but I > do not see any such size limit (other than exceeding the total > machine memory size, probably reported by alloc_page() failing) > imposed there. Am I looking at the wrong place? well at least the len of argv is limited by ~32K (i386) by: bprm->p = PAGE_SIZE*MAX_ARG_PAGES-sizeof(void *); ... bprm->argc = count(argv, bprm->p / sizeof(void *)); -- Vincent Hanquez ``` ## Linus Torvalds, 2005-10-05 22:45 Subject: Re: clone: I'm only doing a max of 256 requests Message-ID: URL: https://gitlist.dev/e/Pine.LNX.4.64.0510051541300.31407%40g5.osdl.org In-Reply-To: <20051005191300.GC17475@hexapodia.org> ``` On Wed, 5 Oct 2005, Andy Isaacson wrote: > > Trying to do a local clone of the linux-mips.org git repo: > > % git clone /home/adi/linux/git/lmo/linux foo > defaulting to local storage area > fatal: I'm only doing a max of 256 requests The pack upload has a totally arbitrary limit of 256 heads. > I got git/lmo/linux from http://www.linux-mips.org/pub/scm/linux.git. Heh. And: git ls-remote http://www.linux-mips.org/pub/scm/linux.git | wc -l returns "295". Seems to have all the old bk history in it. > Am I doing something wrong, or what? No, just change the "MAX_NEEDS" define from 256 to some larger value. There's no real reason for the limit, except that maybe we should have some dynamic allocation for this. Linus ``` ## Linus Torvalds, 2005-10-05 22:48 Subject: Re: clone: I'm only doing a max of 256 requests Message-ID: URL: https://gitlist.dev/e/Pine.LNX.4.64.0510051547090.31407%40g5.osdl.org In-Reply-To: <7vy857iqzh.fsf@assigned-by-dhcp.cox.net> ``` On Wed, 5 Oct 2005, Junio C Hamano wrote: > > Hmph. I was reading linux-2.6/fs/exec.c::copy_strings(), but I > do not see any such size limit (other than exceeding the total > machine memory size, probably reported by alloc_page() failing) > imposed there. Am I looking at the wrong place? Look for "MAX_ARG_PAGES". Ie the limit is about 128kB by default (32 pages). Note that that includes not just arguments, but environment. Linus ``` ## Junio C Hamano, 2005-10-05 23:45 Subject: Re: clone: I'm only doing a max of 256 requests Message-ID: <7vek6zedea.fsf@assigned-by-dhcp.cox.net> URL: https://gitlist.dev/e/7vek6zedea.fsf%40assigned-by-dhcp.cox.net In-Reply-To: ``` Linus Torvalds writes: > No, just change the "MAX_NEEDS" define from 256 to some larger value. > > There's no real reason for the limit, except that maybe we should have > some dynamic allocation for this. If somebody is asking for more than say 20 refs, even if the repository is mature and has 1000 point releases tagged, it might not make that much of a difference if we ship everything back instead of being selective, especially when the downloader said "I do not have anything", i.e. initial cloning. So after the 'rev-list --all' patch, I was actually going to suggest reducing MAX_NEEDS, to say 47 (another arbitrary number), and maybe making MAX_HAS side dynamic to hold more refs for the stop list. Also it may be worthwhile to teach upload-pack.c::got_sha1() to notice when the other side says he has one object and we know that object is reachable from another object he already said he has, and choose not to use the older object on the has_sha1[] list. The "have" list from fetch-pack tends to come from newer to older, so this would save has_sha1[] array entries from being consumed by older commits when we know about the commits he has near the tip of the same branch. ``` ## Alex Riesen, 2005-10-06 13:41 Subject: Re: clone: I'm only doing a max of 256 requests Message-ID: <81b0412b0510060641g3a3c6e02m6827370dd61b5ea6@mail.gmail.com> URL: https://gitlist.dev/e/81b0412b0510060641g3a3c6e02m6827370dd61b5ea6%40mail.gmail.com In-Reply-To: ``` On 10/6/05, Linus Torvalds wrote: > > git ls-remote http://www.linux-mips.org/pub/scm/linux.git | wc -l > Which, btw, failed for me, when I tried to run it home (which has no .git in it yet). Do the scripts git-ls-remote.sh and git-parse-remote.sh really need .git/...? Just in case they don't, the attached patch removes the die("Not a git archive") ``` ## Linus Torvalds, 2005-10-06 14:39 Subject: Re: clone: I'm only doing a max of 256 requests Message-ID: URL: https://gitlist.dev/e/Pine.LNX.4.64.0510060737160.31407%40g5.osdl.org In-Reply-To: <81b0412b0510060641g3a3c6e02m6827370dd61b5ea6@mail.gmail.com> ``` On Thu, 6 Oct 2005, Alex Riesen wrote: > On 10/6/05, Linus Torvalds wrote: > > > > git ls-remote http://www.linux-mips.org/pub/scm/linux.git | wc -l > > > > Which, btw, failed for me, when I tried to run it home (which has no > .git in it yet). Do the scripts git-ls-remote.sh and git-parse-remote.sh > really need .git/...? Good point. No they don't. You should be able to run "git ls-remote" outside of a local git directory. > Just in case they don't, the attached patch removes the die("Not a git archive") Junio, please apply. (It still wants the "git-sh-setup" part if only because it uses "die()" in another place). Linus ``` ## Junio C Hamano, 2005-10-06 20:16 Subject: Re: clone: I'm only doing a max of 256 requests Message-ID: <7v1x2y9z9o.fsf@assigned-by-dhcp.cox.net> URL: https://gitlist.dev/e/7v1x2y9z9o.fsf%40assigned-by-dhcp.cox.net In-Reply-To: ``` Linus Torvalds writes: > Junio, please apply. I've been considering about this myself for quite a while, but haven't done so only because I suspected the removal of the checks are probably not good enough. I'll apply the patch, and we will see what happens. ```