Re: clone: I'm only doing a max of 256 requests

From: Junio C Hamano <junkio@cox.net>
Date: Oct 5, 2005, 21:16 UTC
Message-ID: <7vhdbvk6ln.fsf@assigned-by-dhcp.cox.net>
In-Reply-To: <7virwbu4wz.fsf@assigned-by-dhcp.cox.net>

Junio C Hamano <junkio@cox.net> writes:

Show 12 quoted lines

> Andy Isaacson <adi@hexapodia.org> writes:
>
>> ...  (And how should I be starting to
>> debug this?  The git programs don't seem to have a useful --verbose
>> option.  It would be nice if "git -v clone" would tell me what it is
>> doing.)
>
> $ git grep -n 'max of .* requests'
> upload-pack.c:141:			die("I'm only doing a max of %d requests", MAX_NEEDS);
>
> I suspect that the repository you are cloning has too many
> branch heads and tags under .git/refs/.

We can do three things, the first two being short term, the last one a bit longer term.

1. As a stop gap measure, so that your Linux kernel work can
   continue, please bump MAX_NEEDS definition in upload-pack.c
   from 256 to a bit higher.  That controls the number of
   40-letter SHA1 given to underlying rev-list via execvp(), so
   it cannot be _too_ big like 1M, lest it exceeds the exec
   argument buffer limit.

2. We can add '--all' flag to git-rev-list, and have upload-pack
   use it instead, when it sees more than MAX_NEEDS refs.  I
   have a patch to do this that I am currently testing.

3. In addition, upload-pack should probably be taught to detect
   "I do not have anything.  Please give me objects reachable
   from all your refs" requests, and cache the resulting pack
   somewhere (invalidate whenever any ref changes), so that next
   'clone' request can just send it out instead of rerunning
   rev-list and pack-objects.