Re: Argument list too long when fetching many missing objects from partial clone

From: brian m. carlson <sandals@crustytoothpaste.net>
Date: Mar 3, 2021, 00:42 UTC
Message-ID: <YD7bczBsIR5rkqfc@camp.crustytoothpaste.net>
In-Reply-To: <f185e87e-d8d2-3ac4-9ca1-7b96ffb30562@gmail.com>

[Note: we generally prefer inline responses.]

On 2021-03-02 at 06:55:27, Bagas Sanjaya wrote:

> After trying your suggestion, I now get:
> 
> error: RPC failed; HTTP 413 curl 22 The requested URL returned error: 413
> fatal: unable to write request to remote: Broken pipe

Hmmm, this means that your local Git server is unhappy with the size of the request. That means your request is probably very large, and it's likely that it is kindly asking you not to DoS it.

Show 15 quoted lines

> On 02/03/21 02.22, Bryan Turner wrote:
> > On Mon, Mar 1, 2021 at 5:20 AM Bagas Sanjaya <bagasdotme@gmail.com> wrote:
> > > Inside the partial clone, I tried to convert it to full clone. First, I gather
> > > list of missing objects:
> > > 
> > >      git rev-list --objects --all --missing=print | grep -oP '^\?\K\w+' > .git/missing.list
> > > 
> > > Then I fetched those:
> > > 
> > >      git fetch origin $(cat .git/missing.list)
> > 
> > Since you're on Git 2.30, instead of trying to place all the object
> > IDs on the `git fetch` command line, have you tried adding `--stdin`
> > and having it read from your `missing.list` directly? Something like
> > `git fetch --stdin origin <.git/missing.list` might do what you need.

You may want to try this:

  xargs git fetch origin < .git/missing.list

That will batch the requests to fit on the command line, which will probably also have the pleasant side effect that it makes each individual request smaller to avoid the 413.

-- 
brian m. carlson (he/him or they/them)
Houston, Texas, US