Re: Strategy to deal with slow cloners

From: Eric Wong <e@80x24.org>
Date: Apr 19, 2021, 18:08 UTC
Message-ID: <20210419180803.GA10171@dcvr>
In-Reply-To: <20210419124623.wwps2s35x2mrrhi6@nitro.local>

Konstantin Ryabitsev <konstantin@linuxfoundation.org> wrote:

Show 7 quoted lines

> Hello:
> 
> I try to keep repositories routinely repacked and optimized for clones, in
> hopes that most operations needing lots of objects would be sending packs
> straight from disk. However, every now and again a client from a slow
> connection requests a large clone and then takes half a day downloading it,
> resulting in gigabytes of RAM being occupied by a temporary pack.

Yeah, I'm familiar with the problem.

> Are there any strategies to reduce RAM usage in such cases, other than
> vm.swappiness (which I'm not sure would work, since it's not a sleeping
> process)? Is there a way to write large temporary packs somewhere to disk
> before sendfile'ing them?

public-inbox-httpd actually switched buffering strategies in 2019 to favor hitting ENOSPC instead of ENOMEM :)

  https://public-inbox.org/meta/20190629195951.32160-11-e@80x24.org/

It doesn't support sendfile, currently (I didn't want separate HTTPS vs HTTP code paths), but that's probably not too big of a deal, especially with slow clients.

It's capable of serving non-public-inbox coderepos (and running cgit). Instead of configuring every [coderepo "..."] manually, publicinbox.cgitrc can be set in ~/.public-inbox/config to mass-configure [coderepo] sections. It's only lightly-tested for my setup atm, though.

Mapping publicinbox.<name>.coderepo to [coderepo "..."] entries for solver (blob reconstruction) isn't required; it's a bit of a pain at a large scale and I haven't figured out how to make it easier.