Re: [PATCH] fast-import: implement --min-pack-size parameter

From: Jeff King <peff@peff.net>
Date: Apr 23, 2016, 03:13 UTC
Message-ID: <20160423031313.GA15043@sigill.intra.peff.net>
In-Reply-To: <20160423024225.GA4293@dcvr.yhbt.net>

On Sat, Apr 23, 2016 at 02:42:25AM +0000, Eric Wong wrote:

Show 16 quoted lines

> With many incremental imports, small packs become highly
> inefficient due to the need to readdir scan and load many
> indices to locate even a single object.  Frequent repacking and
> consolidation may be prohibitively expensive in terms of disk
> I/O, especially in large repositories where the initial packs
> were aggressively optimized and marked with .keep files.
> 
> In those cases, users may be better served with loose objects
> and relying on "git gc --auto".
> 
> Signed-off-by: Eric Wong <normalperson@yhbt.net>
> ---
>   There should be a matching config file directive, but I'm
>   not sure how/if it should affect other commands.  So I'm
>   not sure if it should be "pack.packSizeMin" or
>   "fastimport.packSizeMin" or something else.

This same concept exists for pushing/fetching, but there we measure it not in bytes but by the number of objects. Which is probably a better measure. A single 10MB blob is better as a loose object than as a pack, but a thousand 10KB blobs should be a pack.

There we have fetch.unpackLimit and receive.unpackLimit for the two operations, plus transfer.unpackLimit to control both of them. This doesn't necessarily need to be tied to that config, but you could certainly consider it in the same boat. It's a way of transferring a load of objects into the repository.

So it would make some sense to me to have fastimport.unpackLimit, falling back to transfer.unpackLimit.

-Peff