From: Johan Herland Date: Mon, 16 May 2011 01:39:25 GMT Subject: Re: [PATCHv3 3/9] pack-objects: Allow --max-pack-size to be used together with --stdout Message-ID: <201105160339.25737.johan@herland.net> In-Reply-To: On Monday 16 May 2011, Shawn Pearce wrote: > On Sun, May 15, 2011 at 14:37, Johan Herland wrote: > > Currently we refuse combining --max-pack-size with --stdout since > > there's no way to make multiple packs when the pack is written to > > stdout. However, we want to be able to limit the maximum size of the > > pack created by --stdout (and abort pack-objects if we are unable to > > meet that limit). > > > > Therefore, when used together with --stdout, we reinterpret > > --max-pack-size to indicate the maximum pack size which - if exceeded > > - will cause pack-objects to abort with an error message. > > ... > > > if (pack_to_stdout) { > > + if (nr_written != nr_remaining) > > + die("unable to make pack within the pack size" > > + " limit (%lu bytes)", pack_size_limit); > > I think this is too late. We have already output a bunch of data, up > to the size limit at this point. If the size limit is non-trivial > (e.g. 5 MB) we have already sent most of that to the remote side, and > its already written some of that out to disk. > > I'd like this to be a soft limit derived from the reused object sizes. > When planning the pack by looking at where we will reuse an object > from, sum those sizes. If the sum of these sizes would break this > limit, then we abort before even writing the pack header out. I agree, but it's currently late Sunday (early Monday), and after looking at this for a while, I'm no longer thinking straight. If someone that groks the pack-objects internal could help out, I'd be really grateful. AFAICS, we need to drill into prepare_pack() to find the details needed to estimate the total pack size, but I don't know exactly which data structure(s) holds the data needed. We probably need to accumulate a pack size estimate in find_deltas(), and then sum those across the threads, before we finally compare the total estimate to pack_size_limit prior to calling write_pack_file(). ...Johan -- Johan Herland, www.herland.net