Re: [PATCH v3 0/4] multi-pack-index: fix verify on large repos

From: Jeff Hostetler <git@jeffhostetler.com>
Date: Mar 25, 2019, 17:18 UTC
Message-ID: <2eaa3617-326d-4f26-6bc0-7192a12ab192@jeffhostetler.com>
In-Reply-To: <xmqqy357pizb.fsf@gitster-ct.c.googlers.com>

On 3/22/2019 1:37 AM, Junio C Hamano wrote:

Show 24 quoted lines

> "Jeff Hostetler via GitGitGadget" <gitgitgadget@gmail.com> writes:
> 
>> Teach "multi-pack-index verify" to handle cases where the number of
>> packfiles exceeds the open file handle limit.
>>
>> The first commit fixes a problem that prevented the LRU-style
>> close_one_pack() mechanism from working which caused midx verify to run out
>> of file descriptors.
>>
>> The second commit teaches midx verify to sort the set of objects to verify
>> by packfile rather than verifying them in OID order. This eliminates the
>> need to have more than one packfile/idx open at the same time.
>>
>> With the second commit, runtime on 3600 packfiles went from 12 minutes to 25
>> seconds.
> 
> These reference to the first and second commit might have become
> stale across interations, but logically it makes sense---the first
> point is about correctness (i.e. do not die by running out of fds)
> and the second one is about usable-performance.
> 
> But in this round (possibly in the previous one, too?) the "group
> objects by packfile" one addresses both points?
>

Sorry, I forgot to remote the stale content in the cover letter for the V3 version.

This version just has the sorting by packfile commit and because it only keeps 1 packfile open at a time, it does not need the change to add packfiles to the packed-git list because it does not trigger the close_one_pack() problem.

We suspect there are other places (not-yet-observed) where the design of the all-packs and packed-git lists will lead to similar fd exhaustion errors and want to fix it properly in the packfile and/or midx code. We'll address this potential problem in a future patch series.

Jeff