Re: [PATCH] pack-format: correct multi-pack-index description

From: Johannes Berg <johannes@sipsolutions.net>
Date: Feb 10, 2020, 15:06 UTC
Message-ID: <a52c8163abfba107a27b359a1588a68efdc581a8.camel@sipsolutions.net>
In-Reply-To: <08dbc3be-34a7-fb8d-e0bd-56a79ab5b65a@gmail.com>

On Mon, 2020-02-10 at 10:02 -0500, Derrick Stolee wrote:

Show 9 quoted lines

> Git loads the multi-pack-index file, which includes a sorted list of
> the packs it covers. It then scans the "pack" directory for pack-indexes
> and checks if they are covered by the multi-pack-index. If not, then
> Git will add them to the packed_git struct and use them as normal.
> The hope is that this list of "uncovered" packs is small compared to
> the data covered by the multi-pack-index.
> 
> This allows Git to continue functioning after an action like "git fetch"
> that adds a new pack but may not want to rewrite the multi-pack-index.

Ah, ok.

So then perhaps I'll just make bup write the multi-pack-index file as is. This is fine, there's no real need to have multiple, I just didn't want to have to make sure the file was always consistent.

Or maybe just call git to do it, and only be able to read the resulting file :-)

Show 12 quoted lines

> Our background maintenance essentially runs these commands:
> 
>  1. git multi-pack-index write
>  2. git multi-pack-index expire
>  3. git multi-pack-index repack
> 
> Step 1 ensures all packs are pulled into the multi-pack-index. Step 2
> deletes any pack-files whose objects are contained in newer pack-files.
> Step 3 creates a new pack-file containing all objects from a set of
> small pack-files (using the --batch-size=X option). This process helps
> incrementally reduce the size and number of packs. That may be helpful
> for your backup took, too.

I'll have to look at this in more detail later, and understand exactly what the steps do here. Evidently that modifies pack files, which I hadn't expected for a type of "index" command :-)

> Perhaps after an incremental multi-pack-index is added, then Git could
> (optionally) have a mode that only checks the multi-pack-index to
> avoid scanning the packs directory. It would require inserting a
> multi-pack-index write into the index-pack logic so Git.

I guess you'd still want to read non-covered pack files just in case old git was used or something though.

> I'm not sure if that mode would be helpful, since the pack directory
> scan is typically done once per command and is relatively fast.

Right.

Show 6 quoted lines

> > > That said: if someone wanted to contribute an incremental format,
> > > then I would be happy to review it!
> > 
> > I might still get motivated to do so :-)
> 
> YOU CAN DO IT! (Did that help?)

:-)

Thanks, johannes