Re: [PATCH] pack-format: correct multi-pack-index description

From: Johannes Berg <johannes@sipsolutions.net>
Date: Feb 10, 2020, 14:50 UTC
Message-ID: <c077a2100038edf2b0c486c0d364bd00f3921074.camel@sipsolutions.net>
In-Reply-To: <28b6fd7f-85ea-9ef1-1977-888cdd737c6d@gmail.com>

On Mon, 2020-02-10 at 09:46 -0500, Derrick Stolee wrote:

> Part of my initial plan was to have this incremental file format.
> The commit-graph uses a very similar mechanism. The difference may
> be that you likely allow multiple .midx files found by scanning the
> pack directory,

Right, just scan and use any midx that exist, then compare the packs in there against all the packs found, and then remove any packs that actually *are* in an midx from the search list. That leaves you with all information, but optimised by midx where possible.

Show 6 quoted lines

> but I would expect something like the
> "commit-graph-chain" file that provides an ordered list of the
> incremental files. This can be important for deciding when to merge
> layers or delete old files, and would be critical to the possibility
> of converting reachability bitmaps to rely on a stable object order
> stored in the multi-pack-index instead of pack-order.

Right, if we delete then we have to also remove any midx covering the deleted pack, that's pretty rare in bup as a backup tool though.

Show 5 quoted lines

> The reason the multi-pack-index has not become incremental is that
> VFS for Git no longer needs to write it very often. We write the
> entire multi-pack-index during a background job that triggers once
> per day. If we needed to write it more frequently, then the incremental
> format would be more important to us.

So, wait, what if a new pack is created? Does it just get used in addition to the multi-pack-index, if it's not covered by it, like I described above?

If so, I guess it wouldn't actually really matter here. I was afraid (but didn't check yet) that git would always use only the single multi- pack-index file, and not also search additional packs, so that it always has to be maintained in "perfect order" ...

> That said: if someone wanted to contribute an incremental format,
> then I would be happy to review it!

I might still get motivated to do so :-)

johannes