Re: [RFC PATCH 01/18] docs: Multi-Pack Index (MIDX) Design Notes

From: Jonathan Tan <jonathantanmy@google.com>
Date: Jan 8, 2018, 22:06 UTC
Message-ID: <20180108140631.1205a837baab24cb0fe021bc@google.com>
In-Reply-To: <4d7a1fb2-84ca-6bf9-811c-29ad21b4c5a6@gmail.com>

On Mon, 8 Jan 2018 15:35:59 -0500 Derrick Stolee <stolee@gmail.com> wrote:

Show 11 quoted lines

> Thanks! That is certainly the idea. If you know about MIDX, then you can 
> benefit from it. If you do not, then you have all the same data 
> available to you do to your work. Having a MIDX file will not break 
> other tools (libgit2, JGit, etc.).
> 
> One thing I'd like to determine before this patch goes to v1 is how much 
> we should make the other packfile-aware commands also midx-aware. My gut 
> reaction right now is to have git-repack call 'git midx --clear' if 
> core.midx=true and a packfile was deleted. However, this could easily be 
> changed with 'git midx --clear' followed by 'git midx --write 
> --update-head' if midx-head exists.

My opinion is that these are sufficient:
 (a) functionality to create a .midx file from scratch (deleting any
     existing ones)
 (b) either:
     - update of packfile.c to read (one or more) midx files (like in
       patch 18), and possibly usage in a benchmark, or
     - any other usage of midx file (e.g. abbreviations, like in patch
       17)

In general, a way to create them (so that they can be used from a cronjob, etc.), and a way to consume them to show that the new way works and is indeed faster. This functionality in itself might be sufficient to merge in - this would already be useful in situations where it is undesirable for repacks to happen often, and we can "bridge" the intervals between repacks using a cronjob that periodically generates .midx files in order to keep up the object lookup performance.

In particular, I think that it is fine to omit a more sophisticated "repack" for now - the .midx mechanism must tolerate packs referenced by .midx files suddenly disappearing anyway, and in this way, at least we can demonstrate that the .midx mechanism still works in this case.