Re: git gc and worktrees

From: Michael Haggerty <mhagger@alum.mit.edu>
Date: Jun 2, 2016, 04:08 UTC
Message-ID: <574FB126.4090805@alum.mit.edu>
In-Reply-To: <xmqqmvn4y9zq.fsf@gitster.mtv.corp.google.com>

On 06/01/2016 09:39 PM, Junio C Hamano wrote:

Show 36 quoted lines

> Michael Haggerty <mhagger@alum.mit.edu> writes:
> 
>> I argue that the fundamental concept in terms of the implementation
>> should be the individual physical reference stores, and these should be
>> compounded together to form the logical reference collections and the
>> sets of reachability roots that are interesting at the UI level.
> 
> That is very good in principle.  How does that principle translate
> to the current setup (with possible enhancement with pluggable ref
> backends) and multiple worktrees?  Let me try thinking it through
> aloud.
> 
>  * Without pluggable ref backend or worktrees, we start from two
>    "physical reference stores"; packed-refs file lists refs that
>    will be covered (overridden) by loose refs in .git/refs/.
>    Symbolic refs always being in loose falls out as a natural
>    consequence that packed-refs file does not record symrefs.
> 
>  * Throw in multiple worktrees to the mix.  How?  Do we consider
>    selected refs/ hierarchies (like refs/bisect/*) as separate
>    physical store (even though it might be backed by the files in
>    the same .git/refs/ filesystem hierarchy) and represent the
>    "logical" view as an overlay across the traditional two types of
>    physical reference stores?  That is:
> 
>    - loose refs in .git/HEAD, .git/refs/{bisect,...} for
>      per-worktree part form one physical store.  If a ref is found
>      here, that is what we use as a part of the logical view.
> 
>    - loose refs in .git/refs/{branches,tags,notes,...} for common
>      part form one physical store.  For a ref that is not found
>      above but is found here becomes a part of the logical view.
> 
>    - packed refs in .git/packed-refs is another physical store.  For
>      a ref that is not found in the above two but is found here
>      becomes a part of the logical view.

I think I would represent the logical store of a worktree repo as follows. First, I would implement a cached_ref_store that introduces a layer of caching around another ref_store. Then

    def get_files_ref_store(dir) {
        loose = create_cached_ref_store(get_loose_ref_store(dir))
        packed = create_cached_ref_store(get_packed_ref_store(dir))
        return create_files_ref_store(loose, packed)
    }

    common_ref_store = get_files_ref_store(common_dir)

    /*
     * I think we only allow loose refs in worktrees; otherwise
     * this could be an overlay_ref_store too. Actually, we might
     * want to omit the caching here.
     */
    local_ref_store = create_cached_ref_store(
            get_loose_ref_store(git_dir))

    logical_ref_store = create_worktree_ref_store(
        local_ref_store, common_ref_store)

Where worktree_ref_store does something like

    if (is_per_worktree_ref(refname))
        lookup in local_ref_store
    else
        lookup in common_ref_store

for reading, and uses a merge_ref_iterator with a select function that does something similar for iterating.

The files_ref_store would do lookups by looking first in the loose_ref_store then in the packed_ref_store, would use an overlay_ref_iterator for iteration, and would know to do all writes in the loose_ref_store (except for deletes, which also have to go to packed_ref_store). It would have a special "pack-refs" operation, specific to files_ref_store, that shuffles references between its two backends.

Writing to a worktree_ref_store is a bit tricker, because we want to allow ref_transactions to span worktree and common refs (though we probably need to give up atomicity for any such transaction). The worktree_ref_transaction_commit() method has to split the main transaction into two sub-transactions, one for each of its component ref_stores. I planned for this when designing split_under_lock and think it is possible, though I admit I haven't implemented it yet.

One nice thing about this design is that you can skip the worktree_ref_store layer and its overhead entirely for repositories that are not linked. The decision can be made once, at instantiation time, rather than every time a reference is looked up. See the pseudocode below.

Show 11 quoted lines

> Up to this point, I am all for your "separate physical stores are
> composited to give a logical view".  I can see how multi-worktree
> world view fits within that framework.
> 
>  * With pluggable ref backend, we may gain yet another "physical
>    reference store" possibility, e.g. one backed by lmdb.  If it
>    supports symrefs, a repoitory may use lmdb backed reference store
>    without the traditional two.
> 
>    But it is unclear how it would interact with the multi-worktree
>    world order.

Since you could plug-and-play different ref_stores in the above scheme, I don't see any problem here.

    def get_logical_ref_store() {
        local_ref_store = get_local_ref_store(git_dir)
        if (is_linked_repo) {
            common_ref_store = get_ref_store(common_dir)
            return worktree_ref_store(local_ref_store,
                                      common_ref_store)
        } else {
            return local_ref_store;
        }
    }

get_ref_store() would read the git config to decide what the ref store to use for the specified repository, which itself might be an lmdb_ref_store or an overlay_ref_store(loose_ref_store, packed_ref_store).

Michael