Re: [RFC/WIP PATCH] object store classification

From: Stefan Beller <sbeller@google.com>
Date: Jul 11, 2017, 19:49 UTC
Message-ID: <CAGZ79kaCXLtauL5UNT-8gCpzs5A+LU6Ed41+_LdBO7zDyW1Dyg@mail.gmail.com>
In-Reply-To: <xmqqd1963j84.fsf@gitster.mtv.corp.google.com>

On Tue, Jul 11, 2017 at 11:46 AM, Junio C Hamano <gitster@pobox.com> wrote:

Show 20 quoted lines

> Brandon Williams <bmwill@google.com> writes:
>
>>>   the_repository -> the_object_store
>>>   but the object store is a complex beast having different hash tables
>>>   for the different alternates.
>>
>> After looking at the patch and some of the comments here I think that
>> this is probably the best approach with a few tweaks (which may be
>> completely unfounded because I'm not familiar with all the object store
>> code).
>>
>> In an OO world I would envision a single object (let's say 'struct
>> object_store') which is responsible for managing a repository's objects
>> no matter where the individual objects came from (main object store or
>> an alternate for that repository).  And if I understand correctly the
>> single hash table that exists now caches objects like this.
>
> I would say that conceptually an object-store object has an
> interface to answer "give me info on the object I can refer to with
> this object name".

ok.

> At the implementation level, it should have a
> hashtable (lazily populated) for all the objects in a single
> $GIT_OBJECT_DIRECTORY, grafts/replace info, and a set of pointers to
> other object-store instances that are its alternate object stores.

So one repository has one or more object stores?

I would expect that most of the time the question from above "give me info on the object I can refer to with this object name" is asked with the additional information: "and I know it is in this repository", so we rather want to have

  lookup_object(struct *repo, char *name);

instead of

  lookup_object(struct *object_store, char *name);

because the caller most likely would not care if the object comes from the alternate or from the main object store? (There may be special cases in e.g. repacking/gc where we want to instruct the repacker to only repack the main object store, ignoring any alternates or such, but any other command would not care, AFAICT)

So we could have the convenience function

    lookup_object(struct *repo, char *name)
    {
        foreach_objectstore(repo, lookup_object_in_single_store, object)
            if (object)
                return object;
        return NULL;
    }

but such code is related to storing objects, that I would prefer to see it in the object store (object.{h,c}) instead of the repository code. Also my initial plan was to have all objects (including from any alternate) in a single hashmap per repository as that seems to be most efficient assuming all alternates fit into memory.

This whole object store object orientation is only started to not have the memory pressure from lots of submodule objects polluting the main object store. When we have its own hashmap for each alternate our performance would degrade with the number of alternates. (Assuming the hypothetical worst case of one alternate per object, then the lookup time would be linear in time given the above function, I wonder if there are users that heavily use lots of alternates such that this performance characteristics would be measurable in the code to be written)

> You'd need to have a mechanism to avoid creating cycles in these
> pointers, of course.

This is another complication with many hashmaps (=object stores). In the future, is it likely that we would want to drop an alternate store from within a running process? If so we'd want to have hashmaps per alternate, otherwise I only see disadvantages for more than one hashmap (-> more than one object store) per repository. And with that said, I think we can make a wrapper struct object_store, that would encapsulate all needed variables.

+       struct object_store {
+               struct object **obj_hash;
+               int nr_objs, obj_hash_size;
+       } objects;

But instead of defining it at the repository, we'd rather define it in object.h.

Stefan