Re: RFC: Design and code of partial clones (now, missing commits and trees OK) (part 3)

From: Jeff Hostetler <git@jeffhostetler.com>
Date: Sep 26, 2017, 14:25 UTC
Message-ID: <e05a7978-c312-9e26-79a2-fca3ae44f59a@jeffhostetler.com>
In-Reply-To: <20170922155802.ab79717818578a23cc31f6fe@google.com>

On 9/22/2017 6:58 PM, Jonathan Tan wrote:

Show 16 quoted lines

> On Fri, 22 Sep 2017 17:32:00 -0400
> Jeff Hostetler <git@jeffhostetler.com> wrote:
> 
>> I guess I'm afraid that the first call to is_promised() is going
>> cause a very long pause as it loads up a very large hash of objects.
> 
> Yes, the first call will cause a long pause. (I think fsck and gc can
> tolerate this, but a better solution is appreciated.)
> 
>> Perhaps you could augment the OID lookup to remember where the object
>> was found (essentially a .promisor bit set).  Then you wouldn't need
>> to touch them all.
> 
> Sorry - I don't understand this. Are you saying that missing promisor
> objects should go into the global object hashtable, so that we can set a
> flag on them?

I just meant could we add a bit to "struct object_info" to indicate that the object was found in a .promisor packfile ? This could be set in sha1_object_info_extended().

Then the is_promised() calls in fsck and gc would just test that bit.

Given that that bit will be set on promisOR objects (and we won't have object_info for missing objects), you may need to adjust the iterator in the fsck/gc code slightly.

This is a bit of a handwave, but could something like that eliminate the need to build this oidset?

Show 12 quoted lines

> 
>>> The oidset will deduplicate OIDs.
>>
>> Right, but you still have an entry for each object.  For a repo the
>> size of Windows, you may have 25M+ objects your copy of the ODB.
> 
> We have entries only for the "frontier" objects (the objects directly
> referenced by any promisor object). For the Windows repo, for example, I
> foresee that many of the blobs, trees, and commits will be "hiding"
> behind objects that the repository user did not download into their
> repo.
>