Re: Submodule object store

From: David Lang <david.lang@digitalinsight.com>
Date: Mar 26, 2007, 23:40 UTC
Message-ID: <Pine.LNX.4.63.0703261530030.14387@qynat.qvtvafvgr.pbz>
In-Reply-To: <20070326235527.GM22773@admingilde.org>

On Tue, 27 Mar 2007, Martin Waitz wrote:

Show 20 quoted lines

> hoi :)
>
> On Mon, Mar 26, 2007 at 03:20:34PM -0800, David Lang wrote:
>>> I want to be able to list all objects which are not reachable in the
>>> object store, without traversing all submodules at the same time.
>>> The only way I can think of to achieve this is to have one separate
>>> object store per submodule and then do the traversal per submodule.
>>
>> why do you want to optimize for the relativly rare fsck function rather
>> then the more common read functions (which would benifit from shareing
>> object that are identical between projects)?
>
> Because I don't know how to make it _possible_ for large repositories
> otherwise.  Consider a Linux-distribution which handles each package
> as one submodule.
>
> I don't think that it's too much balanced towards fsck.
> The separated object store also helps reduce the memory requirement for
> large pushs/pulls.
> Sharing objects can be achieved by alternates if you want.

alternates require explicitly setting up the sharing.

useing the same object store makes this work automaticaly (think of all the copies of COPYING that would end up being the same as a trivial example)

> If someone comes up with a nice way to handle everything in one big
> object store I would happily use that! :-)

what exactly are the problems with one big object store?

ones that I can think of:

1. when you are doing a fsck you need to walk all the trees and find out the 
list of objects that you know about.

   done as a tree of binary values you can hold a LOT in memory before running 
into swap.

   if it's enough larger then available ram then an option for fsck to use trees 
on disk is an option.

2. when creating a pack you will eventually run into pack-size limits with too 
many objects

   teach the pack creators to make packs that are subsets rather then everything 
(I belive that most of the smarts are there, it just needs the upper control 
logic to tell the existing things what to include)

3. when doing a pull it takes longer to figure out what to pull to get a 
duplicate of _everything_

   add a way to do a 'pull projectlist' that would look at what objects are 
needed by the project(s) requested and only try to pack up those objects

what else is there that I'm not thinking of? so far these look like long-term problems as opposed to short-term problems, and all of them have fairly simple fixes that can be implemented as they become an issue.

David Lang