Re: sharing object packs

From: Shawn O. Pearce <spearce@spearce.org>
Date: Jun 19, 2008, 09:01 UTC
Message-ID: <20080619090129.GJ11793@spearce.org>
In-Reply-To: <20080618195726.GA20440@kernoel.dyndns.org>

marc.zonzon+git@gmail.com wrote:

Show 6 quoted lines

> I have a big bare repository 'main.git' and many small git repositories sub1, sub2, ... subn.
>
> All repositories lie in the same file file system, and each subx
> repository track and fetch main.git in a remote branch.
>
> I would like to avoid duplicating main.git objects

...

Show 5 quoted lines

> - Using an objects/info/alternates with the path of main.git object repository.
> It work well too, but I import objects from main.git inside subx,
> and they don't have the same life time than those in main.git. So
> they can, disapear during a git-prune-packed or gc. (The same
> problem we have with: git clone --share)

This is the approach you want to use. The risk is that you do not allow objects to be added to main.git to later be deleted from main.git. This means main.git cannot rewind/reset/delete a branch.

If that is not acceptable perhaps you could instead create 3 tiers:

	main.git ---
	            \
	            shared.git
	            /
	subx.git ---

Have main.git and subx.git both use shared.git as an alternate (place path of shared.git/objects in their objects/info/alternates). You can still allow subx.git to fetch main.git.

Push only stable commits to shared.git that will never be rewind/reset/deleted. Once something enters shared.git it should never be deleted. This way shared objects will not be removed by git-prune or git-gc. Every so often push newer stable branches from main.git to shared.git, once they cannot be rewind/reset/deleted.

Repack main.git and subx.git using `git gc` as that includes the -l flag to `git repack`. Any objects which are now available from shared.git will not be included in main.git or subx.git, so their usage will shrink after shared.git is updated.

If you also configure gc.packrefs to never in shared.git and symlink shared.git/refs into main.git/refs/shared and also into subx.git/refs/shared and do this configuration on both server and client systems you can have everyone transfer only the minimal objects necessary.

I use basically that arrangement at day-job to avoid 7 copies of roughly 150 MB of shared history across 8 repositories. This reduces the amount of data the OS needs to store in buffer cache by nearly 1050 MB and thus makes things run rather quickly. (Yes, many operations hit all 8 repositories in rather rapid succession, its a submodule sort of arrangement.)

-- 
Shawn.