Re: git clone dies (large git repository)

From: Junio C Hamano <junkio@cox.net>
Date: Aug 22, 2006, 00:23 UTC
Message-ID: <7v3bbpty7j.fsf@assigned-by-dhcp.cox.net>
In-Reply-To: <op.tenp9fv1ies9li@rygel.lnxi.com>

"Troy Telford" <ttelford@linuxnetworx.com> writes:

> I'm thinking of it as an option for git-repack-- that the user can set
> the maximum size of any individual pack, and after that limit is
> reached, a  new pack file is started.  (ie. --max-size 2GB) and will
> end up with two  packs, each 2GB in size.

The way I would suggest you do it is not by size but by distance from the latest. If you want to split the kernel history for example, you repack up to 2.6.14 for example, and then repack the remainder. That way, you can optimize for size for older (presumably less frequently used) data while optimizing for speed for more reent stuff.

There is no wrapper support for the above splitting in git-repack. The low-level plumbing tools can be used this way for example:

	name=`
                git rev-list --objects $list_old_tags_here |
                git pack-objects --window=50 --depth=50 --non-empty .tmp-pack
        ` &&
        mv -f .tmp-pack-$name.{pack,idx} .git/objects/pack/

	name=`
		git revlist --objects --all --not $list_old_tags_here |
		git pack-objects --non-empty .tmp-pack
	` &&
        mv -f .tmp-pack-$name.{pack,idx} .git/objects/pack/

If you are splitting into more than two, you would instead have more than one $list_old_tags_here list, and iterate them through, something like:

	pack_between () {
        	already_done="$1"
                do_this_time="$2"
                w=${3-10}
		name=`
			git rev-list --objects \
                        	$do_this_time \
                                --not $already_done |
			git pack-objects --window=$w --depth=$w \
                        	--non-empty .tmp-pack
                ` &&
                mv -f .tmp-pack-$name.{pack,idx} .git/objects/pack/
	}

	pack_between "" "$prehistoric_tag_list" 100
	pack_between "$prehistoric_tag_list" "$more_recent_tag_list" 50
	pack_between "$more_recent_tag_list" --all

All untested, of course, so do not play with it in your precious repository you do not have any other copy, but hopefully you get an idea ;-).