Historical kernel repository size

  Hi,

  just to test the packing improvements we had achieved over the last
year, I have repacked the historical kernel repository and achieved a
significant improvement:

xpasky@machine[0:0]~/hi/history$ git-repack -a -f
Generating pack...
Done counting 566638 objects.
Deltifying 566638 objects.
 100% (566638/566638) done
Writing 566638 objects.
 100% (566638/566638) done
Total 566638, written 566638 (delta 456212), reused 98435 (delta 0)
Pack pack-4d27038611fe7755938efd4a2745d5d5d35de1c1 created.
xpasky@machine[0:0]~/hi/history$ l .git/objects/pack/
total 476264
-rw-r--r-- 1 xpasky users  13600376 Sep 14 16:18 pack-4d27038611fe7755938efd4a2745d5d5d35de1c1.idx
-rw-r--r-- 1 xpasky users 197168186 Sep 14 16:18 pack-4d27038611fe7755938efd4a2745d5d5d35de1c1.pack
-rw-r--r-- 1 xpasky users  13600376 Sep 14 12:18 pack-cc3517351ecce3ef7ba010559992bdfc10b7acd4.idx
-rw-r--r-- 1 xpasky users 262818936 Sep 14 12:29 pack-cc3517351ecce3ef7ba010559992bdfc10b7acd4.pack

  Since it's a nice place for people to check about how efficient we are
with compressing the repository, perhaps it would be a good idea to
repack the historical repository on kernel.org?

Petr,

I'll do once I'm back home.

    tglx

tglx@linutronix.de wrote:

Is there any reason this isn't a live history. ie that we don't constantly pull linus' master branch into this history to make it a real complete history?

Perhaps that isn't possible ... hmmm. I guess it might only work if linus' repo was actually a grafted version of this history?

/me watches his head explode.

-apw

Andy Whitcroft <apw@shadowen.org> wrote:

Right - the only way to join the two is to graft them together.

Since grafts are a purely local matter anyone can pull both into the same repository and insert the correct grafts to get a complete history. You would just want to publish on the kernel.org website the correct grafts file, so users don't have to figure it out on their own.

Since I'm not a kernel developer I haven't even looked to see if such a grafts file has been published. :-)

Dear diary, on Thu, Sep 14, 2006 at 05:38:29PM CEST, I got a letter where Andy Whitcroft <apw@shadowen.org> said that...

Because at the early times of Git, things were evolving fast and it would be unfeasible to have to drag this old history around in case of format changes and stuff. Also, at that time history was still very big and it would be impractical to require all the kernel developers to grab all the bitkeeper history (it still kind of is).

	http://lkml.org/lkml/2006/6/17/110

may be useful.

It wasn't accepted. Oh well, I may try to resubmit it again soon. :-)

On Thu, 14 Sep 2006, Petr Baudis wrote:

Umm.. Only apparently because the old pack was really really bad. It also has the wrong name, probably because it's using the original naming that had the SHA1 computed on the unsorted input. That was changed a long time ago.

Yours isn't wonderful either.

Mine are:

-rw-r--r-- 1 torvalds torvalds 13600376 Apr 19 10:06 pack-4d27038611fe7755938efd4a2745d5d5d35de1c1.idx -rw-r--r-- 1 torvalds torvalds 185374386 Apr 19 10:06 pack-4d27038611fe7755938efd4a2745d5d5d35de1c1.pack

and as you can see from the date, they aren't exactly very recent, but they shave an additional 6% off the size.

I agree that the _oroginal_ history pack by Thomas seems to be bad, and that's from Aug 9 2005, so it's likely with some really really old packing rules.

For better packing, I think I used a larger depth, ie try something like

	git repack -a -f --depth=50

to get more improvement. For a historical archive that you don't much use, doign the deeper depth is definitely worth it.

		Linus

On Thu, 14 Sep 2006, Linus Torvalds wrote:

Using a larger window helps too. It of course has a direct impact on the processing to perform a full repack, but it has no runtime costs when the pack is used. So I'd suggest adding --window=50 to the above.

[ I made those suggestions in person to Thomas at OLS to which 
  he replied he'd do it when he'd get back home.   ;-) ]

Nicolas

On Thu, 2006-09-14 at 17:23 -0400, Nicolas Pitre wrote:

Thanks for the reminder. I actually logged into kernel.org already :)

	tglx

On Thu, 2006-09-14 at 17:23 -0400, Nicolas Pitre wrote:

Great advise !

git repack neither accepts --depth nor --window

	tglx

On Thu, 14 Sep 2006, Thomas Gleixner wrote:

Is the GIT version on kernel.org _that_ old?

What a shame...

Nicolas

On Thu, 2006-09-14 at 17:42 -0400, Nicolas Pitre wrote:

[tglx@hera history.git]$ git --version git version 1.4.2.1

	tglx

On Thu, 2006-09-14 at 23:54 +0200, Thomas Gleixner wrote:

I know I'm stupid

"git-repack --window=50 --depth=50 -a -f" works "git-repack -a -f --window=50 --depth=50" does not

Intuitive user interfaces are my favorite pitfalls.

-rw-rw-r-- 1 tglx ftpadmin 13600376 Sep 14 22:16 pack-4d27038611fe7755938efd4a2745d5d5d35de1c1.idx -rw-rw-r-- 1 tglx ftpadmin 158679705 Sep 14 22:16 pack-4d27038611fe7755938efd4a2745d5d5d35de1c1.pack

	tglx

Thomas Gleixner <tglx@linutronix.de> writes:

Whaaaat?

I've run them under "sh -x" and both results in a pipe of:

	git-rev-list --objects --all |
        git-pack-objects --non-empty --no-reuse-delta --window=50 --depth=50 \
	.git/.tmp-<somepid>-pack

Now you are making me really worried.

On Fri, 15 Sep 2006, Thomas Gleixner wrote:

OK that's recent enough indeed.

Erm... Both incantations work fine fine here.

And I get the same result as well.

Nicolas

On Thu, Sep 14, 2006 at 09:19:04PM -0400, Nicolas Pitre wrote:

For the curious, a 100/100 parameter gives a size of 154261771. Disminishing returns, here I come.

  OG.

On Fri, 15 Sep 2006, Olivier Galibert wrote:

Right. And then the runtime cost of extracting objects out of such a pack increases due to the deeper delta chain.

The average runtime cost is probably linear with the delta depth, something like f(x) = a*x + k.

But the size reduction follows f(x) = a/x + k.

So to say that infinite delta length does not provide infinite packing size reduction. Anything larger than 50 is probably not worth the small reduction gain.

Nicolas