# Historical kernel repository size 16 messages from 2006-09-14 to 2006-09-15. Participants: Petr Baudis, tglx@linutronix.de, Linus Torvalds, Andy Whitcroft, Shawn Pearce, Nicolas Pitre, Thomas Gleixner, Junio C Hamano, Olivier Galibert. Thread: https://gitlist.dev/t/5560 ## Petr Baudis, 2006-09-14 14:22 Subject: Historical kernel repository size Message-ID: <20060914142249.GK23891@pasky.or.cz> URL: https://gitlist.dev/e/20060914142249.GK23891%40pasky.or.cz ``` Hi, just to test the packing improvements we had achieved over the last year, I have repacked the historical kernel repository and achieved a significant improvement: xpasky@machine[0:0]~/hi/history$ git-repack -a -f Generating pack... Done counting 566638 objects. Deltifying 566638 objects. 100% (566638/566638) done Writing 566638 objects. 100% (566638/566638) done Total 566638, written 566638 (delta 456212), reused 98435 (delta 0) Pack pack-4d27038611fe7755938efd4a2745d5d5d35de1c1 created. xpasky@machine[0:0]~/hi/history$ l .git/objects/pack/ total 476264 -rw-r--r-- 1 xpasky users 13600376 Sep 14 16:18 pack-4d27038611fe7755938efd4a2745d5d5d35de1c1.idx -rw-r--r-- 1 xpasky users 197168186 Sep 14 16:18 pack-4d27038611fe7755938efd4a2745d5d5d35de1c1.pack -rw-r--r-- 1 xpasky users 13600376 Sep 14 12:18 pack-cc3517351ecce3ef7ba010559992bdfc10b7acd4.idx -rw-r--r-- 1 xpasky users 262818936 Sep 14 12:29 pack-cc3517351ecce3ef7ba010559992bdfc10b7acd4.pack Since it's a nice place for people to check about how efficient we are with compressing the repository, perhaps it would be a good idea to repack the historical repository on kernel.org? -- Petr "Pasky" Baudis Stuff: http://pasky.or.cz/ Snow falling on Perl. White noise covering line noise. Hides all the bugs too. -- J. Putnam ``` ## tglx@linutronix.de, 2006-09-14 14:38 Subject: Re: Historical kernel repository size Message-ID: <46900.194.138.39.52.1158244729.squirrel@www.tglx.de> URL: https://gitlist.dev/e/46900.194.138.39.52.1158244729.squirrel%40www.tglx.de In-Reply-To: <20060914142249.GK23891@pasky.or.cz> ``` Petr, > just to test the packing improvements we had achieved over the last > year, I have repacked the historical kernel repository and achieved a > significant improvement: > .... > Since it's a nice place for people to check about how efficient we are > with compressing the repository, perhaps it would be a good idea to > repack the historical repository on kernel.org? I'll do once I'm back home. tglx ``` ## Linus Torvalds, 2006-09-14 15:31 Subject: Re: Historical kernel repository size Message-ID: URL: https://gitlist.dev/e/Pine.LNX.4.64.0609140824580.4388%40g5.osdl.org In-Reply-To: <20060914142249.GK23891@pasky.or.cz> ``` On Thu, 14 Sep 2006, Petr Baudis wrote: > > just to test the packing improvements we had achieved over the last > year, I have repacked the historical kernel repository and achieved a > significant improvement: Umm.. Only apparently because the old pack was really really bad. It also has the wrong name, probably because it's using the original naming that had the SHA1 computed on the unsorted input. That was changed a long time ago. Yours isn't wonderful either. > -rw-r--r-- 1 xpasky users 13600376 Sep 14 16:18 pack-4d27038611fe7755938efd4a2745d5d5d35de1c1.idx > -rw-r--r-- 1 xpasky users 197168186 Sep 14 16:18 pack-4d27038611fe7755938efd4a2745d5d5d35de1c1.pack > -rw-r--r-- 1 xpasky users 13600376 Sep 14 12:18 pack-cc3517351ecce3ef7ba010559992bdfc10b7acd4.idx > -rw-r--r-- 1 xpasky users 262818936 Sep 14 12:29 pack-cc3517351ecce3ef7ba010559992bdfc10b7acd4.pack Mine are: -rw-r--r-- 1 torvalds torvalds 13600376 Apr 19 10:06 pack-4d27038611fe7755938efd4a2745d5d5d35de1c1.idx -rw-r--r-- 1 torvalds torvalds 185374386 Apr 19 10:06 pack-4d27038611fe7755938efd4a2745d5d5d35de1c1.pack and as you can see from the date, they aren't exactly very recent, but they shave an additional 6% off the size. I agree that the _oroginal_ history pack by Thomas seems to be bad, and that's from Aug 9 2005, so it's likely with some really really old packing rules. For better packing, I think I used a larger depth, ie try something like git repack -a -f --depth=50 to get more improvement. For a historical archive that you don't much use, doign the deeper depth is definitely worth it. Linus ``` ## Andy Whitcroft, 2006-09-14 15:38 Subject: Re: Historical kernel repository size Message-ID: <45097775.4040005@shadowen.org> URL: https://gitlist.dev/e/45097775.4040005%40shadowen.org In-Reply-To: <46900.194.138.39.52.1158244729.squirrel@www.tglx.de> ``` tglx@linutronix.de wrote: > Petr, > >> just to test the packing improvements we had achieved over the last >> year, I have repacked the historical kernel repository and achieved a >> significant improvement: >> .... >> Since it's a nice place for people to check about how efficient we are >> with compressing the repository, perhaps it would be a good idea to >> repack the historical repository on kernel.org? > > I'll do once I'm back home. Is there any reason this isn't a live history. ie that we don't constantly pull linus' master branch into this history to make it a real complete history? Perhaps that isn't possible ... hmmm. I guess it might only work if linus' repo was actually a grafted version of this history? /me watches his head explode. -apw ``` ## Shawn Pearce, 2006-09-14 15:52 Subject: Re: Historical kernel repository size Message-ID: <20060914155212.GC9657@spearce.org> URL: https://gitlist.dev/e/20060914155212.GC9657%40spearce.org In-Reply-To: <45097775.4040005@shadowen.org> ``` Andy Whitcroft wrote: > tglx@linutronix.de wrote: > > Petr, > > > >> just to test the packing improvements we had achieved over the last > >> year, I have repacked the historical kernel repository and achieved a > >> significant improvement: > >> .... > >> Since it's a nice place for people to check about how efficient we are > >> with compressing the repository, perhaps it would be a good idea to > >> repack the historical repository on kernel.org? > > > > I'll do once I'm back home. > > Is there any reason this isn't a live history. ie that we don't > constantly pull linus' master branch into this history to make it a real > complete history? > > Perhaps that isn't possible ... hmmm. I guess it might only work if > linus' repo was actually a grafted version of this history? Right - the only way to join the two is to graft them together. Since grafts are a purely local matter anyone can pull both into the same repository and insert the correct grafts to get a complete history. You would just want to publish on the kernel.org website the correct grafts file, so users don't have to figure it out on their own. Since I'm not a kernel developer I haven't even looked to see if such a grafts file has been published. :-) -- Shawn. ``` ## Petr Baudis, 2006-09-14 15:52 Subject: Re: Historical kernel repository size Message-ID: <20060914155231.GU18896@pasky.or.cz> URL: https://gitlist.dev/e/20060914155231.GU18896%40pasky.or.cz In-Reply-To: <45097775.4040005@shadowen.org> ``` Dear diary, on Thu, Sep 14, 2006 at 05:38:29PM CEST, I got a letter where Andy Whitcroft said that... > Is there any reason this isn't a live history. ie that we don't > constantly pull linus' master branch into this history to make it a real > complete history? Because at the early times of Git, things were evolving fast and it would be unfeasible to have to drag this old history around in case of format changes and stuff. Also, at that time history was still very big and it would be impractical to require all the kernel developers to grab all the bitkeeper history (it still kind of is). > Perhaps that isn't possible ... hmmm. I guess it might only work if > linus' repo was actually a grafted version of this history? > > /me watches his head explode. http://lkml.org/lkml/2006/6/17/110 may be useful. It wasn't accepted. Oh well, I may try to resubmit it again soon. :-) -- Petr "Pasky" Baudis Stuff: http://pasky.or.cz/ Snow falling on Perl. White noise covering line noise. Hides all the bugs too. -- J. Putnam ``` ## Nicolas Pitre, 2006-09-14 21:23 Subject: Re: Historical kernel repository size Message-ID: URL: https://gitlist.dev/e/Pine.LNX.4.64.0609141714010.2627%40xanadu.home In-Reply-To: ``` On Thu, 14 Sep 2006, Linus Torvalds wrote: > For better packing, I think I used a larger depth, ie try something like > > git repack -a -f --depth=50 > > to get more improvement. For a historical archive that you don't much use, > doign the deeper depth is definitely worth it. Using a larger window helps too. It of course has a direct impact on the processing to perform a full repack, but it has no runtime costs when the pack is used. So I'd suggest adding --window=50 to the above. [ I made those suggestions in person to Thomas at OLS to which he replied he'd do it when he'd get back home. ;-) ] Nicolas ``` ## Thomas Gleixner, 2006-09-14 21:32 Subject: Re: Historical kernel repository size Message-ID: <1158269538.5724.237.camel@localhost.localdomain> URL: https://gitlist.dev/e/1158269538.5724.237.camel%40localhost.localdomain In-Reply-To: ``` On Thu, 2006-09-14 at 17:23 -0400, Nicolas Pitre wrote: > On Thu, 14 Sep 2006, Linus Torvalds wrote: > > > For better packing, I think I used a larger depth, ie try something like > > > > git repack -a -f --depth=50 > > > > to get more improvement. For a historical archive that you don't much use, > > doign the deeper depth is definitely worth it. > > Using a larger window helps too. It of course has a direct impact on > the processing to perform a full repack, but it has no runtime costs > when the pack is used. So I'd suggest adding --window=50 to the above. > > [ I made those suggestions in person to Thomas at OLS to which > he replied he'd do it when he'd get back home. ;-) ] Thanks for the reminder. I actually logged into kernel.org already :) tglx ``` ## Thomas Gleixner, 2006-09-14 21:37 Subject: Re: Historical kernel repository size Message-ID: <1158269854.5724.240.camel@localhost.localdomain> URL: https://gitlist.dev/e/1158269854.5724.240.camel%40localhost.localdomain In-Reply-To: ``` On Thu, 2006-09-14 at 17:23 -0400, Nicolas Pitre wrote: > On Thu, 14 Sep 2006, Linus Torvalds wrote: > > > For better packing, I think I used a larger depth, ie try something like > > > > git repack -a -f --depth=50 > > > when the pack is used. So I'd suggest adding --window=50 to the above. Great advise ! git repack neither accepts --depth nor --window tglx ``` ## Nicolas Pitre, 2006-09-14 21:42 Subject: Re: Historical kernel repository size Message-ID: URL: https://gitlist.dev/e/Pine.LNX.4.64.0609141742000.2627%40xanadu.home In-Reply-To: <1158269854.5724.240.camel@localhost.localdomain> ``` On Thu, 14 Sep 2006, Thomas Gleixner wrote: > On Thu, 2006-09-14 at 17:23 -0400, Nicolas Pitre wrote: > > On Thu, 14 Sep 2006, Linus Torvalds wrote: > > > > > For better packing, I think I used a larger depth, ie try something like > > > > > > git repack -a -f --depth=50 > > > > > when the pack is used. So I'd suggest adding --window=50 to the above. > > Great advise ! > > git repack neither accepts --depth nor --window Is the GIT version on kernel.org _that_ old? What a shame... Nicolas ``` ## Thomas Gleixner, 2006-09-14 21:54 Subject: Re: Historical kernel repository size Message-ID: <1158270859.5724.244.camel@localhost.localdomain> URL: https://gitlist.dev/e/1158270859.5724.244.camel%40localhost.localdomain In-Reply-To: ``` On Thu, 2006-09-14 at 17:42 -0400, Nicolas Pitre wrote: > > git repack neither accepts --depth nor --window > > Is the GIT version on kernel.org _that_ old? > > What a shame... [tglx@hera history.git]$ git --version git version 1.4.2.1 tglx ``` ## Thomas Gleixner, 2006-09-14 22:24 Subject: Re: Historical kernel repository size Message-ID: <1158272651.5724.251.camel@localhost.localdomain> URL: https://gitlist.dev/e/1158272651.5724.251.camel%40localhost.localdomain In-Reply-To: <1158270859.5724.244.camel@localhost.localdomain> ``` On Thu, 2006-09-14 at 23:54 +0200, Thomas Gleixner wrote: > On Thu, 2006-09-14 at 17:42 -0400, Nicolas Pitre wrote: > > > git repack neither accepts --depth nor --window > > > > Is the GIT version on kernel.org _that_ old? > > > > What a shame... > > [tglx@hera history.git]$ git --version > git version 1.4.2.1 I know I'm stupid "git-repack --window=50 --depth=50 -a -f" works "git-repack -a -f --window=50 --depth=50" does not Intuitive user interfaces are my favorite pitfalls. -rw-rw-r-- 1 tglx ftpadmin 13600376 Sep 14 22:16 pack-4d27038611fe7755938efd4a2745d5d5d35de1c1.idx -rw-rw-r-- 1 tglx ftpadmin 158679705 Sep 14 22:16 pack-4d27038611fe7755938efd4a2745d5d5d35de1c1.pack tglx ``` ## Junio C Hamano, 2006-09-14 23:15 Subject: Re: Historical kernel repository size Message-ID: <7v4pva2g6y.fsf@assigned-by-dhcp.cox.net> URL: https://gitlist.dev/e/7v4pva2g6y.fsf%40assigned-by-dhcp.cox.net In-Reply-To: <1158272651.5724.251.camel@localhost.localdomain> ``` Thomas Gleixner writes: >> [tglx@hera history.git]$ git --version >> git version 1.4.2.1 > > I know I'm stupid > > "git-repack --window=50 --depth=50 -a -f" works > "git-repack -a -f --window=50 --depth=50" does not > > Intuitive user interfaces are my favorite pitfalls. Whaaaat? I've run them under "sh -x" and both results in a pipe of: git-rev-list --objects --all | git-pack-objects --non-empty --no-reuse-delta --window=50 --depth=50 \ .git/.tmp--pack Now you are making me really worried. ``` ## Nicolas Pitre, 2006-09-15 01:19 Subject: Re: Historical kernel repository size Message-ID: URL: https://gitlist.dev/e/Pine.LNX.4.64.0609142115560.2627%40xanadu.home In-Reply-To: <1158272651.5724.251.camel@localhost.localdomain> ``` On Fri, 15 Sep 2006, Thomas Gleixner wrote: > On Thu, 2006-09-14 at 23:54 +0200, Thomas Gleixner wrote: > > On Thu, 2006-09-14 at 17:42 -0400, Nicolas Pitre wrote: > > > > git repack neither accepts --depth nor --window > > > > > > Is the GIT version on kernel.org _that_ old? > > > > > > What a shame... > > > > [tglx@hera history.git]$ git --version > > git version 1.4.2.1 OK that's recent enough indeed. > I know I'm stupid > > "git-repack --window=50 --depth=50 -a -f" works > "git-repack -a -f --window=50 --depth=50" does not > > Intuitive user interfaces are my favorite pitfalls. Erm... Both incantations work fine fine here. > -rw-rw-r-- 1 tglx ftpadmin 13600376 Sep 14 22:16 pack-4d27038611fe7755938efd4a2745d5d5d35de1c1.idx > -rw-rw-r-- 1 tglx ftpadmin 158679705 Sep 14 22:16 pack-4d27038611fe7755938efd4a2745d5d5d35de1c1.pack And I get the same result as well. Nicolas ``` ## Olivier Galibert, 2006-09-15 09:03 Subject: Re: Historical kernel repository size Message-ID: <20060915090305.GC75256@dspnet.fr.eu.org> URL: https://gitlist.dev/e/20060915090305.GC75256%40dspnet.fr.eu.org In-Reply-To: ``` On Thu, Sep 14, 2006 at 09:19:04PM -0400, Nicolas Pitre wrote: > Erm... Both incantations work fine fine here. > > > -rw-rw-r-- 1 tglx ftpadmin 13600376 Sep 14 22:16 pack-4d27038611fe7755938efd4a2745d5d5d35de1c1.idx > > -rw-rw-r-- 1 tglx ftpadmin 158679705 Sep 14 22:16 pack-4d27038611fe7755938efd4a2745d5d5d35de1c1.pack > > And I get the same result as well. For the curious, a 100/100 parameter gives a size of 154261771. Disminishing returns, here I come. OG. ``` ## Nicolas Pitre, 2006-09-15 16:45 Subject: Re: Historical kernel repository size Message-ID: URL: https://gitlist.dev/e/Pine.LNX.4.64.0609151232370.2627%40xanadu.home In-Reply-To: <20060915090305.GC75256@dspnet.fr.eu.org> ``` On Fri, 15 Sep 2006, Olivier Galibert wrote: > On Thu, Sep 14, 2006 at 09:19:04PM -0400, Nicolas Pitre wrote: > > Erm... Both incantations work fine fine here. > > > > > -rw-rw-r-- 1 tglx ftpadmin 13600376 Sep 14 22:16 pack-4d27038611fe7755938efd4a2745d5d5d35de1c1.idx > > > -rw-rw-r-- 1 tglx ftpadmin 158679705 Sep 14 22:16 pack-4d27038611fe7755938efd4a2745d5d5d35de1c1.pack > > > > And I get the same result as well. > > For the curious, a 100/100 parameter gives a size of 154261771. Right. And then the runtime cost of extracting objects out of such a pack increases due to the deeper delta chain. The average runtime cost is probably linear with the delta depth, something like f(x) = a*x + k. But the size reduction follows f(x) = a/x + k. So to say that infinite delta length does not provide infinite packing size reduction. Anything larger than 50 is probably not worth the small reduction gain. Nicolas ```