{"thread":{"id":"5560","subject":"Historical kernel repository size","startedAt":"2006-09-14T14:22:49Z","lastAt":"2006-09-15T16:45:53Z","messageCount":16,"participants":["Petr Baudis","tglx@linutronix.de","Linus Torvalds","Andy Whitcroft","Shawn Pearce","Nicolas Pitre","Thomas Gleixner","Junio C Hamano","Olivier Galibert"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"26883","messageId":"20060914142249.GK23891@pasky.or.cz","threadId":"5560","inReplyTo":null,"subject":"Historical kernel repository size","fromName":"Petr Baudis","fromEmail":"pasky@suse.cz","sentAt":"2006-09-14T14:22:49Z","receivedAt":"2006-09-14T14:22:49Z","isPatch":false,"sender":{"key":"pasky@ucw.cz","avatar":"https://avatars.githubusercontent.com/u/18439?v=4"},"body":"  Hi,\n\n  just to test the packing improvements we had achieved over the last\nyear, I have repacked the historical kernel repository and achieved a\nsignificant improvement:\n\nxpasky@machine[0:0]~/hi/history$ git-repack -a -f\nGenerating pack...\nDone counting 566638 objects.\nDeltifying 566638 objects.\n 100% (566638/566638) done\nWriting 566638 objects.\n 100% (566638/566638) done\nTotal 566638, written 566638 (delta 456212), reused 98435 (delta 0)\nPack pack-4d27038611fe7755938efd4a2745d5d5d35de1c1 created.\nxpasky@machine[0:0]~/hi/history$ l .git/objects/pack/\ntotal 476264\n-rw-r--r-- 1 xpasky users  13600376 Sep 14 16:18 pack-4d27038611fe7755938efd4a2745d5d5d35de1c1.idx\n-rw-r--r-- 1 xpasky users 197168186 Sep 14 16:18 pack-4d27038611fe7755938efd4a2745d5d5d35de1c1.pack\n-rw-r--r-- 1 xpasky users  13600376 Sep 14 12:18 pack-cc3517351ecce3ef7ba010559992bdfc10b7acd4.idx\n-rw-r--r-- 1 xpasky users 262818936 Sep 14 12:29 pack-cc3517351ecce3ef7ba010559992bdfc10b7acd4.pack\n\n  Since it's a nice place for people to check about how efficient we are\nwith compressing the repository, perhaps it would be a good idea to\nrepack the historical repository on kernel.org?\n\n-- \n\t\t\t\tPetr \"Pasky\" Baudis\nStuff: http://pasky.or.cz/\nSnow falling on Perl. White noise covering line noise.\nHides all the bugs too. -- J. Putnam\n"},{"id":"26884","messageId":"46900.194.138.39.52.1158244729.squirrel@www.tglx.de","threadId":"5560","inReplyTo":"20060914142249.GK23891@pasky.or.cz","subject":"Re: Historical kernel repository size","fromName":"","fromEmail":"tglx@linutronix.de","sentAt":"2006-09-14T14:38:49Z","receivedAt":"2006-09-14T14:38:49Z","isPatch":false,"sender":{"key":"tglx@linutronix.de","avatar":null},"body":"Petr,\n\n>   just to test the packing improvements we had achieved over the last\n> year, I have repacked the historical kernel repository and achieved a\n> significant improvement:\n> ....\n>   Since it's a nice place for people to check about how efficient we are\n> with compressing the repository, perhaps it would be a good idea to\n> repack the historical repository on kernel.org?\n\nI'll do once I'm back home.\n\n    tglx\n"},{"id":"26885","messageId":"Pine.LNX.4.64.0609140824580.4388@g5.osdl.org","threadId":"5560","inReplyTo":"20060914142249.GK23891@pasky.or.cz","subject":"Re: Historical kernel repository size","fromName":"Linus Torvalds","fromEmail":"torvalds@osdl.org","sentAt":"2006-09-14T15:31:39Z","receivedAt":"2006-09-14T15:31:39Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Thu, 14 Sep 2006, Petr Baudis wrote:\n> \n>   just to test the packing improvements we had achieved over the last\n> year, I have repacked the historical kernel repository and achieved a\n> significant improvement:\n\nUmm.. Only apparently because the old pack was really really bad. It also \nhas the wrong name, probably because it's using the original naming that \nhad the SHA1 computed on the unsorted input. That was changed a long time \nago.\n\nYours isn't wonderful either.\n\n> -rw-r--r-- 1 xpasky users  13600376 Sep 14 16:18 pack-4d27038611fe7755938efd4a2745d5d5d35de1c1.idx\n> -rw-r--r-- 1 xpasky users 197168186 Sep 14 16:18 pack-4d27038611fe7755938efd4a2745d5d5d35de1c1.pack\n> -rw-r--r-- 1 xpasky users  13600376 Sep 14 12:18 pack-cc3517351ecce3ef7ba010559992bdfc10b7acd4.idx\n> -rw-r--r-- 1 xpasky users 262818936 Sep 14 12:29 pack-cc3517351ecce3ef7ba010559992bdfc10b7acd4.pack\n\nMine are:\n\n-rw-r--r-- 1 torvalds torvalds  13600376 Apr 19 10:06 pack-4d27038611fe7755938efd4a2745d5d5d35de1c1.idx\n-rw-r--r-- 1 torvalds torvalds 185374386 Apr 19 10:06 pack-4d27038611fe7755938efd4a2745d5d5d35de1c1.pack\n\nand as you can see from the date, they aren't exactly very recent, but \nthey shave an additional 6% off the size.\n\nI agree that the _oroginal_ history pack by Thomas seems to be bad, and \nthat's from Aug 9 2005, so it's likely with some really really old packing \nrules.\n\nFor better packing, I think I used a larger depth, ie try something like\n\n\tgit repack -a -f --depth=50\n\nto get more improvement. For a historical archive that you don't much use, \ndoign the deeper depth is definitely worth it.\n\n\t\tLinus\n"},{"id":"26886","messageId":"45097775.4040005@shadowen.org","threadId":"5560","inReplyTo":"46900.194.138.39.52.1158244729.squirrel@www.tglx.de","subject":"Re: Historical kernel repository size","fromName":"Andy Whitcroft","fromEmail":"apw@shadowen.org","sentAt":"2006-09-14T15:38:29Z","receivedAt":"2006-09-14T15:38:29Z","isPatch":false,"sender":{"key":"apw@shadowen.org","avatar":"https://gravatar.com/avatar/d3088262854661a913ef35cc40fedcc270142d4461791142bc1ea0b2a4e2e147?d=mp&s=160"},"body":"tglx@linutronix.de wrote:\n> Petr,\n> \n>>   just to test the packing improvements we had achieved over the last\n>> year, I have repacked the historical kernel repository and achieved a\n>> significant improvement:\n>> ....\n>>   Since it's a nice place for people to check about how efficient we are\n>> with compressing the repository, perhaps it would be a good idea to\n>> repack the historical repository on kernel.org?\n> \n> I'll do once I'm back home.\n\nIs there any reason this isn't a live history.  ie that we don't\nconstantly pull linus' master branch into this history to make it a real\ncomplete history?\n\nPerhaps that isn't possible ... hmmm.  I guess it might only work if\nlinus' repo was actually a grafted version of this history?\n\n/me watches his head explode.\n\n-apw\n"},{"id":"26890","messageId":"20060914155212.GC9657@spearce.org","threadId":"5560","inReplyTo":"45097775.4040005@shadowen.org","subject":"Re: Historical kernel repository size","fromName":"Shawn Pearce","fromEmail":"spearce@spearce.org","sentAt":"2006-09-14T15:52:12Z","receivedAt":"2006-09-14T15:52:12Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Andy Whitcroft <apw@shadowen.org> wrote:\n> tglx@linutronix.de wrote:\n> > Petr,\n> > \n> >>   just to test the packing improvements we had achieved over the last\n> >> year, I have repacked the historical kernel repository and achieved a\n> >> significant improvement:\n> >> ....\n> >>   Since it's a nice place for people to check about how efficient we are\n> >> with compressing the repository, perhaps it would be a good idea to\n> >> repack the historical repository on kernel.org?\n> > \n> > I'll do once I'm back home.\n> \n> Is there any reason this isn't a live history.  ie that we don't\n> constantly pull linus' master branch into this history to make it a real\n> complete history?\n> \n> Perhaps that isn't possible ... hmmm.  I guess it might only work if\n> linus' repo was actually a grafted version of this history?\n\nRight - the only way to join the two is to graft them together.\n\nSince grafts are a purely local matter anyone can pull both into\nthe same repository and insert the correct grafts to get a complete\nhistory.  You would just want to publish on the kernel.org website\nthe correct grafts file, so users don't have to figure it out on\ntheir own.\n\nSince I'm not a kernel developer I haven't even looked to see if\nsuch a grafts file has been published.  :-)\n\n-- \nShawn.\n"},{"id":"26889","messageId":"20060914155231.GU18896@pasky.or.cz","threadId":"5560","inReplyTo":"45097775.4040005@shadowen.org","subject":"Re: Historical kernel repository size","fromName":"Petr Baudis","fromEmail":"pasky@suse.cz","sentAt":"2006-09-14T15:52:31Z","receivedAt":"2006-09-14T15:52:31Z","isPatch":false,"sender":{"key":"pasky@ucw.cz","avatar":"https://avatars.githubusercontent.com/u/18439?v=4"},"body":"Dear diary, on Thu, Sep 14, 2006 at 05:38:29PM CEST, I got a letter\nwhere Andy Whitcroft <apw@shadowen.org> said that...\n> Is there any reason this isn't a live history.  ie that we don't\n> constantly pull linus' master branch into this history to make it a real\n> complete history?\n\nBecause at the early times of Git, things were evolving fast and it\nwould be unfeasible to have to drag this old history around in case of\nformat changes and stuff. Also, at that time history was still very big\nand it would be impractical to require all the kernel developers to grab\nall the bitkeeper history (it still kind of is).\n\n> Perhaps that isn't possible ... hmmm.  I guess it might only work if\n> linus' repo was actually a grafted version of this history?\n> \n> /me watches his head explode.\n\n\thttp://lkml.org/lkml/2006/6/17/110\n\nmay be useful.\n\nIt wasn't accepted. Oh well, I may try to resubmit it again soon. :-)\n\n-- \n\t\t\t\tPetr \"Pasky\" Baudis\nStuff: http://pasky.or.cz/\nSnow falling on Perl. White noise covering line noise.\nHides all the bugs too. -- J. Putnam\n"},{"id":"26925","messageId":"Pine.LNX.4.64.0609141714010.2627@xanadu.home","threadId":"5560","inReplyTo":"Pine.LNX.4.64.0609140824580.4388@g5.osdl.org","subject":"Re: Historical kernel repository size","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2006-09-14T21:23:44Z","receivedAt":"2006-09-14T21:23:44Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Thu, 14 Sep 2006, Linus Torvalds wrote:\n\n> For better packing, I think I used a larger depth, ie try something like\n> \n> \tgit repack -a -f --depth=50\n> \n> to get more improvement. For a historical archive that you don't much use, \n> doign the deeper depth is definitely worth it.\n\nUsing a larger window helps too.  It of course has a direct impact on \nthe processing to perform a full repack, but it has no runtime costs \nwhen the pack is used.  So I'd suggest adding --window=50 to the above.\n\n[ I made those suggestions in person to Thomas at OLS to which \n  he replied he'd do it when he'd get back home.   ;-) ]\n\n\nNicolas\n"},{"id":"26927","messageId":"1158269538.5724.237.camel@localhost.localdomain","threadId":"5560","inReplyTo":"Pine.LNX.4.64.0609141714010.2627@xanadu.home","subject":"Re: Historical kernel repository size","fromName":"Thomas Gleixner","fromEmail":"tglx@linutronix.de","sentAt":"2006-09-14T21:32:18Z","receivedAt":"2006-09-14T21:32:18Z","isPatch":false,"sender":{"key":"tglx@linutronix.de","avatar":null},"body":"On Thu, 2006-09-14 at 17:23 -0400, Nicolas Pitre wrote:\n> On Thu, 14 Sep 2006, Linus Torvalds wrote:\n> \n> > For better packing, I think I used a larger depth, ie try something like\n> > \n> > \tgit repack -a -f --depth=50\n> > \n> > to get more improvement. For a historical archive that you don't much use, \n> > doign the deeper depth is definitely worth it.\n> \n> Using a larger window helps too.  It of course has a direct impact on \n> the processing to perform a full repack, but it has no runtime costs \n> when the pack is used.  So I'd suggest adding --window=50 to the above.\n> \n> [ I made those suggestions in person to Thomas at OLS to which \n>   he replied he'd do it when he'd get back home.   ;-) ]\n\nThanks for the reminder. I actually logged into kernel.org already :)\n\n\ttglx\n"},{"id":"26928","messageId":"1158269854.5724.240.camel@localhost.localdomain","threadId":"5560","inReplyTo":"Pine.LNX.4.64.0609141714010.2627@xanadu.home","subject":"Re: Historical kernel repository size","fromName":"Thomas Gleixner","fromEmail":"tglx@linutronix.de","sentAt":"2006-09-14T21:37:33Z","receivedAt":"2006-09-14T21:37:33Z","isPatch":false,"sender":{"key":"tglx@linutronix.de","avatar":null},"body":"On Thu, 2006-09-14 at 17:23 -0400, Nicolas Pitre wrote:\n> On Thu, 14 Sep 2006, Linus Torvalds wrote:\n> \n> > For better packing, I think I used a larger depth, ie try something like\n> > \n> > \tgit repack -a -f --depth=50\n> > \n> when the pack is used.  So I'd suggest adding --window=50 to the above.\n\nGreat advise !\n\ngit repack neither accepts --depth nor --window\n\n\ttglx\n"},{"id":"26929","messageId":"Pine.LNX.4.64.0609141742000.2627@xanadu.home","threadId":"5560","inReplyTo":"1158269854.5724.240.camel@localhost.localdomain","subject":"Re: Historical kernel repository size","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2006-09-14T21:42:44Z","receivedAt":"2006-09-14T21:42:44Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Thu, 14 Sep 2006, Thomas Gleixner wrote:\n\n> On Thu, 2006-09-14 at 17:23 -0400, Nicolas Pitre wrote:\n> > On Thu, 14 Sep 2006, Linus Torvalds wrote:\n> > \n> > > For better packing, I think I used a larger depth, ie try something like\n> > > \n> > > \tgit repack -a -f --depth=50\n> > > \n> > when the pack is used.  So I'd suggest adding --window=50 to the above.\n> \n> Great advise !\n> \n> git repack neither accepts --depth nor --window\n\nIs the GIT version on kernel.org _that_ old?\n\nWhat a shame...\n\n\nNicolas\n"},{"id":"26931","messageId":"1158270859.5724.244.camel@localhost.localdomain","threadId":"5560","inReplyTo":"Pine.LNX.4.64.0609141742000.2627@xanadu.home","subject":"Re: Historical kernel repository size","fromName":"Thomas Gleixner","fromEmail":"tglx@linutronix.de","sentAt":"2006-09-14T21:54:19Z","receivedAt":"2006-09-14T21:54:19Z","isPatch":false,"sender":{"key":"tglx@linutronix.de","avatar":null},"body":"On Thu, 2006-09-14 at 17:42 -0400, Nicolas Pitre wrote:\n> > git repack neither accepts --depth nor --window\n> \n> Is the GIT version on kernel.org _that_ old?\n> \n> What a shame...\n\n[tglx@hera history.git]$ git --version\ngit version 1.4.2.1\n\n\ttglx\n"},{"id":"26934","messageId":"1158272651.5724.251.camel@localhost.localdomain","threadId":"5560","inReplyTo":"1158270859.5724.244.camel@localhost.localdomain","subject":"Re: Historical kernel repository size","fromName":"Thomas Gleixner","fromEmail":"tglx@linutronix.de","sentAt":"2006-09-14T22:24:11Z","receivedAt":"2006-09-14T22:24:11Z","isPatch":false,"sender":{"key":"tglx@linutronix.de","avatar":null},"body":"On Thu, 2006-09-14 at 23:54 +0200, Thomas Gleixner wrote:\n> On Thu, 2006-09-14 at 17:42 -0400, Nicolas Pitre wrote:\n> > > git repack neither accepts --depth nor --window\n> > \n> > Is the GIT version on kernel.org _that_ old?\n> > \n> > What a shame...\n> \n> [tglx@hera history.git]$ git --version\n> git version 1.4.2.1\n\nI know I'm stupid\n\n\"git-repack --window=50 --depth=50 -a -f\" works\n\"git-repack -a -f --window=50 --depth=50\" does not\n\nIntuitive user interfaces are my favorite pitfalls.\n\n-rw-rw-r-- 1 tglx ftpadmin  13600376 Sep 14 22:16 pack-4d27038611fe7755938efd4a2745d5d5d35de1c1.idx\n-rw-rw-r-- 1 tglx ftpadmin 158679705 Sep 14 22:16 pack-4d27038611fe7755938efd4a2745d5d5d35de1c1.pack\n\n\ttglx\n"},{"id":"26936","messageId":"7v4pva2g6y.fsf@assigned-by-dhcp.cox.net","threadId":"5560","inReplyTo":"1158272651.5724.251.camel@localhost.localdomain","subject":"Re: Historical kernel repository size","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2006-09-14T23:15:17Z","receivedAt":"2006-09-14T23:15:17Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Thomas Gleixner <tglx@linutronix.de> writes:\n\n>> [tglx@hera history.git]$ git --version\n>> git version 1.4.2.1\n>\n> I know I'm stupid\n>\n> \"git-repack --window=50 --depth=50 -a -f\" works\n> \"git-repack -a -f --window=50 --depth=50\" does not\n>\n> Intuitive user interfaces are my favorite pitfalls.\n\nWhaaaat?\n\nI've run them under \"sh -x\" and both results in a pipe of:\n\n\tgit-rev-list --objects --all |\n        git-pack-objects --non-empty --no-reuse-delta --window=50 --depth=50 \\\n\t.git/.tmp-<somepid>-pack\n\nNow you are making me really worried.\n"},{"id":"26937","messageId":"Pine.LNX.4.64.0609142115560.2627@xanadu.home","threadId":"5560","inReplyTo":"1158272651.5724.251.camel@localhost.localdomain","subject":"Re: Historical kernel repository size","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2006-09-15T01:19:04Z","receivedAt":"2006-09-15T01:19:04Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Fri, 15 Sep 2006, Thomas Gleixner wrote:\n\n> On Thu, 2006-09-14 at 23:54 +0200, Thomas Gleixner wrote:\n> > On Thu, 2006-09-14 at 17:42 -0400, Nicolas Pitre wrote:\n> > > > git repack neither accepts --depth nor --window\n> > > \n> > > Is the GIT version on kernel.org _that_ old?\n> > > \n> > > What a shame...\n> > \n> > [tglx@hera history.git]$ git --version\n> > git version 1.4.2.1\n\nOK that's recent enough indeed.\n\n> I know I'm stupid\n> \n> \"git-repack --window=50 --depth=50 -a -f\" works\n> \"git-repack -a -f --window=50 --depth=50\" does not\n> \n> Intuitive user interfaces are my favorite pitfalls.\n\nErm... Both incantations work fine fine here.\n\n> -rw-rw-r-- 1 tglx ftpadmin  13600376 Sep 14 22:16 pack-4d27038611fe7755938efd4a2745d5d5d35de1c1.idx\n> -rw-rw-r-- 1 tglx ftpadmin 158679705 Sep 14 22:16 pack-4d27038611fe7755938efd4a2745d5d5d35de1c1.pack\n\nAnd I get the same result as well.\n\n\nNicolas\n"},{"id":"26954","messageId":"20060915090305.GC75256@dspnet.fr.eu.org","threadId":"5560","inReplyTo":"Pine.LNX.4.64.0609142115560.2627@xanadu.home","subject":"Re: Historical kernel repository size","fromName":"Olivier Galibert","fromEmail":"galibert@pobox.com","sentAt":"2006-09-15T09:03:05Z","receivedAt":"2006-09-15T09:03:05Z","isPatch":false,"sender":{"key":"galibert@pobox.com","avatar":null},"body":"On Thu, Sep 14, 2006 at 09:19:04PM -0400, Nicolas Pitre wrote:\n> Erm... Both incantations work fine fine here.\n> \n> > -rw-rw-r-- 1 tglx ftpadmin  13600376 Sep 14 22:16 pack-4d27038611fe7755938efd4a2745d5d5d35de1c1.idx\n> > -rw-rw-r-- 1 tglx ftpadmin 158679705 Sep 14 22:16 pack-4d27038611fe7755938efd4a2745d5d5d35de1c1.pack\n> \n> And I get the same result as well.\n\nFor the curious, a 100/100 parameter gives a size of 154261771.\nDisminishing returns, here I come.\n\n  OG.\n"},{"id":"26975","messageId":"Pine.LNX.4.64.0609151232370.2627@xanadu.home","threadId":"5560","inReplyTo":"20060915090305.GC75256@dspnet.fr.eu.org","subject":"Re: Historical kernel repository size","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2006-09-15T16:45:53Z","receivedAt":"2006-09-15T16:45:53Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Fri, 15 Sep 2006, Olivier Galibert wrote:\n\n> On Thu, Sep 14, 2006 at 09:19:04PM -0400, Nicolas Pitre wrote:\n> > Erm... Both incantations work fine fine here.\n> > \n> > > -rw-rw-r-- 1 tglx ftpadmin  13600376 Sep 14 22:16 pack-4d27038611fe7755938efd4a2745d5d5d35de1c1.idx\n> > > -rw-rw-r-- 1 tglx ftpadmin 158679705 Sep 14 22:16 pack-4d27038611fe7755938efd4a2745d5d5d35de1c1.pack\n> > \n> > And I get the same result as well.\n> \n> For the curious, a 100/100 parameter gives a size of 154261771.\n\nRight.   And then the runtime cost of extracting objects out of such a \npack increases due to the deeper delta chain.\n\nThe average runtime cost is probably linear with the delta depth, \nsomething like f(x) = a*x + k.\n\nBut the size reduction follows f(x) = a/x + k.\n\nSo to say that infinite delta length does not provide infinite packing \nsize reduction. Anything larger than 50 is probably not worth the \nsmall reduction gain.\n\n\nNicolas\n"}]}