{"thread":{"id":"12988","subject":"git repository size vs. subversion repository size","startedAt":"2008-04-04T22:02:56Z","lastAt":"2008-04-14T15:28:47Z","messageCount":11,"participants":["Sean Brown","Björn Steinbrink","Stephen Bannasch","Steven Walter","Shawn O. Pearce","Jan Hudec","Eric Hanchrow"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"73669","messageId":"1086fb5f0804041502s7ab12202y3b0ec3640425eff@mail.gmail.com","threadId":"12988","inReplyTo":null,"subject":"git repository size vs. subversion repository size","fromName":"Sean Brown","fromEmail":"seanmichaelbrown@gmail.com","sentAt":"2008-04-04T22:02:56Z","receivedAt":"2008-04-04T22:02:56Z","isPatch":false,"sender":{"key":"seanmichaelbrown@gmail.com","avatar":"https://gravatar.com/avatar/6ff2a76c5b8cc3ee811b5c75c65ee629bbbf9a16ac61d06d9b7d223c38838c4e?d=mp&s=160"},"body":"Last night I decided to see what storage size differences I might see\nbetween an svn repo and a git one.  So I imported a highly used\nsubversion repository into git and was shocked to see how huge the git\nversion was.  I used a repo that has a lot of branches and tagged\nreleases just to make sure importing into git would in fact keep all\nof the history.  It did keep the history, but the total disk usage was\nvery different:\n\n$subversionbox # du -hs ./my_sample_website/\n67M\t./my_sample_website\n\n$localhost # du -hs ./git-samplesite/\n3.6GB ./git-samplesite/\n\nHere are the steps I took (locally):\n\nmkdir git-samplesite-tmp\ncd git-samplesite-tmp\ngit-svn init http://subversion.myco.com/my_sample_website --no-metadata\ngit config svn.authorsfile ~/Desktop/users.txt   # mapped svn users to git users\ngit-svn fetch\ngit clone git-samplesite-tmp git-samplesite\n\nI did this based on reading the documents in the git wiki, so I\nassumed they were \"best practice.\"  Did I do something wrong?  If this\nis a normal amount of storage need increase, we'd likely not move to\ngit based on the need for new hardware alone.\n\nAny help would be appreciated.\n\nSean\n"},{"id":"73670","messageId":"20080404221728.GA30728@atjola.homenet","threadId":"12988","inReplyTo":"1086fb5f0804041502s7ab12202y3b0ec3640425eff@mail.gmail.com","subject":"Re: git repository size vs. subversion repository size","fromName":"Björn Steinbrink","fromEmail":"b.steinbrink@gmx.de","sentAt":"2008-04-04T22:17:28Z","receivedAt":"2008-04-04T22:17:28Z","isPatch":false,"sender":{"key":"b.steinbrink@gmx.de","avatar":"https://avatars.githubusercontent.com/u/230962?v=4"},"body":"On 2008.04.04 18:02:56 -0400, Sean Brown wrote:\n> Last night I decided to see what storage size differences I might see\n> between an svn repo and a git one.  So I imported a highly used\n> subversion repository into git and was shocked to see how huge the git\n> version was.  I used a repo that has a lot of branches and tagged\n> releases just to make sure importing into git would in fact keep all\n> of the history.  It did keep the history, but the total disk usage was\n> very different:\n> \n> $subversionbox # du -hs ./my_sample_website/\n> 67M\t./my_sample_website\n> \n> $localhost # du -hs ./git-samplesite/\n> 3.6GB ./git-samplesite/\n\nHow much of that is in the .git/svn directory? The contents of that\ndirectory are used to map git commits to svn revision and git versions\nbefore 1.5.4 had a quite space consuming file format for that. The new\nformat is a lot better. If you want to switch completely, you can even\njust delete the .git/svn directory, as that's only required as long as\nyou want to interact with the corresponding svn repository.\n\nAnd finally, you might want to repack to repository once after the\ninitial import, to get a smaller repo. Something like:\ngit repack -a -d -f --window=100 --depth=100\n\nBjörn\n"},{"id":"73672","messageId":"p06240810c41c6e0993c1@[63.138.152.125]","threadId":"12988","inReplyTo":"20080404221728.GA30728@atjola.homenet","subject":"Re: git repository size vs. subversion repository size","fromName":"Stephen Bannasch","fromEmail":"stephen.bannasch@deanbrook.org","sentAt":"2008-04-04T23:49:24Z","receivedAt":"2008-04-04T23:49:24Z","isPatch":false,"sender":{"key":"stephen.bannasch@deanbrook.org","avatar":null},"body":"I'm just fooling around with git so far but I found a huge space \nsavings after running git gc. Here are the rough numbers:\n\nsvn repo on server:        1GB\nsvn repo checked out:      2GB\ngit svn clone after gc:  384MB\n\nThat's saving the full history in git -- about 13000 revisions.\n\nUsing git version 1.5.4.4.\n"},{"id":"73675","messageId":"20080405000141.GA23633@dervierte","threadId":"12988","inReplyTo":"p06240810c41c6e0993c1@[63.138.152.125]","subject":"Re: git repository size vs. subversion repository size","fromName":"Steven Walter","fromEmail":"stevenrwalter@gmail.com","sentAt":"2008-04-05T00:01:41Z","receivedAt":"2008-04-05T00:01:41Z","isPatch":false,"sender":{"key":"stevenrwalter@gmail.com","avatar":"https://avatars.githubusercontent.com/u/79127?v=4"},"body":"On Fri, Apr 04, 2008 at 07:49:24PM -0400, Stephen Bannasch wrote:\n> I'm just fooling around with git so far but I found a huge space savings \n> after running git gc. Here are the rough numbers:\n>\n> svn repo on server:        1GB\n> svn repo checked out:      2GB\n> git svn clone after gc:  384MB\n>\n> That's saving the full history in git -- about 13000 revisions.\n\ngit-gc is such an important step in importing a repository from svn.\nWhy doesn't git-svn take this vital step automatically?\n-- \n-Steven Walter <stevenrwalter@gmail.com>\nFreedom is the freedom to say that 2 + 2 = 4\nB2F1 0ECC E605 7321 E818  7A65 FC81 9777 DC28 9E8F \n"},{"id":"73676","messageId":"p06240813c41c720f8533@[63.138.152.125]","threadId":"12988","inReplyTo":"20080405000141.GA23633@dervierte","subject":"Re: git repository size vs. subversion repository size","fromName":"Stephen Bannasch","fromEmail":"stephen.bannasch@deanbrook.org","sentAt":"2008-04-05T00:04:54Z","receivedAt":"2008-04-05T00:04:54Z","isPatch":false,"sender":{"key":"stephen.bannasch@deanbrook.org","avatar":null},"body":">On Fri, Apr 04, 2008 at 07:49:24PM -0400, Stephen Bannasch wrote:\n>> I'm just fooling around with git so far but I found a huge space savings\n>> after running git gc. Here are the rough numbers:\n>>\n>> svn repo on server:        1GB\n>> svn repo checked out:      2GB\n>> git svn clone after gc:  384MB\n>>\n>> That's saving the full history in git -- about 13000 revisions.\n>\n>git-gc is such an important step in importing a repository from svn.\n>Why doesn't git-svn take this vital step automatically?\n\nI think because it is not necessary for continued productive use of git and the gc operation is expensive. On the repo above it took about 8 hours running in the background while I was working on other stuff.\n"},{"id":"73677","messageId":"20080405001809.GA3101@atjola.homenet","threadId":"12988","inReplyTo":"20080405000141.GA23633@dervierte","subject":"Re: git repository size vs. subversion repository size","fromName":"Björn Steinbrink","fromEmail":"b.steinbrink@gmx.de","sentAt":"2008-04-05T00:18:09Z","receivedAt":"2008-04-05T00:18:09Z","isPatch":false,"sender":{"key":"b.steinbrink@gmx.de","avatar":"https://avatars.githubusercontent.com/u/230962?v=4"},"body":"[Stephan, please stop dropping me from Cc:, thanks]\n\nOn 2008.04.04 20:01:41 -0400, Steven Walter wrote:\n> On Fri, Apr 04, 2008 at 07:49:24PM -0400, Stephen Bannasch wrote:\n> > I'm just fooling around with git so far but I found a huge space savings \n> > after running git gc. Here are the rough numbers:\n> >\n> > svn repo on server:        1GB\n> > svn repo checked out:      2GB\n> > git svn clone after gc:  384MB\n> >\n> > That's saving the full history in git -- about 13000 revisions.\n> \n> git-gc is such an important step in importing a repository from svn.\n> Why doesn't git-svn take this vital step automatically?\n\nStarting from 1.5.4 (IIRC) git-svn will repack every 1000 revisions (by\ndefault). That won't give you a reeeeally tiny pack but OTOH it won't\ntake ages to do the repacks either.\n\nBjörn\n"},{"id":"73680","messageId":"1086fb5f0804041927m5aba1912sc71a19b8f8221e06@mail.gmail.com","threadId":"12988","inReplyTo":"20080404221728.GA30728@atjola.homenet","subject":"Re: git repository size vs. subversion repository size","fromName":"Sean Brown","fromEmail":"seanmichaelbrown@gmail.com","sentAt":"2008-04-05T02:27:12Z","receivedAt":"2008-04-05T02:27:12Z","isPatch":false,"sender":{"key":"seanmichaelbrown@gmail.com","avatar":"https://gravatar.com/avatar/6ff2a76c5b8cc3ee811b5c75c65ee629bbbf9a16ac61d06d9b7d223c38838c4e?d=mp&s=160"},"body":"On Fri, Apr 4, 2008 at 6:17 PM, Björn Steinbrink <B.Steinbrink@gmx.de> wrote:\n> On 2008.04.04 18:02:56 -0400, Sean Brown wrote:\n>  > Last night I decided to see what storage size differences I might see\n>  > between an svn repo and a git one.  So I imported a highly used\n>  > subversion repository into git and was shocked to see how huge the git\n>  > version was.  I used a repo that has a lot of branches and tagged\n>  > releases just to make sure importing into git would in fact keep all\n>  > of the history.  It did keep the history, but the total disk usage was\n>  > very different:\n>  >\n>  > $subversionbox # du -hs ./my_sample_website/\n>  > 67M   ./my_sample_website\n>  >\n>  > $localhost # du -hs ./git-samplesite/\n>  > 3.6GB ./git-samplesite/\n>\n>  How much of that is in the .git/svn directory? The contents of that\n>  directory are used to map git commits to svn revision and git versions\n>  before 1.5.4 had a quite space consuming file format for that. The new\n>  format is a lot better. If you want to switch completely, you can even\n>  just delete the .git/svn directory, as that's only required as long as\n>  you want to interact with the corresponding svn repository.\n>\n>  And finally, you might want to repack to repository once after the\n>  initial import, to get a smaller repo. Something like:\n>  git repack -a -d -f --window=100 --depth=100\n>\n\nThe svn folder (in the.git directory) was only about 4.2 MB.  After\nrunning the repack (and then after that git gc as mentioned by another\nin this thread), it's still about 3.5 GB.\n\ngit-samplesite (master)]$ du -hs ./*\n2.1G\t./branches\n1.4G\t./tags\n 66M\t./trunk\n\nThe site does have a lot of binary files (PDFs, photographs an such).\nI suppose we could leave all of the branches and tags in subversion\nand just move the trunk to git, but I was hoping to make a clean break\nfrom subversion.\n\nIf anyone has any further suggestions I'd love to hear them.\n\nSean\n\n-- \n\nSean Brown\nseanmichaelbrown@gmail.com\n"},{"id":"73681","messageId":"20080405023437.GA8861@atjola.homenet","threadId":"12988","inReplyTo":"1086fb5f0804041927m5aba1912sc71a19b8f8221e06@mail.gmail.com","subject":"Re: git repository size vs. subversion repository size","fromName":"Björn Steinbrink","fromEmail":"b.steinbrink@gmx.de","sentAt":"2008-04-05T02:34:37Z","receivedAt":"2008-04-05T02:34:37Z","isPatch":false,"sender":{"key":"b.steinbrink@gmx.de","avatar":"https://avatars.githubusercontent.com/u/230962?v=4"},"body":"On 2008.04.04 22:27:12 -0400, Sean Brown wrote:\n> On Fri, Apr 4, 2008 at 6:17 PM, Björn Steinbrink <B.Steinbrink@gmx.de> wrote:\n> > On 2008.04.04 18:02:56 -0400, Sean Brown wrote:\n> >  > Last night I decided to see what storage size differences I might see\n> >  > between an svn repo and a git one.  So I imported a highly used\n> >  > subversion repository into git and was shocked to see how huge the git\n> >  > version was.  I used a repo that has a lot of branches and tagged\n> >  > releases just to make sure importing into git would in fact keep all\n> >  > of the history.  It did keep the history, but the total disk usage was\n> >  > very different:\n> >  >\n> >  > $subversionbox # du -hs ./my_sample_website/\n> >  > 67M   ./my_sample_website\n> >  >\n> >  > $localhost # du -hs ./git-samplesite/\n> >  > 3.6GB ./git-samplesite/\n> >\n> >  How much of that is in the .git/svn directory? The contents of that\n> >  directory are used to map git commits to svn revision and git versions\n> >  before 1.5.4 had a quite space consuming file format for that. The new\n> >  format is a lot better. If you want to switch completely, you can even\n> >  just delete the .git/svn directory, as that's only required as long as\n> >  you want to interact with the corresponding svn repository.\n> >\n> >  And finally, you might want to repack to repository once after the\n> >  initial import, to get a smaller repo. Something like:\n> >  git repack -a -d -f --window=100 --depth=100\n> >\n> \n> The svn folder (in the.git directory) was only about 4.2 MB.  After\n> running the repack (and then after that git gc as mentioned by another\n> in this thread), it's still about 3.5 GB.\n> \n> git-samplesite (master)]$ du -hs ./*\n> 2.1G\t./branches\n> 1.4G\t./tags\n>  66M\t./trunk\n\nUhm, you forgot to use -s when doing the clone. That would have created\nreal git branches instead of the directories... What you are counting is\nthe size of the checked out, uncompressed files of _all_ branches and\n_all_ tags (and trunk). The repo size of basically what \"du -sh .git\"\nwould give.\n\nBjörn\n"},{"id":"73682","messageId":"20080405031150.GQ10274@spearce.org","threadId":"12988","inReplyTo":"1086fb5f0804041502s7ab12202y3b0ec3640425eff@mail.gmail.com","subject":"Re: git repository size vs. subversion repository size","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2008-04-05T03:11:50Z","receivedAt":"2008-04-05T03:11:50Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Sean Brown <seanmichaelbrown@gmail.com> wrote:\n> \n> Here are the steps I took (locally):\n> \n> mkdir git-samplesite-tmp\n> cd git-samplesite-tmp\n> git-svn init http://subversion.myco.com/my_sample_website --no-metadata\n> git config svn.authorsfile ~/Desktop/users.txt   # mapped svn users to git users\n> git-svn fetch\n> git clone git-samplesite-tmp git-samplesite\n> \n> I did this based on reading the documents in the git wiki, so I\n> assumed they were \"best practice.\"  Did I do something wrong?\n\nThe last command there didn't get you the most efficiently packed\nrepository possible.  More recent versions of git-clone will prefer\nto hardlink all of the loose objects and packs from the source to\nthe destination, so the clone can occur more quickly when they are\non the same filesystem.\n\nReally what you want to do here is repack the cloned directory\n(cd git-samplesite && git repack -a -d -f) and maybe include\nsome aggressive --depth and --window options (e.g. 100/100)\nif you have some CPU time to burn and are reasonably certain\nyou will be keeping the result.  You only have to spend that\nCPU time once when converting from SVN, and all future clones\nfrom this one will benefit.\n\nBut your really major disk usage was due to what someone else\npointed out, which was missing the \"-s\" flag to git-svn.  So the\nGit working directory was huge, as we created working files for\nevery single branch and every single tag.  Ouch.\n\n-- \nShawn.\n"},{"id":"74251","messageId":"20080413095708.GA16649@efreet.light.src","threadId":"12988","inReplyTo":"20080405023437.GA8861@atjola.homenet","subject":"Re: git repository size vs. subversion repository size","fromName":"Jan Hudec","fromEmail":"bulb@ucw.cz","sentAt":"2008-04-13T09:57:08Z","receivedAt":"2008-04-13T09:57:08Z","isPatch":false,"sender":{"key":"bulb@ucw.cz","avatar":null},"body":"On Sat, Apr 05, 2008 at 04:34:37 +0200, Björn Steinbrink wrote:\n> On 2008.04.04 22:27:12 -0400, Sean Brown wrote:\n> > On Fri, Apr 4, 2008 at 6:17 PM, Björn Steinbrink <B.Steinbrink@gmx.de> wrote:\n> > > On 2008.04.04 18:02:56 -0400, Sean Brown wrote:\n> > >  > Last night I decided to see what storage size differences I might see\n> > >  > between an svn repo and a git one.  So I imported a highly used\n> > >  > subversion repository into git and was shocked to see how huge the git\n> > >  > version was.  I used a repo that has a lot of branches and tagged\n> > >  > releases just to make sure importing into git would in fact keep all\n> > >  > of the history.  It did keep the history, but the total disk usage was\n> > >  > very different:\n> > >  >\n> > >  > $subversionbox # du -hs ./my_sample_website/\n> > >  > 67M   ./my_sample_website\n> > >  >\n> > >  > $localhost # du -hs ./git-samplesite/\n> > >  > 3.6GB ./git-samplesite/\n> > >\n> > >  How much of that is in the .git/svn directory? The contents of that\n> > >  directory are used to map git commits to svn revision and git versions\n> > >  before 1.5.4 had a quite space consuming file format for that. The new\n> > >  format is a lot better. If you want to switch completely, you can even\n> > >  just delete the .git/svn directory, as that's only required as long as\n> > >  you want to interact with the corresponding svn repository.\n> > >\n> > >  And finally, you might want to repack to repository once after the\n> > >  initial import, to get a smaller repo. Something like:\n> > >  git repack -a -d -f --window=100 --depth=100\n> > >\n> > \n> > The svn folder (in the.git directory) was only about 4.2 MB.  After\n> > running the repack (and then after that git gc as mentioned by another\n> > in this thread), it's still about 3.5 GB.\n> > \n> > git-samplesite (master)]$ du -hs ./*\n> > 2.1G\t./branches\n> > 1.4G\t./tags\n> >  66M\t./trunk\n> \n> Uhm, you forgot to use -s when doing the clone. That would have created\n\nNo, not the clone, but the git svn init.\n\n> real git branches instead of the directories... What you are counting is\n> the size of the checked out, uncompressed files of _all_ branches and\n> _all_ tags (and trunk). The repo size of basically what \"du -sh .git\"\n> would give.\n> \n> Björn\n> --\n> To unsubscribe from this list: send the line \"unsubscribe git\" in\n> the body of a message to majordomo@vger.kernel.org\n> More majordomo info at  http://vger.kernel.org/majordomo-info.html\n-- \n\t\t\t\t\t\t Jan 'Bulb' Hudec <bulb@ucw.cz>\n"},{"id":"74358","messageId":"87y77ga21c.fsf@offby1.atm01.sea.blarg.net","threadId":"12988","inReplyTo":"20080405000141.GA23633@dervierte","subject":"Re: git repository size vs. subversion repository size","fromName":"Eric Hanchrow","fromEmail":"offby1@blarg.net","sentAt":"2008-04-14T15:28:47Z","receivedAt":"2008-04-14T15:28:47Z","isPatch":false,"sender":{"key":"eric.hanchrow@gmail.com","avatar":"https://avatars.githubusercontent.com/u/3145?v=4"},"body":">>>>> \"Steven\" == Steven Walter <stevenrwalter@gmail.com> writes:\n\n    Steven> git-gc is such an important step in importing a repository\n    Steven> from svn.  Why doesn't git-svn take this vital step\n    Steven> automatically?\n\nMine did:\n    git-svn version 1.5.5 (svn 1.3.2)\n    git-svn init file://$HOME/svn-repos\n    git-svn fetch\n...\n\n            M\ttrunk/home/local/bin/spam/print-subjects.ss\n    r5480 = b3edab03f5bacda1db025bd2cca769abbe007f23 (git-svn)\n    Auto packing your repository for optimum performance. You may also\n    run \"git gc\" manually. See \"git help gc\" for more information.\n    Counting objects: 11182, done.\n    Compressing objects: 100% (11021/11021), done.\n    Writing objects: 100% (11182/11182), done.\n    Total 11182 (delta 9378), reused 0 (delta 0)\n    Checked out HEAD:\n      file:///home/erich/svn-repos r5480\n\n    $ du -sh /tmp/ya/.git/ ~/svn-repos/\n    23M\t/tmp/ya/.git/\n    88M\t/home/erich/svn-repos/\n-- \nThe old graybeards in the Smalltalk world may not seem\nrelevant, but if you ask them a question about ORM, they have been\nthinking about it for 20 years.\n        -- Avi Bryant\n"}]}