threads / discuss / 12988

git repository size vs. subversion repository size

Subject: git repository size vs. subversion repository size

## tl;dr

11 messages between Apr 4, 2008 and Apr 14, 2008.

replies: 10people: 7as markdown or json

Sean Brown· Apr 4, 2008, 22:02 UTC · lore

Last night I decided to see what storage size differences I might see between an svn repo and a git one. So I imported a highly used subversion repository into git and was shocked to see how huge the git version was. I used a repo that has a lot of branches and tagged releases just to make sure importing into git would in fact keep all of the history. It did keep the history, but the total disk usage was very different:

$subversionbox # du -hs ./my_sample_website/ 67M ./my_sample_website

$localhost # du -hs ./git-samplesite/ 3.6GB ./git-samplesite/

Here are the steps I took (locally):

mkdir git-samplesite-tmp cd git-samplesite-tmp git-svn init http://subversion.myco.com/my_sample_website --no-metadata git config svn.authorsfile ~/Desktop/users.txt # mapped svn users to git users git-svn fetch git clone git-samplesite-tmp git-samplesite

I did this based on reading the documents in the git wiki, so I assumed they were "best practice." Did I do something wrong? If this is a normal amount of storage need increase, we'd likely not move to git based on the need for new hardware alone.

Any help would be appreciated.
Sean
Björn Steinbrink· Apr 4, 2008, 22:17 UTC · re: Sean Brown · lore

Re: git repository size vs. subversion repository size

On 2008.04.04 18:02:56 -0400, Sean Brown wrote:
Show 13 quoted lines
> Last night I decided to see what storage size differences I might see
> between an svn repo and a git one.  So I imported a highly used
> subversion repository into git and was shocked to see how huge the git
> version was.  I used a repo that has a lot of branches and tagged
> releases just to make sure importing into git would in fact keep all
> of the history.  It did keep the history, but the total disk usage was
> very different:
> 
> $subversionbox # du -hs ./my_sample_website/
> 67M	./my_sample_website
> 
> $localhost # du -hs ./git-samplesite/
> 3.6GB ./git-samplesite/

How much of that is in the .git/svn directory? The contents of that directory are used to map git commits to svn revision and git versions before 1.5.4 had a quite space consuming file format for that. The new format is a lot better. If you want to switch completely, you can even just delete the .git/svn directory, as that's only required as long as you want to interact with the corresponding svn repository.

And finally, you might want to repack to repository once after the initial import, to get a smaller repo. Something like: git repack -a -d -f --window=100 --depth=100

Björn
Stephen Bannasch· Apr 4, 2008, 23:49 UTC · re: Björn Steinbrink · lore

Re: git repository size vs. subversion repository size

I'm just fooling around with git so far but I found a huge space savings after running git gc. Here are the rough numbers:

svn repo on server: 1GB svn repo checked out: 2GB git svn clone after gc: 384MB

That's saving the full history in git -- about 13000 revisions.
Using git version 1.5.4.4.
Steven Walter· Apr 5, 2008, 00:01 UTC · re: Stephen Bannasch · lore

Re: git repository size vs. subversion repository size

On Fri, Apr 04, 2008 at 07:49:24PM -0400, Stephen Bannasch wrote:
Show 8 quoted lines
> I'm just fooling around with git so far but I found a huge space savings 
> after running git gc. Here are the rough numbers:
>
> svn repo on server:        1GB
> svn repo checked out:      2GB
> git svn clone after gc:  384MB
>
> That's saving the full history in git -- about 13000 revisions.

git-gc is such an important step in importing a repository from svn. Why doesn't git-svn take this vital step automatically?

-- 
-Steven Walter <stevenrwalter@gmail.com>
Freedom is the freedom to say that 2 + 2 = 4
B2F1 0ECC E605 7321 E818  7A65 FC81 9777 DC28 9E8F
Stephen Bannasch· Apr 5, 2008, 00:04 UTC · re: Steven Walter · lore

Re: git repository size vs. subversion repository size

Show 12 quoted lines
>On Fri, Apr 04, 2008 at 07:49:24PM -0400, Stephen Bannasch wrote:
>> I'm just fooling around with git so far but I found a huge space savings
>> after running git gc. Here are the rough numbers:
>>
>> svn repo on server:        1GB
>> svn repo checked out:      2GB
>> git svn clone after gc:  384MB
>>
>> That's saving the full history in git -- about 13000 revisions.
>
>git-gc is such an important step in importing a repository from svn.
>Why doesn't git-svn take this vital step automatically?
I think because it is not necessary for continued productive use of git and the gc operation is expensive. On the repo above it took about 8 hours running in the background while I was working on other stuff.
Björn Steinbrink· Apr 5, 2008, 00:18 UTC · re: Steven Walter · lore

Re: git repository size vs. subversion repository size

[Stephan, please stop dropping me from Cc:, thanks]
On 2008.04.04 20:01:41 -0400, Steven Walter wrote:
Show 12 quoted lines
> On Fri, Apr 04, 2008 at 07:49:24PM -0400, Stephen Bannasch wrote:
> > I'm just fooling around with git so far but I found a huge space savings 
> > after running git gc. Here are the rough numbers:
> >
> > svn repo on server:        1GB
> > svn repo checked out:      2GB
> > git svn clone after gc:  384MB
> >
> > That's saving the full history in git -- about 13000 revisions.
> 
> git-gc is such an important step in importing a repository from svn.
> Why doesn't git-svn take this vital step automatically?

Starting from 1.5.4 (IIRC) git-svn will repack every 1000 revisions (by default). That won't give you a reeeeally tiny pack but OTOH it won't take ages to do the repacks either.

Björn
Eric Hanchrow· Apr 14, 2008, 15:28 UTC · re: Steven Walter · lore

Re: git repository size vs. subversion repository size

>>>>> "Steven" == Steven Walter <stevenrwalter@gmail.com> writes:
    Steven> git-gc is such an important step in importing a repository
    Steven> from svn.  Why doesn't git-svn take this vital step
    Steven> automatically?
Mine did:
    git-svn version 1.5.5 (svn 1.3.2)
    git-svn init file://$HOME/svn-repos
    git-svn fetch
...
            M	trunk/home/local/bin/spam/print-subjects.ss
    r5480 = b3edab03f5bacda1db025bd2cca769abbe007f23 (git-svn)
    Auto packing your repository for optimum performance. You may also
    run "git gc" manually. See "git help gc" for more information.
    Counting objects: 11182, done.
    Compressing objects: 100% (11021/11021), done.
    Writing objects: 100% (11182/11182), done.
    Total 11182 (delta 9378), reused 0 (delta 0)
    Checked out HEAD:
      file:///home/erich/svn-repos r5480
    $ du -sh /tmp/ya/.git/ ~/svn-repos/
    23M	/tmp/ya/.git/
    88M	/home/erich/svn-repos/
-- 
The old graybeards in the Smalltalk world may not seem
relevant, but if you ask them a question about ORM, they have been
thinking about it for 20 years.
        -- Avi Bryant
Sean Brown· Apr 5, 2008, 02:27 UTC · re: Björn Steinbrink · lore

Re: git repository size vs. subversion repository size

On Fri, Apr 4, 2008 at 6:17 PM, Björn Steinbrink <B.Steinbrink@gmx.de> wrote:
Show 26 quoted lines
> On 2008.04.04 18:02:56 -0400, Sean Brown wrote:
>  > Last night I decided to see what storage size differences I might see
>  > between an svn repo and a git one.  So I imported a highly used
>  > subversion repository into git and was shocked to see how huge the git
>  > version was.  I used a repo that has a lot of branches and tagged
>  > releases just to make sure importing into git would in fact keep all
>  > of the history.  It did keep the history, but the total disk usage was
>  > very different:
>  >
>  > $subversionbox # du -hs ./my_sample_website/
>  > 67M   ./my_sample_website
>  >
>  > $localhost # du -hs ./git-samplesite/
>  > 3.6GB ./git-samplesite/
>
>  How much of that is in the .git/svn directory? The contents of that
>  directory are used to map git commits to svn revision and git versions
>  before 1.5.4 had a quite space consuming file format for that. The new
>  format is a lot better. If you want to switch completely, you can even
>  just delete the .git/svn directory, as that's only required as long as
>  you want to interact with the corresponding svn repository.
>
>  And finally, you might want to repack to repository once after the
>  initial import, to get a smaller repo. Something like:
>  git repack -a -d -f --window=100 --depth=100
>

The svn folder (in the.git directory) was only about 4.2 MB. After running the repack (and then after that git gc as mentioned by another in this thread), it's still about 3.5 GB.

git-samplesite (master)]$ du -hs ./*
2.1G	./branches
1.4G	./tags
 66M	./trunk

The site does have a lot of binary files (PDFs, photographs an such). I suppose we could leave all of the branches and tags in subversion and just move the trunk to git, but I was hoping to make a clean break from subversion.

If anyone has any further suggestions I'd love to hear them.
Sean
-- 
Sean Brown
seanmichaelbrown@gmail.com
Björn Steinbrink· Apr 5, 2008, 02:34 UTC · re: Sean Brown · lore

Re: git repository size vs. subversion repository size

On 2008.04.04 22:27:12 -0400, Sean Brown wrote:
Show 36 quoted lines
> On Fri, Apr 4, 2008 at 6:17 PM, Björn Steinbrink <B.Steinbrink@gmx.de> wrote:
> > On 2008.04.04 18:02:56 -0400, Sean Brown wrote:
> >  > Last night I decided to see what storage size differences I might see
> >  > between an svn repo and a git one.  So I imported a highly used
> >  > subversion repository into git and was shocked to see how huge the git
> >  > version was.  I used a repo that has a lot of branches and tagged
> >  > releases just to make sure importing into git would in fact keep all
> >  > of the history.  It did keep the history, but the total disk usage was
> >  > very different:
> >  >
> >  > $subversionbox # du -hs ./my_sample_website/
> >  > 67M   ./my_sample_website
> >  >
> >  > $localhost # du -hs ./git-samplesite/
> >  > 3.6GB ./git-samplesite/
> >
> >  How much of that is in the .git/svn directory? The contents of that
> >  directory are used to map git commits to svn revision and git versions
> >  before 1.5.4 had a quite space consuming file format for that. The new
> >  format is a lot better. If you want to switch completely, you can even
> >  just delete the .git/svn directory, as that's only required as long as
> >  you want to interact with the corresponding svn repository.
> >
> >  And finally, you might want to repack to repository once after the
> >  initial import, to get a smaller repo. Something like:
> >  git repack -a -d -f --window=100 --depth=100
> >
> 
> The svn folder (in the.git directory) was only about 4.2 MB.  After
> running the repack (and then after that git gc as mentioned by another
> in this thread), it's still about 3.5 GB.
> 
> git-samplesite (master)]$ du -hs ./*
> 2.1G	./branches
> 1.4G	./tags
>  66M	./trunk

Uhm, you forgot to use -s when doing the clone. That would have created real git branches instead of the directories... What you are counting is the size of the checked out, uncompressed files of _all_ branches and _all_ tags (and trunk). The repo size of basically what "du -sh .git" would give.

Björn
Jan Hudec· Apr 13, 2008, 09:57 UTC · re: Björn Steinbrink · lore

Re: git repository size vs. subversion repository size

On Sat, Apr 05, 2008 at 04:34:37 +0200, Björn Steinbrink wrote:
Show 39 quoted lines
> On 2008.04.04 22:27:12 -0400, Sean Brown wrote:
> > On Fri, Apr 4, 2008 at 6:17 PM, Björn Steinbrink <B.Steinbrink@gmx.de> wrote:
> > > On 2008.04.04 18:02:56 -0400, Sean Brown wrote:
> > >  > Last night I decided to see what storage size differences I might see
> > >  > between an svn repo and a git one.  So I imported a highly used
> > >  > subversion repository into git and was shocked to see how huge the git
> > >  > version was.  I used a repo that has a lot of branches and tagged
> > >  > releases just to make sure importing into git would in fact keep all
> > >  > of the history.  It did keep the history, but the total disk usage was
> > >  > very different:
> > >  >
> > >  > $subversionbox # du -hs ./my_sample_website/
> > >  > 67M   ./my_sample_website
> > >  >
> > >  > $localhost # du -hs ./git-samplesite/
> > >  > 3.6GB ./git-samplesite/
> > >
> > >  How much of that is in the .git/svn directory? The contents of that
> > >  directory are used to map git commits to svn revision and git versions
> > >  before 1.5.4 had a quite space consuming file format for that. The new
> > >  format is a lot better. If you want to switch completely, you can even
> > >  just delete the .git/svn directory, as that's only required as long as
> > >  you want to interact with the corresponding svn repository.
> > >
> > >  And finally, you might want to repack to repository once after the
> > >  initial import, to get a smaller repo. Something like:
> > >  git repack -a -d -f --window=100 --depth=100
> > >
> > 
> > The svn folder (in the.git directory) was only about 4.2 MB.  After
> > running the repack (and then after that git gc as mentioned by another
> > in this thread), it's still about 3.5 GB.
> > 
> > git-samplesite (master)]$ du -hs ./*
> > 2.1G	./branches
> > 1.4G	./tags
> >  66M	./trunk
> 
> Uhm, you forgot to use -s when doing the clone. That would have created
No, not the clone, but the git svn init.
Show 10 quoted lines
> real git branches instead of the directories... What you are counting is
> the size of the checked out, uncompressed files of _all_ branches and
> _all_ tags (and trunk). The repo size of basically what "du -sh .git"
> would give.
> 
> Björn
> --
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
-- 
						 Jan 'Bulb' Hudec <bulb@ucw.cz>
Shawn O. Pearce· Apr 5, 2008, 03:11 UTC · re: Sean Brown · lore

Re: git repository size vs. subversion repository size

Sean Brown <seanmichaelbrown@gmail.com> wrote:
Show 12 quoted lines
> 
> Here are the steps I took (locally):
> 
> mkdir git-samplesite-tmp
> cd git-samplesite-tmp
> git-svn init http://subversion.myco.com/my_sample_website --no-metadata
> git config svn.authorsfile ~/Desktop/users.txt   # mapped svn users to git users
> git-svn fetch
> git clone git-samplesite-tmp git-samplesite
> 
> I did this based on reading the documents in the git wiki, so I
> assumed they were "best practice."  Did I do something wrong?

The last command there didn't get you the most efficiently packed repository possible. More recent versions of git-clone will prefer to hardlink all of the loose objects and packs from the source to the destination, so the clone can occur more quickly when they are on the same filesystem.

Really what you want to do here is repack the cloned directory (cd git-samplesite && git repack -a -d -f) and maybe include some aggressive --depth and --window options (e.g. 100/100) if you have some CPU time to burn and are reasonably certain you will be keeping the result. You only have to spend that CPU time once when converting from SVN, and all future clones from this one will benefit.

But your really major disk usage was due to what someone else pointed out, which was missing the "-s" flag to git-svn. So the Git working directory was huge, as we created working files for every single branch and every single tag. Ouch.

-- 
Shawn.

← back to recent threads