# git-svn has a _lot_ of metadata

2 messages from 2007-10-16 to 2007-10-16. Participants: Karl Hasselström, Sam Vilain.
Thread: https://gitlist.dev/t/10313

## Karl Hasselström, 2007-10-16 10:22

Subject: git-svn has a _lot_ of metadata
Message-ID: <20071016102259.GB5945@diana.vm.bytemark.co.uk>
URL: https://gitlist.dev/e/20071016102259.GB5945%40diana.vm.bytemark.co.uk

```
I just imported an svn repository with about 120 tags and 140
branches, and with some repacking got the pack file down to a
comfortable 80 MB. However, .git is over 600 MB, owing to about 520 MB
of git-svn metadata. (This wasn't a problem when I only tracked a
handful of branches, since they're only a few megs apiece.)

There appears to be two kinds of metadata that takes up a significant
fraction of the space.

  * An index file is saved for each branch and tag. I presume this
    corresponds to the branch head, and is used to speed up importing
    of new revisions to that branch. However, recreating an index with
    git-read-tree is very fast, so I don't think these need to be
    saved between git-svn runs.

  * A "rev_db" file is saved for each branch and tag. This is a text
    file with one sha1 per line -- I seem to remember that line X of
    this file is the commit sha1 of svn revision X. For revisions that
    didn't touch this branch/tag, there's a line of 40 zeros. And
    since every revision touches just one branch, it's almost all
    zeros unless the number of branches is very small.

    This could probably be stored _much_ more efficiently. Just
    gzipping it with the standard options shrinks it by between a
    factor of 4 (for one of the busiest branches) and 300 (for a tag,
    which is written just once). But I understand that we need quick
    random access here?

The index files should be easy enough to erase between runs, if they
indeed just correspond to the branch head. The rev_db files are
trickier; exactly what kind of lookups are required? Could it perhaps
be done with just one file, instead of one per branch/tag?

-- 
Karl Hasselström, kha@treskal.com
      www.treskal.com/kalle

```

## Sam Vilain, 2007-10-16 13:22

Subject: Re: git-svn has a _lot_ of metadata
Message-ID: <4714BB21.3090300@vilain.net>
URL: https://gitlist.dev/e/4714BB21.3090300%40vilain.net
In-Reply-To: <20071016102259.GB5945@diana.vm.bytemark.co.uk>

```
Karl Hasselström wrote:
>     This could probably be stored _much_ more efficiently. Just
>     gzipping it with the standard options shrinks it by between a
>     factor of 4 (for one of the busiest branches) and 300 (for a tag,
>     which is written just once). But I understand that we need quick
>     random access here?
>   

I understand that patches to enhance git-svn to allow it to store its
metadata via DBI (as git-cvsserver currently does) would be more than
welcome.

I made an example for an old version of git-svn, which stored the
information in a DBM.  However using a set of tables would be much less
of a hack.

http://git.catalyst.net.nz/gw?p=git.git;a=commitdiff;h=dbe3fa060

Treat that change as a reference only; it just shows what places in the
code would need to be touched to support a different storage format.  A
mergeable patch to fix this would probably need to support both methods,
detecting which is available or in use.

Sam.

```