{"thread":{"id":"10313","subject":"git-svn has a _lot_ of metadata","startedAt":"2007-10-16T10:22:59Z","lastAt":"2007-10-16T13:22:41Z","messageCount":2,"participants":["Karl Hasselström","Sam Vilain"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"56007","messageId":"20071016102259.GB5945@diana.vm.bytemark.co.uk","threadId":"10313","inReplyTo":null,"subject":"git-svn has a _lot_ of metadata","fromName":"Karl Hasselström","fromEmail":"kha@treskal.com","sentAt":"2007-10-16T10:22:59Z","receivedAt":"2007-10-16T10:22:59Z","isPatch":false,"sender":{"key":"kha@treskal.com","avatar":"https://gravatar.com/avatar/f0120c734b5279b345075a28521e1ac66acb20c9913ffe9bf6ae97e53f7f3f13?d=mp&s=160"},"body":"I just imported an svn repository with about 120 tags and 140\nbranches, and with some repacking got the pack file down to a\ncomfortable 80 MB. However, .git is over 600 MB, owing to about 520 MB\nof git-svn metadata. (This wasn't a problem when I only tracked a\nhandful of branches, since they're only a few megs apiece.)\n\nThere appears to be two kinds of metadata that takes up a significant\nfraction of the space.\n\n  * An index file is saved for each branch and tag. I presume this\n    corresponds to the branch head, and is used to speed up importing\n    of new revisions to that branch. However, recreating an index with\n    git-read-tree is very fast, so I don't think these need to be\n    saved between git-svn runs.\n\n  * A \"rev_db\" file is saved for each branch and tag. This is a text\n    file with one sha1 per line -- I seem to remember that line X of\n    this file is the commit sha1 of svn revision X. For revisions that\n    didn't touch this branch/tag, there's a line of 40 zeros. And\n    since every revision touches just one branch, it's almost all\n    zeros unless the number of branches is very small.\n\n    This could probably be stored _much_ more efficiently. Just\n    gzipping it with the standard options shrinks it by between a\n    factor of 4 (for one of the busiest branches) and 300 (for a tag,\n    which is written just once). But I understand that we need quick\n    random access here?\n\nThe index files should be easy enough to erase between runs, if they\nindeed just correspond to the branch head. The rev_db files are\ntrickier; exactly what kind of lookups are required? Could it perhaps\nbe done with just one file, instead of one per branch/tag?\n\n-- \nKarl Hasselström, kha@treskal.com\n      www.treskal.com/kalle\n"},{"id":"56039","messageId":"4714BB21.3090300@vilain.net","threadId":"10313","inReplyTo":"20071016102259.GB5945@diana.vm.bytemark.co.uk","subject":"Re: git-svn has a _lot_ of metadata","fromName":"Sam Vilain","fromEmail":"sam@vilain.net","sentAt":"2007-10-16T13:22:41Z","receivedAt":"2007-10-16T13:22:41Z","isPatch":false,"sender":{"key":"sam@vilain.net","avatar":"https://gravatar.com/avatar/8fc840ca854dbf6f7065b4335e3b934951c1dca3b11db688e95e471901f8f4a8?d=mp&s=160"},"body":"Karl Hasselström wrote:\n>     This could probably be stored _much_ more efficiently. Just\n>     gzipping it with the standard options shrinks it by between a\n>     factor of 4 (for one of the busiest branches) and 300 (for a tag,\n>     which is written just once). But I understand that we need quick\n>     random access here?\n>   \n\nI understand that patches to enhance git-svn to allow it to store its\nmetadata via DBI (as git-cvsserver currently does) would be more than\nwelcome.\n\nI made an example for an old version of git-svn, which stored the\ninformation in a DBM.  However using a set of tables would be much less\nof a hack.\n\nhttp://git.catalyst.net.nz/gw?p=git.git;a=commitdiff;h=dbe3fa060\n\nTreat that change as a reference only; it just shows what places in the\ncode would need to be touched to support a different storage format.  A\nmergeable patch to fix this would probably need to support both methods,\ndetecting which is available or in use.\n\nSam.\n"}]}