Achieving efficient storage of weirdly structured repos
- From
- Roman Shaposhnik <rvs@sun.com>
- Date
- Apr 3, 2008, 19:42 UTC
- Message-ID
- <7BE3E865-C30D-49B8-A1D9-898109514990@sun.com>
Hi Guys!
First of all, I must admit that I got seriously hooked up on Git when I started to use it internally for FFmpeg development and my life as far as non company related projects are concerned hasn't been the same ever since. The way Git addresses issues like branches (both local and remote), merges and content tracking are, in my opinion, quite superior to the alternative SCMs such as Mercurial. In fact, that's why when I saw my coworkers struggling with exactly the same issues around NetBeans Mercurial repository my natural instinct was to introduce them to Git and show them that life doesn't have to be that complicated or troublesome.
But before I can do that, there's a little issue that I either need
to address or
at least understand better. It seems that the storage requirements
for the
repository that a friend of mine and I created from: http://
hg.netbeans.org/main/
got doubled under Git:
$ du -sh Mercurial- nb-main
491M Mercurial- nb-main
$ du -sh Git-nb-main
1.1G Git-nb-mainThe repository was created using hg2git (the one based on git-fast-
import)
and it was GC'ed and REPACK'ed just in case. I spent some time trying
to understand the issue and here's some statistics on the kind of
objects
that this repository consists of:
75053 commit
334572 blob
750003 treeThe last item (trees) also seem to take the most space and the most reasonable explanation that I can offer is that NetBeans repository has a really weird structure where they have approximately 700 (yes, seven hundred!) top- level subdirectories there. They are clearly Submodules-shy, but that's another issue that I will need to address with them.
So based on my preliminary analysis the biggest culprit seems to be that for every commit at least one pretty hefty tree object needs to be created (remember 700 top level subdirectories). These objects are all in 18-20K range but they are almost identical to each other except for one or two trees that they refer to.
So here's my question: is there anything in Git that I can use to accommodate a repository like http://hg.netbeans.org/main in an efficient manner? Any kind of ideas (like particular command line options for repack, etc.) would be greatly appreciated!
Thanks, Roman.