git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Achieving efficient storage of weirdly structured repos

From
RSRoman Shaposhnik <rvs@sun.com>
Date
Apr 3, 2008, 19:42 UTC
Message-ID
<7BE3E865-C30D-49B8-A1D9-898109514990@sun.com>
Hi Guys!

First of all, I must admit that I got seriously hooked up on Git when I started to use it internally for FFmpeg development and my life as far as non company related projects are concerned hasn't been the same ever since. The way Git addresses issues like branches (both local and remote), merges and content tracking are, in my opinion, quite superior to the alternative SCMs such as Mercurial. In fact, that's why when I saw my coworkers struggling with exactly the same issues around NetBeans Mercurial repository my natural instinct was to introduce them to Git and show them that life doesn't have to be that complicated or troublesome.

But before I can do that, there's a little issue that I either need  
to address or
at least understand better. It seems that the storage requirements  
for the
repository that a friend of mine and I created from: http:// 
hg.netbeans.org/main/
got doubled under Git:
     $  du -sh Mercurial- nb-main
     491M   Mercurial- nb-main
     $ du -sh Git-nb-main
     1.1G     Git-nb-main
The repository was created using hg2git (the one based on git-fast- 
import)
and it was GC'ed and REPACK'ed just in case. I spent some time trying
to understand the issue and here's some statistics on the kind of  
objects
that this repository consists of:
     75053 commit
   334572 blob
   750003 tree

The last item (trees) also seem to take the most space and the most reasonable explanation that I can offer is that NetBeans repository has a really weird structure where they have approximately 700 (yes, seven hundred!) top- level subdirectories there. They are clearly Submodules-shy, but that's another issue that I will need to address with them.

So based on my preliminary analysis the biggest culprit seems to be that for every commit at least one pretty hefty tree object needs to be created (remember 700 top level subdirectories). These objects are all in 18-20K range but they are almost identical to each other except for one or two trees that they refer to.

So here's my question: is there anything in Git that I can use to accommodate a repository like http://hg.netbeans.org/main in an efficient manner? Any kind of ideas (like particular command line options for repack, etc.) would be greatly appreciated!

Thanks, Roman.

Next: Linus Torvalds
Message 1 of 14 in “Achieving efficient storage of weirdly structured repos”
  1. Roman ShaposhnikApr 3, 2008
  2. Linus TorvaldsApr 3, 2008
  3. Jakub NarebskiApr 4, 2008
  4. Nicolas PitreApr 4, 2008
  5. Pieter de BieApr 4, 2008
  6. Shawn O. PearceApr 5, 2008
  7. Roman ShaposhnikApr 4, 2008
  8. Linus TorvaldsApr 4, 2008
  9. Roman ShaposhnikApr 6, 2008
  10. Linus TorvaldsApr 6, 2008
  11. Jeff KingApr 6, 2008
  12. Nicolas PitreApr 7, 2008
  13. Jeff KingApr 7, 2008
  14. Nicolas PitreApr 7, 2008

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.