Re: fast-import and unique objects.

From: Shawn Pearce <spearce@spearce.org>
Date: Aug 8, 2006, 22:45 UTC
Message-ID: <20060808224537.GB18163@spearce.org>
In-Reply-To: <9e4733910608080511t5aa96865p41d6bc1b85e236fa@mail.gmail.com>

Jon Smirl <jonsmirl@gmail.com> wrote:

> We're designing a dumpfile format for git like the one SVN has.

I'm not sure I'd call it a dumpfile format. More like an importfile format. Reading a GIT pack is really pretty trivial; if someone was going to write a parser/reader to pull apart a GIT repository and use that information in another way they would just do it against the pack files. Its really not that much code. But generating a pack efficiently for a large volume of data is slightly less trivial; the attempt here is to produce some tool that can take a relatively trivial data stream and produce a reasonable (but not necessarily absolute smallest) pack from it in the least amount of CPU and disk time necessary to do the job. I would hope that nobody would seriously consider dumping a GIT repository back INTO this format!

[snip]

> AFAIK the svn code doesn't do merge commits. We probably need a post
> processing pass in the git repo that finds the merges and closes off
> the branches. gitk won't be pretty with 1,500 open branches. This may
> need some manual clues.

*wince* 1500 open branches.  Youch.  OK, that answers a lot of
questions for me with regards to memory handling in fast-import.
Which you provide excellent suggestions for below.  I guess I didn't
think you had nearly that many...
 
[snip]

> The file names are used over and over. Alloc a giant chunk of memory
> and keep appending the file name strings to it. Then build a little
> tree so that you can look up existing names. i.e. turn the files names
> into atoms. Never delete anything.

Agreed.  For 1500 branches its worth doing.
 
[snip]

Show 6 quoted lines

> About 100,000 files in the initial change set that builds the repo.
> FInal repo has 120,000 files.
> 
> There are 1,500 branches. I haven't looked at the svn dump file format
> for branches, but I suspect that it sends everything on a branch out
> at once and doesn't intersperse it with the trunk commits.

If you can tell fast-import your are completely done processing a branch I can recycle the memory I have tied up for that branch; but if that's going to be difficult then... hmm.

Right now I'm looking at around 5 MB/branch, based on implementing the memory handling optimizations you suggested. That's still *huge* for 1500 branches. I clearly can't hang onto every branch in memory for the entire life of the import like I was planning on doing. I'll kick that around for a couple of hours and see what I come up with.

-- 
Shawn.