Re: [PATCH 5/6] Introduce the git fast-import-helper

From: Tomas Carnecky <tom@dbservice.com>
Date: Oct 3, 2010, 17:39 UTC
Message-ID: <4CA8BFB7.2050707@dbservice.com>
In-Reply-To: <AANLkTinZ6NCvKeALDBfP4z=ewkwWVwHBk=C_LmXM7OFh@mail.gmail.com>

On 10/3/10 5:53 PM, Sverre Rabbelier wrote:

Show 7 quoted lines

>> I only need two new things from fast-import:
>>  1) support non-numeric marks (and even this is maybe not strictly
>> required)
> 
> If this can be avoided, or worked around somehow, it would be a boon
> to performance. The current marks implementation uses a hash table
> index by the mark number, which is O(1), very efficient.

I also use a hash table (struct hash_table from hash.h). It's indexed by the atom. So it's about equally fast as the existing one but uses slightly more memory. I measured the speed and fih is about 5% slower than fi. Also, I found out that setting max_packfile to 32MB makes the import much faster (from 10 minutes down to 3m to import the sources of git itself).

Show 5 quoted lines

>>  2) dump the mark->sha1 mapping immediately after creating the object
>> (I heard there is a patch somewhere that does just that)
> 
> Why do you need that? Wouldn't the "write created object name to
> stdout" not be sufficient?

I do: fprintf(stdout, "mark :%s %s\n", mark, sha1_to_hex(sha1));
One reason why not just write the plain hash is because that's the same
syntax as the fih accepts in its input. This way you can do:
  $ ( cat marks; cat fast-export-stream ) | git fast-import-helper >> marks
and can restart at any time. Also, making the output a bit more
structured allows it to be easily extended in the future.

tom