Re: [PATCH 0/9] Make git-svn fetch ~1.7x faster

From: Sam Vilain <sam@vilain.net>
Date: Oct 24, 2007, 00:43 UTC
Message-ID: <471E9547.2030100@vilain.net>
In-Reply-To: <20071023060812.GA30978@glandium.org>

Mike Hommey wrote:

Show 9 quoted lines

> On Mon, Oct 22, 2007 at 10:46:28PM -0700, Adam Roben wrote:
>> This patch series makes git-svn fetch about 1.7x faster by reducing the number
>> of forks/execs that occur for each file retrieved from Subversion. To do so, a
>> few new options are added to git-cat-file and git-hash-object to allow
>> continuous input on stdin and continuous output on stdout, so that one instance
>> of each of these commands can be kept running for the duration of the fetch.
> 
> You don't need to do this to avoid forks. Just use git-fast-import
> instead.

git-fast-import only covers the hash-object side of things, not cat-file.

git-fast-import does not currently suit 'gradual deployment' for converters such as git-svn, because it;

  - returns object IDs at the end, when you checkpoint.

    This could be 'fixed' by allowing a marks log file instead of or in
    addition to the current behaviour, though if the exporter is
    continually waiting for the tokens rather than using marks, it will
    slow it down.

  - you can't use plumbing commands, such as rev-parse, cat-file, etc on
    objects which have not been checkpointed yet.

  - can't just stream a file of unknown length to it as you can to
    hash-object

These are the design trade-offs of using fast-import. Using fast-import, you are creating a 'transaction' area which uses user sequences instead of (git)database-issued identifiers. And this transaction is isolated from the other concurrent users of the object database. However the interface does not have the full git CLI available to it, so unlike a regular database transaction, you end up having to care.

Rewriting the importer so as to correctly deal with these problems is quite challenging, and for slow import sources such as Subversion, of limited merit.

Sam.