Re: [PATCH] git-svn: Make it scream by minimizing temp files

From: Marcus Griep <marcus@griep.us>
Date: Aug 9, 2008, 01:12 UTC
Message-ID: <489CEF06.7050204@griep.us>
In-Reply-To: <7vd4kjazaz.fsf@gitster.siamese.dyndns.org>

I am working on that right now; however, against master I am getting checksum mismatches with my svn repository, so generating benchmarks against that requires committing a revert of ffe256f9, which makes things even slower. My work comp is running cygwin, and that could be why ffe256f9 is a problem.

I am, however using a smaller repository, namely that of the Boo Programming Language, to run some benchmarks. I'm running it on a Linux box, and I'll publish the results as soon as they are ready.

I'll include:

ffe256f9 and my patch ffe256f9 and no patch revert ffe256f9 and my patch revert ffe256f9 and no patch

Marcus

Junio C Hamano wrote:

Show 27 quoted lines

> Marcus Griep <marcus@griep.us> writes:
> 
>> Currently, git-svn would create a temp file on four occasions:
>> 1. Reading a blob out of the object db
>> 2. Creating a delta from svn
>> 3. Hashing and writing a blob into the object db
>> 4. Reading a blob out of the object db (in another place in code)
>>
>> Any time git-svn did the above, it would dutifully create and then
>> delete said temp file.  Unfortunately, this means that between 2-4
>> temporary files are created/deleted per file 'add/modify'-ed in
>> svn (O(n)).  This causes significant overhead and helps the inode
>> counter to spin beautifully.
>>
>> By its nature, git-svn is a serial beast.  Thus, reusing a temp file
>> does not pose significant problems.  "truncate and seek" takes much
>> less time than "unlink and create".  This patch centralizes the
>> tempfile creation and holds onto the tempfile until they are deleted
>> on exit.  This significantly reduces file overhead, now requiring
>> at most three (3) temp files per run (O(1)).
> 
> Beautifully written analysis of the issue being tackled.
> 
> But optimization patch should be backed by numbers --- do you have a
> benchmark result of some sort that you would want to include here?
> 
>

-- 
Marcus Griep
GPG Key ID: 0x5E968152
——
http://www.boohaunt.net
את.ψο´