Re: Some git performance measurements..

From: Linus Torvalds <torvalds@linux-foundation.org>
Date: Nov 30, 2007, 06:03 UTC
Message-ID: <alpine.LFD.0.9999.0711292145110.8458@woody.linux-foundation.org>
In-Reply-To: <7v3auos4yi.fsf@gitster.siamese.dyndns.org>

On Thu, 29 Nov 2007, Junio C Hamano wrote:

> 
> I am hoping that "probably 10s of those 17s" can actually be measured
> with the patch I sent out last night.  Has anybody took a look at it?

Sorry, I missed it. But I just did timings.

Your patch helps

	git read-tree -m -u --exclude-per-directory=.gitignore HEAD HEAD

timings enormously, and it's now down to 3s for me (which is the same speed as it is without any per-directory-excludes). That's a big improvement from the ~10s I see without your patch (I've repacked my tree, I have to admit that I don't even know if it's the new or the old older, but I can state that 7s for me was just those .gitignore files).

Sadly, the full "git checkout" itself is not actually improved, due to the

	git update-index --refresh

there, which will end up populating the whole directory cache anyway.

I wonder why I didn't see that as the expensive operation when I timed "git checkout". Probably because I narrowed down on the "git read-tree" as the operation that actually accesses the pack-file and the object directory, while the "git update-index" never touches the actual objects.

Anyway, I think your patch is great. It just doesn't help the full case of a "git checkout", only the read-tree portion of it ;(

As to partitioning the data according to types:

Show 5 quoted lines

> When I do archaeology, I think I often run blame first to see which
> change made the block of text into the current shape first, and then run
> a path limited "git log -p" either starting or ending at that revision.
> In that workflow, the initial blame may get slower with the new layout,
> but I suspect it would help by speeding up the latter "git log -p" step.

I really cannot convince myself one way or the other. I have a suspicion that sometimes it helps to have objects (regardless of type) close to each other, and sometimes it helps to have the trees packed densely. A lot of operations *do* work on both blobs and trees (a *raw* diff doesn't, but they are fairly rare), so this is not at all clear-cut like the commit case.

So sorting the commits together is a no-brainer, since a lot of really important ops only look at them. But blobs and trees? The numbers certainly go both ways, and I suspect we are probably better off not messing with the sort order unless we have some unambiguous real results.

Oh, well. I was hoping that I'd have a number of cases that showed good improvements, with perhaps the bulk of it not showing much difference at all. But while I saw the good improvements, the very first try at "git blame" also showed quite worse numbers, so I think we should consider it an interesting idea, but probably shelve it.

			Linus