Re: Handling large files with GIT

Btw, some actual numbers: I did the recent kernel networking merge (which is a trivial in-index merge) with the standard three-way

	git-read-tree -m <base> <branch> <branch>

and with the new git-merge-tree to compare performance.

Doing git-read-tree takes ~0.35s, while git-merge-tree took 0.015s.

Now, that's not a really fair comparison, because the end result is very different: the git-read-tree has populated the index, ready for a git-writet-ree, while the git-merge-tree has not.

However, the interesting part is that especially for a trivial merge, we don't actually _want_ to necessarily populate the index, because doing a "git-write-tree" is actually a pretty expensive operation (on the kernel, it will try to write 1000+ directory trees, most of which already exist. Admittedly we don't actually have to write the objects, since we figure out that they already exist, but we have to do the SHA1 calculations to do so).

So if we made the git-merge-tree based merge work entirely on trees all the way, and never even necessarily populate the index at all (unless it has to, due to actual data conflicts that want to be fixed up), that would actually be another performance advantage. The only downside there is that we would literally have to write the resulting tree objects by hand (ie we'd need a new helper for doing that, and another thing to validate).

Anyway, that should almost certainly make it possible to scale up git merges to hundreds of thousands of files without huge performance problems (still, that depends a bit on layout - again, flat directory structures won't scale as well, so it might not be enough for maildir handling).

But just at a guess, I think there's at least an order of magnitude to be had there. So if a maildir merge currently takes an hour, at least we should be able to get it down to a few minutes.

Ben, are you interested in trying this out in your maildir experiments?

		Linus