Git benchmark - comparison with Bazaar, Darcs, Git and Mercurial

I have lately added new Git speed benchmark, from Bryan Murdock blog. The repository is bit untypical:

<quote>  
  By performance, I mean that I used the UNIX time command to see how
  long various basic operations took. Performing the various basic
  operations gave me some insight into the usability of each as well.
  For this test I used a directory with 266 MB of files, 258 KB of which
  were text files, with the rest being image files. I know, kind of
  weird to version all those binary files, but that was the project I
  was interested in testing this out on. Your mileage may vary and all
  that. Here’s a table summarizing the real times reported by time(1):
</quote>

If I remember correctly there were some patches to git which tried to better deal with large blobs. In this simple benchmark git was outperformed by Mercurial and even Bazaar-NG a bit.

http://git.or.cz/gitwiki/GitBenchmarks#head-5657b8361895b5a02c0de39337c410e4d8dcdbce http://bryan-murdock.blogspot.com/2007/03/cutting-edge-revision-control.html

On Wed, 1 Aug 2007, Jakub Narebski wrote:

It's almost certainly not the binary blobs.

I think almost all the difference is from the cloning, without repacking the souce or using a local clone.

The default action for a git clone is to create a pack-file, and do a local clone as if you did it over the network. That is obviously much slower than using the "-l" flag for the _clone_ action, but it tends to be better for the end result - since you get a nice packed starting point, and none of the confusion with hardlinks etc.

[ Maybe I'm just a worry-wart, but hardlinking two repos still makes me 
  worried. Even though we never modify the object files. 

  Quite frankly, I almost wish we hadn't ever done "-l" at all, and I 
  cannot really suggest using it. Either use "-s" for the truly shared 
  repository, or use the default pack-generating one. The hardlinking one 
  was simple and made sense, but it's really not very nice.

  But that aversion to "git clone -l" is really totally illogical. The way 
  we do the object handling, hardlinking object files in git is just about 
  the most safe operation you can think of - and I *still* shudder at it ]

Now, I think the "always act as if you were network transparent" by default is great, but especially if you have never run "git gc" to generate a pack to begin with, it's going to be a very costly thing. And I think that's what the numbers show. That's the only op we do a *lot* worse on than we should.

(The "nonconflicting merge" is probably - once more - the diffstat generation that bites us. That's generally the most costly thing of the whole merge, but I *love* the diffstat).

That said, even if he had done a "git gc", to be fair he would have had to include the cost of that first garbage collect in the "initial import", so the end result would have been exactly the same. Git _does_ end up having a very odd performance profile, and while it's optimized for certain thing, the "initial import" is not one of them.

(Which admittedly is a bit odd. The reason I didn't ever seriously even consider monotone was that the initial import was so *incredibly* sucky, and took hours for the kernel. So use "-l" for benchmarks, and damn my "I hate hardlinking repos" idiocy).

So the only way to truly do a fast initial import *and* get a reasonably good initial clone is likely one of:

 - take full advantage of git, and use local branches, instead of 
   bothering with lots of clones.

   I think that this is often the right thing to do, but it's obviously 
   not fair for comparisons, since it's really something different from 
   what's likely available in the other SCM's. But it's the "git way".

 - use "git clone -s" (or "-l").

   I think the hg numbers are the result of hg defaulting to "-l" 
   behaviour.  Which makes sense for hg, since people need to clone more 
   (in git, you'd generally work with local branches instead).

 - or the initial import would be done with some "git fast-import" thing, 
   rather than "git add ." We don't do it now, and the resulting pack-file 
   wouldn't be optimal, but it would be reasonable. It would at least cut 
   down a _bit_ on the clone cost.

The other reaction I took away from that (quite reasonable, I think) comparison is that I think Murdock would have been much happier if git diff defaulted to "-C". We don't do that (for the best of reasons: interoperability), but maybe we should document the "-M/-C" options more.

The options do show up in the man-page, but apparently not obviously enough, since he hadn't noticed.

			Linus

Linus Torvalds <torvalds@linux-foundation.org> writes:

I would call aversion to -l a superstition, while aversion to -s has a sound technical reasons. The latter means you need to know what you are doing --- namely, you are making the clone still dependent on the original.

Junio C Hamano <gitster@pobox.com> writes:

Well, I'd not call the -l aversy a complete superstition: it means that cloning a repository won't provide any redundancy worth noting against file system corruption.

On Tue, Jul 31, 2007 at 10:50:48PM -0700, Junio C Hamano wrote:

So would you accept a patch which adds a git-config variable which specifies whether or not local clones should use hard links by default (defaulting to yes), and which adds a --no-hard-links option to git-clone to override the config option?

I could imagine a situation where if you are using a git repository exclusively on a local system, with no remote repositories to act as backups, where you might want git clone to to make full copies to provide backups in case of filesystem or disk induced corruption. But most of the time there are enough copies of the the repo on other machines that the need for making separate copies of the git objects/packs isn't really needed.

					- Ted

Theodore Tso <tytso@mit.edu> writes:

Are you suggesting to make -l the default for local, in other words? I personally do not make local clone often enough that I am not disturbed having to type extra " -l" on the command line.

But giving a way to force "copy not hardlink" while still avoiding "the same as the networked case by doing pack transfer" overhead may be a good thing to do.

Perhaps if the destination is local,

         - if -s is given, just set up alternates, do nothing else;
         - by default, do "always copy never hardlink";
         - with -l, do "hardlink if possible";

Hmmmm...

On 8/1/07, Junio C Hamano <gitster@pobox.com> wrote:

...as long as the underlying filesystem _supports_ hardlinks.

BTW, we need a warning when falling back to normal copy, if git-clone -l is used. The user _asked_ for a hard-linked clone, but silently got something else. Something like this:

On 8/1/07, Alex Riesen <raa.lkml@gmail.com> wrote:

Err, the other way around, of course.

On 8/1/07, Alex Riesen <raa.lkml@gmail.com> wrote:

On Wed, 01 Aug 2007 03:15:25 -0700, Junio C Hamano wrote:

Personally, I think it would be a great default.

And I think the frequency with which you type this command is not a good metric for deciding if a command-line option should be required.

Instead, the focus should be on having good defaults for a good user experience, (for example, the benchmarking that started this thread that gave a bad first impression of git).

So, just making git-clone go as fast as possible when local, without requiring any additional options from the user, would be a very good thing.

As for the concern that new users might do local clones in the hope to get some redundancy, hopefully the fact that the operation is instantaneous will give plenty of clue to the user that no redundancy has been provided. That should be enough to send the user looking for the documentation to find the --no-hard-links option.

-Carl

On Wed, 1 Aug 2007, Carl Worth wrote:

I suspect it probably *would* make sense to default to "-l". Even if it makes me get goose-bumps. I freely admit that my worries are totally illogical.

We might make it something like: "if you use an url, we don't default to local", so the difference would be that

	git clone file:///directory/to/repo

would work the way it does now, but

	git clone /directory/to/repo

would default to "-l" behaviour. That kind of would make sense (and should be easy to implement: it would be a trivial fixup to "connect.c".

Something like this adds support for "file://". And then git-clone could just do something like

	# if the source is a local directory, default to local
	if [ -d "$src" ]; then
		use_local=yes
	fi

or similar.

		Linus

---
 connect.c |   12 +++++++-----
 1 files changed, 7 insertions(+), 5 deletions(-)

Linus Torvalds <torvalds@linux-foundation.org> writes:

Couldn't git clone http://host/directory/to/repo tell the proxy that it should enter off-line mode and stop updating?

* David Kastrup:

Huh? I don't see how this is relevant to the current thread.

Anyway, I don't think the max-stale cache control mechanism is widely implemented. If you want effective expiry controls, you need to implement them on the server side.

Linus Torvalds <torvalds@linux-foundation.org> writes:

The attached does not default to "-l", but filesystem level copy behaviour, which is what happens with "clone -l" across filesystem boundaries with the current code.

Clone of linux-2.6 repository (the source is well packed)

(hardlink -- obviously, almost no cost) $ /usr/bin/time git clone -l --bare linux-2.6 l-clone.git Initialized empty Git repository in /git/l-clone.git/ 0 blocks 0.55user 1.00system 0:01.56elapsed 99%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (0major+206724minor)pagefaults 0swaps

(same-as-network)
$ /usr/bin/time git clone --bare file://`pwd`/linux-2.6 n-clone.git
Initialized empty Git repository in /git/n-clone.git/
remote: Generating pack...
remote: Counting objects: 1076746
remote: Done counting 1169654 objects.
remote: Deltifying 1169654 objects...
remote:  100% (1169654/1169654) done
Indexing 1169654 objects...
 100% (1169654/1169654) done
remote: Total 1169654 (delta 959223), reused 1160595 (delta 950164)
Resolving 959223 deltas...
 100% (959223/959223) done
172.85user 20.94system 4:25.88elapsed 72%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (6294major+2019874minor)pagefaults 0swaps

(copy -- takes a lot more than hardlink but cheaper than net) $ /usr/bin/time git clone --bare linux-2.6 c-clone.git Initialized empty Git repository in /git/c-clone.git/ 1266644 blocks 0.92user 10.81system 0:38.38elapsed 30%CPU (0avgtext+0avgdata 0maxresident)k 0inputs+0outputs (406major+204775minor)pagefaults 0swaps

I am ambivalent between -l vs no -l.

 * Without -l (i.e. have all objects/ copied via cpio) would not
   catch the source repository corruption, and also risks
   corrupted recipient repository if an alpha-particle hits
   memory cell while indexing and resolving deltas.  As long as
   the recipient is made uncorrupted, you have a good back-up.

 * same-as-network is expensive, but it would catch if the
   source is already corrupted.  It still risks corrupted
   recipient repository.  As long as the recipient is made
   uncorrupted, you have a good back-up.

 * With -l, as long as the source repository is healthy, it is
   very likely that the recipient would be, too.  Also it is
   very cheap.  You do not get any back-up benefit.

None of the method is resilient against the source repository corruption, so let's discount that from the comparison. Then the differences between -l and non -l matters primarily if you value the back-up benefit or not. If you want to use the cloned repository as a back-up, then it is cheaper to do a non -l clone and two git-fsck (source before clone, recipient after clone) than same-as-network clone, especially as you are likely to do a git-fsck on the recipient if you are so paranoid anyway.

Which leads me to believe that being able to use file:/// is probably a good idea, if only for testability, but probably of little practical value, and we can default to -l for everyday use, and paranoids can use non -l as a way to make a back-up.

---

 git-clone.sh               |   61 +++++++++++++++++++++++---------------------
 t/t5500-fetch-pack.sh      |    2 +-
 t/t5700-clone-reference.sh |    2 +-
 t/t5701-clone-local.sh     |   17 ++++++++++++
 4 files changed, 51 insertions(+), 31 deletions(-)

Junio C Hamano <gitster@pobox.com> writes:

Oh, but one does: an overzealous prune or rm -oopswrongoption in one repo does not hurt the other.

Sane enough, I guess.

David Kastrup <dak@gnu.org> writes:

That's not "back-up" benefit I was thinking about. It is more about protecting your data from hardware failure. You physically have bits in two places, preferrably on separate disk drives.

And that is what you do not get from hardlinked clone.

Junio C Hamano <gitster@pobox.com> writes:

Not at the inode/blob level, but at least the directory manipulations of one are safe from the other.

On Friday 03 August 2007, Junio C Hamano wrote:

If one is serious about backing up ones repo to protect it from hardware failure, there is not much use at all in cloning (by copy, hardlink, or otherwise) to a different location on the _same_ filesystem. In order for a backup to be at least marginally useful, it should be on a different disk drive (which you hint at below), or even better; on a different continent...

My point is as follows: One has to clone a repo onto (at least) a different filesystem if one is serious about backup. But if one is cloning to a different filesystem, hardlinking is no longer an option; git _has_ to make a copy of some sort. Therefore we might as well hardlink as long as we're on a single filesystem (since the extra copy would not be worth much, backup-wise).

If the two copies are on separate disk drives (i.e. separate filesystems), you cannot make a hardlink in the first place. If the two copies are on the same filesystem, they're not much more worth than a single copy (backup-wise).

Given the clone-to-same-filesystem(-with-hardlink-capability) scenario (which is the only scenario where we have the option of using hardlinks), we have the following pros and cons when using hardlinks instead of copying:

Pros:
- Hardlink is _much_ faster (for big repos, we're talking orders of 
magnitude faster)

Cons:
- Hardlink will not leave two copies on the disk. But I'm arguing that the 
additional copy will have pretty much _no_ value from a redundancy POV, 
since the copy is still left on the _same_ filesystem. Some would even go 
as far as to say that the second copy provides a false sense of security as 
long as it is located on the same filesystem.

Have fun!

...Johan

On Wed, Aug 01, 2007 at 03:15:25AM -0700, Junio C Hamano wrote:

Yeah, essentially, with a git-config option (and comand-line option) to override the default for those people who are "squeamish" about git clone -l. Linus's suggestion of using file:// as a way to indicate non-local also makes a lot of sense to me.

As I understand it, the main objection with making -s the default is surprising result that could happen if you do a git-prune in the base repository which causes objects which are borrowed from the base repository via .git/objects/info/alternates, right?

What if git clone -s appended the repository which is borrowing objects via alternates to a file located in the base repository, .git/objects/info/shared-repos?

Then git-prune could also use the refs marked in each of the downstream repositories that are sharing objects with base repository and not make those objects go away. That way, git-gc --prune won't do anything surprising to any shared repositories, since it will scan those shared repositories automatically. Would that be considered acceptable? That way you can get instant clones even on filesystems that don't support hard links, such as Windows systems.

The way I would suggest doing it is once we implement support for .git/objects/info/shared-repos is to do the following with git-clone by default:

   	* If the source repo is specified via a file:// URL, per Linus's
          suggestion, do the clone via copying.

	* If the source repo is specified via a local pathname, and
          .git/objects/info/shared-repos in the source repository is
          writeable by the user who is doing the clone, then do a
          clone -s.

	* If the above fails, try clone -l

	* If the above fails, do a clone via copying over a new pack

Would this be acceptable? Patches to do the following should be fairly easy to whip up. Obviously this wouldn't be for 1.5.3. :-)

       	       	    	 	   	- Ted

Theodore Tso wrote:

-s would be a lot safer to use if repack -a -d (as used by git-gc) was smarter. -a -d has the nasty side effect of doing what it seems only prune is intended to do... that is to remove unreferenced objects.

-s usage currently has to be very well thought out, unless you're just using it for a short-lived temporary branch. If this unintended pruning could be avoided then an average user could go about their merry business repacking and git-gc'ing without a care, and only when doing a git-gc --prune would they need to do something special.

-brandon

On 2007-08-01T18:03:50-0400, Theodore Tso wrote:

I would expect /something and file:///something to behave exactly the same way (the latter just having bit extra syntax sugar).

/Allan

On Thu, 2 Aug 2007, Allan Wind wrote:

I do agree that they should be basically the same, but from an implementation standpoint it actually makes a lot of sense to separate them. Also, there's actually a small amount of "logic" in it: the /something is obviously a "raw filename", while the "file:://something" clearly is something a lot more abstract.

I don't actually have a very strong opinion, but I do think that "file://" makes sense regardless (ie the patch I sent out is probably a good idea).

I also strongly dispute that "file://something" is _identical_ to just "something". There's a huge difference, as anybody who has ever tried to do

	cp file://file-A file-B

will have hopefully found out. They may mean the same thing, but they have totally different levels of abstraction, so it does actually make some sense that you end up *cloning* the same thing, but different ways.

		Linus

Junio C Hamano wrote:

That I think it is the best solution, together with support for file:///path/to/repo.git scheme which would turn on old repacking behavior. I'm all for it.

Jakub Narebski wrote:

By the way, with "-l" you have hardlinks only till repack ("git gc"), isn't it?

Junio C Hamano wrote:

About six weeks ago, I finally got around to installing Linux (ubuntu 7.04) on my laptop. Naturally, I cloned my sparse and git repositories over from the Windows XP partition. Unfortunately, that left me with a sparse repo that I could not modify; during the clone cpio copied the object directory, with perhaps a little too much fidelity, which resulted in a .git/objects tree with 555 permissions (both files and directories). [It also set the file timestamps with utime(), BTW]. A quick chmod fixed it up without problem, but still ...

When I cloned the git repo, however, I forgot the -l parameter and git-clone effectively did a "git-fetch-pack --all -k $repo", leaving me with a working, and fully repacked, repository. Nice.

So, I was about to suggest that when invoked with -l, if the object database cannot be linked, due to EXDEV for example, it should fall back to the "fetch-pack" behaviour. Since I don't have access to a large repo, I can't compare the filesystem-copy time versus the fetch-pack time for a "realistic" repo, but I suppose the copy would always be faster. Oh Well.

Just a data point.

ATB,

Ramsay Jones

Linus Torvalds wrote:

http://bryan-murdock.blogspot.com/2007/03/cutting-edge-revision-control.html doesn't tell what is the directory structure of imported files. If it is flat, then git does not use advantage of hierarchical tree structure.

By the way, I guess that "nonconflicting merge" is trivial tree-level merge, as "no changes" merge should be faster (or fast-forward).

About clone: there was "pack loose, copy existing packs" idea. I don't remember what happened with it. At least for local clone it would be nice.

Jakub Narebski <jnareb@gmail.com> writes:

Can you give more details --- I do not recall such an "idea" discussed.

Junio C Hamano wrote:

The idea was to avoid repacking, and just pack loose, unpacked objects (and save this pack if possible), then concatenate all packs and send this concatenated pack as the result. This saves a bit (quite a bit) of CPU at the cost of additional bandwidth usage if packfiles are not optimized.

The only result of the discussion was that it would be fairly easy to send multiple packs concatenated into one pack, without need to add some multi-pack extension, as there would be required minor changes to split "concatenated" packfiles.

Jakub Narebski <jnareb@gmail.com> wrote:

Yes.  And we backed them out more recently.  :-(

A while ago someone had issues with large binary blobs being added to the repository as loose objects (e.g. by git-add/git-update-index). Repacking that repository (for just git-gc or for transport/clone) was ugly as the large binary blob had to be deflated then reinflated to encode it in the packfile. The solution was the core.legacyheaders = false configuration setting, which used packfile encoding for loose objects, thereby allowing the packer to just copy the already compressed data into the output packfile.

Unfortunately we backed that out recently to "simplify the code". We can still read that loose object format, but we cannot create it and during packing we don't copy the data (we deflate/inflate anyway). So we're back to the horrible deflate/inflate problem. That probably explains the large clone time seen by the author.

I wonder if hg realizes that the two repositories are on the same filesystem and automatically uses hardlinks if possible (aka git clone -l). That would easily explain how they can clone so dang fast. Maybe we should do the same in git-clone, its a pretty simple thing to do.

I do have to question the author's timing method. I don't know if this was hot-cache or not, and he doesn't say. I don't know if the system was 100% idle when running these times, or the times were averaged over a few runs. Usually the first run of anything can give inaccurate timings, as for example the executable code may not be paged in from disk. One of the tools may have had a bias as maybe he poked around with that tool first, before starting the timings, so its executables were still hot in cache. Etc.

However assuming everything was actually done in a way that the timings can be accurately relied upon...

Regarding the initial file import it looks like we about broke even with bzr if you add the "initial file import" and "initial commit" times together. Remember we have to hash and compress the data during git-add; bzr probably delayed their equivilant operation(s) until the commit operation. Summing these two times is probably needed to really compare them.

We were also rather close to hg if you again sum the times up. But we do appear to be slower, by about 27s. I guess I find that hard to believe, but sure, maybe hg somehow has a faster codepath for their file revision disk IO than we do. Maybe its because hg is streaming data and we're loading it all in-core first; maybe the author's system had to swap get enough virtual memory for git-add. Maybe it is just because the author's testing methodology was not very good and one or more of these numbers are just bunk.

Our merge time is pretty respectible giving the competition.
Its probably within the margin of error of the author's testing
methodology.