Re: question about: Facebook makes Mercurial faster than Git

From: Ondřej Bílka <neleai@seznam.cz>
Date: Mar 10, 2014, 17:51 UTC
Message-ID: <20140310175102.GA17336@domone.podge>
In-Reply-To: <alpine.DEB.2.02.1403100310080.25193@nftneq.ynat.uz>

On Mon, Mar 10, 2014 at 03:13:45AM -0700, David Lang wrote:

Show 19 quoted lines

> On Mon, 10 Mar 2014, Dennis Luehring wrote:
> 
> >according to these blog posts
> >
> >http://www.infoq.com/news/2014/01/facebook-scaling-hg
> >https://code.facebook.com/posts/218678814984400/scaling-mercurial-at-facebook/
> >
> >mercurial "can" be faster then git
> >
> >but i don't found any reply from the git community if it is a real problem
> >or if there a ongoing (maybe git 2.0) changes to compete better in this case
> 
> As I understand this, the biggest part of what happened is that
> Facebook made a tweak to mercurial so that when it needs to know
> what files have changed in their massive tree, their version asks
> their special storage array, while git would have to look at it
> through the filesystem interface (by doing stat calls on the
> directories and files to see if anything has changed)
>

That is mostly a kernel problem. Long ago there was proposed patch to add a recursive mtime so you could check what subtrees changed. If somebody ressurected that patch it would gave similar boost.

There are two issues that need to be handled, first if you are concerned about one mtime change doing lot of updates a application needs to mark all directories it is interested on, when we do update we unmark directory and by that we update each directory at most once per application run.

Second problem were hard links where probably a best course is keep list of these and stat them separately.