Re: Debugging git-commit slowness on a large repo

From: Nguyen Thai Ngoc Duy <pclouds@gmail.com>
Date: Dec 8, 2011, 01:39 UTC
Message-ID: <CACsJy8DiWWr7eo86gzb-XcqfDv4_ENkqWxswTNb-k84xO18c=A@mail.gmail.com>
In-Reply-To: <CB051EFC.2C795%joshua.redstone@fb.com>

On Thu, Dec 8, 2011 at 5:48 AM, Joshua Redstone <joshua.redstone@fb.com> wrote:

Show 5 quoted lines

> Hi Duy,
> Thanks for the documentation link.
>
> git ls-files shows 100k files, which matches # of files in the working
> tree ('find . -type f -print | wc -l').

Any chance you can split it into smaller repositories, or remove files from working directory (e.g. if you store logs, you don't have to keep logs from all time in working directory, they can be retrieved from history).

> I added a 'git read-tree HEAD' before the git-add, and a 'git write-tree'
> after the add.  With that, the commit time slowed down to 8 seconds per
> commit, plus 4 more seconds for the read-tree/add/write-tree ops.  The
> read-tree/add/write-tree each took about a second.

read-tree destroys stat info in index, refreshing 100k entries in index in this case may take some time. Try this to see if commit time reduces and how much time update-index takes

read-tree HEAD update-index --refresh add .... write-tree commit -q

> As an experiment, I also tried removing the 'git read-tree' and just
> having the git-write-tree.  That sped up commits to 0.6 seconds, but the
> overall time for add/write-tree/commit was still 3 to 6 seconds.

overall time is not really important because we duplicate work here (write-tree is done as part of commit again). What I'm trying to do is to determine how much time each operation in commit may take.

-- 
Duy