Re: git slow unless piped to cat

From: Uwe Kleine-König <u.kleine-koenig@pengutronix.de>
Date: Aug 27, 2019, 21:32 UTC
Message-ID: <20190827213248.kzpszutt46ipboq4@pengutronix.de>
In-Reply-To: <20190827185929.GA3150@sigill.intra.peff.net>

On Tue, Aug 27, 2019 at 02:59:30PM -0400, Jeff King wrote:

Show 21 quoted lines

> On Tue, Aug 27, 2019 at 12:04:27PM +0200, Uwe Kleine-König wrote:
> 
> > 	$ sudo sh -c "echo 3 > /proc/sys/vm/drop_caches"; time env GIT_CONFIG_NOSYSTEM=1 HOME=/nonexistant XDG_CONFIG_HOME=/nonexistant git --no-pager show --no-color --no-decorate v5.2
> > 	...
> > 
> > 	real	0m1.041s
> > 	user	0m0.000s
> > 	sys	0m0.021s
> > 
> > 	$ git for-each-ref |wc -l
> > 	10013
> > 
> > So this is indeed the problem.
> 
> That's not that many refs. Are they mostly unpacked? Loose refs are
> awful on a cold cache, since it's one inode per ref (or worse, since it
> has walk the directories, too).
> 
> Try:
> 
>   find .git/refs -type f | wc -l

ukl@dude.ptx:/ptx/src/git/linux.git$ find refs -type f | wc -l 2874

(Note this is a bare repository. So I used refs instead of .git/refs.)

Show 5 quoted lines

> to see the loose ones. If there are a lot, try:
> 
>   git pack-refs --prune --all
> 
> (or just "git gc", which does this).

This is a daily updated mirror that is also incrementally backed up. I repack from time to time, but hesitate to do this regularily as each time I repack the next backup run is "expensive" and I get many small packs. I did this now anyhow.

After that I have:

	$ find objects/[0-9a-f][0-9a-f] -type f | wc -l
	166
	$ find objects/pack | wc -l
	98

Show 6 quoted lines

> Modern versions of Git should delete any leftover empty directories
> after pruning the loose variants, but I think won't go out of their way
> to remove ones they didn't delete from. And those are still expensive to
> walk on a cold cache. If it's still slow, try:
> 
>   find .git/refs/* -type d | xargs rmdir -v

There were a few:

$ find refs/ -depth -type d -empty -print -delete | wc -l 147

(Oops, after the repacking above this also removed ./refs which I had to restore to make git recognize this as a repository.)

With these two things done it is a bit faster:

	$ sudo sh -c "echo 3 > /proc/sys/vm/drop_caches"; time env GIT_CONFIG_NOSYSTEM=1 HOME=/nonexistant XDG_CONFIG_HOME=/nonexistant git --no-pager show --no-color v5.2 
	tag v5.2

	...
	real	0m11.022s
	user	0m0.321s
	sys	0m0.248s

Best regards Uwe

-- 
Pengutronix e.K.                           | Uwe Kleine-König            |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |