Re: git stash takes excessively long when many untracked files present

From: Anders Darander <anders.darander@gmail.com>
Date: Aug 13, 2013, 10:11 UTC
Message-ID: <loom.20130813T120243-481@post.gmane.org>
In-Reply-To: <20130810214453.GA5719@jtriplet-mobl1>

Josh Triplett <josh <at> joshtriplett.org> writes:

Show 6 quoted lines

> [CCing folks involved in the recent "stash-refuse-to-kill" merge.]
> 
> I keep portions of my home directory in git.  I tried to "git stash"
> some local changes, and it ran for several minutes with no progress.  ps
> showed that it was running "git ls-files --killed", which was taking
> 100% CPU, and occasionally reading the disk very slowly.

I've recently got the same problem, though in this case it's my openembedded directory that's giving me those problems. (Having an untracked build-directory of quiet a few GB takes some time).

I worked around it by locally patching git-stash: -------------------------------------------

diff --git a/git-stash.sh b/git-stash.sh
index 85c9e2c..e5a2043 100755
--- a/git-stash.sh
+++ b/git-stash.sh
@@ -263,7 +263,7 @@ save_stash () {
                exit 0
        fi
        if test -z "$untracked$force" &&
-          test -n "$(git ls-files --killed | head -n 1)"
+          test -n "$(git ls-files --killed --directory | head -n 1)"
        then
                say "$(gettext "The following untracked files would NOT be 
saved
                test -n "$GIT_QUIET" || git ls-files --killed | sed 
's/^/\t/'
-------------------------------------------

It seems to work in my extremely limited testing. Though, I'm pretty sure 
that there'll be quite a few error cases... (Especially, as I just made
a naive attempt at patching git-stash, so I could go on with a few other 
things).

Do anyone have any better idea on how to approach this?

> strace shows that git ls-files --killed is doing a full recursive
> enumeration of my entire home directory.  That's a Really Bad Idea:
> 
> ~$ find | wc -l
> 3248997
> ~$ find -type d | wc -l
> 350680
> 
> Not only that, but it also appears to be attempting to stat and open
> several files in every single directory; for instance:
> 
> stat(".ccache/1/3/.git", 0x7fff254bc7a0) = -1 ENOENT (No such file or 
directory)
> open(".ccache/1/3/.git/HEAD", O_RDONLY) = -1 ENOENT (No such file or 
directory)
> stat(".ccache/1/3/.git", 0x7fff254bc770) = -1 ENOENT (No such file or 
directory)
> open(".ccache/1/3/.git/packed-refs", O_RDONLY) = -1 ENOENT (No such file 
or directory)
> 
> (Yes, in that order.)
> 
> I see a lot of room for optimization here.  Most importantly, git
> ls-files --killed really doesn't need to look at any directory entry
> unless something in the index would conflict with it.

I guess that this would be a good optimization. Or, are ls-files --killed 
used in other cases where the current behaviour would be requiered?

Cheers,
Anders