Re: git-reflog 70 minutes at 100% cpu and counting

From: Nicolas Pitre <nico@fluxnic.net>
Date: Dec 17, 2009, 05:38 UTC
Message-ID: <alpine.LFD.2.00.0912161841210.23173@xanadu.home>
In-Reply-To: <1261003041.2788.142.camel@localhost>

On Wed, 16 Dec 2009, Eric Paris wrote:

Show 32 quoted lines

> On Wed, 2009-12-16 at 16:06 -0500, Nicolas Pitre wrote:
> > On Wed, 16 Dec 2009, Eric Paris wrote:
> > 
> > > On Tue, 2009-12-15 at 22:03 -0500, Nicolas Pitre wrote:
> > > > On Mon, 14 Dec 2009, Eric Paris wrote:
> > > > 
> > > > > The alternative repo is slowing pushing up to that same location.  That
> > > > > tar is 855838982, so just a tad bit smaller.
> > > > 
> > > > It doesn't appear to be complete yet, and not progressing either.
> > > 
> > > The alternative repo is now available (but the original is down)
> > > 
> > > I tried to run git gc --aggressive last night while I slept and got this
> > > as output, maybe it helps point to a solution/problem?  The git reflog
> > > portion ran for 5 hours and 36 minutes and appears to have finished.
> > 
> > Yes.  I was able to reproduce your issue.  And because of the *horrible* 
> > repository packing, the reflog expiration process is taking ages when 
> > determining object reachability at a rate of one reflog entry every 2 
> > seconds or so.  With 4214 entries for the fsnotify-syscall branch, and 
> > 1352 entries for the fsnotify branch, this already takes up asignificant 
> > portion of the actual run time.  I'm sure if your repository was 
> > properly packed this would take less than a minute.
> 
> I'm guessing this is a result of stgit.?  These branches really should
> be just a branch from a tag (which exists in kernel-1) and about 30-50
> patches linearly applied on top.  I don't know how I get that many
> objects.  I'm guessing many/most of them are crap that should be able to
> be cleaned/deleted entirely as the rebasing/pushing/poping/updating that
> stgit does under the covers should have rendered them pointless.  Not
> really sure when/how that should/could have happened.

Possible. Commit operations (including patch applications) always create loose objects because this is fast, with the expectation that they get collected in a pack later.

> Should I be running git-gc every night?

This is certainly a good thing to do given your heavy stgit usage.

Show 10 quoted lines

> > Now, repacking doesn't work because...
> > 
> > > $ git gc --aggressive
> > > error: Could not read d936ff8a7b0841b51ddf96afa24a30b016824cb2
> > > error: Could not read 29b6c2fb1390b4fd350a5ecc78f1156fc5d91e9f
> 
> /me is pretty git dumb, but is there some way to figure out the parents
> or children of these?  I just trolled through all of my directories
> doing git show and didn't get any hits.  I guess I'll just clean up and
> start over....

Moving the reflog data aside (i.e. mv .git/logs .git/logs.bak) it seems that d936ff8 is not referenced anymore.

I found the other one as follows:

First I tried

$ git rev-list --all --objects

This resulted in:

[...] 4f7911b0b0dbd187131a109cf00161a0c6a9d727 arch/x86 ea868257c1eabc31e0ea7941efa42b543978b3fa arch/x86/kvm a0c11ead723956c667172a9f3fb6787684fe7ff5 arch/x86/kvm/paging_tmpl.h b556b6aad8b1aacfecb1dd4a56dbd389674687b5 arch/x86/kvm/x86.c 68a9733ae3315d7e2bfec2037dfeee4db8a6f6a1 drivers error: Could not read 29b6c2fb1390b4fd350a5ecc78f1156fc5d91e9f fatal: bad tree object 29b6c2fb1390b4fd350a5ecc78f1156fc5d91e9f

Because of the way objects are enumerated, we can be pretty sure that the bad tree object is referenced by the tree object 68a9733a corresponding to drivers/. Let's verify that:

$ git ls-tree 68a9733a 100644 blob 00cf9553f74065291612b0971337f79995933a06 Kconfig 100644 blob c1bf41737936ab00be4a87563a0bb0638074785d Makefile 040000 tree d4e847de9bf2450842936582ea7cc6778413825b accessibility 040000 tree 29b6c2fb1390b4fd350a5ecc78f1156fc5d91e9f acpi [...]

Yep, we found it there. So the missing tree object corresponds to drivers/acpi/. So to find the latest commit to which this particular tree object is referenced by, we just need to look at the same rev-list output above (piped into less is handy here) and scroll up until an object with no name is found. This would usually be the first root tree object referencing the named objects that follow. Here I get aafb68eb. To be sure, let's list it so to confirm it really contains a reference to the 68a9733a drivers tree:

$ git ls-tree aafb68eb [...] 040000 tree 68a9733ae3315d7e2bfec2037dfeee4db8a6f6a1 drivers [...]

So yes, we've got the right root tree object. Now finding the corresponding commit should be easy:

$ git log --all --pretty=raw

Then within less, a search for aafb68eb brings us to this:

commit 2e765e9c87a337131aad3014f9a7e5e878c7d0a0 tree aafb68eb84f96c9ab5697c6e8d10d5006d1e7209 parent a2c2de42295b3ac29758f454a7072338e5555ca3 author Eric Paris <eparis@redhat.com> 1237233261 -0400 committer Eric Paris <eparis@redhat.com> 1237233261 -0400

    refresh     64d34c511b1125d9efd2926e683e019f15dec5b4

So this is referenced by a commit that you made on the 1237233261th second since January 1, 1970 i.e. 2009-03-16 19:54:21 +0000 which is quite a while ago. Or given the nature of the commit log, this is probably some stgit branch.

Note that the missing tree didn't necessarily appear with that commit. Because of the recency ordering from rev-list, all we can say is that this is the last commit on that particular branch to reference that tree, but it might have been introduced in the repository way before that point in time.

Now let's try to find out what branch(es) actually link(s) to this commit:

$ git branch -a --contains 2e765e9c

This comes empty. This is because 'git branch' looks only in the refs/heads/ and refs/remotes namespace (or only one of them without -a).

Scripting something around 'git for-each-ref' and 'git merge-base' could be done, such as:

	TARGET=2e765e9c87a337131aad3014f9a7e5e878c7d0a0
	git for-each-ref refs/* |
	while read sha1 type ref; do
		if [ "$(git merge-base $sha1 $TARGET)" = "$TARGET" ]; then
			echo "referenced by $type $ref"
		fi
	done

But this is slow, for the same reason as 'git reflog expire' above. But letting it run for a while should give you at least one answer.

Show 11 quoted lines

> > Of course, usage of alternates is recommended _only_ with repositories 
> > that are stable, i.e. don't ever add repositories to 
> > .git/objects/info/alternates if those repositories are rewinded/rebased 
> > and/or branches in them are deleted/replaced.  That could be a reason 
> > why some objects are now missing from the repository using alternates.
> 
> So I'm not sure how I did things wrong.  my kernel-1 has those bunch of
> remotes.  The linux-next remote, like I said, basically rebases to
> linus' tree, then merges 150 random branches.  It tags that tree every
> day and I pull those tags.  So I would never expect any objects from
> those remote trees to ever disappear.

Right.

Show 22 quoted lines

> Now I created branches in kernel-1 and I certainly have done lots of
> things like so
> 
> git checkout -b testing remotes/linux-next/master
> [edit]
> git commit -a
> git checkout -b testing1 remotes/linux-next/master
> git branch -D testing
> 
> My assumption though was that this wouldn't ever affect my other
> repositories.  My other repository branches always started by checking
> out a branch with remotes/*/* as the base.
> 
> My understanding was that I would only run into problems if I used
> something on a branch I created myself in the alternatives repo in other
> repos (and I didn't remove remotes)
> 
> I guess it's not impossible to believe that at some point in time i
> would have exported patches to and mbox from kernel-1 and applied them
> to kernel-2 or vice versa.  I guess this would end up with the same
> objects, right?  Then if I deleted the branch in kernel-1 I would have
> problems in kernel-2?

Eventually, yes. After a while the auto repack in kernel2 would notice that some objects are in kernel1 already and purge them from kernel2. And if those objects were part of a deleted branch then kernel1 would get rid of those objects too once the reflog with a reference to that deleted branch expires. The unsuspecting kernel2 repo then gets broken.

Show 8 quoted lines

> I guess I'll rebuild my setup
> 
> new kernel-alt has just the remotes, and my kernel-1,2,3 all alt to it
> I'll never have local branches in my kernel-alt
> I'll run git-gc every night
> I'll hope to never have problem again.
> 
> Sound good?

Yes. And make sure not to fetch rebasing repositories, such as linux-next, into kernel-alt without keeping a tag for each fetched state otherwise you'll accumulate unreferenced objects which the other repositories might rely upon.

Nicolas