git-reflog 70 minutes at 100% cpu and counting

So I have no idea what is interesting or relevant what I can collect, what you want to know or anything like that, so this is a bit of a dump of info and I'm sorry to whoever tries to pick anything useful out of it. Somone who understands git might glean some interesting information (or tell me what a fool I am) I'm going to lay out my whole working process here and maybe people will even point out how to improve what I do....

git-1.5.5.6-4.el5 (git in extras for RHEL5)

I have about 5 local trees that way I can work on different things without having to rebuild quite a much especially as I go back and change history so often with stgit. Each local tree has a .git/config file that has about 5 different kernel trees set up as remotes. They look something like this.

[core]
	repositoryformatversion = 0
	filemode = true
	bare = false
	logallrefupdates = true
[remote "origin"]
	url = git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git
	fetch = +refs/heads/*:refs/remotes/origin/*
[remote "linus"]
	url = git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git
	fetch = +refs/heads/master:refs/remotes/linus/master
[remote "linux-next"]
	url = git://git.kernel.org/pub/scm/linux/kernel/git/sfr/linux-next.git
	fetch = +refs/heads/*:refs/remotes/linux-next/*
{snipped}

Each of these trees also has a .git/objects/info/alternatives file which looks like so (am I using alternatives right?)

/export/kernel/kernel-1/.git/objects /storage/kernel/kernel-1/.git/objects

On this particular machine (a very beefy dual quad core Nehalem) /storage is a bind mount of /export. On other machines I will mount these using NFS in which case /export doesn't exist and /storage is the mount point. Typically (but not always) the only thing I do over the NFS mount point is 'make install'.

In this particular tree I use stgit and on a daily basis will update my remotes and rebase my stgit patch series on top of linux-next. I don't know the details of the git commands going on under the covers, I just do git remote update; stg rebase remotes/linux-next/master; I don't know if that's relevant, but it might leave me lots of crap in the tree?

Today I decided to make a clean branch to ask Linus to pull. I exported a patch series (about 80 patches) to an mbox file from one of my other 5 trees and I did the following

git remote update git checkout -b working remotes/linus/master git-am -3 -k mbox.file

all 80 or so patches in the mbox file applied and then I got

Auto packing your repository for optimum performance. You may also run "git gc" manually. See "git help gc" for more information.

I waited for a while, but it still hasn't come back.

#ps -ef | grep git paris 24134 22057 0 14:09 pts/12 00:00:00 /bin/sh /usr/bin/git-am -3 -k /tmp/fanotify.mbox paris 25638 24134 0 14:09 pts/12 00:00:00 git gc --auto paris 25640 25638 99 14:09 pts/12 00:58:07 git-reflog expire --all

#top 25640 paris 25 0 920m 211m 149m R 99.8 2.1 69:09.55 git-reflog

#ls -ld /proc/pid/cwd lrwxrwxrwx 1 paris paris 0 Dec 14 15:02 /proc/25640/cwd -> /storage/kernel/kernel-2

#strace -p -T -ttt 1260821793.751746 stat(".git/objects/b2/ad3c1470e751c53bf7a4d3d53514e0debab1fc", {st_mode=S_IFREG|0444, st_size=291, ...}) = 0 <0.000043> 1260821793.751917 open(".git/objects/b2/ad3c1470e751c53bf7a4d3d53514e0debab1fc", O_RDONLY|O_NOATIME) = 40 <0.000041> 1260821793.752032 mmap(NULL, 291, PROT_READ, MAP_PRIVATE, 40, 0) = 0x2b4d6a90e000 <0.000041> 1260821793.752148 close(40) = 0 <0.000026> 1260821793.752286 munmap(0x2b4d6a90e000, 291) = 0 <0.000035> 1260821793.752538 stat(".git/objects/85/7d99d3a4f9780402fbff3d59b6b3de8d614cc7", {st_mode=S_IFREG|0444, st_size=330, ...}) = 0 <0.000138> 1260821793.752743 open(".git/objects/85/7d99d3a4f9780402fbff3d59b6b3de8d614cc7", O_RDONLY|O_NOATIME) = 40 <0.000027> 1260821793.752942 mmap(NULL, 330, PROT_READ, MAP_PRIVATE, 40, 0) = 0x2b4d6a90e000 <0.000024> 1260821793.753018 close(40) = 0 <0.000038> 1260821793.753289 munmap(0x2b4d6a90e000, 330) = 0 <0.000040> 1260821796.796243 stat(".git/objects/85/7d99d3a4f9780402fbff3d59b6b3de8d614cc7", {st_mode=S_IFREG|0444, st_size=330, ...}) = 0 <0.000076> 1260821796.796440 open(".git/objects/85/7d99d3a4f9780402fbff3d59b6b3de8d614cc7", O_RDONLY|O_NOATIME) = 40 <0.000036> 1260821796.796553 mmap(NULL, 330, PROT_READ, MAP_PRIVATE, 40, 0) = 0x2b4d6a90e000 <0.000031> 1260821796.796624 close(40) = 0 <0.000017> 1260821796.796828 munmap(0x2b4d6a90e000, 330) = 0 <0.000042> 1260821796.797124 stat(".git/objects/40/c92d2149426ea7fd8c70bf7c7727af15eed75d", {st_mode=S_IFREG|0444, st_size=293, ...}) = 0 <0.008584> 1260821796.805844 open(".git/objects/40/c92d2149426ea7fd8c70bf7c7727af15eed75d", O_RDONLY|O_NOATIME) = 40 <0.000114> 1260821796.806062 mmap(NULL, 293, PROT_READ, MAP_PRIVATE, 40, 0) = 0x2b4d6a90e000 <0.000041> 1260821796.806144 close(40) = 0 <0.000018> 1260821796.806341 munmap(0x2b4d6a90e000, 293) = 0 <0.000023> 1260821799.863480 stat(".git/objects/40/c92d2149426ea7fd8c70bf7c7727af15eed75d", {st_mode=S_IFREG|0444, st_size=293, ...}) = 0 <0.000118> 1260821799.863737 open(".git/objects/40/c92d2149426ea7fd8c70bf7c7727af15eed75d", O_RDONLY|O_NOATIME) = 40 <0.000042> 1260821799.863855 mmap(NULL, 293, PROT_READ, MAP_PRIVATE, 40, 0) = 0x2b4d6a90e000 <0.000075> 1260821799.863973 close(40) = 0 <0.000021> 1260821799.864101 munmap(0x2b4d6a90e000, 293) = 0 <0.000033> 1260821799.864306 stat(".git/objects/43/77e6fe8ac62e7b3a1b65a83665f172550440b6", {st_mode=S_IFREG|0444, st_size=272, ...}) = 0 <0.000177> 1260821799.864551 open(".git/objects/43/77e6fe8ac62e7b3a1b65a83665f172550440b6", O_RDONLY|O_NOATIME) = 40 <0.000025> 1260821799.864635 mmap(NULL, 272, PROT_READ, MAP_PRIVATE, 40, 0) = 0x2b4d6a90e000 <0.000041> 1260821799.864729 close(40) = 0 <0.000058> 1260821799.865064 munmap(0x2b4d6a90e000, 272) = 0 <0.000031>

First things I notice in the strace is that git is opening the same objects multiple times, and there are seconds between the munmap of the last object and the second stat of that same object....

What can I collect, do, whatever?

-Eric

Heya,

On Mon, Dec 14, 2009 at 21:28, Eric Paris <eparis@redhat.com> wrote:

If this really is a case that we end up wanting to optimize somehow, it would probably be very helpful to make a copy of the repository state _before_ the gc is done.

Also, 1.5.5 is really really old in git terms, consider compiling your own. Something post 1.6.4 might be a good idea :).

On Mon, Dec 14, 2009 at 03:28:04PM -0500, Eric Paris wrote:

It sounds like you might have found an infinite loop, as reflog should never really need a lot of CPU. Is it possible to tar the whole repository and make it available publicly for us to look at?

-Peff

On Mon, 2009-12-14 at 16:11 -0500, Jeff King wrote:

Updated to git-1.6.5.3-1 from Fedora rawhide and still git reflog ran for >5 minutes at 100% cpu (I killed it, it didn't finish)

I'm pushing a copy of the whole repo (all 1.9G after bzip compression) to

http://people.redhat.com/~eparis/git-tar/

But it's going to take a couple hours.

-Eric

On Mon, Dec 14, 2009 at 04:20:29PM -0500, Eric Paris wrote:

Wowzers, that's big. Can you send just what's in .git?

-Peff

On Mon, 2009-12-14 at 16:23 -0500, Jeff King wrote:

So I zipped up just .git 1.2G. I did a make clean and zipped up the whole repo 1.3G.

Just started pushing the 1.3G file.

Maybe having a .git directory that large is the problem?

Heya,

On Mon, Dec 14, 2009 at 22:56, Eric Paris <eparis@redhat.com> wrote:

What did you say this repository contained again? Your home video's? Ah, well that explains ;).

On Mon, 14 Dec 2009, Sverre Rabbelier wrote:

That would explain the size, but not the reflog CPU time.

Nicolas

On Mon, Dec 14, 2009 at 04:56:30PM -0500, Eric Paris wrote:

It could be, but I doubt it. If you have a lot of loose objects that could make things slow due to the disk access, but it is not likely to use that much CPU time (we do have to zlib uncompress more, but still...70 minutes is a lot).

-Peff

On Mon, 14 Dec 2009, Eric Paris wrote:

Shouldn't be, unless your repo is really badly packed.

What's the output of 'git count-objects -v' ?

Nicolas

Nicolas Pitre <nico@fluxnic.net> writes:

Didn't somebody say that the trace hints an infinite loop not "slow because of bad packing"?

On Mon, 14 Dec 2009, Junio C Hamano wrote:

Maybe. But I was curious about the size too, which turns out to be really bad packing. Of course bad packing shouldn't affect the correctness of the repository.

Nicolas

On Mon, 2009-12-14 at 19:26 -0500, Nicolas Pitre wrote:

count: 87065 size: 866744 in-pack: 1203497 packs: 148 size-pack: 976474 prune-packable: 1611 garbage: 0

It's not home movies :) . It's a kernel trees with about 5 'upstream' trees that are remotes, which I update daily. One of the remotes constantly rebases every day starting with Linus' tree and pulling in about 150+ branches of work from others all of which might rebase. I have (needlessly) the tags he keeps of that repo every day. I daily rebase my work on top of that constantly rebasing tree (linux-next) using stgit.

I noticed just blindly poking at sizes in my .git/object/pack that the largest pack is a lot larger than the second and third largest....

-r--r--r-- 1 paris paris 108031039 Feb 12 2009 pack-71a9c0f08c76b8ffd1cf0a14d7cfe991fbc9db80.pack -r--r--r-- 1 paris paris 32670479 Apr 7 2009 pack-5c8333301012d9b70d70648b287cf540afcc63ed.pack -r--r--r-- 1 paris paris 26728958 Dec 30 2008 pack-fb8ceb5a33d9881fe771860c6006f55f73ecdf65.pack

And all total there is almost 1G of data in .git/object/pack

If the answer really is that I just have too much data and it can't be handled, I'm fine exporting my patches getting some clean trees and starting over till I get in this situation again, but if it really is a problem/bug that can be solved, the full tar ball of my repo is at

http://people.redhat.com/~eparis/git-tar/

-Eric

On Mon, 14 Dec 2009, Eric Paris wrote:

So basically 87K loose objects occupying 846 MB and 1.2M packed objects occupying 954 MB across 148 packs. That's an horrible repository layout which would definitely gain by being repacked.

That's expected.

Nope. git should handle that kind of data set perfectly fine. And once repacked, you should end up with a single pack containing everything and the total size of your .git/objects directory will probably shrink by 50% or more.

But to be able to repack, your 'git reflog' needs to work correctly, and the problem is unlikely to be related to the repository size.

Nicolas

On Mon, Dec 14, 2009 at 04:20:29PM -0500, Eric Paris wrote:

Holy cow. Almost 150 packs, and that's not even everything. The tarball is missing a bunch of objects, because it points to your kernel-1 as an alternate. So I suspect we would need that, as well, to recreate.

-Peff

On Mon, 14 Dec 2009, Jeff King wrote:

Hmmm... Rebasing repositories mixed with alternates... I wonder if the infinite loop might not actually be due to a delta cycle, especially if the alternate is also rebasing.

So having the alternate, too, would certainly be interesting.

Nicolas

On Mon, 2009-12-14 at 22:50 -0500, Nicolas Pitre wrote:

The alternative repo is slowing pushing up to that same location. That tar is 855838982, so just a tad bit smaller.

-Eric

On Mon, 14 Dec 2009, Eric Paris wrote:

It doesn't appear to be complete yet, and not progressing either.

Nicolas

On Tue, 2009-12-15 at 22:03 -0500, Nicolas Pitre wrote:

I ran out of quota and ask for more, but IT departments moves at the speed of IT departments. I'll delete the first one and just push the alternative repo. Once I get more space I'll try to get them both up at once....

-Eric

On Tue, 2009-12-15 at 22:03 -0500, Nicolas Pitre wrote:

The alternative repo is now available (but the original is down)

I tried to run git gc --aggressive last night while I slept and got this as output, maybe it helps point to a solution/problem? The git reflog portion ran for 5 hours and 36 minutes and appears to have finished.

$ git gc --aggressive error: Could not read d936ff8a7b0841b51ddf96afa24a30b016824cb2 error: Could not read d936ff8a7b0841b51ddf96afa24a30b016824cb2 error: Could not read d936ff8a7b0841b51ddf96afa24a30b016824cb2 error: Could not read d936ff8a7b0841b51ddf96afa24a30b016824cb2 error: Could not read d936ff8a7b0841b51ddf96afa24a30b016824cb2 error: Could not read d936ff8a7b0841b51ddf96afa24a30b016824cb2 error: Could not read d936ff8a7b0841b51ddf96afa24a30b016824cb2 error: Could not read d936ff8a7b0841b51ddf96afa24a30b016824cb2 error: Could not read d936ff8a7b0841b51ddf96afa24a30b016824cb2 error: Could not read d936ff8a7b0841b51ddf96afa24a30b016824cb2 error: Could not read d936ff8a7b0841b51ddf96afa24a30b016824cb2 error: Could not read d936ff8a7b0841b51ddf96afa24a30b016824cb2 error: Could not read d936ff8a7b0841b51ddf96afa24a30b016824cb2 error: Could not read d936ff8a7b0841b51ddf96afa24a30b016824cb2 error: Could not read d936ff8a7b0841b51ddf96afa24a30b016824cb2 error: Could not read d936ff8a7b0841b51ddf96afa24a30b016824cb2 error: Could not read d936ff8a7b0841b51ddf96afa24a30b016824cb2 error: Could not read d936ff8a7b0841b51ddf96afa24a30b016824cb2 error: Could not read d936ff8a7b0841b51ddf96afa24a30b016824cb2 error: Could not read d936ff8a7b0841b51ddf96afa24a30b016824cb2 error: Could not read d936ff8a7b0841b51ddf96afa24a30b016824cb2 error: Could not read d936ff8a7b0841b51ddf96afa24a30b016824cb2 warning: reflog of 'refs/remotes/audit/master' references pruned commits warning: reflog of 'refs/remotes/btrfs/enospc' references pruned commits warning: reflog of 'refs/remotes/btrfs/merge' references pruned commits warning: reflog of 'refs/remotes/btrfs/for-linus' references pruned commits warning: reflog of 'refs/remotes/security-testing/for-linus' references pruned commits error: Could not read 29b6c2fb1390b4fd350a5ecc78f1156fc5d91e9f fatal: bad tree object 29b6c2fb1390b4fd350a5ecc78f1156fc5d91e9f error: failed to run repack

On Wed, 16 Dec 2009, Eric Paris wrote:

Yes. I was able to reproduce your issue. And because of the *horrible* repository packing, the reflog expiration process is taking ages when determining object reachability at a rate of one reflog entry every 2 seconds or so. With 4214 entries for the fsnotify-syscall branch, and 1352 entries for the fsnotify branch, this already takes up asignificant portion of the actual run time. I'm sure if your repository was properly packed this would take less than a minute.

Now, repacking doesn't work because...

Those objects are indeed missing from the repository. Without them your repository is "broken". Either you can find them somewhere else and copy them over, or salvage as much as you can by fetching the interesting branches into another freshly made repository. This is unfortunate because I would have liked to see by how much this repository would have shrunk after a successful repack.

Of course, usage of alternates is recommended _only_ with repositories that are stable, i.e. don't ever add repositories to .git/objects/info/alternates if those repositories are rewinded/rebased and/or branches in them are deleted/replaced. That could be a reason why some objects are now missing from the repository using alternates.

Nicolas

On Wed, 2009-12-16 at 16:06 -0500, Nicolas Pitre wrote:

I'm guessing this is a result of stgit.? These branches really should be just a branch from a tag (which exists in kernel-1) and about 30-50 patches linearly applied on top. I don't know how I get that many objects. I'm guessing many/most of them are crap that should be able to be cleaned/deleted entirely as the rebasing/pushing/poping/updating that stgit does under the covers should have rendered them pointless. Not really sure when/how that should/could have happened.

Should I be running git-gc every night?

/me is pretty git dumb, but is there some way to figure out the parents or children of these? I just trolled through all of my directories doing git show and didn't get any hits. I guess I'll just clean up and start over....

So I'm not sure how I did things wrong. my kernel-1 has those bunch of remotes. The linux-next remote, like I said, basically rebases to linus' tree, then merges 150 random branches. It tags that tree every day and I pull those tags. So I would never expect any objects from those remote trees to ever disappear.

Now I created branches in kernel-1 and I certainly have done lots of things like so

git checkout -b testing remotes/linux-next/master [edit] git commit -a git checkout -b testing1 remotes/linux-next/master git branch -D testing

My assumption though was that this wouldn't ever affect my other repositories. My other repository branches always started by checking out a branch with remotes/*/* as the base.

My understanding was that I would only run into problems if I used something on a branch I created myself in the alternatives repo in other repos (and I didn't remove remotes)

I guess it's not impossible to believe that at some point in time i would have exported patches to and mbox from kernel-1 and applied them to kernel-2 or vice versa. I guess this would end up with the same objects, right? Then if I deleted the branch in kernel-1 I would have problems in kernel-2?

I guess I'll rebuild my setup

new kernel-alt has just the remotes, and my kernel-1,2,3 all alt to it I'll never have local branches in my kernel-alt I'll run git-gc every night I'll hope to never have problem again.

Sound good?

On Wed, 16 Dec 2009, Eric Paris wrote:

Possible. Commit operations (including patch applications) always create loose objects because this is fast, with the expectation that they get collected in a pack later.

This is certainly a good thing to do given your heavy stgit usage.

Moving the reflog data aside (i.e. mv .git/logs .git/logs.bak) it seems that d936ff8 is not referenced anymore.

I found the other one as follows:

First I tried

$ git rev-list --all --objects

This resulted in:

[...] 4f7911b0b0dbd187131a109cf00161a0c6a9d727 arch/x86 ea868257c1eabc31e0ea7941efa42b543978b3fa arch/x86/kvm a0c11ead723956c667172a9f3fb6787684fe7ff5 arch/x86/kvm/paging_tmpl.h b556b6aad8b1aacfecb1dd4a56dbd389674687b5 arch/x86/kvm/x86.c 68a9733ae3315d7e2bfec2037dfeee4db8a6f6a1 drivers error: Could not read 29b6c2fb1390b4fd350a5ecc78f1156fc5d91e9f fatal: bad tree object 29b6c2fb1390b4fd350a5ecc78f1156fc5d91e9f

Because of the way objects are enumerated, we can be pretty sure that the bad tree object is referenced by the tree object 68a9733a corresponding to drivers/. Let's verify that:

$ git ls-tree 68a9733a 100644 blob 00cf9553f74065291612b0971337f79995933a06 Kconfig 100644 blob c1bf41737936ab00be4a87563a0bb0638074785d Makefile 040000 tree d4e847de9bf2450842936582ea7cc6778413825b accessibility 040000 tree 29b6c2fb1390b4fd350a5ecc78f1156fc5d91e9f acpi [...]

Yep, we found it there. So the missing tree object corresponds to drivers/acpi/. So to find the latest commit to which this particular tree object is referenced by, we just need to look at the same rev-list output above (piped into less is handy here) and scroll up until an object with no name is found. This would usually be the first root tree object referencing the named objects that follow. Here I get aafb68eb. To be sure, let's list it so to confirm it really contains a reference to the 68a9733a drivers tree:

$ git ls-tree aafb68eb [...] 040000 tree 68a9733ae3315d7e2bfec2037dfeee4db8a6f6a1 drivers [...]

So yes, we've got the right root tree object. Now finding the corresponding commit should be easy:

$ git log --all --pretty=raw

Then within less, a search for aafb68eb brings us to this:

commit 2e765e9c87a337131aad3014f9a7e5e878c7d0a0 tree aafb68eb84f96c9ab5697c6e8d10d5006d1e7209 parent a2c2de42295b3ac29758f454a7072338e5555ca3 author Eric Paris <eparis@redhat.com> 1237233261 -0400 committer Eric Paris <eparis@redhat.com> 1237233261 -0400

    refresh     64d34c511b1125d9efd2926e683e019f15dec5b4

So this is referenced by a commit that you made on the 1237233261th second since January 1, 1970 i.e. 2009-03-16 19:54:21 +0000 which is quite a while ago. Or given the nature of the commit log, this is probably some stgit branch.

Note that the missing tree didn't necessarily appear with that commit. Because of the recency ordering from rev-list, all we can say is that this is the last commit on that particular branch to reference that tree, but it might have been introduced in the repository way before that point in time.

Now let's try to find out what branch(es) actually link(s) to this commit:

$ git branch -a --contains 2e765e9c

This comes empty. This is because 'git branch' looks only in the refs/heads/ and refs/remotes namespace (or only one of them without -a).

Scripting something around 'git for-each-ref' and 'git merge-base' could be done, such as:

	TARGET=2e765e9c87a337131aad3014f9a7e5e878c7d0a0
	git for-each-ref refs/* |
	while read sha1 type ref; do
		if [ "$(git merge-base $sha1 $TARGET)" = "$TARGET" ]; then
			echo "referenced by $type $ref"
		fi
	done

But this is slow, for the same reason as 'git reflog expire' above. But letting it run for a while should give you at least one answer.

Right.

Eventually, yes. After a while the auto repack in kernel2 would notice that some objects are in kernel1 already and purge them from kernel2. And if those objects were part of a deleted branch then kernel1 would get rid of those objects too once the reflog with a reference to that deleted branch expires. The unsuspecting kernel2 repo then gets broken.

Yes. And make sure not to fetch rebasing repositories, such as linux-next, into kernel-alt without keeping a tag for each fetched state otherwise you'll accumulate unreferenced objects which the other repositories might rely upon.

Nicolas

On Thu, 2009-12-17 at 00:38 -0500, Nicolas Pitre wrote:

This alone almost certainly tells me how I broke it.

For quite some time (a period of months) linux-next was broken and I had to carry a patch to ACPI to make it boot. I dropped that patch at the head of my stgit trees in all of my repositories. So I wouldn't be at all surprised to learn that eventually kernel-2 found that object in kernel-1. Sometime when I dropped that patch from kernel-1 (because it finally got fixed upstream) I can see how it broke.

But now that patch shouldn't be needed by any tree since I have long since dropped it from the stgit stack. So if we cleaned up all of the useless objects in this tree I bet this object wouldn't be needed. Not exactly a situation that I'd expect git to be able to dig out of itself thought.

I'm creating clean repos and going to do no work in my -alt    :)

Thanks everyone!

-Eric

On Thu, 17 Dec 2009, Eric Paris wrote:

I let the script I provided previously ran for a while. And the commit I found to contain the missing object belongs to refs/patches/fsnotify/fsnotify-group-priorities.log. So I simply deleted that branch entirely and now the repack can proceed. And with a 'git gc --aggressive' the 1.2GB repository shrank to a mere 5.2 MB. :-) Of course I didn't bring back all the reflogs though. But I would have expected a repository reduction of the same magnitude even with them.

Nicolas

On Thu, Dec 17, 2009 at 7:33 PM, Nicolas Pitre <nico@fluxnic.net> wrote:

Are we talking about the same Linux kernel repository as before? Because if so, that reduction in size doesn't make any sense to me. The smallest size I've seen for the Linux kernel repository (in the past year) is 250MB.

- Steven

On Thu, 2009-12-17 at 19:44 -0800, Steven Noonan wrote:

Remember that the real code object are in an alternative repository which isn't going to shrink like this. (A nicely packed repo with the majority of the objects in question is around 500M)

On Thu, 17 Dec 2009, Steven Noonan wrote:

As before in this thread.

Sure it does.

Depends if you have an alternate repository from which you may borrow objects from, which was the case here. In that context, 1.2 GB of disk space was completely insane.

Nicolas

On Thu, Dec 17, 2009 at 7:57 PM, Nicolas Pitre <nico@fluxnic.net> wrote:

Ahh. That makes sense. I should really read up on alternates then.

- Steven

On Thu, 2009-12-17 at 22:33 -0500, Nicolas Pitre wrote:

At least when I thought it was in ACPI I could imagine what I had done wrong. Now I'm not so sure.

In any case, I've redesigned with a clear alternative repo that I never work in and a cron job to clean up garbage every night. So hopefully noone will hear from me again.

Nicolas, thanks so much for hunting this down!

-Eric