Re: Help with "fatal: unable to read ...." error during GC?

From: Jeff King <peff@peff.net>
Date: Aug 9, 2018, 17:06 UTC
Message-ID: <20180809170609.GE1439@sigill.intra.peff.net>
In-Reply-To: <249be5d3dada9a4b1b5282896a9a11e12c1ffd2a.camel@mad-scientist.net>

On Wed, Aug 08, 2018 at 10:45:49PM -0400, Paul Smith wrote:

Show 15 quoted lines

> On Wed, 2018-08-08 at 14:24 -0400, Jeff King wrote:
> > If so, can you try running it under gdb and getting a stack trace?
> > Something like:
> > 
> >   gdb git
> >   [and then inside gdb...]
> >   set args pack-objects --all --reflog --indexed-objects foo </dev/null
> >   break die
> >   run
> >   bt
> > 
> > That might give us a clue where the broken object reference is coming
> 
> Here we go.  I can rebuild with -Og or -O0 if more detailed debugging
> is needed; most everything appears to be optimized out:

No, I think this is enough to give a general sense of the problem location.

Show 18 quoted lines

> Compressing objects: 100% (107777/107777), done.
> Writing objects:  54% (274416/508176)   
> Thread 1 "git" hit Breakpoint 1, die (err=err@entry=0x5a373a "unable to read %s") at usage.c:119
> 119     {
> (gdb) bt
> #0  die (err=err@entry=0x5a373a "unable to read %s") at usage.c:119
> #1  0x00000000004563f3 in get_delta (entry=<optimized out>) at builtin/pack-objects.c:143
> #2  write_no_reuse_object () at builtin/pack-objects.c:308
> #3  0x0000000000456592 in write_reuse_object (usable_delta=<optimized out>, limit=<optimized out>, entry=<optimized out>, f=<optimized out>) at builtin/pack-objects.c:516
> #4  write_object (write_offset=<optimized out>, entry=0x7fffc9a8d940, f=0x198fb70) at builtin/pack-objects.c:518
> #5  write_one () at builtin/pack-objects.c:576
> #6  0x00000000004592f0 in write_pack_file () at builtin/pack-objects.c:849
> #7  cmd_pack_objects (argc=<optimized out>, argv=<optimized out>, prefix=<optimized out>) at builtin/pack-objects.c:3354
> #8  0x0000000000404f06 in run_builtin (argv=<optimized out>, argc=<optimized out>, p=<optimized out>) at git.c:417
> #9  handle_builtin (argc=<optimized out>, argv=<optimized out>) at git.c:632
> #10 0x0000000000405f21 in run_argv (argv=0x7fffffffe210, argcp=0x7fffffffe21c) at git.c:761
> #11 cmd_main (argc=<optimized out>, argc@entry=6, argv=<optimized out>, argv@entry=0x7fffffffe448) at git.c:761
> #12 0x0000000000404b15 in main (argc=6, argv=0x7fffffffe448) at common-main.c:45

So that's quite unexpected. I assumed we'd have hit this problem while deciding _which_ objects to write. But we get all the way to the point of writing out the result before we notice it's missing.

I don't think I've run such a case before, but I wonder if "pack-objects --all" is too lax about adding missing blobs during its object traversal (especially during the "unreachable but recent" part of the traversal that I mentioned, which should silently omit missing objects). I played around with recreating this situation, though, and I don't think it's possible to cause the results you're seeing. We come up with a list of recent objects, but we only use it as a look-up index for discarding too-old objects. So:

  - it wouldn't ever cause us to choose to write an object into a pack,
    which is what you're seeing

  - we'd never consider a missing object; it's a pure lookup table, and
    the actual list of objects we consider is found by walking the set
    of packs

So that's probably a dead end.

What I really wonder is where we found out about that object name in the first place. Can you instrument your Git build like this:

diff --git a/builtin/pack-objects.c b/builtin/pack-objects.c
index 71056d8294..5ff6de5ddf 100644
--- a/builtin/pack-objects.c
+++ b/builtin/pack-objects.c
@@ -1112,6 +1112,13 @@ static int add_object_entry(const struct object_id *oid, enum object_type type,
 	struct packed_git *found_pack = NULL;
 	off_t found_offset = 0;
 	uint32_t index_pos;
+	static const struct object_id funny_oid = {
+		"\xc1\x04\xb8\xfb\x36\x31\xb5\xc5\x46\x95"
+		"\x20\x6b\x2f\x73\x31\x0c\x02\x3c\x99\x63"
+	};
+
+	if (!oidcmp(oid, &funny_oid))
+		warning("found funny oid");
 
 	display_progress(progress_state, ++nr_seen);
 

and similarly get a backtrace when we hit that warning()? (Or if you're
a gdb expert, you could probably use a conditional breakpoint, but I
find just modifying the source easier).

-Peff