threads / discuss / 8502

fatal: serious inflate inconsistency

Subject: fatal: serious inflate inconsistency

## tl;dr

17 messages between Jun 8, 2007 and Jun 17, 2007.

replies: 16people: 5as markdown or json

Thomas Glanzmann· Jun 8, 2007, 11:05 UTC · lore

Hello, and here is another one I guess: ;-(

        (tomcat-05) [/usr/src] git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git
        Initialized empty Git repository in /usr/src/linux-2.6/.git/
        remote: Generating pack...
        remote: Done counting 496911 objects.
        remote: Deltifying 496911 objects...
        remote:  100% (496911/496911) done
        Indexing 496911 objects...
        remote: Total 496911 (delta 401539), reused 493712 (delta 398653)
        100% (496911/496911) done
        Resolving 401539 deltas...
        fatal: serious inflate inconsistency
        fatal: index-pack died with error code 128
        fetch-pack from
        'git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git' failed.
        (tomcat-05) [/usr/src] git version
        git version 1.5.2.1

Greetings,
        Thomas

Nicolas Pitre· Jun 8, 2007, 13:48 UTC · re: Thomas Glanzmann · lore

Re: fatal: serious inflate inconsistency

On Fri, 8 Jun 2007, Thomas Glanzmann wrote:

Show 19 quoted lines

> Hello,
> and here is another one I guess: ;-(
> 
>         (tomcat-05) [/usr/src] git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git
>         Initialized empty Git repository in /usr/src/linux-2.6/.git/
>         remote: Generating pack...
>         remote: Done counting 496911 objects.
>         remote: Deltifying 496911 objects...
>         remote:  100% (496911/496911) done
>         Indexing 496911 objects...
>         remote: Total 496911 (delta 401539), reused 493712 (delta 398653)
>         100% (496911/496911) done
>         Resolving 401539 deltas...
>         fatal: serious inflate inconsistency
>         fatal: index-pack died with error code 128
>         fetch-pack from
>         'git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git' failed.
>         (tomcat-05) [/usr/src] git version
>         git version 1.5.2.1

This is bad.

index-pack works like this:

1) it receives data over the network:
   1a) it writes a copy of that data to disk to create a pack
   1b) it deflates that data in parallel to compute its SHA1

2) when done, it resolves deltas and to do so:
   2a) it deflates the base object data from 1a
   2b) it applies the delta over the base object data then compute its 
       SHA1

In your case it seems that 1B and 2A don't produce the same deflated data somehow.

Is it reproducible? Are you using Linux? What kernel version? Are you using NFS?

If it isn't reproducible then you might possibly have been the victim of faulty hardware.

Nicolas

Thomas Glanzmann· Jun 8, 2007, 13:50 UTC · re: Nicolas Pitre · lore

Re: fatal: serious inflate inconsistency

Hello,

> Is it reproducible?

I can try. But that produces 150 Mbyte Traffic on Kernel.org

> Are you using Linux?  What kernel version?

Yes, I do. Debian etch; 2.6.20

> Are you using NFS?

Nope.

> If it isn't reproducible then you might possibly have been the victim
> of faulty hardware.

I give it a try and come back to you. I am off the net for 1 hour.

        Thomas

Thomas Glanzmann· Jun 8, 2007, 15:37 UTC · re: Thomas Glanzmann · lore

Re: fatal: serious inflate inconsistency

Hello,

> I give it a try and come back to you. I am off the net for 1 hour.

        (tomcat-05) [/usr/src] git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git
        Initialized empty Git repository in /usr/src/linux-2.6/.git/
        remote: Generating pack...
        remote: Done counting 496911 objects.
        remote: Deltifying 496911 objects...
        remote:  100% (496911/496911) done
        Indexing 496911 objects...
        remote: Total 496911 (delta 401539), reused 493712 (delta 398653)
        100% (496911/496911) done
        Resolving 401539 deltas...
        fatal: serious inflate inconsistency
        fatal: index-pack died with error code 128
        fetch-pack from 'git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git' failed.

yes, it is reproducable. Again I am off the net for an hour.

        Thomas

Thomas Glanzmann· Jun 8, 2007, 15:39 UTC · re: Thomas Glanzmann · lore

Re: fatal: serious inflate inconsistency

Hello again,

> yes, it is reproducable. Again I am off the net for an hour.

I used the debian packages that built a friend of mine:

        deb http://rmdir.de/~michael/git/ ./

If you want to reproduce it by yourself.

        Thomas

Nicolas Pitre· Jun 8, 2007, 16:15 UTC · re: Thomas Glanzmann · lore

Re: fatal: serious inflate inconsistency

On Fri, 8 Jun 2007, Thomas Glanzmann wrote:

Show 19 quoted lines

> Hello,
> 
> > I give it a try and come back to you. I am off the net for 1 hour.
> 
>         (tomcat-05) [/usr/src] git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git
>         Initialized empty Git repository in /usr/src/linux-2.6/.git/
>         remote: Generating pack...
>         remote: Done counting 496911 objects.
>         remote: Deltifying 496911 objects...
>         remote:  100% (496911/496911) done
>         Indexing 496911 objects...
>         remote: Total 496911 (delta 401539), reused 493712 (delta 398653)
>         100% (496911/496911) done
>         Resolving 401539 deltas...
>         fatal: serious inflate inconsistency
>         fatal: index-pack died with error code 128
>         fetch-pack from 'git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git' failed.
> 
> yes, it is reproducable. Again I am off the net for an hour.

I can't reproduce that.

$ git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git
Initialized empty Git repository in /home/nico/git/tst/linux-2.6/.git/
remote: Generating pack...
remote: Done counting 496911 objects.
remote: Deltifying 496911 objects...
remote:  100% (496911/496911) done
Indexing 496911 objects...
remote: Total 496911 (delta 401539), reused 493712 (delta 398653)
 100% (496911/496911) done
Resolving 401539 deltas...
 100% (401539/401539) done
Checking 22413 files out...
 100% (22413/22413) done
$ git version
git version 1.5.2.1

Show 5 quoted lines

> I used the debian packages that built a friend of mine:
>
>         deb http://rmdir.de/~michael/git/ ./
>
> If you want to reproduce it by yourself.

I don't use Debian, sorry.

Nicolas

Linus Torvalds· Jun 8, 2007, 17:01 UTC · re: Thomas Glanzmann · lore

Re: fatal: serious inflate inconsistency

On Fri, 8 Jun 2007, Thomas Glanzmann wrote:

> 
> yes, it is reproducable. Again I am off the net for an hour.

Could you compile your own git? If so, please try this patch on top of current git, to make the error message a bit more informative.

It does sound like a corrupt stream, but maybe debian has some strange zlib version that git doesn't like.

		Linus
---
 index-pack.c |    8 ++++++--
 1 files changed, 6 insertions(+), 2 deletions(-)

diff --git a/index-pack.c b/index-pack.c
index 82c8da3..542687b 100644
--- a/index-pack.c
+++ b/index-pack.c
@@ -277,9 +277,13 @@ static void *get_data_from_pack(struct object_entry *obj)
 	stream.avail_in = len;
 	inflateInit(&stream);
 	while ((st = inflate(&stream, Z_FINISH)) == Z_OK);
+	if (st != Z_STREAM_END)
+		die("serious inflate inconsistency: %d (%s)",
+			st, stream.msg ? stream.msg : "no message");
+	if (stream.total_out != obj->size)
+		die("serious inflate inconsistency: got %lu bytes, expected %lu",
+			stream.total_out, obj->size);
 	inflateEnd(&stream);
-	if (st != Z_STREAM_END || stream.total_out != obj->size)
-		die("serious inflate inconsistency");
 	free(src);
 	return data;
 }

Thomas Glanzmann· Jun 8, 2007, 18:24 UTC · re: Linus Torvalds · lore

Re: fatal: serious inflate inconsistency

Hello Linus,

> Could you compile your own git? If so, please try this patch on top of 
> current git, to make the error message a bit more informative.

        (tomcat-05) [/usr/src] scp 131.188.30.102:/var/tmp/sithglan/git-core-1.5.2.1/../git-core_1.5.2.1-1_i386.deb .
        Warning: Permanently added '131.188.30.102' (RSA) to the list of known hosts.
        git-core_1.5.2.1-1_i386.deb                                                    100% 3511KB   1.7MB/s   00:02
        (tomcat-05) [/usr/src] / dpkg -i git-core_1.5.2.1-1_i386.deb
        (Reading database ... 13273 files and directories currently installed.)
        Preparing to replace git-core 1:1.5.2.1-1 (using git-core_1.5.2.1-1_i386.deb) ...
        Unpacking replacement git-core ...
        Setting up git-core (1.5.2.1-1) ...
        (tomcat-05) [/usr/src] git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git
        Initialized empty Git repository in /usr/src/linux-2.6/.git/
        remote: Generating pack...
        remote: Done counting 496911 objects.
        remote: Deltifying 496911 objects...
        remote:  100% (496911/496911) done
        Indexing 496911 objects...
        remote: Total 496911 (delta 401539), reused 493712 (delta 398653)
        100% (496911/496911) done
        Resolving 401539 deltas...
        fatal: serious inflate inconsistency: -3 (invalid distance too far back)
        fatal: index-pack died with error code 128
        fetch-pack from 'git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git' failed.

I talked to Michael and he is certain that the machine has a memory corruption the next thing I am going to do is compiling a kernel in an endless loop. Again if somone needs shell access just send me your ssh key.

The complete build log is on (well not the complete log the first line is missing because my screen backlog has _only_ 10000 lines (a memory issue)):

The first line is: wget http://ftp.debian.org/debian/pool/main/g/git-core/git-core_1.5.2.1-1.dsc

The rest is here:

http://wwwcip.informatik.uni-erlangen.de/~sithglan/screen-exchange.bz2 (32k)

Thanks,
        Thomas

Linus Torvalds· Jun 8, 2007, 18:52 UTC · re: Thomas Glanzmann · lore

Re: fatal: serious inflate inconsistency

On Fri, 8 Jun 2007, Thomas Glanzmann wrote:

>
>         fatal: serious inflate inconsistency: -3 (invalid distance too far back)

Whee. That's Z_DATA_ERROR, and yeah, I think the only thing that causes it is a corrupt zlib input stream.

> I talked to Michael and he is certain that the machine has a memory
> corruption the next thing I am going to do is compiling a kernel in an
> endless loop. Again if somone needs shell access just send me your ssh
> key.

I'd not expect memory corruption to be that *repeatable*. But the message certainly implies zlib data stream corruption _somewhere_, although the fact that it seems to be so repeatable does make me suspect program error.

Of course, it could be a hard bit-error in memory, but even then it would kind of have to hit the same page allocation history each time to be repeatable. And the page cache is the only thing that is that sticky under Linux, so it would have to be something like the zlib static data or code that had hit the memory corruption.

Sounds strange. I wonder if we might have finally hit a case of different versions of zlib acting differently?

But it _could_ just be a git bug too.

I don't think it's on the kernel.org side (or we would have had more reports of this), but maybe there is some wild pointer thing inside git, and nobody else noticed because it depends on some specific memory allocation pattern (which in turn depends on things like kernel memory layout choices, and on libc 'malloc()' implementation etc).

I'll build with ElectricFence, and see if it says anything for me. It passes all the tests, but fetching the kernel is almost certainly going to do things that we don't test for..

			Linus

Linus Torvalds· Jun 8, 2007, 19:06 UTC · re: Linus Torvalds · lore

Re: fatal: serious inflate inconsistency

On Fri, 8 Jun 2007, Linus Torvalds wrote:

> 
> I'll build with ElectricFence, and see if it says anything for me. It 
> passes all the tests, but fetching the kernel is almost certainly going to 
> do things that we don't test for..

It passes smaller clones fine with ElectricFence (and bigger ones fail because EF uses *so* much memory for its checking that you cannot actually complete them), so I don't think this is a git bug. Sounds like either the hw problems are real, or there is some zlib (or other library) issue..

		Linus

Thomas Glanzmann· Jun 8, 2007, 19:33 UTC · re: Linus Torvalds · lore

Re: fatal: serious inflate inconsistency

Hello Linus,

> It passes smaller clones fine with ElectricFence (and bigger ones fail 
> because EF uses *so* much memory for its checking that you cannot actually 
> complete them), so I don't think this is a git bug. Sounds like either the 
> hw problems are real, or there is some zlib (or other library) issue..

        (mailrelay) [/var/tmp] git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git
        Initialized empty Git repository in /var/tmp/linux-2.6/.git/
        remote: Generating pack...
        remote: Done counting 496911 objects.
        remote: Deltifying 496911 objects...
        remote:  100% (496911/496911) done
        Indexing 496911 objects...
        remote: Total 496911 (delta 401539), reused 493712 (delta 398653)
        100% (496911/496911) done
        Resolving 401539 deltas...
        100% (401539/401539) done
        Checking 22413 files out...
        100% (22413/22413) done

I have a very similar machine. For this one it works. It has 32 bit kernel and userland. Debian etch again. v2.6.20.

Does someone know a command for debian that checks the checksums for every installed packages and the file that are included in that package? I know there is one for rpm so there must be one for debian, too.

        Thomas

Marco Roeland· Jun 8, 2007, 20:45 UTC · re: Thomas Glanzmann · lore

Re: fatal: serious inflate inconsistency

On Friday June 8th 2007 at 21:33 Thomas Glanzmann wrote:

> Does someone know a command for debian that checks the checksums for every
> installed packages and the file that are included in that package?

Try debsums. Without arguments it checks all installed packages and their files, but you can also check individual packages.

-- 
Marco Roeland

Thomas Glanzmann· Jun 9, 2007, 00:13 UTC · re: Marco Roeland · lore

Re: fatal: serious inflate inconsistency

Hello, the OS is not corrupt. So I am going to compile a few kernels and wait for gcc to segfault.

        Thomas

Thomas Glanzmann· Jun 9, 2007, 21:12 UTC · re: Thomas Glanzmann · lore

Re: fatal: serious inflate inconsistency

Hello again, the machine that had the issues just got down because of a broken power supply and took half the rack with it. So it was definitive a hardware fault. Sorry for all the trouble. And the bottom line is: It was _not_ git, it was broken hardware!

        Thomas

Linus Torvalds· Jun 10, 2007, 00:30 UTC · re: Thomas Glanzmann · lore

Re: fatal: serious inflate inconsistency

On Sat, 9 Jun 2007, Thomas Glanzmann wrote:

Show 5 quoted lines

>
> the machine that had the issues just got down because of a broken power
> supply and took half the rack with it. So it was definitive a hardware
> fault. Sorry for all the trouble. And the bottom line is: It was _not_
> git, it was broken hardware!

Good. That was one of the design goals for git (ie the "you know you can trust the data" thing relies on very strong protection at all levels, even in the presense of disk/memory/cpu corruption).

I think we have a pretty good record on this: the only real git-induced corruption I can remember was the embarrassing bug with use-after-free of the CRLF auto-translation. But let's not get complacent.

		Linus

Martin Langhoff· Jun 10, 2007, 00:50 UTC · re: Thomas Glanzmann · lore

Re: fatal: serious inflate inconsistency

On 6/9/07, Thomas Glanzmann <thomas@glanzmann.de> wrote:

> I have a very similar machine. For this one it works. It has 32 bit kernel and
> userland. Debian etch again. v2.6.20.

Wanted to mention -- we've been using git on debian sarge/etch on several arches a lot and never seen any lib compatiblity problem.

> Does someone know a command for debian that checks the checksums

apt-get install debsums man debsums # sorry, I don't use it often enough to remember ;-)

cheers,

Thomas Glanzmann· Jun 17, 2007, 08:01 UTC · re: Thomas Glanzmann · lore

Re: fatal: serious inflate inconsistency

Hello, the machine came back with a new power supply. And guess what:

(tomcat-05) [/usr/src] git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git Initialized empty Git repository in /usr/src/linux-2.6/.git/ remote: Generating pack... remote: Done counting 499634 objects. remote: Deltifying 499634 objects... remote: 100% (499634/499634) done Indexing 499634 objects... remote: Total 499634 (delta 403845), reused 495761 (delta 400337) 100% (499634/499634) done Resolving 403845 deltas... 100% (403845/403845) done Checking 22411 files out... 100% (22411/22411) done

        Thomas