# fatal: serious inflate inconsistency 17 messages from 2007-06-08 to 2007-06-17. Participants: Thomas Glanzmann, Nicolas Pitre, Linus Torvalds, Marco Roeland, Martin Langhoff. Thread: https://gitlist.dev/t/8502 ## Thomas Glanzmann, 2007-06-08 11:05 Subject: fatal: serious inflate inconsistency Message-ID: <20070608110536.GD2335@cip.informatik.uni-erlangen.de> URL: https://gitlist.dev/e/20070608110536.GD2335%40cip.informatik.uni-erlangen.de ``` Hello, and here is another one I guess: ;-( (tomcat-05) [/usr/src] git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git Initialized empty Git repository in /usr/src/linux-2.6/.git/ remote: Generating pack... remote: Done counting 496911 objects. remote: Deltifying 496911 objects... remote: 100% (496911/496911) done Indexing 496911 objects... remote: Total 496911 (delta 401539), reused 493712 (delta 398653) 100% (496911/496911) done Resolving 401539 deltas... fatal: serious inflate inconsistency fatal: index-pack died with error code 128 fetch-pack from 'git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git' failed. (tomcat-05) [/usr/src] git version git version 1.5.2.1 Greetings, Thomas ``` ## Nicolas Pitre, 2007-06-08 13:48 Subject: Re: fatal: serious inflate inconsistency Message-ID: URL: https://gitlist.dev/e/alpine.LFD.0.99.0706080931330.12885%40xanadu.home In-Reply-To: <20070608110536.GD2335@cip.informatik.uni-erlangen.de> ``` On Fri, 8 Jun 2007, Thomas Glanzmann wrote: > Hello, > and here is another one I guess: ;-( > > (tomcat-05) [/usr/src] git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git > Initialized empty Git repository in /usr/src/linux-2.6/.git/ > remote: Generating pack... > remote: Done counting 496911 objects. > remote: Deltifying 496911 objects... > remote: 100% (496911/496911) done > Indexing 496911 objects... > remote: Total 496911 (delta 401539), reused 493712 (delta 398653) > 100% (496911/496911) done > Resolving 401539 deltas... > fatal: serious inflate inconsistency > fatal: index-pack died with error code 128 > fetch-pack from > 'git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git' failed. > (tomcat-05) [/usr/src] git version > git version 1.5.2.1 This is bad. index-pack works like this: 1) it receives data over the network: 1a) it writes a copy of that data to disk to create a pack 1b) it deflates that data in parallel to compute its SHA1 2) when done, it resolves deltas and to do so: 2a) it deflates the base object data from 1a 2b) it applies the delta over the base object data then compute its SHA1 In your case it seems that 1B and 2A don't produce the same deflated data somehow. Is it reproducible? Are you using Linux? What kernel version? Are you using NFS? If it isn't reproducible then you might possibly have been the victim of faulty hardware. Nicolas ``` ## Thomas Glanzmann, 2007-06-08 13:50 Subject: Re: fatal: serious inflate inconsistency Message-ID: <20070608135054.GK2335@cip.informatik.uni-erlangen.de> URL: https://gitlist.dev/e/20070608135054.GK2335%40cip.informatik.uni-erlangen.de In-Reply-To: ``` Hello, > Is it reproducible? I can try. But that produces 150 Mbyte Traffic on Kernel.org > Are you using Linux? What kernel version? Yes, I do. Debian etch; 2.6.20 > Are you using NFS? Nope. > If it isn't reproducible then you might possibly have been the victim > of faulty hardware. I give it a try and come back to you. I am off the net for 1 hour. Thomas ``` ## Thomas Glanzmann, 2007-06-08 15:37 Subject: Re: fatal: serious inflate inconsistency Message-ID: <20070608153722.GL2335@cip.informatik.uni-erlangen.de> URL: https://gitlist.dev/e/20070608153722.GL2335%40cip.informatik.uni-erlangen.de In-Reply-To: <20070608135054.GK2335@cip.informatik.uni-erlangen.de> ``` Hello, > I give it a try and come back to you. I am off the net for 1 hour. (tomcat-05) [/usr/src] git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git Initialized empty Git repository in /usr/src/linux-2.6/.git/ remote: Generating pack... remote: Done counting 496911 objects. remote: Deltifying 496911 objects... remote: 100% (496911/496911) done Indexing 496911 objects... remote: Total 496911 (delta 401539), reused 493712 (delta 398653) 100% (496911/496911) done Resolving 401539 deltas... fatal: serious inflate inconsistency fatal: index-pack died with error code 128 fetch-pack from 'git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git' failed. yes, it is reproducable. Again I am off the net for an hour. Thomas ``` ## Thomas Glanzmann, 2007-06-08 15:39 Subject: Re: fatal: serious inflate inconsistency Message-ID: <20070608153916.GM2335@cip.informatik.uni-erlangen.de> URL: https://gitlist.dev/e/20070608153916.GM2335%40cip.informatik.uni-erlangen.de In-Reply-To: <20070608153722.GL2335@cip.informatik.uni-erlangen.de> ``` Hello again, > yes, it is reproducable. Again I am off the net for an hour. I used the debian packages that built a friend of mine: deb http://rmdir.de/~michael/git/ ./ If you want to reproduce it by yourself. Thomas ``` ## Nicolas Pitre, 2007-06-08 16:15 Subject: Re: fatal: serious inflate inconsistency Message-ID: URL: https://gitlist.dev/e/alpine.LFD.0.99.0706081210130.12885%40xanadu.home In-Reply-To: <20070608153722.GL2335@cip.informatik.uni-erlangen.de> ``` On Fri, 8 Jun 2007, Thomas Glanzmann wrote: > Hello, > > > I give it a try and come back to you. I am off the net for 1 hour. > > (tomcat-05) [/usr/src] git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git > Initialized empty Git repository in /usr/src/linux-2.6/.git/ > remote: Generating pack... > remote: Done counting 496911 objects. > remote: Deltifying 496911 objects... > remote: 100% (496911/496911) done > Indexing 496911 objects... > remote: Total 496911 (delta 401539), reused 493712 (delta 398653) > 100% (496911/496911) done > Resolving 401539 deltas... > fatal: serious inflate inconsistency > fatal: index-pack died with error code 128 > fetch-pack from 'git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git' failed. > > yes, it is reproducable. Again I am off the net for an hour. I can't reproduce that. $ git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git Initialized empty Git repository in /home/nico/git/tst/linux-2.6/.git/ remote: Generating pack... remote: Done counting 496911 objects. remote: Deltifying 496911 objects... remote: 100% (496911/496911) done Indexing 496911 objects... remote: Total 496911 (delta 401539), reused 493712 (delta 398653) 100% (496911/496911) done Resolving 401539 deltas... 100% (401539/401539) done Checking 22413 files out... 100% (22413/22413) done $ git version git version 1.5.2.1 > I used the debian packages that built a friend of mine: > > deb http://rmdir.de/~michael/git/ ./ > > If you want to reproduce it by yourself. I don't use Debian, sorry. Nicolas ``` ## Linus Torvalds, 2007-06-08 17:01 Subject: Re: fatal: serious inflate inconsistency Message-ID: URL: https://gitlist.dev/e/alpine.LFD.0.98.0706080959570.4205%40woody.linux-foundation.org In-Reply-To: <20070608153722.GL2335@cip.informatik.uni-erlangen.de> ``` On Fri, 8 Jun 2007, Thomas Glanzmann wrote: > > yes, it is reproducable. Again I am off the net for an hour. Could you compile your own git? If so, please try this patch on top of current git, to make the error message a bit more informative. It does sound like a corrupt stream, but maybe debian has some strange zlib version that git doesn't like. Linus --- index-pack.c | 8 ++++++-- 1 files changed, 6 insertions(+), 2 deletions(-) diff --git a/index-pack.c b/index-pack.c index 82c8da3..542687b 100644 --- a/index-pack.c +++ b/index-pack.c @@ -277,9 +277,13 @@ static void *get_data_from_pack(struct object_entry *obj) stream.avail_in = len; inflateInit(&stream); while ((st = inflate(&stream, Z_FINISH)) == Z_OK); + if (st != Z_STREAM_END) + die("serious inflate inconsistency: %d (%s)", + st, stream.msg ? stream.msg : "no message"); + if (stream.total_out != obj->size) + die("serious inflate inconsistency: got %lu bytes, expected %lu", + stream.total_out, obj->size); inflateEnd(&stream); - if (st != Z_STREAM_END || stream.total_out != obj->size) - die("serious inflate inconsistency"); free(src); return data; } ``` ## Thomas Glanzmann, 2007-06-08 18:24 Subject: Re: fatal: serious inflate inconsistency Message-ID: <20070608182412.GQ2335@cip.informatik.uni-erlangen.de> URL: https://gitlist.dev/e/20070608182412.GQ2335%40cip.informatik.uni-erlangen.de In-Reply-To: ``` Hello Linus, > Could you compile your own git? If so, please try this patch on top of > current git, to make the error message a bit more informative. (tomcat-05) [/usr/src] scp 131.188.30.102:/var/tmp/sithglan/git-core-1.5.2.1/../git-core_1.5.2.1-1_i386.deb . Warning: Permanently added '131.188.30.102' (RSA) to the list of known hosts. git-core_1.5.2.1-1_i386.deb 100% 3511KB 1.7MB/s 00:02 (tomcat-05) [/usr/src] / dpkg -i git-core_1.5.2.1-1_i386.deb (Reading database ... 13273 files and directories currently installed.) Preparing to replace git-core 1:1.5.2.1-1 (using git-core_1.5.2.1-1_i386.deb) ... Unpacking replacement git-core ... Setting up git-core (1.5.2.1-1) ... (tomcat-05) [/usr/src] git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git Initialized empty Git repository in /usr/src/linux-2.6/.git/ remote: Generating pack... remote: Done counting 496911 objects. remote: Deltifying 496911 objects... remote: 100% (496911/496911) done Indexing 496911 objects... remote: Total 496911 (delta 401539), reused 493712 (delta 398653) 100% (496911/496911) done Resolving 401539 deltas... fatal: serious inflate inconsistency: -3 (invalid distance too far back) fatal: index-pack died with error code 128 fetch-pack from 'git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git' failed. I talked to Michael and he is certain that the machine has a memory corruption the next thing I am going to do is compiling a kernel in an endless loop. Again if somone needs shell access just send me your ssh key. The complete build log is on (well not the complete log the first line is missing because my screen backlog has _only_ 10000 lines (a memory issue)): The first line is: wget http://ftp.debian.org/debian/pool/main/g/git-core/git-core_1.5.2.1-1.dsc The rest is here: http://wwwcip.informatik.uni-erlangen.de/~sithglan/screen-exchange.bz2 (32k) Thanks, Thomas ``` ## Linus Torvalds, 2007-06-08 18:52 Subject: Re: fatal: serious inflate inconsistency Message-ID: URL: https://gitlist.dev/e/alpine.LFD.0.98.0706081136480.4205%40woody.linux-foundation.org In-Reply-To: <20070608182412.GQ2335@cip.informatik.uni-erlangen.de> ``` On Fri, 8 Jun 2007, Thomas Glanzmann wrote: > > fatal: serious inflate inconsistency: -3 (invalid distance too far back) Whee. That's Z_DATA_ERROR, and yeah, I think the only thing that causes it is a corrupt zlib input stream. > I talked to Michael and he is certain that the machine has a memory > corruption the next thing I am going to do is compiling a kernel in an > endless loop. Again if somone needs shell access just send me your ssh > key. I'd not expect memory corruption to be that *repeatable*. But the message certainly implies zlib data stream corruption _somewhere_, although the fact that it seems to be so repeatable does make me suspect program error. Of course, it could be a hard bit-error in memory, but even then it would kind of have to hit the same page allocation history each time to be repeatable. And the page cache is the only thing that is that sticky under Linux, so it would have to be something like the zlib static data or code that had hit the memory corruption. Sounds strange. I wonder if we might have finally hit a case of different versions of zlib acting differently? But it _could_ just be a git bug too. I don't think it's on the kernel.org side (or we would have had more reports of this), but maybe there is some wild pointer thing inside git, and nobody else noticed because it depends on some specific memory allocation pattern (which in turn depends on things like kernel memory layout choices, and on libc 'malloc()' implementation etc). I'll build with ElectricFence, and see if it says anything for me. It passes all the tests, but fetching the kernel is almost certainly going to do things that we don't test for.. Linus ``` ## Linus Torvalds, 2007-06-08 19:06 Subject: Re: fatal: serious inflate inconsistency Message-ID: URL: https://gitlist.dev/e/alpine.LFD.0.98.0706081202580.4205%40woody.linux-foundation.org In-Reply-To: ``` On Fri, 8 Jun 2007, Linus Torvalds wrote: > > I'll build with ElectricFence, and see if it says anything for me. It > passes all the tests, but fetching the kernel is almost certainly going to > do things that we don't test for.. It passes smaller clones fine with ElectricFence (and bigger ones fail because EF uses *so* much memory for its checking that you cannot actually complete them), so I don't think this is a git bug. Sounds like either the hw problems are real, or there is some zlib (or other library) issue.. Linus ``` ## Thomas Glanzmann, 2007-06-08 19:33 Subject: Re: fatal: serious inflate inconsistency Message-ID: <20070608193319.GS2335@cip.informatik.uni-erlangen.de> URL: https://gitlist.dev/e/20070608193319.GS2335%40cip.informatik.uni-erlangen.de In-Reply-To: ``` Hello Linus, > It passes smaller clones fine with ElectricFence (and bigger ones fail > because EF uses *so* much memory for its checking that you cannot actually > complete them), so I don't think this is a git bug. Sounds like either the > hw problems are real, or there is some zlib (or other library) issue.. (mailrelay) [/var/tmp] git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git Initialized empty Git repository in /var/tmp/linux-2.6/.git/ remote: Generating pack... remote: Done counting 496911 objects. remote: Deltifying 496911 objects... remote: 100% (496911/496911) done Indexing 496911 objects... remote: Total 496911 (delta 401539), reused 493712 (delta 398653) 100% (496911/496911) done Resolving 401539 deltas... 100% (401539/401539) done Checking 22413 files out... 100% (22413/22413) done I have a very similar machine. For this one it works. It has 32 bit kernel and userland. Debian etch again. v2.6.20. Does someone know a command for debian that checks the checksums for every installed packages and the file that are included in that package? I know there is one for rpm so there must be one for debian, too. Thomas ``` ## Marco Roeland, 2007-06-08 20:45 Subject: Re: fatal: serious inflate inconsistency Message-ID: <20070608204557.GA18840@fiberbit.xs4all.nl> URL: https://gitlist.dev/e/20070608204557.GA18840%40fiberbit.xs4all.nl In-Reply-To: <20070608193319.GS2335@cip.informatik.uni-erlangen.de> ``` On Friday June 8th 2007 at 21:33 Thomas Glanzmann wrote: > Does someone know a command for debian that checks the checksums for every > installed packages and the file that are included in that package? Try debsums. Without arguments it checks all installed packages and their files, but you can also check individual packages. -- Marco Roeland ``` ## Thomas Glanzmann, 2007-06-09 00:13 Subject: Re: fatal: serious inflate inconsistency Message-ID: <20070609001333.GA4058@cip.informatik.uni-erlangen.de> URL: https://gitlist.dev/e/20070609001333.GA4058%40cip.informatik.uni-erlangen.de In-Reply-To: <20070608204557.GA18840@fiberbit.xs4all.nl> ``` Hello, the OS is not corrupt. So I am going to compile a few kernels and wait for gcc to segfault. Thomas ``` ## Thomas Glanzmann, 2007-06-09 21:12 Subject: Re: fatal: serious inflate inconsistency Message-ID: <20070609211213.GA18761@cip.informatik.uni-erlangen.de> URL: https://gitlist.dev/e/20070609211213.GA18761%40cip.informatik.uni-erlangen.de In-Reply-To: <20070609001333.GA4058@cip.informatik.uni-erlangen.de> ``` Hello again, the machine that had the issues just got down because of a broken power supply and took half the rack with it. So it was definitive a hardware fault. Sorry for all the trouble. And the bottom line is: It was _not_ git, it was broken hardware! Thomas ``` ## Linus Torvalds, 2007-06-10 00:30 Subject: Re: fatal: serious inflate inconsistency Message-ID: URL: https://gitlist.dev/e/alpine.LFD.0.98.0706091727050.20321%40woody.linux-foundation.org In-Reply-To: <20070609211213.GA18761@cip.informatik.uni-erlangen.de> ``` On Sat, 9 Jun 2007, Thomas Glanzmann wrote: > > the machine that had the issues just got down because of a broken power > supply and took half the rack with it. So it was definitive a hardware > fault. Sorry for all the trouble. And the bottom line is: It was _not_ > git, it was broken hardware! Good. That was one of the design goals for git (ie the "you know you can trust the data" thing relies on very strong protection at all levels, even in the presense of disk/memory/cpu corruption). I think we have a pretty good record on this: the only real git-induced corruption I can remember was the embarrassing bug with use-after-free of the CRLF auto-translation. But let's not get complacent. Linus ``` ## Martin Langhoff, 2007-06-10 00:50 Subject: Re: fatal: serious inflate inconsistency Message-ID: <46a038f90706091750x446c96bfkdc1d5dd0cc59e5d2@mail.gmail.com> URL: https://gitlist.dev/e/46a038f90706091750x446c96bfkdc1d5dd0cc59e5d2%40mail.gmail.com In-Reply-To: <20070608193319.GS2335@cip.informatik.uni-erlangen.de> ``` On 6/9/07, Thomas Glanzmann wrote: > I have a very similar machine. For this one it works. It has 32 bit kernel and > userland. Debian etch again. v2.6.20. Wanted to mention -- we've been using git on debian sarge/etch on several arches a lot and never seen any lib compatiblity problem. > Does someone know a command for debian that checks the checksums apt-get install debsums man debsums # sorry, I don't use it often enough to remember ;-) cheers, m ``` ## Thomas Glanzmann, 2007-06-17 08:01 Subject: Re: fatal: serious inflate inconsistency Message-ID: <20070617080143.GG23473@cip.informatik.uni-erlangen.de> URL: https://gitlist.dev/e/20070617080143.GG23473%40cip.informatik.uni-erlangen.de In-Reply-To: <20070608110536.GD2335@cip.informatik.uni-erlangen.de> ``` Hello, the machine came back with a new power supply. And guess what: (tomcat-05) [/usr/src] git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git Initialized empty Git repository in /usr/src/linux-2.6/.git/ remote: Generating pack... remote: Done counting 499634 objects. remote: Deltifying 499634 objects... remote: 100% (499634/499634) done Indexing 499634 objects... remote: Total 499634 (delta 403845), reused 495761 (delta 400337) 100% (499634/499634) done Resolving 403845 deltas... 100% (403845/403845) done Checking 22411 files out... 100% (22411/22411) done Thomas ```