{"thread":{"id":"34639","subject":"Reproducible, corrupt packfile after fresh","startedAt":"2013-08-08T11:56:30Z","lastAt":"2013-08-15T14:46:55Z","messageCount":26,"participants":["gitml.jexpert@recursor.net","Thomas Rast","Stefan Beller","Matthieu Moy","Ben Tebulin","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"224797","messageId":"f1d8f80d6fa3678ac043bfdb19bebf6bf4261273@localhost","threadId":"34639","inReplyTo":"52037F84.9060006@exxcellent.de","subject":"Reproducible, corrupt packfile after fresh","fromName":"","fromEmail":"gitml.jexpert@recursor.net","sentAt":"2013-08-08T11:56:30Z","receivedAt":"2013-08-08T11:56:30Z","isPatch":false,"sender":{"key":"gitml.jexpert@recursor.net","avatar":null},"body":"I'm a heavy user of git-svn and experience an issue with one specific\n(git-svn) repository: 'git fsck' reports a corrupt packfile after every\ncheckout.\n\nNow I'm totally puzzled about the cause and what do about it.\nThis is what I do:\n\n git svn init -s http://svn.foo.com/myproject myproject.git\n cd myproject.git\n git svn fetch # Much more reliable than 'git-svn clone'\n\nThis checks out ~2100 commits and executes 2 git-gc during checkout. The\nfinal .git repo size is about 940MB. Then I run\n\n git fsck\n\n➜ myproject.git git:(master) git fsck\nChecking object directories: 100% (256/256), done.\nerror: packed 0f5f33639bfc1a781fe080c31a1f076d9a25c1d3 from\n.git/objects/pack/pack-6a6f5355584a5d71215d5fc867ce09602ceab533.pack is\ncorrupt\n*** glibc detected *** git: free(): invalid pointer: 0x00007f46a09e9010 ***\n======= Backtrace: =========\n/lib/x86_64-linux-gnu/libc.so.6(+0x7eb96)[0x7f46d9ebab96]\ngit[0x4ddf46]\ngit[0x4b4123]\ngit[0x431524]\ngit[0x405ce8]\ngit[0x4050e2]\n/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed)[0x7f46d9e5d76d]\ngit[0x405529]\n======= Memory map: ========\n00400000-0055f000 r-xp 00000000 fc:01 12452043\n /usr/bin/git\n0075e000-0075f000 r--p 0015e000 fc:01 12452043\n /usr/bin/git\n\n....\n\n\nThis only affects this very particular git-svn repo (and I have dozens\nof them). The error happens reproducible on every fresh checkout as\ndescribed above. The backtrace does not appear always. The object is a\nvery large blob\n\n git show 0f5f33639bfc1a781fe080c31a1f076d9a25c1d3 | wc -c\n 39524691\n\n\nAny hints what to do?\n\nThanks\n- Ben\n"},{"id":"224798","messageId":"877gfw4byx.fsf@linux-k42r.v.cablecom.net","threadId":"34639","inReplyTo":"f1d8f80d6fa3678ac043bfdb19bebf6bf4261273@localhost","subject":"Re: Reproducible, corrupt packfile after fresh","fromName":"Thomas Rast","fromEmail":"trast@inf.ethz.ch","sentAt":"2013-08-08T12:13:26Z","receivedAt":"2013-08-08T12:13:26Z","isPatch":false,"sender":{"key":"tr@thomasrast.ch","avatar":"https://avatars.githubusercontent.com/u/153510?v=4"},"body":"gitml.jexpert@recursor.net writes:\n\n> ➜ myproject.git git:(master) git fsck\n> Checking object directories: 100% (256/256), done.\n> error: packed 0f5f33639bfc1a781fe080c31a1f076d9a25c1d3 from\n> .git/objects/pack/pack-6a6f5355584a5d71215d5fc867ce09602ceab533.pack is\n> corrupt\n> *** glibc detected *** git: free(): invalid pointer: 0x00007f46a09e9010 ***\n> ======= Backtrace: =========\n> /lib/x86_64-linux-gnu/libc.so.6(+0x7eb96)[0x7f46d9ebab96]\n> git[0x4ddf46]\n> git[0x4b4123]\n> git[0x431524]\n> git[0x405ce8]\n> git[0x4050e2]\n> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed)[0x7f46d9e5d76d]\n> git[0x405529]\n> ======= Memory map: ========\n> 00400000-0055f000 r-xp 00000000 fc:01 12452043\n> /usr/bin/git\n> 0075e000-0075f000 r--p 0015e000 fc:01 12452043\n> /usr/bin/git\n[...]\n> Any hints what to do?\n\nRegardless of any possible fault in git-svn, there's an obvious bug here\nwith git-fsck. Can you share the pack (if the project is public) or\ncompile a git-fsck without optimization and with debugging, and run it\nunder valgrind, to hopefully get us a backtrace of where the memory\nmanagement goes off the rails?\n\n-- \nThomas Rast\ntrast@{inf,student}.ethz.ch\n"},{"id":"224799","messageId":"52038CA9.5090602@googlemail.com","threadId":"34639","inReplyTo":"f1d8f80d6fa3678ac043bfdb19bebf6bf4261273@localhost","subject":"Re: Reproducible, corrupt packfile after fresh","fromName":"Stefan Beller","fromEmail":"stefanbeller@googlemail.com","sentAt":"2013-08-08T12:18:49Z","receivedAt":"2013-08-08T12:18:49Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On 08/08/2013 01:56 PM, gitml.jexpert@recursor.net wrote:\n> I'm a heavy user of git-svn and experience an issue with one specific\n> (git-svn) repository: 'git fsck' reports a corrupt packfile after every\n> checkout.\n> \n> Now I'm totally puzzled about the cause and what do about it.\n> This is what I do:\n> \n> git svn init -s http://svn.foo.com/myproject myproject.git\n> cd myproject.git\n> git svn fetch # Much more reliable than 'git-svn clone'\n> \n> This checks out ~2100 commits and executes 2 git-gc during checkout. The\n> final .git repo size is about 940MB. Then I run\n> \n> git fsck\n> \n> ➜ myproject.git git:(master) git fsck\n> Checking object directories: 100% (256/256), done.\n> error: packed 0f5f33639bfc1a781fe080c31a1f076d9a25c1d3 from\n> .git/objects/pack/pack-6a6f5355584a5d71215d5fc867ce09602ceab533.pack is\n> corrupt\n> *** glibc detected *** git: free(): invalid pointer: 0x00007f46a09e9010 ***\n> ======= Backtrace: =========\n> /lib/x86_64-linux-gnu/libc.so.6(+0x7eb96)[0x7f46d9ebab96]\n> git[0x4ddf46]\n> git[0x4b4123]\n> git[0x431524]\n> git[0x405ce8]\n> git[0x4050e2]\n> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed)[0x7f46d9e5d76d]\n> git[0x405529]\n> ======= Memory map: ========\n> 00400000-0055f000 r-xp 00000000 fc:01 12452043\n> /usr/bin/git\n> 0075e000-0075f000 r--p 0015e000 fc:01 12452043\n> /usr/bin/git\n> \n> ....\n> \n> \n> This only affects this very particular git-svn repo (and I have dozens\n> of them). The error happens reproducible on every fresh checkout as\n> described above. The backtrace does not appear always. The object is a\n> very large blob\n> \n> git show 0f5f33639bfc1a781fe080c31a1f076d9a25c1d3 | wc -c\n> 39524691\n> \n> \n> Any hints what to do?\n> \n> Thanks\n> - Ben\n> \n> --\n> To unsubscribe from this list: send the line \"unsubscribe git\" in\n> the body of a message to majordomo@vger.kernel.org\n> More majordomo info at http://vger.kernel.org/majordomo-info.html\n> \n\nHi,\n\nRegarding the backtrace, would it be possible to install debug symbols,\nso the backtrace is a little more meaningful?\nIf you distribution doesn't provide debug symbols, compiling git \nyourself is rather easy (git clone && make && make install \n[defaults to ~/bin, so not root required for installing apart \nfrom missing dependencies])\n\nStefan\n\n\n"},{"id":"224800","messageId":"262a9f8309a3812970f47ac9f4e4b49bb972ca49@localhost","threadId":"34639","inReplyTo":"877gfw4byx.fsf@linux-k42r.v.cablecom.net","subject":"Re: Reproducible, corrupt packfile after fresh git-svn checkout","fromName":"","fromEmail":"gitml.jexpert@recursor.net","sentAt":"2013-08-08T12:23:06Z","receivedAt":"2013-08-08T12:23:06Z","isPatch":false,"sender":{"key":"gitml.jexpert@recursor.net","avatar":null},"body":"> Regardless of any possible fault in git-svn, there's an obvious bug here\n> with git-fsck. Can you share the pack (if the project is public) or\n> compile a git-fsck without optimization and with debugging, and run it\n> under valgrind, to hopefully get us a backtrace of where the memory\n> management goes off the rails?\n\nUnfortunately I'm unable to share the pack.\n\nAs Java Developer I'm note very savy, but I'd try.\nDo you have me any pointers on \".. without optimization and with\ndebugging\" and \"run it under valgrind\"?\n\nCurrently I used\n deb http://ppa.launchpad.net/git-core/ppa/ubuntu quantal main\nas source.\n"},{"id":"224801","messageId":"87fvuk2wl0.fsf@linux-k42r.v.cablecom.net","threadId":"34639","inReplyTo":"262a9f8309a3812970f47ac9f4e4b49bb972ca49@localhost","subject":"Re: Reproducible, corrupt packfile after fresh git-svn checkout","fromName":"Thomas Rast","fromEmail":"trast@inf.ethz.ch","sentAt":"2013-08-08T12:31:07Z","receivedAt":"2013-08-08T12:31:07Z","isPatch":false,"sender":{"key":"tr@thomasrast.ch","avatar":"https://avatars.githubusercontent.com/u/153510?v=4"},"body":"gitml.jexpert@recursor.net writes:\n\n>> Regardless of any possible fault in git-svn, there's an obvious bug here\n>> with git-fsck. Can you share the pack (if the project is public) or\n>> compile a git-fsck without optimization and with debugging, and run it\n>> under valgrind, to hopefully get us a backtrace of where the memory\n>> management goes off the rails?\n>\n> Unfortunately I'm unable to share the pack.\n>\n> As Java Developer I'm note very savy, but I'd try.\n> Do you have me any pointers on \".. without optimization and with\n> debugging\" and \"run it under valgrind\"?\n>\n> Currently I used\n> deb http://ppa.launchpad.net/git-core/ppa/ubuntu quantal main\n> as source.\n\nTry something like\n\n # The package names might be wrong and/or you may need additional -dev\n # packages, I don't know the specifics for ubuntu\n apt-get install gcc make valgrind libcurl-dev zlib-dev\n cd\n git clone git://git.kernel.org/pub/scm/git/git.git\n cd git\n echo 'CFLAGS = -O0 -g' >>config.mak\n make\n cd /path/to/repo\n valgrind --track-origins=yes ~/git/git-fsck\n\nIt'll be very slow, at least 20x the normal runtime, so don't be\nsurprised if it doesn't seem to get anywhere at first.\n\n-- \nThomas Rast\ntrast@{inf,student}.ethz.ch\n"},{"id":"224802","messageId":"vpqpptofjd3.fsf@anie.imag.fr","threadId":"34639","inReplyTo":"262a9f8309a3812970f47ac9f4e4b49bb972ca49@localhost","subject":"Re: Reproducible, corrupt packfile after fresh git-svn checkout","fromName":"Matthieu Moy","fromEmail":"matthieu.moy@grenoble-inp.fr","sentAt":"2013-08-08T12:38:16Z","receivedAt":"2013-08-08T12:38:16Z","isPatch":false,"sender":{"key":"matthieu.moy@grenoble-inp.fr","avatar":"https://gravatar.com/avatar/72c8a2705971a25dfaff23cece15130d405685845d911aedd5667ace277f3fc5?d=mp&s=160"},"body":"gitml.jexpert@recursor.net writes:\n\n>> Regardless of any possible fault in git-svn, there's an obvious bug here\n>> with git-fsck. Can you share the pack (if the project is public) or\n>> compile a git-fsck without optimization and with debugging, and run it\n>> under valgrind, to hopefully get us a backtrace of where the memory\n>> management goes off the rails?\n>\n> Unfortunately I'm unable to share the pack.\n>\n> As Java Developer I'm note very savy, but I'd try.\n> Do you have me any pointers on \".. without optimization and with\n> debugging\" and \"run it under valgrind\"?\n\nTo fetch sources and compile:\n\n git clone https://github.com/git/git.git\n cd git/\n make CFLAGS='-g -O0'\n\nTo run gdb:\n\n gdb --args ./git fsck\n\nThen type \"run\" at the gdb prompt. When the program crashes, you should\nget the gdb prompt back. Then, typing \"bt\" gives you a backtrace, and\n\"bt full\" gives you the same with more information (be careful, there\nmay be private information in it).\n\nTo use valgrind, just do:\n\n valgrind ./git fsck\n\n(essentially, it runs the program adding a bunch of runtime checks hence\nmay provide better diagnosis)\n\n-- \nMatthieu Moy\nhttp://www-verimag.imag.fr/~moy/\n"},{"id":"224803","messageId":"52039205.5060307@googlemail.com","threadId":"34639","inReplyTo":"262a9f8309a3812970f47ac9f4e4b49bb972ca49@localhost","subject":"Re: Reproducible, corrupt packfile after fresh git-svn checkout","fromName":"Stefan Beller","fromEmail":"stefanbeller@googlemail.com","sentAt":"2013-08-08T12:41:41Z","receivedAt":"2013-08-08T12:41:41Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On 08/08/2013 02:23 PM, gitml.jexpert@recursor.net wrote:\n>> Regardless of any possible fault in git-svn, there's an obvious bug here\n>> with git-fsck. Can you share the pack (if the project is public) or\n>> compile a git-fsck without optimization and with debugging, and run it\n>> under valgrind, to hopefully get us a backtrace of where the memory\n>> management goes off the rails?\n> \n> Unfortunately I'm unable to share the pack.\n> \n> As Java Developer I'm note very savy, but I'd try.\n> Do you have me any pointers on \".. without optimization and with\n> debugging\" and \"run it under valgrind\"?\n> \n> Currently I used\n> deb http://ppa.launchpad.net/git-core/ppa/ubuntu quantal main\n> as source.\n\nThe version from the ppa seems to be 1.7.0.4-1ubuntu0.2 but the version\nof git being in the native ubuntu quantal repository \n(I assume you're running Ubuntu?) is already 1.7.10.4-1ubuntu1 \n(http://packages.ubuntu.com/quantal/git)\n\n\nHowever to get an unoptimized version with debug symbols you could do this:\n\n\t# get the source:\n\tgit clone git://git.kernel.org/pub/scm/git/git.git/\n\tcd git\n\n\t# read INSTALL, the section starting at \n\t# > Git is reasonably self-sufficient, but does depend on a few external\n\t# > programs and libraries\n\t# this is a guess, maybe you need more or less packages, but you'd need the -dev packages for the C-headers\n\tsudo apt-get install libz-dev libopenssl-dev libcurl-dev libexpat-dev \n\t\n\t# Now disable optimizing by overwriting default compile flags:\n\techo \"CFLAGS=-g\" > config.mak\n\n\t# compile and install\t\n\tmake \n\tmake install # installs to ~/bin\n\n\t# open a new shell or relogin, and see if you're using the version you just build:\n\tgit --version\n\t# git version 1.8.4.rc1....\n\nStefan\n\n"},{"id":"224805","messageId":"9aa1672d5c04994f416dccd84b5983c960c0fdf9@localhost","threadId":"34639","inReplyTo":"87fvuk2wl0.fsf@linux-k42r.v.cablecom.net","subject":"Re: Reproducible, corrupt packfile after fresh git-svn checkout message 3 of 20)","fromName":"","fromEmail":"gitml.jexpert@recursor.net","sentAt":"2013-08-08T13:12:34Z","receivedAt":"2013-08-08T13:12:34Z","isPatch":false,"sender":{"key":"gitml.jexpert@recursor.net","avatar":null},"body":"Great! Thank you all guys for your extensive instructions!\n\n@Thomas: I only had to add libexpat1-dev to the list.\n\nI checked out Git v1.8.3.4 as this was my used verion and built as\ninstructed. The error is still reproducible using the \"CFLAGS = -O0 -g\"\nbuild.\n\nSo - now the puzzling thing: With valgrind it seems to work! \nIf I run it plain, it doesn't:\n\n/tmp/project.git $ valgrind --track-origins=yes ~/projects/git.git/git-fsck\n==3431== Memcheck, a memory error detector\n==3431== Copyright (C) 2002-2011, and GNU GPL'd, by Julian Seward et al.\n==3431== Using Valgrind-3.7.0 and LibVEX; rerun with -h for copyright info\n==3431== Command: /home/ben/projects/git.git/git-fsck\n==3431== \n==3431== Warning: set address range perms: large range [0x70b5000, 0x2b0ff000) (defined)\nChecking object directories: 100% (256/256), done.\n==3431== Warning: set address range perms: large range [0x3959d000, 0x49d53000) (defined)\nChecking objects: 100% (63588/63588), done.\nChecking connectivity: 68696, done.\n==3431== \n==3431== HEAP SUMMARY:\n==3431== in use at exit: 7,574,911 bytes in 15,428 blocks\n==3431== total heap usage: 422,498 allocs, 407,070 frees, 4,396,939,465 bytes allocated\n==3431== \n==3431== LEAK SUMMARY:\n==3431== definitely lost: 0 bytes in 0 blocks\n==3431== indirectly lost: 0 bytes in 0 blocks\n==3431== possibly lost: 0 bytes in 0 blocks\n==3431== still reachable: 7,574,911 bytes in 15,428 blocks\n==3431== suppressed: 0 bytes in 0 blocks\n==3431== Rerun with --leak-check=full to see details of leaked memory\n==3431== \n==3431== For counts of detected and suppressed errors, rerun with: -v\n==3431== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 2 from 2)\n/tmp/project.git $ ~/projects/git.git/git-fsck\nChecking object directories: 100% (256/256), done.\nerror: packed 49cdd0b21a351f3366008615d2cf8d03ca943978 from .git/objects/pack/pack-6a6f5355584a5d71215d5fc867ce09602ceab533.pack is corrupt\n*** glibc detected *** : free(): invalid pointer: 0x00007f8576682010 ***\n======= Backtrace: =========\n/lib/x86_64-linux-gnu/libc.so.6(+0x7eb96)[0x7f85ad765b96]\n[0x4e727c]\n\n\nI'm using a i5-3340M on Linux Mint 14 with a offical Linux 3.9.9\ngcc (Ubuntu/Linaro 4.7.2-2ubuntu1) 4.7.2\n\nAny further hints?\n\n\n> Try something like\n>\n> # The package names might be wrong and/or you may need additional -dev\n> # packages, I don't know the specifics for ubuntu\n> apt-get install gcc make valgrind libcurl-dev zlib-dev\n> cd\n> git clone git://git.kernel.org/pub/scm/git/git.git\n> cd git\n> echo 'CFLAGS = -O0 -g' >>config.mak\n> make\n> cd /path/to/repo\n> valgrind --track-origins=yes ~/git/git-fsck\n>\n> It'll be very slow, at least 20x the normal runtime, so don't be\n> surprised if it doesn't seem to get anywhere at first.\n>\n"},{"id":"224807","messageId":"vpqli4cfhhe.fsf@anie.imag.fr","threadId":"34639","inReplyTo":"9aa1672d5c04994f416dccd84b5983c960c0fdf9@localhost","subject":"Re: Reproducible, corrupt packfile after fresh git-svn checkout message 3 of 20)","fromName":"Matthieu Moy","fromEmail":"matthieu.moy@grenoble-inp.fr","sentAt":"2013-08-08T13:18:53Z","receivedAt":"2013-08-08T13:18:53Z","isPatch":false,"sender":{"key":"matthieu.moy@grenoble-inp.fr","avatar":"https://gravatar.com/avatar/72c8a2705971a25dfaff23cece15130d405685845d911aedd5667ace277f3fc5?d=mp&s=160"},"body":"gitml.jexpert@recursor.net writes:\n\n> So - now the puzzling thing: With valgrind it seems to work! \n\nWeird, indeed. What about GDB ?\n\n-- \nMatthieu Moy\nhttp://www-verimag.imag.fr/~moy/\n"},{"id":"224808","messageId":"87y58c1fpo.fsf@linux-k42r.v.cablecom.net","threadId":"34639","inReplyTo":"9aa1672d5c04994f416dccd84b5983c960c0fdf9@localhost","subject":"Re: Reproducible, corrupt packfile after fresh git-svn checkout message 3 of 20)","fromName":"Thomas Rast","fromEmail":"trast@inf.ethz.ch","sentAt":"2013-08-08T13:20:51Z","receivedAt":"2013-08-08T13:20:51Z","isPatch":false,"sender":{"key":"tr@thomasrast.ch","avatar":"https://avatars.githubusercontent.com/u/153510?v=4"},"body":"gitml.jexpert@recursor.net writes:\n\n> So - now the puzzling thing: With valgrind it seems to work! \n> If I run it plain, it doesn't:\n>\n> /tmp/project.git $ valgrind --track-origins=yes ~/projects/git.git/git-fsck\n[...]\n> ==3431== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 2 from 2)\n> /tmp/project.git $ ~/projects/git.git/git-fsck\n> Checking object directories: 100% (256/256), done.\n> error: packed 49cdd0b21a351f3366008615d2cf8d03ca943978 from\n> .git/objects/pack/pack-6a6f5355584a5d71215d5fc867ce09602ceab533.pack\n> is corrupt\n> *** glibc detected *** : free(): invalid pointer: 0x00007f8576682010 ***\n> ======= Backtrace: =========\n> /lib/x86_64-linux-gnu/libc.so.6(+0x7eb96)[0x7f85ad765b96]\n> [0x4e727c]\n[...]\n> Any further hints?\n\nHrm. Can you try getting a backtrace from\n\n $ gdb ~/projects/git.git/git-fsck\n (gdb) run\n ... wait until it crashes ...\n (gdb) backtrace full\n\nI would have been more interested in error output from valgrind, because\nmemory corruption invariably happens long before glibc finally figures\nout that something is amiss. But as things stand, the backtrace is\nprobably the only thing we have...\n\n-- \nThomas Rast\ntrast@{inf,student}.ethz.ch\n"},{"id":"224809","messageId":"813506281f7a4d3a8af650c5f8ee33dd7224f193@localhost","threadId":"34639","inReplyTo":"vpqli4cfhhe.fsf@anie.imag.fr","subject":"Re: Reproducible, corrupt packfile after fresh git-svn checkout message","fromName":"","fromEmail":"gitml.jexpert@recursor.net","sentAt":"2013-08-08T13:27:12Z","receivedAt":"2013-08-08T13:27:12Z","isPatch":false,"sender":{"key":"gitml.jexpert@recursor.net","avatar":null},"body":"Am 08.08.2013 15:18, schrieb Matthieu Moy - Matthieu.Moy@grenoble-inp.fr:> gitml.jexpert@recursor.net writes:\n>> So - now the puzzling thing: With valgrind it seems to work! \n> Weird, indeed. What about GDB ?\n\n\nAh - come on. Is this another Heisenberg bug ?\nStill trying to reproduce it using gdb and/or valgrind...\n\n\n\nben@n179 /tmp/project.git $ gdb --args ~/projects/git.git/git-fsck\nGNU gdb (GDB) 7.5-ubuntu\nCopyright (C) 2012 Free Software Foundation, Inc.\nLicense GPLv3+: GNU GPL version 3 or later \nThis is free software: you are free to change and redistribute it.\nThere is NO WARRANTY, to the extent permitted by law. Type \"show copying\"\nand \"show warranty\" for details.\nThis GDB was configured as \"x86_64-linux-gnu\".\nFor bug reporting instructions, please see:\n...\nReading symbols from /home/ben/projects/git.git/git-fsck...done.\n(gdb) run\nStarting program: /home/ben/projects/git.git/git-fsck \nwarning: no loadable sections found in added symbol-file system-supplied DSO at 0x7ffff7ffa000\n[Thread debugging using libthread_db enabled]\nUsing host libthread_db library \"/lib/x86_64-linux-gnu/libthread_db.so.1\".\nChecking object directories: 100% (256/256), done.\nChecking objects: 100% (63588/63588), done.\n[Inferior 1 (process 3795) exited normally]\n(gdb) quit\n\nben@n179 /tmp/project.git $ ~/projects/git.git/git-fsck\nChecking object directories: 100% (256/256), done.\nerror: packed 49cdd0b21a351f3366008615d2cf8d03ca943978 from .git/objects/pack/pack-6a6f5355584a5d71215d5fc867ce09602ceab533.pack is corrupt\nChecking objects: 100% (63588/63588), done.\n\nben@n179 /tmp/project.git $ ~/projects/git.git/git-fsck\nChecking object directories: 100% (256/256), done.\nerror: packed 49cdd0b21a351f3366008615d2cf8d03ca943978 from .git/objects/pack/pack-6a6f5355584a5d71215d5fc867ce09602ceab533.pack is corrupt\nChecking objects: 100% (63588/63588), done.\n\nben@n179 /tmp/project.git $ ~/projects/git.git/git-fsck\nChecking object directories: 100% (256/256), done.\nChecking objects: 100% (63588/63588), done.\n\nben@n179 /tmp/project.git $ ~/projects/git.git/git-fsck\nChecking object directories: 100% (256/256), done.\nerror: packed 0f5f33639bfc1a781fe080c31a1f076d9a25c1d3 from .git/objects/pack/pack-6a6f5355584a5d71215d5fc867ce09602ceab533.pack is corrupt\n*** glibc detected *** : free(): invalid pointer: 0x00007fe129669010 ***\n======= Backtrace: =========\n/lib/x86_64-linux-gnu/libc.so.6(+0x7eb96)[0x7fe162d3eb96]\n[0x51e401]\n[0x51e53c]\n[0x51ecc3]\n[0x4e707b]\n[0x4e7485]\n[0x43d433]\n[0x405158]\n[0x4052ee]\n[0x4054ba]\n/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed)[0x7fe162ce176d]\n[0x404429]\n======= Memory map: ========\n00400000-005bb000 r-xp 00000000 fc:01 3939047 /home/ben/projects/git.git/git-fsck\n007ba000-007bb000 r--p 001ba000 fc:01 3939047 /home/ben/projects/git.git/git-fsck\n007bb000-007c2000 rw-p 001bb000 fc:01 3939047 /home/ben/projects/git.git/git-fsck\n007c2000-00822000 rw-p 00000000 00:00 0 \n02770000-05809000 rw-p 00000000 00:00 0 [heap]\n7fe129669000-7fe12bc38000 rw-p 00000000 00:00 0 \n7fe12dff5000-7fe12e00a000 r-xp 00000000 fc:01 7081526 /lib/x86_64-linux-gnu/libgcc_s.so.1\n7fe12e00a000-7fe12e209000 ---p 00015000 fc:01 7081526 /lib/x86_64-linux-gnu/libgcc_s.so.1\n7fe12e209000-7fe12e20a000 r--p 00014000 fc:01 7081526 /lib/x86_64-linux-gnu/libgcc_s.so.1\n7fe12e20a000-7fe12e20b000 rw-p 00015000 fc:01 7081526 /lib/x86_64-linux-gnu/libgcc_s.so.1\n7fe12e20b000-7fe13e9c1000 r--p 00000000 fc:01 10879034 /tmp/project.git/.git/objects/pack/pack-6a6f5355584a5d71215d5fc867ce09602ceab533.pack\n7fe13e9c1000-7fe162a0b000 r--p 00000000 fc:01 10881032 /tmp/project.git/.git/objects/pack/pack-5b132454d5acb542969a939fe55186a65d9dd697.pack\n7fe162a0b000-7fe162abc000 r--p 00000000 fc:01 10902665 /tmp/project.git/.git/objects/pack/pack-6a6f5355584a5d71215d5fc867ce09602ceab533.idx\n7fe162abc000-7fe162abe000 r-xp 00000000 fc:01 7085848 /lib/x86_64-linux-gnu/libdl-2.15.so\n7fe162abe000-7fe162cbe000 ---p 00002000 fc:01 7085848 /lib/x86_64-linux-gnu/libdl-2.15.so\n7fe162cbe000-7fe162cbf000 r--p 00002000 fc:01 7085848 /lib/x86_64-linux-gnu/libdl-2.15.so\n7fe162cbf000-7fe162cc0000 rw-p 00003000 fc:01 7085848 /lib/x86_64-linux-gnu/libdl-2.15.so\n7fe162cc0000-7fe162e75000 r-xp 00000000 fc:01 7085850 /lib/x86_64-linux-gnu/libc-2.15.so\n7fe162e75000-7fe163074000 ---p 001b5000 fc:01 7085850 /lib/x86_64-linux-gnu/libc-2.15.so\n7fe163074000-7fe163078000 r--p 001b4000 fc:01 7085850 /lib/x86_64-linux-gnu/libc-2.15.so\n7fe163078000-7fe16307a000 rw-p 001b8000 fc:01 7085850 /lib/x86_64-linux-gnu/libc-2.15.so\n7fe16307a000-7fe16307f000 rw-p 00000000 00:00 0 \n7fe16307f000-7fe163097000 r-xp 00000000 fc:01 7085833 /lib/x86_64-linux-gnu/libpthread-2.15.so\n7fe163097000-7fe163296000 ---p 00018000 fc:01 7085833 /lib/x86_64-linux-gnu/libpthread-2.15.so\n7fe163296000-7fe163297000 r--p 00017000 fc:01 7085833 /lib/x86_64-linux-gnu/libpthread-2.15.so\n7fe163297000-7fe163298000 rw-p 00018000 fc:01 7085833 /lib/x86_64-linux-gnu/libpthread-2.15.so\n7fe163298000-7fe16329c000 rw-p 00000000 00:00 0 \n7fe16329c000-7fe16344d000 r-xp 00000000 fc:01 7078062 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0\n7fe16344d000-7fe16364d000 ---p 001b1000 fc:01 7078062 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0\n7fe16364d000-7fe163668000 r--p 001b1000 fc:01 7078062 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0\n7fe163668000-7fe163673000 rw-p 001cc000 fc:01 7078062 /lib/x86_64-linux-gnu/libcrypto.so.1.0.0\n7fe163673000-7fe163677000 rw-p 00000000 00:00 0 \n7fe163677000-7fe16368d000 r-xp 00000000 fc:01 7081635 /lib/x86_64-linux-gnu/libz.so.1.2.7\n7fe16368d000-7fe16388c000 ---p 00016000 fc:01 7081635 /lib/x86_64-linux-gnu/libz.so.1.2.7\n7fe16388c000-7fe16388d000 r--p 00015000 fc:01 7081635 /lib/x86_64-linux-gnu/libz.so.1.2.7\n7fe16388d000-7fe16388e000 rw-p 00016000 fc:01 7081635 /lib/x86_64-linux-gnu/libz.so.1.2.7\n7fe16388e000-7fe1638b0000 r-xp 00000000 fc:01 7085838 /lib/x86_64-linux-gnu/ld-2.15.so\n7fe16394a000-7fe163a4d000 r--p 00000000 fc:01 10888685 /tmp/project.git/.git/objects/pack/pack-5b132454d5acb542969a939fe55186a65d9dd697.idx\n7fe163a4d000-7fe163a8c000 r--p 00000000 fc:01 1048722 /usr/lib/locale/de_DE.utf8/LC_CTYPE\n7fe163a8c000-7fe163a90000 rw-p 00000000 00:00 0 \n7fe163aa5000-7fe163aa6000 rw-p 00000000 00:00 0 \n7fe163aa6000-7fe163aad000 r--s 00000000 fc:01 12987862 /usr/lib/x86_64-linux-gnu/gconv/gconv-modules.cache\n7fe163aad000-7fe163aae000 r--p 00000000 fc:01 1050508 /usr/lib/locale/de_DE.utf8/LC_MESSAGES/SYS_LC_MESSAGES\n7fe163aae000-7fe163ab0000 rw-p 00000000 00:00 0 \n7fe163ab0000-7fe163ab1000 r--p 00022000 fc:01 7085838 /lib/x86_64-linux-gnu/ld-2.15.so\n7fe163ab1000-7fe163ab3000 rw-p 00023000 fc:01 7085838 /lib/x86_64-linux-gnu/ld-2.15.so\n7fff8299b000-7fff829bc000 rw-p 00000000 00:00 0 [stack]\n7fff829fe000-7fff82a00000 r-xp 00000000 00:00 0 [vdso]\nffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]\nAbgebrochen\n"},{"id":"224811","messageId":"vpqfvukfgr5.fsf@anie.imag.fr","threadId":"34639","inReplyTo":"813506281f7a4d3a8af650c5f8ee33dd7224f193@localhost","subject":"Re: Reproducible, corrupt packfile after fresh git-svn checkout message","fromName":"Matthieu Moy","fromEmail":"matthieu.moy@grenoble-inp.fr","sentAt":"2013-08-08T13:34:38Z","receivedAt":"2013-08-08T13:34:38Z","isPatch":false,"sender":{"key":"matthieu.moy@grenoble-inp.fr","avatar":"https://gravatar.com/avatar/72c8a2705971a25dfaff23cece15130d405685845d911aedd5667ace277f3fc5?d=mp&s=160"},"body":"gitml.jexpert@recursor.net writes:\n\n> /lib/x86_64-linux-gnu/libc.so.6(+0x7eb96)[0x7fe162d3eb96]\n> [0x51e401]\n> [0x51e53c]\n> [0x51ecc3]\n> [0x4e707b]\n> [0x4e7485]\n> [0x43d433]\n> [0x405158]\n> [0x4052ee]\n> [0x4054ba]\n\nUsing\n\n addr2line -e ~/projects/git.git/git-fsck\n\non these addresses may help a little, but not sure it's going to be\nsufficient :-(.\n\n-- \nMatthieu Moy\nhttp://www-verimag.imag.fr/~moy/\n"},{"id":"224812","messageId":"edc9f41b4ffca9ce36dc50e749de7a86e4618a0e@localhost","threadId":"34639","inReplyTo":"vpqfvukfgr5.fsf@anie.imag.fr","subject":"Re: Reproducible, corrupt packfile after fresh git-svn checkout message (gitml: message 5 of 20)","fromName":"","fromEmail":"gitml.jexpert@recursor.net","sentAt":"2013-08-08T13:43:36Z","receivedAt":"2013-08-08T13:43:36Z","isPatch":false,"sender":{"key":"gitml.jexpert@recursor.net","avatar":null},"body":"> Using\n> \n> addr2line -e ~/projects/git.git/git-fsck\n> \n> on these addresses may help a little, but not sure it's going to be\n> sufficient :-(.\n\n\nI'm still trying to reproduce this issue using gdb.\nAlso I'm trying to reproduce this issue with my git repo on another machine.\n\nben@n179 /tmp $ addr2line -e ~/projects/git.git/git-fsck\n0x51e401\n0x51e53c\n0x51ecc3\n0x4e707b\n0x4e7485\n0x43d433\n0x405158\n0x4052ee\n0x4054ba\n/home/ben/projects/git.git/sha1_file.c:1901\n/home/ben/projects/git.git/sha1_file.c:1928\n/home/ben/projects/git.git/sha1_file.c:2096\n/home/ben/projects/git.git/pack-check.c:119\n/home/ben/projects/git.git/pack-check.c:177\n/home/ben/projects/git.git/builtin/fsck.c:678\n/home/ben/projects/git.git/git.c:291\n/home/ben/projects/git.git/git.c:453\n/home/ben/projects/git.git/git.c:543\n"},{"id":"224813","messageId":"87fvuk1cyl.fsf@linux-k42r.v.cablecom.net","threadId":"34639","inReplyTo":"edc9f41b4ffca9ce36dc50e749de7a86e4618a0e@localhost","subject":"Re: Reproducible, corrupt packfile after fresh git-svn checkout message (gitml: message 5 of 20)","fromName":"Thomas Rast","fromEmail":"trast@inf.ethz.ch","sentAt":"2013-08-08T14:20:18Z","receivedAt":"2013-08-08T14:20:18Z","isPatch":false,"sender":{"key":"tr@thomasrast.ch","avatar":"https://avatars.githubusercontent.com/u/153510?v=4"},"body":"gitml.jexpert@recursor.net writes:\n\n>> Using\n>> \n>> addr2line -e ~/projects/git.git/git-fsck\n>> \n>> on these addresses may help a little, but not sure it's going to be\n>> sufficient :-(.\n>\n>\n> I'm still trying to reproduce this issue using gdb.\n> Also I'm trying to reproduce this issue with my git repo on another machine.\n>\n> ben@n179 /tmp $ addr2line -e ~/projects/git.git/git-fsck\n> 0x51e401\n> 0x51e53c\n> 0x51ecc3\n> 0x4e707b\n> 0x4e7485\n> 0x43d433\n> 0x405158\n> 0x4052ee\n> 0x4054ba\n> /home/ben/projects/git.git/sha1_file.c:1901\n> /home/ben/projects/git.git/sha1_file.c:1928\n> /home/ben/projects/git.git/sha1_file.c:2096\n> /home/ben/projects/git.git/pack-check.c:119\n> /home/ben/projects/git.git/pack-check.c:177\n> /home/ben/projects/git.git/builtin/fsck.c:678\n> /home/ben/projects/git.git/git.c:291\n> /home/ben/projects/git.git/git.c:453\n> /home/ben/projects/git.git/git.c:543\n\nCan you try to reproduce with a version older than v1.8.3?\nE.g. v1.8.2.3.\n\nI'm asking because the above points at packed_object_info(), which I\nrecently rewrote to be nonrecursive.\n\n\nAlso, can you please stop losing the Cc list? The etiquette on this\nlist is to Cc everyone who was involved so far, usually meaning everyone\nwho was already a recipient of the mail you are replying to.\n\n-- \nThomas Rast\ntrast@{inf,student}.ethz.ch\n"},{"id":"224816","messageId":"5203B100.30008@gmail.com","threadId":"34639","inReplyTo":"87fvuk1cyl.fsf@linux-k42r.v.cablecom.net","subject":"Re: Reproducible, corrupt packfile after fresh git-svn checkout message (gitml: message 5 of 20) (gitml: message 6 of 20)","fromName":"Ben Tebulin","fromEmail":"tebulin@googlemail.com","sentAt":"2013-08-08T14:53:52Z","receivedAt":"2013-08-08T14:53:52Z","isPatch":false,"sender":{"key":"tebulin@googlemail.com","avatar":null},"body":"\nAm 08.08.2013 16:20, schrieb Thomas Rast - trast@inf.ethz.ch:\n> Can you try to reproduce with a version older than v1.8.3?\n> E.g. v1.8.2.3.\n> I'm asking because the above points at packed_object_info(), which I\n> recently rewrote to be nonrecursive.\n\nIt seems to run 'much better' \n v1.8.2.3 : 3/10 runs do fail\n fb56570 : 9/10 runs do fail\n\nThey always fail on a big blob (39MB) as I wrote in my first e-mail:\n\nben@n179 /tmp/project.git $ ~/projects/git.git/git-show 49cdd0b21a351f3366008615d2cf8d03ca943978 | wc -c\nerror: sha1 mismatch 49cdd0b21a351f3366008615d2cf8d03ca943978\nfatal: bad object 49cdd0b21a351f3366008615d2cf8d03ca943978\n0\nben@n179 /tmp/project.git $ ~/projects/git.git/git-show 49cdd0b21a351f3366008615d2cf8d03ca943978 | wc -c\n39517156\n\n\n> Also, can you please stop losing the Cc list? \nI'm _very_ sorry for this. I was trying to hide my e-mail \naddress from spam robots using spamgourmet.com as remailer.\nUnfortunately it breaks CC. Switching e-mail now.\n"},{"id":"224822","messageId":"87txj0i4mc.fsf@inf.ethz.ch","threadId":"34639","inReplyTo":"5203B100.30008@gmail.com","subject":"Re: Reproducible, corrupt packfile after fresh git-svn checkout message (gitml: message 5 of 20) (gitml: message 6 of 20)","fromName":"Thomas Rast","fromEmail":"trast@inf.ethz.ch","sentAt":"2013-08-08T15:28:27Z","receivedAt":"2013-08-08T15:28:27Z","isPatch":false,"sender":{"key":"tr@thomasrast.ch","avatar":"https://avatars.githubusercontent.com/u/153510?v=4"},"body":"Ben Tebulin writes:\n\n> Am 08.08.2013 16:20, schrieb Thomas Rast - trast@inf.ethz.ch:\n>> Can you try to reproduce with a version older than v1.8.3?\n>> E.g. v1.8.2.3.\n>> I'm asking because the above points at packed_object_info(), which I\n>> recently rewrote to be nonrecursive.\n>\n> It seems to run 'much better' \n> v1.8.2.3 : 3/10 runs do fail\n> fb56570 : 9/10 runs do fail\n\nThe good news is that this shifts the blame away from my commit ;-) as\nthe problem clearly existed even before that.\n\nThe bad news, of course, is that this is another hunch that turned out\nto be wrong. I'm running out of ideas.\n\n> They always fail on a big blob (39MB) as I wrote in my first e-mail:\n>\n> ben@n179 /tmp/project.git $ ~/projects/git.git/git-show 49cdd0b21a351f3366008615d2cf8d03ca943978 | wc -c\n> error: sha1 mismatch 49cdd0b21a351f3366008615d2cf8d03ca943978\n> fatal: bad object 49cdd0b21a351f3366008615d2cf8d03ca943978\n> 0\n> ben@n179 /tmp/project.git $ ~/projects/git.git/git-show 49cdd0b21a351f3366008615d2cf8d03ca943978 | wc -c\n> 39517156\n\nHrmm. I wonder about the significance of those 39MB. What is your\ncore.packedGitWindowSize? (Judging from the pastes you seem to be on\n64bit, so the default would be 1GB.)\n\n-- \nThomas Rast\ntrast@{inf,student}.ethz.ch\n"},{"id":"224824","messageId":"5203C923.8060905@gmail.com","threadId":"34639","inReplyTo":"87txj0i4mc.fsf@inf.ethz.ch","subject":"Re: Reproducible, corrupt packfile after fresh git-svn checkout message","fromName":"Ben Tebulin","fromEmail":"tebulin@googlemail.com","sentAt":"2013-08-08T16:36:51Z","receivedAt":"2013-08-08T16:36:51Z","isPatch":false,"sender":{"key":"tebulin@googlemail.com","avatar":null},"body":"I was unable to reproduce the error with the same repo and same Git\nversion on a different machine (Debian Squeeze x64 on a AMD Phenom x6\n1045T).\n\n\n> I'm running out of ideas.\nMe, too. Based on out current observations I'd assume one of:\n\na) a rare, timing-sensitive bug in Git\nb) a compiler/distribution/environment sensitive issue\nc) or defect/buggy Hardware (CPU, Memory)\n\n> Hrmm. I wonder about the significance of those 39MB. What is your\n> core.packedGitWindowSize? (Judging from the pastes you seem to be on\n> 64bit, so the default would be 1GB.)\nThe default. I have 12GB RAM.\n\nSo I'll try to rule out b) and c) as far as I can and report in if I\nhave any new findings.\n\nNevertheless thank you very much for your extensive assistance!\n\n- Ben\n"},{"id":"224825","messageId":"vpqob98cena.fsf@anie.imag.fr","threadId":"34639","inReplyTo":"5203C923.8060905@gmail.com","subject":"Re: Reproducible, corrupt packfile after fresh git-svn checkout message","fromName":"Matthieu Moy","fromEmail":"matthieu.moy@grenoble-inp.fr","sentAt":"2013-08-08T16:48:25Z","receivedAt":"2013-08-08T16:48:25Z","isPatch":false,"sender":{"key":"matthieu.moy@grenoble-inp.fr","avatar":"https://gravatar.com/avatar/72c8a2705971a25dfaff23cece15130d405685845d911aedd5667ace277f3fc5?d=mp&s=160"},"body":"Ben Tebulin writes:\n\n> c) or defect/buggy Hardware (CPU, Memory)\n\nYou may want to try a memcheck86 on your machine for a few hours, in\ncase ... (if your RAM is broken, you do want to know it!).\n\n-- \nMatthieu Moy\nhttp://www-verimag.imag.fr/~moy/\n"},{"id":"224832","messageId":"7vsiyk9j6a.fsf@alter.siamese.dyndns.org","threadId":"34639","inReplyTo":"87txj0i4mc.fsf@inf.ethz.ch","subject":"Re: Reproducible, corrupt packfile after fresh git-svn checkout message (gitml: message 5 of 20) (gitml: message 6 of 20)","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-08-08T17:38:53Z","receivedAt":"2013-08-08T17:38:53Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Thomas Rast writes:\n\n> Ben Tebulin writes:\n>\n>> Am 08.08.2013 16:20, schrieb Thomas Rast - trast@inf.ethz.ch:\n>>> Can you try to reproduce with a version older than v1.8.3?\n>>> E.g. v1.8.2.3.\n>>> I'm asking because the above points at packed_object_info(), which I\n>>> recently rewrote to be nonrecursive.\n>>\n>> It seems to run 'much better' \n>> v1.8.2.3 : 3/10 runs do fail\n>> fb56570 : 9/10 runs do fail\n>\n> The good news is that this shifts the blame away from my commit ;-) as\n> the problem clearly existed even before that.\n>\n> The bad news, of course, is that this is another hunch that turned out\n> to be wrong. I'm running out of ideas.\n>\n>> They always fail on a big blob (39MB) as I wrote in my first e-mail:\n>>\n>> ben@n179 /tmp/project.git $ ~/projects/git.git/git-show 49cdd0b21a351f3366008615d2cf8d03ca943978 | wc -c\n>> error: sha1 mismatch 49cdd0b21a351f3366008615d2cf8d03ca943978\n>> fatal: bad object 49cdd0b21a351f3366008615d2cf8d03ca943978\n>> 0\n>> ben@n179 /tmp/project.git $ ~/projects/git.git/git-show 49cdd0b21a351f3366008615d2cf8d03ca943978 | wc -c\n>> 39517156\n\nHmm, from this, and a later one ...\n\n> Ben Tebulin writes:\n>\n> I was unable to reproduce the error with the same repo and same Git\n> version on a different machine (Debian Squeeze x64 on a AMD Phenom x6\n> 1045T).\n\n... I am reading that (1) the packfile and repository is basically\nOK, (2) reading that object sometimes fails, and (3) the symptom is\nnot limited to fsck but anything that reads the object with\nparse_object(). And that symptom exists only on that single machine\n(I am assuming that the repository was bit-for-bit copied, not\n\"cloned\", for the purpose of testing it on the other machine). That\nmakes me suspect something outside the control of Git (e.g. faulty\nmemory or disk controller cable).\n\nAre there other big blobs in the repository, and would \"show | wc\" fail\nif you attempt to read it on that machine?\n\nThanks all for helping to track this, by the way.\n"},{"id":"224895","messageId":"5204E018.4000808@gmail.com","threadId":"34639","inReplyTo":"7vsiyk9j6a.fsf@alter.siamese.dyndns.org","subject":"Re: Reproducible, corrupt packfile after fresh git-svn checkout message","fromName":"Ben Tebulin","fromEmail":"tebulin@googlemail.com","sentAt":"2013-08-09T12:27:04Z","receivedAt":"2013-08-09T12:27:04Z","isPatch":false,"sender":{"key":"tebulin@googlemail.com","avatar":null},"body":"Hello everybody!\n\nI have some _very interesting_ news regarding this issue!\nHere is the deal:\n\n 1. I was able to *reproduce the error on a machine of a coworker!*\n\n 2. I was able to rule out\n - HDD: It's reproducible from /dev/shm\n - Memory: Memory tests works fine\n\nnow the interesting part:\n\n 3. Occurs on Linux kernels 3.7.10, 3.8.x, 3.9.11, 3.10.5\n _but not on:_ 3.6.11, 3.5.7 and 2.6.32\n\nBoth machines showing this problem are »Dell Latitude E6330« with an\n»i5-3340M @ 2.70GHz« CPU. Mine running stock kernels, coworker using\nplain Ubuntu 13.04.\n\nFurthermore I need to stress, that we never had any issues with our\ndevices during daily _at all_.\n\nSo what to do best now?\n\n- Ben\n\n\nAm 08.08.2013 19:38, schrieb Junio C Hamano:\n>>>> Can you try to reproduce with a version older than v1.8.3?\n>>>> E.g. v1.8.2.3.\n>>>\n>>> It seems to run 'much better' \n>>> v1.8.2.3 : 3/10 runs do fail\n>>> fb56570 : 9/10 runs do fail\n[..]\n>>> They always fail on a big blob (39MB) as I wrote in my first e-mail:\n>>>\n>>> ben@n179 /tmp/project.git $ ~/projects/git.git/git-show 49cdd0b21a351f3366008615d2cf8d03ca943978 | wc -c\n>>> error: sha1 mismatch 49cdd0b21a351f3366008615d2cf8d03ca943978\n>>> fatal: bad object 49cdd0b21a351f3366008615d2cf8d03ca943978\n>>> 0\n>>> ben@n179 /tmp/project.git $ ~/projects/git.git/git-show 49cdd0b21a351f3366008615d2cf8d03ca943978 | wc -c\n>>> 39517156\n> \n> Hmm, from this, and a later one ...\n> \n> ... I am reading that (1) the packfile and repository is basically\n> OK, (2) reading that object sometimes fails, and (3) the symptom is\n> not limited to fsck but anything that reads the object with\n> parse_object(). And that symptom exists only on that single machine\n> (I am assuming that the repository was bit-for-bit copied, not\n> \"cloned\", for the purpose of testing it on the other machine). That\n> makes me suspect something outside the control of Git (e.g. faulty\n> memory or disk controller cable).\n> \n> Are there other big blobs in the repository, and would \"show | wc\" fail\n> if you attempt to read it on that machine?\n"},{"id":"224899","messageId":"5204E67D.9090106@googlemail.com","threadId":"34639","inReplyTo":"5204E018.4000808@gmail.com","subject":"Re: Reproducible, corrupt packfile after fresh git-svn checkout message","fromName":"Stefan Beller","fromEmail":"stefanbeller@googlemail.com","sentAt":"2013-08-09T12:54:21Z","receivedAt":"2013-08-09T12:54:21Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On 08/09/2013 02:27 PM, Ben Tebulin wrote:\n> Hello everybody!\n> \n> I have some _very interesting_ news regarding this issue!\n> Here is the deal:\n> \n> 1. I was able to *reproduce the error on a machine of a coworker!*\n> \n> 2. I was able to rule out\n> - HDD: It's reproducible from /dev/shm\n> - Memory: Memory tests works fine\n> \n> now the interesting part:\n> \n> 3. Occurs on Linux kernels 3.7.10, 3.8.x, 3.9.11, 3.10.5\n> _but not on:_ 3.6.11, 3.5.7 and 2.6.32\n> \n> Both machines showing this problem are »Dell Latitude E6330« with an\n> »i5-3340M @ 2.70GHz« CPU. Mine running stock kernels, coworker using\n> plain Ubuntu 13.04.\n> \n> Furthermore I need to stress, that we never had any issues with our\n> devices during daily _at all_.\n> \n> So what to do best now?\n\nMaybe include the kernel mailing list, they know what's up ;)\nLinking to the start of the discussion $gmane/33602\nMaybe you could also run git through strace and diff the output of a\nfaulty machine with a correct machine?\n\nStefan\n\n\n"},{"id":"225192","messageId":"877gfok59m.fsf@linux-k42r.v.cablecom.net","threadId":"34639","inReplyTo":"5204E018.4000808@gmail.com","subject":"Re: Reproducible, corrupt packfile after fresh git-svn checkout message","fromName":"Thomas Rast","fromEmail":"trast@inf.ethz.ch","sentAt":"2013-08-14T09:09:41Z","receivedAt":"2013-08-14T09:09:41Z","isPatch":false,"sender":{"key":"tr@thomasrast.ch","avatar":"https://avatars.githubusercontent.com/u/153510?v=4"},"body":"Ben Tebulin writes:\n\n> Hello everybody!\n>\n> I have some _very interesting_ news regarding this issue!\n> Here is the deal:\n>\n> 1. I was able to *reproduce the error on a machine of a coworker!*\n>\n> 2. I was able to rule out\n> - HDD: It's reproducible from /dev/shm\n> - Memory: Memory tests works fine\n>\n> now the interesting part:\n>\n> 3. Occurs on Linux kernels 3.7.10, 3.8.x, 3.9.11, 3.10.5\n> _but not on:_ 3.6.11, 3.5.7 and 2.6.32\n>\n> Both machines showing this problem are »Dell Latitude E6330« with an\n> »i5-3340M @ 2.70GHz« CPU. Mine running stock kernels, coworker using\n> plain Ubuntu 13.04.\n>\n> Furthermore I need to stress, that we never had any issues with our\n> devices during daily _at all_.\n>\n> So what to do best now?\n\nIf you're still interested...\n\nWhat did memcheck86 say about these machines?\n\n\nMy best theory so far:\n\nmalloc()/free() actually use mmap()/munmap() for large allocations.\nmallopt(3) tells me that \"large\" here means a dynamic limit on modern\nglibc, but with an upper limit of 32MB on 64-bit. So your 39MB blob\nwould always result in an mmap() type allocation.\n\nAnd it should be easy for glibc to check mmap() type allocations against\ndouble free(), at least if the address has not been mapped again in the\nsame process. So it probably does that, which aborts the program.\n\nYou could try to gather some data on this by looking at\n\n strace -o fsck.trace -f git fsck\n\nfor a crash case. If the theory is correct, you should see an munmap()\nfor an address similar to the one pointed out by the glibc error\nmessage, as follows: if you get\n\n *** glibc detected *** : free(): invalid pointer: 0x00007fe129669010 ***\n\ncheck for the address 0x7fe129669000 (the start of the page that the\naddress is in).\n\nNote that this is a pretty weak theory, because double free() detection\nis squarely up valgrind's alley. So valgrind's lack of complaints is a\nstrong argument against it. But it's the best I've got so far.\n\n-- \nThomas Rast\ntrast@{inf,student}.ethz.ch\n"},{"id":"225195","messageId":"520B4DF2.207@gmail.com","threadId":"34639","inReplyTo":"5204E67D.9090106@googlemail.com","subject":"Reproducible, corrupt packfile after fresh git-svn checkouts - IDENTIFIED","fromName":"Ben Tebulin","fromEmail":"tebulin@googlemail.com","sentAt":"2013-08-14T09:29:22Z","receivedAt":"2013-08-14T09:29:22Z","isPatch":false,"sender":{"key":"tebulin@googlemail.com","avatar":null},"body":"Hello everybody!\n\nI'm the guy having the issue, that in a particular Git repository the\nreads of an object fail the SHA1 reproducible on *two independent\nmachines* .\n\nThe issue occurs on a very specific repository starting with Linux\nKernel 3.7.2. Unfortunately my E-Mail to the kernel list dit get no\nresponse.\n\nAfter a long session of rebooting and bisecting the Linux kernel\n(fortunately I had a SSD and ccache!) I was able to pinpoint this to the\nfollowing patch related to _memory management_ causing my data\ncorruption with git:\n\n*\"mm: limit mmu_gather batching to fix soft lockups on !CONFIG_PREEMPT\"*\n 787f7301074ccd07a3e82236ca41eefd245f4e07 linux stable [1]\n 53a59fc67f97374758e63a9c785891ec62324c81 upstream commit [2]\n\nThe patch speaks about \"batching [memory] pages to be freed\" so I'm a\nbit unsure: Is this a Git or a Linux kernel issue?\n\n\nThanks\n- Ben\n\n\n[1] https://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/commit/?id=787f7301074ccd07a3e82236ca41eefd245f4e07\n[2] https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=53a59fc67f97374758e63a9c785891ec62324c81\n"},{"id":"225197","messageId":"520B5221.3040308@gmail.com","threadId":"34639","inReplyTo":"877gfok59m.fsf@linux-k42r.v.cablecom.net","subject":"Re: Reproducible, corrupt packfile after fresh git-svn checkout message","fromName":"Ben Tebulin","fromEmail":"tebulin@googlemail.com","sentAt":"2013-08-14T09:47:13Z","receivedAt":"2013-08-14T09:47:13Z","isPatch":false,"sender":{"key":"tebulin@googlemail.com","avatar":null},"body":"\n> My best theory so far:\n> malloc()/free() actually use mmap()/munmap() for large allocations.\n> mallopt(3) tells me that [...]\n\nMany things I don't grab at first go :-)\n\nLast night I did a long git-bisect of the kernel and was able to\npinpoint a change in the Linux memory management as cause (see my\npre-prepared email I just sent).\n\nWould that fit into your theory?\n\n> You could try to gather some data on this by [...]\nBefore I dive into that maybe you could throw a quick look at the\ncausing Kernel change (which reading from the words _could_ be related)\nif this fits into the picture?\n\nIf yes, I'm happy to try to assist you in further pinning this issue down!\n"},{"id":"225255","messageId":"520CA02C.1020801@gmail.com","threadId":"34639","inReplyTo":"877gfok59m.fsf@linux-k42r.v.cablecom.net","subject":"Re: Reproducible, corrupt packfile after fresh git-svn checkout message","fromName":"Ben Tebulin","fromEmail":"tebulin@googlemail.com","sentAt":"2013-08-15T09:32:28Z","receivedAt":"2013-08-15T09:32:28Z","isPatch":false,"sender":{"key":"tebulin@googlemail.com","avatar":null},"body":"Just as a catchup for everybody being interested:\n\nI finally wrote to the linux-mm newsgroup and Linus pointed out, that\nthis might be a known bug yet not fixed in mainline.\n\nUnfortunately this doesn't seem to stand the test; but as far as Git is\nconcerned, it appears that that they are willing to take actions.\n\nhttp://thread.gmane.org/gmane.linux.kernel/1541707/focus=105432\n"},{"id":"225263","messageId":"CAPc5daUWd==-cy7qaj6Y3F2KVAH=P2me19LxLuKFC5BCcPbMMg@mail.gmail.com","threadId":"34639","inReplyTo":"520CA02C.1020801@gmail.com","subject":"Re: Reproducible, corrupt packfile after fresh git-svn checkout message","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-08-15T14:46:55Z","receivedAt":"2013-08-15T14:46:55Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"On Thu, Aug 15, 2013 at 2:32 AM, Ben Tebulin wrote:\n> Just as a catchup for everybody being interested:\n>\n> I finally wrote to the linux-mm newsgroup and Linus pointed out, that\n> this might be a known bug yet not fixed in mainline.\n>\n> Unfortunately this doesn't seem to stand the test; but as far as Git is\n> concerned, it appears that that they are willing to take actions.\n>\n> http://thread.gmane.org/gmane.linux.kernel/1541707/focus=105432\n\nThanks for following this up.\n"}]}