{"thread":{"id":"24379","subject":"Massive repository corruptions","startedAt":"2010-07-13T01:56:00Z","lastAt":"2010-08-05T20:42:07Z","messageCount":13,"participants":["Enrico Weigelt","Avery Pennarun","Valeo de Vries"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"145418","messageId":"20100713015600.GA29392@nibiru.local","threadId":"24379","inReplyTo":null,"subject":"Massive repository corruptions","fromName":"Enrico Weigelt","fromEmail":"weigelt@metux.de","sentAt":"2010-07-13T01:56:00Z","receivedAt":"2010-07-13T01:56:00Z","isPatch":false,"sender":{"key":"weigelt@metux.de","avatar":null},"body":"\nHi folks,\n\n\nI've just reorganized several repositories (eg. splitted off a large \nrepo into several small ones), and then I had massive corruptions\n(broken pack files) in the new repos (after they already had been clean).\n\nMaybe it has something to do with a cronjob which frequently GC's\nall the repos, and it could get even worse if the fs sometimes \ngoes full within this process.\n\nCould multiple GCs running on the same repo cause this ?\n\n\ncu\n-- \n----------------------------------------------------------------------\n Enrico Weigelt, metux IT service -- http://www.metux.de/\n\n phone: +49 36207 519931 email: weigelt@metux.de\n mobile: +49 151 27565287 icq: 210169427 skype: nekrad666\n----------------------------------------------------------------------\n Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme\n----------------------------------------------------------------------\n"},{"id":"145419","messageId":"AANLkTilXQ3VgPjihf0pjt4QPN-nCjwAWyHwoosLMeRpH@mail.gmail.com","threadId":"24379","inReplyTo":"20100713015600.GA29392@nibiru.local","subject":"Re: Massive repository corruptions","fromName":"Avery Pennarun","fromEmail":"apenwarr@gmail.com","sentAt":"2010-07-13T03:23:58Z","receivedAt":"2010-07-13T03:23:58Z","isPatch":false,"sender":{"key":"apenwarr@gmail.com","avatar":"https://avatars.githubusercontent.com/u/20592?v=4"},"body":"On Mon, Jul 12, 2010 at 9:56 PM, Enrico Weigelt wrote:\n> I've just reorganized several repositories (eg. splitted off a large\n> repo into several small ones), and then I had massive corruptions\n> (broken pack files) in the new repos (after they already had been clean).\n>\n> Maybe it has something to do with a cronjob which frequently GC's\n> all the repos, and it could get even worse if the fs sometimes\n> goes full within this process.\n>\n> Could multiple GCs running on the same repo cause this ?\n\nMultiple simultaneous gc's shouldn't be a problem - git locks things\nas it needs them. Plus, git only removes objects after it has safely\ncreated a new packfile that contains them. Maybe a filesystem filling\nup could cause a problem, but git should be detecting that if it\nhappens (maybe there's a bug that causes it to not notice, though).\n\nYou could experience corruption if your computer crashed before\neverything was synced to disk.\n\nDo you know which packfiles are corrupted? Does 'git index-pack' on\nthe files reveal anything?\n\nBe sure to make a backup copy of your corrupted repositories before\ndoing any experiments, or you might accidentally fix the problem and\nmake it harder to trace.\n\nGood luck.\n\nAvery\n"},{"id":"145422","messageId":"20100713050350.GB29392@nibiru.local","threadId":"24379","inReplyTo":"AANLkTilXQ3VgPjihf0pjt4QPN-nCjwAWyHwoosLMeRpH@mail.gmail.com","subject":"Re: Massive repository corruptions","fromName":"Enrico Weigelt","fromEmail":"weigelt@metux.de","sentAt":"2010-07-13T05:03:51Z","receivedAt":"2010-07-13T05:03:51Z","isPatch":false,"sender":{"key":"weigelt@metux.de","avatar":null},"body":"* Avery Pennarun wrote:\n\n> Multiple simultaneous gc's shouldn't be a problem - git locks things\n> as it needs them. Plus, git only removes objects after it has safely\n> created a new packfile that contains them. Maybe a filesystem filling\n> up could cause a problem, but git should be detecting that if it\n> happens (maybe there's a bug that causes it to not notice, though).\n\nOkay.\n\n> You could experience corruption if your computer crashed before\n> everything was synced to disk.\n\nNo machine crash, and no sign of filesystem or disk problems\n(according to kernel log).\n\n> Do you know which packfiles are corrupted? Does 'git index-pack' on\n> the files reveal anything?\n\ngit@blackwidow ~/metux/work.git/pack $ git index-pack pack-3b6cbd5dc5f54cf390cfaa479cac6a99d7401375.pack\nerror: inflate: data stream error (incorrect data check)\nfatal: pack has bad object at offset 37075832: inflate returned -3\n\n(that's essentially the same git-gc says)\n\n\ngit@blackwidow ~/metux/work.git/pack $ git unpack-objects -r < pack-3b6cbd5dc5f54cf390cfaa479cac6a99d7401375.pack \nerror: inflate: data stream error (incorrect data check)\nerror: inflate returned -3\n\nerror: inflate: data stream error (incorrect data check)\nerror: inflate returned -3\n\nUnpacking objects: 100% (1223/1223), done.\nfatal: final sha1 did not match\n\n\n\ncu\n-- \n----------------------------------------------------------------------\n Enrico Weigelt, metux IT service -- http://www.metux.de/\n\n phone: +49 36207 519931 email: weigelt@metux.de\n mobile: +49 151 27565287 icq: 210169427 skype: nekrad666\n----------------------------------------------------------------------\n Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme\n----------------------------------------------------------------------\n"},{"id":"145425","messageId":"20100713053154.GC29392@nibiru.local","threadId":"24379","inReplyTo":"20100713050350.GB29392@nibiru.local","subject":"Re: Massive repository corruptions","fromName":"Enrico Weigelt","fromEmail":"weigelt@metux.de","sentAt":"2010-07-13T05:31:54Z","receivedAt":"2010-07-13T05:31:54Z","isPatch":false,"sender":{"key":"weigelt@metux.de","avatar":null},"body":"* Enrico Weigelt wrote:\n\n\n\nWhat's strange: \n\nwhen copying pack files from another machine to this box and\nrun git index-pack there, it fails with the same error. \n\nalso: pushing into a new (bare) repo sometimes fails with \ninflate errors, sometimes succeeds but leaves an broken packfile.\n\n\ncu\n-- \n----------------------------------------------------------------------\n Enrico Weigelt, metux IT service -- http://www.metux.de/\n\n phone: +49 36207 519931 email: weigelt@metux.de\n mobile: +49 151 27565287 icq: 210169427 skype: nekrad666\n----------------------------------------------------------------------\n Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme\n----------------------------------------------------------------------\n"},{"id":"145430","messageId":"20100713064648.GD29392@nibiru.local","threadId":"24379","inReplyTo":"20100713053154.GC29392@nibiru.local","subject":"Re: Massive repository corruptions","fromName":"Enrico Weigelt","fromEmail":"weigelt@metux.de","sentAt":"2010-07-13T06:46:48Z","receivedAt":"2010-07-13T06:46:48Z","isPatch":false,"sender":{"key":"weigelt@metux.de","avatar":null},"body":"* Enrico Weigelt wrote:\n> * Enrico Weigelt wrote:\n> \n> \n> \n> What's strange: \n> \n> when copying pack files from another machine to this box and\n> run git index-pack there, it fails with the same error. \n> \n> also: pushing into a new (bare) repo sometimes fails with \n> inflate errors, sometimes succeeds but leaves an broken packfile.\n\nInteresting: if I limit the pack size on the local repository,\nand manually copy them over via scp, git index-pack runs fine\non them.\n\nSubsequent push doesnt seem to recognize the already transferred\npacks and still sends the big one which gets broken, but running\ngit-gc multiple times seems to clean up the mess.\n\nIs there any way for limiting the pack size on push ?\n(pack.packSizeLimit only affects git-repack, not remote transfers).\n\n\ncu\n-- \n----------------------------------------------------------------------\n Enrico Weigelt, metux IT service -- http://www.metux.de/\n\n phone: +49 36207 519931 email: weigelt@metux.de\n mobile: +49 151 27565287 icq: 210169427 skype: nekrad666\n----------------------------------------------------------------------\n Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme\n----------------------------------------------------------------------\n"},{"id":"145441","messageId":"AANLkTimQPv5MhLo4wwVTt2LiaWxqWwoYykEbz3wBS-OY@mail.gmail.com","threadId":"24379","inReplyTo":"20100713050350.GB29392@nibiru.local","subject":"Re: Massive repository corruptions","fromName":"Avery Pennarun","fromEmail":"apenwarr@gmail.com","sentAt":"2010-07-13T09:40:58Z","receivedAt":"2010-07-13T09:40:58Z","isPatch":false,"sender":{"key":"apenwarr@gmail.com","avatar":"https://avatars.githubusercontent.com/u/20592?v=4"},"body":"On Tue, Jul 13, 2010 at 1:03 AM, Enrico Weigelt wrote:\n> * Avery Pennarun wrote:\n>> Do you know which packfiles are corrupted?  Does 'git index-pack' on\n>> the files reveal anything?\n>\n> git@blackwidow ~/metux/work.git/pack $ git index-pack pack-3b6cbd5dc5f54cf390cfaa479cac6a99d7401375.pack\n> error: inflate: data stream error (incorrect data check)\n> fatal: pack has bad object at offset 37075832: inflate returned -3\n>\n> (that's essentially the same git-gc says)\n\nWhat's the size of that .pack file?\n\nAvery\n"},{"id":"145442","messageId":"AANLkTil5XgnUqm3hy4OAEBS9-ylUu7rSmKUAQXD7LcHX@mail.gmail.com","threadId":"24379","inReplyTo":"20100713053154.GC29392@nibiru.local","subject":"Re: Massive repository corruptions","fromName":"Valeo de Vries","fromEmail":"valeo@valeo.co.cc","sentAt":"2010-07-13T10:17:25Z","receivedAt":"2010-07-13T10:17:25Z","isPatch":false,"sender":{"key":"valeo@valeo.co.cc","avatar":null},"body":"On 13 July 2010 06:31, Enrico Weigelt wrote:\n> * Enrico Weigelt wrote:\n>\n> \n>\n> What's strange:\n>\n> when copying pack files from another machine to this box and\n> run git index-pack there, it fails with the same error.\n>\n> also: pushing into a new (bare) repo sometimes fails with\n> inflate errors, sometimes succeeds but leaves an broken packfile.\n\nThe pack files you copied over from another machine, were they sane\n(i.e. non-corrupt)? If so, that perhaps smells like your hard drive\ncould be on its last legs...\n"},{"id":"145443","messageId":"20100713102245.GE29392@nibiru.local","threadId":"24379","inReplyTo":"AANLkTimQPv5MhLo4wwVTt2LiaWxqWwoYykEbz3wBS-OY@mail.gmail.com","subject":"Re: Massive repository corruptions","fromName":"Enrico Weigelt","fromEmail":"weigelt@metux.de","sentAt":"2010-07-13T10:22:45Z","receivedAt":"2010-07-13T10:22:45Z","isPatch":false,"sender":{"key":"weigelt@metux.de","avatar":null},"body":"* Avery Pennarun wrote:\n> On Tue, Jul 13, 2010 at 1:03 AM, Enrico Weigelt wrote:\n> > * Avery Pennarun wrote:\n> >> Do you know which packfiles are corrupted?  Does 'git index-pack' on\n> >> the files reveal anything?\n> >\n> > git@blackwidow ~/metux/work.git/pack $ git index-pack pack-3b6cbd5dc5f54cf390cfaa479cac6a99d7401375.pack\n> > error: inflate: data stream error (incorrect data check)\n> > fatal: pack has bad object at offset 37075832: inflate returned -3\n> >\n> > (that's essentially the same git-gc says)\n> \n> What's the size of that .pack file?\n\nSomewhat over 300MB. \n\nLowering the packfile size seemed to help.\n(but I still only can do that for git-repack, not remote transfers)\n\n\ncu\n-- \n----------------------------------------------------------------------\n Enrico Weigelt, metux IT service -- http://www.metux.de/\n\n phone: +49 36207 519931 email: weigelt@metux.de\n mobile: +49 151 27565287 icq: 210169427 skype: nekrad666\n----------------------------------------------------------------------\n Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme\n----------------------------------------------------------------------\n"},{"id":"145453","messageId":"AANLkTimYeKr0asVE9mo8VcQEp5kdC18Wk5ykY9OFwixN@mail.gmail.com","threadId":"24379","inReplyTo":"20100713102245.GE29392@nibiru.local","subject":"Re: Massive repository corruptions","fromName":"Avery Pennarun","fromEmail":"apenwarr@gmail.com","sentAt":"2010-07-13T17:59:31Z","receivedAt":"2010-07-13T17:59:31Z","isPatch":false,"sender":{"key":"apenwarr@gmail.com","avatar":"https://avatars.githubusercontent.com/u/20592?v=4"},"body":"On Tue, Jul 13, 2010 at 6:22 AM, Enrico Weigelt wrote:\n> * Avery Pennarun wrote:\n>> On Tue, Jul 13, 2010 at 1:03 AM, Enrico Weigelt wrote:\n>> > * Avery Pennarun wrote:\n>> >> Do you know which packfiles are corrupted?  Does 'git index-pack' on\n>> >> the files reveal anything?\n>> >\n>> > git@blackwidow ~/metux/work.git/pack $ git index-pack pack-3b6cbd5dc5f54cf390cfaa479cac6a99d7401375.pack\n>> > error: inflate: data stream error (incorrect data check)\n>> > fatal: pack has bad object at offset 37075832: inflate returned -3\n>> >\n>> > (that's essentially the same git-gc says)\n>>\n>> What's the size of that .pack file?\n>\n> Somewhat over 300MB.\n>\n> Lowering the packfile size seemed to help.\n> (but I still only can do that for git-repack, not remote transfers)\n\nIf you got corruption at offset 37,075,832 (about 37 megs) and the\npack is over 300 megs, then the file itself is corrupted right in the\nmiddle (not truncated) and this couldn't have been caused by disk full\nerrors. Either you have memory corruption problems, or disk\ncorruption problems, or filesystem corruption problems. You'd better\nwatch out.\n\nForcing the packfile size to be smaller probably just changes your\nmemory access patterns and moves your errors around. But it doesn't\nsound like a git bug at this point.\n\nAvery\n"},{"id":"145536","messageId":"20100714132224.GF29392@nibiru.local","threadId":"24379","inReplyTo":"AANLkTimYeKr0asVE9mo8VcQEp5kdC18Wk5ykY9OFwixN@mail.gmail.com","subject":"Re: Massive repository corruptions","fromName":"Enrico Weigelt","fromEmail":"weigelt@metux.de","sentAt":"2010-07-14T13:22:24Z","receivedAt":"2010-07-14T13:22:24Z","isPatch":false,"sender":{"key":"weigelt@metux.de","avatar":null},"body":"* Avery Pennarun wrote:\n\n> If you got corruption at offset 37,075,832 (about 37 megs) and the\n> pack is over 300 megs, then the file itself is corrupted right in the\n> middle (not truncated) and this couldn't have been caused by disk full\n> errors. Either you have memory corruption problems, or disk\n> corruption problems, or filesystem corruption problems. You'd better\n> watch out.\n\nhmm, I have no signs of any hw corruption, but I had a patched\nversion of zlib installed. Maybe some of my patches broke it, \nso some strange overflow or sth like that caused that trouble.\n\nMeanwhile, after reinstalling (unpatched) zlib and recloning the\nbroken repos, everything seems fine again. Maybe some of you would\nlike to have a look at my zlib patches ;-o\n\n\ncu\n-- \n----------------------------------------------------------------------\n Enrico Weigelt, metux IT service -- http://www.metux.de/\n\n phone: +49 36207 519931 email: weigelt@metux.de\n mobile: +49 151 27565287 icq: 210169427 skype: nekrad666\n----------------------------------------------------------------------\n Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme\n----------------------------------------------------------------------\n"},{"id":"147228","messageId":"20100805173156.GB2630@nibiru.local","threadId":"24379","inReplyTo":"20100714132224.GF29392@nibiru.local","subject":"Re: Massive repository corruptions","fromName":"Enrico Weigelt","fromEmail":"weigelt@metux.de","sentAt":"2010-08-05T17:31:56Z","receivedAt":"2010-08-05T17:31:56Z","isPatch":false,"sender":{"key":"weigelt@metux.de","avatar":null},"body":"* Enrico Weigelt wrote:\n\n> hmm, I have no signs of any hw corruption, but I had a patched\n> version of zlib installed. Maybe some of my patches broke it, \n> so some strange overflow or sth like that caused that trouble.\n> \n> Meanwhile, after reinstalling (unpatched) zlib and recloning the\n> broken repos, everything seems fine again. Maybe some of you would\n> like to have a look at my zlib patches ;-o\n\nThis only seemed to help for a while. Again have trouble w/ broken \nrepos. But the strange thing: seems to affect only large ones. For \nexample, could got clone and repeatedly gc --aggressive the git\nsource w/ trouble.\n\nIf it *is* any hw problem (which isnt that unplausible since that\nmachine is the only one making trouble now), how can I detect it ?\nShouldnt broken memory or disk raise some kernel log message ?\n\n\ncu\n-- \n----------------------------------------------------------------------\n Enrico Weigelt, metux IT service -- http://www.metux.de/\n\n phone: +49 36207 519931 email: weigelt@metux.de\n mobile: +49 151 27565287 icq: 210169427 skype: nekrad666\n----------------------------------------------------------------------\n Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme\n----------------------------------------------------------------------\n"},{"id":"147243","messageId":"20100805201026.GC2630@nibiru.local","threadId":"24379","inReplyTo":"AANLkTikFypx3e-=+8J2925A++_jY-aJCDYHHw6dry5s6@mail.gmail.com","subject":"Re: Massive repository corruptions","fromName":"Enrico Weigelt","fromEmail":"weigelt@metux.de","sentAt":"2010-08-05T20:10:27Z","receivedAt":"2010-08-05T20:10:27Z","isPatch":false,"sender":{"key":"weigelt@metux.de","avatar":null},"body":"* Jussi Sirpoma wrote:\n\n> I once had a difficult to trace memory problem on a box when one of the last\n> memory banks\n> was bad. It was only used during high load situations while compiling the\n> kernel or something\n> similar. The problem was finally pinpointed by memtest86 which stresses all\n> memory.\n\nhmm, you know some way to do a memory-stresstest w/o rebooting ?\n\n\ncu\n-- \n----------------------------------------------------------------------\n Enrico Weigelt, metux IT service -- http://www.metux.de/\n\n phone: +49 36207 519931 email: weigelt@metux.de\n mobile: +49 151 27565287 icq: 210169427 skype: nekrad666\n----------------------------------------------------------------------\n Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme\n----------------------------------------------------------------------\n"},{"id":"147245","messageId":"20100805204206.GD2630@nibiru.local","threadId":"24379","inReplyTo":"AANLkTi=5HVQ2kSEt7O+OXdMRtvy8amufpFKgRpj2VLEy@mail.gmail.com","subject":"Re: Massive repository corruptions","fromName":"Enrico Weigelt","fromEmail":"weigelt@metux.de","sentAt":"2010-08-05T20:42:07Z","receivedAt":"2010-08-05T20:42:07Z","isPatch":false,"sender":{"key":"weigelt@metux.de","avatar":null},"body":"* Jussi Sirpoma wrote:\n\n> Sorry not really. Maybe compiling kernel with lots of jobs would reveal some\n> problems?\n\nokay, I'll have a try :)\n\nBTW: I'm currently recreating one of the broken repos (mozilla,\nwhich might be large enough for stresstest ;-)) by adding and\nfetching the remotes step by step. Reposity size now about 250M\n(reduced the packsize to 32M, since this already helped on some\nother repos) - yet no breaks occoured.\n\nLet's see where it goes ...\n\n\ncu\n-- \n----------------------------------------------------------------------\n Enrico Weigelt, metux IT service -- http://www.metux.de/\n\n phone: +49 36207 519931 email: weigelt@metux.de\n mobile: +49 151 27565287 icq: 210169427 skype: nekrad666\n----------------------------------------------------------------------\n Embedded-Linux / Portierung / Opensource-QM / Verteilte Systeme\n----------------------------------------------------------------------\n"}]}