{"thread":{"id":"25539","subject":"fsck errors on newly cloned, newly imported git repository","startedAt":"2010-10-24T15:54:04Z","lastAt":"2010-10-28T23:40:15Z","messageCount":5,"participants":["Mike Herrick","Drew Northup"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"154339","messageId":"AANLkTimxXXNxUOMQyDDoW9+vT9aKL5C5m+VD51jk0zL9@mail.gmail.com","threadId":"25539","inReplyTo":null,"subject":"fsck errors on newly cloned, newly imported git repository","fromName":"Mike Herrick","fromEmail":"mike.herrick@gmail.com","sentAt":"2010-10-24T15:54:04Z","receivedAt":"2010-10-24T15:54:04Z","isPatch":false,"sender":{"key":"mike.herrick@gmail.com","avatar":null},"body":"This weekend we're cutting over to use git for our source code control\nsystem. I've imported about 20 years worth of previous history using\n\"git cvsimport\" (takes about four hours). I then cloned the resulting\nrepository onto five different machines (four Linux, one Solaris).\nI've set up a cron job to do a nightly \"git fsck\" on each of the five\nmachines, and last night, two of the machines reported fsck errors on\ntheir initial run. Here's a sample of the errors:\n\nerror: packed cd00921f75f91985d1b67181632a4764af50d4e8 from\n.git/objects/pack/pack-b17f2e0a970084fed1f7a6c7664601e78059063f.pack\nis corrupt\nerror: sha1 mismatch 20abcd833a10aad51ff7f59b6a5e179d77e9a388\n\nerror: 20abcd833a10aad51ff7f59b6a5e179d77e9a388: object corrupt or missing\nerror: sha1 mismatch 343c28f127a5e5b9b85b0bdc5419e131b10ff2f0\n...\nbroken link from tree df17bc72fd5f7cea686f97e14f71f8464149ed25\n to blob d085a51be07285bec9ccf0323a7cf47856dbb31f\nbroken link from tree ab9e5b7383bcde71680abe552e30ae5abf64cf6d\n to blob 83e8475441911692d1a63d0272e17d62d1b7b8d1\n...\nmissing blob ad3209e27bbc3676bf06f889779908928948b65a\nmissing blob 4d4829314e64e2a0524fa520f59f7d18482e2b0a\nmissing blob 9ed481a1970e5f38b1479241ed21a2296c09cda0\n...\n\nThe errors reported on these two machines were different, but what's\ninteresting is that all of the missing blobs refer to various\nrevisions of the same file, namely our \"Changes\" file (which is\nupdated with each change). It's also the largest file in our\nrepository (3.3M). I immediately started looking at logs to see if\nthere was any indication of disk corruption and found none (no SMART\nerrors either). Both of these machines have been stable over a\nmulti-year period of time (no unexplained crashes). They're also\nolder Linux machines (running 2.6.5-1.358 and 2.6.1-1.65, with\nrelatively little memory: 1GB and .5GB), but with newly installed\nversion of git (1.7.3.1). I initially used git-daemon for the clone\nprocess, but even using ssh, I still see fsck errors on the resulting\nclones on these two machines.\n\nIf I don't find an explanation for this behavior, our conversion to\ngit will need to be backed out before development begins tomorrow :-(.\n\nThanks for any pointers.\n\nMike.\n"},{"id":"154400","messageId":"1288004282.819.26.camel@drew-northup.unet.maine.edu","threadId":"25539","inReplyTo":"AANLkTimxXXNxUOMQyDDoW9+vT9aKL5C5m+VD51jk0zL9@mail.gmail.com","subject":"Re: fsck errors on newly cloned, newly imported git repository","fromName":"Drew Northup","fromEmail":"drew.northup@maine.edu","sentAt":"2010-10-25T10:58:02Z","receivedAt":"2010-10-25T10:58:02Z","isPatch":false,"sender":{"key":"drew.northup@maine.edu","avatar":"https://avatars.githubusercontent.com/u/18331571?v=4"},"body":"\nOn Sun, 2010-10-24 at 11:54 -0400, Mike Herrick wrote:\n> This weekend we're cutting over to use git for our source code control\n> system. I've imported about 20 years worth of previous history using\n> \"git cvsimport\" (takes about four hours). I then cloned the resulting\n> repository onto five different machines (four Linux, one Solaris).\n> I've set up a cron job to do a nightly \"git fsck\" on each of the five\n> machines, and last night, two of the machines reported fsck errors on\n> their initial run. \n\n\n> The errors reported on these two machines were different, but what's\n> interesting is that all of the missing blobs refer to various\n> revisions of the same file, namely our \"Changes\" file (which is\n> updated with each change). It's also the largest file in our\n> repository (3.3M). I immediately started looking at logs to see if\n> there was any indication of disk corruption and found none (no SMART\n> errors either). Both of these machines have been stable over a\n> multi-year period of time (no unexplained crashes). They're also\n> older Linux machines (running 2.6.5-1.358 and 2.6.1-1.65, with\n> relatively little memory: 1GB and .5GB), but with newly installed\n> version of git (1.7.3.1). I initially used git-daemon for the clone\n> process, but even using ssh, I still see fsck errors on the resulting\n> clones on these two machines.\n\nDid you \"git fsck\" BEFORE you attempted to clone? Is it ONLY clones\nshowing errors? Alas, no blatant evidence of disk corruption is not\nevidence of no disk corruption as well.\n\n-- \n-Drew Northup N1XIM\n AKA RvnPhnx on OPN\n________________________________________________\n\"As opposed to vegetable or mineral error?\"\n-John Pescatore, SANS NewsBites Vol. 12 Num. 59\n"},{"id":"154402","messageId":"AANLkTi=7mg-1Q1hHPA52MiHa3BXF95SGTTTTLFR9+VZ3@mail.gmail.com","threadId":"25539","inReplyTo":"1288004282.819.26.camel@drew-northup.unet.maine.edu","subject":"Re: fsck errors on newly cloned, newly imported git repository","fromName":"Mike Herrick","fromEmail":"mike.herrick@gmail.com","sentAt":"2010-10-25T12:25:10Z","receivedAt":"2010-10-25T12:25:10Z","isPatch":false,"sender":{"key":"mike.herrick@gmail.com","avatar":null},"body":"On Mon, Oct 25, 2010 at 6:58 AM, Drew Northup wrote:\n>\n> On Sun, 2010-10-24 at 11:54 -0400, Mike Herrick wrote:\n>> This weekend we're cutting over to use git for our source code control\n>> system.  I've imported about 20 years worth of previous history using\n>> \"git cvsimport\" (takes about four hours).  I then cloned the resulting\n>> repository onto five different machines (four Linux, one Solaris).\n>> I've set up a cron job to do a nightly \"git fsck\" on each of the five\n>> machines, and last night, two of the machines reported fsck errors on\n>> their initial run.\n> \n>\n>> The errors reported on these two machines were different, but what's\n>> interesting is that all of the missing blobs refer to various\n>> revisions of the same file, namely our \"Changes\" file (which is\n>> updated with each change).  It's also the largest file in our\n>> repository (3.3M).  I immediately started looking at logs to see if\n>> there was any indication of disk corruption and found none (no SMART\n>> errors either).  Both of these machines have been stable over a\n>> multi-year period of time (no unexplained crashes).  They're also\n>> older Linux machines (running  2.6.5-1.358 and  2.6.1-1.65, with\n>> relatively little memory: 1GB and .5GB), but with newly installed\n>> version of git (1.7.3.1).  I initially used git-daemon for the clone\n>> process, but even using ssh, I still see fsck errors on the resulting\n>> clones on these two machines.\n>\n> Did you \"git fsck\" BEFORE you attempted to clone? Is it ONLY clones\n> showing errors? Alas, no blatant evidence of disk corruption is not\n> evidence of no disk corruption as well.\n\nThanks for your reply.\n\nOnly two of the five clones exhibit fsck errors and the server\nrepository has no fsck errors.\n\nThe two machines report different sets of missing blobs, but always in\nthe \"Changes\" file (which has the somewhat unique characteristics that\nit is the \"most changed\" file in the repository, the largest, and one\nwhich is almost always only added to).\n\nI've since created two more clones on one of the machines (one using\ngit-daemon and the other ssh) and both of these clones have the exact\nsame set of missing blobs! For me this rules out disk corruption.\n\nThe good(?) news is that the process is repeatable on one machine:\ncloning from a known good repository results in different (but\nrepeatable) errors. Performing a second clone on the other \"bad\"\nmachine also results in missing blobs, but different ones than the\nfirst (although all in the Changes file).\n\nMy current thought is that somehow it's related to very old kernels?\nApparently these machines are FC2 vintage.\n\nMike.\n"},{"id":"154407","messageId":"AANLkTinfDh5_-u6nJc=MMs_vPQGuBPhOT-TqgVOvAMPo@mail.gmail.com","threadId":"25539","inReplyTo":"AANLkTi=7mg-1Q1hHPA52MiHa3BXF95SGTTTTLFR9+VZ3@mail.gmail.com","subject":"Re: fsck errors on newly cloned, newly imported git repository","fromName":"Mike Herrick","fromEmail":"mike.herrick@gmail.com","sentAt":"2010-10-25T14:02:31Z","receivedAt":"2010-10-25T14:02:31Z","isPatch":false,"sender":{"key":"mike.herrick@gmail.com","avatar":null},"body":"On Mon, Oct 25, 2010 at 8:25 AM, Mike Herrick wrote:\n> On Mon, Oct 25, 2010 at 6:58 AM, Drew Northup wrote:\n>>\n>> On Sun, 2010-10-24 at 11:54 -0400, Mike Herrick wrote:\n>>> This weekend we're cutting over to use git for our source code control\n>>> system.  I've imported about 20 years worth of previous history using\n>>> \"git cvsimport\" (takes about four hours).  I then cloned the resulting\n>>> repository onto five different machines (four Linux, one Solaris).\n>>> I've set up a cron job to do a nightly \"git fsck\" on each of the five\n>>> machines, and last night, two of the machines reported fsck errors on\n>>> their initial run.\n>> \n>>\n>>> The errors reported on these two machines were different, but what's\n>>> interesting is that all of the missing blobs refer to various\n>>> revisions of the same file, namely our \"Changes\" file (which is\n>>> updated with each change).  It's also the largest file in our\n>>> repository (3.3M).  I immediately started looking at logs to see if\n>>> there was any indication of disk corruption and found none (no SMART\n>>> errors either).  Both of these machines have been stable over a\n>>> multi-year period of time (no unexplained crashes).  They're also\n>>> older Linux machines (running  2.6.5-1.358 and  2.6.1-1.65, with\n>>> relatively little memory: 1GB and .5GB), but with newly installed\n>>> version of git (1.7.3.1).  I initially used git-daemon for the clone\n>>> process, but even using ssh, I still see fsck errors on the resulting\n>>> clones on these two machines.\n>>\n>> Did you \"git fsck\" BEFORE you attempted to clone? Is it ONLY clones\n>> showing errors? Alas, no blatant evidence of disk corruption is not\n>> evidence of no disk corruption as well.\n>\n> Thanks for your reply.\n>\n> Only two of the five clones exhibit fsck errors and the server\n> repository has no fsck errors.\n>\n> The two machines report different sets of missing blobs, but always in\n> the \"Changes\" file (which has the somewhat unique characteristics that\n> it is the \"most changed\" file in the repository, the largest, and one\n> which is almost always only added to).\n>\n> I've since created two more clones on one of the machines (one using\n> git-daemon and the other ssh) and both of these clones have the exact\n> same set of missing blobs!  For me this rules out disk corruption.\n>\n> The good(?) news is that the process is repeatable on one machine:\n> cloning from a known good repository results in different (but\n> repeatable) errors.  Performing a second clone on the other \"bad\"\n> machine also results in missing blobs, but different ones than the\n> first (although all in the Changes file).\n>\n> My current thought is that somehow it's related to very old kernels?\n> Apparently these machines are FC2 vintage.\n\nWe've backed out of our git cutover due to these errors.\n\nI should also point out that on the machine where the errors are\nrepeatable, two of the clones were made to a local disk and one to an\nNFS disk, and all three showed the same missing blobs (another\nindication that it is unlikely to be a disk problem).\n\nIt's also interesting that the missing blobs seem to be in the same\ngeneral timeframe, 2001-2002 on one machine and 2008-2009 on the other\nmachine (as evidenced by the file sizes of the missing blobs):\n\n[mikeh@mac5 src]$ for i in `cat /tmp/lin4`; do git cat-file -s $i ; done\n1494474\n1667992\n1496198\n1643008\n1666070\n1724686\n1494201\n1643297\n1665137\n1640569\n1726140\n[mikeh@mac5 src]$ for i in `cat /tmp/toulouse`; do git cat-file -s $i ; done\n3055178\n2858902\n3060252\n2887177\n3038051\n3033691\n3008232\n2981567\n3000575\n3081501\n2995707\n3070232\n3076036\n3059223\n3075351\n3070343\n3054573\n3033120\n3028284\n3078443\n2896078\n2895094\n2973070\n2859356\n\nI was hoping that these would be on some type of boundary (and hence\npowers of two), but that doesn't seem to be the case.\n\nMike.\n"},{"id":"154700","messageId":"AANLkTimr_11_F7WJK3Uv8N=ORqDYfrLTGyHB7c-npNaU@mail.gmail.com","threadId":"25539","inReplyTo":"AANLkTinfDh5_-u6nJc=MMs_vPQGuBPhOT-TqgVOvAMPo@mail.gmail.com","subject":"Re: fsck errors on newly cloned, newly imported git repository","fromName":"Mike Herrick","fromEmail":"mike.herrick@gmail.com","sentAt":"2010-10-28T23:40:15Z","receivedAt":"2010-10-28T23:40:15Z","isPatch":false,"sender":{"key":"mike.herrick@gmail.com","avatar":null},"body":"Following up to my own post, after some serious debugging, it turned\nout that the libcrypto library was to blame.\n\nI found that after the clone, a git index-pack on the pack-file would\ngenerate a different index each time (from the same pack-file). I\nnarrowed it down to this routine returning an incorrect sha1 on \"bad\"\nmachines:\n\nstatic void write_sha1_file_prepare(const void *buf, unsigned long len,\n const char *type, unsigned char *sha1,\n char *hdr, int *hdrlen)\n{\n git_SHA_CTX c;\n\n /* Generate the header */\n *hdrlen = sprintf(hdr, \"%s %lu\", type, len)+1;\n\n /* Sha1.. */\n git_SHA1_Init(&c);\n git_SHA1_Update(&c, hdr, *hdrlen);\n git_SHA1_Update(&c, buf, len);\n git_SHA1_Final(sha1, &c);\n}\n\nUpgrading the libcrypto solved the problem on both machines.\n\nFYI, the offending versions were openssl-0.9.7a-26 and\nopenssl-0.9.7a-35 (these were Fedora Core 2 vintage). Hopefully\nno-one has any systems this old lying around any longer.\n\nMike.\n"}]}