{"thread":{"id":"32592","subject":"missing objects -- prevention","startedAt":"2013-01-11T11:10:38Z","lastAt":"2013-01-13T12:30:36Z","messageCount":7,"participants":["Sitaram Chamarty","Jeff King"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"206506","messageId":"CAMK1S_jpofLRO02XTYryOP98g7rnrJXs7Mh2zvi=SoVUAs0dUw@mail.gmail.com","threadId":"32592","inReplyTo":null,"subject":"missing objects -- prevention","fromName":"Sitaram Chamarty","fromEmail":"sitaramc@gmail.com","sentAt":"2013-01-11T11:10:38Z","receivedAt":"2013-01-11T11:10:38Z","isPatch":false,"sender":{"key":"sitaramc@gmail.com","avatar":"https://avatars.githubusercontent.com/u/43316?v=4"},"body":"Hi all,\n\nI find a lot of info on how to recover from and/or repair a repo that\nhas missing (or corrupted) objects.\n\nWhat I need is info on common reasons (other than disk errors -- we've\nchecked for those) for such errors to occur, any preventive measures\nwe can take, and so on.  For example, can *any* type of network error\nor race condition cause this?  (Say, can one push writes an object,\nthen fails an update check, and a later push succeeds and races\nagainst a gc that removes the unreachable object?)  Or... the repo is\npretty large -- about 6-7 GB, so could size cause a race that would\nnot show up on a smaller repo?\n\nAnything else I can watch out for or caution the team about?\n\nThe symptom is usually a disk space crunch caused by tmp_pack_* files\nleft around by auto-gc.  Presumably the missing objects failed the gc\nand so it left the files around, and they eventually accumulate enough\nto cause disk full errors.  (If a gc ever succeeded, I suspect these\nfiles would go away, but that requires manual intervention).\n\nI would appreciate any thoughts...  Thanks!\n\n-- \nSitaram\n"},{"id":"206540","messageId":"20130111164202.GB5219@sigill.intra.peff.net","threadId":"32592","inReplyTo":"CAMK1S_jpofLRO02XTYryOP98g7rnrJXs7Mh2zvi=SoVUAs0dUw@mail.gmail.com","subject":"Re: missing objects -- prevention","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2013-01-11T16:42:03Z","receivedAt":"2013-01-11T16:42:03Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Jan 11, 2013 at 04:40:38PM +0530, Sitaram Chamarty wrote:\n\n> I find a lot of info on how to recover from and/or repair a repo that\n> has missing (or corrupted) objects.\n> \n> What I need is info on common reasons (other than disk errors -- we've\n> checked for those) for such errors to occur, any preventive measures\n> we can take, and so on.\n\nI don't think any race can cause corruption of the object or packfiles\nbecause of the way they are written. At GitHub, every case of file-level\ncorruption we've seen has been a filesystem issue.\n\nSo I think the main thing systemic/race issue to worry about is missing\nobjects. And since git only deletes objects during a prune (assuming you\nare using git-gc or \"repack -A\" so that repack cannot drop objects), I\nthink prune is the only thing to watch out for.\n\nThe --expire time saves us from the obvious races where you write object\nX but have not yet referenced it, and a simultaneous prune wants to\ndelete it. However, it's possible that you have an old object that is\nunreferenced, but would become referenced as a result of an in-progress\noperation. For example, commit X is unreferenced and ready to be pruned,\nyou build commit Y on top of it, but before you write the ref, git-prune\nremoves X.\n\nThe server-side version of that would happen via receive-pack, and is\neven more unlikely, because X would have to be referenced initially for\nus to advertise it. So it's something like:\n\n  1. The repo has a ref R pointing at commit X.\n\n  2. A user starts a push to another ref, Q, of commit Y that builds on\n     X. Git advertises ref R, so the sender knows they do not need to\n     send X, but only Y. The user then proceeds to send the packfile\n     (which might take a very long time).\n\n  3. Meanwhile, another user deletes ref R. X becomes unreferenced.\n\n  4. After step 3 but before step 2 has finished, somebody runs prune\n     (this might sound unlikely, but if you kick off a \"gc\" job after\n     each push, or after N pushes, it's not so unlikely).  It sees that\n     X is unreferenced, and it may very well be older than the --expire\n     setting. Prune deletes X.\n\n  5. The packfile in (2) arrives, and receive-pack attempts to update\n     the refs.\n\nSo it's even a bit more unlikely than the local case, because\nreceive-pack would not otherwise build on dangling objects. You have\nto race steps (2) and (3) just to create the situation.\n\nThen we have an extra protection in the form of\ncheck_everything_connected, which receive-pack runs before writing the\nrefs into place. So if step 4 happens while the packfile is being sent\n(which is the most likely case, since it is the longest stretch of\nreceive-pack's time), we would still catch it there and reject the push\n(annoying to the user, but the repo remains consistent).\n\nHowever, that's not foolproof. We might hit step 4 after we've checked\nthat everything is connected but right before we write the ref. In which\ncase we drop X, which has just become referenced, and we have a missing\nobject.\n\nSo I think it's possible. But I have never actually seen it in practice,\nand come up with this scenario only by brainstorming \"what could go\nwrong\" scenarios.\n\nThis could be mitigated if there was a \"proposed refs\" storage.\nReceive-pack would write a note saying \"consider Y for pruning purposes,\nbut it's not really referenced yet\", check connectivity for Y against\nthe current refs, and then eventually write Y to its real ref (or reject\nit if there are problems). Prune would either run before the \"proposed\"\nnote is written, which would mean it deletes X, but the connectivity\ncheck fails. Or it would run after, in which case it would leave X\nalone.\n\n> For example, can *any* type of network error or race condition cause\n> this?  (Say, can one push writes an object, then fails an update\n> check, and a later push succeeds and races against a gc that removes\n> the unreachable object?)  Or... the repo is pretty large -- about 6-7\n> GB, so could size cause a race that would not show up on a smaller\n> repo?\n\nThe above is the only open issue I know about. I don't think it is\ndependent on repo size, but the window is widened for a really large\npush, because rev-list takes longer to run. It does not widen if you\nhave receive.fsckobjects set, because that happens before we do the\nconnectivity check (and the connectivity check is run in a sub-process,\nso the race timer starts when we exec rev-list, which may open and mmap\npackfiles or otherwise cache the presence of X in memory).\n\n> Anything else I can watch out for or caution the team about?\n\nThat's the only open issue I know about for missing objects.\n\nThere is a race with simultaneously deleting and packing refs. It\ndoesn't cause object db corruption, but it will cause refs to \"rewind\"\nback to their packed versions. I have seen that one in practice (though\nrelatively rare). I fixed it in b3f1280, which is not yet in any\nreleased version.\n\n> The symptom is usually a disk space crunch caused by tmp_pack_* files\n> left around by auto-gc.  Presumably the missing objects failed the gc\n> and so it left the files around, and they eventually accumulate enough\n> to cause disk full errors.  (If a gc ever succeeded, I suspect these\n> files would go away, but that requires manual intervention).\n\nGc shouldn't be leaving around tmp_pack_* unless it is actually dying\nduring the pack-objects phase. In my experience, stale tmp_pack_*\nobjects are more likely a sign of a failed push (e.g., the client hangs\nup in the middle, or fsck rejects the pack). We have historically left\nthem in place to facilitate analysis of the failure.\n\nAt GitHub, we've taken to just cleaning them up aggressively (I think\nafter an hour), though I am tempted to put in an optional signal/atexit\nhandler to clean them up when index-pack fails. The forensics are\noccasionally interesting (e.g., finding weird fsck problems), but large\npushes can waste a lot of disk space in the interim.\n\n-Peff\n"},{"id":"206569","messageId":"CAMK1S_jN7=Antz-5D7yf0KV8m-YEy93tZP_zziTXPDzbdyjUrw@mail.gmail.com","threadId":"32592","inReplyTo":"20130111164202.GB5219@sigill.intra.peff.net","subject":"Re: missing objects -- prevention","fromName":"Sitaram Chamarty","fromEmail":"sitaramc@gmail.com","sentAt":"2013-01-12T01:09:52Z","receivedAt":"2013-01-12T01:09:52Z","isPatch":false,"sender":{"key":"sitaramc@gmail.com","avatar":"https://avatars.githubusercontent.com/u/43316?v=4"},"body":"Thanks for the very detailed answer.\n\nOn Fri, Jan 11, 2013 at 10:12 PM, Jeff King <peff@peff.net> wrote:\n> On Fri, Jan 11, 2013 at 04:40:38PM +0530, Sitaram Chamarty wrote:\n>\n>> I find a lot of info on how to recover from and/or repair a repo that\n>> has missing (or corrupted) objects.\n>>\n>> What I need is info on common reasons (other than disk errors -- we've\n>> checked for those) for such errors to occur, any preventive measures\n>> we can take, and so on.\n>\n> I don't think any race can cause corruption of the object or packfiles\n> because of the way they are written. At GitHub, every case of file-level\n> corruption we've seen has been a filesystem issue.\n>\n> So I think the main thing systemic/race issue to worry about is missing\n> objects. And since git only deletes objects during a prune (assuming you\n> are using git-gc or \"repack -A\" so that repack cannot drop objects), I\n> think prune is the only thing to watch out for.\n\nNo one runs anything manually under normal conditions.  If there's any\ngc happening, it's gc --auto.\n\n> The --expire time saves us from the obvious races where you write object\n> X but have not yet referenced it, and a simultaneous prune wants to\n> delete it. However, it's possible that you have an old object that is\n> unreferenced, but would become referenced as a result of an in-progress\n> operation. For example, commit X is unreferenced and ready to be pruned,\n> you build commit Y on top of it, but before you write the ref, git-prune\n> removes X.\n>\n> The server-side version of that would happen via receive-pack, and is\n> even more unlikely, because X would have to be referenced initially for\n> us to advertise it. So it's something like:\n>\n>   1. The repo has a ref R pointing at commit X.\n>\n>   2. A user starts a push to another ref, Q, of commit Y that builds on\n>      X. Git advertises ref R, so the sender knows they do not need to\n>      send X, but only Y. The user then proceeds to send the packfile\n>      (which might take a very long time).\n>\n>   3. Meanwhile, another user deletes ref R. X becomes unreferenced.\n\nThe gitolite logs show that no deletion of refs has happened.\n\n>   4. After step 3 but before step 2 has finished, somebody runs prune\n>      (this might sound unlikely, but if you kick off a \"gc\" job after\n>      each push, or after N pushes, it's not so unlikely).  It sees that\n>      X is unreferenced, and it may very well be older than the --expire\n>      setting. Prune deletes X.\n>\n>   5. The packfile in (2) arrives, and receive-pack attempts to update\n>      the refs.\n>\n> So it's even a bit more unlikely than the local case, because\n> receive-pack would not otherwise build on dangling objects. You have\n> to race steps (2) and (3) just to create the situation.\n>\n> Then we have an extra protection in the form of\n> check_everything_connected, which receive-pack runs before writing the\n> refs into place. So if step 4 happens while the packfile is being sent\n> (which is the most likely case, since it is the longest stretch of\n> receive-pack's time), we would still catch it there and reject the push\n> (annoying to the user, but the repo remains consistent).\n>\n> However, that's not foolproof. We might hit step 4 after we've checked\n> that everything is connected but right before we write the ref. In which\n> case we drop X, which has just become referenced, and we have a missing\n> object.\n>\n> So I think it's possible. But I have never actually seen it in practice,\n> and come up with this scenario only by brainstorming \"what could go\n> wrong\" scenarios.\n>\n> This could be mitigated if there was a \"proposed refs\" storage.\n> Receive-pack would write a note saying \"consider Y for pruning purposes,\n> but it's not really referenced yet\", check connectivity for Y against\n> the current refs, and then eventually write Y to its real ref (or reject\n> it if there are problems). Prune would either run before the \"proposed\"\n> note is written, which would mean it deletes X, but the connectivity\n> check fails. Or it would run after, in which case it would leave X\n> alone.\n>\n>> For example, can *any* type of network error or race condition cause\n>> this?  (Say, can one push writes an object, then fails an update\n>> check, and a later push succeeds and races against a gc that removes\n>> the unreachable object?)  Or... the repo is pretty large -- about 6-7\n>> GB, so could size cause a race that would not show up on a smaller\n>> repo?\n>\n> The above is the only open issue I know about. I don't think it is\n> dependent on repo size, but the window is widened for a really large\n> push, because rev-list takes longer to run. It does not widen if you\n> have receive.fsckobjects set, because that happens before we do the\n> connectivity check (and the connectivity check is run in a sub-process,\n> so the race timer starts when we exec rev-list, which may open and mmap\n> packfiles or otherwise cache the presence of X in memory).\n>\n>> Anything else I can watch out for or caution the team about?\n>\n> That's the only open issue I know about for missing objects.\n>\n> There is a race with simultaneously deleting and packing refs. It\n> doesn't cause object db corruption, but it will cause refs to \"rewind\"\n> back to their packed versions. I have seen that one in practice (though\n> relatively rare). I fixed it in b3f1280, which is not yet in any\n> released version.\n\nThis is for the packed-refs file right?  And it could result in a ref\ngetting deleted right?\n\nI said above that the gitolite logs say no ref was deleted.  What if\nthe ref \"deletion\" happened because of this race, making the rest of\nyour 4-step scenario above possible?\n\n>> The symptom is usually a disk space crunch caused by tmp_pack_* files\n>> left around by auto-gc.  Presumably the missing objects failed the gc\n>> and so it left the files around, and they eventually accumulate enough\n>> to cause disk full errors.  (If a gc ever succeeded, I suspect these\n>> files would go away, but that requires manual intervention).\n>\n> Gc shouldn't be leaving around tmp_pack_* unless it is actually dying\n> during the pack-objects phase. In my experience, stale tmp_pack_*\n> objects are more likely a sign of a failed push (e.g., the client hangs\n\nOh.  I did not know that!  That explains why I sometimes saw that even\nwhen there were less than 6700 loose objects (i.e., auto-gc should not\nhave kicked in at all).\n\n> up in the middle, or fsck rejects the pack). We have historically left\n\nfsck... you mean if I had 'receive.fsckObjects' true, right?  I don't.\n Should I?  Would it help this overall situation?  As I understand it,\nthats only about the internals of each object to check corruption, and\ncannot detect a *missing* object on the local object store.\n\n> them in place to facilitate analysis of the failure.\n>\n> At GitHub, we've taken to just cleaning them up aggressively (I think\n> after an hour), though I am tempted to put in an optional signal/atexit\n\nOK; I'll do the same then.  I suppose a cron job is the best way; I\ndidn't find any config for expiring these files.\n\nThanks again for your help.  I'm going to treat it (for now) as a\ndisk/fs error after hearing from you about the other possibility I\nmentioned above, although I find it hard to believe one repo can be\nhit buy *two* races occurring together!\n\n> handler to clean them up when index-pack fails. The forensics are\n> occasionally interesting (e.g., finding weird fsck problems), but large\n> pushes can waste a lot of disk space in the interim.\n>\n> -Peff\n\n\n\n-- \nSitaram\n"},{"id":"206595","messageId":"20130112131358.GB21875@sigill.intra.peff.net","threadId":"32592","inReplyTo":"CAMK1S_jN7=Antz-5D7yf0KV8m-YEy93tZP_zziTXPDzbdyjUrw@mail.gmail.com","subject":"Re: missing objects -- prevention","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2013-01-12T13:13:58Z","receivedAt":"2013-01-12T13:13:58Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sat, Jan 12, 2013 at 06:39:52AM +0530, Sitaram Chamarty wrote:\n\n> >   1. The repo has a ref R pointing at commit X.\n> >\n> >   2. A user starts a push to another ref, Q, of commit Y that builds on\n> >      X. Git advertises ref R, so the sender knows they do not need to\n> >      send X, but only Y. The user then proceeds to send the packfile\n> >      (which might take a very long time).\n> >\n> >   3. Meanwhile, another user deletes ref R. X becomes unreferenced.\n> \n> The gitolite logs show that no deletion of refs has happened.\n\nTo be pedantic, step 3 could also be rewinding R to a commit before X.\nAnything that causes X to become unreferenced.\n\n> > There is a race with simultaneously deleting and packing refs. It\n> > doesn't cause object db corruption, but it will cause refs to \"rewind\"\n> > back to their packed versions. I have seen that one in practice (though\n> > relatively rare). I fixed it in b3f1280, which is not yet in any\n> > released version.\n> \n> This is for the packed-refs file right?  And it could result in a ref\n> getting deleted right?\n\nYes, if the ref was not previously packed, it could result in the ref\nbeing deleted entirely.\n\n> I said above that the gitolite logs say no ref was deleted.  What if\n> the ref \"deletion\" happened because of this race, making the rest of\n> your 4-step scenario above possible?\n\nIt's possible. I do want to highlight how unlikely it is, though.\n\n> > up in the middle, or fsck rejects the pack). We have historically left\n> \n> fsck... you mean if I had 'receive.fsckObjects' true, right?  I don't.\n>  Should I?  Would it help this overall situation?  As I understand it,\n> thats only about the internals of each object to check corruption, and\n> cannot detect a *missing* object on the local object store.\n\nRight, I meant if you have receive.fsckObjects on. It won't help this\nsituation at all, as we already do a connectivity check separate from\nthe fsck. But I do recommend it in general, just because it helps catch\nbad objects before they gets disseminated to a wider audience (at which\npoint it is often infeasible to rewind history). And it has found git\nbugs (e.g., null sha1s in tree entries).\n\n> > At GitHub, we've taken to just cleaning them up aggressively (I think\n> > after an hour), though I am tempted to put in an optional signal/atexit\n> \n> OK; I'll do the same then.  I suppose a cron job is the best way; I\n> didn't find any config for expiring these files.\n\nIf you run \"git prune --expire=1.hour.ago\", it should prune stale\ntmp_pack_* files more than an hour old. But you may not be comfortable\nwith such a short expiration for the objects themselves. :)\n\n> Thanks again for your help.  I'm going to treat it (for now) as a\n> disk/fs error after hearing from you about the other possibility I\n> mentioned above, although I find it hard to believe one repo can be\n> hit buy *two* races occurring together!\n\nYeah, the race seems pretty unlikely (though it could be just the one\nrace with a rewind). As I said, I haven't actually ever seen it in\npractice. In my experience, though, disk/fs issues do not manifest as\njust missing objects, but as corrupted packfiles (e.g., the packfile\ndirectory entry ends up pointing to the wrong inode, which is easy to\nsee because the inode's content is actually a reflog). And then of\ncourse with the packfile unreadable, you have missing objects. But YMMV,\ndepending on the fs and what's happened to the machine to cause the fs\nproblem.\n\n-Peff\n"},{"id":"206631","messageId":"CAMK1S_iKARYqi_Dv90og0No7NN=WxFg+ixmRvnkvfdrcOi1r=Q@mail.gmail.com","threadId":"32592","inReplyTo":"20130112131358.GB21875@sigill.intra.peff.net","subject":"Re: missing objects -- prevention","fromName":"Sitaram Chamarty","fromEmail":"sitaramc@gmail.com","sentAt":"2013-01-13T00:56:53Z","receivedAt":"2013-01-13T00:56:53Z","isPatch":false,"sender":{"key":"sitaramc@gmail.com","avatar":"https://avatars.githubusercontent.com/u/43316?v=4"},"body":"On Sat, Jan 12, 2013 at 6:43 PM, Jeff King <peff@peff.net> wrote:\n> On Sat, Jan 12, 2013 at 06:39:52AM +0530, Sitaram Chamarty wrote:\n>\n>> >   1. The repo has a ref R pointing at commit X.\n>> >\n>> >   2. A user starts a push to another ref, Q, of commit Y that builds on\n>> >      X. Git advertises ref R, so the sender knows they do not need to\n>> >      send X, but only Y. The user then proceeds to send the packfile\n>> >      (which might take a very long time).\n>> >\n>> >   3. Meanwhile, another user deletes ref R. X becomes unreferenced.\n>>\n>> The gitolite logs show that no deletion of refs has happened.\n>\n> To be pedantic, step 3 could also be rewinding R to a commit before X.\n> Anything that causes X to become unreferenced.\n\nRight, but there were no rewinds also; I should have mentioned that.\n(Gitolite log files mark rewinds and deletes specially, so they're\neasy to search.  There were two attempted rewinds but they failed the\ngitolite update hook so -- while the new objects would have landed in\nthe object store -- the old ones were not dereferenced).\n\n>> > There is a race with simultaneously deleting and packing refs. It\n>> > doesn't cause object db corruption, but it will cause refs to \"rewind\"\n>> > back to their packed versions. I have seen that one in practice (though\n>> > relatively rare). I fixed it in b3f1280, which is not yet in any\n>> > released version.\n>>\n>> This is for the packed-refs file right?  And it could result in a ref\n>> getting deleted right?\n>\n> Yes, if the ref was not previously packed, it could result in the ref\n> being deleted entirely.\n>\n>> I said above that the gitolite logs say no ref was deleted.  What if\n>> the ref \"deletion\" happened because of this race, making the rest of\n>> your 4-step scenario above possible?\n>\n> It's possible. I do want to highlight how unlikely it is, though.\n\nAgreed.\n\n>> > up in the middle, or fsck rejects the pack). We have historically left\n>>\n>> fsck... you mean if I had 'receive.fsckObjects' true, right?  I don't.\n>>  Should I?  Would it help this overall situation?  As I understand it,\n>> thats only about the internals of each object to check corruption, and\n>> cannot detect a *missing* object on the local object store.\n>\n> Right, I meant if you have receive.fsckObjects on. It won't help this\n> situation at all, as we already do a connectivity check separate from\n> the fsck. But I do recommend it in general, just because it helps catch\n> bad objects before they gets disseminated to a wider audience (at which\n> point it is often infeasible to rewind history). And it has found git\n> bugs (e.g., null sha1s in tree entries).\n\nI will add this.  Any idea if there's a significant performance hit?\n\n>> > At GitHub, we've taken to just cleaning them up aggressively (I think\n>> > after an hour), though I am tempted to put in an optional signal/atexit\n>>\n>> OK; I'll do the same then.  I suppose a cron job is the best way; I\n>> didn't find any config for expiring these files.\n>\n> If you run \"git prune --expire=1.hour.ago\", it should prune stale\n> tmp_pack_* files more than an hour old. But you may not be comfortable\n> with such a short expiration for the objects themselves. :)\n>\n>> Thanks again for your help.  I'm going to treat it (for now) as a\n>> disk/fs error after hearing from you about the other possibility I\n>> mentioned above, although I find it hard to believe one repo can be\n>> hit buy *two* races occurring together!\n>\n> Yeah, the race seems pretty unlikely (though it could be just the one\n> race with a rewind). As I said, I haven't actually ever seen it in\n> practice. In my experience, though, disk/fs issues do not manifest as\n> just missing objects, but as corrupted packfiles (e.g., the packfile\n> directory entry ends up pointing to the wrong inode, which is easy to\n> see because the inode's content is actually a reflog). And then of\n> course with the packfile unreadable, you have missing objects. But YMMV,\n> depending on the fs and what's happened to the machine to cause the fs\n> problem.\n\nThat's always the hard part.  System admins (at the Unix level) insist\nthere's nothing wrong and no disk errors and so on...  that is why I\nwas interested in network errors causing problems and so on.\n\nAnyway, now that I know the tmp_pack_* files are caused mostly by\nfailed pushes than by failed auto-gc, at least I can deal with the\nimmediate problem easily!\n\nThanks once again for your patient replies!\n\nsitaram\n\n\n-- \nSitaram\n"},{"id":"206632","messageId":"CAMK1S_gMsxuP-L=+1kcM2Zmh=vzpYqdCm+Rder8jnChYhuYVOw@mail.gmail.com","threadId":"32592","inReplyTo":"CAMK1S_iKARYqi_Dv90og0No7NN=WxFg+ixmRvnkvfdrcOi1r=Q@mail.gmail.com","subject":"Re: missing objects -- prevention","fromName":"Sitaram Chamarty","fromEmail":"sitaramc@gmail.com","sentAt":"2013-01-13T00:57:56Z","receivedAt":"2013-01-13T00:57:56Z","isPatch":false,"sender":{"key":"sitaramc@gmail.com","avatar":"https://avatars.githubusercontent.com/u/43316?v=4"},"body":"Uggh...\n\nOn Sun, Jan 13, 2013 at 6:26 AM, Sitaram Chamarty <sitaramc@gmail.com> wrote:\n\n> the object store -- the old ones were not dereferenced).\n\nI meant *un* referenced of course :)\n"},{"id":"206650","messageId":"20130113123036.GA498@sigill.intra.peff.net","threadId":"32592","inReplyTo":"CAMK1S_iKARYqi_Dv90og0No7NN=WxFg+ixmRvnkvfdrcOi1r=Q@mail.gmail.com","subject":"Re: missing objects -- prevention","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2013-01-13T12:30:36Z","receivedAt":"2013-01-13T12:30:36Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sun, Jan 13, 2013 at 06:26:53AM +0530, Sitaram Chamarty wrote:\n\n> > Right, I meant if you have receive.fsckObjects on. It won't help this\n> > situation at all, as we already do a connectivity check separate from\n> > the fsck. But I do recommend it in general, just because it helps catch\n> > bad objects before they gets disseminated to a wider audience (at which\n> > point it is often infeasible to rewind history). And it has found git\n> > bugs (e.g., null sha1s in tree entries).\n> \n> I will add this.  Any idea if there's a significant performance hit?\n\nNot usually; we are already resolving all of the sent deltas as a\nprecaution, anyway. I do notice after a push to GitHub there is\nsometimes a second or two of pause from the server before the push\nstatus is shown. But I haven't narrowed it down to fsck (versus\nconnectivity check, versus our post-receive hook).\n\nSo you may want to keep an eye on the effects (and if you have numbers,\nplease share :) ).\n\n> That's always the hard part.  System admins (at the Unix level) insist\n> there's nothing wrong and no disk errors and so on...  that is why I\n> was interested in network errors causing problems and so on.\n\nYeah, I feel bad saying \"well, this repo is totally corrupted, but it\ncouldn't possibly be git's fault, because that's not what its failure\nmodes look like\". But luckily our Ops people are very understanding, and\nmost of the problems I have seen have turned out to be fs corruption\nafter all (the pack-refs things is the big exception).\n\n> Thanks once again for your patient replies!\n\nNo problem. There aren't many people dealing with large-scale\nserver-side issues, so it's something that doesn't come up much on the\nlist. I'm happy to talk about it.\n\n-Peff\n"}]}