{"thread":{"id":"32128","subject":"Auto-repo-repair","startedAt":"2012-11-16T17:51:45Z","lastAt":"2012-11-22T23:16:34Z","messageCount":9,"participants":["Enrico Weigelt","Jeff King","Drew Northup"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"203353","messageId":"dbae3a06-c14b-4c06-9863-ae4771968fe1@zcs","threadId":"32128","inReplyTo":"0c0e34a4-16ab-40a0-9293-af94e34e4290@zcs","subject":"Auto-repo-repair","fromName":"Enrico Weigelt","fromEmail":"enrico.weigelt@vnc.biz","sentAt":"2012-11-16T17:51:45Z","receivedAt":"2012-11-16T17:51:45Z","isPatch":false,"sender":{"key":"enrico.weigelt@vnc.biz","avatar":"https://gravatar.com/avatar/15479654ea5be5109582e0713a8ab4bb258846099c516dd320dc686ef70a8a3e?d=mp&s=160"},"body":"Hi folks,\n\nsuppose the following scenario:\n\nI've broken some repo (missing objects), eg by messing something up\nw/ alternates, broken filesystem, or whatever. And I've got a bunch\nof remotes which (together) contain all of the lost objects.\n\nNow I'd like to run some $magic_command which automatically fetches\nall the missing objects and so repair my local repo.\n\nIs this already possible right now ?\n\n\nthx\n-- \nMit freundlichen Grüßen / Kind regards \n\nEnrico Weigelt \nVNC - Virtual Network Consult GmbH \nHead Of Development \n\nPariser Platz 4a, D-10117 Berlin\nTel.: +49 (30) 3464615-20\nFax: +49 (30) 3464615-59\n\nenrico.weigelt@vnc.biz; www.vnc.de \n"},{"id":"203358","messageId":"20121116190004.GA2310@sigill.intra.peff.net","threadId":"32128","inReplyTo":"dbae3a06-c14b-4c06-9863-ae4771968fe1@zcs","subject":"Re: Auto-repo-repair","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2012-11-16T19:00:04Z","receivedAt":"2012-11-16T19:00:04Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Nov 16, 2012 at 06:51:45PM +0100, Enrico Weigelt wrote:\n\n> I've broken some repo (missing objects), eg by messing something up\n> w/ alternates, broken filesystem, or whatever. And I've got a bunch\n> of remotes which (together) contain all of the lost objects.\n> \n> Now I'd like to run some $magic_command which automatically fetches\n> all the missing objects and so repair my local repo.\n> \n> Is this already possible right now ?\n\nYou can't reliably just grab the broken objects, because most transports\ndon't support grabbing arbitrary objects (you can do it if you have\nshell access to a known-good repository, but it's not automated).\n\nThe simplest thing is usually to re-clone the known-good remotes, then\ncopy the resulting packfiles into your original repository. You'll have\nduplicated objects until your next \"gc\", but the resulting repack should\nskip any corrupted objects and use the known-good ones.\n\n-Peff\n"},{"id":"203398","messageId":"075f74ff-cafb-4021-ba4d-2474b6fb1853@zcs","threadId":"32128","inReplyTo":"20121116190004.GA2310@sigill.intra.peff.net","subject":"Re: Auto-repo-repair","fromName":"Enrico Weigelt","fromEmail":"enrico.weigelt@vnc.biz","sentAt":"2012-11-17T09:21:49Z","receivedAt":"2012-11-17T09:21:49Z","isPatch":false,"sender":{"key":"enrico.weigelt@vnc.biz","avatar":"https://gravatar.com/avatar/15479654ea5be5109582e0713a8ab4bb258846099c516dd320dc686ef70a8a3e?d=mp&s=160"},"body":"Hi,\n\n> You can't reliably just grab the broken objects, because most\n> transports\n> don't support grabbing arbitrary objects (you can do it if you have\n> shell access to a known-good repository, but it's not automated).\n\ncan we introduce a new or extend existing transports to support that ?\n\n\ncu\n-- \nMit freundlichen Grüßen / Kind regards \n\nEnrico Weigelt \nVNC - Virtual Network Consult GmbH \nHead Of Development \n\nPariser Platz 4a, D-10117 Berlin\nTel.: +49 (30) 3464615-20\nFax: +49 (30) 3464615-59\n\nenrico.weigelt@vnc.biz; www.vnc.de \n"},{"id":"203490","messageId":"CAM9Z-nn1S9JvfcymotOvSis4CoYco0Htn4uoETZn2kCto0z9zw@mail.gmail.com","threadId":"32128","inReplyTo":"075f74ff-cafb-4021-ba4d-2474b6fb1853@zcs","subject":"Re: Auto-repo-repair","fromName":"Drew Northup","fromEmail":"n1xim.email@gmail.com","sentAt":"2012-11-18T16:37:57Z","receivedAt":"2012-11-18T16:37:57Z","isPatch":false,"sender":{"key":"n1xim.email@gmail.com","avatar":null},"body":"On Sat, Nov 17, 2012 at 4:21 AM, Enrico Weigelt wrote:\n> Hi,\n>\n>> You can't reliably just grab the broken objects, because most\n>> transports\n>> don't support grabbing arbitrary objects (you can do it if you have\n>> shell access to a known-good repository, but it's not automated).\n>\n> can we introduce a new or extend existing transports to support that ?\n\nHow would the broken repository be sure of what it is missing to\nrequest it from the other side?\n\n-- \n-Drew Northup\n--------------------------------------------------------------\n\"As opposed to vegetable or mineral error?\"\n-John Pescatore, SANS NewsBites Vol. 12 Num. 59\n"},{"id":"203493","messageId":"66719060-db68-45b4-8453-2fd996f27657@zcs","threadId":"32128","inReplyTo":"CAM9Z-nn1S9JvfcymotOvSis4CoYco0Htn4uoETZn2kCto0z9zw@mail.gmail.com","subject":"Re: Auto-repo-repair","fromName":"Enrico Weigelt","fromEmail":"enrico.weigelt@vnc.biz","sentAt":"2012-11-18T16:55:56Z","receivedAt":"2012-11-18T16:55:56Z","isPatch":false,"sender":{"key":"enrico.weigelt@vnc.biz","avatar":"https://gravatar.com/avatar/15479654ea5be5109582e0713a8ab4bb258846099c516dd320dc686ef70a8a3e?d=mp&s=160"},"body":"\n> How would the broken repository be sure of what it is missing to\n> request it from the other side?\n\nfsck will find missing objects.\n\n\n> \n> --\n> -Drew Northup\n> --------------------------------------------------------------\n> \"As opposed to vegetable or mineral error?\"\n> -John Pescatore, SANS NewsBites Vol. 12 Num. 59\n> \n\n-- \nMit freundlichen Grüßen / Kind regards \n\nEnrico Weigelt \nVNC - Virtual Network Consult GmbH \nHead Of Development \n\nPariser Platz 4a, D-10117 Berlin\nTel.: +49 (30) 3464615-20\nFax: +49 (30) 3464615-59\n\nenrico.weigelt@vnc.biz; www.vnc.de \n"},{"id":"203499","messageId":"CAM9Z-nmu2MiE9vF9T6Aw8vFTR8mTkuR3akHgZX6+=n3uA4fmpA@mail.gmail.com","threadId":"32128","inReplyTo":"66719060-db68-45b4-8453-2fd996f27657@zcs","subject":"Re: Auto-repo-repair","fromName":"Drew Northup","fromEmail":"n1xim.email@gmail.com","sentAt":"2012-11-19T03:30:26Z","receivedAt":"2012-11-19T03:30:26Z","isPatch":false,"sender":{"key":"n1xim.email@gmail.com","avatar":null},"body":"On Sun, Nov 18, 2012 at 11:55 AM, Enrico Weigelt wrote:\n>\n>> How would the broken repository be sure of what it is missing to\n>> request it from the other side?\n>\n> fsck will find missing objects.\n\nAnd what about the objects referred to by objects that are missing?\nJeff's solution doesn't suffer from this recursivity problem.\n\n-- \n-Drew Northup\n--------------------------------------------------------------\n\"As opposed to vegetable or mineral error?\"\n-John Pescatore, SANS NewsBites Vol. 12 Num. 59\n"},{"id":"203520","messageId":"fd162e10-46a8-433c-80b2-c1c4185c2032@zcs","threadId":"32128","inReplyTo":"CAM9Z-nmu2MiE9vF9T6Aw8vFTR8mTkuR3akHgZX6+=n3uA4fmpA@mail.gmail.com","subject":"Re: Auto-repo-repair","fromName":"Enrico Weigelt","fromEmail":"enrico.weigelt@vnc.biz","sentAt":"2012-11-19T22:35:34Z","receivedAt":"2012-11-19T22:35:34Z","isPatch":false,"sender":{"key":"enrico.weigelt@vnc.biz","avatar":"https://gravatar.com/avatar/15479654ea5be5109582e0713a8ab4bb258846099c516dd320dc686ef70a8a3e?d=mp&s=160"},"body":"\n> >> How would the broken repository be sure of what it is missing to\n> >> request it from the other side?\n> >\n> > fsck will find missing objects.\n> \n> And what about the objects referred to by objects that are missing?\n\nWill be fetched after multiple iterations.\nWe could even introduce some 'fsck --autorepair' mode, which triggers\nit to fetch any missing object from its remotes.\n\nMaybe even introduce a concept of peer object stores, which (a bit like\nalternates) are asked for objects that arent locally availabe - that\ncould be even a plain venti store.\n\n\ncu\n-- \nMit freundlichen Grüßen / Kind regards \n\nEnrico Weigelt \nVNC - Virtual Network Consult GmbH \nHead Of Development \n\nPariser Platz 4a, D-10117 Berlin\nTel.: +49 (30) 3464615-20\nFax: +49 (30) 3464615-59\n\nenrico.weigelt@vnc.biz; www.vnc.de \n"},{"id":"203552","messageId":"CAM9Z-n=tAcpQHTU7WHhzZkoVL_ar9vcH8G1tKd-026+djAiJ4A@mail.gmail.com","threadId":"32128","inReplyTo":"fd162e10-46a8-433c-80b2-c1c4185c2032@zcs","subject":"Re: Auto-repo-repair","fromName":"Drew Northup","fromEmail":"n1xim.email@gmail.com","sentAt":"2012-11-20T11:56:17Z","receivedAt":"2012-11-20T11:56:17Z","isPatch":false,"sender":{"key":"n1xim.email@gmail.com","avatar":null},"body":"On Mon, Nov 19, 2012 at 5:35 PM, Enrico Weigelt wrote:\n>\n>> >> How would the broken repository be sure of what it is missing to\n>> >> request it from the other side?\n>> >\n>> > fsck will find missing objects.\n>>\n>> And what about the objects referred to by objects that are missing?\n>\n> Will be fetched after multiple iterations.\n> We could even introduce some 'fsck --autorepair' mode, which triggers\n> it to fetch any missing object from its remotes.\n>\n> Maybe even introduce a concept of peer object stores, which (a bit like\n> alternates) are asked for objects that arent locally availabe - that\n> could be even a plain venti store.\n\nI still think that it would make the most sense to do the following\n(if you insist on some sort of automated repair):\n(1) Fetch a \"good\" clone (or clones) into a temporary directory;\n(2) Cannibalize the objects from it (them);\n(3) Re-run git fsck and check for still-missing / unreachable items;\n(4) IF THE RESULT OF (3) IS ACCEPTABLE, run git gc to clean up the\nmess, discard / \"merge\" duplicate objects, and fix up the packfiles.\n\nIt is step (4) that requires the most user interaction. I could see\nbuilding up a shell script that does all but (4) nearly automatically.\nNone of this requires modifying Git itself.\n\n-- \n-Drew Northup\n--------------------------------------------------------------\n\"As opposed to vegetable or mineral error?\"\n-John Pescatore, SANS NewsBites Vol. 12 Num. 59\n"},{"id":"203675","messageId":"cd8ee2cf-2d4a-4d0c-931f-61de4c7f6348@zcs","threadId":"32128","inReplyTo":"CAM9Z-n=tAcpQHTU7WHhzZkoVL_ar9vcH8G1tKd-026+djAiJ4A@mail.gmail.com","subject":"Re: Auto-repo-repair","fromName":"Enrico Weigelt","fromEmail":"enrico.weigelt@vnc.biz","sentAt":"2012-11-22T23:16:34Z","receivedAt":"2012-11-22T23:16:34Z","isPatch":false,"sender":{"key":"enrico.weigelt@vnc.biz","avatar":"https://gravatar.com/avatar/15479654ea5be5109582e0713a8ab4bb258846099c516dd320dc686ef70a8a3e?d=mp&s=160"},"body":"Hi,\n\n> I still think that it would make the most sense to do the following\n> (if you insist on some sort of automated repair):\n> (1) Fetch a \"good\" clone (or clones) into a temporary directory;\n> (2) Cannibalize the objects from it (them);\n> (3) Re-run git fsck and check for still-missing / unreachable items;\n> (4) IF THE RESULT OF (3) IS ACCEPTABLE, run git gc to clean up the\n> mess, discard / \"merge\" duplicate objects, and fix up the packfiles.\n> \n> It is step (4) that requires the most user interaction. I could see\n> building up a shell script that does all but (4) nearly\n> automatically.\n> None of this requires modifying Git itself.\n\nWell, I'd like to have some really automatic mode, which does\neverything ondemand.\n\nOnce we've got this not just for repair, but also to support\nquick partial clones that fetch more objects when required.\n\nIn fact, finally, I'd like to have some storage cloud where\ndata automatically gets replicated to nodes which need the data,\nnot just for VCS, but other purposes (backup, filestore, etc) too.\nBut before inventing someting completely new (reinventing much\nof the wheel), I'd like to investigate whether git can be\nextended into this direction step by step.\n\n\ncu\n-- \nMit freundlichen Grüßen / Kind regards \n\nEnrico Weigelt \nVNC - Virtual Network Consult GmbH \nHead Of Development \n\nPariser Platz 4a, D-10117 Berlin\nTel.: +49 (30) 3464615-20\nFax: +49 (30) 3464615-59\n\nenrico.weigelt@vnc.biz; www.vnc.de \n"}]}