# Auto-repo-repair 9 messages from 2012-11-16 to 2012-11-22. Participants: Enrico Weigelt, Jeff King, Drew Northup. Thread: https://gitlist.dev/t/32128 ## Enrico Weigelt, 2012-11-16 17:51 Subject: Auto-repo-repair Message-ID: URL: https://gitlist.dev/e/dbae3a06-c14b-4c06-9863-ae4771968fe1%40zcs In-Reply-To: <0c0e34a4-16ab-40a0-9293-af94e34e4290@zcs> ``` Hi folks, suppose the following scenario: I've broken some repo (missing objects), eg by messing something up w/ alternates, broken filesystem, or whatever. And I've got a bunch of remotes which (together) contain all of the lost objects. Now I'd like to run some $magic_command which automatically fetches all the missing objects and so repair my local repo. Is this already possible right now ? thx -- Mit freundlichen Grüßen / Kind regards Enrico Weigelt VNC - Virtual Network Consult GmbH Head Of Development Pariser Platz 4a, D-10117 Berlin Tel.: +49 (30) 3464615-20 Fax: +49 (30) 3464615-59 enrico.weigelt@vnc.biz; www.vnc.de ``` ## Jeff King, 2012-11-16 19:00 Subject: Re: Auto-repo-repair Message-ID: <20121116190004.GA2310@sigill.intra.peff.net> URL: https://gitlist.dev/e/20121116190004.GA2310%40sigill.intra.peff.net In-Reply-To: ``` On Fri, Nov 16, 2012 at 06:51:45PM +0100, Enrico Weigelt wrote: > I've broken some repo (missing objects), eg by messing something up > w/ alternates, broken filesystem, or whatever. And I've got a bunch > of remotes which (together) contain all of the lost objects. > > Now I'd like to run some $magic_command which automatically fetches > all the missing objects and so repair my local repo. > > Is this already possible right now ? You can't reliably just grab the broken objects, because most transports don't support grabbing arbitrary objects (you can do it if you have shell access to a known-good repository, but it's not automated). The simplest thing is usually to re-clone the known-good remotes, then copy the resulting packfiles into your original repository. You'll have duplicated objects until your next "gc", but the resulting repack should skip any corrupted objects and use the known-good ones. -Peff ``` ## Enrico Weigelt, 2012-11-17 09:21 Subject: Re: Auto-repo-repair Message-ID: <075f74ff-cafb-4021-ba4d-2474b6fb1853@zcs> URL: https://gitlist.dev/e/075f74ff-cafb-4021-ba4d-2474b6fb1853%40zcs In-Reply-To: <20121116190004.GA2310@sigill.intra.peff.net> ``` Hi, > You can't reliably just grab the broken objects, because most > transports > don't support grabbing arbitrary objects (you can do it if you have > shell access to a known-good repository, but it's not automated). can we introduce a new or extend existing transports to support that ? cu -- Mit freundlichen Grüßen / Kind regards Enrico Weigelt VNC - Virtual Network Consult GmbH Head Of Development Pariser Platz 4a, D-10117 Berlin Tel.: +49 (30) 3464615-20 Fax: +49 (30) 3464615-59 enrico.weigelt@vnc.biz; www.vnc.de ``` ## Drew Northup, 2012-11-18 16:37 Subject: Re: Auto-repo-repair Message-ID: URL: https://gitlist.dev/e/CAM9Z-nn1S9JvfcymotOvSis4CoYco0Htn4uoETZn2kCto0z9zw%40mail.gmail.com In-Reply-To: <075f74ff-cafb-4021-ba4d-2474b6fb1853@zcs> ``` On Sat, Nov 17, 2012 at 4:21 AM, Enrico Weigelt wrote: > Hi, > >> You can't reliably just grab the broken objects, because most >> transports >> don't support grabbing arbitrary objects (you can do it if you have >> shell access to a known-good repository, but it's not automated). > > can we introduce a new or extend existing transports to support that ? How would the broken repository be sure of what it is missing to request it from the other side? -- -Drew Northup -------------------------------------------------------------- "As opposed to vegetable or mineral error?" -John Pescatore, SANS NewsBites Vol. 12 Num. 59 ``` ## Enrico Weigelt, 2012-11-18 16:55 Subject: Re: Auto-repo-repair Message-ID: <66719060-db68-45b4-8453-2fd996f27657@zcs> URL: https://gitlist.dev/e/66719060-db68-45b4-8453-2fd996f27657%40zcs In-Reply-To: ``` > How would the broken repository be sure of what it is missing to > request it from the other side? fsck will find missing objects. > > -- > -Drew Northup > -------------------------------------------------------------- > "As opposed to vegetable or mineral error?" > -John Pescatore, SANS NewsBites Vol. 12 Num. 59 > -- Mit freundlichen Grüßen / Kind regards Enrico Weigelt VNC - Virtual Network Consult GmbH Head Of Development Pariser Platz 4a, D-10117 Berlin Tel.: +49 (30) 3464615-20 Fax: +49 (30) 3464615-59 enrico.weigelt@vnc.biz; www.vnc.de ``` ## Drew Northup, 2012-11-19 03:30 Subject: Re: Auto-repo-repair Message-ID: URL: https://gitlist.dev/e/CAM9Z-nmu2MiE9vF9T6Aw8vFTR8mTkuR3akHgZX6%2B%3Dn3uA4fmpA%40mail.gmail.com In-Reply-To: <66719060-db68-45b4-8453-2fd996f27657@zcs> ``` On Sun, Nov 18, 2012 at 11:55 AM, Enrico Weigelt wrote: > >> How would the broken repository be sure of what it is missing to >> request it from the other side? > > fsck will find missing objects. And what about the objects referred to by objects that are missing? Jeff's solution doesn't suffer from this recursivity problem. -- -Drew Northup -------------------------------------------------------------- "As opposed to vegetable or mineral error?" -John Pescatore, SANS NewsBites Vol. 12 Num. 59 ``` ## Enrico Weigelt, 2012-11-19 22:35 Subject: Re: Auto-repo-repair Message-ID: URL: https://gitlist.dev/e/fd162e10-46a8-433c-80b2-c1c4185c2032%40zcs In-Reply-To: ``` > >> How would the broken repository be sure of what it is missing to > >> request it from the other side? > > > > fsck will find missing objects. > > And what about the objects referred to by objects that are missing? Will be fetched after multiple iterations. We could even introduce some 'fsck --autorepair' mode, which triggers it to fetch any missing object from its remotes. Maybe even introduce a concept of peer object stores, which (a bit like alternates) are asked for objects that arent locally availabe - that could be even a plain venti store. cu -- Mit freundlichen Grüßen / Kind regards Enrico Weigelt VNC - Virtual Network Consult GmbH Head Of Development Pariser Platz 4a, D-10117 Berlin Tel.: +49 (30) 3464615-20 Fax: +49 (30) 3464615-59 enrico.weigelt@vnc.biz; www.vnc.de ``` ## Drew Northup, 2012-11-20 11:56 Subject: Re: Auto-repo-repair Message-ID: URL: https://gitlist.dev/e/CAM9Z-n%3DtAcpQHTU7WHhzZkoVL_ar9vcH8G1tKd-026%2BdjAiJ4A%40mail.gmail.com In-Reply-To: ``` On Mon, Nov 19, 2012 at 5:35 PM, Enrico Weigelt wrote: > >> >> How would the broken repository be sure of what it is missing to >> >> request it from the other side? >> > >> > fsck will find missing objects. >> >> And what about the objects referred to by objects that are missing? > > Will be fetched after multiple iterations. > We could even introduce some 'fsck --autorepair' mode, which triggers > it to fetch any missing object from its remotes. > > Maybe even introduce a concept of peer object stores, which (a bit like > alternates) are asked for objects that arent locally availabe - that > could be even a plain venti store. I still think that it would make the most sense to do the following (if you insist on some sort of automated repair): (1) Fetch a "good" clone (or clones) into a temporary directory; (2) Cannibalize the objects from it (them); (3) Re-run git fsck and check for still-missing / unreachable items; (4) IF THE RESULT OF (3) IS ACCEPTABLE, run git gc to clean up the mess, discard / "merge" duplicate objects, and fix up the packfiles. It is step (4) that requires the most user interaction. I could see building up a shell script that does all but (4) nearly automatically. None of this requires modifying Git itself. -- -Drew Northup -------------------------------------------------------------- "As opposed to vegetable or mineral error?" -John Pescatore, SANS NewsBites Vol. 12 Num. 59 ``` ## Enrico Weigelt, 2012-11-22 23:16 Subject: Re: Auto-repo-repair Message-ID: URL: https://gitlist.dev/e/cd8ee2cf-2d4a-4d0c-931f-61de4c7f6348%40zcs In-Reply-To: ``` Hi, > I still think that it would make the most sense to do the following > (if you insist on some sort of automated repair): > (1) Fetch a "good" clone (or clones) into a temporary directory; > (2) Cannibalize the objects from it (them); > (3) Re-run git fsck and check for still-missing / unreachable items; > (4) IF THE RESULT OF (3) IS ACCEPTABLE, run git gc to clean up the > mess, discard / "merge" duplicate objects, and fix up the packfiles. > > It is step (4) that requires the most user interaction. I could see > building up a shell script that does all but (4) nearly > automatically. > None of this requires modifying Git itself. Well, I'd like to have some really automatic mode, which does everything ondemand. Once we've got this not just for repair, but also to support quick partial clones that fetch more objects when required. In fact, finally, I'd like to have some storage cloud where data automatically gets replicated to nodes which need the data, not just for VCS, but other purposes (backup, filestore, etc) too. But before inventing someting completely new (reinventing much of the wheel), I'd like to investigate whether git can be extended into this direction step by step. cu -- Mit freundlichen Grüßen / Kind regards Enrico Weigelt VNC - Virtual Network Consult GmbH Head Of Development Pariser Platz 4a, D-10117 Berlin Tel.: +49 (30) 3464615-20 Fax: +49 (30) 3464615-59 enrico.weigelt@vnc.biz; www.vnc.de ```