{"thread":{"id":"43221","subject":"Re: should git download missing objects?","startedAt":"2006-11-12T15:44:25Z","lastAt":"2006-11-14T20:08:27Z","messageCount":10,"participants":["Shawn Pearce","Anand Kumria","Alex Riesen","Petr Baudis","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"295322","messageId":"ej7fgp$8ca$1@sea.gmane.org","threadId":"43221","inReplyTo":null,"subject":"should git download missing objects?","fromName":"Anand Kumria","fromEmail":"wildfire@progsoc.org","sentAt":"2006-11-12T15:44:25Z","receivedAt":"2006-11-12T15:44:25Z","isPatch":false,"sender":{"key":"wildfire@progsoc.org","avatar":null},"body":"Hi,\n\nI did an initial clone of Linus' linux-2.6.git tree, via the git protocol,\nand then managed to accidently delete one of the .pack and\ncorresponding .idx files.\n\nI thought that 'cg-fetch' would do the job of bring down the missing pack\nagain, and all would be well. Alas this isn't the case.\n\n\n\nPasky, on IRC, indicated that this might be because git-fetch-pack isn't\ndownloading missing objects when the git:// protocol is being used.\n\nShould it? Is there a magic invocation of git fetch I can use to fix this\nup. I can always re-clone completely (since this is just a tracking repo)\nbut it would be nice to fix this with the tools themselves.\n\nAny hints?\n\nThanks,\nAnand\n"},{"id":"298582","messageId":"7vwt60bggs.fsf@assigned-by-dhcp.cox.net","threadId":"43221","inReplyTo":"ej7fgp$8ca$1@sea.gmane.org","subject":"Re: should git download missing objects?","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2006-11-12T19:41:23Z","receivedAt":"2006-11-12T19:41:23Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Anand Kumria\" writes:\n\n> I did an initial clone of Linus' linux-2.6.git tree, via the git protocol,\n> and then managed to accidently delete one of the .pack and\n> corresponding .idx files.\n>\n> I thought that 'cg-fetch' would do the job of bring down the missing pack\n> again, and all would be well. Alas this isn't the case.\n>\n> \n>\n> Pasky, on IRC, indicated that this might be because git-fetch-pack isn't\n> downloading missing objects when the git:// protocol is being used.\n\nThere are the invariants between refs and objects:\n\n - objects that its refs (files under .git/refs/ hierarchy that\n record 40-byte hexadecimal object names) point at are never\n missing, or the repository is corrupt.\n\n - objects that are reachable via pointers in another object\n that is not missing (a tag points at another object, a commit\n points at its tree and its parent commits, and a tree points\n at its subtrees and blobs) are never missing, or the repository\n is corrupt.\n\nGit tools first fetch missing objects and then update your refs\nonly when fetch succeeds completely, in order to maintain the\nabove invariants (a partial fetch does not update your refs).\nAnd these invariants are why:\n\n - fsck-objects start reachability check from the refs;\n\n - commit walkers can stop at your existing refs;\n\n - git native protocols only need to tell the other end what\n refs you have, in order for the other end to exclude what you\n already have from the set of objects it sends you.\n\nWhat's missing needs to be determined in a reasonably efficient\nmanner, and the above invariants allow us not have to do the\nequivalent of fsck-objects every time. Being able to trust refs\nis fairly fundamental in the fetch operation of git.\n\nI am not opposed to the idea of a new tool to fix a corrupted\nrepository that has broken the above invariants, perhaps caused\nby accidental removal of objects and packs by end users. What\nit needs to do would be:\n\n - run fsck-objects to notice what are missing, by noting\n \"broken link from foo to bar\" output messages. Object 'bar'\n is what you _ought_ to have according to your refs but you\n don't (because you removed the objects that should be there),\n and everything that is reachable from it from the other side\n needs to be retrieved. Because you do not have 'bar', your\n end cannot determine what other objects you happen to have in\n your object store are reachable from it and would result in\n redundant download.\n\n - run fetch-pack equivalent to get everything reachable\n starting at the above missing objects, pretending you do not\n have any object, because your refs are not trustworthy.\n\n - run fsck-objects again to make sure that your refs can now be\n trusted again.\n\nTo implement the second step above, you need to implement a\nmodified fetch-pack that does not trust any of your refs. It\nalso needs to ignore what are offered from the other end but\nasks the objects you know are missing ('bar' in the above\nexample). This program needs to talk to a modified upload-pack\nrunning at the other end (let's call it upload-pack-recover),\nbecause usual upload-pack does not serve starting from a random\nobject that happen to be in its repository, but only starting\nfrom objects that are pointed by its own set of refs to ensure\nintegrity.\n\nThe upload-pack-recover program would need to start traversal\nfrom object 'bar' in the above example, and when it does so, it\nshould not just run 'rev-list --objects' starting at 'bar'. It\nfirst needs to prove that its object store has everything that\nis reachable from 'bar' (the recipient would still end up with\nan incomplete repository if it didn't).\n\nWhat this means is that it needs to prove some of its refs can\nreach 'bar' (again, on the upstream end, only refs are trusted,\nnot mere existence of object is not enough) before sending\nobjects back. Usual upload-pack do not have to do it because it\nrefuses to serve starting from anything but what its refs point\nat (and by the invariants, the objects pointed at by refs are\nguaranteed to be complete [an object is \"complete\" if no object\nthat can be reachable is not missing]).\n\nThis is needed because the repository might have discarded\nbranch that used to reach 'bar', and while the object 'bar' was\nin a pack but some of its ancestors or component trees and/or\nblobs were loose and subsequent git-prune have removed the\nlatter without removing 'bar'. Mere existence of the object\n'bar' does not mean 'bar' is complete.\n\nSo coming up with such a pair of programs is not a rocket\nscience, but it is fairly delicate. I would rather have them as\nspecialized commands, not a part of everyday commands, even if\nyou were to implement it.\n\nSince this is not everyday anyway, a far easier way would be to\nclone-pack from the upstream into a new repository, take the\npack you downloaded from that new repository and mv it into your\ncorrupt repository. You can run fsck-objects to see if you got\nback everything you lost earlier.\n"},{"id":"295648","messageId":"20061113194532.GA4547@steel.home","threadId":"43221","inReplyTo":"7vwt60bggs.fsf@assigned-by-dhcp.cox.net","subject":"Re: should git download missing objects?","fromName":"Alex Riesen","fromEmail":"fork0@t-online.de","sentAt":"2006-11-13T19:45:32Z","receivedAt":"2006-11-13T19:45:32Z","isPatch":false,"sender":{"key":"raa.lkml@gmail.com","avatar":"https://avatars.githubusercontent.com/u/324101?v=4"},"body":"Junio C Hamano, Sun, Nov 12, 2006 20:41:23 +0100:\n> Since this is not everyday anyway, a far easier way would be to\n> clone-pack from the upstream into a new repository, take the\n> pack you downloaded from that new repository and mv it into your\n> corrupt repository. You can run fsck-objects to see if you got\n> back everything you lost earlier.\n\nI get into such a situation annoyingly often, by using\n\"git clone -l -s from to\" and doing some \"cleanup\" in the\norigin repository. For example, it happens that I remove a tag,\nor a branch, and do a repack or prune afterwards. The related\nrepositories, which had \"accidentally\" referenced the pruned\nobjects become \"corrupt\", as you put it.\n\nAt the moment, if I run into the situation, I copy packs/objects from\nall repos I have (objects/info/alternates are useful here too), run a\nfsck-objects/repack and hope nothing is lost. It works, as I almost\nalways have \"accidental\" backups somewhere, but is kind of annoying to\nsetup. A tool to do this job more effectively will be very handy (at\nleast, it wont have to copy gigabytes of data over switched windows\nnetwork. Not often, I hope. Not _so_ many gigabytes, possibly).\n"},{"id":"294186","messageId":"20061113195414.GD17244@spearce.org","threadId":"43221","inReplyTo":"20061113194532.GA4547@steel.home","subject":"Re: should git download missing objects?","fromName":"Shawn Pearce","fromEmail":"spearce@spearce.org","sentAt":"2006-11-13T19:54:14Z","receivedAt":"2006-11-13T19:54:14Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Alex Riesen wrote:\n> Junio C Hamano, Sun, Nov 12, 2006 20:41:23 +0100:\n> > Since this is not everyday anyway, a far easier way would be to\n> > clone-pack from the upstream into a new repository, take the\n> > pack you downloaded from that new repository and mv it into your\n> > corrupt repository. You can run fsck-objects to see if you got\n> > back everything you lost earlier.\n> \n> I get into such a situation annoyingly often, by using\n> \"git clone -l -s from to\" and doing some \"cleanup\" in the\n> origin repository. For example, it happens that I remove a tag,\n> or a branch, and do a repack or prune afterwards. The related\n> repositories, which had \"accidentally\" referenced the pruned\n> objects become \"corrupt\", as you put it.\n> \n> At the moment, if I run into the situation, I copy packs/objects from\n> all repos I have (objects/info/alternates are useful here too), run a\n> fsck-objects/repack and hope nothing is lost. It works, as I almost\n> always have \"accidental\" backups somewhere, but is kind of annoying to\n> setup. A tool to do this job more effectively will be very handy (at\n> least, it wont have to copy gigabytes of data over switched windows\n> network. Not often, I hope. Not _so_ many gigabytes, possibly).\n\n\nOne of my coworkers recently lost a single loose tree object.\nWe suspect his Windows virus scanner deleted the file. :-(\n\nCopying the one bad object from another repository immediately fixed\nthe breakage caused, but it was very annoying to not be able to run a\n\"git fetch --missing-objects\" or some such. Fortunately it was just\nthe one object and it was also still loose in another repository.\nscp was handy. :-)\n\n-- \n"},{"id":"297173","messageId":"20061113200358.GF7201@pasky.or.cz","threadId":"43221","inReplyTo":"20061113195414.GD17244@spearce.org","subject":"Re: should git download missing objects?","fromName":"Petr Baudis","fromEmail":"pasky@suse.cz","sentAt":"2006-11-13T20:03:58Z","receivedAt":"2006-11-13T20:03:58Z","isPatch":false,"sender":{"key":"pasky@ucw.cz","avatar":"https://avatars.githubusercontent.com/u/18439?v=4"},"body":"On Mon, Nov 13, 2006 at 08:54:14PM CET, Shawn Pearce wrote:\n> Alex Riesen wrote:\n> > Junio C Hamano, Sun, Nov 12, 2006 20:41:23 +0100:\n> > > Since this is not everyday anyway, a far easier way would be to\n> > > clone-pack from the upstream into a new repository, take the\n> > > pack you downloaded from that new repository and mv it into your\n> > > corrupt repository. You can run fsck-objects to see if you got\n> > > back everything you lost earlier.\n> > \n> > I get into such a situation annoyingly often, by using\n> > \"git clone -l -s from to\" and doing some \"cleanup\" in the\n> > origin repository. For example, it happens that I remove a tag,\n> > or a branch, and do a repack or prune afterwards. The related\n> > repositories, which had \"accidentally\" referenced the pruned\n> > objects become \"corrupt\", as you put it.\n> >\n> > At the moment, if I run into the situation, I copy packs/objects from\n> > all repos I have (objects/info/alternates are useful here too), run a\n> > fsck-objects/repack and hope nothing is lost. It works, as I almost\n> > always have \"accidental\" backups somewhere, but is kind of annoying to\n> > setup. A tool to do this job more effectively will be very handy (at\n> > least, it wont have to copy gigabytes of data over switched windows\n> > network. Not often, I hope. Not _so_ many gigabytes, possibly).\n\ncg-fetch -f locally or over HTTP should be able to fix that up, if used\ncleverly.\n\n> One of my coworkers recently lost a single loose tree object.\n> We suspect his Windows virus scanner deleted the file. :-(\n> \n> Copying the one bad object from another repository immediately fixed\n> the breakage caused, but it was very annoying to not be able to run a\n> \"git fetch --missing-objects\" or some such. Fortunately it was just\n> the one object and it was also still loose in another repository.\n> scp was handy. :-)\n\nIf it's over ssh, this is still where the heavily dusty (and heavily\n\"plumby\") git-ssh-fetch command is useful, since it can get passed an\nundocumented --recover argument and then it will fetch _all_ the objects\nyou are missing, not assuming anything.\n\nPerhaps I should reintroduce support for git-ssh-fetch to cg-fetch to be\nused in case of -f over SSH. But it would be silly if I did that and\nnext Git would remove the command from its suite. Junio, what's its life\nexpectancy? I guess this usage scenario is something to take into\naccount when thinking about removing it, I know that I wanted to get rid\nof it in the past but now my opinion is changing.\n\n-- \n\t\t\t\tPetr \"Pasky\" Baudis\nStuff: http://pasky.or.cz/\n#!/bin/perl -sp0777i Junio C Hamano, Sun, Nov 12, 2006 20:41:23 +0100:\n>> Since this is not everyday anyway, a far easier way would be to\n>> clone-pack from the upstream into a new repository, take the\n>> pack you downloaded from that new repository and mv it into your\n>> corrupt repository. You can run fsck-objects to see if you got\n>> back everything you lost earlier.\n>\n> I get into such a situation annoyingly often, by using\n> \"git clone -l -s from to\" and doing some \"cleanup\" in the\n> origin repository. For example, it happens that I remove a tag,\n> or a branch, and do a repack or prune afterwards. The related\n> repositories, which had \"accidentally\" referenced the pruned\n> objects become \"corrupt\", as you put it.\n>\n> At the moment, if I run into the situation, I copy packs/objects from\n> all repos I have (objects/info/alternates are useful here too), run a\n> fsck-objects/repack and hope nothing is lost. It works, as I almost\n> always have \"accidental\" backups somewhere, but is kind of annoying to\n> setup. A tool to do this job more effectively will be very handy (at\n> least, it wont have to copy gigabytes of data over switched windows\n> network. Not often, I hope. Not _so_ many gigabytes, possibly).\n\nI suspect it is a different issue. Maybe you would need reverse\nlinks from the origin directory to .git/refs/ directroy of\nrepositories that borrow from it to prevent pruning. No amount\nof butchering fetch-pack to look behind incomplete refs that lie\nand claim they are complete would solve your problem if you do\nnot have any \"accidental backups\".\n\nIn general, 'git clone -l -s' origin directories may not be\nwritable by the person who is making the clone, so we should not\ndo this inside 'git clone'. Also you could add alternates after\nyou set up your repository, so maybe something like this would\nhelp?\n\n\t#!/bin/sh\n\t#\n\t# Usage: git-add-alternates other_repo\n #\n : ${GIT_DIR=.git}\n\tmy_refs=`cd $GIT_DIR/refs && pwd`\n\tother=$1\n test -d \"$other/.git\" && test -d \"$other/objects\" || {\n \techo >&2 \"I do not see a repository at $other\"\n exit 1\n\t}\n\tmkdir -p \"$other/.git/refs/borrowers\" || {\n \techo >&2 \"You cannot write in $other\"\n \techo >&2 \"Arrange with the owner of it to make\"\n \techo >&2 \"sure the objects you need are not pruned.\"\n exit 2\n }\n\tcnt=0\n while test -d \"$other/.git/refs/borrowers/$cnt\"\n do\n\t\tcnt=$(($cnt + 1))\n\tdone\n ln -s \"$my_refs\" \"$other/.git/refs/borrowers/$cnt\"\n"},{"id":"294062","messageId":"20061113201027.GE17244@spearce.org","threadId":"43221","inReplyTo":"20061113200358.GF7201@pasky.or.cz","subject":"Re: should git download missing objects?","fromName":"Shawn Pearce","fromEmail":"spearce@spearce.org","sentAt":"2006-11-13T20:10:27Z","receivedAt":"2006-11-13T20:10:27Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Petr Baudis wrote:\n> On Mon, Nov 13, 2006 at 08:54:14PM CET, Shawn Pearce wrote:\n> > Copying the one bad object from another repository immediately fixed\n> > the breakage caused, but it was very annoying to not be able to run a\n> > \"git fetch --missing-objects\" or some such. Fortunately it was just\n> > the one object and it was also still loose in another repository.\n> > scp was handy. :-)\n> \n> If it's over ssh, this is still where the heavily dusty (and heavily\n> \"plumby\") git-ssh-fetch command is useful, since it can get passed an\n> undocumented --recover argument and then it will fetch _all_ the objects\n> you are missing, not assuming anything.\n\nInteresting. Since its undocumented I didn't know it existed\nuntil now. :)\n\nI'm thinking though that a --recover should just be part of\ngit-fetch, and that it should work on all transports, not just SSH.\n\nOf course you could get into a whole world of hurt where you keep\ndoing fsck-objects --full (listing out the missing), fetching them,\nonly to find more missing, etc. After a coule of cycles of that it\nmay just be better to claim to the other end that you have nothing\nbut want everything (e.g. an initial clone) and get a new pack from\nwhich you can pull objects.\n\nBut I think that was sort of Junio's point on this topic. I'm just\ntrying to throw in my +1 in favor of a feature that would have\nrecovered that sole missing object without making the end user\nreclone their entire repository and move pack files around by hand.\nAnd I'm being more verbose about it than just +1. :)\n\n-- \n"},{"id":"297986","messageId":"7vac2v6qru.fsf@assigned-by-dhcp.cox.net","threadId":"43221","inReplyTo":"20061113200358.GF7201@pasky.or.cz","subject":"Re: should git download missing objects?","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2006-11-13T20:22:13Z","receivedAt":"2006-11-13T20:22:13Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Petr Baudis writes:\n\n> ... Junio, what's its life\n> expectancy? I guess this usage scenario is something to take into\n> account when thinking about removing it, I know that I wanted to get rid\n> of it in the past but now my opinion is changing.\n\nIt uses the same commit walker semantics and mechanism so I do\nnot think it is too much burden to carry it, but I'd rather have\nsomething that works over git native protocol if we really care\nabout this. People without ssh access needs to be able to\nrecover over git:// protocol.\n\n"},{"id":"295594","messageId":"20061113225239.GA9411@steel.home","threadId":"43221","inReplyTo":"7virhj6rj7.fsf@assigned-by-dhcp.cox.net","subject":"Re: should git download missing objects?","fromName":"Alex Riesen","fromEmail":"fork0@t-online.de","sentAt":"2006-11-13T22:52:39Z","receivedAt":"2006-11-13T22:52:39Z","isPatch":false,"sender":{"key":"raa.lkml@gmail.com","avatar":"https://avatars.githubusercontent.com/u/324101?v=4"},"body":"Junio C Hamano, Mon, Nov 13, 2006 21:05:48 +0100:\n> > Junio C Hamano, Sun, Nov 12, 2006 20:41:23 +0100:\n> >> Since this is not everyday anyway, a far easier way would be to\n> >> clone-pack from the upstream into a new repository, take the\n> >> pack you downloaded from that new repository and mv it into your\n> >> corrupt repository. You can run fsck-objects to see if you got\n> >> back everything you lost earlier.\n> >\n> > I get into such a situation annoyingly often, by using\n> > \"git clone -l -s from to\" and doing some \"cleanup\" in the\n> > origin repository. For example, it happens that I remove a tag,\n> > or a branch, and do a repack or prune afterwards. The related\n> > repositories, which had \"accidentally\" referenced the pruned\n> > objects become \"corrupt\", as you put it.\n> >\n> > At the moment, if I run into the situation, I copy packs/objects from\n> > all repos I have (objects/info/alternates are useful here too), run a\n> > fsck-objects/repack and hope nothing is lost. It works, as I almost\n> > always have \"accidental\" backups somewhere, but is kind of annoying to\n> > setup. A tool to do this job more effectively will be very handy (at\n> > least, it wont have to copy gigabytes of data over switched windows\n> > network. Not often, I hope. Not _so_ many gigabytes, possibly).\n> \n> I suspect it is a different issue. Maybe you would need reverse\n> links from the origin directory to .git/refs/ directroy of\n> repositories that borrow from it to prevent pruning. No amount\n> of butchering fetch-pack to look behind incomplete refs that lie\n> and claim they are complete would solve your problem if you do\n> not have any \"accidental backups\".\n\nIt's is not about preventing this from happening. It is about\nrecovering from user error (which I plainly did). The discussion about\n\"git fetch --recover\" sound very much like what would helped in that\nsituation. I'll just try not doing it next time, but if I do, it'd be\nnice to have a tool to help me recover from it. Not prevent, not\nseeing it possible, just help.\n\nAnyway, it's kind of too late for that repositories. And not very\nconvenient to work with: the branches in the slave repos come and go\noften, they pull from each other and push into central (aka origin)\nrepo. Maintain the borrowed refs in sync would be nightmare (as is: \"I\npromise to forget doing it\").\n"},{"id":"297267","messageId":"20061114200827.GI7201@pasky.or.cz","threadId":"43221","inReplyTo":"7vac2v6qru.fsf@assigned-by-dhcp.cox.net","subject":"Re: should git download missing objects?","fromName":"Petr Baudis","fromEmail":"pasky@suse.cz","sentAt":"2006-11-14T20:08:27Z","receivedAt":"2006-11-14T20:08:27Z","isPatch":false,"sender":{"key":"pasky@ucw.cz","avatar":"https://avatars.githubusercontent.com/u/18439?v=4"},"body":"On Mon, Nov 13, 2006 at 09:22:13PM CET, Junio C Hamano wrote:\n> Petr Baudis writes:\n> \n> > ... Junio, what's its life\n> > expectancy? I guess this usage scenario is something to take into\n> > account when thinking about removing it, I know that I wanted to get rid\n> > of it in the past but now my opinion is changing.\n> \n> It uses the same commit walker semantics and mechanism so I do\n> not think it is too much burden to carry it, but I'd rather have\n> something that works over git native protocol if we really care\n> about this. People without ssh access needs to be able to\n> recover over git:// protocol.\n\nEven though I obviously agree with the above, it would be useful to have\nthe flag even though git:// (which is apparently harder to get right\nthan the others) is not supported. After all, most repositories I've\nseen that are available over git:// are available over HTTP as well.\n\n-- \n\t\t\t\tPetr \"Pasky\" Baudis\nStuff: http://pasky.or.cz/\n#!/bin/perl -sp0777i