{"thread":{"id":"13887","subject":"Recovering from repository corruption","startedAt":"2008-06-10T17:26:51Z","lastAt":"2008-06-12T12:20:16Z","messageCount":31,"participants":["Denis Bueno","Jakub Narebski","Nicolas Pitre","Linus Torvalds","Tarmigan","Junio C Hamano","Stephen R. van den Berg","Johan Herland","Jeff King"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"79369","messageId":"6dbd4d000806101026m458513ecqa8141f509bad7602@mail.gmail.com","threadId":"13887","inReplyTo":null,"subject":"Recovering from repository corruption","fromName":"Denis Bueno","fromEmail":"dbueno@gmail.com","sentAt":"2008-06-10T17:26:51Z","receivedAt":"2008-06-10T17:26:51Z","isPatch":false,"sender":{"key":"dbueno@gmail.com","avatar":"https://gravatar.com/avatar/591be06714161d33b59a485e6b822cbd40ecc0b91fd188b9924ea22c405996f7?d=mp&s=160"},"body":"I started a thread a while back about repository corruption.  It\nmanifested as a clone error and the thread is here:\n\n    http://kerneltrap.org/mailarchive/git/2007/7/31/253475\n\nI just ran, again, into corruption after my laptop kernel-panic'd.\n(Ironically, at the moment I ran into the corruption I was trying to\npush my repo to a backup location.)  Since that thread took place it\nseems a section about recovering from repo corruption was added to the\nmanual --- but it assumes you can (or care to painstakingly) recreate\neach corrupted version.\n\nI made several changes to one file, home.html, and now have the\nfollowing corruption:\n\n    identity.corrupt[56] > git fsck --full\n    error: 320bd6e82267b71dd2ca7043ea3f61dbbca16109: object corrupt or missing\n    error: 4d0be2816d5eea5ae2b40990235e2225c1715927: object corrupt or missing\n    missing blob 320bd6e82267b71dd2ca7043ea3f61dbbca16109\n    missing blob 4d0be2816d5eea5ae2b40990235e2225c1715927\n\nI know which commits these hashes correspond to, and I know roughly\nwhat I did in those commits, but, I really don't care that much, and\nanyway it will be painful to recreate them because of\nwhitespace/formatting issues.  Here are the commits, in case it is\nrelevant:\n\n    commit 163a93df14d246dee91c3a503e6372b8313f337d\n    Author: Denis Bueno <dbueno@gmail.com>\n    Date:   Tue Jun 10 09:45:41 2008 -0400\n\n        Add lambda-the-ultimate link\n\n    :100644 100644 320bd6e... 2ab4775... M  home.html\n\n    [... intervenent commits ...]\n\n    commit 4737fea59fdc8325e09b5206cc7a6ac593446ce3\n    Author: Denis Bueno <dbueno@gmail.com>\n    Date:   Tue Jun 10 09:37:12 2008 -0400\n\n        Hoogle up top too\n\n    :100644 100644 4d0be28... c6fe111... M  home.html\n\nAssuming I can't recreate the hashed files, what are my options?\n\nI was told in the thread above that I could use grafts and \"git\nfilter-branch\" to create a new repository that simply got rid of the\noffending object.  That case was simpler, as it was the initial import\nof a file that had only two commits total that was corrupted.\nHowever, in this case there are changes between the initial and latest\nversion of the file, and commits between the corrupted versions, so, I\ncan imagine that it would be hard to get rid of in-between commits.\n\nThe thing that makes sense intuitively (read: not as a Git expert, but\nas a user) is to just let me replace the commits associated with the\nproblematic objects with new versions of those commits (e.g. make\nchange described in the commit message, which is different from the\nactual change that was recorded, due to whitespace/formatting issues).\n Is this what I should do?  And to do so, should I be reading chapter\n5 of the manual?\n\nThanks.\n\n-- \n                              Denis\n"},{"id":"79374","messageId":"m3abhtp42o.fsf@localhost.localdomain","threadId":"13887","inReplyTo":"6dbd4d000806101026m458513ecqa8141f509bad7602@mail.gmail.com","subject":"Re: Recovering from repository corruption","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2008-06-10T17:55:52Z","receivedAt":"2008-06-10T17:55:52Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"\"Denis Bueno\" <dbueno@gmail.com> writes:\n\n> I was told in the thread above that I could use grafts and \"git\n> filter-branch\" to create a new repository that simply got rid of the\n> offending object.  That case was simpler, as it was the initial import\n> of a file that had only two commits total that was corrupted.\n> However, in this case there are changes between the initial and latest\n> version of the file, and commits between the corrupted versions, so, I\n> can imagine that it would be hard to get rid of in-between commits.\n> \n> The thing that makes sense intuitively (read: not as a Git expert, but\n> as a user) is to just let me replace the commits associated with the\n> problematic objects with new versions of those commits (e.g. make\n> change described in the commit message, which is different from the\n> actual change that was recorded, due to whitespace/formatting issues).\n>  Is this what I should do?  And to do so, should I be reading chapter\n> 5 of the manual?\n\nWithout checking Git User's Manual, I think the solution could go as\nthe following.\n\nAssume that history looks like this\n\n    ...---.---a---*---b---.---...\n\nwhere by '*' is marked corruped commit (commit shich tree contains\ncorrupted blobs).\n\nFirst, you can check the commit message for '*' using git-cat-file or\ngit-show, you can get the difference between 'a' and 'b' using \n\"git diff a b\".  When you know how repaired commit 'X' should look\nlike, do something like:\n\n  $ git checkout -b <temp-branch> 'a'\n  $ <edit edit edit>\n  $ git commit\n\nThen history would look like this\n\n    ...---.---a---*---b---.---...\n               \\\n                \\-X\n\nNow with grafts make 'b' be a child of 'X', i.e. modify parent of 'b'\nfor history to look like below:\n\n    ...---.---a---*   b---.---...\n               \\     /\n                \\-X-/\n\nExamine history using git-log, git-show, check tree with git-ls-tree\nand examining files, use graphical history browser like gitk.\n\nThen if possible use git-filter-branch to make history recorded in\ngrafts file permanent...\n\nHTH\n-- \nJakub Narebski\nPoland\nShadeHawk on #git\n"},{"id":"79377","messageId":"6dbd4d000806101238v2bb975abqd39916e45d4bf866@mail.gmail.com","threadId":"13887","inReplyTo":"m3abhtp42o.fsf@localhost.localdomain","subject":"Re: Recovering from repository corruption","fromName":"Denis Bueno","fromEmail":"dbueno@gmail.com","sentAt":"2008-06-10T19:38:09Z","receivedAt":"2008-06-10T19:38:09Z","isPatch":false,"sender":{"key":"dbueno@gmail.com","avatar":"https://gravatar.com/avatar/591be06714161d33b59a485e6b822cbd40ecc0b91fd188b9924ea22c405996f7?d=mp&s=160"},"body":"On Tue, Jun 10, 2008 at 13:55, Jakub Narebski <jnareb@gmail.com> wrote:\n> Assume that history looks like this\n>\n>    ...---.---a---*---b---.---...\n>\n> where by '*' is marked corruped commit (commit shich tree contains\n> corrupted blobs).\n>\n> First, you can check the commit message for '*' using git-cat-file or\n> git-show, you can get the difference between 'a' and 'b' using\n> \"git diff a b\".  When you know how repaired commit 'X' should look\n> like, do something like:\n>\n>  $ git checkout -b <temp-branch> 'a'\n>  $ <edit edit edit>\n>  $ git commit\n>\n> Then history would look like this\n>\n>    ...---.---a---*---b---.---...\n>               \\\n>                \\-X\n>\n> Now with grafts make 'b' be a child of 'X', i.e. modify parent of 'b'\n> for history to look like below:\n>\n>    ...---.---a---*   b---.---...\n>               \\     /\n>                \\-X-/\n>\n> Examine history using git-log, git-show, check tree with git-ls-tree\n> and examining files, use graphical history browser like gitk.\n>\n> Then if possible use git-filter-branch to make history recorded in\n> grafts file permanent...\n>\n> HTH\n> --\n> Jakub Narebski\n> Poland\n> ShadeHawk on #git\n>\n\nThanks for the help.\n\nMy situation was:\n\n    ...---a---*---b---c---d---*---e---...\n\nFollowing your example, I believe I got this to:\n\n    ...---a---*   b---c---d---*   e---...\n           \\     /         \\     /\n            \\-X-/           \\---/\n\nThat is, I replaced the first problematic commit and deleted the\nsecond, since I forgot how I changed 'd' to get that commit.  I put\nthe following in .git/info/grafts:\n\n    'b' X\n    'e' 'd'\n\n(which I gathered from here:\nhttp://thread.gmane.org/gmane.comp.version-control.git/66398/focus=66402.\n I've never use grafts before.  A bit about them should be put in the\nmanual, if it's not there already. =])\n\nThen I ran:\n\n    git-filter-branch HEAD ^X ^'d'\n\nNow \"git log --raw --all\" doesn't show any of the problematic SHA-1\nhashes anymore!\n\nHowever:\n\nidentity.fb[173] > git fsck --full\n    error: 320bd6e82267b71dd2ca7043ea3f61dbbca16109: object corrupt or missing\n    error: 4d0be2816d5eea5ae2b40990235e2225c1715927: object corrupt or missing\n    missing blob 320bd6e82267b71dd2ca7043ea3f61dbbca16109\n    missing blob 4d0be2816d5eea5ae2b40990235e2225c1715927\n\nShouldn't these be unreferenced now that I've run filter-branch?\n\n-- \n                              Denis\n"},{"id":"79378","messageId":"alpine.LFD.1.10.0806101537440.23110@xanadu.home","threadId":"13887","inReplyTo":"6dbd4d000806101026m458513ecqa8141f509bad7602@mail.gmail.com","subject":"Re: Recovering from repository corruption","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2008-06-10T19:40:04Z","receivedAt":"2008-06-10T19:40:04Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Tue, 10 Jun 2008, Denis Bueno wrote:\n\n> I started a thread a while back about repository corruption.  It\n> manifested as a clone error and the thread is here:\n> \n>     http://kerneltrap.org/mailarchive/git/2007/7/31/253475\n> \n> I just ran, again, into corruption after my laptop kernel-panic'd.\n> (Ironically, at the moment I ran into the corruption I was trying to\n> push my repo to a backup location.)  Since that thread took place it\n> seems a section about recovering from repo corruption was added to the\n> manual --- but it assumes you can (or care to painstakingly) recreate\n> each corrupted version.\n\nWould you happen, by chance, to have another instance of that repository \nsomewhere else with the concerned objects in it?\n\n\nNicolas\n"},{"id":"79380","messageId":"6dbd4d000806101242s23e79a4fyfeb7ae0a40b1078f@mail.gmail.com","threadId":"13887","inReplyTo":"alpine.LFD.1.10.0806101537440.23110@xanadu.home","subject":"Re: Recovering from repository corruption","fromName":"Denis Bueno","fromEmail":"dbueno@gmail.com","sentAt":"2008-06-10T19:42:38Z","receivedAt":"2008-06-10T19:42:38Z","isPatch":false,"sender":{"key":"dbueno@gmail.com","avatar":"https://gravatar.com/avatar/591be06714161d33b59a485e6b822cbd40ecc0b91fd188b9924ea22c405996f7?d=mp&s=160"},"body":"On Tue, Jun 10, 2008 at 15:40, Nicolas Pitre <nico@cam.org> wrote:\n>> (Ironically, at the moment I ran into the corruption I was trying to\n>> push my repo to a backup location.)\n>\n> Would you happen, by chance, to have another instance of that repository\n> somewhere else with the concerned objects in it?\n\nNope.  I was *just* about to back it up.\n\n-- \n                              Denis\n"},{"id":"79381","messageId":"200806102159.02875.jnareb@gmail.com","threadId":"13887","inReplyTo":"6dbd4d000806101238v2bb975abqd39916e45d4bf866@mail.gmail.com","subject":"Re: Recovering from repository corruption","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2008-06-10T19:59:02Z","receivedAt":"2008-06-10T19:59:02Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"On Tue, 10 Jun 2008, Denis Bueno wrote:\n\n> However:\n> \n> identity.fb[173] > git fsck --full\n>     error: 320bd6e82267b71dd2ca7043ea3f61dbbca16109: object corrupt or missing\n>     error: 4d0be2816d5eea5ae2b40990235e2225c1715927: object corrupt or missing\n>     missing blob 320bd6e82267b71dd2ca7043ea3f61dbbca16109\n>     missing blob 4d0be2816d5eea5ae2b40990235e2225c1715927\n> \n> Shouldn't these be unreferenced now that I've run filter-branch?\n\nTry to clone this repository (using file:/// pseudo-protocol to force\ntransfer of objects instead of hardlinking them), and chek if the\nproblem persists in the clone too.  If not, error/missing might be\nin \"garbage\".\n\nBut I'm not sure...\n-- \nJakub Narebski\nPoland\n"},{"id":"79382","messageId":"6dbd4d000806101303j4b2032ajc6e004e0a82e4db5@mail.gmail.com","threadId":"13887","inReplyTo":"200806102159.02875.jnareb@gmail.com","subject":"Re: Recovering from repository corruption","fromName":"Denis Bueno","fromEmail":"dbueno@gmail.com","sentAt":"2008-06-10T20:03:46Z","receivedAt":"2008-06-10T20:03:46Z","isPatch":false,"sender":{"key":"dbueno@gmail.com","avatar":"https://gravatar.com/avatar/591be06714161d33b59a485e6b822cbd40ecc0b91fd188b9924ea22c405996f7?d=mp&s=160"},"body":"On Tue, Jun 10, 2008 at 15:59, Jakub Narebski <jnareb@gmail.com> wrote:\n>> Shouldn't these be unreferenced now that I've run filter-branch?\n>\n> Try to clone this repository (using file:/// pseudo-protocol to force\n> transfer of objects instead of hardlinking them), and chek if the\n> problem persists in the clone too.  If not, error/missing might be\n> in \"garbage\".\n>\n> But I'm not sure...\n\nYou're onto something:\n\n[dorothy.local /tmp <Tue Jun 10> <16:02:08>]\ntmp[176] > git clone file:///Volumes/work/identity.fb/\nInitialized empty Git repository in /tmp/identity.fb/.git/\nremote: Counting objects: 401, done.\nremote: Compressing objects: 100% (364/364), done.\nremote: Total 401 (delta 170), reused 0 (delta 0)\nReceiving objects: 100% (401/401), 233.76 KiB, done.\nResolving deltas: 100% (170/170), done.\n\n[dorothy.local /tmp <Tue Jun 10> <16:02:22>]\ntmp[177] > cd identity.fb/\n/tmp/identity.fb\n\n[dorothy.local /tmp/identity.fb <Tue Jun 10> <16:02:24>]\nidentity.fb[178] > git fsck --full\nbroken link from  commit 4737fea59fdc8325e09b5206cc7a6ac593446ce3\n              to  commit fe431b4b69453ad9207a5528cf9b9d12ef69c988\ndangling commit 28aa69aafc8ae901e588f6d341b3e6d3558c6d26\ndangling commit 884a8024fbcb9367726abb25f8bb6ac539712d46\nmissing commit fe431b4b69453ad9207a5528cf9b9d12ef69c988\n\nBut I've just substituted one error for another.  Are these errors\neasier to fix?\n\n\n-- \n                              Denis\n"},{"id":"79384","messageId":"200806102214.40805.jnareb@gmail.com","threadId":"13887","inReplyTo":"6dbd4d000806101303j4b2032ajc6e004e0a82e4db5@mail.gmail.com","subject":"Re: Recovering from repository corruption","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2008-06-10T20:14:40Z","receivedAt":"2008-06-10T20:14:40Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"On Tue, 10 Jun 2008, Denis Bueno wrote:\n> On Tue, Jun 10, 2008, Jakub Narebski <jnareb@gmail.com> wrote: \n>> Denis Bueno wrote:\n>>>\n>>> Shouldn't these be unreferenced now that I've run filter-branch?\n>>\n>> Try to clone this repository (using file:/// pseudo-protocol to force \n>> transfer of objects instead of hardlinking them), and chek if the\n>> problem persists in the clone too.  If not, error/missing might be\n>> in \"garbage\".\n>>\n>> But I'm not sure...\n> \n> You're onto something:\n> \n> [dorothy.local /tmp <Tue Jun 10> <16:02:08>]\n> tmp[176] > git clone file:///Volumes/work/identity.fb/\n> Initialized empty Git repository in /tmp/identity.fb/.git/\n> remote: Counting objects: 401, done.\n> remote: Compressing objects: 100% (364/364), done.\n> remote: Total 401 (delta 170), reused 0 (delta 0)\n> Receiving objects: 100% (401/401), 233.76 KiB, done.\n> Resolving deltas: 100% (170/170), done.\n> \n> [dorothy.local /tmp <Tue Jun 10> <16:02:22>]\n> tmp[177] > cd identity.fb/\n> /tmp/identity.fb\n> \n> [dorothy.local /tmp/identity.fb <Tue Jun 10> <16:02:24>]\n> identity.fb[178] > git fsck --full\n> broken link from  commit 4737fea59fdc8325e09b5206cc7a6ac593446ce3\n>               to  commit fe431b4b69453ad9207a5528cf9b9d12ef69c988\n> dangling commit 28aa69aafc8ae901e588f6d341b3e6d3558c6d26\n> dangling commit 884a8024fbcb9367726abb25f8bb6ac539712d46\n> missing commit fe431b4b69453ad9207a5528cf9b9d12ef69c988\n> \n> But I've just substituted one error for another.  Are these errors\n> easier to fix?\n\nPlease remember that in such clone you _don't_ have grafts info (unless\nyou copy it manually), so it is a good test if you correctly rewrote \nhistory using git-filter-branch.  So take a look at history in your \nclone using gitk or some similar tool.\n\nIn the history you mentioned:\n\n    ...---a---*   b---c---d---*   e---...\n           \\     /         \\     /\n            \\-X-/           \\---/\n\nyou should rewritr from 'a'=='X^' to, and including 'e' (and not only \nfrom 'd').\n\n\nBut if it is not the case I'm afraid I wouldn't be able to offer any \nfurther insight...\n\n-- \nJakub Narebski\nPoland\n"},{"id":"79385","messageId":"alpine.LFD.1.10.0806101317100.3101@woody.linux-foundation.org","threadId":"13887","inReplyTo":"6dbd4d000806101303j4b2032ajc6e004e0a82e4db5@mail.gmail.com","subject":"Re: Recovering from repository corruption","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2008-06-10T20:23:56Z","receivedAt":"2008-06-10T20:23:56Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Tue, 10 Jun 2008, Denis Bueno wrote:\n>\n> You're onto something:\n> \n> [dorothy.local /tmp <Tue Jun 10> <16:02:08>]\n> tmp[176] > git clone file:///Volumes/work/identity.fb/\n\n[ successful ]\n\nHmm. Scary. That should *not* have been successful with a corrupt repo.\n\nUnless you have done a .grafts file to hide the corruption, or something \nlike that?\n\nHave you saved away the original corrupt repo (the whole .git directory as \na tar-ball, for example)? And is the data public and non-embarrassing \nenough so that you could make it available for some post-corruption \nanalysis? Even if we cannot help recover it, real-life corruption is \nalways interesting to see if only as a test-case to make sure that git \nnotices it as quickly as possible.\n\n\t\t\tLinus\n"},{"id":"79386","messageId":"6dbd4d000806101328k1fc913f2ia55c3e44273ec5ad@mail.gmail.com","threadId":"13887","inReplyTo":"alpine.LFD.1.10.0806101317100.3101@woody.linux-foundation.org","subject":"Re: Recovering from repository corruption","fromName":"Denis Bueno","fromEmail":"dbueno@gmail.com","sentAt":"2008-06-10T20:28:43Z","receivedAt":"2008-06-10T20:28:43Z","isPatch":false,"sender":{"key":"dbueno@gmail.com","avatar":"https://gravatar.com/avatar/591be06714161d33b59a485e6b822cbd40ecc0b91fd188b9924ea22c405996f7?d=mp&s=160"},"body":"On Tue, Jun 10, 2008 at 16:23, Linus Torvalds\n<torvalds@linux-foundation.org> wrote:\n>\n>\n> On Tue, 10 Jun 2008, Denis Bueno wrote:\n>>\n>> You're onto something:\n>>\n>> [dorothy.local /tmp <Tue Jun 10> <16:02:08>]\n>> tmp[176] > git clone file:///Volumes/work/identity.fb/\n>\n> [ successful ]\n>\n> Hmm. Scary. That should *not* have been successful with a corrupt repo.\n>\n> Unless you have done a .grafts file to hide the corruption, or something\n> like that?\n\nI intended to do that, yes, and I think I was successful.  (I only say\nI \"intended to\" --- instead of \"I did\" --- because I read the\ndocumentation for the grafts file elsewhere on this list, and not in\nsome more \"blessed\" location.)\n\n> Have you saved away the original corrupt repo (the whole .git directory as\n> a tar-ball, for example)? And is the data public and non-embarrassing\n> enough so that you could make it available for some post-corruption\n> analysis? Even if we cannot help recover it, real-life corruption is\n> always interesting to see if only as a test-case to make sure that git\n> notices it as quickly as possible.\n\nI do have bunches of personal information in the repo, unfortunately.\nThe particular *file* involved in the corruption, however, is fine for\nall to view.  Is that useful?\n\n\n-- \n                              Denis\n"},{"id":"79387","messageId":"6dbd4d000806101335y5bb7660cge5bbaf571ebce98e@mail.gmail.com","threadId":"13887","inReplyTo":"200806102214.40805.jnareb@gmail.com","subject":"Re: Recovering from repository corruption","fromName":"Denis Bueno","fromEmail":"dbueno@gmail.com","sentAt":"2008-06-10T20:35:51Z","receivedAt":"2008-06-10T20:35:51Z","isPatch":false,"sender":{"key":"dbueno@gmail.com","avatar":"https://gravatar.com/avatar/591be06714161d33b59a485e6b822cbd40ecc0b91fd188b9924ea22c405996f7?d=mp&s=160"},"body":"On Tue, Jun 10, 2008 at 16:14, Jakub Narebski <jnareb@gmail.com> wrote:\n> Please remember that in such clone you _don't_ have grafts info (unless\n> you copy it manually), so it is a good test if you correctly rewrote\n> history using git-filter-branch.  So take a look at history in your\n> clone using gitk or some similar tool.\n>\n> In the history you mentioned:\n>\n>    ...---a---*   b---c---d---*   e---...\n>           \\     /         \\     /\n>            \\-X-/           \\---/\n>\n> you should rewritr from 'a'=='X^' to, and including 'e' (and not only\n> from 'd').\n\nSo I re-did the filter-branch as:\n\n    git-filter-branch HEAD\n28aa69aafc8ae901e588f6d341b3e6d3558c6d26^..163a93df14d246dee91c3a503e6372b8313f337d\n\nNow cloning still works and only shows dangling commits --- no errors!\n\n-- \n                              Denis\n"},{"id":"79388","messageId":"alpine.LFD.1.10.0806101403080.3101@woody.linux-foundation.org","threadId":"13887","inReplyTo":"6dbd4d000806101328k1fc913f2ia55c3e44273ec5ad@mail.gmail.com","subject":"Re: Recovering from repository corruption","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2008-06-10T21:09:42Z","receivedAt":"2008-06-10T21:09:42Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Tue, 10 Jun 2008, Denis Bueno wrote:\n> >\n> > Hmm. Scary. That should *not* have been successful with a corrupt repo.\n> >\n> > Unless you have done a .grafts file to hide the corruption, or something\n> > like that?\n> \n> I intended to do that, yes, and I think I was successful.\n\nAhh, ok. Yes, we should probably re-think our 'grafts' file thing, or at \nleast not document it, because it's actually a wondeful way to just cause \nmore corruption by hiding things (ie if you clone a repo with a grafts \nfile, the result will now have neither the grafts file _nor_ the state \nthat was hidden by it, so the result is guaranteed to be corrupt).\n\nBut that explains why your clone worked, and why the resulting repo had \ndifferent corruption - it avoided the original corruption, but because of \nthe grafts file it avoided it by just not having those commits at all..\n\n> I do have bunches of personal information in the repo, unfortunately.\n> The particular *file* involved in the corruption, however, is fine for\n> all to view.  Is that useful?\n\nNo, almost all the interest is basically in how the whole repo ties \ntogether. The individual corrupt files may be interesting, though, ie from \nyour original report:\n\n    error: 320bd6e82267b71dd2ca7043ea3f61dbbca16109: object corrupt or missing\n    error: 4d0be2816d5eea5ae2b40990235e2225c1715927: object corrupt or missing\n\nthen *if* you have the files\n\n\t.git/objects/32/0bd6e82267b71dd2ca7043ea3f61dbbca16109\n\t.git/objects/4d/0be2816d5eea5ae2b40990235e2225c1715927\n\nthen those two files are interesting in themselves (most likely they are \nnot there at all, or are zero-sized, but if you have them, please post \nthem).\n\nAnd as this was a result of a real filesystem crash, it *is* possible that \nyou have something in the /lost+found directory for that filesystem. If \nso, those missing files may be found there.\n\n\t\tLinus\n"},{"id":"79390","messageId":"6dbd4d000806101422j39709906x1b4b03b82b504e62@mail.gmail.com","threadId":"13887","inReplyTo":"alpine.LFD.1.10.0806101403080.3101@woody.linux-foundation.org","subject":"Re: Recovering from repository corruption","fromName":"Denis Bueno","fromEmail":"dbueno@gmail.com","sentAt":"2008-06-10T21:22:49Z","receivedAt":"2008-06-10T21:22:49Z","isPatch":false,"sender":{"key":"dbueno@gmail.com","avatar":"https://gravatar.com/avatar/591be06714161d33b59a485e6b822cbd40ecc0b91fd188b9924ea22c405996f7?d=mp&s=160"},"body":"On Tue, Jun 10, 2008 at 17:09, Linus Torvalds\n<torvalds@linux-foundation.org> wrote:\n> No, almost all the interest is basically in how the whole repo ties\n> together. The individual corrupt files may be interesting, though, ie from\n> your original report:\n>\n>    error: 320bd6e82267b71dd2ca7043ea3f61dbbca16109: object corrupt or missing\n>    error: 4d0be2816d5eea5ae2b40990235e2225c1715927: object corrupt or missing\n>\n> then *if* you have the files\n>\n>        .git/objects/32/0bd6e82267b71dd2ca7043ea3f61dbbca16109\n>        .git/objects/4d/0be2816d5eea5ae2b40990235e2225c1715927\n>\n> then those two files are interesting in themselves (most likely they are\n> not there at all, or are zero-sized, but if you have them, please post\n> them).\n\nThey are attached, and they are not zero-sized.\n\n> And as this was a result of a real filesystem crash, it *is* possible that\n> you have something in the /lost+found directory for that filesystem. If\n> so, those missing files may be found there.\n\nI checked; no such luck.\n\n-- \n                              Denis\n"},{"id":"79391","messageId":"6dbd4d000806101427p338b52bate470f6f8b68221df@mail.gmail.com","threadId":"13887","inReplyTo":"alpine.LFD.1.10.0806101403080.3101@woody.linux-foundation.org","subject":"Re: Recovering from repository corruption","fromName":"Denis Bueno","fromEmail":"dbueno@gmail.com","sentAt":"2008-06-10T21:27:19Z","receivedAt":"2008-06-10T21:27:19Z","isPatch":false,"sender":{"key":"dbueno@gmail.com","avatar":"https://gravatar.com/avatar/591be06714161d33b59a485e6b822cbd40ecc0b91fd188b9924ea22c405996f7?d=mp&s=160"},"body":"On Tue, Jun 10, 2008 at 17:09, Linus Torvalds\n<torvalds@linux-foundation.org> wrote:\n> Ahh, ok. Yes, we should probably re-think our 'grafts' file thing, or at\n> least not document it, because it's actually a wondeful way to just cause\n> more corruption by hiding things (ie if you clone a repo with a grafts\n> file, the result will now have neither the grafts file _nor_ the state\n> that was hidden by it, so the result is guaranteed to be corrupt).\n\nI'd argue in favor of documenting it, even if it's dangerous, unless\nthere's some other mechanism (rebase?) that would let me do what I\ndid?  That is, to recover from corruption in a way that lets me\nregenerate or ignore inexact, corrupted commits.\n\n-- \n Denis\n"},{"id":"79392","messageId":"alpine.LFD.1.10.0806101431410.3101@woody.linux-foundation.org","threadId":"13887","inReplyTo":"6dbd4d000806101422j39709906x1b4b03b82b504e62@mail.gmail.com","subject":"Re: Recovering from repository corruption","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2008-06-10T21:48:12Z","receivedAt":"2008-06-10T21:48:12Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Tue, 10 Jun 2008, Denis Bueno wrote:\n> >\n> > then *if* you have the files\n> >\n> >        .git/objects/32/0bd6e82267b71dd2ca7043ea3f61dbbca16109\n> >        .git/objects/4d/0be2816d5eea5ae2b40990235e2225c1715927\n> >\n> > then those two files are interesting in themselves (most likely they are\n> > not there at all, or are zero-sized, but if you have them, please post\n> > them).\n> \n> They are attached, and they are not zero-sized.\n\nVery interesting.\n\nBoth of them look fairly sane as objects (ie random - it's supposed to eb \nzlib-compressed), but both of them have the first 512 bytes *identically* \ncorrupted:\n\n\t0000000 6564 626e 6575 406e 6f64 6f72 6874 2e79\n\t          d   e   n   b   u   e   n   @   d   o   r   o   t   h   y   .\n\t0000020 6f6c 6163 2e6c 3634 0033 0000 0000 0000\n\t          l   o   c   a   l   .   4   6   3  \\0  \\0  \\0  \\0  \\0  \\0  \\0\n\t0000040 0000 0000 0000 0000 0000 0000 0000 0000\n\t         \\0  \\0  \\0  \\0  \\0  \\0  \\0  \\0  \\0  \\0  \\0  \\0  \\0  \\0  \\0  \\0\n\t*\n\nie it's an all-zero block, except for that email-looking thing at the \nhead. \n\nSadly, I don't think there is any way to find the missing block that got \noverwritten. And quite frankly, there's no way to really know whether the \nrest was really fine either - it just looks more likely, but quite \nfrankly, it could have been random old contents on your disk too that just \nhappens to look like the expected random pattern (which you'll get with \nany compression format - compression by definition removes patterns).\n\nOne thign that strikes me is that you seem to be really prone to this \nproblem, since it happened to you a year ago too. I cannot swear to this, \nbut I literally suspect your last case (July-2007) was the previous time \nwe had a corruption issue. Why does it seem to happen to you, but not \nothers?\n\nDo you have some odd filesystem in play? Was the current corruption in a \nsimilar environment as the old one? IOW, I'm trying to find a pattern \nhere, to see if there might be something we can do about it..\n\nBut it *sounds* like the objects you lost were literally old ones, no? Ie \nthe lost stuff wasn't something you had committed in the last five minutes \nor so? If so, then you really do seem to have a filesystem that corrupts \n*old* files when it crashes. That's fairly scary. What FS is it?\n\n\t\tLinus\n"},{"id":"79393","messageId":"6dbd4d000806101509l516cf467me06fadee6ead0964@mail.gmail.com","threadId":"13887","inReplyTo":"alpine.LFD.1.10.0806101431410.3101@woody.linux-foundation.org","subject":"Re: Recovering from repository corruption","fromName":"Denis Bueno","fromEmail":"dbueno@gmail.com","sentAt":"2008-06-10T22:09:20Z","receivedAt":"2008-06-10T22:09:20Z","isPatch":false,"sender":{"key":"dbueno@gmail.com","avatar":"https://gravatar.com/avatar/591be06714161d33b59a485e6b822cbd40ecc0b91fd188b9924ea22c405996f7?d=mp&s=160"},"body":"On Tue, Jun 10, 2008 at 17:48, Linus Torvalds\n<torvalds@linux-foundation.org> wrote:\n>\n>\n> On Tue, 10 Jun 2008, Denis Bueno wrote:\n>> >\n>> > then *if* you have the files\n>> >\n>> >        .git/objects/32/0bd6e82267b71dd2ca7043ea3f61dbbca16109\n>> >        .git/objects/4d/0be2816d5eea5ae2b40990235e2225c1715927\n>> >\n>> > then those two files are interesting in themselves (most likely they are\n>> > not there at all, or are zero-sized, but if you have them, please post\n>> > them).\n>>\n>> They are attached, and they are not zero-sized.\n>\n> Very interesting.\n>\n> Both of them look fairly sane as objects (ie random - it's supposed to eb\n> zlib-compressed), but both of them have the first 512 bytes *identically*\n> corrupted:\n>\n>        0000000 6564 626e 6575 406e 6f64 6f72 6874 2e79\n>                  d   e   n   b   u   e   n   @   d   o   r   o   t   h   y   .\n>        0000020 6f6c 6163 2e6c 3634 0033 0000 0000 0000\n>                  l   o   c   a   l   .   4   6   3  \\0  \\0  \\0  \\0  \\0  \\0  \\0\n>        0000040 0000 0000 0000 0000 0000 0000 0000 0000\n>                 \\0  \\0  \\0  \\0  \\0  \\0  \\0  \\0  \\0  \\0  \\0  \\0  \\0  \\0  \\0  \\0\n>        *\n>\n> ie it's an all-zero block, except for that email-looking thing at the\n> head.\n\nRight --- that's my username and computer's hostname... for some\nreason.  [You are not expected to understand this.  My computer's name\nmysteriously changed.  It should not be \"dorothy.local\" but it is.  I\nwill have to find out why....]\n\n> One thign that strikes me is that you seem to be really prone to this\n> problem, since it happened to you a year ago too. I cannot swear to this,\n> but I literally suspect your last case (July-2007) was the previous time\n> we had a corruption issue. Why does it seem to happen to you, but not\n> others?\n\nIt is the same computer on which the problem occurred last time.  It's\nan OS X 10.4 macbook pro.  I haven't noticed corruption in other\nplaces, but it's fair to assume it's occurring.  I'll have to boot off\nmy install disk and fsck the drive....\n\n> Do you have some odd filesystem in play? Was the current corruption in a\n> similar environment as the old one? IOW, I'm trying to find a pattern\n> here, to see if there might be something we can do about it..\n\nI can't remember if the old one happened after a panic or not, but I'd\nbet it did.  The filesystem is HFS+, as indeed most OS X 10.4\ninstallations are.  Maybe the HD has been going south?  However, that\ndoesn't seem likely, since when I got the computer it was new, and\nthat was around Jun 2007.\n\n> But it *sounds* like the objects you lost were literally old ones, no? Ie\n> the lost stuff wasn't something you had committed in the last five minutes\n> or so? If so, then you really do seem to have a filesystem that corrupts\n> *old* files when it crashes. That's fairly scary. What FS is it?\n\nNo, in fact I had just committed those changes not 10 minutes before\nthe panic.  Last time they were also fresh changes, although perhaps\nolder than 10 minutes.  I can't remember.\n\n\n-- \n Denis\n"},{"id":"79394","messageId":"905315640806101525n26a0a4eic7943613ab9e1a8c@mail.gmail.com","threadId":"13887","inReplyTo":"6dbd4d000806101509l516cf467me06fadee6ead0964@mail.gmail.com","subject":"Re: Recovering from repository corruption","fromName":"Tarmigan","fromEmail":"tarmigan+git@gmail.com","sentAt":"2008-06-10T22:25:08Z","receivedAt":"2008-06-10T22:25:08Z","isPatch":false,"sender":{"key":"tarmigan+git@gmail.com","avatar":null},"body":"On Tue, Jun 10, 2008 at 3:09 PM, Denis Bueno <dbueno@gmail.com> wrote:\n> It is the same computer on which the problem occurred last time.  It's\n> an OS X 10.4 macbook pro.  I haven't noticed corruption in other\n> places, but it's fair to assume it's occurring.  I'll have to boot off\n> my install disk and fsck the drive....\n\nDo you have fink installed?  Do you have the openssl fink package\ninstalled?  Vger seems to have swallowed my original reply, but see\nthis thread:\nhttp://marc.info/?l=git&m=120787191106549&w=2\nIf so, try removing the fink openssl packages and reinstalling git.\n\nDo you push from this machine often?  If you do, then this probably is\nnot your problem as you would have seen it earlier.\n\n-Tarmigan\n"},{"id":"79397","messageId":"6dbd4d000806101541g32d57f18r2372a360f6c3ba2f@mail.gmail.com","threadId":"13887","inReplyTo":"905315640806101525n26a0a4eic7943613ab9e1a8c@mail.gmail.com","subject":"Re: Recovering from repository corruption","fromName":"Denis Bueno","fromEmail":"dbueno@gmail.com","sentAt":"2008-06-10T22:41:48Z","receivedAt":"2008-06-10T22:41:48Z","isPatch":false,"sender":{"key":"dbueno@gmail.com","avatar":"https://gravatar.com/avatar/591be06714161d33b59a485e6b822cbd40ecc0b91fd188b9924ea22c405996f7?d=mp&s=160"},"body":"On Tue, Jun 10, 2008 at 18:25, Tarmigan <tarmigan+git@gmail.com> wrote:\n> Do you have fink installed?  Do you have the openssl fink package\n> installed?  Vger seems to have swallowed my original reply, but see\n> this thread:\n> http://marc.info/?l=git&m=120787191106549&w=2\n> If so, try removing the fink openssl packages and reinstalling git.\n\nI use macports.\n\n> Do you push from this machine often?  If you do, then this probably is\n> not your problem as you would have seen it earlier.\n\nYes, almost exclusively.  ... That is an odd problem.  Thanks for the\nsuggestion.\n\n-- \n Denis\n"},{"id":"79398","messageId":"alpine.LFD.1.10.0806101518590.3101@woody.linux-foundation.org","threadId":"13887","inReplyTo":"6dbd4d000806101509l516cf467me06fadee6ead0964@mail.gmail.com","subject":"Re: Recovering from repository corruption","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2008-06-10T22:45:05Z","receivedAt":"2008-06-10T22:45:05Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Tue, 10 Jun 2008, Denis Bueno wrote:\n> \n> > Do you have some odd filesystem in play? Was the current corruption in a\n> > similar environment as the old one? IOW, I'm trying to find a pattern\n> > here, to see if there might be something we can do about it..\n> \n> I can't remember if the old one happened after a panic or not, but I'd\n> bet it did.  The filesystem is HFS+, as indeed most OS X 10.4\n> installations are.  Maybe the HD has been going south?  However, that\n> doesn't seem likely, since when I got the computer it was new, and\n> that was around Jun 2007.\n\nYeah, it's almost certainly not the disk. Disks do go bad, but the \nbehavior tends to be rather different when they do (usually you will get \nread errors with uncorrectably CRC failures, and you'd know that _very_ \nclearly).\n\nSure, I could imagine something like the sector remapping could be flaking \nout on you, but that sounds really unlikely. Especially since:\n\n> > But it *sounds* like the objects you lost were literally old ones, no? Ie\n> > the lost stuff wasn't something you had committed in the last five minutes\n> > or so? If so, then you really do seem to have a filesystem that corrupts\n> > *old* files when it crashes. That's fairly scary. What FS is it?\n> \n> No, in fact I had just committed those changes not 10 minutes before\n> the panic.  Last time they were also fresh changes, although perhaps\n> older than 10 minutes.  I can't remember.\n\nOh, ok. If so, then this is much less worrisome, and is in fact almost \n\"normal\" HFS+ behaviour. It is a journaling filesystem, but it only \njournals metadata, so the filenames and inodes will be fine after a crash, \nbut the contents will be random.\n\n[ Yeah, yeah, I know - it sounds rather stupid, but it's a common kind of \n  stupidity. The journaling essentially protects the only thing that fsck \n  can find. Ext3 does similar things in \"writeback\" mode - but you should \n  use \"data=ordered\" which writes out the data before metadata.\n\n  Basically, such journaling doesn't help data integrity per se, but it \n  does mean that the metadata is ok, and that in turn means that while the \n  file contents won't be dependable, at least things like free block \n  bitmaps etc hopefully are.\n\n  That in turn hopefully means that new file allocations won't be \n  crapping out all over old ones etc due to bad resource allocations, so \n  while it doesn't mean that the data is trust-worthy, it at least means \n  that you can trust _some_ things ]\n\nIf your machine crashes often, you could trivially add a \"sync\" to your \ncommit hook. That would make things better. And maybe we should have a \n\"safe mode\" that does these things more carefully. You would definitely \nwant to turn it on on that machine.\n\nAre you doing something special to make the machine crash so much? Or do \nOS X machines always crash, and Apple PR is just so good that people \naren't aware of it?\n\nAnyway, I'll think about sane ways to add a \"safe\" mode without making it \n_too_ painful. In the meantime, here's a trial patch that you should \nprobably use. It does slow things down, but hopefully not too much.\n\n(I really don't much like it - but I think this is a good change, and I \njust need to come up with a better way to do the fsync() than to be \ntotally synchronous about it.)\n\nIt's going to make big \"git add\" calls *much* slower, so I'm not very \nhappy about it (especially since we don't actually care that deeply about \nthe files really being there until much later, so doing something \nasynchronous would be perfectly acceptable), but for you this is \ndefinitely worth-while.\n\n\t\t\tLinus\n\n---\n sha1_file.c |   17 +++++++++++------\n 1 files changed, 11 insertions(+), 6 deletions(-)\n\ndiff --git a/sha1_file.c b/sha1_file.c\nindex adcf37c..86a653b 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -2105,6 +2105,15 @@ int hash_sha1_file(const void *buf, unsigned long len, const char *type,\n \treturn 0;\n }\n \n+/* Finalize a file on disk, and close it. */\n+static void close_sha1_file(int fd)\n+{\n+\tfsync_or_die(fd, \"sha1 file\");\n+\tfchmod(fd, 0444);\n+\tif (close(fd) != 0)\n+\t\tdie(\"unable to write sha1 file\");\n+}\n+\n static int write_loose_object(const unsigned char *sha1, char *hdr, int hdrlen,\n \t\t\t      void *buf, unsigned long len, time_t mtime)\n {\n@@ -2170,9 +2179,7 @@ static int write_loose_object(const unsigned char *sha1, char *hdr, int hdrlen,\n \n \tif (write_buffer(fd, compressed, size) < 0)\n \t\tdie(\"unable to write sha1 file\");\n-\tfchmod(fd, 0444);\n-\tif (close(fd))\n-\t\tdie(\"unable to write sha1 file\");\n+\tclose_sha1_file(fd);\n \tfree(compressed);\n \n \tif (mtime) {\n@@ -2350,9 +2357,7 @@ int write_sha1_from_fd(const unsigned char *sha1, int fd, char *buffer,\n \t} while (1);\n \tinflateEnd(&stream);\n \n-\tfchmod(local, 0444);\n-\tif (close(local) != 0)\n-\t\tdie(\"unable to write sha1 file\");\n+\tclose_sha1_file(local);\n \tSHA1_Final(real_sha1, &c);\n \tif (ret != Z_STREAM_END) {\n \t\tunlink(tmpfile);\n"},{"id":"79399","messageId":"7v63sgdhs1.fsf@gitster.siamese.dyndns.org","threadId":"13887","inReplyTo":"alpine.LFD.1.10.0806101403080.3101@woody.linux-foundation.org","subject":"Re: Recovering from repository corruption","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2008-06-10T22:52:46Z","receivedAt":"2008-06-10T22:52:46Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Linus Torvalds <torvalds@linux-foundation.org> writes:\n\n> Ahh, ok. Yes, we should probably re-think our 'grafts' file thing, or at \n> least not document it, because it's actually a wondeful way to just cause \n> more corruption by hiding things (ie if you clone a repo with a grafts \n> file, the result will now have neither the grafts file _nor_ the state \n> that was hidden by it, so the result is guaranteed to be corrupt).\n\n\"Graft and then clone\" will not make the copied repository Ok.  You need\nto propagate the graft in some other way.\n\nHowever, \"Graft and then filter-branch\" is a way to hide and get rid of\nthe the broken thing in history etched in the objects.  After that the\nrepository itself and a clone from it will not need the graft.  So I'd\nrather argue we should document it _differently_ (or just _better_) than\nnot document it.\n"},{"id":"79401","messageId":"alpine.LFD.1.10.0806101554490.3101@woody.linux-foundation.org","threadId":"13887","inReplyTo":"alpine.LFD.1.10.0806101518590.3101@woody.linux-foundation.org","subject":"Re: Recovering from repository corruption","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2008-06-10T23:00:49Z","receivedAt":"2008-06-10T23:00:49Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Tue, 10 Jun 2008, Linus Torvalds wrote:\n> \n> It's going to make big \"git add\" calls *much* slower, so I'm not very \n> happy about it (especially since we don't actually care that deeply about \n> the files really being there until much later, so doing something \n> asynchronous would be perfectly acceptable), but for you this is \n> definitely worth-while.\n\nFor me, on the whole kernel, on a pretty good system:\n\n - before:\n\n\t[torvalds@woody test-it-out]$ time git add .\n\n\treal    0m7.986s\n\tuser    0m6.404s\n\tsys     0m1.456s\n\n - after:\n\n\t[torvalds@woody test-it-out]$ time ~/git/git-add .\n\n\treal    0m52.693s\n\tuser    0m7.416s\n\tsys     0m2.516s\n\nso it's definitely quite noticeable in that simplistic form. \n\nA more interesting patch would use aio_fsync(), and then just wait for \nthem at the end with aio_return(). Not that I love AIO, but this is \ndefinitely a case where it would make sense to do (of course, systems \nwithout AIO support would then fall back to regular fsync()).\n\nI will have to think about this.\n\n\t\t\tLinus\n"},{"id":"79407","messageId":"alpine.LFD.1.10.0806102026430.23110@xanadu.home","threadId":"13887","inReplyTo":"alpine.LFD.1.10.0806101518590.3101@woody.linux-foundation.org","subject":"Re: Recovering from repository corruption","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2008-06-11T00:43:58Z","receivedAt":"2008-06-11T00:43:58Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Tue, 10 Jun 2008, Linus Torvalds wrote:\n\n> Anyway, I'll think about sane ways to add a \"safe\" mode without making it \n> _too_ painful. In the meantime, here's a trial patch that you should \n> probably use. It does slow things down, but hopefully not too much.\n> \n> (I really don't much like it - but I think this is a good change, and I \n> just need to come up with a better way to do the fsync() than to be \n> totally synchronous about it.)\n> \n> It's going to make big \"git add\" calls *much* slower, so I'm not very \n> happy about it (especially since we don't actually care that deeply about \n> the files really being there until much later, so doing something \n> asynchronous would be perfectly acceptable), but for you this is \n> definitely worth-while.\n\nI don't like it at all.\n\nI think this only gives a false sense of security with a huge \nperformance cost.  If the machine crashes at the right moment, the \nobject will still be half written/fsync'd and you'll be in the same \nsituation again.\n\nAnd because we don't overwrite existing objects (again for performance \nreasons), then a corrupted blob object will remain corrupted even if you \nreattempt the commit later.  So doing the fsync only when the commit \nobject is written isn't a good solution either.\n\nI wonder if supporting crashy systems is worth that cost.  If Denis' \nlaptop is the odd case then a sync in the commit hook might be plenty \nsufficient.  Personally I'd simply replace the OS or the machine for \nsomething more reliable.\n\n\nNicolas\n"},{"id":"79414","messageId":"alpine.LFD.1.10.0806101836510.3101@woody.linux-foundation.org","threadId":"13887","inReplyTo":"alpine.LFD.1.10.0806102026430.23110@xanadu.home","subject":"Re: Recovering from repository corruption","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2008-06-11T01:39:51Z","receivedAt":"2008-06-11T01:39:51Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Tue, 10 Jun 2008, Nicolas Pitre wrote:\n> \n> I think this only gives a false sense of security with a huge \n> performance cost.  If the machine crashes at the right moment, the \n> object will still be half written/fsync'd and you'll be in the same \n> situation again.\n\nNo you wouldn't.\n\nWe do the write and the fsync() of the write to a _temporary_ filename. We \ndo the rename _after_ the fsync.\n\nSo you'd never have a half-written object file.\n\nThat said, I do agree that the bigger problem is that Denis' machine is \nsimply so unreliable.\n\n\t\t\tLinus\n"},{"id":"79416","messageId":"alpine.LFD.1.10.0806102143080.23110@xanadu.home","threadId":"13887","inReplyTo":"alpine.LFD.1.10.0806101836510.3101@woody.linux-foundation.org","subject":"Re: Recovering from repository corruption","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2008-06-11T01:47:38Z","receivedAt":"2008-06-11T01:47:38Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Tue, 10 Jun 2008, Linus Torvalds wrote:\n\n> \n> \n> On Tue, 10 Jun 2008, Nicolas Pitre wrote:\n> > \n> > I think this only gives a false sense of security with a huge \n> > performance cost.  If the machine crashes at the right moment, the \n> > object will still be half written/fsync'd and you'll be in the same \n> > situation again.\n> \n> No you wouldn't.\n> \n> We do the write and the fsync() of the write to a _temporary_ filename. We \n> do the rename _after_ the fsync.\n\nAh, true.  That part somehow evaded my mind.\n\n\nNicolas\n"},{"id":"79524","messageId":"20080611232126.GA9054@cuci.nl","threadId":"13887","inReplyTo":"alpine.LFD.1.10.0806101403080.3101@woody.linux-foundation.org","subject":"To graft or not to graft... (Re: Recovering from repository corruption)","fromName":"Stephen R. van den Berg","fromEmail":"srb@cuci.nl","sentAt":"2008-06-11T23:21:26Z","receivedAt":"2008-06-11T23:21:26Z","isPatch":false,"sender":{"key":"srb@cuci.nl","avatar":"https://gravatar.com/avatar/f75389059e827634d38e9df2a9b6ecbd50028b5a454442efa1c7205b7ff29c6a?d=mp&s=160"},"body":"Linus Torvalds wrote:\n>more corruption by hiding things (ie if you clone a repo with a grafts \n>file, the result will now have neither the grafts file _nor_ the state \n>that was hidden by it, so the result is guaranteed to be corrupt).\n\nThis is kind of confusing.\nAs I understood it from the few shreds of documentation that actually\nmention the grafts file, the grafts file is *not* being cloned.\nTherefore, my assumption was that cloning a repository that has a grafts\nfile gives an identical result to cloning the same repository *without*\nthe grafts file present.\n\nAs I understand it now, the cloning process actually peeks at the grafts\nfile while cloning, and then doesn't copy it.  This results in a rather\nconfusingly corrupt clone.\n\nI suggest two things:\na. That during the cloning process, the grafts file is completely\n   disregarded in any case at first.\nb. Preferably the grafts file is copied as well (after cloning).  I\n   never really understood why the file is not being copied in the first\n   place (anyone care to explain that?).\n-- \nSincerely,\n           Stephen R. van den Berg.\n\nDifferentiation is an integral part of calculus.\n"},{"id":"79526","messageId":"m3ve0fr1fg.fsf@localhost.localdomain","threadId":"13887","inReplyTo":"20080611232126.GA9054@cuci.nl","subject":"Re: To graft or not to graft... (Re: Recovering from repository corruption)","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2008-06-11T23:34:31Z","receivedAt":"2008-06-11T23:34:31Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"\"Stephen R. van den Berg\" <srb@cuci.nl> writes:\n\n> This is kind of confusing.\n>\n> As I understood it from the few shreds of documentation that actually\n> mention the grafts file, the grafts file is *not* being cloned.\n> Therefore, my assumption was that cloning a repository that has a grafts\n> file gives an identical result to cloning the same repository *without*\n> the grafts file present.\n> \n> As I understand it now, the cloning process actually peeks at the grafts\n> file while cloning, and then doesn't copy it.  This results in a rather\n> confusingly corrupt clone.\n> \n> I suggest two things:\n> a. That during the cloning process, the grafts file is completely\n>    disregarded in any case at first.\n> b. Preferably the grafts file is copied as well (after cloning).  I\n>    never really understood why the file is not being copied in the first\n>    place (anyone care to explain that?).\n\nA bit of explanation: initially I think grafts were created as a means\nto \"graft\" historical repository (conversion from BitKeeper and from\npatches) to current work repository (from when git was deemed suitable\nas SCM for Linux kernel development).  Nevertheless the machenism is\ngeneric enough to change history _locally_ in many strange ways (for\nexample shallow clone uses kind of grafts).\n\nBecause graft file can be used to alter history, this totally\n_bypases_ the check given by sha1 of commit and cryptographically\nsigned tags.  It negates security given by sha-1 signing.  That's why\nusing grafs must be _conscious_ decision - therefore they are purely\nlocal and not propagated.\n\n(Also there were no place for grafts in the \"smart\" trasport, i.e. git\nand ssh protocols.  Thinking about what happens if both sides have\ngrafs files which differ...)\n\nOn the other hand history _without_ grafts might not validate.  I\nthink that it is why current confusing behavior...\n\n-- \nJakub Narebski\nPoland\nShadeHawk on #git\n"},{"id":"79527","messageId":"alpine.LFD.1.10.0806111635200.3101@woody.linux-foundation.org","threadId":"13887","inReplyTo":"20080611232126.GA9054@cuci.nl","subject":"Re: To graft or not to graft... (Re: Recovering from repository corruption)","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2008-06-11T23:39:44Z","receivedAt":"2008-06-11T23:39:44Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Thu, 12 Jun 2008, Stephen R. van den Berg wrote:\n>\n> As I understood it from the few shreds of documentation that actually\n> mention the grafts file, the grafts file is *not* being cloned.\n> Therefore, my assumption was that cloning a repository that has a grafts\n> file gives an identical result to cloning the same repository *without*\n> the grafts file present.\n\nThat would probably be the right behaviour, but no - all our commit \nwalkers honor the grafts file.\n\nIncluding the ones used for creating pack-files and thus a clone.\n\n> As I understand it now, the cloning process actually peeks at the grafts\n> file while cloning, and then doesn't copy it.  This results in a rather\n> confusingly corrupt clone.\n\nYes. The grafts-file was a mistake, but it's just barely useful to some \npeople that it's stayed alive. Sadly, those \"some people\" don't tend to \ncare enough about the problems it can cause.\n\n> I suggest two things:\n> a. That during the cloning process, the grafts file is completely\n>    disregarded in any case at first.\n\nYes.\n\nAnd (a'): git-fsck and repacking should just consider it to be an \n_additional_ source of parenthood rather than a _replacement_ source.\n\n> b. Preferably the grafts file is copied as well (after cloning).  I\n>    never really understood why the file is not being copied in the first\n>    place (anyone care to explain that?).\n\nThe grafts file isn't part of the object stream and refs, and clones (and \nfetches) very much just copy the object database.\n\n\t\tLinus\n"},{"id":"79570","messageId":"200806120914.22083.johan@herland.net","threadId":"13887","inReplyTo":"alpine.LFD.1.10.0806111635200.3101@woody.linux-foundation.org","subject":"Re: To graft or not to graft... (Re: Recovering from repository corruption)","fromName":"Johan Herland","fromEmail":"johan@herland.net","sentAt":"2008-06-12T07:14:21Z","receivedAt":"2008-06-12T07:14:21Z","isPatch":false,"sender":{"key":"johan@herland.net","avatar":"https://avatars.githubusercontent.com/u/547031?v=4"},"body":"On Thursday 12 June 2008, Linus Torvalds wrote:\n> On Thu, 12 Jun 2008, Stephen R. van den Berg wrote:\n> > As I understood it from the few shreds of documentation that actually\n> > mention the grafts file, the grafts file is *not* being cloned.\n> > Therefore, my assumption was that cloning a repository that has a\n> > grafts file gives an identical result to cloning the same repository\n> > *without* the grafts file present.\n>\n> That would probably be the right behaviour, but no - all our commit\n> walkers honor the grafts file.\n>\n> Including the ones used for creating pack-files and thus a clone.\n>\n> > As I understand it now, the cloning process actually peeks at the\n> > grafts file while cloning, and then doesn't copy it.  This results in a\n> > rather confusingly corrupt clone.\n>\n> Yes. The grafts-file was a mistake, but it's just barely useful to some\n> people that it's stayed alive. Sadly, those \"some people\" don't tend to\n> care enough about the problems it can cause.\n>\n> > I suggest two things:\n> > a. That during the cloning process, the grafts file is completely\n> >    disregarded in any case at first.\n>\n> Yes.\n>\n> And (a'): git-fsck and repacking should just consider it to be an\n> _additional_ source of parenthood rather than a _replacement_ source.\n>\n> > b. Preferably the grafts file is copied as well (after cloning).  I\n> >    never really understood why the file is not being copied in the\n> > first place (anyone care to explain that?).\n>\n> The grafts file isn't part of the object stream and refs, and clones (and\n> fetches) very much just copy the object database.\n\nAFAICS, there's already a perfectly fine way to distribute grafted history:\n1. Add a grafts file\n2. Run git-filter-branch\n3. Remove grafts file\n4. Distribute repo\n5. Profit!\n\nSince git-filter-branch turns grafted parentage into _real_ parentage,\nthere's no point in ever having a grafts file at all (except transiently\nfor telling git-filter-branch what to do).\n\nI suggest we make commit walkers NOT obey the grafts file by default, but\ninstead require a --follow-grafts option to restore the current behaviour.\nThen, we teach git-filter-branch to obey the grafts file (probably by\nemploying said --follow-grafts option).\n\nFor those who want to hang on to the current behaviour, they can create\nsome config option that is equivalent to always running with\n--follow-grafts.\n\n\nThe following is ugly, untested, undocumented, and obviously unfit for\ninclusion:\n\n\ndiff --git a/commit.c b/commit.c\nindex 94d5b3d..3e9ebf7 100644\n--- a/commit.c\n+++ b/commit.c\n@@ -7,6 +7,7 @@\n #include \"revision.h\"\n \n int save_commit_buffer = 1;\n+int use_grafts = 0;\n \n const char *commit_type = \"commit\";\n \n@@ -242,7 +243,7 @@ int parse_commit_buffer(struct commit *item, void *buffer, unsigned long size)\n \tchar *bufptr = buffer;\n \tunsigned char parent[20];\n \tstruct commit_list **pptr;\n-\tstruct commit_graft *graft;\n+\tstruct commit_graft *graft = NULL;\n \tunsigned n_refs = 0;\n \n \tif (item->object.parsed)\n@@ -260,7 +261,8 @@ int parse_commit_buffer(struct commit *item, void *buffer, unsigned long size)\n \tbufptr += 46; /* \"tree \" + \"hex sha1\" + \"\\n\" */\n \tpptr = &item->parents;\n \n-\tgraft = lookup_commit_graft(item->object.sha1);\n+\tif (use_grafts)\n+\t\tgraft = lookup_commit_graft(item->object.sha1);\n \twhile (bufptr + 48 < tail && !memcmp(bufptr, \"parent \", 7)) {\n \t\tstruct commit *new_parent;\n \ndiff --git a/commit.h b/commit.h\nindex 2d94d41..3e30aa0 100644\n--- a/commit.h\n+++ b/commit.h\n@@ -22,6 +22,7 @@ struct commit {\n };\n \n extern int save_commit_buffer;\n+extern int use_grafts;\n extern const char *commit_type;\n \n /* While we can decorate any object with a name, it's only used for commits.. */\ndiff --git a/git-filter-branch.sh b/git-filter-branch.sh\nindex d04c346..5ebe7cd 100755\n--- a/git-filter-branch.sh\n+++ b/git-filter-branch.sh\n@@ -230,11 +230,11 @@ mkdir ../map || die \"Could not create map/ directory\"\n case \"$filter_subdir\" in\n \"\")\n \tgit rev-list --reverse --topo-order --default HEAD \\\n-\t\t--parents \"$@\"\n+\t\t--follow-grafts --parents \"$@\"\n \t;;\n *)\n \tgit rev-list --reverse --topo-order --default HEAD \\\n-\t\t--parents \"$@\" -- \"$filter_subdir\"\n+\t\t--follow-grafts --parents \"$@\" -- \"$filter_subdir\"\n esac > ../revs || die \"Could not get the commits\"\n commits=$(wc -l <../revs | tr -d \" \")\n \ndiff --git a/revision.c b/revision.c\nindex 5a1a948..ca98815 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -1059,6 +1059,10 @@ int setup_revisions(int argc, const char **argv, struct rev_info *revs, const ch\n \t\t\t\trevs->first_parent_only = 1;\n \t\t\t\tcontinue;\n \t\t\t}\n+\t\t\tif (!strcmp(arg, \"--follow-grafts\")) {\n+\t\t\t\tuse_grafts = 1;\n+\t\t\t\tcontinue;\n+\t\t\t}\n \t\t\tif (!strcmp(arg, \"--reflog\")) {\n \t\t\t\thandle_reflog(revs, flags);\n \t\t\t\tcontinue;\n-- \n1.5.6.rc2.128.gf64ae\n\n\nHave fun! :)\n\n...Johan\n\n-- \nJohan Herland, <johan@herland.net>\nwww.herland.net\n"},{"id":"79574","messageId":"20080612074752.GA507@sigill.intra.peff.net","threadId":"13887","inReplyTo":"200806120914.22083.johan@herland.net","subject":"Re: To graft or not to graft... (Re: Recovering from repository corruption)","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2008-06-12T07:47:53Z","receivedAt":"2008-06-12T07:47:53Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Jun 12, 2008 at 09:14:21AM +0200, Johan Herland wrote:\n\n> > The grafts file isn't part of the object stream and refs, and clones (and\n> > fetches) very much just copy the object database.\n> \n> AFAICS, there's already a perfectly fine way to distribute grafted history:\n> 1. Add a grafts file\n> 2. Run git-filter-branch\n> 3. Remove grafts file\n> 4. Distribute repo\n> 5. Profit!\n> \n> Since git-filter-branch turns grafted parentage into _real_ parentage,\n> there's no point in ever having a grafts file at all (except transiently\n> for telling git-filter-branch what to do).\n\nBut then you have rewritten all of the later commits, so you can no\nlonger talk to other people about them.\n\nThe kernel repo is split into \"historical\" and active repos. You can\ngraft the historical repo and get more far-reaching answers to things\nlike \"git log\" and \"git blame\". But if you run filter-branch, you can't\nshare development on that repo via push / pull to people who _don't_ use\nthe graft, since they don't share your history (and they probably don't\nwant to, because of the extra resources required to pull in the\nhistorical chunk).\n\nThat being said, I don't know how common such a setup is. And you did\nmention a \"follow-grafts\" config option for such people.\n\n-Peff\n"},{"id":"79581","messageId":"200806121221.02287.johan@herland.net","threadId":"13887","inReplyTo":"20080612074752.GA507@sigill.intra.peff.net","subject":"Re: To graft or not to graft... (Re: Recovering from repository corruption)","fromName":"Johan Herland","fromEmail":"johan@herland.net","sentAt":"2008-06-12T10:21:02Z","receivedAt":"2008-06-12T10:21:02Z","isPatch":false,"sender":{"key":"johan@herland.net","avatar":"https://avatars.githubusercontent.com/u/547031?v=4"},"body":"On Thursday 12 June 2008, Jeff King wrote:\n> On Thu, Jun 12, 2008 at 09:14:21AM +0200, Johan Herland wrote:\n> > > The grafts file isn't part of the object stream and refs, and\n> > > clones (and fetches) very much just copy the object database.\n> >\n> > AFAICS, there's already a perfectly fine way to distribute grafted\n> > history: 1. Add a grafts file\n> > 2. Run git-filter-branch\n> > 3. Remove grafts file\n> > 4. Distribute repo\n> > 5. Profit!\n> >\n> > Since git-filter-branch turns grafted parentage into _real_\n> > parentage, there's no point in ever having a grafts file at all\n> > (except transiently for telling git-filter-branch what to do).\n>\n> But then you have rewritten all of the later commits, so you can no\n> longer talk to other people about them.\n\nCorrect. My point is that if you want to talk to people about revisions, \nyou'd better do it from a repo where people agree on the entire \nhistory. On the other hand, if you want to do archaeology with grafts, \nyou should be aware that you are subverting one of the core guarantees \nprovided by Git (i.e. a commit id verifies full ancestry of a commit), \nand therefore shouldn't communicate with other repos _at_ _all_, as \nother repos can easily be confused (see [1]).\n\n> The kernel repo is split into \"historical\" and active repos. You can\n> graft the historical repo and get more far-reaching answers to things\n> like \"git log\" and \"git blame\". But if you run filter-branch, you\n> can't share development on that repo via push / pull to people who\n> _don't_ use the graft, since they don't share your history (and they\n> probably don't want to, because of the extra resources required to\n> pull in the historical chunk).\n\nYes, by forcing git-filter-branch, you can no longer push/pull to/from \nsuch a historical repo. But as this thread has already demonstrated, \nwith grafts you can't clone from such a repo today (nor pull in certain \ncircumstances, see [1]); so the way I see it, communication with this \nrepo is _already_ limited. By disallowing grafts and forcing a rewrite \nof the entire repo, we force these communication problems to be more \nexplicit/visible.\n\n> That being said, I don't know how common such a setup is. And you did\n> mention a \"follow-grafts\" config option for such people.\n\nIndeed. :)\n\nAFAICS, there's two use cases for grafts:\n1. As a preparation for rewriting the history with git-filter-branch.\n2. For providing historical repos (like you mention above).\n\nMy suggestion only makes life harder for people in the second use case.\nIf there are many people in the second use case, and they deem \nthe \"follow-grafts\" config option unacceptable, I expect them to flame \nmy suggestion to a crisp, and we'll have to think of something else...\n\n\nHave fun! :)\n\n...Johan\n\n[1]: Consider the following:\n\n### Create a repo with one commit, A\n$ mkdir foo\n$ cd foo\n$ git init\nInitialized empty Git repository in /path/to/foo/.git/\n$ echo foo > foo\n$ git add foo\n$ git commit -mA\nCreated initial commit fe2ec02: A\n 1 files changed, 1 insertions(+), 0 deletions(-)\n create mode 100644 foo\n### Clone the repo\n$ cd ..\n$ git clone /path/to/foo bar\nInitialize bar/.git\nInitialized empty Git repository in /path/to/bar/.git/\n### Create 3 more commits in the original repo: A---B---C---D\n$ cd foo\n$ echo bar >> foo && git commit -a -mB\nCreated commit ad10f00: B\n 1 files changed, 1 insertions(+), 0 deletions(-)\n$ echo baz >> foo && git commit -a -mC\nCreated commit be96559: C\n 1 files changed, 1 insertions(+), 0 deletions(-)\n$ echo xyzzy >> foo && git commit -a -mD\nCreated commit f2bafe5: D\n 1 files changed, 1 insertions(+), 0 deletions(-)\n### Create a graft removing C from the history: A---B---D\n$ echo \"f2bafe58175e132077285e7fbbcec30859101d2e \\ \nad10f005205f61429dccda95e1442dabe31fbfbe\" > .git/info/grafts\n### Pull the recent changes into the clone\n$ cd ../bar\n$ git pull\nremote: Counting objects: 8, done.\nremote: Compressing objects: 100% (2/2), done.\nUnpacking objects: 100% (6/6), done.\nremote: Total 6 (delta 0), reused 0 (delta 0)\nerror: Could not read be965599d99192f624b8d8bbf3cab412872586fc\nFrom /path/to/foo/\n + fe2ec02...f2bafe5 master     -> origin/master  (forced update)\nerror: Could not read be965599d99192f624b8d8bbf3cab412872586fc\nerror: Could not read be965599d99192f624b8d8bbf3cab412872586fc\nAuto-merged foo\nCONFLICT (add/add): Merge conflict in foo\nAutomatic merge failed; fix conflicts and then commit the result.\n\nAFAICS, git-pull can easily become just as confused by grafts as \ngit-clone. I wouldn't be surprised by a similar example for git-push.\n\nI can only draw the conclusion that with current versions of Git, repos \nwith grafts should _never_ be made public.\n\n-- \nJohan Herland, <johan@herland.net>\nwww.herland.net\n"},{"id":"79592","messageId":"20080612122016.GA25926@cuci.nl","threadId":"13887","inReplyTo":"200806121221.02287.johan@herland.net","subject":"Re: To graft or not to graft... (Re: Recovering from repository corruption)","fromName":"Stephen R. van den Berg","fromEmail":"srb@cuci.nl","sentAt":"2008-06-12T12:20:16Z","receivedAt":"2008-06-12T12:20:16Z","isPatch":false,"sender":{"key":"srb@cuci.nl","avatar":"https://gravatar.com/avatar/f75389059e827634d38e9df2a9b6ecbd50028b5a454442efa1c7205b7ff29c6a?d=mp&s=160"},"body":"Johan Herland wrote:\n>I can only draw the conclusion that with current versions of Git, repos \n>with grafts should _never_ be made public.\n\nCorrect.\n\nI still prefer my original suggestion, i.e. allow repos with grafts to\nbe cloned, yet disregard the grafts during the cloning process.\nThe trouble is that with your suggestion, it becomes a bit convoluted\nwhen grafts are being used and when not.  It already is complicated as\nit is, so I suggest we try and keep git honest so that it does exactly\nwhat one would expect (instead of documenting awkward behaviour).\n\nAs soon as time permits, I'll submit appropriate patches to implement\nthis, as well as some other sanity check patches which I've been\ncontemplating to help the grafter detect \"bad\" grafts as early as\npossible.\n-- \nSincerely,\n           Stephen R. van den Berg.\n\n\"Always look on the bright side of life!\"\n"}]}