{"thread":{"id":"10891","subject":"New repo quickly corrupted","startedAt":"2007-11-15T21:50:15Z","lastAt":"2007-11-17T14:13:37Z","messageCount":11,"participants":["Jason Sewall","Junio C Hamano","Linus Torvalds","Nicolas Pitre","Christian Couder","Andreas Ericsson","Robin Rosenberg"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"60026","messageId":"31e9dd080711151350u6c2ae40foc7c05e59496260fa@mail.gmail.com","threadId":"10891","inReplyTo":null,"subject":"New repo quickly corrupted","fromName":"Jason Sewall","fromEmail":"jasonsewall@gmail.com","sentAt":"2007-11-15T21:50:15Z","receivedAt":"2007-11-15T21:50:15Z","isPatch":false,"sender":{"key":"jasonsewall@gmail.com","avatar":null},"body":"I was working on a new repo created from a tar snapshot of an old\nproject, and after about a half-hour and about 5 commits for hacking,\nI had fixed the problem I set out to take care of.\n\nHowever, I tried to run git gc to clean it up and put it away and I\nwas met with the following:\n\n[sewall@hobo SWIFT++_1.2 (master)]$ git gc\nCounting objects: 227, done.\nerror: corrupt loose object '680aba836639d14a36b81f9e29da52bd9af69770'\nfatal: object 680aba836639d14a36b81f9e29da52bd9af69770 cannot be read\nerror: failed to run repack\n\nI've followed discussions on the list about corrupted repos and did\n[sewall@hobo SWIFT++_1.2 (master)]$ git fsck\nerror: corrupt loose object '24bbb41919ef906db6f40c3c80f246b7e0cdf9f4'\nerror: 24bbb41919ef906db6f40c3c80f246b7e0cdf9f4: object corrupt or missing\nerror: corrupt loose object '680aba836639d14a36b81f9e29da52bd9af69770'\nerror: 680aba836639d14a36b81f9e29da52bd9af69770: object corrupt or missing\nerror: 8de8382bdc3cc5f936670df414b33ee63927f3a4: object corrupt or missing\nerror: corrupt loose object 'd186d23e87ddf344acc56f48e3cf2f61c7a47e16'\nerror: d186d23e87ddf344acc56f48e3cf2f61c7a47e16: object corrupt or missing\nmissing blob 680aba836639d14a36b81f9e29da52bd9af69770\nmissing blob 8de8382bdc3cc5f936670df414b33ee63927f3a4\nmissing blob d186d23e87ddf344acc56f48e3cf2f61c7a47e16\nmissing blob 24bbb41919ef906db6f40c3c80f246b7e0cdf9f4\n\nAnd since these were all blobs, I looked for em with git-log:\n[sewall@hobo SWIFT++_1.2 (master)]$ git log --raw --all | grep 24bbb*\n[sewall@hobo SWIFT++_1.2 (master)]$ git log --raw --all | grep d186*\n[sewall@hobo SWIFT++_1.2 (master)]$ git log --raw --all | grep 8de83*\n[sewall@hobo SWIFT++_1.2 (master)]$ git log --raw --all | grep 680ab*\n\nSo it would assume that these are dangling objects. Given the simple\nhistory of the repo, the only think I can think of that could have\ncaused it would be that my first commit was of the original files, and\nI immediately remembered that I wanted to run dos2unix on the files so\nI did and did a commit -a --amend.\n\nI untarred the original snapshot again and ran git-hash-object over\nall the files, and sure enough each of those missing hashes\ncorresponds to a blob in the original (non-dos2unix'd) file.\n\nI copied the corrupted repo, manually deleted those objects, then\nhash-object -w the appropriate originals back in. git-fsck was\nmollified.\n\nFinally, I tried repeating the process and the same error occurs.\nIn summary:\nunzip original.zip\ngit init\ngit add .\ngit commit -m \"Initial commit\"\nfind . -type f -exec dos2unix {} +\ngit commit -a --amend\n[accept same message]\ngit fsck\n[same error message as above]\n\nI just built my git today, unmodified:\ngit version 1.5.3.5.721.g039b\n\nWhat's going on?\n\nJason\n\nP.S. These files aren't particularly secret, so I'm happy to post them\nif need be. Its actually someone else's code (a fellow student who\ngraduated some time before I arrived in grad school); I was asked to\nmodernize it enough to compile on on the latest gcc.\n"},{"id":"60028","messageId":"7vejer41ib.fsf@gitster.siamese.dyndns.org","threadId":"10891","inReplyTo":"31e9dd080711151350u6c2ae40foc7c05e59496260fa@mail.gmail.com","subject":"Re: New repo quickly corrupted","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2007-11-15T21:55:40Z","receivedAt":"2007-11-15T21:55:40Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Jason Sewall\" <jasonsewall@gmail.com> writes:\n\n> find . -type f -exec dos2unix {} +\n> git commit -a --amend\n> [accept same message]\n> git fsck\n> [same error message as above]\n>\n> I just built my git today, unmodified:\n> git version 1.5.3.5.721.g039b\n>\n> What's going on?\n\nCarelessness is what is going on.\n\nNotice how nicely your \"find . -type f -exec dos2unix {}\" goes\ndown to .git/objects and eats your loose objects\n\nYou probably wanted to do\n\n\tgit ls-files | xargs dos2unix\n\ninstead.\n"},{"id":"60030","messageId":"31e9dd080711151359y1504e58cp994e828d44660c7@mail.gmail.com","threadId":"10891","inReplyTo":"7vejer41ib.fsf@gitster.siamese.dyndns.org","subject":"Re: New repo quickly corrupted","fromName":"Jason Sewall","fromEmail":"jasonsewall@gmail.com","sentAt":"2007-11-15T21:59:33Z","receivedAt":"2007-11-15T21:59:33Z","isPatch":false,"sender":{"key":"jasonsewall@gmail.com","avatar":null},"body":"On Nov 15, 2007 4:55 PM, Junio C Hamano <gitster@pobox.com> wrote:\n> \"Jason Sewall\" <jasonsewall@gmail.com> writes:\n>\n> > find . -type f -exec dos2unix {} +\n> > git commit -a --amend\n> > [accept same message]\n> > git fsck\n> > [same error message as above]\n> >\n> > I just built my git today, unmodified:\n> > git version 1.5.3.5.721.g039b\n> >\n> > What's going on?\n>\n> Carelessness is what is going on.\n>\n> Notice how nicely your \"find . -type f -exec dos2unix {}\" goes\n> down to .git/objects and eats your loose objects\n>\n> You probably wanted to do\n>\n>         git ls-files | xargs dos2unix\n>\n> instead.\n>\n\nSure enough. I'll chalk that one up to too many mind-numbing meetings today.\n\nThanks for the wake-up...\n"},{"id":"60032","messageId":"alpine.LFD.0.9999.0711151358180.4260@woody.linux-foundation.org","threadId":"10891","inReplyTo":"31e9dd080711151350u6c2ae40foc7c05e59496260fa@mail.gmail.com","subject":"Re: New repo quickly corrupted","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2007-11-15T22:01:26Z","receivedAt":"2007-11-15T22:01:26Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Thu, 15 Nov 2007, Jason Sewall wrote:\n>\n> find . -type f -exec dos2unix {} +\n\nOops.\n\nYou just ran \"dos2unix\" on all the *git* files!\n\nWhich certainly explains why all your old objects got corrupted!\n\n> What's going on?\n\nPEBKAC ;)\n\nTo avoid this error in the future, may I suggest using\n\n\tgit ls-files -z | xargs -0 dos2unix\n\nor similar. Please DO NOT write to the .git/objects directory.\n\n\t\tLinus\n"},{"id":"60033","messageId":"31e9dd080711151406u6df05debs56ac473977dfede3@mail.gmail.com","threadId":"10891","inReplyTo":"alpine.LFD.0.9999.0711151358180.4260@woody.linux-foundation.org","subject":"Re: New repo quickly corrupted","fromName":"Jason Sewall","fromEmail":"jasonsewall@gmail.com","sentAt":"2007-11-15T22:06:14Z","receivedAt":"2007-11-15T22:06:14Z","isPatch":false,"sender":{"key":"jasonsewall@gmail.com","avatar":null},"body":"On Nov 15, 2007 5:01 PM, Linus Torvalds <torvalds@linux-foundation.org> wrote:\n>\n>\n> On Thu, 15 Nov 2007, Jason Sewall wrote:\n> >\n> > find . -type f -exec dos2unix {} +\n>\n> Oops.\n>\n> You just ran \"dos2unix\" on all the *git* files!\n>\n> Which certainly explains why all your old objects got corrupted!\n>\n> > What's going on?\n>\n> PEBKAC ;)\n\nYeah, Junio beat you to it. :)\n\nI take comfort in the fact that others have suffered far worse\nconsequences from find misuse....\n"},{"id":"60034","messageId":"alpine.LFD.0.9999.0711151708470.21255@xanadu.home","threadId":"10891","inReplyTo":"alpine.LFD.0.9999.0711151358180.4260@woody.linux-foundation.org","subject":"Re: New repo quickly corrupted","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2007-11-15T22:12:23Z","receivedAt":"2007-11-15T22:12:23Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Thu, 15 Nov 2007, Linus Torvalds wrote:\n\n> \n> \n> On Thu, 15 Nov 2007, Jason Sewall wrote:\n> >\n> > find . -type f -exec dos2unix {} +\n> \n> Oops.\n> \n> You just ran \"dos2unix\" on all the *git* files!\n\nDoes \"dos2unix\" override file access bits?  Because the object store is \nalways made read-only.\n\n\nNicolas\n"},{"id":"60041","messageId":"alpine.LFD.0.9999.0711151434290.4260@woody.linux-foundation.org","threadId":"10891","inReplyTo":"alpine.LFD.0.9999.0711151708470.21255@xanadu.home","subject":"Re: New repo quickly corrupted","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2007-11-15T22:37:29Z","receivedAt":"2007-11-15T22:37:29Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Thu, 15 Nov 2007, Nicolas Pitre wrote:\n> \n> Does \"dos2unix\" override file access bits?  Because the object store is \n> always made read-only.\n\nAlmost all programs like that will entirely ignor the fact that something \nis read-only.\n\nWhy? Becuase you end up having to create a new file *anyway*. So nobody \ndoes modify-in-place, they literally end up doing\n\n - create temp-file\n - while (data) \n\tread old file, write to tempfile\n - rename temp-file over oldfile\n\nand unless you *explicitly* look at the permission bits you'll never even \nnotice that the old file was read-only, because none of the steps above \ncare at all!\n\nSo marking things read-only will give only limited protection. It will \nprotect against most editors, and will protect against things that change \nfiles in-place and literally try to open the original file as read-write, \nbut not much else.\n\n\t\t\tLinus\n"},{"id":"60055","messageId":"200711160645.04352.chriscool@tuxfamily.org","threadId":"10891","inReplyTo":"alpine.LFD.0.9999.0711151434290.4260@woody.linux-foundation.org","subject":"Re: New repo quickly corrupted","fromName":"Christian Couder","fromEmail":"chriscool@tuxfamily.org","sentAt":"2007-11-16T05:45:03Z","receivedAt":"2007-11-16T05:45:03Z","isPatch":false,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Le jeudi 15 novembre 2007, Linus Torvalds a écrit :\n> On Thu, 15 Nov 2007, Nicolas Pitre wrote:\n> > Does \"dos2unix\" override file access bits?  Because the object store is\n> > always made read-only.\n>\n> Almost all programs like that will entirely ignor the fact that something\n> is read-only.\n\nWhat if the .git/objects/ sudirectories were also read-only ?\n\nChristian.\n"},{"id":"60062","messageId":"473D4827.1060109@op5.se","threadId":"10891","inReplyTo":"200711160645.04352.chriscool@tuxfamily.org","subject":"Re: New repo quickly corrupted","fromName":"Andreas Ericsson","fromEmail":"ae@op5.se","sentAt":"2007-11-16T07:35:03Z","receivedAt":"2007-11-16T07:35:03Z","isPatch":false,"sender":{"key":"ae@op5.se","avatar":"https://gravatar.com/avatar/426e89595c75a8f5252dd0c989e5fabe5bcac616e68557427ad9aef6b0ca342a?d=mp&s=160"},"body":"Christian Couder wrote:\n> Le jeudi 15 novembre 2007, Linus Torvalds a écrit :\n>> On Thu, 15 Nov 2007, Nicolas Pitre wrote:\n>>> Does \"dos2unix\" override file access bits?  Because the object store is\n>>> always made read-only.\n>> Almost all programs like that will entirely ignor the fact that something\n>> is read-only.\n> \n> What if the .git/objects/ sudirectories were also read-only ?\n> \n\nThen git wouldn't be able to write to it without chmod()'ing it each time.\n\n-- \nAndreas Ericsson                   andreas.ericsson@op5.se\nOP5 AB                             www.op5.se\nTel: +46 8-230225                  Fax: +46 8-230231\n"},{"id":"60134","messageId":"200711171353.45310.chriscool@tuxfamily.org","threadId":"10891","inReplyTo":"473D4827.1060109@op5.se","subject":"Re: New repo quickly corrupted","fromName":"Christian Couder","fromEmail":"chriscool@tuxfamily.org","sentAt":"2007-11-17T12:53:45Z","receivedAt":"2007-11-17T12:53:45Z","isPatch":false,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"Le vendredi 16 novembre 2007, Andreas Ericsson a écrit :\n> Christian Couder wrote:\n> > Le jeudi 15 novembre 2007, Linus Torvalds a écrit :\n> >> On Thu, 15 Nov 2007, Nicolas Pitre wrote:\n> >>> Does \"dos2unix\" override file access bits?  Because the object store\n> >>> is always made read-only.\n> >>\n> >> Almost all programs like that will entirely ignor the fact that\n> >> something is read-only.\n> >\n> > What if the .git/objects/ sudirectories were also read-only ?\n>\n> Then git wouldn't be able to write to it without chmod()'ing it each\n> time.\n\nYes, but some (not manly enough) people might want the extra safety even if \nit means a performance penalty.\n\nChristian.\n"},{"id":"60143","messageId":"200711171513.38243.robin.rosenberg.lists@dewire.com","threadId":"10891","inReplyTo":"200711171353.45310.chriscool@tuxfamily.org","subject":"Re: New repo quickly corrupted","fromName":"Robin Rosenberg","fromEmail":"robin.rosenberg.lists@dewire.com","sentAt":"2007-11-17T14:13:37Z","receivedAt":"2007-11-17T14:13:37Z","isPatch":false,"sender":{"key":"robin.rosenberg@dewire.com","avatar":"https://avatars.githubusercontent.com/u/46357?v=4"},"body":"lördag 17 november 2007 skrev Christian Couder:\n> Le vendredi 16 novembre 2007, Andreas Ericsson a écrit :\n> > Christian Couder wrote:\n> > > Le jeudi 15 novembre 2007, Linus Torvalds a écrit :\n> > >> On Thu, 15 Nov 2007, Nicolas Pitre wrote:\n> > >>> Does \"dos2unix\" override file access bits?  Because the object store\n> > >>> is always made read-only.\n> > >>\n> > >> Almost all programs like that will entirely ignor the fact that\n> > >> something is read-only.\n> > >\n> > > What if the .git/objects/ sudirectories were also read-only ?\n> >\n> > Then git wouldn't be able to write to it without chmod()'ing it each\n> > time.\n> \n> Yes, but some (not manly enough) people might want the extra safety even if \n> it means a performance penalty.\n\nThose do manly enough to do run find . -exec  ;)\n\n-- robin\n"}]}