{"thread":{"id":"5438","subject":"Re: Problematic git pack","startedAt":"2006-08-31T08:45:12Z","lastAt":"2006-08-31T21:33:06Z","messageCount":4,"participants":["Sergio Callegari","Johannes Schindelin","Nicolas Pitre","Linus Torvalds"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"26129","messageId":"44F6A198.4040902@arces.unibo.it","threadId":"5438","inReplyTo":null,"subject":"Re: Problematic git pack","fromName":"Sergio Callegari","fromEmail":"scallegari@arces.unibo.it","sentAt":"2006-08-31T08:45:12Z","receivedAt":"2006-08-31T08:45:12Z","isPatch":false,"sender":{"key":"scallegari@arces.unibo.it","avatar":null},"body":"What can I say... I had never seen before such an action at such a rapid \npace following the indication of a potential problem.\nThanks Linus and Junio and everybody who might have contributed.\n>   Junio could then generate a new pack with the one corrupted object \n>   fixed, which obviously meant that all the deltas now worked too.\n>   \nExcellent news...\n>   This is my (probably final) analysis of the resulting differences.. ]\n>\n> On Wed, 30 Aug 2006, Junio C Hamano wrote:\n> > \n> > Ok, I was going to attach the resurrected pack that should\n> > contain everything your corrupt pack had, but it is a bit too\n> > large, so I'll place it here [*1*].  Drop me a note when you\n> > retrieved it, so that I can remove it.\n>   \nJunio, can you please send me privately details about [*1*] so I can \nretrieve the pack also?\n\nI also have another question... (maybe it was answered in some previous \nthread on this list, in this case a pointer would be enough).\nNow I am going to have the fixed archive and also a new archive, which I \nrestarted from the latest working copy I had of my project.\nIs there any way to automatically do real \"surgery\" to attach one to the \nother and get a single archive with all the history?\nObviously, if I try to change a commit object to modify its parents, its \nsignature changes, so I need to modify its childs and so on, is this \ncorrect?\nAlternatively I belive that grafts should be a way to go... I had never \nused them before, do all git tools support them? Particularly do they \nget pushed and pulled correctly?\n> So the _real_ difference is literally just the one byte at offset 0151000 \n> (decimal 53760) which in the fixed pack is 0x96, and in the corrupt pack \n> it is 0x94. That's a single-bit difference (bit #1 has been cleared).\n>\n>   \nSo, possibly, the alpha particle theory could be the plausible one in \nthe end...\n> Now, that makes me feel happy on one level, because it's almost certainly \n> a hardware problem - subtle memory corruption, or disk corruption that \n> happened when either reading or writing the image. Sergio may not be that \n> happy about it, of course.\n>   \nThe bad thing is that I don't know which of my two machines (the laptop \nor the desktop) caused the issue!\n\n> Finally, this also points out that the corrupted packs _can_ be fixed, but \n> I think Sergio was a bit lucky (to offset all the bad luck). Sergio still \n> had access to the original file that had had its object corrupted. \nActually, this could possibly be a not so rare case... In my tree I had \nthe development of some LaTeX documents and packages (code like, the \nreally \"precious\" files) and a few binary objects (images and openoffice \nfiles mainly, by far less precious).\nSince the binary objects were so much overwhelming in size with regard \nto the text ones, assuming a single error the probability of having it \nin a non-code object was much larger than that of having it in a \nprecious code object. Also commit and tree objects should be much \nsmaller than data objects.\nThis assumption is the reason which initally pushed me to ask help to \ntry to unpack at least all the correct objects (one of my first \nquestions was: does git unpack-objects die on the first error or is \nthere a way to convince it to simply skip the wrong object (or the delta \nagainst a wrong object)...\nIf git unpack-objects can gain an option like --continue-on-errors and \nif checkout/reset can also get an option to do the same (i.e. in a tree \nwith missing objects, checkout all that can be found), I believe that \none is at a good point already...\nFinally, having a command to create an object out of a single file \n(contrary of git cat-file) could help re-creating the missing objects...\n> And it \n> took a fair amount of work, and some git hacking by somebody who really \n> understood git (Junio).\n>\n> Maybe we'll end up having some of that effort being useful and checked in, \n> and we'll eventually have more infrastructure for fixing these things, but \n> I suspect that in most cases, even a _single_ bit of corruption will \n> generally result in so much havoc that nobody should depend on that. It's \n> a lot better to have backups.\n>\n> \t\t\tLinus\n"},{"id":"26133","messageId":"Pine.LNX.4.63.0608311301260.28360@wbgn013.biozentrum.uni-wuerzburg.de","threadId":"5438","inReplyTo":"44F6A198.4040902@arces.unibo.it","subject":"Re: Problematic git pack","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2006-08-31T11:15:07Z","receivedAt":"2006-08-31T11:15:07Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Thu, 31 Aug 2006, Sergio Callegari wrote:\n\n> Now I am going to have the fixed archive and also a new archive, which I\n> restarted from the latest working copy I had of my project.\n> Is there any way to automatically do real \"surgery\" to attach one to the other\n> and get a single archive with all the history?\n\nYou can \"graft\" the new onto the old branch:\n\nIf <40-hex-chars-old> is the commit id of the youngest commit of the \nreconstructed branch, and <40-hex-chars-new> is the commit id of the \ninitial commit of the newly started branch, you can put this line into \n.git/info/grafts:\n\n<40-hex-chars-new> <40-hex-chars-old>\n\nThis will make git believe that the initial commit is no initial commit, \nbut has the old head as single parent. And yes, AFAICT all git tools \nsupport this. I used this technique many times to be able to merge \nunrelated developments.\n\nNOTE! This is the quickest way if you want to have the history _locally_.\n\nIf you want to be able to distribute it (or synchronize it between your \nlaptop and PC _with git!_), you can rewrite the history by either \ngit-rebase, or by using cg-admin-rewritehist if you are using cogito.\n\nUnfortunately, I do not use cogito nor git-rebase, so if you want to walk \nthat path, others have to help. (And most likely, we'd put the result into \nDocumentation/howto/.)\n\nCiao,\nDscho\n\nP.S.: Of course, if you do not insist on a super clean history, you can \nfake a merge. Just put <40-hex-chars-old> into .git/MERGE_HEAD and commit. \nThis will pretend that your new head and your old head were merged, and \nthe result is the new head. This _should_ even work with git-bisect, but \nit is slightly ugly.\n"},{"id":"26151","messageId":"Pine.LNX.4.64.0608311216480.9796@xanadu.home","threadId":"5438","inReplyTo":"44F6A198.4040902@arces.unibo.it","subject":"Re: Problematic git pack","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2006-08-31T16:23:07Z","receivedAt":"2006-08-31T16:23:07Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Thu, 31 Aug 2006, Sergio Callegari wrote:\n\n> The bad thing is that I don't know which of my two machines (the laptop or the\n> desktop) caused the issue!\n\nmemtest86 is your friend: http://www.memtest.org\n\n\nNicolas\n"},{"id":"26160","messageId":"Pine.LNX.4.64.0608311416060.27779@g5.osdl.org","threadId":"5438","inReplyTo":"44F6A198.4040902@arces.unibo.it","subject":"Re: Problematic git pack","fromName":"Linus Torvalds","fromEmail":"torvalds@osdl.org","sentAt":"2006-08-31T21:33:06Z","receivedAt":"2006-08-31T21:33:06Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Thu, 31 Aug 2006, Sergio Callegari wrote:\n>\n> Junio, can you please send me privately details about [*1*] so I can retrieve\n> the pack also?\n\nHe already did, search for \"members.cox.net\" in your email archive (it's \nMessage-ID: <7v7j0qihwl.fsf@assigned-by-dhcp.cox.net> to be precise).\n\n> I also have another question... (maybe it was answered in some previous thread\n> on this list, in this case a pointer would be enough).\n> Now I am going to have the fixed archive and also a new archive, which I\n> restarted from the latest working copy I had of my project.\n> Is there any way to automatically do real \"surgery\" to attach one to the other\n> and get a single archive with all the history?\n\nYes. This is just what a \"grafts\" file is for.\n\nPut the old pack/idx files into the .git/objects/packs directory, and then \nyou can create \"fake parenthood\" information in a \".git/info/grafts\" file \nby just adding text-lines of the format \"<sha1> <fakeparentsha1>\" (with \neach SHA being the regular 40-byte hex representation).\n\n> Obviously, if I try to change a commit object to modify its parents, its\n> signature changes, so I need to modify its childs and so on, is this correct?\n> Alternatively I belive that grafts should be a way to go... I had never used\n> them before, do all git tools support them? Particularly do they get pushed\n> and pulled correctly?\n\nNope, they won't get pushed and pulled correctly, you need to put the \ngrafts files in all repositories. Alternatively, you can re-create the \nwhole history, I think cogito had some history re-writing tool.\n\n> > So the _real_ difference is literally just the one byte at offset 0151000\n> > (decimal 53760) which in the fixed pack is 0x96, and in the corrupt pack it\n> > is 0x94. That's a single-bit difference (bit #1 has been cleared).\n> \n> So, possibly, the alpha particle theory could be the plausible one in the\n> end...\n\nYes. It's just that Junio's original theory required it to not just hit a \nmemory cell, it also had to hit it at _just_ the right time in between \nbeing written and the SHA1 of the buffer being computed. So the original \ntheory was very unlikely indeed.\n\nMy theory of the corruption just causing a re-computed SHA1 when repacking \n(and silently copying the corruption without realizing it) meant that \nthere was no such small and unlikely window, but that any regular memory \n(or disk) corruption could easily have caused it at any time, and then a \nsubsequent re-pack \"fixed\" the SHA1 to match the corruption..\n\n> The bad thing is that I don't know which of my two machines (the laptop or the\n> desktop) caused the issue!\n\nI'd suggest running memtest86 for a few days on both (not necessarily at \nthe same time - keep one working machine to do you job on ;)\n\n> > Finally, this also points out that the corrupted packs _can_ be fixed, but I\n> > think Sergio was a bit lucky (to offset all the bad luck). Sergio still had\n> > access to the original file that had had its object corrupted. \n>\n> Actually, this could possibly be a not so rare case... In my tree I had the\n> development of some LaTeX documents and packages (code like, the really\n> \"precious\" files) and a few binary objects (images and openoffice files\n> mainly, by far less precious).\n\nSure. In your case you had checked in generated files too, and yes, they \nwere the larger ones. That's not true in general - in many other projects, \nthe _directory_ structure (ie the git \"tree\" objects) will be a large \nportion of the project, and probably more likely to be corrupt. Now, to \nsome degree the tree objects are likely the ones easiest to \"repair\" \n(because you can try to look at the history and figure things out by \nhand), but at the same time, people also tend to have deeper delta-chains \nand it would just be _very_ painful.\n\nSo I do think you were somewhat lucky.\n\n> Finally, having a command to create an object out of a single file (contrary\n> of git cat-file) could help re-creating the missing objects...\n\nHmm. Like \"git-hash-object\"?\n\n\t\t\tLinus\n"}]}