{"thread":{"id":"10575","subject":"Debugging corrupt object generation","startedAt":"2007-11-01T22:57:26Z","lastAt":"2007-11-02T05:15:55Z","messageCount":4,"participants":["Julian Phillips","Nicolas Pitre","Shawn O. Pearce"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"57896","messageId":"Pine.LNX.4.64.0711012256080.17812@beast.quantumfyre.co.uk","threadId":"10575","inReplyTo":null,"subject":"Debugging corrupt object generation","fromName":"Julian Phillips","fromEmail":"julian@quantumfyre.co.uk","sentAt":"2007-11-01T22:57:26Z","receivedAt":"2007-11-01T22:57:26Z","isPatch":false,"sender":{"key":"julian@quantumfyre.co.uk","avatar":"https://avatars.githubusercontent.com/u/948888?v=4"},"body":"I have been trying to import a repository using fast-import, and as part \nof this I have extended fast-import to support copying a path from any \narbitrary commit. This works fine on my small test repository.\n\nHowever, when I run against my (vastly more complicated) real work \nrepository things don't go so well. I get a few thousand commits in, and \nthen it breaks. It appears that I have somehow managed to create a \ncorrupt object or something.\n\nAfter a certain commit, and attempt to use that commit generates a \"fatal:\nunable to apply delta\". This appears to be coming from unpack_delta_entry\nin sha1_file.c.\n\nCan anyone give me any hints as to how I find out what is causing the\nproblem? I'm not even sure what it is that isn't working ... and all\nattempts to replicate the problem with my test repository have failed.\n\n-- \nJulian\n\n ---\n...the prevailing Catholic odor - incense, wax, centuries of mild bleating\nfrom the lips of the flock.\n-- Thomas Pynchon, _Gravity's Rainbow_\n"},{"id":"57901","messageId":"alpine.LFD.0.9999.0711011947220.21255@xanadu.home","threadId":"10575","inReplyTo":"Pine.LNX.4.64.0711012256080.17812@beast.quantumfyre.co.uk","subject":"Re: Debugging corrupt object generation","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2007-11-01T23:57:00Z","receivedAt":"2007-11-01T23:57:00Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Thu, 1 Nov 2007, Julian Phillips wrote:\n\n> I have been trying to import a repository using fast-import, and as part of\n> this I have extended fast-import to support copying a path from any arbitrary\n> commit. This works fine on my small test repository.\n> \n> However, when I run against my (vastly more complicated) real work repository\n> things don't go so well. I get a few thousand commits in, and then it breaks.\n> It appears that I have somehow managed to create a corrupt object or\n> something.\n> \n> After a certain commit, and attempt to use that commit generates a \"fatal:\n> unable to apply delta\". This appears to be coming from unpack_delta_entry\n> in sha1_file.c.\n\nI suppose you mean \"fatal: failed to apply delta\", because \"unable to \napply delta\" doesn't appear anywhere in the current source tree.\n\n> Can anyone give me any hints as to how I find out what is causing the\n> problem? I'm not even sure what it is that isn't working ... and all\n> attempts to replicate the problem with my test repository have failed.\n\nWell, something is screwed for sure. Some object you're requesting is \nmade of a delta, and that delta is bad, therefore patch_delta() returns \nNULL (you could instrument it to determine exactly why).\n\nMaybe fast-import hasn't flushed the needed data to the pack yet?\n\n\nNicolas\n"},{"id":"57905","messageId":"Pine.LNX.4.64.0711020018400.18429@beast.quantumfyre.co.uk","threadId":"10575","inReplyTo":"alpine.LFD.0.9999.0711011947220.21255@xanadu.home","subject":"Re: Debugging corrupt object generation","fromName":"Julian Phillips","fromEmail":"julian@quantumfyre.co.uk","sentAt":"2007-11-02T00:26:34Z","receivedAt":"2007-11-02T00:26:34Z","isPatch":false,"sender":{"key":"julian@quantumfyre.co.uk","avatar":"https://avatars.githubusercontent.com/u/948888?v=4"},"body":"On Thu, 1 Nov 2007, Nicolas Pitre wrote:\n\n> On Thu, 1 Nov 2007, Julian Phillips wrote:\n>\n>> After a certain commit, and attempt to use that commit generates a \"fatal:\n>> unable to apply delta\". This appears to be coming from unpack_delta_entry\n>> in sha1_file.c.\n>\n> I suppose you mean \"fatal: failed to apply delta\", because \"unable to\n> apply delta\" doesn't appear anywhere in the current source tree.\n\nI did indeed. Read 'n' type never seems to be as reliable as copy 'n' \npaste ...\n\n>> Can anyone give me any hints as to how I find out what is causing the\n>> problem? I'm not even sure what it is that isn't working ... and all\n>> attempts to replicate the problem with my test repository have failed.\n>\n> Well, something is screwed for sure. Some object you're requesting is\n> made of a delta, and that delta is bad, therefore patch_delta() returns\n> NULL (you could instrument it to determine exactly why).\n\nAh, ok - that sounds like it might be useful. I guess that looking at the \nopposite side where fast-import is creating the data might also prove \nfruitful.\n\n> Maybe fast-import hasn't flushed the needed data to the pack yet?\n\nWell, fast-import completes quite happily and outputs the normal summary \nstatus. I can look at logs and trees etc provided that I don't try and \nlook at one particular part of the tree on one particular commit. I \nthink the problem is that I've managed to do something inside fast-import \nthat corrupts one particular tree object (though I've no idea what that \nmight be).\n\nI shall have to start instrumenting and try again.\n\nThanks.\n\n-- \nJulian\n\n ---\nMany changes of mind and mood; do not hesitate too long.\n"},{"id":"57927","messageId":"20071102051555.GB14735@spearce.org","threadId":"10575","inReplyTo":"Pine.LNX.4.64.0711020018400.18429@beast.quantumfyre.co.uk","subject":"Re: Debugging corrupt object generation","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2007-11-02T05:15:55Z","receivedAt":"2007-11-02T05:15:55Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Julian Phillips wrote:\n> On Thu, 1 Nov 2007, Nicolas Pitre wrote:\n> >\n> >Maybe fast-import hasn't flushed the needed data to the pack yet?\n> \n> Well, fast-import completes quite happily and outputs the normal summary \n> status. I can look at logs and trees etc provided that I don't try and \n> look at one particular part of the tree on one particular commit. I \n> think the problem is that I've managed to do something inside fast-import \n> that corrupts one particular tree object (though I've no idea what that \n> might be).\n\nAhhh. I'm betting you messed up the version 0 and version 1 arrays\ninside of the struct tree_entry. This could cause the delta\ngenerator to look at the wrong base information when it creates\nthe delta, thus causing the delta to be created for a different\nbase than what the object is actually using in the packfile.\n\nThe version[0] is meant to hold the mode and SHA-1 of the tree_entry\nin the base object. The version[1] is meant to hold the current\nmode and SHA-1 of the tree_entry in the new object. A mode of 0\nmeans the entry doesn't exist in that particular tree; so an add\nis shown as \"version[0].mode = 0; version[1].mode = 0100644\".\n\nLook at the store_tree function, this is where we regenerate the\ncanonical representation of both the version[0] and the version[1]\ntrees so the store_object function can generate a delta. Note that\nwe assume the base object name is root->versions[0].sha1. Maybe the\nversion[0] array doesn't actually match the tree named by that sha1?\n\n-- \nShawn.\n"}]}