{"thread":{"id":"5381","subject":"Re: Problem with pack","startedAt":"2006-08-25T10:07:30Z","lastAt":"2006-08-26T10:31:09Z","messageCount":6,"participants":["Sergio Callegari","Andreas Ericsson","Jakub Narebski","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"25884","messageId":"44EECBE2.7090801@arces.unibo.it","threadId":"5381","inReplyTo":null,"subject":"Re: Problem with pack","fromName":"Sergio Callegari","fromEmail":"scallegari@arces.unibo.it","sentAt":"2006-08-25T10:07:30Z","receivedAt":"2006-08-25T10:07:30Z","isPatch":false,"sender":{"key":"scallegari@arces.unibo.it","avatar":null},"body":">\n> > git verify-pack -v pack-ebcdfbbda07e5a3e4136aa1f499990b35685bab4.idx\n> > fatal: failed to read delta-pack base object 2849bd2bd8a76bbca37df2a4c8e8b990811d01a7\n>\n> Eeeh! Not good.\n>\n> > 1) I am working on both a pc and a notebook, syncing the two everytime I move\n> > from one to the other.\n>\n> So, you still have one \"good\" version? Please make a backup immediately. \n> (If only to reproduce the problem.)\n>   \nI have a good working tree, but unfortunately I realized that there was \na problem with the pack only _after_ the sync:\nI was not expecting this kind of problem, so I silly did a repack as the \nlast thing, I went home, I attached the laptop to the net, I run unison, \nI started to work and I realized that there was a problem when I \nattempted a new repack which failed complaining about the corrupted pack...\n\nSo actually, I do not even know where the corruption came from (an hd \nerror, the sync tool, ...)\n\nI only have the corrupted pack and its index and a good last working tree.\n\nBTW, it would be nice to have some \"security measure\" in git reset... \ne.g. an option to trigger the following behavior:\n\n- saving all current changes in a temporary commit\n- checking that the current HEAD can be re-checked out before the reset\n\n> Since unpack-objects does not use the index, it cannot extract anything \n> after the first error. We _could_ enhance unpack-objects to be nice and \n> optionally take a pack-index to try to reconstruct as many objects as \n> possible.\n>   \nThat would be very useful...\nBtw, even without that, if I understand correctly, git packs are \ncollections of compressed objects, each of which has its own header \nstating how long is the compressed object itself. In my case, the error \nis in inflating one object (git unpack-objects says inflate returns \n-3)... so shouldn't there be a way to try to skip to the next object \neven in this case?\n> BTW I'd recommend not syncing with unison, but with the git transports: If \n> your PC and Laptop are connected, you could do something like\n>\n> \tgit pull laptop:my_project/.git\n>   \nActually, the project, including the git archive gets syncronized as a \npart of a syncronization process including all my Documents directory \n(the project is in fact a LaTeX manual with somehow complex LaTeX \npackages and classes). Syncronizing in this way actually worked very \nwell so far, because at once I was getting in sync all my working trees \nand all my repos...\n\nSergio\n"},{"id":"25885","messageId":"44EECEDC.7090608@op5.se","threadId":"5381","inReplyTo":"44EECBE2.7090801@arces.unibo.it","subject":"Re: Problem with pack","fromName":"Andreas Ericsson","fromEmail":"ae@op5.se","sentAt":"2006-08-25T10:20:12Z","receivedAt":"2006-08-25T10:20:12Z","isPatch":false,"sender":{"key":"ae@op5.se","avatar":"https://gravatar.com/avatar/426e89595c75a8f5252dd0c989e5fabe5bcac616e68557427ad9aef6b0ca342a?d=mp&s=160"},"body":"Sergio Callegari wrote:\n>>\n>> > git verify-pack -v pack-ebcdfbbda07e5a3e4136aa1f499990b35685bab4.idx\n>> > fatal: failed to read delta-pack base object \n>> 2849bd2bd8a76bbca37df2a4c8e8b990811d01a7\n>>\n>> Eeeh! Not good.\n>>\n>> > 1) I am working on both a pc and a notebook, syncing the two \n>> everytime I move\n>> > from one to the other.\n>>\n>> So, you still have one \"good\" version? Please make a backup \n>> immediately. (If only to reproduce the problem.)\n>>   \n> I have a good working tree, but unfortunately I realized that there was \n> a problem with the pack only _after_ the sync:\n> I was not expecting this kind of problem, so I silly did a repack as the \n> last thing, I went home, I attached the laptop to the net, I run unison, \n> I started to work and I realized that there was a problem when I \n> attempted a new repack which failed complaining about the corrupted pack...\n> \n> So actually, I do not even know where the corruption came from (an hd \n> error, the sync tool, ...)\n> \n> I only have the corrupted pack and its index and a good last working tree.\n> \n> BTW, it would be nice to have some \"security measure\" in git reset... \n> e.g. an option to trigger the following behavior:\n> \n> - saving all current changes in a temporary commit\n> - checking that the current HEAD can be re-checked out before the reset\n> \n\nThe recommended way is to do a throw-away branch to commit your \ntemporary commit to (or the 'master' so long as you remember to use \nreset). The current HEAD can always, barring object database errors, be \nchecked out if 'git status' reports no changes in the working tree. \nUnfortunately, the object database is often enormous, so doing a full \nfsck-objects before each change to the branch you're on (which is \nbasically what a reset is), would take far too long to be viable.\n\n>> Since unpack-objects does not use the index, it cannot extract \n>> anything after the first error. We _could_ enhance unpack-objects to \n>> be nice and optionally take a pack-index to try to reconstruct as many \n>> objects as possible.\n>>   \n> That would be very useful...\n> Btw, even without that, if I understand correctly, git packs are \n> collections of compressed objects, each of which has its own header \n> stating how long is the compressed object itself. In my case, the error \n> is in inflating one object (git unpack-objects says inflate returns \n> -3)... so shouldn't there be a way to try to skip to the next object \n> even in this case?\n\nIt should be possible, assuming the pack index is still intact. The pack \nindex is where the headers are stored, afaik.\n\n\n>> BTW I'd recommend not syncing with unison, but with the git \n>> transports: If your PC and Laptop are connected, you could do \n>> something like\n>>\n>>     git pull laptop:my_project/.git\n>>   \n> Actually, the project, including the git archive gets syncronized as a \n> part of a syncronization process including all my Documents directory \n> (the project is in fact a LaTeX manual with somehow complex LaTeX \n> packages and classes). Syncronizing in this way actually worked very \n> well so far, because at once I was getting in sync all my working trees \n> and all my repos...\n> \n\nThe largest benefit of using git's synchronization methods is that you \nimmediately get a pack-file verification, and also that you never risk \noverwriting anything in either repo if you've forgotten to sync between \nthe two (say you've made changes on your laptop, forgot to send them to \nyour workstation, then made changes on your workstation and then you try \nto sync them). It's possible to recover from such a situation using the \nlost-found tool, but it can be cumbersome, and uncommitted changes, as \nwell as changes to the working tree, are lost forever.\n\n-- \nAndreas Ericsson                   andreas.ericsson@op5.se\nOP5 AB                             www.op5.se\nTel: +46 8-230225                  Fax: +46 8-230231\n"},{"id":"25886","messageId":"ecmk3o$sbs$1@sea.gmane.org","threadId":"5381","inReplyTo":"44EECEDC.7090608@op5.se","subject":"Re: Problem with pack","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2006-08-25T10:41:00Z","receivedAt":"2006-08-25T10:41:00Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Andreas Ericsson wrote:\n\n>>> BTW I'd recommend not syncing with unison, but with the git \n>>> transports: If your PC and Laptop are connected, you could do \n>>> something like\n>>>\n>>>     git pull laptop:my_project/.git\n>>>   \n>> Actually, the project, including the git archive gets syncronized as a \n>> part of a syncronization process including all my Documents directory \n>> (the project is in fact a LaTeX manual with somehow complex LaTeX \n>> packages and classes). Syncronizing in this way actually worked very \n>> well so far, because at once I was getting in sync all my working trees \n>> and all my repos...\n>> \n> \n> The largest benefit of using git's synchronization methods is that you \n> immediately get a pack-file verification, and also that you never risk \n> overwriting anything in either repo if you've forgotten to sync between \n> the two (say you've made changes on your laptop, forgot to send them to \n> your workstation, then made changes on your workstation and then you try \n> to sync them). It's possible to recover from such a situation using the \n> lost-found tool, but it can be cumbersome, and uncommitted changes, as \n> well as changes to the working tree, are lost forever.\n\nUnison (which if I remember correctly uses rsync, or rsync over ssh) detect\nsuch case and ask user what to do if both sides changed a file (copy from\none side, copy from second side, view diff, merge,...).\n\nBut you can always tell unison to ignore git object database\n  ignore = Name .git\nand perhaps also ignore working directories under git control\n  ignore = Path path/to/working/dir/\n-- \nJakub Narebski\nWarsaw, Poland\nShadeHawk on #git\n"},{"id":"25887","messageId":"7vbqq95bfq.fsf@assigned-by-dhcp.cox.net","threadId":"5381","inReplyTo":"44EECEDC.7090608@op5.se","subject":"Re: Problem with pack","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2006-08-25T10:58:33Z","receivedAt":"2006-08-25T10:58:33Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Andreas Ericsson <ae@op5.se> writes:\n\n>> Btw, even without that, if I understand correctly, git packs are\n>> collections of compressed objects, each of which has its own header\n>> stating how long is the compressed object itself. In my case, the\n>> error is in inflating one object (git unpack-objects says inflate\n>> returns -3)... so shouldn't there be a way to try to skip to the\n>> next object even in this case?\n>\n> It should be possible, assuming the pack index is still intact. The\n> pack index is where the headers are stored, afaik.\n\nThe problem Sergio seems to be having is because somehow he does\nnot have a base object that another object that is in the pack\ndepends on, because the latter object is stored in deltified\nform.\n\nThis should never happen unless .pack itself is corrupted\n(git-pack-objects, unless explicitly told to do so with --thin\nflag to git-rev-list upstream, would not make a delta against\nobjects not in the same pack).\n\nWhen a delta is written to the pack file, unless its base object\nhas already written out, git-pack-objects writes out the base\nobject immediately after that deltified object.  So one\npossibility is that the pack was truncated soon after the delta\nthat is having trouble with finding its base object.  In such a\ncase, the proposed recovery measure of skipping the corruption\nand keep going would not buy you that much.  On the other hand,\nif the corruption is in the middle (e.g. a single disk block was\nwiped out), having .idx file might help you resync.\n\nDoes the pack pass git-verify-pack test, I wonder?\n"},{"id":"25931","messageId":"7vu03z3j1y.fsf@assigned-by-dhcp.cox.net","threadId":"5381","inReplyTo":"44EECBE2.7090801@arces.unibo.it","subject":"Re: Problem with pack","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2006-08-26T10:09:13Z","receivedAt":"2006-08-26T10:09:13Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Sergio Callegari <scallegari@arces.unibo.it> writes:\n\n> I was not expecting this kind of problem, so I silly did a repack as\n> the last thing, I went home, I attached the laptop to the net, I run\n> unison, I started to work and I realized that there was a problem when\n> I attempted a new repack which failed complaining about the corrupted\n> pack...\n\nSorry about the mixed \"intended audience\" of this message,\nasking Sergio for a bit more info as the end user who had\nproblems with git, and at the same time describing the code\nlevel analysis of possible cause to ask for help from git\ndevelopers.  Nico CC'ed because he seems to be the person who\nknows the best around this area including zlib.\n\n- - -\n\nEarlier you said that the mothership has 1.4.2, and the note has\n1.4.0.  The sequence of events as I understand are:\n\n\t- repack -a -d on the mothership with 1.4.2; no problems\n          observed.\n\n        - transfer the results to note; this was done behind git so\n          no problems observed.\n\n        - tried to repack on note with 1.4.0; got \"failed to\n          read delta-pack base object\" error.\n\nCan you make the pack/idx available to the public for\npostmortem?\n\nAlso I wonder if the pack can be read by 1.4.2.\n\nEarlier you said \"unpack-objects <$that-pack.pack\" fails with\n\"error code -3 in inflate...\"  What exact error do you get?\nI am guessing that it is get_data() which says:\n\n\t\"inflate returned %d\\n\"\n\n(side note: we should not say \\n there).\n\n        static void *get_data(unsigned long size)\n        {\n                z_stream stream;\n                void *buf = xmalloc(size);\n\n                memset(&stream, 0, sizeof(stream));\n\n                stream.next_out = buf;\n                stream.avail_out = size;\n                stream.next_in = fill(1);\n                stream.avail_in = len;\n                inflateInit(&stream);\n\n                for (;;) {\n                        int ret = inflate(&stream, 0);\n                        use(len - stream.avail_in);\n                        if (stream.total_out == size && ret == Z_STREAM_END)\n                                break;\n                        if (ret != Z_OK)\n                                die(\"inflate returned %d\\n\", ret);\n                        stream.next_in = fill(1);\n                        stream.avail_in = len;\n                }\n                inflateEnd(&stream);\n                return buf;\n        }\n\nThis pattern appears practically everywhere.  When inflate()\nreturns, we expect its return value to be either Z_OK or\nZ_STREAM_END and everything else is treated as an error.  Also\nwhen we receive Z_STREAM_END the resulting length had better be\nthe size we expect.  I do not have any problem with the latter\nbut have always felt uneasy about the former but being no zlib\nexpert had been using the code as is.  zlib.h says Z_BUF_ERROR\nis not fatal -- it just means this round of call with the given\ninput and output buffer did not make any progress.  I've been\nwondering if it is possible for inflate to eat some input but\nthat was not enough to produce one byte of output, and what\nwould return in such a case...\n\nAbout the error message you got while attempting to repack, the\nexact same error message appears in two places, but the one that\nis emitted is the one in sha1_file.c::unpack_delta_entry().  The\nother one is in unpack-objects.c and is not run by repack.\n\nThis function is called after:\n\n\t- we find that we need to access an object;\n\n\t- we decided to use the .pack/.idx pair; when mmap'ing\n          the .idx file, we validate that the pair is not\n          corrupt by calling check_packed_git_idx().  This does\n          the sha-1 checksum of both files;\n\n\t- we find the location of the deltified object in the\n          .pack file by looking at the corresponding .idx;\n\n\t- we read from that location a handful bytes, find that\n          it is deltified and learn its base object name.\n\nSo it is not likely that the .pack/.idx pair was corrupted after\nit was written (i.e. not a bit rot).\n\nNow, in unpack_delta_entry_function():\n\n\t- we find the location of its base object in the .pack\n          file by looking at the .idx again; if this fails, we\n          would die with a different error message \"failed to\n          find delta-pack base object\";\n\n\t- we call unpack_entry_gently(); we would see the error\n          message you saw only when this function returns NULL.\n\nSo unpack_entry_gently() while reading the base object returned\nNULL.  Let's see how it can:\n\n\t- we read the data for the base object in the pack we\n          identified earlier.  First we learn its type and size.\n\n\t- the base object could be also a deltified object, in\n          which case it would recursively call unpack_delta_entry();\n\t  however, that function would never return NULL.  it\n          either succeeds or die()s.  So the base must not have\n          been OBJ_DELTA type.\n\n        - the object type recorded there for the base object\n          could have been something bogus, in which case we\n          would return NULL.  But that would mean pack-object in\n          1.4.2 generated a bogus pack on the mothership.\n\n\t- if the object type is not bogus, unpack_non_delta_entry()\n          is called to extract the data for the base object.\n          this decompresses the data stream, and if it does\n          not inflate well it would return NULL.\n\nSo there are only a few ways you can get that error message.\n\n\t- pack-objects in 1.4.2 produced an invalid pack by\n          recording:\n\n          - bogus object type for the base object, or\n\n          - incorrectly recording the offset of the base object in\n            the pack file in .idx, or\n\n\t  - incorrectly recording the size of the base object in\n            the pack file; \n \n \t  and checksumed the bogus resulting pack/idx pair as if\n \t  nothing was wrong.\n\n\t- pack-objects in 1.4.2 produced a deflated stream that\n          made unpack_non_delta_entry() unhappy.\n\nOther changes between 1.4.0 and 1.4.2 that I do not think are\nrelated are:\n\n\t- we slightly changed the way data is deflated with\n          commit 12f6c30 to favour speed over compression, but\n          this only affects loose objects and not packs.\n\n\t- we introduced a new file format for loose objects with\n          commit 93821bd, but this needs to be explicitly\n          enabled by .git/config option.  Even if the mothership\n          1.4.2 had recorded loose objects in the new format,\n          the process to create the pack by first expanding and\n          then recompressing with the old-and-proven code, so\n          this should not affect the resulting pack.  Even if\n          such a loose object were copied to the note with 1.4.0\n          together with the pack, object reading code always\n          favours what's in the pack, so it should not even be\n          touched.  Actually, the codepath that emits the error\n          message does not read the base object from anywhere\n          other than from the same pack.\n\n\t- we slightly changed the way data for the object to be\n          deltified and the base object is read in pack-objects\n          with commit 560b25a; I did not see anything obviously\n          wrong with that change.\n\n        - we slightly changed the way we pick the base object\n          when making a delta with commits 8dbbd14 and 51d1e83;\n          these should not change what happens after a pair is\n          decided to be used as delta and its base.\n"},{"id":"25934","messageId":"7vlkpb3i1e.fsf@assigned-by-dhcp.cox.net","threadId":"5381","inReplyTo":"7vu03z3j1y.fsf@assigned-by-dhcp.cox.net","subject":"Re: Problem with pack","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2006-08-26T10:31:09Z","receivedAt":"2006-08-26T10:31:09Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano <junkio@cox.net> writes:\n\n> Earlier you said \"unpack-objects <$that-pack.pack\" fails with\n> \"error code -3 in inflate...\"  What exact error do you get?\n> I am guessing that it is get_data() which says:\n>\n> \t\"inflate returned %d\\n\"\n>\n> (side note: we should not say \\n there).\n> ...\n> This pattern appears practically everywhere...\n> ...  I've been\n> wondering if it is possible for inflate to eat some input but\n> that was not enough to produce one byte of output, and what [it]\n> would return in such a case...\n\nI do not think this fear does not apply to this particular case;\nreturn value -3 is Z_DATA_ERROR, so the deflated stream is\ncorrupt.\n\n> So there are only a few ways you can get that error message.\n> ...\n\nI just realized there is another not so inplausible explanation.\n\nWhen the problematic pack was made on the mothership,\ncsum-file.c::sha1write_compressed() gave the data for the base\nobject to zlib, an alpha particle hit a memory cell that\ncontained zlib output buffer (resulting in a corrupt deflated\nstream in variable \"out\"), and sha1write() wrote it out while\ncomputing the right checksum.\n\nIs the memory on your mothership reliable (I do not want to make\nthis message sound like one on the kernel list, but memtest86\nmight be in order)?\n"}]}