{"thread":{"id":"14891","subject":"How to replace a single corrupt, packed object?","startedAt":"2008-08-08T14:39:18Z","lastAt":"2008-09-15T16:34:12Z","messageCount":18,"participants":["Johannes Schindelin","Shawn O. Pearce","Pieter de Bie","Nicolas Pitre"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"86504","messageId":"20080808143918.GB9152@spearce.org","threadId":"14891","inReplyTo":"alpine.DEB.1.00.0808081639490.24820@pacific.mpi-cbg.de.mpi-cbg.de","subject":"Re: How to replace a single corrupt, packed object?","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2008-08-08T14:39:18Z","receivedAt":"2008-08-08T14:39:18Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Johannes Schindelin wrote:\n> my auto gc kicked in, and shows this:\n> \n> fatal: corrupt packed object for 2c1e128aa51e3a64bd61556c0cd488628b423ccf\n> error: failed to run repack\n> \n> Fortunately, I have the uncorrupted object somewhere else. So I copy the \n> single object as a loose one, and all is fine. Right?\n> \n> Wrong.\n> \n> Repack still picks up the corrupt object instead of the good one. What's \n> the best way out?\n\nUse a 1.6.0 rc? Nico I thought fixed this in 1.6 to automatically\ntry and find another copy of an object if the first copy considered\nfor inclusion into the pack was corrupt.\n\n-- \nShawn.\n"},{"id":"86503","messageId":"alpine.DEB.1.00.0808081639490.24820@pacific.mpi-cbg.de.mpi-cbg.de","threadId":"14891","inReplyTo":null,"subject":"How to replace a single corrupt, packed object?","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2008-08-08T14:41:49Z","receivedAt":"2008-08-08T14:41:49Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nmy auto gc kicked in, and shows this:\n\nfatal: corrupt packed object for 2c1e128aa51e3a64bd61556c0cd488628b423ccf\nerror: failed to run repack\n\nFortunately, I have the uncorrupted object somewhere else. So I copy the \nsingle object as a loose one, and all is fine. Right?\n\nWrong.\n\nRepack still picks up the corrupt object instead of the good one. What's \nthe best way out?\n\nCiao,\nDscho\n"},{"id":"86507","messageId":"alpine.DEB.1.00.0808081647580.24820@pacific.mpi-cbg.de.mpi-cbg.de","threadId":"14891","inReplyTo":"20080808143918.GB9152@spearce.org","subject":"Re: How to replace a single corrupt, packed object?","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2008-08-08T14:50:00Z","receivedAt":"2008-08-08T14:50:00Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Fri, 8 Aug 2008, Shawn O. Pearce wrote:\n\n> Johannes Schindelin wrote:\n> > my auto gc kicked in, and shows this:\n> > \n> > fatal: corrupt packed object for 2c1e128aa51e3a64bd61556c0cd488628b423ccf\n> > error: failed to run repack\n> > \n> > Fortunately, I have the uncorrupted object somewhere else. So I copy the \n> > single object as a loose one, and all is fine. Right?\n> > \n> > Wrong.\n> > \n> > Repack still picks up the corrupt object instead of the good one. What's \n> > the best way out?\n> \n> Use a 1.6.0 rc? Nico I thought fixed this in 1.6 to automatically\n> try and find another copy of an object if the first copy considered\n> for inclusion into the pack was corrupt.\n\nThanks for the quick reply, but no joy (You should know me better than \nthat ;-) :\n\n$ git --version\ngit version 1.6.0.rc1.112.gebbe4\n\nCiao,\nDscho\n"},{"id":"86512","messageId":"alpine.DEB.1.00.0808081716010.24820@pacific.mpi-cbg.de.mpi-cbg.de","threadId":"14891","inReplyTo":"alpine.DEB.1.00.0808081647580.24820@pacific.mpi-cbg.de.mpi-cbg.de","subject":"Re: How to replace a single corrupt, packed object?","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2008-08-08T15:21:44Z","receivedAt":"2008-08-08T15:21:44Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Fri, 8 Aug 2008, Johannes Schindelin wrote:\n\n> On Fri, 8 Aug 2008, Shawn O. Pearce wrote:\n> \n> > Johannes Schindelin wrote:\n> > > my auto gc kicked in, and shows this:\n> > > \n> > > fatal: corrupt packed object for 2c1e128aa51e3a64bd61556c0cd488628b423ccf\n> > > error: failed to run repack\n> > > \n> > > Fortunately, I have the uncorrupted object somewhere else. So I copy the \n> > > single object as a loose one, and all is fine. Right?\n> > > \n> > > Wrong.\n> > > \n> > > Repack still picks up the corrupt object instead of the good one. What's \n> > > the best way out?\n> > \n> > Use a 1.6.0 rc? Nico I thought fixed this in 1.6 to automatically\n> > try and find another copy of an object if the first copy considered\n> > for inclusion into the pack was corrupt.\n> \n> Thanks for the quick reply, but no joy (You should know me better than \n> that ;-) :\n> \n> $ git --version\n> git version 1.6.0.rc1.112.gebbe4\n\nJust for bullocks, I gave it a try with git version 1.6.0.rc1.141.g0d7ea.\nThat is rc2+26 patches, in case you are interested.\n\nUnfortunately, each test costs me\n\n630.25user 19.74system 22:01.43elapsed 49%CPU (0avgtext+0avgdata 0maxresident)k\n7533688inputs+3154520outputs (55275major+2011583minor)pagefaults 0swaps\n\n(including the inability to work properly because of all the swapping) \nwhich is not really nice, as the corrupt packed object occurs during the \nlast 20% of the _writing_ phase of 60104 objects.\n\nOh, and they are big objects. The corrupt one is 65 megabyte, for \nexample.\n\nCiao,\nDscho\n"},{"id":"86514","messageId":"0BF03F86-8E4E-46D2-9B04-4385CEBD6902@ai.rug.nl","threadId":"14891","inReplyTo":"alpine.DEB.1.00.0808081639490.24820@pacific.mpi-cbg.de.mpi-cbg.de","subject":"Re: How to replace a single corrupt, packed object?","fromName":"Pieter de Bie","fromEmail":"pdebie@ai.rug.nl","sentAt":"2008-08-08T15:30:19Z","receivedAt":"2008-08-08T15:30:19Z","isPatch":false,"sender":{"key":"pdebie@ai.rug.nl","avatar":null},"body":"On 8 aug 2008, at 16:41, Johannes Schindelin wrote:\n\n> fatal: corrupt packed object for \n> 2c1e128aa51e3a64bd61556c0cd488628b423ccf\n> error: failed to run repack\n>\n> Fortunately, I have the uncorrupted object somewhere else. So I \n> copy the\n> single object as a loose one, and all is fine. Right?\n>\n> Wrong.\n>\n> Repack still picks up the corrupt object instead of the good one. \n> What's\n> the best way out?\n\nI don't know how to replace the object, but you can expand the pack, \nreplace\nthe loose file, delete the old pack, and then create a new one, as \nalso explained\nin this thread:\n\n\thttp://thread.gmane.org/gmane.comp.version-control.git/40893/focus=40896\n\n- Pieter\n"},{"id":"86518","messageId":"20080808161937.GC9152@spearce.org","threadId":"14891","inReplyTo":"0BF03F86-8E4E-46D2-9B04-4385CEBD6902@ai.rug.nl","subject":"Re: How to replace a single corrupt, packed object?","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2008-08-08T16:19:37Z","receivedAt":"2008-08-08T16:19:37Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Pieter de Bie wrote:\n> On 8 aug 2008, at 16:41, Johannes Schindelin wrote:\n>\n>> fatal: corrupt packed object for \n>> 2c1e128aa51e3a64bd61556c0cd488628b423ccf\n>> error: failed to run repack\n>>\n>> What's\n>> the best way out?\n>\n> I don't know how to replace the object, but you can expand the pack, \n> replace\n> the loose file, delete the old pack, and then create a new one\n\nI don't think that will work here. The unpack-objects process will\nfail when it finds this bad object, and everything after that in\nthe pack file will be dropped on the floor and not get unpacked.\n\n-- \nShawn.\n"},{"id":"86519","messageId":"20080808162347.GD9152@spearce.org","threadId":"14891","inReplyTo":"alpine.DEB.1.00.0808081716010.24820@pacific.mpi-cbg.de.mpi-cbg.de","subject":"Re: How to replace a single corrupt, packed object?","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2008-08-08T16:23:47Z","receivedAt":"2008-08-08T16:23:47Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Johannes Schindelin wrote:\n> On Fri, 8 Aug 2008, Johannes Schindelin wrote:\n> > On Fri, 8 Aug 2008, Shawn O. Pearce wrote:\n> > \n> > > Johannes Schindelin wrote:\n> > > > my auto gc kicked in, and shows this:\n> > > > \n> > > > fatal: corrupt packed object for 2c1e128aa51e3a64bd61556c0cd488628b423ccf\n> > > > error: failed to run repack\n\nHere's another idea.\n\nYou have the good object loose. So do something like this:\n\n $ mkdir foo\n $ cd foo\n $ git init\n $ cp ../../the_good_loose_obj .git/objects/??/....\n\n $ cd ../corrupt_repo\n $ (\n cd ../foo;\n echo blob; \n echo data $(git cat-file -s the_good_loose_obj);\n git cat-file blob the_good_loose_obj;\n ) | git fast-import\n $ git repack -a -d\n\nThe new pack file created by gfi will have a newer timestamp than the\none with the corruption. This will cause it to sort higher in the\npack list, and we'll take objects from the first pack we find it in.\n\n-- \nShawn.\n"},{"id":"86523","messageId":"20080808163628.GF9152@spearce.org","threadId":"14891","inReplyTo":"90E12BC7-1950-41DF-8BE5-C6B63CE060D9@ai.rug.nl","subject":"Re: How to replace a single corrupt, packed object?","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2008-08-08T16:36:28Z","receivedAt":"2008-08-08T16:36:28Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Pieter de Bie wrote:\n> On 8 aug 2008, at 18:19, Shawn O. Pearce wrote:\n>\n>> The unpack-objects process will\n>> fail when it finds this bad object, and everything after that in\n>> the pack file will be dropped on the floor and not get unpacked.\n>\n> Even with the -r switch?\n>\n> -r When unpacking a corrupt packfile, the command dies at the \n> first corruption. This flag tells it to keep going and make\n> the best effort to recover as many objects as possible.\n\nOh, thanks for reminding me. I had forgotten that Linus added -r\nwhen someone else had corruption in a packed object. Yea, if you\nuse -r it may be able to resume and pick up where it left off.\n\nI haven't studied the -r code so I'm not sure how it knows where\nits safe to restart unpacking from. But if you have a .idx file\nwe probably could make a really good guess based on the offsets\nit stores. I doubt unpack-objects makes use of the .idx.\n\n-- \nShawn.\n"},{"id":"86524","messageId":"alpine.DEB.1.00.0808081841290.24820@pacific.mpi-cbg.de.mpi-cbg.de","threadId":"14891","inReplyTo":"90E12BC7-1950-41DF-8BE5-C6B63CE060D9@ai.rug.nl","subject":"Re: How to replace a single corrupt, packed object?","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2008-08-08T16:48:33Z","receivedAt":"2008-08-08T16:48:33Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Fri, 8 Aug 2008, Pieter de Bie wrote:\n\n> On 8 aug 2008, at 18:19, Shawn O. Pearce wrote:\n> \n> >The unpack-objects process will fail when it finds this bad object, and \n> >everything after that in the pack file will be dropped on the floor and \n> >not get unpacked.\n> \n> Even with the -r switch?\n> \n> -r When unpacking a corrupt packfile, the command dies at the first\n> corruption. This flag tells it to keep going and make\n> the best effort to recover as many objects as possible.\n\nIn any case, the pack is too large for me to let my computer repack \neverything, when only one object needs repacking.\n\nCiao,\nDscho\n"},{"id":"86740","messageId":"alpine.LFD.1.10.0808102139050.22892@xanadu.home","threadId":"14891","inReplyTo":"alpine.DEB.1.00.0808081639490.24820@pacific.mpi-cbg.de.mpi-cbg.de","subject":"Re: How to replace a single corrupt, packed object?","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2008-08-11T01:43:40Z","receivedAt":"2008-08-11T01:43:40Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"\n[ sorry for the delay -- I just returned from vacation ]\n\nOn Fri, 8 Aug 2008, Johannes Schindelin wrote:\n\n> Hi,\n> \n> my auto gc kicked in, and shows this:\n> \n> fatal: corrupt packed object for 2c1e128aa51e3a64bd61556c0cd488628b423ccf\n> error: failed to run repack\n> \n> Fortunately, I have the uncorrupted object somewhere else. So I copy the \n> single object as a loose one, and all is fine. Right?\n> \n> Wrong.\n\nWell, to be sure things are then right or wrong, just do a \n\n\tgit show 2c1e128aa51e3a64bd61556c0cd488628b423ccf\n\nIf you can't see the object before, and are able to see it once it has \nbeen copied over, then things are \"right\".\n\n> Repack still picks up the corrupt object instead of the good one. What's \n> the best way out?\n\nHow do you repack? The only way to get rid of a corrupted object in \nthat case is to 'git repack -a -f'.\n\n\nNicolas\n"},{"id":"86744","messageId":"alpine.LFD.1.10.0808102146050.22892@xanadu.home","threadId":"14891","inReplyTo":"alpine.DEB.1.00.0808081841290.24820@pacific.mpi-cbg.de.mpi-cbg.de","subject":"Re: How to replace a single corrupt, packed object?","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2008-08-11T02:55:22Z","receivedAt":"2008-08-11T02:55:22Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Fri, 8 Aug 2008, Johannes Schindelin wrote:\n\n> In any case, the pack is too large for me to let my computer repack \n> everything, when only one object needs repacking.\n\nBy that you mean you cannot/don't want to use repack -f, right?\n\nThere _could_ be a way to hack pack-objects so not to reuse bad objects. \nHowever I don't want that to impact the code too much for an \nevent that hopefully should almost never happens, especially if using -f \ndoes work around it already.\n\nWell, let's see.\n\n[...]\n\nOK, here's what the patch to allow repacking without -f and still using \nredundant objects in presence of pack corruption might look like. \nPlease tell me if that works for you.\n\ndiff --git a/builtin-pack-objects.c b/builtin-pack-objects.c\nindex 2dadec1..88e73f3 100644\n--- a/builtin-pack-objects.c\n+++ b/builtin-pack-objects.c\n@@ -277,6 +277,7 @@ static unsigned long write_object(struct sha1file *f,\n \t\t\t\t */\n \n \tif (!to_reuse) {\n+\t\tno_reuse:\n \t\tif (!usable_delta) {\n \t\t\tbuf = read_sha1_file(entry->idx.sha1, &type, &size);\n \t\t\tif (!buf)\n@@ -364,14 +365,28 @@ static unsigned long write_object(struct sha1file *f,\n \t\t\treused_delta++;\n \t\t}\n \t\thdrlen = encode_header(type, entry->size, header);\n+\n \t\toffset = entry->in_pack_offset;\n \t\trevidx = find_pack_revindex(p, offset);\n \t\tdatalen = revidx[1].offset - offset;\n \t\tif (!pack_to_stdout && p->index_version > 1 &&\n-\t\t check_pack_crc(p, &w_curs, offset, datalen, revidx->nr))\n-\t\t\tdie(\"bad packed object CRC for %s\", sha1_to_hex(entry->idx.sha1));\n+\t\t check_pack_crc(p, &w_curs, offset, datalen, revidx->nr)) {\n+\t\t\terror(\"bad packed object CRC for %s\", sha1_to_hex(entry->idx.sha1));\n+\t\t\tif (entry->delta)\n+\t\t\t\treused_delta--;\n+\t\t\tgoto no_reuse;\n+\t\t}\n+\n \t\toffset += entry->in_pack_header_size;\n \t\tdatalen -= entry->in_pack_header_size;\n+\t\tif (!pack_to_stdout && p->index_version == 1 &&\n+\t\t check_pack_inflate(p, &w_curs, offset, datalen, entry->size)) {\n+\t\t\tdie(\"corrupt packed object for %s\", sha1_to_hex(entry->idx.sha1));\n+\t\t\tif (entry->delta)\n+\t\t\t\treused_delta--;\n+\t\t\tgoto no_reuse;\n+\t\t}\n+\n \t\tif (type == OBJ_OFS_DELTA) {\n \t\t\toff_t ofs = entry->idx.offset - entry->delta->idx.offset;\n \t\t\tunsigned pos = sizeof(dheader) - 1;\n@@ -394,10 +409,6 @@ static unsigned long write_object(struct sha1file *f,\n \t\t\t\treturn 0;\n \t\t\tsha1write(f, header, hdrlen);\n \t\t}\n-\n-\t\tif (!pack_to_stdout && p->index_version == 1 &&\n-\t\t check_pack_inflate(p, &w_curs, offset, datalen, entry->size))\n-\t\t\tdie(\"corrupt packed object for %s\", sha1_to_hex(entry->idx.sha1));\n \t\tcopy_pack_data(f, p, &w_curs, offset, datalen);\n \t\tunuse_pack(&w_curs);\n \t\treused++;\n\n\nNicolas\n"},{"id":"86746","messageId":"20080811030744.GD27195@spearce.org","threadId":"14891","inReplyTo":"alpine.LFD.1.10.0808102146050.22892@xanadu.home","subject":"Re: How to replace a single corrupt, packed object?","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2008-08-11T03:07:44Z","receivedAt":"2008-08-11T03:07:44Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Nicolas Pitre wrote:\n> OK, here's what the patch to allow repacking without -f and still using \n> redundant objects in presence of pack corruption might look like. \n> Please tell me if that works for you.\n\nAside from goto being considered harmful by some really smart people,\nthis patch makes a lot of sense. Its only downside is a backwards\ngoto within this function, but the code is actually still quite\nclear to me.\n\nIf this allows git to magically fix Dscho's bad pack, it may be\nworth including in the core tree.\n\n \n> diff --git a/builtin-pack-objects.c b/builtin-pack-objects.c\n> index 2dadec1..88e73f3 100644\n> --- a/builtin-pack-objects.c\n> +++ b/builtin-pack-objects.c\n> @@ -277,6 +277,7 @@ static unsigned long write_object(struct sha1file *f,\n> \t\t\t\t */\n> \n> \tif (!to_reuse) {\n> +\t\tno_reuse:\n> \t\tif (!usable_delta) {\n> \t\t\tbuf = read_sha1_file(entry->idx.sha1, &type, &size);\n> \t\t\tif (!buf)\n> @@ -364,14 +365,28 @@ static unsigned long write_object(struct sha1file *f,\n> \t\t\treused_delta++;\n> \t\t}\n> \t\thdrlen = encode_header(type, entry->size, header);\n> +\n> \t\toffset = entry->in_pack_offset;\n> \t\trevidx = find_pack_revindex(p, offset);\n> \t\tdatalen = revidx[1].offset - offset;\n> \t\tif (!pack_to_stdout && p->index_version > 1 &&\n> -\t\t check_pack_crc(p, &w_curs, offset, datalen, revidx->nr))\n> -\t\t\tdie(\"bad packed object CRC for %s\", sha1_to_hex(entry->idx.sha1));\n> +\t\t check_pack_crc(p, &w_curs, offset, datalen, revidx->nr)) {\n> +\t\t\terror(\"bad packed object CRC for %s\", sha1_to_hex(entry->idx.sha1));\n> +\t\t\tif (entry->delta)\n> +\t\t\t\treused_delta--;\n> +\t\t\tgoto no_reuse;\n> +\t\t}\n> +\n> \t\toffset += entry->in_pack_header_size;\n> \t\tdatalen -= entry->in_pack_header_size;\n> +\t\tif (!pack_to_stdout && p->index_version == 1 &&\n> +\t\t check_pack_inflate(p, &w_curs, offset, datalen, entry->size)) {\n> +\t\t\tdie(\"corrupt packed object for %s\", sha1_to_hex(entry->idx.sha1));\n> +\t\t\tif (entry->delta)\n> +\t\t\t\treused_delta--;\n> +\t\t\tgoto no_reuse;\n> +\t\t}\n> +\n> \t\tif (type == OBJ_OFS_DELTA) {\n> \t\t\toff_t ofs = entry->idx.offset - entry->delta->idx.offset;\n> \t\t\tunsigned pos = sizeof(dheader) - 1;\n> @@ -394,10 +409,6 @@ static unsigned long write_object(struct sha1file *f,\n> \t\t\t\treturn 0;\n> \t\t\tsha1write(f, header, hdrlen);\n> \t\t}\n> -\n> -\t\tif (!pack_to_stdout && p->index_version == 1 &&\n> -\t\t check_pack_inflate(p, &w_curs, offset, datalen, entry->size))\n> -\t\t\tdie(\"corrupt packed object for %s\", sha1_to_hex(entry->idx.sha1));\n> \t\tcopy_pack_data(f, p, &w_curs, offset, datalen);\n> \t\tunuse_pack(&w_curs);\n> \t\treused++;\n> \n> \n> Nicolas\n\n-- \nShawn.\n"},{"id":"86747","messageId":"alpine.LFD.1.10.0808102325280.22892@xanadu.home","threadId":"14891","inReplyTo":"20080811030744.GD27195@spearce.org","subject":"Re: How to replace a single corrupt, packed object?","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2008-08-11T03:40:30Z","receivedAt":"2008-08-11T03:40:30Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Sun, 10 Aug 2008, Shawn O. Pearce wrote:\n\n> Nicolas Pitre wrote:\n> > OK, here's what the patch to allow repacking without -f and still using \n> > redundant objects in presence of pack corruption might look like. \n> > Please tell me if that works for you.\n> \n> Aside from goto being considered harmful by some really smart people,\n\nWell, other really smart people consider gotos perfectly fine when used \njudiciously. So this ends up being a question of belief and taste.\n\n> this patch makes a lot of sense. Its only downside is a backwards\n> goto within this function, but the code is actually still quite\n> clear to me.\n\nThe actual downside I see with this patch is the fact that real data \ncorruptions might be \"fixed\" automagically with user unaware of it. \nThis could be a serious sign that the hardware is going bad and \nrequiring the user to consciously use -f to fix things is good. However \nit is most unlikely that redundant objects will be kept around in the \nnormal case, hence manual intervention will be needed anyway to bring a \ncopy of bad object into the repository. So not having to use -f might \nnot be such an issue.\n\n> If this allows git to magically fix Dscho's bad pack, it may be\n> worth including in the core tree.\n\nYep.\n\n\nNicolas\n"},{"id":"86748","messageId":"20080811034608.GA27684@spearce.org","threadId":"14891","inReplyTo":"alpine.LFD.1.10.0808102325280.22892@xanadu.home","subject":"Re: How to replace a single corrupt, packed object?","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2008-08-11T03:46:08Z","receivedAt":"2008-08-11T03:46:08Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Nicolas Pitre wrote:\n> The actual downside I see with this patch is the fact that real data \n> corruptions might be \"fixed\" automagically with user unaware of it. \n> This could be a serious sign that the hardware is going bad and \n> requiring the user to consciously use -f to fix things is good. However \n> it is most unlikely that redundant objects will be kept around in the \n> normal case, hence manual intervention will be needed anyway to bring a \n> copy of bad object into the repository. So not having to use -f might \n> not be such an issue.\n\nYup, I agree completely.\n\nDuplicates should be rare, and likely are only the fault of a\ndumb transport fetch, or the user trying to fix their repository\nby placing copies of corrupt objects obtained from elsewhere.\nRequiring -f to fix such cases is heavy-handed. Some trees can\ntake many hours to repack with -f; think gcc or OOo.\n\n-- \nShawn.\n"},{"id":"86806","messageId":"alpine.DEB.1.00.0808111818460.24820@pacific.mpi-cbg.de.mpi-cbg.de","threadId":"14891","inReplyTo":"alpine.LFD.1.10.0808102146050.22892@xanadu.home","subject":"Re: How to replace a single corrupt, packed object?","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2008-08-11T18:59:46Z","receivedAt":"2008-08-11T18:59:46Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Sun, 10 Aug 2008, Nicolas Pitre wrote:\n\n> On Fri, 8 Aug 2008, Johannes Schindelin wrote:\n> \n> > In any case, the pack is too large for me to let my computer repack \n> > everything, when only one object needs repacking.\n> \n> By that you mean you cannot/don't want to use repack -f, right?\n\nRight. However, I had a relatively fast machine standing nearby today, \nso that scp was not too painful.\n\n> There _could_ be a way to hack pack-objects so not to reuse bad objects. \n> However I don't want that to impact the code too much for an event that \n> hopefully should almost never happens, especially if using -f does work \n> around it already.\n> \n> Well, let's see.\n> \n> [...]\n> \n> OK, here's what the patch to allow repacking without -f and still using \n> redundant objects in presence of pack corruption might look like. \n> Please tell me if that works for you.\n\nThe testing took quite a while unfortunately, mainly because I followed \nShawn's advice, and added not only a loose object, but also a single pack \nwith the single object in it, and a newer timestamp.\n\nThis resulted in my CPU being hogged when Git tried to read the object. I \ndo not know exactly what is happening, but I suspect an infinite loop due \nto the funny interaction between a valid and a corrupt pack containing the \nsame object. Or maybe the issue described later in this mail.\n\nOnly when I removed the pack did things actually go further, so there is \nstill a bug lurking.\n\nYour patch worked _almost_:\n\n> \t\toffset += entry->in_pack_header_size;\n> \t\tdatalen -= entry->in_pack_header_size;\n> +\t\tif (!pack_to_stdout && p->index_version == 1 &&\n> +\t\t check_pack_inflate(p, &w_curs, offset, datalen, entry->size)) {\n> +\t\t\tdie(\"corrupt packed object for %s\", sha1_to_hex(entry->idx.sha1));\n\nThis needs to be an error(), obviously.\n\n> +\t\t\tif (entry->delta)\n> +\t\t\t\treused_delta--;\n> +\t\t\tgoto no_reuse;\n> +\t\t}\n> +\n> \t\tif (type == OBJ_OFS_DELTA) {\n> \t\t\toff_t ofs = entry->idx.offset - entry->delta->idx.offset;\n> \t\t\tunsigned pos = sizeof(dheader) - 1;\n\nWith that, it took quite a while, then it told me about the corrupt \nobject.\n\nAnd then it hangs in the loop sha1_file.c:1511. The function inflate() \nreturns Z_BUF_ERROR, and nothing is read.\n\nOh, and it still tries to access the same corrupt pack.\n\nThanks,\nDscho\n\nP.S.: I have to wrap up my work at my current (interim) job, and will be \nmoving in the next days, so do not expect too much from my side before \nMonday.\n"},{"id":"90739","messageId":"alpine.DEB.1.00.0809151603390.13830@pacific.mpi-cbg.de.mpi-cbg.de","threadId":"14891","inReplyTo":"alpine.LFD.1.10.0808102139050.22892@xanadu.home","subject":"Re: How to replace a single corrupt, packed object?","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2008-09-15T14:05:52Z","receivedAt":"2008-09-15T14:05:52Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Sun, 10 Aug 2008, Nicolas Pitre wrote:\n\n> [ sorry for the delay -- I just returned from vacation ]\n\nNo need to be sorry; my delay is even worse...\n\n> On Fri, 8 Aug 2008, Johannes Schindelin wrote:\n> \n> > my auto gc kicked in, and shows this:\n> > \n> > fatal: corrupt packed object for 2c1e128aa51e3a64bd61556c0cd488628b423ccf\n> > error: failed to run repack\n> > \n> > Fortunately, I have the uncorrupted object somewhere else. So I copy the \n> > single object as a loose one, and all is fine. Right?\n> > \n> > Wrong.\n> \n> Well, to be sure things are then right or wrong, just do a \n> \n> \tgit show 2c1e128aa51e3a64bd61556c0cd488628b423ccf\n> \n> If you can't see the object before, and are able to see it once it has \n> been copied over, then things are \"right\".\n> \n> > Repack still picks up the corrupt object instead of the good one. \n> > What's the best way out?\n> \n> How do you repack? The only way to get rid of a corrupted object in \n> that case is to 'git repack -a -f'.\n\nTurns out I am a complete, utter moron. And I am sure René will quote me \non that.\n\nGit would probably have taken the copied-over object, and now took the \ncopied-over pack (finally!).\n\nMy mistake was to keep the .keep file. And the corrupt object was -- you \nguessed it -- in the corresponding .pack file.\n\nAaaaargh,\nDscho\n"},{"id":"90750","messageId":"alpine.LFD.1.10.0809151223570.6279@xanadu.home","threadId":"14891","inReplyTo":"alpine.DEB.1.00.0809151603390.13830@pacific.mpi-cbg.de.mpi-cbg.de","subject":"Re: How to replace a single corrupt, packed object?","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2008-09-15T16:26:36Z","receivedAt":"2008-09-15T16:26:36Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Mon, 15 Sep 2008, Johannes Schindelin wrote:\n\n> On Sun, 10 Aug 2008, Nicolas Pitre wrote:\n> \n> > On Fri, 8 Aug 2008, Johannes Schindelin wrote:\n> > \n> > > my auto gc kicked in, and shows this:\n> > > \n> > > fatal: corrupt packed object for 2c1e128aa51e3a64bd61556c0cd488628b423ccf\n> > > error: failed to run repack\n> > > \n> > > Fortunately, I have the uncorrupted object somewhere else. So I copy the \n> > > single object as a loose one, and all is fine. Right?\n> > > \n> > > Wrong.\n> > \n> > Well, to be sure things are then right or wrong, just do a \n> > \n> > \tgit show 2c1e128aa51e3a64bd61556c0cd488628b423ccf\n> > \n> > If you can't see the object before, and are able to see it once it has \n> > been copied over, then things are \"right\".\n> > \n> > > Repack still picks up the corrupt object instead of the good one. \n> > > What's the best way out?\n> > \n> > How do you repack? The only way to get rid of a corrupted object in \n> > that case is to 'git repack -a -f'.\n> \n> Turns out I am a complete, utter moron. And I am sure René will quote me \n> on that.\n> \n> Git would probably have taken the copied-over object, and now took the \n> copied-over pack (finally!).\n> \n> My mistake was to keep the .keep file. And the corrupt object was -- you \n> guessed it -- in the corresponding .pack file.\n\nOK. Then I'll dig my patch out and write a test for it before \nsubmitting it to Junio.\n\n\nNicolas\n"},{"id":"90754","messageId":"alpine.DEB.1.00.0809151733550.14329@racer","threadId":"14891","inReplyTo":"alpine.LFD.1.10.0809151223570.6279@xanadu.home","subject":"Re: How to replace a single corrupt, packed object?","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2008-09-15T16:34:12Z","receivedAt":"2008-09-15T16:34:12Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Mon, 15 Sep 2008, Nicolas Pitre wrote:\n\n> On Mon, 15 Sep 2008, Johannes Schindelin wrote:\n> \n> > On Sun, 10 Aug 2008, Nicolas Pitre wrote:\n> > \n> > > On Fri, 8 Aug 2008, Johannes Schindelin wrote:\n> > > \n> > > > my auto gc kicked in, and shows this:\n> > > > \n> > > > fatal: corrupt packed object for 2c1e128aa51e3a64bd61556c0cd488628b423ccf\n> > > > error: failed to run repack\n> > > > \n> > > > Fortunately, I have the uncorrupted object somewhere else. So I copy the \n> > > > single object as a loose one, and all is fine. Right?\n> > > > \n> > > > Wrong.\n> > > \n> > > Well, to be sure things are then right or wrong, just do a \n> > > \n> > > \tgit show 2c1e128aa51e3a64bd61556c0cd488628b423ccf\n> > > \n> > > If you can't see the object before, and are able to see it once it has \n> > > been copied over, then things are \"right\".\n> > > \n> > > > Repack still picks up the corrupt object instead of the good one. \n> > > > What's the best way out?\n> > > \n> > > How do you repack? The only way to get rid of a corrupted object in \n> > > that case is to 'git repack -a -f'.\n> > \n> > Turns out I am a complete, utter moron. And I am sure René will quote me \n> > on that.\n> > \n> > Git would probably have taken the copied-over object, and now took the \n> > copied-over pack (finally!).\n> > \n> > My mistake was to keep the .keep file. And the corrupt object was -- you \n> > guessed it -- in the corresponding .pack file.\n> \n> OK. Then I'll dig my patch out and write a test for it before \n> submitting it to Junio.\n\nThank you very much & sorry for the trouble,\nDscho\n"}]}