{"thread":{"id":"53945","subject":"I have gone and done a bad thing - malformed tree objects","startedAt":"2020-07-29T00:47:10Z","lastAt":"2020-08-02T03:02:37Z","messageCount":9,"participants":["Jason Pyeron","Junio C Hamano","Jeff King"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"402315","messageId":"19ca801d66541$cf872af0$6e9580d0$@pdinc.us","threadId":"53945","inReplyTo":null,"subject":"I have gone and done a bad thing - malformed tree objects","fromName":"Jason Pyeron","fromEmail":"jpyeron@pdinc.us","sentAt":"2020-07-29T00:47:18Z","receivedAt":"2020-07-29T00:47:10Z","isPatch":false,"sender":{"key":"jpyeron@pdinc.us","avatar":"https://gravatar.com/avatar/c2e53452caa53d940768a1ffc9cf76196d851b9b534b7a39cd39852a70a0508f?d=mp&s=160"},"body":"I was trying to \"do stuff\" using hash-object -t tree --stdin -w, but I accidentally created trees where other trees were marked as blobs. They were dangling and not connected to any actual commits on my branches.\n\nAfter gc and fsck clean ups, everything reports well...\n\nExcept:\n\n$ GIT_TRACE=1 git cat-file --batch-all-objects --batch=objecttype\n20:41:33.323399 git.c:442               trace: built-in: git cat-file --batch-all-objects --batch=objecttype\nobjecttype\nfatal: object 00009623a06b8dea7c151542fc789539599c07d0 changed type!?\n\nWhere to start on this? I think the unused malformed tree referring to that object got put in to a pack. No idea how to find the malformed tree's id.\n\nFor that fact, cat-file views the object properly:\n\n$ git cat-file -p 00009623a06b8dea7c151542fc789539599c07d0\n100644 blob e465d57c345e2dcb117b5a30f9272b7fc5ec77cd    .p-truncated-names-because-they-are-not-needed-in-this-email\n100755 blob 7f16c1d4cbb75cf7bd635970a2588ced6ccea8ad    Ap\n040000 tree 5261c0a3f3b4c688a082c3c5eaf03f8039bf153c    CA\n100644 blob 188c0d0541523016352b6851e0f7200c18a372e6    CM\n100644 blob c8b040ec356b21fcc06911c544149dc6f5d5b861    CM\n100644 blob e441983f0fd4d57fb7bf640de31f728529f12c29    CM\n100644 blob fd06c9c6ad662e099341f4e0a05b272c6370e64b    CM\n100644 blob d433fb05ebca807f4487ae4cecf48ec3b66cce78    CM\n100755 blob 4b1b049b83bcc7821a7b62977124bfcaa024d960    CM\n040000 tree 150d60813c913ec9a178c4230b18fbda84edc2af    RE\n100644 blob ec54be851b811bff55a2034886a683969ef39880    ac\n040000 tree f541fe035a345b5b3fcf84c83c64135899adadda    ad\n040000 tree 80f08d7c97c88da471495a7e93e1466a737a7d29    au\n040000 tree 7130737ca26ee3f1ac7d437bd8123d024c1cbfeb    cf\n100755 blob 9637152c0a45a7ea716c5c05a7a3a68e2f655bc2    cm\n100755 blob bde5db875d0c5d2bc6874ac62bff026bafd91914    co\n100755 blob 308678b6e4f4aec8c03c6ced43750c84f8a0659d    cr\n040000 tree 1657e8a77ed7430e4845b60cc1e1fb277b385d8a    da\n100644 blob ed2eb8b686d008b895c52ddfca2ef2692174a722    de\n100755 blob 6fe10b9f4d0df0d0c357b7ddee56e5612143c6b9    de\n040000 tree 58a3cb399b25cf54f3ec92d52b79e3f081614d68    de\n100755 blob f09ba30b1d5be4015dbcd8e8cec37449fae504e1    er\n040000 tree 951642bc2f183e984e1879eab68f3414c16e535a    fa\n100755 blob 5b796b1747de423f1fb81f976c860f50c3b1d02a    fi\n100755 blob c2074af9676af85beb9f08bb1a5339b14d6ee485    ge\n100755 blob 22ca54226a6c8e2bb2c9eb6d0db1731553e4b5ae    gl\n100755 blob 57060dc44448e5b6c14a0df59bf81050a8464472    he\n040000 tree af5e7c654837923730d15cc051f843d42ef84b56    hr\n040000 tree 0a0f5e8c1b2968d516b501f7995d6e85897b26ab    im\n040000 tree 6abe0be44fad7a58c0035010137431d9c581aabc    in\n100755 blob 156ee5024cd8fb361552e873b843a8d90b861fb6    in\n100644 blob 3371f55d5057ea4f4d29eae0184dfe17ca2919ff    jb\n040000 tree eef927bf134d3c3777e7c02f6e81f3de06be0bdc    li\n100755 blob 8d0cfbb604a92c2b24b9761efc5320f851c96ac6    lo\n100755 blob 25efc4fd596e651d22c4bfdd8d694acefb5aa004    lo\n100755 blob 92b3f44a17c12485923cf0143938c6f2490bced9    lo\n100755 blob af171c4a7b37c1763d05275ddeeb994fd6ec5d01    ma\n100755 blob 4047be1da363bf55fa9773218665f723d4d9941d    mc\n100755 blob 0a28fb9de61bb5fb9de7876514dabbbc35b8d444    me\n040000 tree b9b374e4925ece4b5781ef70c6590fab8a85b0f5    mi\n100755 blob 6bb288db5b11856ea79577b52d57bf8b0c8ea422    po\n040000 tree 2e18b952c4c796afdcdf4d41230c2e8a4212a8b9    re\n040000 tree 002c945c849e2b976ae193481ad299181322663f    re\n040000 tree 40dd932e877ad3aee47628277d9e75c53284dea5    re\n100755 blob ea520d8ccda4207e732c3549d4c18d1ae0e6136b    sc\n040000 tree 9a7014ee2249be58fd79bbb1a76f8ef94ef1af8e    sc\n100644 blob dfbbb48c461b7739c211ae84a7b1355a03a0bd5e    se\n040000 tree cdb9bdb60dc857cb04e707582852e700cc5c1d63    se\n100755 blob 87542658ec3f45076443a278f7aac8cdcdc649cd    se\n100755 blob bc50a5c1ba42b95abef98c31c36ad1c154b480b4    si\n100644 blob 6e96fb004b88d5537bec9fcfeab581253886a61f    sq\n100644 blob 6e96fb004b88d5537bec9fcfeab581253886a61f    sq\n040000 tree a614f05126c49e0ad2c35243907d3d7300db85a2    te\n040000 tree 1f7e4a51903a860fb4710060aa035beda35e18df    te\n100755 blob ebb1a7029de2062a00b851463922560ed1a369ae    te\n100644 blob e794743d0bf1c99b3ba211bdcf5d60e8261aa8f5    te\n100644 blob da41b803b2a86c90417b3ed1f7fda1b9524fa897    te\n100755 blob 43803c936c25f29895dc1c78a8f1ebcd10ab77b8    te\n040000 tree 042ca993b1f44d4cd7abd846f669c9b0ee9d5545    tr\n040000 tree de84b6f6c8a562a001a492efe640c43e158b2dd8    tr\n100755 blob 90dbabed658c0cbf9f74eed9eff6da0ef6155464    un\n100644 blob ebc83b2afb7bed3c360b7cc5da8750ab3038a2ee    vd\n100644 blob 7c689ba4641860b103a6afda8dc2c0cc803eabdf    xl\n\nThanks in advance, and sorry for my silliness.\n\n-Jason\n\n"},{"id":"402317","messageId":"xmqqh7trb0sr.fsf@gitster.c.googlers.com","threadId":"53945","inReplyTo":"19ca801d66541$cf872af0$6e9580d0$@pdinc.us","subject":"Re: I have gone and done a bad thing - malformed tree objects","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-07-29T00:52:04Z","receivedAt":"2020-07-29T00:52:10Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Jason Pyeron\" <jpyeron@pdinc.us> writes:\n\n> I was trying to \"do stuff\" using hash-object -t tree --stdin -w,\n> but I accidentally created trees where other trees were marked as\n> blobs. They were dangling and not connected to any actual commits\n> on my branches.\n>\n> After gc and fsck clean ups, everything reports well...\n>\n> Except:\n>\n> $ GIT_TRACE=1 git cat-file --batch-all-objects --batch=objecttype\n\ngc and fsck may not have pruned the dangling object yet, but\n--batch-all-objects is a request to enumerate objects that exist in\nthe repository, regardless of their reachability from any ref.\n\nPerhaps \"git prune --expire=now\" would get rid of it?\n"},{"id":"402318","messageId":"19cab01d66544$ecb402d0$c61c0870$@pdinc.us","threadId":"53945","inReplyTo":"xmqqh7trb0sr.fsf@gitster.c.googlers.com","subject":"RE: I have gone and done a bad thing - malformed tree objects","fromName":"Jason Pyeron","fromEmail":"jpyeron@pdinc.us","sentAt":"2020-07-29T01:09:35Z","receivedAt":"2020-07-29T01:09:29Z","isPatch":false,"sender":{"key":"jpyeron@pdinc.us","avatar":"https://gravatar.com/avatar/c2e53452caa53d940768a1ffc9cf76196d851b9b534b7a39cd39852a70a0508f?d=mp&s=160"},"body":"> From: Junio C Hamano\n> Sent: Tuesday, July 28, 2020 8:52 PM\n> \n> \"Jason Pyeron\" writes:\n> \n> > I was trying to \"do stuff\" using hash-object -t tree --stdin -w,\n> > but I accidentally created trees where other trees were marked as\n> > blobs. They were dangling and not connected to any actual commits\n> > on my branches.\n> >\n> > After gc and fsck clean ups, everything reports well...\n> >\n> > Except:\n> >\n> > $ GIT_TRACE=1 git cat-file --batch-all-objects --batch=objecttype\n> \n> gc and fsck may not have pruned the dangling object yet, but\n> --batch-all-objects is a request to enumerate objects that exist in\n> the repository, regardless of their reachability from any ref.\n> \n> Perhaps \"git prune --expire=now\" would get rid of it?\n\nBoth that and\n\ngit -c gc.reflogExpire=now -c gc.reflogExpireUnreachable=now   -c gc.rerereresolved=now -c gc.rerereunresolved=now   -c gc.pruneExpire=now -c gc.worktreePruneExpire=now gc --prune=now --aggressive\n\nleave it in.\n\n"},{"id":"402341","messageId":"xmqq8sf2b3be.fsf@gitster.c.googlers.com","threadId":"53945","inReplyTo":"19cab01d66544$ecb402d0$c61c0870$@pdinc.us","subject":"Re: I have gone and done a bad thing - malformed tree objects","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-07-29T18:09:57Z","receivedAt":"2020-07-29T18:10:04Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Jason Pyeron\" <jpyeron@pdinc.us> writes:\n\n>> gc and fsck may not have pruned the dangling object yet, but\n>> --batch-all-objects is a request to enumerate objects that exist in\n>> the repository, regardless of their reachability from any ref.\n>> \n>> Perhaps \"git prune --expire=now\" would get rid of it?\n>\n> Both that and\n>\n> git -c gc.reflogExpire=now -c gc.reflogExpireUnreachable=now   -c gc.rerereresolved=now -c gc.rerereunresolved=now   -c gc.pruneExpire=now -c gc.worktreePruneExpire=now gc --prune=now --aggressive\n>\n> leave it in.\n\nIf the cruft has already been stored in a packfile, then prune would\nnot touch it.  \"git repack -a -d && git prune --expire=now\" would be\nthe next thing to do.\n"},{"id":"402606","messageId":"045701d6678f$1f03df20$5d0b9d60$@pdinc.us","threadId":"53945","inReplyTo":"xmqq8sf2b3be.fsf@gitster.c.googlers.com","subject":"RE: I have gone and done a bad thing - malformed tree objects","fromName":"Jason Pyeron","fromEmail":"jpyeron@pdinc.us","sentAt":"2020-07-31T23:05:42Z","receivedAt":"2020-07-31T23:05:41Z","isPatch":false,"sender":{"key":"jpyeron@pdinc.us","avatar":"https://gravatar.com/avatar/c2e53452caa53d940768a1ffc9cf76196d851b9b534b7a39cd39852a70a0508f?d=mp&s=160"},"body":"> From: Junio C Hamano\n> Sent: Wednesday, July 29, 2020 2:10 PM\n> \n> \"Jason Pyeron\" writes:\n> \n> >> gc and fsck may not have pruned the dangling object yet, but\n> >> --batch-all-objects is a request to enumerate objects that exist in\n> >> the repository, regardless of their reachability from any ref.\n> >>\n> >> Perhaps \"git prune --expire=now\" would get rid of it?\n> >\n> > Both that and\n> >\n> > git -c gc.reflogExpire=now -c gc.reflogExpireUnreachable=now   -c gc.rerereresolved=now -c\n> gc.rerereunresolved=now   -c gc.pruneExpire=now -c gc.worktreePruneExpire=now gc --prune=now --\n> aggressive\n> >\n> > leave it in.\n> \n> If the cruft has already been stored in a packfile, then prune would\n> not touch it.  \"git repack -a -d && git prune --expire=now\" would be\n> the next thing to do.\n\n$ git repack -a -d && git prune --expire=now\nEnumerating objects: 327236, done.\nCounting objects: 100% (327125/327125), done.\nDelta compression using up to 8 threads\nCompressing objects: 100% (104728/104728), done.\nWriting objects: 100% (327125/327125), done.\nTotal 327125 (delta 205244), reused 326116 (delta 204678), pack-reused 0\n\n$ git cat-file --batch-all-objects --batch=objecttype\nfatal: object 00009623a06b8dea7c151542fc789539599c07d0 changed type!?\n\nGrrrrr....\n\n\n$ mkdir bak/objects/pack -p\n\n$ mv objects/pack/* bak/objects/pack/\n\n$ for i in bak/objects/pack/*.pack; do git unpack-objects < $i ; done\nUnpacking objects: 100% (111/111), 142.57 KiB | 41.00 KiB/s, done.\nUnpacking objects: 100% (303/303), 1.16 MiB | 182.00 KiB/s, done.\nUnpacking objects: 100% (327125/327125), 1.39 GiB | 158.00 KiB/s, done.\n\n$ git -c gc.reflogExpire=now -c gc.reflogExpireUnreachable=now   -c gc.rerereresolved=now -c gc.rerereunresolved=now   -c gc.pruneExpire=now -c gc.worktreePruneExpire=now gc --prune=now --aggressive\nEnumerating objects: 327314, done.\nCounting objects: 100% (327314/327314), done.\nDelta compression using up to 8 threads\nCompressing objects: 100% (309595/309595), done.\nWriting objects: 100% (327314/327314), done.\nSelecting bitmap commits: 33000, done.\nBuilding bitmaps: 100% (311/311), done.\nTotal 327314 (delta 170421), reused 0 (delta 0), pack-reused 0\nRemoving duplicate objects: 100% (256/256), done.\n\n$ ls -Gg objects/pack/\ntotal 1620740\n-r--r-----+ 1    1979274 Jul 30 19:50 pack-45501d1b4064e03537158f14ea1cd40158424086.bitmap\n-r--r-----+ 1    9165864 Jul 30 17:00 pack-45501d1b4064e03537158f14ea1cd40158424086.idx\n-r--r-----+ 1 1648486198 Jul 30 17:00 pack-45501d1b4064e03537158f14ea1cd40158424086.pack\n\n$ git fsck\nChecking object directories: 100% (256/256), done.\nwarning in tree 64dee778f5cb68573709e04d5bdd0f391c002599: zeroPaddedFilemode: contains zero-padded file modes\nChecking objects: 100% (327314/327314), done.\nChecking connectivity: 327540, done.\n\njpyeron@blackfat /projects/disa-cmis/cmis.git\n$ git cat-file -p 64dee778f5cb68573709e04d5bdd0f391c002599\n040000 tree cee1ca9232b589fd8721cbe8c9a2910413f5607e    00\n040000 tree d8e1f140eaa746fe358e5eb8c3d1573b432158c2    01\n040000 tree 5b1b2826f56a9d7ce4ae4040cdf809bded41557c    02\n...\n040000 tree 49d3bcf3bf2dcaac80c12a6075b16ef6d067e807    fd\n040000 tree 64a163e509b2c061927133eacc68826df202f933    fe\n040000 tree 6572fafefd58be1364e7a8dc8230a13d0fd1c9b2    ff\n\n$ git cat-file --batch-all-objects --batch=objecttype\nobjecttype\nfatal: object 00009623a06b8dea7c151542fc789539599c07d0 changed type!?\n\nDouble Grrrrr.....\n\n\n$ mv bak bak2\n\n$ mkdir bak/objects/pack -p\n\n$ mv -v objects/pack/* bak/objects/pack/\n'objects/pack/pack-45501d1b4064e03537158f14ea1cd40158424086.bitmap' -> 'bak/objects/pack/pack-45501d1b4064e03537158f14ea1cd40158424086.bitmap'\n'objects/pack/pack-45501d1b4064e03537158f14ea1cd40158424086.idx' -> 'bak/objects/pack/pack-45501d1b4064e03537158f14ea1cd40158424086.idx'\n'objects/pack/pack-45501d1b4064e03537158f14ea1cd40158424086.pack' -> 'bak/objects/pack/pack-45501d1b4064e03537158f14ea1cd40158424086.pack'\n\n$ for i in bak/objects/pack/*.pack; do git unpack-objects < $i ; done\nUnpacking objects: 100% (327314/327314), 1.53 GiB | 179.00 KiB/s, done.\n\n$ git fsck --lost-found --no-reflogs\nwarning in tree 64dee778f5cb68573709e04d5bdd0f391c002599: zeroPaddedFilemode: contains zero-padded file modes\nChecking object directories: 100% (256/256), done.\nChecking connectivity: 326145, done.\ndangling tree 9e6a1003b4999aacfe522160843059484350f34d\ndangling tree ff42e36dc8727cb5c664ba42add1b42e70de209d\ndangling blob d761c46714331a15dbcef84fd9af92678cd3c79d\ndangling tree cfdf35b71ec160a5cf37cb52647f766dee0a3737\ndangling tree 78dbd6e5a37fea6b708952a00e8bf1d1eb3eb5b5\ndangling tree 798377228d213226f3b068aeb346b84c4df916a1\ndangling tree d4d84773d2523d4d3290ebb669fbc37a3f16808d\ndangling tree fa1de9e451e8f51f51f31ba15d1f0d941be0e51e\nmissing commit 37b7f95fd6b1a015559d0a1dfb0df8f0524fc660\ndangling commit e3300a5773d39735a8930303f58411fc145eff93\ndangling tree 9e3bba142e03a3b9f08a58ff88385e030553540d\ndangling blob e5474ae6ae5fde0d0dfe83a32fa72b3c71601317\ndangling tree ba486ad81526bc6bfed81a292ad4713a170f9cc6\ndangling commit 8cba1b102d044d4098087c4b5fed1f02f3b63ad7\ndangling tree 90657cf7213475972835035494edfa0ae2fea3fc\ndangling commit e0d29cbbdf18cbaf6b23781f08f199d9bd11b7ef\ndangling commit d22f7d69cb188b067c1ae385537aac67063a0bcd\ndangling tree b7f16f536f3c5f6b0642fb2cae2e969da4c81f22\n\n$ git cat-file --batch-all-objects --batch=objecttype\nobjecttype\nfatal: object 00009623a06b8dea7c151542fc789539599c07d0 changed type!?\n\nI'm running fsck a second time with no packs to see what happens. Thoughts?\n\n-Jason\n\n"},{"id":"402608","messageId":"20200731231521.GB1461090@coredump.intra.peff.net","threadId":"53945","inReplyTo":"045701d6678f$1f03df20$5d0b9d60$@pdinc.us","subject":"Re: I have gone and done a bad thing - malformed tree objects","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2020-07-31T23:15:21Z","receivedAt":"2020-07-31T23:15:24Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Jul 31, 2020 at 07:05:42PM -0400, Jason Pyeron wrote:\n\n> > If the cruft has already been stored in a packfile, then prune would\n> > not touch it.  \"git repack -a -d && git prune --expire=now\" would be\n> > the next thing to do.\n> \n> $ git repack -a -d && git prune --expire=now\n> Enumerating objects: 327236, done.\n> Counting objects: 100% (327125/327125), done.\n> Delta compression using up to 8 threads\n> Compressing objects: 100% (104728/104728), done.\n> Writing objects: 100% (327125/327125), done.\n> Total 327125 (delta 205244), reused 326116 (delta 204678), pack-reused 0\n> \n> $ git cat-file --batch-all-objects --batch=objecttype\n> fatal: object 00009623a06b8dea7c151542fc789539599c07d0 changed type!?\n\nThat should be dropping everything that isn't reachable. I'd suggest to\nexpire reflogs, though it looks like you've also tried \"git gc\" with\nreflog expiration. Does removing .git/logs entirely help?\n\nIf not, are you sure it isn't actually reachable from your history? What\ndoes:\n\n  git rev-list --all --objects | grep 00009623a06\n\nsay? If no hits, does adding --reflogs to the command-line change it?\n\nWe also consider blobs in the index reachable. I don't recall offhand\nwhether that applies to trees mentioned by the cache-trees extension. I\ndon't _think_ that would apply to your broken tree, since they'd have\nbeen generated by Git itself, but possibly removing .git/index (if this\nisn't a bare repo) would help?\n\n-Peff\n"},{"id":"402614","messageId":"046201d66796$fb575bd0$f2061370$@pdinc.us","threadId":"53945","inReplyTo":"20200731231521.GB1461090@coredump.intra.peff.net","subject":"RE: I have gone and done a bad thing - malformed tree objects","fromName":"Jason Pyeron","fromEmail":"jpyeron@pdinc.us","sentAt":"2020-08-01T00:01:58Z","receivedAt":"2020-08-01T00:01:54Z","isPatch":false,"sender":{"key":"jpyeron@pdinc.us","avatar":"https://gravatar.com/avatar/c2e53452caa53d940768a1ffc9cf76196d851b9b534b7a39cd39852a70a0508f?d=mp&s=160"},"body":"> From: Jeff King\n> Sent: Friday, July 31, 2020 7:15 PM\n> \n> On Fri, Jul 31, 2020 at 07:05:42PM -0400, Jason Pyeron wrote:\n> \n> > > If the cruft has already been stored in a packfile, then prune would\n> > > not touch it.  \"git repack -a -d && git prune --expire=now\" would be\n> > > the next thing to do.\n> >\n> > $ git repack -a -d && git prune --expire=now\n> > Enumerating objects: 327236, done.\n> > Counting objects: 100% (327125/327125), done.\n> > Delta compression using up to 8 threads\n> > Compressing objects: 100% (104728/104728), done.\n> > Writing objects: 100% (327125/327125), done.\n> > Total 327125 (delta 205244), reused 326116 (delta 204678), pack-reused 0\n> >\n> > $ git cat-file --batch-all-objects --batch=objecttype\n> > fatal: object 00009623a06b8dea7c151542fc789539599c07d0 changed type!?\n> \n> That should be dropping everything that isn't reachable. I'd suggest to\n> expire reflogs, though it looks like you've also tried \"git gc\" with\n> reflog expiration. Does removing .git/logs entirely help?\n> \n> If not, are you sure it isn't actually reachable from your history? What\n> does:\n> \n>   git rev-list --all --objects | grep 00009623a06\n\n$ git rev-list --all --objects | grep 00009623a06\n00009623a06b8dea7c151542fc789539599c07d0 src/htdocs\n(it is still running...)\n\nBut that is an expected result, I will be back at work on Sunday.\n\n\n> \n> say? If no hits, does adding --reflogs to the command-line change it?\n> \n> We also consider blobs in the index reachable. I don't recall offhand\n> whether that applies to trees mentioned by the cache-trees extension. I\n> don't _think_ that would apply to your broken tree, since they'd have\n> been generated by Git itself, but possibly removing .git/index (if this\n> isn't a bare repo) would help?\n> \n> -Peff\n\n"},{"id":"402615","messageId":"20200801014443.GA1464485@coredump.intra.peff.net","threadId":"53945","inReplyTo":"046201d66796$fb575bd0$f2061370$@pdinc.us","subject":"Re: I have gone and done a bad thing - malformed tree objects","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2020-08-01T01:44:43Z","receivedAt":"2020-08-01T01:44:46Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Jul 31, 2020 at 08:01:58PM -0400, Jason Pyeron wrote:\n\n> > That should be dropping everything that isn't reachable. I'd suggest to\n> > expire reflogs, though it looks like you've also tried \"git gc\" with\n> > reflog expiration. Does removing .git/logs entirely help?\n> > \n> > If not, are you sure it isn't actually reachable from your history? What\n> > does:\n> > \n> >   git rev-list --all --objects | grep 00009623a06\n> \n> $ git rev-list --all --objects | grep 00009623a06\n> 00009623a06b8dea7c151542fc789539599c07d0 src/htdocs\n> (it is still running...)\n> \n> But that is an expected result, I will be back at work on Sunday.\n\nSo it sounds like it's still reachable, and you'd need to rewrite\nhistory to get rid of it. Or is that object OK, and it's a containing\ntree that mentions it with the wrong mode the problem? In that case,\nsame question: is the containing tree reachable?\n\n-Peff\n"},{"id":"402636","messageId":"052601d66877$a191c450$e4b54cf0$@pdinc.us","threadId":"53945","inReplyTo":"20200801014443.GA1464485@coredump.intra.peff.net","subject":"RE: I have gone and done a bad thing - malformed tree objects","fromName":"Jason Pyeron","fromEmail":"jpyeron@pdinc.us","sentAt":"2020-08-02T02:50:04Z","receivedAt":"2020-08-02T03:02:37Z","isPatch":false,"sender":{"key":"jpyeron@pdinc.us","avatar":"https://gravatar.com/avatar/c2e53452caa53d940768a1ffc9cf76196d851b9b534b7a39cd39852a70a0508f?d=mp&s=160"},"body":"> -----Original Message-----\n> From: Jeff King\n> Sent: Friday, July 31, 2020 9:45 PM\n> \n> On Fri, Jul 31, 2020 at 08:01:58PM -0400, Jason Pyeron wrote:\n> \n> > > That should be dropping everything that isn't reachable. I'd suggest to\n> > > expire reflogs, though it looks like you've also tried \"git gc\" with\n> > > reflog expiration. Does removing .git/logs entirely help?\n> > >\n> > > If not, are you sure it isn't actually reachable from your history? What\n> > > does:\n> > >\n> > >   git rev-list --all --objects | grep 00009623a06\n> >\n> > $ git rev-list --all --objects | grep 00009623a06\n> > 00009623a06b8dea7c151542fc789539599c07d0 src/htdocs\n> > (it is still running...)\n\n$ git rev-list --all --objects | grep 00009623a06\n00009623a06b8dea7c151542fc789539599c07d0 src/htdocs\n\n$ git rev-list --all --objects --reflog | grep 00009623a06\n00009623a06b8dea7c151542fc789539599c07d0 src/htdocs\n\nNo unexpected results, just the correct tree pointing to the 00009623a06 as a tree.\n\n> >\n> > But that is an expected result, I will be back at work on Sunday.\n> \n> So it sounds like it's still reachable, and you'd need to rewrite\n> history to get rid of it. Or is that object OK, and it's a containing\n> tree that mentions it with the wrong mode the problem? In that case,\n> same question: is the containing tree reachable?\n\nBacking up to the beginning.\n\nThere has always been a tree with a tree entry for that blob - which is reachable.\n\nI then created a tree manually, in it I added that tree id as a blob reference when it should have been a tree. (e.g. 100644 blob 00009623a06b8dea7c151542fc789539599c07d0 00009623a06b8dea7c151542fc789539599c07d0.blob) \n\nI realized my mistake, dropped the commit referring to my butchered tree object (reset to new correct commit with correct tree).\n\nThe tree I created should no longer be reachable.\n\n$ mv logs logs.bak\n\n$ git cat-file --batch-all-objects --batch=objecttype --unordered --allow-unknown-type\nobjecttype\nfatal: object 00009623a06b8dea7c151542fc789539599c07d0 changed type!?\n\n$ git cat-file -t 00009623a06b8dea7c151542fc789539599c07d0\ntree\n\n$ git cat-file -s 00009623a06b8dea7c151542fc789539599c07d0\n2375\n\n$ echo -e 'import zlib\\nfrom hashlib import sha1\\ndecompressed_contents=zlib.decompress(open(\"objects/00/009623a06b8dea7c151542fc789539599c07d0\", \"rb\").read())\\nprint sha1(decompressed_contents).hexdigest()\\nprint decompressed_contents[0:19]' | python | hexdump -C\n00000000  30 30 30 30 39 36 32 33  61 30 36 62 38 64 65 61  |00009623a06b8dea|\n00000010  37 63 31 35 31 35 34 32  66 63 37 38 39 35 33 39  |7c151542fc789539|\n00000020  35 39 39 63 30 37 64 30  0a 74 72 65 65 20 32 33  |599c07d0.tree 23|\n00000030  37 35 00 31 30 30 36 34  34 20 2e 70 0a           |75.100644 .p.|\n0000003d\n\n$ git cat-file -p 00009623a06b8dea7c151542fc789539599c07d0 | cut -c -55 | head\n100644 blob e465d57c345e2dcb117b5a30f9272b7fc5ec77cd    .p\n100755 blob 7f16c1d4cbb75cf7bd635970a2588ced6ccea8ad    Ap\n040000 tree 5261c0a3f3b4c688a082c3c5eaf03f8039bf153c    CA\n100644 blob 188c0d0541523016352b6851e0f7200c18a372e6    CM\n100644 blob c8b040ec356b21fcc06911c544149dc6f5d5b861    CM\n100644 blob e441983f0fd4d57fb7bf640de31f728529f12c29    CM\n100644 blob fd06c9c6ad662e099341f4e0a05b272c6370e64b    CM\n100644 blob d433fb05ebca807f4487ae4cecf48ec3b66cce78    CM\n100755 blob 4b1b049b83bcc7821a7b62977124bfcaa024d960    CM\n040000 tree 150d60813c913ec9a178c4230b18fbda84edc2af    RE\n\n\n\n"}]}