{"thread":{"id":"54296","subject":"Segfault during fsck and spreading brokenness","startedAt":"2020-09-25T16:29:29Z","lastAt":"2020-09-28T06:43:41Z","messageCount":3,"participants":["Rich","Jeff King"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"406347","messageId":"2aa62ff7-0780-8e20-e87d-387152524971@artfulrobot.uk","threadId":"54296","inReplyTo":null,"subject":"Segfault during fsck and spreading brokenness","fromName":"Rich","fromEmail":"forums@artfulrobot.uk","sentAt":"2020-09-25T15:48:39Z","receivedAt":"2020-09-25T16:29:29Z","isPatch":false,"sender":{"key":"forums@artfulrobot.uk","avatar":null},"body":"I was encouraged by a couple of people on stackoverflow to post to this \nlist. Apols in advance if it's not the right place.\n\n\nI encountered a Segfault during fsck on a damaged bare repo (probably \ndue to a powercut. Possibly during an operation, although not sure):\n\ngit --version\ngit version 2.11.0\n\ngit fsck --full -v\nChecking HEAD link\nChecking object directory\nChecking tree 11bbc847cf1b4422b3e37830a9eac2e7af6559de\nChecking tree 11be4abeb20314de6145dfc0e6180807a74c03dc\n--->8 snip 8<---------------------------------------------------------------\nChecking tree 14a4423e86f06c7ad75bf391d138e0cf7790508f\nChecking tree 147aeaec72b2f29bf1813494c942fbce497be679\nzsh: segmentation fault  git fsck --full -v\n\nHost: Debian 10. Linux 4.9.0-13-amd64 #1 SMP Debian 4.9.228-1 \n(2020-07-05) x86_64 GNU/Linux\n\nFS: BTRFS\n\n\nI learnt about the damage while trying to `git push`  from my dev \nworking tree. I think I did a `git pull` when the `git push` failed. The \nresult of this was that my *local* repo was also damaged: there was an \nempty file created that would stop git operations and on deleting that I \ngot a page full of errors.\n\nOriginal SO question: \nhttps://stackoverflow.com/questions/64044995/can-one-corrupt-git-repo-corrupt-another?noredirect=1\n\n\nUnfortunately I don't think I'm at liberty to give access to the repo as \nit possibly contains sensitive info/personal data.\n\nI have solved the problem for myself (took a morning to do it!) but \nreporting here in case it's useful to do so. I use git all the time and \nI'm grateful for it. I'm not a systems-level coder, so not sure what use \nI'll be, but happy to answer any Qs or try things on the broken copies.\n\n\nThanks,\n\nRich\n\n"},{"id":"406369","messageId":"20200925174718.GA70005@coredump.intra.peff.net","threadId":"54296","inReplyTo":"2aa62ff7-0780-8e20-e87d-387152524971@artfulrobot.uk","subject":"Re: Segfault during fsck and spreading brokenness","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2020-09-25T17:47:18Z","receivedAt":"2020-09-25T17:47:21Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Sep 25, 2020 at 04:48:39PM +0100, Rich wrote:\n\n> I was encouraged by a couple of people on stackoverflow to post to this\n> list. Apols in advance if it's not the right place.\n>\n> I encountered a Segfault during fsck on a damaged bare repo (probably due to\n> a powercut. Possibly during an operation, although not sure):\n\nThanks for the report. This is definitely the right place.\n\n> git --version\n> git version 2.11.0\n> \n> git fsck --full -v\n> Checking HEAD link\n> Checking object directory\n> Checking tree 11bbc847cf1b4422b3e37830a9eac2e7af6559de\n> Checking tree 11be4abeb20314de6145dfc0e6180807a74c03dc\n> --->8 snip 8<---------------------------------------------------------------\n> Checking tree 14a4423e86f06c7ad75bf391d138e0cf7790508f\n> Checking tree 147aeaec72b2f29bf1813494c942fbce497be679\n> zsh: segmentation fault  git fsck --full -v\n\nIt's tough to say from this where the problem might be. If you still\nhave the broken repo and can reproduce, two things that might help:\n\n  - trying with a more recent version of Git; we've fixed quite a few\n    segfaults around corrupted data in the past few years\n\n  - getting a backtrace; if you can build Git from source, the simplest\n    thing is just running \"gdb --args /path/to/your/git fsck --full -v\",\n    waiting for it to crash, and then running \"bt\" in the debugger\n\n> I learnt about the damage while trying to `git push`  from my dev working\n> tree. I think I did a `git pull` when the `git push` failed. The result of\n> this was that my *local* repo was also damaged: there was an empty file\n> created that would stop git operations and on deleting that I got a page\n> full of errors.\n\nThat's concerning; push/pull generally try hard not to let corruption\nspread. What transport do you use between the two servers? The usual\ngit-over-ssh and git-over-http protocols should be pretty resilient, but\nI would not be surprised if the old dumb-http protocol, which just\ndownloads remote files wholesale, would be confused by a zero-length\nloose object file or similar.\n\n-Peff\n"},{"id":"406523","messageId":"5cb3e4da-05ef-ac6a-2b8c-fab3137cdc68@artfulrobot.uk","threadId":"54296","inReplyTo":"20200925174718.GA70005@coredump.intra.peff.net","subject":"Re: Segfault during fsck and spreading brokenness","fromName":"Rich","fromEmail":"forums@artfulrobot.uk","sentAt":"2020-09-28T06:43:34Z","receivedAt":"2020-09-28T06:43:41Z","isPatch":false,"sender":{"key":"forums@artfulrobot.uk","avatar":null},"body":"Hi Jeff,\n\nThanks for a friendly and welcoming response (I was a bit scared posting \nhere I suppose because the project's creator has a rep for not suffering \nfools!).\n\nI copied the repo that causes a segfault on 2.11.0 to my laptop which \nruns 2.25.1 and I'm pleased to report that there's no segfault. I've \nposted the errors it does show below this message in case they're useful.\n\n>   What transport do you use between the two servers? The usual\n> git-over-ssh and git-over-http protocols should be pretty resilient, but\n> I would not be surprised if the old dumb-http protocol, which just\n> downloads remote files wholesale, would be confused by a zero-length\n> loose object file or similar.\n\nThe two servers are connected by SSH. As in:\n\n% git config remote.origin.url sshalias:path/to/repo\n\nwhere sshalias is defined in my .ssh/config file.\n\nTo test again,I made a bare clone of the fixed repo on my 2.25.1 host. I \nstill had some dangling trees/blobs so I ran the following to make sure \nwe were all clean:\n\ngit reflog expire --expire-unreachable=now --all\ngit gc --prune=now\ngit fsck --full\n\n\nThen I added the broken repo as a remote (ssh) repo and ran fetch, which \ncrashed (presumably correctly):\n\n% git fetch broke\nerror: object file ./objects/3e/9c7cd338d2ea3603d0c23b4b56902a04937833 \nis empty\nerror: object file ./objects/3e/9c7cd338d2ea3603d0c23b4b56902a04937833 \nis empty\nfatal: loose object 3e9c7cd338d2ea3603d0c23b4b56902a04937833 (stored in \n./objects/3e/9c7cd338d2ea3603d0c23b4b56902a04937833) is corrupt\nfatal: the remote end hung up unexpectedly\n\nI re-ran fsck and it was still all clean. So 2.25.1 seems safe.I \nrepeated the same procedure on the 2.11.0 version (Debian 10 host), and \nthe same thing happened.\n\nSo I'm not sure what happened with the two repos getting broken at once; \ncan't repeat it.\n\nLet's hope that this was something that was fixed as you say!\n\nThanks,\nRich\n\n\n\n\n## Errors shown when running fsck on 2.25.1 on the broken repo:\n\ngit --version\ngit version\n% git fsck --full\nerror: object file ./objects/14/2d73242bba1e4aee89a887ec37d3b1ef00461c \nis empty\nerror: unable to mmap \n./objects/14/2d73242bba1e4aee89a887ec37d3b1ef00461c: No such file or \ndirectory\nerror: 142d73242bba1e4aee89a887ec37d3b1ef00461c: object corrupt or \nmissing: ./objects/14/2d73242bba1e4aee89a887ec37d3b1ef00461c\nerror: object file ./objects/3e/9c7cd338d2ea3603d0c23b4b56902a04937833 \nis empty\nerror: unable to mmap \n./objects/3e/9c7cd338d2ea3603d0c23b4b56902a04937833: No such file or \ndirectory\nerror: 3e9c7cd338d2ea3603d0c23b4b56902a04937833: object corrupt or \nmissing: ./objects/3e/9c7cd338d2ea3603d0c23b4b56902a04937833\nerror: object file ./objects/45/aaa8e78abffa54ee2a620ef453d2b73443a3f9 \nis empty\nerror: unable to mmap \n./objects/45/aaa8e78abffa54ee2a620ef453d2b73443a3f9: No such file or \ndirectory\nerror: 45aaa8e78abffa54ee2a620ef453d2b73443a3f9: object corrupt or \nmissing: ./objects/45/aaa8e78abffa54ee2a620ef453d2b73443a3f9\nerror: object file ./objects/51/c79dda59372c0bd44457fbf810c66a6b184bb4 \nis empty\nerror: unable to mmap \n./objects/51/c79dda59372c0bd44457fbf810c66a6b184bb4: No such file or \ndirectory\nerror: 51c79dda59372c0bd44457fbf810c66a6b184bb4: object corrupt or \nmissing: ./objects/51/c79dda59372c0bd44457fbf810c66a6b184bb4\nerror: object file ./objects/92/92c90c7f01b5f9252a4631071d8ed39b121d52 \nis empty\nerror: unable to mmap \n./objects/92/92c90c7f01b5f9252a4631071d8ed39b121d52: No such file or \ndirectory\nerror: 9292c90c7f01b5f9252a4631071d8ed39b121d52: object corrupt or \nmissing: ./objects/92/92c90c7f01b5f9252a4631071d8ed39b121d52\nerror: object file ./objects/a0/1c6d43bb04b86b8263848c9d8daa20d4a68be4 \nis empty\nerror: unable to mmap \n./objects/a0/1c6d43bb04b86b8263848c9d8daa20d4a68be4: No such file or \ndirectory\nerror: a01c6d43bb04b86b8263848c9d8daa20d4a68be4: object corrupt or \nmissing: ./objects/a0/1c6d43bb04b86b8263848c9d8daa20d4a68be4\nerror: object file ./objects/c6/480c5f0be01c6860a17123bef79aa3fb98dd8b \nis empty\nerror: unable to mmap \n./objects/c6/480c5f0be01c6860a17123bef79aa3fb98dd8b: No such file or \ndirectory\nerror: c6480c5f0be01c6860a17123bef79aa3fb98dd8b: object corrupt or \nmissing: ./objects/c6/480c5f0be01c6860a17123bef79aa3fb98dd8b\nerror: object file ./objects/e7/e8f34edee57ed77b240fa024df2f472dd9776d \nis empty\nerror: unable to mmap \n./objects/e7/e8f34edee57ed77b240fa024df2f472dd9776d: No such file or \ndirectory\nerror: e7e8f34edee57ed77b240fa024df2f472dd9776d: object corrupt or \nmissing: ./objects/e7/e8f34edee57ed77b240fa024df2f472dd9776d\nChecking object directories: 100% (256/256), done.\nChecking objects: 100% (310949/310949), done.\nerror: object file ./objects/3e/9c7cd338d2ea3603d0c23b4b56902a04937833 \nis empty\nerror: object file ./objects/3e/9c7cd338d2ea3603d0c23b4b56902a04937833 \nis empty\nfatal: loose object 3e9c7cd338d2ea3603d0c23b4b56902a04937833 (stored in \n./objects/3e/9c7cd338d2ea3603d0c23b4b56902a04937833) is corrupt\n\n"}]}