Segfault during fsck and spreading brokenness

I was encouraged by a couple of people on stackoverflow to post to this list. Apols in advance if it's not the right place.

I encountered a Segfault during fsck on a damaged bare repo (probably due to a powercut. Possibly during an operation, although not sure):

git --version git version 2.11.0

git fsck --full -v Checking HEAD link Checking object directory Checking tree 11bbc847cf1b4422b3e37830a9eac2e7af6559de Checking tree 11be4abeb20314de6145dfc0e6180807a74c03dc --->8 snip 8<--------------------------------------------------------------- Checking tree 14a4423e86f06c7ad75bf391d138e0cf7790508f Checking tree 147aeaec72b2f29bf1813494c942fbce497be679 zsh: segmentation fault  git fsck --full -v

Host: Debian 10. Linux 4.9.0-13-amd64 #1 SMP Debian 4.9.228-1 
(2020-07-05) x86_64 GNU/Linux

FS: BTRFS

I learnt about the damage while trying to `git push`  from my dev working tree. I think I did a `git pull` when the `git push` failed. The result of this was that my *local* repo was also damaged: there was an empty file created that would stop git operations and on deleting that I got a page full of errors.

Original SO question: https://stackoverflow.com/questions/64044995/can-one-corrupt-git-repo-corrupt-another?noredirect=1

Unfortunately I don't think I'm at liberty to give access to the repo as it possibly contains sensitive info/personal data.

I have solved the problem for myself (took a morning to do it!) but reporting here in case it's useful to do so. I use git all the time and I'm grateful for it. I'm not a systems-level coder, so not sure what use I'll be, but happy to answer any Qs or try things on the broken copies.

Thanks,

Rich

On Fri, Sep 25, 2020 at 04:48:39PM +0100, Rich wrote:

Thanks for the report. This is definitely the right place.

It's tough to say from this where the problem might be. If you still have the broken repo and can reproduce, two things that might help:

  - trying with a more recent version of Git; we've fixed quite a few
    segfaults around corrupted data in the past few years

  - getting a backtrace; if you can build Git from source, the simplest
    thing is just running "gdb --args /path/to/your/git fsck --full -v",
    waiting for it to crash, and then running "bt" in the debugger

That's concerning; push/pull generally try hard not to let corruption spread. What transport do you use between the two servers? The usual git-over-ssh and git-over-http protocols should be pretty resilient, but I would not be surprised if the old dumb-http protocol, which just downloads remote files wholesale, would be confused by a zero-length loose object file or similar.

-Peff

Hi Jeff,

Thanks for a friendly and welcoming response (I was a bit scared posting here I suppose because the project's creator has a rep for not suffering fools!).

I copied the repo that causes a segfault on 2.11.0 to my laptop which runs 2.25.1 and I'm pleased to report that there's no segfault. I've posted the errors it does show below this message in case they're useful.

The two servers are connected by SSH. As in:

% git config remote.origin.url sshalias:path/to/repo

where sshalias is defined in my .ssh/config file.

To test again,I made a bare clone of the fixed repo on my 2.25.1 host. I still had some dangling trees/blobs so I ran the following to make sure we were all clean:

git reflog expire --expire-unreachable=now --all git gc --prune=now git fsck --full

Then I added the broken repo as a remote (ssh) repo and ran fetch, which crashed (presumably correctly):

% git fetch broke error: object file ./objects/3e/9c7cd338d2ea3603d0c23b4b56902a04937833 is empty error: object file ./objects/3e/9c7cd338d2ea3603d0c23b4b56902a04937833 is empty fatal: loose object 3e9c7cd338d2ea3603d0c23b4b56902a04937833 (stored in ./objects/3e/9c7cd338d2ea3603d0c23b4b56902a04937833) is corrupt fatal: the remote end hung up unexpectedly

I re-ran fsck and it was still all clean. So 2.25.1 seems safe.I repeated the same procedure on the 2.11.0 version (Debian 10 host), and the same thing happened.

So I'm not sure what happened with the two repos getting broken at once; can't repeat it.

Let's hope that this was something that was fixed as you say!

Thanks, Rich

## Errors shown when running fsck on 2.25.1 on the broken repo:

git --version git version % git fsck --full error: object file ./objects/14/2d73242bba1e4aee89a887ec37d3b1ef00461c is empty error: unable to mmap ./objects/14/2d73242bba1e4aee89a887ec37d3b1ef00461c: No such file or directory error: 142d73242bba1e4aee89a887ec37d3b1ef00461c: object corrupt or missing: ./objects/14/2d73242bba1e4aee89a887ec37d3b1ef00461c error: object file ./objects/3e/9c7cd338d2ea3603d0c23b4b56902a04937833 is empty error: unable to mmap ./objects/3e/9c7cd338d2ea3603d0c23b4b56902a04937833: No such file or directory error: 3e9c7cd338d2ea3603d0c23b4b56902a04937833: object corrupt or missing: ./objects/3e/9c7cd338d2ea3603d0c23b4b56902a04937833 error: object file ./objects/45/aaa8e78abffa54ee2a620ef453d2b73443a3f9 is empty error: unable to mmap ./objects/45/aaa8e78abffa54ee2a620ef453d2b73443a3f9: No such file or directory error: 45aaa8e78abffa54ee2a620ef453d2b73443a3f9: object corrupt or missing: ./objects/45/aaa8e78abffa54ee2a620ef453d2b73443a3f9 error: object file ./objects/51/c79dda59372c0bd44457fbf810c66a6b184bb4 is empty error: unable to mmap ./objects/51/c79dda59372c0bd44457fbf810c66a6b184bb4: No such file or directory error: 51c79dda59372c0bd44457fbf810c66a6b184bb4: object corrupt or missing: ./objects/51/c79dda59372c0bd44457fbf810c66a6b184bb4 error: object file ./objects/92/92c90c7f01b5f9252a4631071d8ed39b121d52 is empty error: unable to mmap ./objects/92/92c90c7f01b5f9252a4631071d8ed39b121d52: No such file or directory error: 9292c90c7f01b5f9252a4631071d8ed39b121d52: object corrupt or missing: ./objects/92/92c90c7f01b5f9252a4631071d8ed39b121d52 error: object file ./objects/a0/1c6d43bb04b86b8263848c9d8daa20d4a68be4 is empty error: unable to mmap ./objects/a0/1c6d43bb04b86b8263848c9d8daa20d4a68be4: No such file or directory error: a01c6d43bb04b86b8263848c9d8daa20d4a68be4: object corrupt or missing: ./objects/a0/1c6d43bb04b86b8263848c9d8daa20d4a68be4 error: object file ./objects/c6/480c5f0be01c6860a17123bef79aa3fb98dd8b is empty error: unable to mmap ./objects/c6/480c5f0be01c6860a17123bef79aa3fb98dd8b: No such file or directory error: c6480c5f0be01c6860a17123bef79aa3fb98dd8b: object corrupt or missing: ./objects/c6/480c5f0be01c6860a17123bef79aa3fb98dd8b error: object file ./objects/e7/e8f34edee57ed77b240fa024df2f472dd9776d is empty error: unable to mmap ./objects/e7/e8f34edee57ed77b240fa024df2f472dd9776d: No such file or directory error: e7e8f34edee57ed77b240fa024df2f472dd9776d: object corrupt or missing: ./objects/e7/e8f34edee57ed77b240fa024df2f472dd9776d Checking object directories: 100% (256/256), done. Checking objects: 100% (310949/310949), done. error: object file ./objects/3e/9c7cd338d2ea3603d0c23b4b56902a04937833 is empty error: object file ./objects/3e/9c7cd338d2ea3603d0c23b4b56902a04937833 is empty fatal: loose object 3e9c7cd338d2ea3603d0c23b4b56902a04937833 (stored in ./objects/3e/9c7cd338d2ea3603d0c23b4b56902a04937833) is corrupt