Re: broken repo after power cut

From: Johannes Schindelin <johannes.schindelin@gmx.de>
Date: Jun 21, 2015, 12:28 UTC
Message-ID: <330ab8f498e1b435d5b210384200b649@www.dscho.org>
In-Reply-To: <5585C1B6.50407@nod.at>

Hi Richard,

On 2015-06-20 21:40, Richard Weinberger wrote:

Show 19 quoted lines

> Yesterday our git server faced a power cut and a git repository broke.
> The server is running a ext4 filesystem on top of Linux 3.16 (stable
> from openSUSE) and git 2.1.4.
> We had a backup, so no data was lost but I really would like to figure out
> what happened.
> 
> This is the output of git fsck:
> Checking object directories: 100% (256/256), done.
> error: object file objects/ce/f7627fc160ad7294b1f728db0c1ddb65a38b1d is empty
> error: object file objects/ce/f7627fc160ad7294b1f728db0c1ddb65a38b1d is empty
> fatal: loose object cef7627fc160ad7294b1f728db0c1ddb65a38b1d (stored
> in objects/ce/f7627fc160ad7294b1f728db0c1ddb65a38b1d) is corrupt
> 
> To me it seems like git was creating a new object and got interrupted
> before fsync/fdatasync'ing it.
> As the object was referenced before syncing the data to disk the repo broke.
> Could this have happened?
> Also, is git designed to survive power cuts? Then referencing an
> object before synching it do disk would make no sense.

I had similar issues with ext4 in the past, even with local repositories when using Git without pushing. My then-current laptop would not report battery power correctly, so I ran into out-of-power situations that would result in a loose object file that was simply empty, i.e. its length was zero. As far as my analysis back then went, this was not Git's fault, because its `write_loose_object()` function would write to a temporary file first and only move that file into place once it was written fully.

I was then shocked to learn that ext4 apparently has a default setting that allows it to truncate files upon power failure (something about a full journal vs a fast journal or some such) when I had expected the default to be a true journaled file system with proper atomicity regarding writes and moves. I remember that back then, I angrily fixed that setting to make my file system fully journaled.

Maybe this leads you into the direction of a work-around in your setup?

Ciao, Johannes