{"thread":{"id":"39682","subject":"broken repo after power cut","startedAt":"2015-06-20T19:40:38Z","lastAt":"2015-06-22T12:31:35Z","messageCount":8,"participants":["Richard Weinberger","Johannes Schindelin","Christoph Hellwig","Theodore Ts'o"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"264412","messageId":"5585C1B6.50407@nod.at","threadId":"39682","inReplyTo":null,"subject":"broken repo after power cut","fromName":"Richard Weinberger","fromEmail":"richard@nod.at","sentAt":"2015-06-20T19:40:38Z","receivedAt":"2015-06-20T19:40:38Z","isPatch":false,"sender":{"key":"richard@nod.at","avatar":"https://avatars.githubusercontent.com/u/1149549?v=4"},"body":"Hi!\n\nYesterday our git server faced a power cut and a git repository broke.\nThe server is running a ext4 filesystem on top of Linux 3.16 (stable from openSUSE) and git 2.1.4.\nWe had a backup, so no data was lost but I really would like to figure out\nwhat happened.\n\nThis is the output of git fsck:\nChecking object directories: 100% (256/256), done.\nerror: object file objects/ce/f7627fc160ad7294b1f728db0c1ddb65a38b1d is empty\nerror: object file objects/ce/f7627fc160ad7294b1f728db0c1ddb65a38b1d is empty\nfatal: loose object cef7627fc160ad7294b1f728db0c1ddb65a38b1d (stored in objects/ce/f7627fc160ad7294b1f728db0c1ddb65a38b1d) is corrupt\n\nTo me it seems like git was creating a new object and got interrupted before fsync/fdatasync'ing it.\nAs the object was referenced before syncing the data to disk the repo broke.\nCould this have happened?\nAlso, is git designed to survive power cuts? Then referencing an object before synching it do disk would make no sense.\n\nThanks,\n//richard\n"},{"id":"264428","messageId":"330ab8f498e1b435d5b210384200b649@www.dscho.org","threadId":"39682","inReplyTo":"5585C1B6.50407@nod.at","subject":"Re: broken repo after power cut","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2015-06-21T12:28:20Z","receivedAt":"2015-06-21T12:28:20Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Richard,\n\nOn 2015-06-20 21:40, Richard Weinberger wrote:\n\n> Yesterday our git server faced a power cut and a git repository broke.\n> The server is running a ext4 filesystem on top of Linux 3.16 (stable\n> from openSUSE) and git 2.1.4.\n> We had a backup, so no data was lost but I really would like to figure out\n> what happened.\n> \n> This is the output of git fsck:\n> Checking object directories: 100% (256/256), done.\n> error: object file objects/ce/f7627fc160ad7294b1f728db0c1ddb65a38b1d is empty\n> error: object file objects/ce/f7627fc160ad7294b1f728db0c1ddb65a38b1d is empty\n> fatal: loose object cef7627fc160ad7294b1f728db0c1ddb65a38b1d (stored\n> in objects/ce/f7627fc160ad7294b1f728db0c1ddb65a38b1d) is corrupt\n> \n> To me it seems like git was creating a new object and got interrupted\n> before fsync/fdatasync'ing it.\n> As the object was referenced before syncing the data to disk the repo broke.\n> Could this have happened?\n> Also, is git designed to survive power cuts? Then referencing an\n> object before synching it do disk would make no sense.\n\nI had similar issues with ext4 in the past, even with local repositories when using Git without pushing. My then-current laptop would not report battery power correctly, so I ran into out-of-power situations that would result in a loose object file that was simply empty, i.e. its length was zero. As far as my analysis back then went, this was not Git's fault, because its `write_loose_object()` function would write to a temporary file first and only move that file into place once it was written fully.\n\nI was then shocked to learn that ext4 apparently has a default setting that allows it to truncate files upon power failure (something about a full journal vs a fast journal or some such) when I had expected the default to be a true journaled file system with proper atomicity regarding writes and moves. I remember that back then, I angrily fixed that setting to make my file system fully journaled.\n\nMaybe this leads you into the direction of a work-around in your setup?\n\nCiao,\nJohannes\n"},{"id":"264433","messageId":"5586B71D.2070407@nod.at","threadId":"39682","inReplyTo":"330ab8f498e1b435d5b210384200b649@www.dscho.org","subject":"Re: broken repo after power cut","fromName":"Richard Weinberger","fromEmail":"richard@nod.at","sentAt":"2015-06-21T13:07:41Z","receivedAt":"2015-06-21T13:07:41Z","isPatch":false,"sender":{"key":"richard@nod.at","avatar":"https://avatars.githubusercontent.com/u/1149549?v=4"},"body":"Hi Johannes,\n\n[CC'ing linux-fsdevel and tytso]\n\nAm 21.06.2015 um 14:28 schrieb Johannes Schindelin:\n> Hi Richard,\n> \n> On 2015-06-20 21:40, Richard Weinberger wrote:\n> \n>> Yesterday our git server faced a power cut and a git repository broke.\n>> The server is running a ext4 filesystem on top of Linux 3.16 (stable\n>> from openSUSE) and git 2.1.4.\n>> We had a backup, so no data was lost but I really would like to figure out\n>> what happened.\n>>\n>> This is the output of git fsck:\n>> Checking object directories: 100% (256/256), done.\n>> error: object file objects/ce/f7627fc160ad7294b1f728db0c1ddb65a38b1d is empty\n>> error: object file objects/ce/f7627fc160ad7294b1f728db0c1ddb65a38b1d is empty\n>> fatal: loose object cef7627fc160ad7294b1f728db0c1ddb65a38b1d (stored\n>> in objects/ce/f7627fc160ad7294b1f728db0c1ddb65a38b1d) is corrupt\n>>\n>> To me it seems like git was creating a new object and got interrupted\n>> before fsync/fdatasync'ing it.\n>> As the object was referenced before syncing the data to disk the repo broke.\n>> Could this have happened?\n>> Also, is git designed to survive power cuts? Then referencing an\n>> object before synching it do disk would make no sense.\n> \n> I had similar issues with ext4 in the past, even with local repositories when using Git without pushing. My then-current laptop would not report battery power correctly, so I ran into out-of-power situations that would result in a loose object file that was simply empty, i.e. its length was zero. As far as my analysis back then went, this was not Git's fault, because its `write_loose_object()` function would write to a temporary file first and only move that file into place once it was written fully.\n> \n> I was then shocked to learn that ext4 apparently has a default setting that allows it to truncate files upon power failure (something about a full journal vs a fast journal or some such) when I had expected the default to be a true journaled file system with proper atomicity regarding writes and moves. I remember that back then, I angrily fixed that setting to make my file system fully journaled.\n\nYou mean the ext4 delayed block allocation feature/issue?\nIIRC Ted added some hacks to ext4 to detect misbehaving applications (Gnome and KDE).\nBut to my knowledge such an file corruption must not happen if the application behaves well. And it can happen on all file systems.\nTed, maybe you can help us? BTW: I'm using ext4's default mount options from openSUSE, data=ordered.\n\n> Maybe this leads you into the direction of a work-around in your setup?\n\nI'm still not sure who to blame. ;-)\n\nThanks,\n//richard\n--\nTo unsubscribe from this list: send the line \"unsubscribe linux-fsdevel\" in\n"},{"id":"264437","messageId":"20150621135903.GA18719@infradead.org","threadId":"39682","inReplyTo":"5586B71D.2070407@nod.at","subject":"Re: broken repo after power cut","fromName":"Christoph Hellwig","fromEmail":"hch@infradead.org","sentAt":"2015-06-21T13:59:03Z","receivedAt":"2015-06-21T13:59:03Z","isPatch":false,"sender":{"key":"hch@infradead.org","avatar":null},"body":"On Sun, Jun 21, 2015 at 03:07:41PM +0200, Richard Weinberger wrote:\n> >> To me it seems like git was creating a new object and got interrupted\n> >> before fsync/fdatasync'ing it.\n> >> As the object was referenced before syncing the data to disk the repo broke.\n\nGit doesn't fsync by default, and because of that I've seen similar\ndata losses on ext4/xfs/btrfs.\n\nYou can set the core.fsyncobjectfiles to mitigate it, but even with\nthat I've seen corrupted index files.\n\nNote that I've been mostly on old git versions from various distros,\nso in case this was fixed recently I'll take everything I said back.\n"},{"id":"264440","messageId":"5586C54F.1090705@nod.at","threadId":"39682","inReplyTo":"20150621135903.GA18719@infradead.org","subject":"Re: broken repo after power cut","fromName":"Richard Weinberger","fromEmail":"richard@nod.at","sentAt":"2015-06-21T14:08:15Z","receivedAt":"2015-06-21T14:08:15Z","isPatch":false,"sender":{"key":"richard@nod.at","avatar":"https://avatars.githubusercontent.com/u/1149549?v=4"},"body":"Am 21.06.2015 um 15:59 schrieb Christoph Hellwig:\n> On Sun, Jun 21, 2015 at 03:07:41PM +0200, Richard Weinberger wrote:\n>>>> To me it seems like git was creating a new object and got interrupted\n>>>> before fsync/fdatasync'ing it.\n>>>> As the object was referenced before syncing the data to disk the repo broke.\n> \n> Git doesn't fsync by default, and because of that I've seen similar\n> data losses on ext4/xfs/btrfs.\n> \n> You can set the core.fsyncobjectfiles to mitigate it, but even with\n> that I've seen corrupted index files.\n\nYeah, after inspecting git's source I've found that config option too.\nNow it's also crystal clear that git is not power cut safe at all by default. ;-\\\n\nSo, anyone that cares about his repos has to enable core.fsyncobjectfiles,\nwhich is IMHO kind of sad.\n\nThanks,\n//richard\n--\nTo unsubscribe from this list: send the line \"unsubscribe linux-fsdevel\" in\n"},{"id":"264480","messageId":"20150622003551.GP29480@thunk.org","threadId":"39682","inReplyTo":"5586B71D.2070407@nod.at","subject":"Re: broken repo after power cut","fromName":"Theodore Ts'o","fromEmail":"tytso@mit.edu","sentAt":"2015-06-22T00:35:51Z","receivedAt":"2015-06-22T00:35:51Z","isPatch":false,"sender":{"key":"tytso@mit.edu","avatar":"https://avatars.githubusercontent.com/u/51416?v=4"},"body":"On Sun, Jun 21, 2015 at 03:07:41PM +0200, Richard Weinberger wrote:\n\n> > I was then shocked to learn that ext4 apparently has a default\n> > setting that allows it to truncate files upon power failure\n> > (something about a full journal vs a fast journal or some such)\n\ns/ext4/all modern file systems/\n\nPOSIX makes **no guarantees** about what happens after a power failure\nunless you use fsync() --- which git does not do by default (see below).\n\n> You mean the ext4 delayed block allocation feature/issue?\n> IIRC Ted added some hacks to ext4 to detect misbehaving applications (Gnome and KDE).\n> But to my knowledge such an file corruption must not happen if the application behaves well. And it can happen on all file systems.\n> Ted, maybe you can help us? BTW: I'm using ext4's default mount options from openSUSE, data=ordered.\n\nThe hacks (which were agreed upon by all of the major file system\ndevelopers --- ext4, btfs, xfs --- at the Linux File Systems and\nStorage summit a couple of years ago --- protects against the default\ntext editors of GNOME and KDE which were saving file without using\nfsync(), and in one particularly egregious example (although I don't\nremember which program was doing this), updated files by opening the\nfile with O_TRUNC and then rewritng the new contents of the file.  So\nif you crashed just after the open(2), and before the file data was\nwritten, you were guaranteed to lose data.\n\nThe hack protects against data loss when programs updated a file\nincompetently.  What we agreed to do was that upon renaming a fileA on\ntop of another fileB, there is an implicit writeback initiated of\nfileA.  If the program properly called fsync(2) before closing the\nfile descriptor for fileA and doing the rename, this implicit\nwriteback would be no-op.  Simiarly, if a file descriptor was opened\nwith O_TRUNC, when the file descriptor is closed, we start an implicit\nwriteback at that point.  Note that this is not the same as a full\nfsync; it merely closes the race window from 30 seconds down to a\nsecond or so (depending on how busy the disk is).\n\nBut this hack does not protect against freshly written files, which is\nthe case of git object files or git pack files.  The basic idea here\nis that you could have just as easily crashed before the commit as\nafter the commit, and doing an implicit writeback after all file\ncloses would have destroyed performance and penalized progams that\ndidn't really care so much about the file hitting disk.  (For example,\nif you do a compile, and you crash, it's not such a big deal.)\n\nThe bottome lins is that if you care about files being written, you\nneed to use fsync().  Should git use fsync() by default?  Well, if you\nare willing to accept that if your system crashes within a second or\nso of your last git operation, you might need to run \"git fsck\" and\npotentially recover from a busted repo, maybe speed is more important\nfor you (and git is known for its speed/performance, after all. :-)\n\nThe actual state of the source tree would have been written using a\ntext editor which tends to be paranoid about using fsync (at least, if\nyou use a real editor like Emacs or Vi, as opposed to the toy notepad\neditors shipped with GNOME or KDE :-).  So as long as you know what\nyou're doing, it's unlikely that you will actually lose any work.\n\nPersonally, I have core.fsyncobjectfiles set to yes in my .gitconfig.\nPart of this is because I have an SSD, so the speed hit really doesn't\nbother me, and needing to recover a corrupted git repository is a pain\n(although I have certainly done it in the past).\n\n\t\t\t\t\t\t- Ted\n--\nTo unsubscribe from this list: send the line \"unsubscribe linux-fsdevel\" in\n"},{"id":"264506","messageId":"5587EF5F.90207@nod.at","threadId":"39682","inReplyTo":"20150622003551.GP29480@thunk.org","subject":"Re: broken repo after power cut","fromName":"Richard Weinberger","fromEmail":"richard@nod.at","sentAt":"2015-06-22T11:19:59Z","receivedAt":"2015-06-22T11:19:59Z","isPatch":false,"sender":{"key":"richard@nod.at","avatar":"https://avatars.githubusercontent.com/u/1149549?v=4"},"body":"Am 22.06.2015 um 02:35 schrieb Theodore Ts'o:\n> On Sun, Jun 21, 2015 at 03:07:41PM +0200, Richard Weinberger wrote:\n> \n>>> I was then shocked to learn that ext4 apparently has a default\n>>> setting that allows it to truncate files upon power failure\n>>> (something about a full journal vs a fast journal or some such)\n> \n> s/ext4/all modern file systems/\n> \n> POSIX makes **no guarantees** about what happens after a power failure\n> unless you use fsync() --- which git does not do by default (see below).\n\nThanks for pointing this out.\n\n> The bottome lins is that if you care about files being written, you\n> need to use fsync().  Should git use fsync() by default?  Well, if you\n> are willing to accept that if your system crashes within a second or\n> so of your last git operation, you might need to run \"git fsck\" and\n> potentially recover from a busted repo, maybe speed is more important\n> for you (and git is known for its speed/performance, after all. :-)\n> \n> The actual state of the source tree would have been written using a\n> text editor which tends to be paranoid about using fsync (at least, if\n> you use a real editor like Emacs or Vi, as opposed to the toy notepad\n> editors shipped with GNOME or KDE :-).  So as long as you know what\n> you're doing, it's unlikely that you will actually lose any work.\n> \n> Personally, I have core.fsyncobjectfiles set to yes in my .gitconfig.\n> Part of this is because I have an SSD, so the speed hit really doesn't\n> bother me, and needing to recover a corrupted git repository is a pain\n> (although I have certainly done it in the past).\n\nI think core.fsyncObjectFiles documentation really needs an update.\nWhat about this one?\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex 43bb53c..b08fa11 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -693,10 +693,16 @@ core.whitespace::\n core.fsyncObjectFiles::\n \tThis boolean will enable 'fsync()' when writing object files.\n +\n-This is a total waste of time and effort on a filesystem that orders\n-data writes properly, but can be useful for filesystems that do not use\n-journalling (traditional UNIX filesystems) or that only journal metadata\n-and not file contents (OS X's HFS+, or Linux ext3 with \"data=writeback\").\n+For performance reasons git does not call 'fsync()' after writing object\n+files. This means that after a power cut your git repository can get\n+corrupted as not all data hit the storage media. Especially on modern\n+filesystems like ext4, xfs or btrfs this can happen very easily.\n+If you have to face power cuts and care about your data it is strongly\n+recommended to enable this setting.\n+Please note that git's behavior used to be safe on ext3 with data=ordered,\n+for any other filesystems or mount settings this is not the case as\n+POSIX clearly states that you have to call 'fsync()' to make sure that\n+all data is written.\n\n core.preloadIndex::\n \tEnable parallel index preload for operations like 'git diff'\n\n--\nThanks,\n//richard\n"},{"id":"264508","messageId":"20150622123135.GU29480@thunk.org","threadId":"39682","inReplyTo":"5587EF5F.90207@nod.at","subject":"Re: broken repo after power cut","fromName":"Theodore Ts'o","fromEmail":"tytso@mit.edu","sentAt":"2015-06-22T12:31:35Z","receivedAt":"2015-06-22T12:31:35Z","isPatch":false,"sender":{"key":"tytso@mit.edu","avatar":"https://avatars.githubusercontent.com/u/51416?v=4"},"body":"On Mon, Jun 22, 2015 at 01:19:59PM +0200, Richard Weinberger wrote:\n> \n> > The bottome lins is that if you care about files being written, you\n> > need to use fsync().  Should git use fsync() by default?  Well, if you\n> > are willing to accept that if your system crashes within a second or\n> > so of your last git operation, you might need to run \"git fsck\" and\n> > potentially recover from a busted repo, maybe speed is more important\n> > for you (and git is known for its speed/performance, after all. :-)\n\nI made a typo in the above.  s/second/minute/.  (Linux's writeback\ntimer is 30 seconds, but if the disk is busy it might take a bit\nlonger to get all of the data blocks written out to disk and\ncommitted.)\n\n> I think core.fsyncObjectFiles documentation really needs an update.\n> What about this one?\n> \n> diff --git a/Documentation/config.txt b/Documentation/config.txt\n> index 43bb53c..b08fa11 100644\n> --- a/Documentation/config.txt\n> +++ b/Documentation/config.txt\n> @@ -693,10 +693,16 @@ core.whitespace::\n>  core.fsyncObjectFiles::\n>  \tThis boolean will enable 'fsync()' when writing object files.\n>  +\n> -This is a total waste of time and effort on a filesystem that orders\n> -data writes properly, but can be useful for filesystems that do not use\n> -journalling (traditional UNIX filesystems) or that only journal metadata\n> -and not file contents (OS X's HFS+, or Linux ext3 with \"data=writeback\").\n> +For performance reasons git does not call 'fsync()' after writing object\n> +files. This means that after a power cut your git repository can get\n> +corrupted as not all data hit the storage media. Especially on modern\n> +filesystems like ext4, xfs or btrfs this can happen very easily.\n> +If you have to face power cuts and care about your data it is strongly\n> +recommended to enable this setting.\n> +Please note that git's behavior used to be safe on ext3 with data=ordered,\n> +for any other filesystems or mount settings this is not the case as\n> +POSIX clearly states that you have to call 'fsync()' to make sure that\n> +all data is written.\n\n\nMy main complaint about this is that it's a bit Linux-centric.  For\nexample, the fact that fsync(2) is needed to push data out of the\ncache is also true for MacOS (and indeed all other Unix systems going\nback three decades) as well as Windows.  In fact, it's not a matter of\n\"POSIX says\", but \"POSIX documented\", but since standards are held in\nhigh esteem, it's sometimes a bit more convenient to use them as an\nappeal to authority.  :-)\n\n(Ext3's data=ordered behaviour is an outlier, and in fact, the reason\nwhy it mostly safe to skip fsync(2) calls when using ext3 data=ordered\nwas an accidental side effect of another problem which was trying to\nsolve based on the relatively primitive way it handled block\nallocation.)\n\nCheers,\n\n\t\t\t\t\t\t- Ted\n--\nTo unsubscribe from this list: send the line \"unsubscribe linux-fsdevel\" in\n"}]}