# [PATCH] Enable core.fsyncObjectFiles by default 8 messages from 2015-06-23 to 2015-06-24. Participants: Stefan Beller, Junio C Hamano, Theodore Ts'o, Duy Nguyen, Jeff King. Thread: https://gitlist.dev/t/39705 ## Stefan Beller, 2015-06-23 21:57 Subject: [PATCH] Enable core.fsyncObjectFiles by default Message-ID: <1435096643-18159-1-git-send-email-sbeller@google.com> URL: https://gitlist.dev/e/1435096643-18159-1-git-send-email-sbeller%40google.com ``` Linus Torvalds started a discussion[1] if we want to play rather safe than use defaults which make sense only for the most power users of Git: > So git is "safe" in the sense that you won't really lose any data, > but you may well be inconvenienced. The "fsync each object" config > option is there in case you don't want that inconvenience, but it > should be noted that it can make for a hell of a performance impact. > Of course, it might well be the case that the actual default > might be worth turning around. Most git users probably don't > care about that kind of "apply two hundred patches from Andrew > Morton" kind of workload, although "rebase a big patch-series" > does end up doing basically the same thing, and might be more > common. This patch enables fsync_object_files by default. [1] https://plus.google.com/u/1/+JonathanCorbet/posts/JBxiKPe3VXa Signed-off-by: Stefan Beller --- Documentation/config.txt | 8 ++++---- environment.c | 2 +- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/Documentation/config.txt b/Documentation/config.txt index 43bb53c..dce2640 100644 --- a/Documentation/config.txt +++ b/Documentation/config.txt @@ -693,10 +693,10 @@ core.whitespace:: core.fsyncObjectFiles:: This boolean will enable 'fsync()' when writing object files. + -This is a total waste of time and effort on a filesystem that orders -data writes properly, but can be useful for filesystems that do not use -journalling (traditional UNIX filesystems) or that only journal metadata -and not file contents (OS X's HFS+, or Linux ext3 with "data=writeback"). +This ensures objects are written to disk instead of relying on the +operating systems cache and eventual write. Disabling this option will +yield performance with a trade off in safety for repository corruption +during power loss. core.preloadIndex:: Enable parallel index preload for operations like 'git diff' diff --git a/environment.c b/environment.c index 61c685b..b406f5e 100644 --- a/environment.c +++ b/environment.c @@ -35,7 +35,7 @@ const char *git_attributes_file; int zlib_compression_level = Z_BEST_SPEED; int core_compression_level; int core_compression_seen; -int fsync_object_files; +int fsync_object_files = 1; size_t packed_git_window_size = DEFAULT_PACKED_GIT_WINDOW_SIZE; size_t packed_git_limit = DEFAULT_PACKED_GIT_LIMIT; size_t delta_base_cache_limit = 96 * 1024 * 1024; -- 2.4.1.345.gab207b6.dirty ``` ## Junio C Hamano, 2015-06-23 22:21 Subject: Re: [PATCH] Enable core.fsyncObjectFiles by default Message-ID: URL: https://gitlist.dev/e/xmqqpp4maww7.fsf%40gitster.dls.corp.google.com In-Reply-To: <1435096643-18159-1-git-send-email-sbeller@google.com> ``` Stefan Beller writes: > Linus Torvalds started a discussion[1] if we want to play rather safe > than use defaults which make sense only for the most power users of Git: > >> So git is "safe" in the sense that you won't really lose any data, >> but you may well be inconvenienced. The "fsync each object" config >> option is there in case you don't want that inconvenience, but it >> should be noted that it can make for a hell of a performance impact. > >> Of course, it might well be the case that the actual default >> might be worth turning around. Most git users probably don't >> care about that kind of "apply two hundred patches from Andrew >> Morton" kind of workload, although "rebase a big patch-series" >> does end up doing basically the same thing, and might be more >> common. > > This patch enables fsync_object_files by default. Sorry, but I fail to see which part of what Linus said (which is the only thing you quoted from the discussion) argues for this patch. If anything, I read that as cautioning people against making a tradeoff based on an incorrect perception of risks and blindly flipping this bit ON (the original discussion a few days ago, where Ted says he has this bit ON while clarifying that he does so on SSD, is also a sensible description on how he made his trade-off). It's a different matter whom you would want to align with when assessing your own risk tolerance. If you quoted Corbet's original message, then that would have been more consistent. > > [1] https://plus.google.com/u/1/+JonathanCorbet/posts/JBxiKPe3VXa > > Signed-off-by: Stefan Beller > --- > Documentation/config.txt | 8 ++++---- > environment.c | 2 +- > 2 files changed, 5 insertions(+), 5 deletions(-) > > diff --git a/Documentation/config.txt b/Documentation/config.txt > index 43bb53c..dce2640 100644 > --- a/Documentation/config.txt > +++ b/Documentation/config.txt > @@ -693,10 +693,10 @@ core.whitespace:: > core.fsyncObjectFiles:: > This boolean will enable 'fsync()' when writing object files. > + > -This is a total waste of time and effort on a filesystem that orders > -data writes properly, but can be useful for filesystems that do not use > -journalling (traditional UNIX filesystems) or that only journal metadata > -and not file contents (OS X's HFS+, or Linux ext3 with "data=writeback"). > +This ensures objects are written to disk instead of relying on the > +operating systems cache and eventual write. Disabling this option will > +yield performance with a trade off in safety for repository corruption > +during power loss. > > core.preloadIndex:: > Enable parallel index preload for operations like 'git diff' > diff --git a/environment.c b/environment.c > index 61c685b..b406f5e 100644 > --- a/environment.c > +++ b/environment.c > @@ -35,7 +35,7 @@ const char *git_attributes_file; > int zlib_compression_level = Z_BEST_SPEED; > int core_compression_level; > int core_compression_seen; > -int fsync_object_files; > +int fsync_object_files = 1; > size_t packed_git_window_size = DEFAULT_PACKED_GIT_WINDOW_SIZE; > size_t packed_git_limit = DEFAULT_PACKED_GIT_LIMIT; > size_t delta_base_cache_limit = 96 * 1024 * 1024; ``` ## Theodore Ts'o, 2015-06-23 23:29 Subject: Re: [PATCH] Enable core.fsyncObjectFiles by default Message-ID: <20150623232921.GB14324@thunk.org> URL: https://gitlist.dev/e/20150623232921.GB14324%40thunk.org In-Reply-To: ``` The main issue is that non-expert users might not realize that they really need to run "git fsck" after a crash; otherwise, what might happen is that although git is only appending, that you might have some zero-length (or partially written) git object or pack files, and while you might not notice at that moment, it might come and bite you later. If you do try to recover immediately after a crash, in the worst case you might have to do that "git am -s /tmp/mbox-filled-with patches" command, but otherwise you won't lose much data. So perhaps one alternative strategy to calling fsync(2) after every single git object file write might be to have git create a zero-length ".git/in-progress-" file, which gets fsync'ed, and then it can do the "git am -s /tmp/akpm-big-bag-o-patches" processing nice and fast, and once git is done, then we call call sync(2) and then delete the in-progress file. If there is an in-progress file in the .git directory, git would then automatically run git fsck to make sure the repository is consistent. For people who care, maybe that's a good compromise. (Me, the way things are right now is just fine since I have a nice fast SSD, and so setting fsyncObjectfiles is a perfectly fine thing as far as I am concerned. :-) - Ted ``` ## Duy Nguyen, 2015-06-24 01:07 Subject: Re: [PATCH] Enable core.fsyncObjectFiles by default Message-ID: URL: https://gitlist.dev/e/CACsJy8AZ7TOqZLKUD%3DtVaBVMH4af_jg42Y04ZryQ9dhTSoGu5A%40mail.gmail.com In-Reply-To: <1435096643-18159-1-git-send-email-sbeller@google.com> ``` On Wed, Jun 24, 2015 at 4:57 AM, Stefan Beller wrote: > Linus Torvalds started a discussion[1] if we want to play rather safe > than use defaults which make sense only for the most power users of Git: > >> So git is "safe" in the sense that you won't really lose any data, >> but you may well be inconvenienced. The "fsync each object" config >> option is there in case you don't want that inconvenience, but it >> should be noted that it can make for a hell of a performance impact. > >> Of course, it might well be the case that the actual default >> might be worth turning around. Most git users probably don't >> care about that kind of "apply two hundred patches from Andrew >> Morton" kind of workload, although "rebase a big patch-series" >> does end up doing basically the same thing, and might be more >> common. > > This patch enables fsync_object_files by default. Will this make nfs performance a lot worse or still within acceptable range? -- Duy ``` ## Jeff King, 2015-06-24 03:37 Subject: Re: [PATCH] Enable core.fsyncObjectFiles by default Message-ID: <20150624033725.GA19321@peff.net> URL: https://gitlist.dev/e/20150624033725.GA19321%40peff.net In-Reply-To: <1435096643-18159-1-git-send-email-sbeller@google.com> ``` On Tue, Jun 23, 2015 at 02:57:23PM -0700, Stefan Beller wrote: > Linus Torvalds started a discussion[1] if we want to play rather safe > than use defaults which make sense only for the most power users of Git: > > > So git is "safe" in the sense that you won't really lose any data, > > but you may well be inconvenienced. The "fsync each object" config > > option is there in case you don't want that inconvenience, but it > > should be noted that it can make for a hell of a performance impact. > > > Of course, it might well be the case that the actual default > > might be worth turning around. Most git users probably don't > > care about that kind of "apply two hundred patches from Andrew > > Morton" kind of workload, although "rebase a big patch-series" > > does end up doing basically the same thing, and might be more > > common. > > This patch enables fsync_object_files by default. If you are looking for safety out of the box, I think this falls far short, as we do not fsync all of the other files. For instance, we do not fsync refs before they are written (nor anything else that uses the commit_lock_file() interface to rename, such as the index). We do always fsync packfiles and their indices. I had always assumed this was fine on ext4 with data=ordered (i.e., either the rename and its pointed-to content will go through, or not; so you either get your update or the old state, but not a garbage or empty file). But it sounds from what Ted wrote in: http://article.gmane.org/gmane.linux.file-systems/97255 that this may not be the case. If it's not, I think we should consider fsyncing ref writes. -Peff ``` ## Junio C Hamano, 2015-06-24 05:20 Subject: Re: [PATCH] Enable core.fsyncObjectFiles by default Message-ID: URL: https://gitlist.dev/e/xmqqlhf9bs2b.fsf%40gitster.dls.corp.google.com In-Reply-To: <20150624033725.GA19321@peff.net> ``` Jeff King writes: > I had always assumed this was fine on ext4 with data=ordered (i.e., > either the rename and its pointed-to content will go through, or not; so > you either get your update or the old state, but not a garbage or empty > file). But it sounds from what Ted wrote in: > > http://article.gmane.org/gmane.linux.file-systems/97255 > > that this may not be the case. If it's not, I think we should consider > fsyncing ref writes. That matches my understanding. IIRC, we do fsync (without a knob to turn it off) packs, we do not fsync refs (as you described above), the index, or working tree files (via write_entry()). Among these: * If we are so paranoid that loss of new loose objects matter (as opposed to "Heh, it is just the matter of 'git add' again") to cause us to set core.fsyncObjectFiles, we should definitely fsync ref writes. They are even more precious. * The index is much less precious. In the worst case, you can 'git reset' (no flags) and re-add from the working tree and nothing unrecoverable is lost. I do not mind a knob to force us fsync, but we may want it to be separate from core.fsyncObjectFiles. * The working tree files? I am not sure. Again, recovery is just to run "git diff" to notice what was eaten by the power failure and then "git checkout $path" to restore from a known state, so... ``` ## Junio C Hamano, 2015-06-24 05:32 Subject: Re: [PATCH] Enable core.fsyncObjectFiles by default Message-ID: URL: https://gitlist.dev/e/xmqqegl1brjb.fsf%40gitster.dls.corp.google.com In-Reply-To: <20150623232921.GB14324@thunk.org> ``` Theodore Ts'o writes: > The main issue is that non-expert users might not realize that they > really need to run "git fsck" after a crash; otherwise, what might > happen is that although git is only appending, that you might have > some zero-length (or partially written) git object or pack files, and > while you might not notice at that moment, it might come and bite you > later. Regarding loose object files, given that we write to a temporary, optionally fsync, close and then move to the final name, would we still see partially written file if we omit the fsync, or would the corruption be limited to either empty or missing? The reason I am wondering is because the codepath to create an object (i.e. "update-index --add", "hash-object -w", or "add") first checks if a packed or a loose object file _exists_ and if so bypasses writing the same thing anew, but the existence check for a loose object is to merely making sure that access(F_OK) (and optionally utime()) succeeds. If the potential breakage is limited to truncation to empty, then we could replace it with stat(2) and st.st_size check, as no loose object file can be empty. ``` ## Theodore Ts'o, 2015-06-24 14:30 Subject: Re: [PATCH] Enable core.fsyncObjectFiles by default Message-ID: <20150624143007.GC14324@thunk.org> URL: https://gitlist.dev/e/20150624143007.GC14324%40thunk.org In-Reply-To: ``` On Tue, Jun 23, 2015 at 10:32:08PM -0700, Junio C Hamano wrote: > > Regarding loose object files, given that we write to a temporary, > optionally fsync, close and then move to the final name, would we > still see partially written file if we omit the fsync, or would the > corruption be limited to either empty or missing? *Most* of the time the corruption will be an empty or missing file. It's possible that the file could be partially written. This is a relatively low-probability event, with the probability going up if the object file is large, and/or if the system is under memory pressure. > The reason I am wondering is because the codepath to create an > object (i.e. "update-index --add", "hash-object -w", or "add") first > checks if a packed or a loose object file _exists_ and if so > bypasses writing the same thing anew, but the existence check for a > loose object is to merely making sure that access(F_OK) (and > optionally utime()) succeeds. If the potential breakage is limited > to truncation to empty, then we could replace it with stat(2) and > st.st_size check, as no loose object file can be empty. It would certainly be a good thing to do a st_size check; it can't possible hurt, and it will catch a large number of failures after a power failure. I could also imagine some hueristics that force an fsync if the object file is larger than a certain size (say, 4k if you are very paranoid, a few hundred kilobytes if you are less so), but past a certain point, it might be better just to tell the user to use fsyncObjectFiles and be done with it. - Ted ```