threads / patch / 39705

patchEnable core.fsyncObjectFiles by default

Subject: [PATCH] Enable core.fsyncObjectFiles by default

## tl;dr

8 messages between Jun 23, 2015 and Jun 24, 2015. Diffs are folded; open one to read it.

replies: 7people: 5as markdown or json

Stefan Beller· Jun 23, 2015, 21:57 UTC · lore

Linus Torvalds started a discussion[1] if we want to play rather safe than use defaults which make sense only for the most power users of Git:

> So git is "safe" in the sense that you won't really lose any data,
> but you may well be inconvenienced.  The "fsync each object" config
> option is there in case you don't want that inconvenience, but it
> should be noted that it can make for a hell of a performance impact.
Show 6 quoted lines
> Of course, it might well be the case that the actual default
> might be worth turning around. Most git users probably don't
> care about that kind of "apply two hundred patches from Andrew
> Morton" kind of workload, although "rebase a big patch-series"
> does end up doing basically the same thing, and might be more
> common.
This patch enables fsync_object_files by default.
[1] https://plus.google.com/u/1/+JonathanCorbet/posts/JBxiKPe3VXa
Signed-off-by: Stefan Beller <sbeller@google.com>
---
 Documentation/config.txt | 8 ++++----
 environment.c            | 2 +-
 2 files changed, 5 insertions(+), 5 deletions(-)
Show changes to 2 files +5 −5

Documentation/config.txt, environment.c

diff --git a/Documentation/config.txt b/Documentation/config.txt
index 43bb53c..dce2640 100644
--- a/Documentation/config.txt
+++ b/Documentation/config.txt
@@ -693,10 +693,10 @@ core.whitespace::
 core.fsyncObjectFiles::
 	This boolean will enable 'fsync()' when writing object files.
 +
-This is a total waste of time and effort on a filesystem that orders
-data writes properly, but can be useful for filesystems that do not use
-journalling (traditional UNIX filesystems) or that only journal metadata
-and not file contents (OS X's HFS+, or Linux ext3 with "data=writeback").
+This ensures objects are written to disk instead of relying on the
+operating systems cache and eventual write. Disabling this option will
+yield performance with a trade off in safety for repository corruption
+during power loss.
 
 core.preloadIndex::
 	Enable parallel index preload for operations like 'git diff'
diff --git a/environment.c b/environment.c
index 61c685b..b406f5e 100644
--- a/environment.c
+++ b/environment.c
@@ -35,7 +35,7 @@ const char *git_attributes_file;
 int zlib_compression_level = Z_BEST_SPEED;
 int core_compression_level;
 int core_compression_seen;
-int fsync_object_files;
+int fsync_object_files = 1;
 size_t packed_git_window_size = DEFAULT_PACKED_GIT_WINDOW_SIZE;
 size_t packed_git_limit = DEFAULT_PACKED_GIT_LIMIT;
 size_t delta_base_cache_limit = 96 * 1024 * 1024;
-- 
2.4.1.345.gab207b6.dirty
Junio C Hamano· Jun 23, 2015, 22:21 UTC · re: Stefan Beller · lore

Re: [PATCH] Enable core.fsyncObjectFiles by default

Stefan Beller <sbeller@google.com> writes:
Show 16 quoted lines
> Linus Torvalds started a discussion[1] if we want to play rather safe
> than use defaults which make sense only for the most power users of Git:
>
>> So git is "safe" in the sense that you won't really lose any data,
>> but you may well be inconvenienced.  The "fsync each object" config
>> option is there in case you don't want that inconvenience, but it
>> should be noted that it can make for a hell of a performance impact.
>
>> Of course, it might well be the case that the actual default
>> might be worth turning around. Most git users probably don't
>> care about that kind of "apply two hundred patches from Andrew
>> Morton" kind of workload, although "rebase a big patch-series"
>> does end up doing basically the same thing, and might be more
>> common.
>
> This patch enables fsync_object_files by default.

Sorry, but I fail to see which part of what Linus said (which is the only thing you quoted from the discussion) argues for this patch. If anything, I read that as cautioning people against making a tradeoff based on an incorrect perception of risks and blindly flipping this bit ON (the original discussion a few days ago, where Ted says he has this bit ON while clarifying that he does so on SSD, is also a sensible description on how he made his trade-off).

It's a different matter whom you would want to align with when assessing your own risk tolerance. If you quoted Corbet's original message, then that would have been more consistent.

Show 41 quoted lines
>
> [1] https://plus.google.com/u/1/+JonathanCorbet/posts/JBxiKPe3VXa
>
> Signed-off-by: Stefan Beller <sbeller@google.com>
> ---
>  Documentation/config.txt | 8 ++++----
>  environment.c            | 2 +-
>  2 files changed, 5 insertions(+), 5 deletions(-)
>
> diff --git a/Documentation/config.txt b/Documentation/config.txt
> index 43bb53c..dce2640 100644
> --- a/Documentation/config.txt
> +++ b/Documentation/config.txt
> @@ -693,10 +693,10 @@ core.whitespace::
>  core.fsyncObjectFiles::
>  	This boolean will enable 'fsync()' when writing object files.
>  +
> -This is a total waste of time and effort on a filesystem that orders
> -data writes properly, but can be useful for filesystems that do not use
> -journalling (traditional UNIX filesystems) or that only journal metadata
> -and not file contents (OS X's HFS+, or Linux ext3 with "data=writeback").
> +This ensures objects are written to disk instead of relying on the
> +operating systems cache and eventual write. Disabling this option will
> +yield performance with a trade off in safety for repository corruption
> +during power loss.
>  
>  core.preloadIndex::
>  	Enable parallel index preload for operations like 'git diff'
> diff --git a/environment.c b/environment.c
> index 61c685b..b406f5e 100644
> --- a/environment.c
> +++ b/environment.c
> @@ -35,7 +35,7 @@ const char *git_attributes_file;
>  int zlib_compression_level = Z_BEST_SPEED;
>  int core_compression_level;
>  int core_compression_seen;
> -int fsync_object_files;
> +int fsync_object_files = 1;
>  size_t packed_git_window_size = DEFAULT_PACKED_GIT_WINDOW_SIZE;
>  size_t packed_git_limit = DEFAULT_PACKED_GIT_LIMIT;
>  size_t delta_base_cache_limit = 96 * 1024 * 1024;
Theodore Ts'o· Jun 23, 2015, 23:29 UTC · re: Junio C Hamano · lore

Re: [PATCH] Enable core.fsyncObjectFiles by default

The main issue is that non-expert users might not realize that they really need to run "git fsck" after a crash; otherwise, what might happen is that although git is only appending, that you might have some zero-length (or partially written) git object or pack files, and while you might not notice at that moment, it might come and bite you later. If you do try to recover immediately after a crash, in the worst case you might have to do that "git am -s /tmp/mbox-filled-with patches" command, but otherwise you won't lose much data.

So perhaps one alternative strategy to calling fsync(2) after every single git object file write might be to have git create a zero-length ".git/in-progress-<pid>" file, which gets fsync'ed, and then it can do the "git am -s /tmp/akpm-big-bag-o-patches" processing nice and fast, and once git is done, then we call call sync(2) and then delete the in-progress file.

If there is an in-progress file in the .git directory, git would then automatically run git fsck to make sure the repository is consistent.

For people who care, maybe that's a good compromise. (Me, the way things are right now is just fine since I have a nice fast SSD, and so setting fsyncObjectfiles is a perfectly fine thing as far as I am concerned. :-)

		      		     	   - Ted
Junio C Hamano· Jun 24, 2015, 05:32 UTC · re: Theodore Ts'o · lore

Re: [PATCH] Enable core.fsyncObjectFiles by default

Theodore Ts'o <tytso@mit.edu> writes:
Show 6 quoted lines
> The main issue is that non-expert users might not realize that they
> really need to run "git fsck" after a crash; otherwise, what might
> happen is that although git is only appending, that you might have
> some zero-length (or partially written) git object or pack files, and
> while you might not notice at that moment, it might come and bite you
> later.

Regarding loose object files, given that we write to a temporary, optionally fsync, close and then move to the final name, would we still see partially written file if we omit the fsync, or would the corruption be limited to either empty or missing?

The reason I am wondering is because the codepath to create an object (i.e. "update-index --add", "hash-object -w", or "add") first checks if a packed or a loose object file _exists_ and if so bypasses writing the same thing anew, but the existence check for a loose object is to merely making sure that access(F_OK) (and optionally utime()) succeeds. If the potential breakage is limited to truncation to empty, then we could replace it with stat(2) and st.st_size check, as no loose object file can be empty.

Theodore Ts'o· Jun 24, 2015, 14:30 UTC · re: Junio C Hamano · lore

Re: [PATCH] Enable core.fsyncObjectFiles by default

On Tue, Jun 23, 2015 at 10:32:08PM -0700, Junio C Hamano wrote:
Show 5 quoted lines
> 
> Regarding loose object files, given that we write to a temporary,
> optionally fsync, close and then move to the final name, would we
> still see partially written file if we omit the fsync, or would the
> corruption be limited to either empty or missing?

*Most* of the time the corruption will be an empty or missing file. It's possible that the file could be partially written. This is a relatively low-probability event, with the probability going up if the object file is large, and/or if the system is under memory pressure.

Show 8 quoted lines
> The reason I am wondering is because the codepath to create an
> object (i.e. "update-index --add", "hash-object -w", or "add") first
> checks if a packed or a loose object file _exists_ and if so
> bypasses writing the same thing anew, but the existence check for a
> loose object is to merely making sure that access(F_OK) (and
> optionally utime()) succeeds.  If the potential breakage is limited
> to truncation to empty, then we could replace it with stat(2) and
> st.st_size check, as no loose object file can be empty.

It would certainly be a good thing to do a st_size check; it can't possible hurt, and it will catch a large number of failures after a power failure. I could also imagine some hueristics that force an fsync if the object file is larger than a certain size (say, 4k if you are very paranoid, a few hundred kilobytes if you are less so), but past a certain point, it might be better just to tell the user to use fsyncObjectFiles and be done with it.

						- Ted
Duy Nguyen· Jun 24, 2015, 01:07 UTC · re: Stefan Beller · lore

Re: [PATCH] Enable core.fsyncObjectFiles by default

On Wed, Jun 24, 2015 at 4:57 AM, Stefan Beller <sbeller@google.com> wrote:
Show 16 quoted lines
> Linus Torvalds started a discussion[1] if we want to play rather safe
> than use defaults which make sense only for the most power users of Git:
>
>> So git is "safe" in the sense that you won't really lose any data,
>> but you may well be inconvenienced.  The "fsync each object" config
>> option is there in case you don't want that inconvenience, but it
>> should be noted that it can make for a hell of a performance impact.
>
>> Of course, it might well be the case that the actual default
>> might be worth turning around. Most git users probably don't
>> care about that kind of "apply two hundred patches from Andrew
>> Morton" kind of workload, although "rebase a big patch-series"
>> does end up doing basically the same thing, and might be more
>> common.
>
> This patch enables fsync_object_files by default.
Will this make nfs performance a lot worse or still within acceptable range?
-- 
Duy
Jeff King· Jun 24, 2015, 03:37 UTC · re: Stefan Beller · lore

Re: [PATCH] Enable core.fsyncObjectFiles by default

On Tue, Jun 23, 2015 at 02:57:23PM -0700, Stefan Beller wrote:
Show 16 quoted lines
> Linus Torvalds started a discussion[1] if we want to play rather safe
> than use defaults which make sense only for the most power users of Git:
> 
> > So git is "safe" in the sense that you won't really lose any data,
> > but you may well be inconvenienced.  The "fsync each object" config
> > option is there in case you don't want that inconvenience, but it
> > should be noted that it can make for a hell of a performance impact.
> 
> > Of course, it might well be the case that the actual default
> > might be worth turning around. Most git users probably don't
> > care about that kind of "apply two hundred patches from Andrew
> > Morton" kind of workload, although "rebase a big patch-series"
> > does end up doing basically the same thing, and might be more
> > common.
> 
> This patch enables fsync_object_files by default.

If you are looking for safety out of the box, I think this falls far short, as we do not fsync all of the other files. For instance, we do not fsync refs before they are written (nor anything else that uses the commit_lock_file() interface to rename, such as the index). We do always fsync packfiles and their indices.

I had always assumed this was fine on ext4 with data=ordered (i.e., either the rename and its pointed-to content will go through, or not; so you either get your update or the old state, but not a garbage or empty file). But it sounds from what Ted wrote in:

  http://article.gmane.org/gmane.linux.file-systems/97255

that this may not be the case. If it's not, I think we should consider fsyncing ref writes.

-Peff
Junio C Hamano· Jun 24, 2015, 05:20 UTC · re: Jeff King · lore

Re: [PATCH] Enable core.fsyncObjectFiles by default

Jeff King <peff@peff.net> writes:
Show 9 quoted lines
> I had always assumed this was fine on ext4 with data=ordered (i.e.,
> either the rename and its pointed-to content will go through, or not; so
> you either get your update or the old state, but not a garbage or empty
> file). But it sounds from what Ted wrote in:
>
>   http://article.gmane.org/gmane.linux.file-systems/97255
>
> that this may not be the case. If it's not, I think we should consider
> fsyncing ref writes.

That matches my understanding. IIRC, we do fsync (without a knob to turn it off) packs, we do not fsync refs (as you described above), the index, or working tree files (via write_entry()). Among these:

 * If we are so paranoid that loss of new loose objects matter (as
   opposed to "Heh, it is just the matter of 'git add' again") to
   cause us to set core.fsyncObjectFiles, we should definitely fsync
   ref writes.  They are even more precious.
 * The index is much less precious.  In the worst case, you can 'git
   reset' (no flags) and re-add from the working tree and nothing
   unrecoverable is lost.  I do not mind a knob to force us fsync,
   but we may want it to be separate from core.fsyncObjectFiles.
 * The working tree files?  I am not sure.  Again, recovery is just
   to run "git diff" to notice what was eaten by the power failure
   and then "git checkout $path" to restore from a known state,
   so...

← back to recent threads