git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[RFC PATCH 2/2] core.fsyncObjectFiles: make the docs less flippant

From
Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Date
Sep 17, 2020, 11:28 UTC
Message-ID
<20200917112830.26606-3-avarab@gmail.com>
In-Reply-To
<87sgbghdbp.fsf@evledraar.gmail.com>

As amusing as Linus's original prose[1] is here it doesn't really explain in any detail to the uninitiated why you would or wouldn't enable this, and the counter-intuitive reason for why git wouldn't fsync your precious data.

So elaborate (a lot) on why this may or may not be needed. This is my best-effort attempt to summarize the various points raised in the last ML[2] discussion about this.

1.  aafe9fbaf4 ("Add config option to enable 'fsync()' of object
    files", 2008-06-18)
2. https://lore.kernel.org/git/20180117184828.31816-1-hch@lst.de/
Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 Documentation/config/core.txt | 42 ++++++++++++++++++++++++++++++-----
 1 file changed, 36 insertions(+), 6 deletions(-)
diff --git a/Documentation/config/core.txt b/Documentation/config/core.txt
index 74619a9c03..5b47670c16 100644
--- a/Documentation/config/core.txt
+++ b/Documentation/config/core.txt
@@ -548,12 +548,42 @@ core.whitespace::
   errors. The default tab width is 8. Allowed values are 1 to 63.
 
 core.fsyncObjectFiles::
-	This boolean will enable 'fsync()' when writing object files.
-+
-This is a total waste of time and effort on a filesystem that orders
-data writes properly, but can be useful for filesystems that do not use
-journalling (traditional UNIX filesystems) or that only journal metadata
-and not file contents (OS X's HFS+, or Linux ext3 with "data=writeback").
+	This boolean will enable 'fsync()' when writing loose object
+	files. Both the file itself and its containng directory will
+	be fsynced.
++
+When git writes data any required object writes will precede the
+corresponding reference update(s). For example, a
+linkgit:git-receive-pack[1] accepting a push might write a pack or
+loose objects (depending on settings such as `transfer.unpackLimit`).
++
+Therefore on a journaled file system which ensures that data is
+flushed to disk in chronological order an fsync shouldn't be
+needed. The loose objects might be lost with a crash, but so will the
+ref update that would have referenced them. Git's own state in such a
+crash will remain consistent.
++
+This option exists because that assumption doesn't hold on filesystems
+where the data ordering is not preserved, such as on ext3 and ext4
+with "data=writeback". On such a filesystem the `rename()` that drops
+the new reference in place might be preserved, but the contents or
+directory entry for the loose object(s) might not have been synced to
+disk.
++
+Enabling this option might slow git down by a lot in some
+cases. E.g. in the case of a naïve bulk import tool which might create
+a million loose objects before a final ref update and `gc`. In other
+more common cases such as on a server being pushed to with default
+`transfer.unpackLimit` settings the difference might not be noticable.
++
+However, that's highly filesystem-dependent, on some filesystems
+simply calling fsync() might force an unrelated bulk background write
+to be serialized to disk. Such edge cases are the reason this option
+is off by default. That default setting might change in future
+versions.
++
+In older versions of git only the descriptor for the file itself was
+fsynced, not its directory entry.
 
 core.preloadIndex::
 	Enable parallel index preload for operations like 'git diff'
-- 
2.28.0.297.g1956fa8f8d
Previous: Ævar Arnfjörð BjarmasonNext: Junio C Hamano
Message 26 of 52 in “enable core.fsyncObjectFiles by default”
  1. enable core.fsyncObjectFiles by defaultChristoph Hellwig, Jan 17, 2018
  2. Junio C HamanoJan 17, 2018
  3. Christoph HellwigJan 17, 2018
  4. Andreas SchwabJan 17, 2018
  5. Matthew WilcoxJan 17, 2018
  6. Christoph HellwigJan 17, 2018
  7. Ævar Arnfjörð BjarmasonJan 17, 2018
  8. Linus TorvaldsJan 17, 2018
  9. Linus TorvaldsJan 17, 2018
  10. Ævar Arnfjörð BjarmasonJan 17, 2018
  11. Linus TorvaldsJan 17, 2018
  12. Theodore Ts'oJan 17, 2018
  13. Linus TorvaldsJan 17, 2018
  14. Christoph HellwigJan 18, 2018
  15. Junio C HamanoJan 19, 2018
  16. Theodore Ts'oJan 20, 2018
  17. Junio C HamanoJan 20, 2018
  18. Ævar Arnfjörð BjarmasonJan 22, 2018
  19. Theodore Ts'oJan 22, 2018
  20. Jeff KingJan 23, 2018
  21. Theodore Ts'oJan 23, 2018
  22. Jeff KingJan 23, 2018
  23. Jeff KingJan 23, 2018
  24. Chris MasonJan 21, 2018
  25. Ævar Arnfjörð BjarmasonSep 17, 2020
  26. 2/2 core.fsyncObjectFiles: make the docs less flippantÆvar Arnfjörð Bjarmason, Sep 17, 2020
  27. Junio C HamanoSep 17, 2020
  28. Johannes SixtSep 17, 2020
  29. Johannes SchindelinOct 8, 2020
  30. Ævar Arnfjörð BjarmasonOct 8, 2020
  31. Junio C HamanoOct 8, 2020
  32. Johannes SchindelinOct 9, 2020
  33. Christoph HellwigSep 17, 2020
  34. Marc BranchaudSep 17, 2020
  35. 0/2 should core.fsyncObjectFiles fsync the dir entry + docsÆvar Arnfjörð Bjarmason, Sep 17, 2020
  36. 1/2 sha1-file: fsync() loose dir entry when core.fsyncObjectFilesÆvar Arnfjörð Bjarmason, Sep 17, 2020
  37. Jeff KingSep 17, 2020
  38. Christoph HellwigSep 17, 2020
  39. Christoph HellwigSep 17, 2020
  40. Jeff KingSep 17, 2020
  41. Christoph HellwigSep 17, 2020
  42. Junio C HamanoSep 17, 2020
  43. Jeff KingSep 17, 2020
  44. Taylor BlauSep 17, 2020
  45. Ævar Arnfjörð BjarmasonSep 22, 2020
  46. Johannes SixtSep 17, 2020
  47. Ævar Arnfjörð BjarmasonSep 22, 2020
  48. Johannes SchindelinNov 19, 2020
  49. Christoph HellwigSep 17, 2020
  50. Junio C HamanoSep 17, 2020
  51. Jeff KingJan 17, 2018
  52. Christoph HellwigJan 17, 2018

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.