git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [RFC PATCH 1/2] sha1-file: fsync() loose dir entry when core.fsyncObjectFiles

From
CHChristoph Hellwig <hch@lst.de>
Date
Sep 17, 2020, 15:09 UTC
Message-ID
<20200917150958.GA31693@lst.de>
In-Reply-To
<20200917131605.GC3024501@coredump.intra.peff.net>
On Thu, Sep 17, 2020 at 09:16:05AM -0400, Jeff King wrote:
Show 11 quoted lines
> I've also often wondered whether this is necessary. Given the symptom of
> "oops, this object is there but with 0 bytes" after a hard crash (power
> off, etc), my assumption is that the metadata is being journaled but the
> actual data is not. Which would imply this isn't needed, but may just be
> revealing my naive view of how filesystems work.
> 
> And of course all of my experience is on ext4 (which doubly confuses me,
> because my systems typically have data=ordered, which I thought would
> solve this). Non-journalling filesystems or other modes likely behave
> differently, but if this extra fsync carries a cost, we may want to make
> it optional.

I hope my other mail clarified how this works at a high level, if not feel free to ask more questions.

Show 13 quoted lines
> >  sha1-file.c | 19 ++++++++++++++-----
> >  1 file changed, 14 insertions(+), 5 deletions(-)
> 
> We already fsync pack files, but we don't fsync their directories. If
> this is important to do, we should be doing it there, too.
> 
> We also don't fsync ref files (nor packed-refs) at all. If fsyncing
> files is important for reliability, we should be including those, too.
> It may be tempting to say that the important stuff is in objects and the
> refs can be salvaged from the commit graph, but my experience says
> otherwise. Missing, broken, or mysteriously-rewound refs cause confusing
> user-visible behavior, and when compounded with pruning operations like
> "git gc" they _do_ result in losing objects.

True, this probably needs to do for the directories of other files as well.

One interesting optimization under linux is the syncfs syscall, that syncs all files on a file system - if you need to do a large number of fsyncs that do not depend on each other for transaction semantics it can provide a huge speedup.

Previous: Jeff KingNext: Christoph Hellwig
Message 38 of 52 in “enable core.fsyncObjectFiles by default”
  1. enable core.fsyncObjectFiles by defaultChristoph Hellwig, Jan 17, 2018
  2. Junio C HamanoJan 17, 2018
  3. Christoph HellwigJan 17, 2018
  4. Andreas SchwabJan 17, 2018
  5. Matthew WilcoxJan 17, 2018
  6. Christoph HellwigJan 17, 2018
  7. Ævar Arnfjörð BjarmasonJan 17, 2018
  8. Linus TorvaldsJan 17, 2018
  9. Linus TorvaldsJan 17, 2018
  10. Ævar Arnfjörð BjarmasonJan 17, 2018
  11. Linus TorvaldsJan 17, 2018
  12. Theodore Ts'oJan 17, 2018
  13. Linus TorvaldsJan 17, 2018
  14. Christoph HellwigJan 18, 2018
  15. Junio C HamanoJan 19, 2018
  16. Theodore Ts'oJan 20, 2018
  17. Junio C HamanoJan 20, 2018
  18. Ævar Arnfjörð BjarmasonJan 22, 2018
  19. Theodore Ts'oJan 22, 2018
  20. Jeff KingJan 23, 2018
  21. Theodore Ts'oJan 23, 2018
  22. Jeff KingJan 23, 2018
  23. Jeff KingJan 23, 2018
  24. Chris MasonJan 21, 2018
  25. Ævar Arnfjörð BjarmasonSep 17, 2020
  26. 2/2 core.fsyncObjectFiles: make the docs less flippantÆvar Arnfjörð Bjarmason, Sep 17, 2020
  27. Junio C HamanoSep 17, 2020
  28. Johannes SixtSep 17, 2020
  29. Johannes SchindelinOct 8, 2020
  30. Ævar Arnfjörð BjarmasonOct 8, 2020
  31. Junio C HamanoOct 8, 2020
  32. Johannes SchindelinOct 9, 2020
  33. Christoph HellwigSep 17, 2020
  34. Marc BranchaudSep 17, 2020
  35. 0/2 should core.fsyncObjectFiles fsync the dir entry + docsÆvar Arnfjörð Bjarmason, Sep 17, 2020
  36. 1/2 sha1-file: fsync() loose dir entry when core.fsyncObjectFilesÆvar Arnfjörð Bjarmason, Sep 17, 2020
  37. Jeff KingSep 17, 2020
  38. Christoph HellwigSep 17, 2020
  39. Christoph HellwigSep 17, 2020
  40. Jeff KingSep 17, 2020
  41. Christoph HellwigSep 17, 2020
  42. Junio C HamanoSep 17, 2020
  43. Jeff KingSep 17, 2020
  44. Taylor BlauSep 17, 2020
  45. Ævar Arnfjörð BjarmasonSep 22, 2020
  46. Johannes SixtSep 17, 2020
  47. Ævar Arnfjörð BjarmasonSep 22, 2020
  48. Johannes SchindelinNov 19, 2020
  49. Christoph HellwigSep 17, 2020
  50. Junio C HamanoSep 17, 2020
  51. Jeff KingJan 17, 2018
  52. Christoph HellwigJan 17, 2018

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.