git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: do we have too much fsync() configuration in 'next'? (was: [PATCH v7] core.fsync: documentation and user-friendly aggregate options)

From
Patrick Steinhardt <ps@pks.im>
Date
Mar 28, 2022, 10:56 UTC
Message-ID
<YkGUeQH4y1KIAdCc@ncase>
In-Reply-To
<220327.86y20veeua.gmgdl@evledraar.gmail.com>
On Sun, Mar 27, 2022 at 02:43:48PM +0200, Ævar Arnfjörð Bjarmason wrote:
Show 88 quoted lines
> 
> On Sat, Mar 26 2022, Neeraj Singh wrote:
> 
> > On Sat, Mar 26, 2022 at 8:34 AM Ævar Arnfjörð Bjarmason
> > <avarab@gmail.com> wrote:
> >>
> >>
> >> On Fri, Mar 25 2022, Neeraj Singh wrote:
> >>
> >> > On Fri, Mar 25, 2022 at 5:33 PM Ævar Arnfjörð Bjarmason
> >> > <avarab@gmail.com> wrote:
> [...]
> >> > I want to make a comment about the Index here.  Syncing the index is
> >> > strictly required for the "added" level of consistency, so that we
> >> > don't lose stuff that leaves the work tree but is staged.  But my
> >> > Windows enlistment has an index that's 266MB, which would be painful
> >> > to sync even with all the optimizations.  Maybe with split-index, this
> >> > wouldn't be so bad, but I just wanted to call out that some advanced
> >> > users may really care about the configurability.
> >>
> >> So for that use-case you'd like to fsync the loose objects (if any), but
> >> not the index? So the FS will "flush" up to the index, and then queue
> >> the index for later syncing to platter?
> >>
> >>
> >> But even in that case don't the settings need to be tied to one another,
> >> because in the method=bulk sync=index && sync=!loose case wouldn't we be
> >> syncing "loose" in any case?
> >>
> >> > As Git's various database implementations improve, the fsync stuff
> >> > will hopefully be more optimal and self-tuning.  But as that happens,
> >> > Git could just start ignoring settings that lose meaning without tying
> >> > anyones hands.
> >>
> >> Yeah that would alleviate most of my concerns here, but the docs aren't
> >> saying anything like that. Since you added them & they just landed, do
> >> you mind doing a small follow-up where we e.g. say that these new
> >> settings are "EXPERIMENTAL" or whatever, and subject to drastic change?
> >
> > The doc is already pretty prescriptive.  It has this line at the end
> > of the first  paragraph:
> > "Unless you
> > have special requirements, it is recommended that you leave
> > this option empty or pick one of `committed`, `added`,
> > or `all`."
> >
> > Those values are already designed to change as Git changes.
> 
> I'm referring to the documentation as it stands not being marked as
> experimental in the sense that we might decide to re-do this to a large
> extent, i.e. something like the diff I suggested upthread in
> https://lore.kernel.org/git/220323.86fsn8ohg8.gmgdl@evledraar.gmail.com/
> 
> So yes, I agree that it e.g. clearly states that you can add a new
> core.git=foobar or whatever down the line, but it clearly doesn't
> suggest that e.g. core.fsync might have boolean semantics in some later
> version, or that the rest might simply be ignored, even if that
> e.g. means that we wouldn't sync loose objects on
> core.fsync=loose-object, as we'd just warn with a "we don't provide this
> anymore".
> 
> Or do you disagree with that? IOW I mean that we'd do something like
> this, either in docs or code:
> 
> diff --git a/config.c b/config.c
> index 3c9b6b589ab..94548566073 100644
> --- a/config.c
> +++ b/config.c
> @@ -1675,6 +1675,9 @@ static int git_default_core_config(const char *var, const char *value, void *cb)
>  	}
>  
>  	if (!strcmp(var, "core.fsync")) {
> +		if (!the_repository->settings.feature_experimental)
> +			warning(_("the '%s' configuration option is EXPERIMENTAL. opt-in to use it with feature.experimental=true"),
> +				var);
>  		if (!value)
>  			return config_error_nonbool(var);
>  		fsync_components = parse_fsync_components(var, value);
> @@ -1682,6 +1685,9 @@ static int git_default_core_config(const char *var, const char *value, void *cb)
>  	}
>  
>  	if (!strcmp(var, "core.fsyncmethod")) {
> +		if (!the_repository->settings.feature_experimental)
> +			warning(_("the '%s' configuration option is EXPERIMENTAL. opt-in to use it with feature.experimental=true"),
> +				var);
>  		if (!value)
>  			return config_error_nonbool(var);
>  		if (!strcmp(value, "fsync"))

Let's please not tie this to `feature.experimental=true`. Setting that option has unintended sideeffects and will also change defaults which we may not want to have in production. I don't mind adding a warning in the docs though that the specific items which can be configured may be subject to change in the future.

At GitLab, we've got a three-step plan:
    1. We need to migrate to `core.fsync` in the first place. In order
       to not migrate and change behaviour at the same point in time we
       already benefit from the fine-grainedness of this config because
       we can simply say `core.fsync=loose-objects` and have the same
       behaviour as before with `core.fsyncLooseObjects=true`.
    2. We'll want to enable syncing of packfiles, which I think wasn't
       previously covered by `core.fsyncLooseobjects`.
    3. We'll add `refs` to also sync loose refs to disk.

So while the end result will be the same as `committed`, having this level of control helps us to assess the impact in a nicer way by being able to do this step by step with feature flags.

On the other hand, many of the other parts we don't really care about. Auxiliary metadata like the commit-graph or pack indices are data that can in the worst case be regenerated by us, so it's not clear to me whether it makes to also enable fsyncing those in production.

So altogether, I agree with Neeraj: having the fine-grainedness greatly helps us to roll out changes like this and be able to pick what we deem to be important. Personally I would be fine with explicitly pointing out that there are two groups of this config in our docs though:

    1. The "porcelain" group: "committed", "added", "all", "none". These
       are abstract groups whose behaviour should adapt as we change
       implementations, and are those that should typically be set by a
       user, if intended.
    2. The "plumbing" or "expert" group: these are fine-grained options
       which shouldn't typically be used by Git users. They still have
       merit though in hosting environments, where requirements are
       typically a lot more specific.

We may also provide different guarantees for both groups. The first one should definitely be stable, but we might state that the second group is subject to change in the future.

Patrick
Previous: Ævar Arnfjörð BjarmasonNext: Ævar Arnfjörð Bjarmason
Message 113 of 122 in “A design for future-proofing fsync() configuration”
  1. 0/2 A design for future-proofing fsync() configurationNeeraj K. Singh via GitGitGadget, Dec 4, 2021
  2. 1/2 fsync: add writeout-only mode for fsyncing repo dataNeeraj Singh via GitGitGadget, Dec 4, 2021
  3. Neeraj SinghDec 6, 2021
  4. 2/2 core.fsync: introduce granular fsync controlNeeraj Singh via GitGitGadget, Dec 4, 2021
  5. 0/3 A design for future-proofing fsync() configurationNeeraj K. Singh via GitGitGadget, Dec 7, 2021
  6. 1/3 core.fsyncmethod: add writeout-only modeNeeraj Singh via GitGitGadget, Dec 7, 2021
  7. Patrick SteinhardtDec 7, 2021
  8. Ævar Arnfjörð BjarmasonDec 7, 2021
  9. Neeraj SinghDec 7, 2021
  10. Ævar Arnfjörð BjarmasonDec 7, 2021
  11. Neeraj SinghDec 7, 2021
  12. 2/3 core.fsync: introduce granular fsync controlNeeraj Singh via GitGitGadget, Dec 7, 2021
  13. Patrick SteinhardtDec 7, 2021
  14. Neeraj SinghDec 7, 2021
  15. Ævar Arnfjörð BjarmasonDec 7, 2021
  16. Neeraj SinghDec 7, 2021
  17. Ævar Arnfjörð BjarmasonDec 8, 2021
  18. Neeraj SinghDec 9, 2021
  19. Junio C HamanoDec 9, 2021
  20. Ævar Arnfjörð BjarmasonDec 9, 2021
  21. Neeraj SinghDec 9, 2021
  22. Neeraj SinghJan 18, 2022
  23. Ævar Arnfjörð BjarmasonJan 19, 2022
  24. Ævar Arnfjörð BjarmasonJan 19, 2022
  25. Neeraj SinghJan 28, 2022
  26. 3/3 core.fsync: new option to harden the indexNeeraj Singh via GitGitGadget, Dec 7, 2021
  27. Patrick SteinhardtDec 7, 2021
  28. Neeraj SinghDec 8, 2021
  29. 0/4 A design for future-proofing fsync() configurationNeeraj K. Singh via GitGitGadget, Dec 9, 2021
  30. 1/4 core.fsyncmethod: add writeout-only modeNeeraj Singh via GitGitGadget, Dec 9, 2021
  31. 2/4 core.fsync: introduce granular fsync controlNeeraj Singh via GitGitGadget, Dec 9, 2021
  32. 3/4 core.fsync: new option to harden the indexNeeraj Singh via GitGitGadget, Dec 9, 2021
  33. 4/4 core.fsync: add a `derived-metadata` aggregate optionNeeraj Singh via GitGitGadget, Dec 9, 2021
  34. Neeraj SinghJan 8, 2022
  35. rsbecker@nexbridge.comJan 9, 2022
  36. Neeraj SinghJan 10, 2022
  37. 0/4 A design for future-proofing fsync() configurationNeeraj K. Singh via GitGitGadget, Feb 1, 2022
  38. 1/4 core.fsyncmethod: add writeout-only modeNeeraj Singh via GitGitGadget, Feb 1, 2022
  39. 2/4 core.fsync: introduce granular fsync controlNeeraj Singh via GitGitGadget, Feb 1, 2022
  40. Junio C HamanoFeb 2, 2022
  41. Junio C HamanoFeb 2, 2022
  42. Neeraj SinghFeb 11, 2022
  43. Junio C HamanoFeb 11, 2022
  44. Neeraj SinghFeb 11, 2022
  45. Junio C HamanoFeb 11, 2022
  46. rsbecker@nexbridge.comFeb 12, 2022
  47. Patrick SteinhardtFeb 14, 2022
  48. Junio C HamanoFeb 14, 2022
  49. Patrick SteinhardtMar 9, 2022
  50. Ævar Arnfjörð BjarmasonMar 9, 2022
  51. Junio C HamanoMar 9, 2022
  52. Patrick SteinhardtMar 10, 2022
  53. Junio C HamanoMar 10, 2022
  54. Neeraj SinghMar 9, 2022
  55. Neeraj SinghFeb 11, 2022
  56. 3/4 core.fsync: new option to harden the indexNeeraj Singh via GitGitGadget, Feb 1, 2022
  57. 4/4 core.fsync: add a `derived-metadata` aggregate optionNeeraj Singh via GitGitGadget, Feb 1, 2022
  58. 0/5 A design for future-proofing fsync() configurationNeeraj K. Singh via GitGitGadget, Mar 9, 2022
  59. 1/5 wrapper: move inclusion of CSPRNG headers the wrapper.c fileNeeraj Singh via GitGitGadget, Mar 9, 2022
  60. Junio C HamanoMar 9, 2022
  61. Neeraj SinghMar 10, 2022
  62. brian m. carlsonMar 10, 2022
  63. Neeraj SinghMar 10, 2022
  64. 2/5 core.fsyncmethod: add writeout-only modeNeeraj Singh via GitGitGadget, Mar 9, 2022
  65. Junio C HamanoMar 9, 2022
  66. 3/5 core.fsync: introduce granular fsync controlNeeraj Singh via GitGitGadget, Mar 9, 2022
  67. Junio C HamanoMar 10, 2022
  68. Neeraj SinghMar 10, 2022
  69. Junio C HamanoMar 10, 2022
  70. Neeraj SinghMar 10, 2022
  71. Junio C HamanoMar 10, 2022
  72. Junio C HamanoMar 10, 2022
  73. Neeraj SinghMar 10, 2022
  74. Junio C HamanoMar 10, 2022
  75. Johannes SchindelinMar 10, 2022
  76. Junio C HamanoMar 10, 2022
  77. 4/5 core.fsync: new option to harden the indexNeeraj Singh via GitGitGadget, Mar 9, 2022
  78. 5/5 core.fsync: documentation and user-friendly aggregate optionsNeeraj Singh via GitGitGadget, Mar 9, 2022
  79. Future-proofed syncing of refsPatrick Steinhardt, Mar 10, 2022
  80. 6/8 core.fsync: add `fsync_component()` wrapper which doesn't diePatrick Steinhardt, Mar 10, 2022
  81. Junio C HamanoMar 10, 2022
  82. Neeraj SinghMar 10, 2022
  83. 7/8 core.fsync: new option to harden loose referencesPatrick Steinhardt, Mar 10, 2022
  84. Junio C HamanoMar 10, 2022
  85. Neeraj SinghMar 10, 2022
  86. Neeraj SinghMar 10, 2022
  87. Junio C HamanoMar 11, 2022
  88. Patrick SteinhardtMar 11, 2022
  89. Ævar Arnfjörð BjarmasonMar 11, 2022
  90. 8/8 core.fsync: new option to harden packed referencesPatrick Steinhardt, Mar 10, 2022
  91. Junio C HamanoMar 10, 2022
  92. Patrick SteinhardtMar 11, 2022
  93. 0/6 A design for future-proofing fsync() configurationNeeraj K. Singh via GitGitGadget, Mar 10, 2022
  94. 1/6 wrapper: make inclusion of Windows csprng header tightly scopedNeeraj Singh via GitGitGadget, Mar 10, 2022
  95. 3/6 core.fsync: introduce granular fsync control infrastructureNeeraj Singh via GitGitGadget, Mar 10, 2022
  96. 4/6 core.fsync: add configuration parsingNeeraj Singh via GitGitGadget, Mar 10, 2022
  97. Jiang XinMar 28, 2022
  98. Neeraj SinghMar 28, 2022
  99. 5/6 core.fsync: new option to harden the indexNeeraj Singh via GitGitGadget, Mar 10, 2022
  100. 2/6 core.fsyncmethod: add writeout-only modeNeeraj Singh via GitGitGadget, Mar 10, 2022
  101. 6/6 core.fsync: documentation and user-friendly aggregate optionsNeeraj Singh via GitGitGadget, Mar 10, 2022
  102. core.fsync: documentation and user-friendly aggregate optionsNeeraj Singh, Mar 15, 2022
  103. Junio C HamanoMar 15, 2022
  104. Neeraj SinghMar 15, 2022
  105. do we have too much fsync() configuration in 'next'? (was: [PATCH v7] core.fsync: documentation and user-friendly aggregate options)Ævar Arnfjörð Bjarmason, Mar 23, 2022
  106. Neeraj SinghMar 25, 2022
  107. Ævar Arnfjörð BjarmasonMar 26, 2022
  108. Junio C HamanoMar 26, 2022
  109. Neeraj SinghMar 26, 2022
  110. Ævar Arnfjörð BjarmasonMar 26, 2022
  111. Neeraj SinghMar 27, 2022
  112. Ævar Arnfjörð BjarmasonMar 27, 2022
  113. Patrick SteinhardtMar 28, 2022
  114. Ævar Arnfjörð BjarmasonMar 28, 2022
  115. Neeraj SinghMar 28, 2022
  116. Neeraj SinghMar 30, 2022
  117. Junio C HamanoMar 10, 2022
  118. Neeraj SinghMar 11, 2022
  119. Neeraj SinghMar 11, 2022
  120. Junio C HamanoMar 13, 2022
  121. core.fsync: new option to harden referencesPatrick Steinhardt, Mar 11, 2022
  122. SZEDER GáborMar 25, 2022

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.