git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH v2 05/18] maintenance: add commit-graph task

From
Derrick Stolee <stolee@gmail.com>
Date
Jul 30, 2020, 13:59 UTC
Message-ID
<b20e2e46-ff71-b84f-19da-9d8873ef08c3@gmail.com>
In-Reply-To
<20200725015240.GD35171@syl.lan>
On 7/24/2020 9:52 PM, Taylor Blau wrote:
Show 36 quoted lines
> On Fri, Jul 24, 2020 at 12:47:00PM -0700, Junio C Hamano wrote:
>> Derrick Stolee <stolee@gmail.com> writes:
>>
>>> But you are discussing here how the _behavior_ can change when
>>> --auto is specified. And specifically, "git gc --auto" really
>>> meant "This is running after a foreground command, so only do
>>> work if necessary and do it quickly to minimize blocking time."
>>>
>>> I'd be happy to replace "--auto" with "--quick" in the
>>> maintenance builtin.
>>>
>>> This opens up some extra design space for how the individual
>>> tasks perform depending on "--quick" being specified or not.
>>> My intention was to create tasks that are already in "quick"
>>> mode:
>>>
>>> * loose-objects have a maximum batch size.
>>> * incremental-repack is capped in size.
>>> * commit-graph uses the --split option.
>>>
>>> But this "quick" distinction might be important for some of
>>> the tasks we intend to extract from the gc builtin.
>>
>> Yup.  To be honest, I came to this topic from a completely different
>> direction.  The field name "auto" alone (and no other field name)
>> had to have an extra cruft (i.e. "_flag") attached to it, which is
>> understandable but ugly.  Then I started thinking if 'auto(matic)'
>> is really the right word to describe what we want out of the option,
>> and came to the realization that there may be better words.
> 
> I wonder what the quick and slow paths are here. For the commit-graph
> code, what you wrote here seems to match what I'd expect with passing
> '--auto' in the sense of running 'git gc'. That is, I'm leaving it up to
> the commit-graph machinery's idea of the normal '--split' rules to
> figure out when to roll up layers of a commit-graph, as opposed to
> creating a new layer and extending the chain.

I had intended all of my new tasks to be the "quick" version of their operations. The "slow" version would abandon hope of doing a small amount of work to create the best possible world for the repository. This would include:

 * The commit-graph would collapse all layers into one file.
 * The multi-pack-index repack would rewrite all object data into one
   pack-file.
 * The loose-objects task would not stop at a maximum number of loose
   objects (and would probably want to repack everything, anyway).

I'm open to making this possibility more explicit by renaming "--auto" and just performing a translation to 'git gc --auto'. So, what should the name be? Here are a few options to consider:

 --quick
 --fast
 --limited
 --incremental
 -O[0|1|2...] (think GCC optimization flags, exposing granularity)
 --[non-]aggressive

Regardless, this makes me rethink that the --[no-]maintenance option from PATCH 03/18 is better than --[no-]auto-maintenance, since we are really saying "run _some_ maintenance or _no_ maintenance" and the "how" of the maintenance is left intentionally vague. I've already made the change locally to add "auto-" so I'll wait for confirmation before reverting that change.

Show 15 quoted lines
> So, I think that makes sense if the caller gave '--auto'. But, I'm not
> sure that it makes sense if they didn't, in which case I'd imagine
> something quicker to happen. There, I'd expect something more like:
> 
>   1. Run 'git commit-graph write --reachable --split=no-merge'.
>   2. Run 'git commit-graph verify'.
>   3. If 'git commit-graph verify' failed, drop the existing commit graph
>      and rebuild it with 'git commit-graph --reachable --split=replace'.
>   4. Otherwise, do nothing.
> 
> I'm biased, of course, but I think that that matches roughly what I'd
> expect to happen in the fast/slow path. Granted, the steps to rebuild
> the commit graph are going to be slow no matter what (depending on the
> size of the repository), and so in that case maybe the commit-graph
> should just be dropped. I'm not really sure what to do about that...

I think this approach is the best we can do given the current behavior inside the commit-graph builtin. Perhaps in the future we could change the commit-graph builtin to include a "--verify" option so it could do the "git commit-graph verify --shallow" on the new layer before committing the new commit-graph-chain file and expiring old layers. That way, we would not need to delete and rewrite the whole thing when there is a problem writing the top layer.

Show 15 quoted lines
>>> Since the tasks are frequently running subcommands, returning
>>> 0 for success and non-zero for error matches the error codes
>>> returned by those subcommands.
>>
>> As long as these will _never_ be called from other helper functions
>> but from the cmd_foo() top-level and their return values are only
>> used directly as the top-level's return value, I do not mind too
>> much.
>>
>> But whenever I am writing such a code, I find myself not brave
>> enough to make such a bold promise (I saw other people call the
>> helpers I wrote in unintended ways and had to adjust the semantics
>> of them to accomodate the new callers too many times), so I'd rather
>> see the caller do "return !!helper_fn()" to allow helper_fn() to be
>> written more naturally (e.g. letting them return error(...)).
I will try to be consistent here with the behavior:
 * 0 is success
 * 1 is failure
Which is what I think you are implying by "return !!helper_fn()".

Thanks, -Stolee

Previous: Taylor BlauNext: Jeff King
Message 86 of 164 in “Maintenance builtin, allowing 'gc --auto' customization”
  1. 00/21 Maintenance builtin, allowing 'gc --auto' customizationDerrick Stolee via GitGitGadget, Jul 7, 2020
  2. 01/21 gc: use the_repository less oftenDerrick Stolee via GitGitGadget, Jul 7, 2020
  3. 02/21 gc: use repository in too_many_loose_objects()Derrick Stolee via GitGitGadget, Jul 7, 2020
  4. 03/21 gc: use repo configDerrick Stolee via GitGitGadget, Jul 7, 2020
  5. 05/21 maintenance: create basic maintenance runnerDerrick Stolee via GitGitGadget, Jul 7, 2020
  6. 06/21 maintenance: add --quiet optionDerrick Stolee via GitGitGadget, Jul 7, 2020
  7. 08/21 maintenance: initialize task array and hashmapDerrick Stolee via GitGitGadget, Jul 7, 2020
  8. Jonathan TanJul 9, 2020
  9. Derrick StoleeJul 9, 2020
  10. Junio C HamanoJul 9, 2020
  11. 07/21 maintenance: replace run_auto_gc()Derrick Stolee via GitGitGadget, Jul 7, 2020
  12. 09/21 maintenance: add commit-graph taskDerrick Stolee via GitGitGadget, Jul 7, 2020
  13. Jonathan TanJul 9, 2020
  14. Derrick StoleeJul 9, 2020
  15. Jeff KingJul 9, 2020
  16. Derrick StoleeJul 9, 2020
  17. 10/21 maintenance: add --task optionDerrick Stolee via GitGitGadget, Jul 7, 2020
  18. 11/21 maintenance: take a lock on the objects directoryDerrick Stolee via GitGitGadget, Jul 7, 2020
  19. 17/21 maintenance: use pointers to check --autoDerrick Stolee via GitGitGadget, Jul 7, 2020
  20. 19/21 maintenance: create auto condition for loose-objectsDerrick Stolee via GitGitGadget, Jul 7, 2020
  21. 20/21 maintenance: add pack-files auto conditionDerrick Stolee via GitGitGadget, Jul 7, 2020
  22. 21/21 midx: use start_delayed_progress()Derrick Stolee via GitGitGadget, Jul 7, 2020
  23. 16/21 maintenance: create maintenance.<task>.enabled configDerrick Stolee via GitGitGadget, Jul 7, 2020
  24. 18/21 maintenance: add auto condition for commit-graph taskDerrick Stolee via GitGitGadget, Jul 7, 2020
  25. 13/21 maintenance: add loose-objects taskDerrick Stolee via GitGitGadget, Jul 7, 2020
  26. 15/21 maintenance: auto-size pack-files batchDerrick Stolee via GitGitGadget, Jul 7, 2020
  27. 14/21 maintenance: add pack-files taskDerrick Stolee via GitGitGadget, Jul 7, 2020
  28. 04/21 gc: drop the_repository in log locationDerrick Stolee via GitGitGadget, Jul 7, 2020
  29. Jonathan TanJul 9, 2020
  30. Derrick StoleeJul 9, 2020
  31. 12/21 maintenance: add fetch taskDerrick Stolee via GitGitGadget, Jul 7, 2020
  32. Jonathan TanJul 9, 2020
  33. Emily ShafferJul 8, 2020
  34. Derrick StoleeJul 9, 2020
  35. Derrick StoleeJul 9, 2020
  36. Jeff KingJul 9, 2020
  37. Derrick StoleeJul 9, 2020
  38. Emily ShafferJul 10, 2020
  39. Son Luong NgocJul 10, 2020
  40. Junio C HamanoJul 9, 2020
  41. Derrick StoleeJul 9, 2020
  42. Junio C HamanoJul 9, 2020
  43. Derrick StoleeJul 9, 2020
  44. 00/18 Maintenance builtin, allowing 'gc --auto' customizationDerrick Stolee via GitGitGadget, Jul 23, 2020
  45. 01/18 maintenance: create basic maintenance runnerDerrick Stolee via GitGitGadget, Jul 23, 2020
  46. Taylor BlauJul 25, 2020
  47. Đoàn Trần Công DanhJul 25, 2020
  48. Jonathan NiederJul 29, 2020
  49. Derrick StoleeJul 30, 2020
  50. Jonathan NiederJul 31, 2020
  51. Derrick StoleeAug 3, 2020
  52. Jonathan NiederAug 3, 2020
  53. Taylor BlauAug 3, 2020
  54. Jonathan NiederAug 3, 2020
  55. Taylor BlauAug 3, 2020
  56. Jonathan NiederAug 3, 2020
  57. Junio C HamanoAug 4, 2020
  58. Derrick StoleeAug 4, 2020
  59. Jonathan NiederAug 4, 2020
  60. Derrick StoleeAug 4, 2020
  61. Jonathan NiederAug 4, 2020
  62. Derrick StoleeAug 4, 2020
  63. Derrick StoleeAug 5, 2020
  64. Jonathan NiederJul 31, 2020
  65. Jonathan NiederAug 3, 2020
  66. 02/18 maintenance: add --quiet optionDerrick Stolee via GitGitGadget, Jul 23, 2020
  67. 03/18 maintenance: replace run_auto_gc()Derrick Stolee via GitGitGadget, Jul 23, 2020
  68. Junio C HamanoJul 23, 2020
  69. Taylor BlauJul 25, 2020
  70. Derrick StoleeJul 30, 2020
  71. Derrick StoleeJul 30, 2020
  72. Eric SunshineJul 30, 2020
  73. Derrick StoleeJul 30, 2020
  74. 04/18 maintenance: initialize task arrayDerrick Stolee via GitGitGadget, Jul 23, 2020
  75. Junio C HamanoJul 23, 2020
  76. Derrick StoleeJul 24, 2020
  77. Derrick StoleeJul 24, 2020
  78. Junio C HamanoJul 24, 2020
  79. Taylor BlauJul 25, 2020
  80. Emily ShafferJul 29, 2020
  81. 05/18 maintenance: add commit-graph taskDerrick Stolee via GitGitGadget, Jul 23, 2020
  82. Junio C HamanoJul 23, 2020
  83. Derrick StoleeJul 24, 2020
  84. Junio C HamanoJul 24, 2020
  85. Taylor BlauJul 25, 2020
  86. Derrick StoleeJul 30, 2020
  87. Jeff KingJul 29, 2020
  88. 07/18 maintenance: take a lock on the objects directoryDerrick Stolee via GitGitGadget, Jul 23, 2020
  89. 06/18 maintenance: add --task optionDerrick Stolee via GitGitGadget, Jul 23, 2020
  90. Junio C HamanoJul 23, 2020
  91. Junio C HamanoJul 23, 2020
  92. Derrick StoleeJul 24, 2020
  93. Junio C HamanoJul 24, 2020
  94. 08/18 maintenance: add prefetch taskDerrick Stolee via GitGitGadget, Jul 23, 2020
  95. Junio C HamanoJul 23, 2020
  96. Derrick StoleeJul 24, 2020
  97. Junio C HamanoJul 24, 2020
  98. Đoàn Trần Công DanhJul 25, 2020
  99. Junio C HamanoJul 25, 2020
  100. Derrick StoleeJul 27, 2020
  101. Junio C HamanoJul 27, 2020
  102. Derrick StoleeJul 27, 2020
  103. fetch: optionally allow disabling FETCH_HEAD updateJunio C Hamano, Jul 28, 2020
  104. Phillip WoodJul 29, 2020
  105. Phillip WoodJul 29, 2020
  106. Derrick StoleeJul 30, 2020
  107. 09/18 maintenance: add loose-objects taskDerrick Stolee via GitGitGadget, Jul 23, 2020
  108. Junio C HamanoJul 23, 2020
  109. Derrick StoleeJul 24, 2020
  110. Junio C HamanoJul 24, 2020
  111. Emily ShafferJul 29, 2020
  112. Derrick StoleeJul 30, 2020
  113. 10/18 maintenance: add incremental-repack taskDerrick Stolee via GitGitGadget, Jul 23, 2020
  114. Junio C HamanoJul 23, 2020
  115. Derrick StoleeJul 24, 2020
  116. Emily ShafferJul 29, 2020
  117. 11/18 maintenance: auto-size incremental-repack batchDerrick Stolee via GitGitGadget, Jul 23, 2020
  118. Junio C HamanoJul 23, 2020
  119. Eric SunshineJul 23, 2020
  120. Junio C HamanoJul 23, 2020
  121. Derrick StoleeJul 24, 2020
  122. Derrick StoleeJul 24, 2020
  123. Junio C HamanoJul 24, 2020
  124. Emily ShafferJul 29, 2020
  125. Derrick StoleeJul 30, 2020
  126. Derrick StoleeJul 30, 2020
  127. Chris TorekJul 30, 2020
  128. Đoàn Trần Công DanhAug 5, 2020
  129. Derrick StoleeAug 6, 2020
  130. 13/18 maintenance: use pointers to check --autoDerrick Stolee via GitGitGadget, Jul 23, 2020
  131. 14/18 maintenance: add auto condition for commit-graph taskDerrick Stolee via GitGitGadget, Jul 23, 2020
  132. 12/18 maintenance: create maintenance.<task>.enabled configDerrick Stolee via GitGitGadget, Jul 23, 2020
  133. 18/18 maintenance: add trace2 regions for task executionDerrick Stolee via GitGitGadget, Jul 23, 2020
  134. 17/18 midx: use start_delayed_progress()Derrick Stolee via GitGitGadget, Jul 23, 2020
  135. 16/18 maintenance: add incremental-repack auto conditionDerrick Stolee via GitGitGadget, Jul 23, 2020
  136. 15/18 maintenance: create auto condition for loose-objectsDerrick Stolee via GitGitGadget, Jul 23, 2020
  137. Emily ShafferJul 29, 2020
  138. 00/20 Maintenance builtin, allowing 'gc --auto' customizationDerrick Stolee via GitGitGadget, Jul 30, 2020
  139. 01/20 maintenance: create basic maintenance runnerDerrick Stolee via GitGitGadget, Jul 30, 2020
  140. 02/20 maintenance: add --quiet optionDerrick Stolee via GitGitGadget, Jul 30, 2020
  141. 04/20 maintenance: initialize task arrayDerrick Stolee via GitGitGadget, Jul 30, 2020
  142. 03/20 maintenance: replace run_auto_gc()Derrick Stolee via GitGitGadget, Jul 30, 2020
  143. 05/20 maintenance: add commit-graph taskDerrick Stolee via GitGitGadget, Jul 30, 2020
  144. 06/20 maintenance: add --task optionDerrick Stolee via GitGitGadget, Jul 30, 2020
  145. 08/20 fetch: optionally allow disabling FETCH_HEAD updateJunio C Hamano via GitGitGadget, Jul 30, 2020
  146. 07/20 maintenance: take a lock on the objects directoryDerrick Stolee via GitGitGadget, Jul 30, 2020
  147. 10/20 maintenance: add loose-objects taskDerrick Stolee via GitGitGadget, Jul 30, 2020
  148. 09/20 maintenance: add prefetch taskDerrick Stolee via GitGitGadget, Jul 30, 2020
  149. 11/20 midx: enable core.multiPackIndex by defaultDerrick Stolee via GitGitGadget, Jul 30, 2020
  150. 14/20 maintenance: create maintenance.<task>.enabled configDerrick Stolee via GitGitGadget, Jul 30, 2020
  151. 15/20 maintenance: use pointers to check --autoDerrick Stolee via GitGitGadget, Jul 30, 2020
  152. 13/20 maintenance: auto-size incremental-repack batchDerrick Stolee via GitGitGadget, Jul 30, 2020
  153. Chris TorekJul 30, 2020
  154. Derrick StoleeAug 3, 2020
  155. 12/20 maintenance: add incremental-repack taskDerrick Stolee via GitGitGadget, Jul 30, 2020
  156. 20/20 maintenance: add trace2 regions for task executionDerrick Stolee via GitGitGadget, Jul 30, 2020
  157. 18/20 maintenance: add incremental-repack auto conditionDerrick Stolee via GitGitGadget, Jul 30, 2020
  158. 17/20 maintenance: create auto condition for loose-objectsDerrick Stolee via GitGitGadget, Jul 30, 2020
  159. 19/20 midx: use start_delayed_progress()Derrick Stolee via GitGitGadget, Jul 30, 2020
  160. 16/20 maintenance: add auto condition for commit-graph taskDerrick Stolee via GitGitGadget, Jul 30, 2020
  161. Junio C HamanoJul 30, 2020
  162. Junio C HamanoJul 30, 2020
  163. Junio C HamanoJul 31, 2020
  164. Derrick StoleeAug 6, 2020

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.