git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [RFC/PATCH] gc: run more pre-detach operations under lock

From
Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Date
Jun 19, 2019, 18:01 UTC
Message-ID
<87k1dh8ne4.fsf@evledraar.gmail.com>
In-Reply-To
<CACsJy8AqA3TmNP62ko4c5Et39jsADYf9nKQByz28y-YQjNyKag@mail.gmail.com>
On Wed, Jun 19 2019, Duy Nguyen wrote:
Show 42 quoted lines
> On Wed, Jun 19, 2019 at 5:26 PM Ævar Arnfjörð Bjarmason
> <avarab@gmail.com> wrote:
>> This patch is part of a WIP branch I have that's a bit of a mess. It's
>> more-gc-detach-under-lock on github.com/avar/git.git. It doesn't apply
>> on master because it relies on some previous test work, but for RFC
>> purposes I figured it was better to send it stand-alone.
>>
>> But I think this sort of approach is better than Duy's proposed patch,
>> because...
>>
>> On Wed, Jun 19 2019, Nguyễn Thái Ngọc Duy wrote:
>>
>> > So let's try to avoid that. We should only need one 'gc' run after all
>> > objects and references are added anyway. Add a new option --no-auto-gc
>> > that will be used by those n-1 processes. 'gc --auto' will always run on
>> > the main fetch process (*).
>> >
>> > (*) even if we fetch remotes in parallel at some point in future, this
>> >     should still be fine because we should "join" all those processes
>> >     before this step.
>>
>> This is what I'm trying to fix in my version of this patch. This is
>> only true for yours if you assume that the user is going to be
>> invoking "fetch" in a single terminal window, IOW that we have an
>> implicit global mutex of one top-level git command at a time.
>>
>> Wheras mine fixes e.g. the same issue for:
>>
>>     parallel 'git fetch {}' ::: $(git remote)
>>
>> Ditto for you running a "git" command and your editor running a
>> "fetch" at the same time.
>
> You could sort of avoid the problem here too with
>
> parallel 'git fetch --no-auto-gc {}' ::: $(git remote)
> git gc --auto
>
> It's definitely simpler, but of course we have to manually add
> --no-auto-gc in everywhere we need, so not quite as elegant.
>
> Actually you could already do that with 'git -c gc.auto=false fetch', I guess.

The point of the 'parallel' example is to show disconnected git commands, think trying to run 'git' in a terminal while your editor asynchronously runs a polling 'fetch', or a server with multiple concurrent clients running 'gc --auto'.

That's the question my RFC patch raises. As far as I can tell the approach in your patch is only needed because our locking for gc is buggy, rather than introduce the caveat that an fetch(N) operation won't do "gc" until it's finished (we may have hundreds, thousands of remotes, I use that for some more obscure use-cases) shouldn't we just fix the locking?

Previous: Duy NguyenNext: Jeff King
Message 4 of 61 in “fetch: only run 'gc' once when fetching multiple remotes”
  1. fetch: only run 'gc' once when fetching multiple remotesNguyễn Thái Ngọc Duy, Jun 19, 2019
  2. gc: run more pre-detach operations under lockÆvar Arnfjörð Bjarmason, Jun 19, 2019
  3. Duy NguyenJun 19, 2019
  4. Ævar Arnfjörð BjarmasonJun 19, 2019
  5. Jeff KingJun 19, 2019
  6. Ævar Arnfjörð BjarmasonJun 19, 2019
  7. 0/6 Change <non-empty?> GIT_TEST_* variables to <boolean>Ævar Arnfjörð Bjarmason, Jun 19, 2019
  8. Junio C HamanoJun 20, 2019
  9. Ævar Arnfjörð BjarmasonJun 20, 2019
  10. Junio C HamanoJun 20, 2019
  11. 0/8 Change <non-empty?> GIT_TEST_* variables to <boolean>Ævar Arnfjörð Bjarmason, Jun 20, 2019
  12. 1/8 config tests: simplify include cycle testÆvar Arnfjörð Bjarmason, Jun 21, 2019
  13. 0/8 Change <non-empty?> GIT_TEST_* variables to <boolean>Ævar Arnfjörð Bjarmason, Jun 21, 2019
  14. 2/8 env--helper: new undocumented builtin wrapping git_env_*()Ævar Arnfjörð Bjarmason, Jun 21, 2019
  15. Junio C HamanoJun 21, 2019
  16. 3/8 config.c: refactor die_bad_number() to not call gettext() earlyÆvar Arnfjörð Bjarmason, Jun 21, 2019
  17. 4/8 t6040 test: stop using global "script" variableÆvar Arnfjörð Bjarmason, Jun 21, 2019
  18. 6/8 tests README: re-flow a previously changed paragraphÆvar Arnfjörð Bjarmason, Jun 21, 2019
  19. 5/8 tests: make GIT_TEST_GETTEXT_POISON a booleanÆvar Arnfjörð Bjarmason, Jun 21, 2019
  20. Junio C HamanoJun 24, 2019
  21. 7/8 tests: replace test_tristate with "git env--helper"Ævar Arnfjörð Bjarmason, Jun 21, 2019
  22. 1/2 t/lib-git-svn.sh: check GIT_TEST_SVN_HTTPD when running SVN HTTP testsSZEDER Gábor, Sep 6, 2019
  23. 2/2 ci: restore running httpd testsSZEDER Gábor, Sep 6, 2019
  24. Junio C HamanoSep 6, 2019
  25. Jeff KingSep 6, 2019
  26. SZEDER GáborSep 7, 2019
  27. 0/2 tests: catch non-bool GIT_TEST_* valuesSZEDER Gábor, Nov 22, 2019
  28. 1/2 tests: add 'test_bool_env' to catch non-bool GIT_TEST_* valuesSZEDER Gábor, Nov 22, 2019
  29. Jeff KingNov 25, 2019
  30. 2/2 t5608-clone-2gb.sh: turn GIT_TEST_CLONE_2GB into a boolSZEDER Gábor, Nov 22, 2019
  31. Jeff KingNov 25, 2019
  32. 8/8 tests: make GIT_TEST_FAIL_PREREQS a booleanÆvar Arnfjörð Bjarmason, Jun 21, 2019
  33. 1/8 config tests: simplify include cycle testÆvar Arnfjörð Bjarmason, Jun 20, 2019
  34. 2/8 env--helper: new undocumented builtin wrapping git_env_*()Ævar Arnfjörð Bjarmason, Jun 20, 2019
  35. Junio C HamanoJun 20, 2019
  36. Junio C HamanoJun 20, 2019
  37. Ævar Arnfjörð BjarmasonJun 21, 2019
  38. Junio C HamanoJun 21, 2019
  39. 3/8 config.c: refactor die_bad_number() to not call gettext() earlyÆvar Arnfjörð Bjarmason, Jun 20, 2019
  40. 4/8 t6040 test: stop using global "script" variableÆvar Arnfjörð Bjarmason, Jun 20, 2019
  41. 5/8 tests: make GIT_TEST_GETTEXT_POISON a booleanÆvar Arnfjörð Bjarmason, Jun 20, 2019
  42. 7/8 tests: replace test_tristate with "git env--helper"Ævar Arnfjörð Bjarmason, Jun 20, 2019
  43. 6/8 tests README: re-flow a previously changed paragraphÆvar Arnfjörð Bjarmason, Jun 20, 2019
  44. 8/8 tests: make GIT_TEST_FAIL_PREREQS a booleanÆvar Arnfjörð Bjarmason, Jun 20, 2019
  45. 1/6 env--helper: new undocumented builtin wrapping git_env_*()Ævar Arnfjörð Bjarmason, Jun 19, 2019
  46. Junio C HamanoJun 20, 2019
  47. 2/6 t6040 test: stop using global "script" variableÆvar Arnfjörð Bjarmason, Jun 19, 2019
  48. Junio C HamanoJun 20, 2019
  49. 3/6 tests: make GIT_TEST_GETTEXT_POISON a booleanÆvar Arnfjörð Bjarmason, Jun 19, 2019
  50. Junio C HamanoJun 20, 2019
  51. 5/6 tests: replace test_tristate with "git env--helper"Ævar Arnfjörð Bjarmason, Jun 19, 2019
  52. 4/6 tests README: re-flow a previously changed paragraphÆvar Arnfjörð Bjarmason, Jun 19, 2019
  53. 6/6 tests: make GIT_TEST_FAIL_PREREQS a booleanÆvar Arnfjörð Bjarmason, Jun 19, 2019
  54. Duy NguyenJun 20, 2019
  55. Ævar Arnfjörð BjarmasonJun 20, 2019
  56. Jeff KingJun 20, 2019
  57. Junio C HamanoJun 20, 2019
  58. Jeff KingJun 19, 2019
  59. Jeff KingJun 19, 2019
  60. Duy NguyenJun 20, 2019
  61. Jeff KingJun 20, 2019

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.