git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH 0/4] gc docs: modernize and fix the documentation

From
Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Date
May 9, 2019, 23:20 UTC
Message-ID
<8736lnxlig.fsf@evledraar.gmail.com>
In-Reply-To
<20190507075158.GG28060@sigill.intra.peff.net>
On Tue, May 07 2019, Jeff King wrote:
Show 32 quoted lines
> On Mon, May 06, 2019 at 11:44:06AM +0200, Ævar Arnfjörð Bjarmason wrote:
>
>> Maybe there's some case I haven't thought of that makes this stupid, but
>> I wonder if something like a "gc quarantine" might be a fix fo both of
>> the the issues you noted above.
>>
>> I.e. it seems to me that the main issue is that we conflate "mtime 2
>> weeks old because it's unreferenced for 2 weeks" v.s. "mtime 2 weeks old
>> because we haven't gotten around to a 'gc'".
>>
>> So in such a "gc quarantine" mode when we discover an object/pack that's
>> unreachable/purely made up of unreachable objects we'd move the relevant
>> loose object/"loose" pack to such a quarantine, which would just be
>> .git/unreferenced-objects/{??,pack}/ or whatever.
>>
>> AFAICT both cases you mentioned above would be mitigated by this because
>> we'd no longer conflate "haven't gc'd this yet and it's 2 weeks old"
>> v.s. "hasn't been referenced in 2 weeks".
>
> Michael Haggerty and I have (off-list) discussed variations on that, but
> it opens up a lot of new issues.  Moving something into quarantine isn't
> atomic. So you've still corrupted the repo, but now it's recoverable by
> reaching into the quarantine. Who notices that the repo is corrupt, and
> how? When do we expire objects from quarantine?
>
> I think the heart of the issue is really the lack of atomicity in the
> operations. You need some way to mark "I am using this now" in a way
> that cannot race with "looks like nobody is using this, so I'll delete
> it".
>
> And ideally without traversing large bits of the graph on the writing
> side, and without requiring any stop-the-world locks during pruning.

I was thinking (but realize now that I didn't articulate) that the "gc quarantine" would be another "alternate" implementing a copy-on-write "lockless delete-but-be-able-to-rollback scheme" as you put it.

So "gc" would decide (racily) what's unreachable, but instead of unlink()-ing it would "mv" the loose object/pack into the "unreferenced-objects" quarantine.

Then in your example #1 "wants to reference ABCD. It sees that we have it." would race on the "other side". I.e. maybe ABCD was *just* moved to the quarantine. But in that case we'd move it back, which would bump the mtime and thus make it ineligible for expiry.

Similarly for example #2, the "ABCD is ancient" would be moved, but then promptely moved back on the next GC as we notice ABCD has been re-referenced.

Maybe it's just the same problem all over again, but I don't see how yet.

Aside from that, I have a hunch that while it's theoretically true that you can at any time re-reference some loose blob/tree/commit again, that the likelyhood of that in practice goes down as it ages, since a user is likely to e.g. re-push or rename some branch they pushed last week, not last year.

Hence the mention of creating "unreferenced packs" with some new --keep-unreachable mode. Since we'd pack those together they wouldn't create the "ref explosion" problem we have with the loose refs, and thus you could afford to keep them longer (even though the deltas would be shittier).

Whereas now you either need --keep-unreachable (keep stuff forever) or a more aggressive gc.pruneExpire if you'd like to not end up with a ginormous amount of loose objects.

Show 12 quoted lines
>> I started looking at this initially because I was wondering if the
>> --keep-unreachable mode you modified in e26a8c4721 ("repack: extend
>> --keep-unreachable to loose objects", 2016-06-13) could be made to write
>> out such "unreferenced" objects into their *own* pack, so we could
>> delete them all at once as a batch, and wouldn't create the "ref
>> explosions" mentioned in [1].
>>
>> But of course without an accompanying quarantine described above doing
>> that would just make this race condition worse.
>
> I'm not sure it really makes it worse. The pack would have the same
> mtime as the loose objects would.
Previous: Jeff KingNext: Jeff King
Message 25 of 64 in “gc docs: modernize and fix the documentation”
  1. 0/4 gc docs: modernize and fix the documentationÆvar Arnfjörð Bjarmason, Mar 18, 2019
  2. 1/4 gc docs: modernize the advice for manually running "gc"Ævar Arnfjörð Bjarmason, Mar 18, 2019
  3. Jeff KingMar 18, 2019
  4. Ævar Arnfjörð BjarmasonMar 18, 2019
  5. 2/4 gc docs: include the "gc.*" section from "config" in "gc"Ævar Arnfjörð Bjarmason, Mar 18, 2019
  6. Jeff KingMar 18, 2019
  7. Andreas HeidukMar 21, 2019
  8. Duy NguyenMar 19, 2019
  9. 3/4 gc docs: de-duplicate "OPTIONS" and "CONFIGURATION"Ævar Arnfjörð Bjarmason, Mar 18, 2019
  10. Jeff KingMar 18, 2019
  11. Ævar Arnfjörð BjarmasonMar 18, 2019
  12. Jeff KingMar 18, 2019
  13. 4/4 gc docs: downplay the usefulness of --aggressiveÆvar Arnfjörð Bjarmason, Mar 18, 2019
  14. Jonathan NiederMar 18, 2019
  15. Jeff KingMar 18, 2019
  16. Ævar Arnfjörð BjarmasonMar 18, 2019
  17. Jeff KingMar 18, 2019
  18. Johannes SixtMar 19, 2019
  19. Ævar Arnfjörð BjarmasonMar 19, 2019
  20. Jeff KingMar 18, 2019
  21. Ævar Arnfjörð BjarmasonMar 18, 2019
  22. Jeff KingMar 19, 2019
  23. Ævar Arnfjörð BjarmasonMay 6, 2019
  24. Jeff KingMay 7, 2019
  25. Ævar Arnfjörð BjarmasonMay 9, 2019
  26. Jeff KingJul 31, 2019
  27. Ævar Arnfjörð BjarmasonJul 31, 2019
  28. 00/10 gc docs: modernize and fix the documentationÆvar Arnfjörð Bjarmason, Mar 21, 2019
  29. 00/11 gc docs: modernize the advice for manually running "gc"Ævar Arnfjörð Bjarmason, Mar 22, 2019
  30. 01/11 gc docs: modernize the advice for manually running "gc"Ævar Arnfjörð Bjarmason, Mar 22, 2019
  31. 02/11 gc docs: stop noting "repack" flagsÆvar Arnfjörð Bjarmason, Mar 22, 2019
  32. 03/11 gc docs: clean grammar for "gc.bigPackThreshold"Ævar Arnfjörð Bjarmason, Mar 22, 2019
  33. 06/11 gc docs: fix formatting for "gc.writeCommitGraph"Ævar Arnfjörð Bjarmason, Mar 22, 2019
  34. 05/11 gc docs: re-flow the "gc.*" section in "config"Ævar Arnfjörð Bjarmason, Mar 22, 2019
  35. 07/11 gc docs: note how --aggressive impacts --window & --depthÆvar Arnfjörð Bjarmason, Mar 22, 2019
  36. 04/11 gc docs: include the "gc.*" section from "config" in "gc"Ævar Arnfjörð Bjarmason, Mar 22, 2019
  37. Todd ZullingerMar 30, 2019
  38. 00/11 gc docs: modernize and fix the documentationÆvar Arnfjörð Bjarmason, Apr 7, 2019
  39. 01/11 gc docs: modernize the advice for manually running "gc"Ævar Arnfjörð Bjarmason, Apr 7, 2019
  40. 02/11 gc docs: stop noting "repack" flagsÆvar Arnfjörð Bjarmason, Apr 7, 2019
  41. 03/11 gc docs: clean grammar for "gc.bigPackThreshold"Ævar Arnfjörð Bjarmason, Apr 7, 2019
  42. 04/11 gc docs: include the "gc.*" section from "config" in "gc"Ævar Arnfjörð Bjarmason, Apr 7, 2019
  43. 05/11 gc docs: re-flow the "gc.*" section in "config"Ævar Arnfjörð Bjarmason, Apr 7, 2019
  44. 06/11 gc docs: fix formatting for "gc.writeCommitGraph"Ævar Arnfjörð Bjarmason, Apr 7, 2019
  45. 07/11 gc docs: note how --aggressive impacts --window & --depthÆvar Arnfjörð Bjarmason, Apr 7, 2019
  46. 09/11 gc docs: note "gc --aggressive" in "fast-import"Ævar Arnfjörð Bjarmason, Apr 7, 2019
  47. 10/11 gc docs: clarify that "gc" doesn't throw away referenced objectsÆvar Arnfjörð Bjarmason, Apr 7, 2019
  48. 11/11 gc docs: remove incorrect reference to gc.auto=0Ævar Arnfjörð Bjarmason, Apr 7, 2019
  49. 08/11 gc docs: downplay the usefulness of --aggressiveÆvar Arnfjörð Bjarmason, Apr 7, 2019
  50. 10/11 gc docs: clarify that "gc" doesn't throw away referenced objectsÆvar Arnfjörð Bjarmason, Mar 22, 2019
  51. 08/11 gc docs: downplay the usefulness of --aggressiveÆvar Arnfjörð Bjarmason, Mar 22, 2019
  52. 11/11 gc docs: remove incorrect reference to gc.auto=0Ævar Arnfjörð Bjarmason, Mar 22, 2019
  53. 09/11 gc docs: note "gc --aggressive" in "fast-import"Ævar Arnfjörð Bjarmason, Mar 22, 2019
  54. 02/10 gc docs: stop noting "repack" flagsÆvar Arnfjörð Bjarmason, Mar 21, 2019
  55. 01/10 gc docs: modernize the advice for manually running "gc"Ævar Arnfjörð Bjarmason, Mar 21, 2019
  56. Junio C HamanoMar 22, 2019
  57. 03/10 gc docs: clean grammar for "gc.bigPackThreshold"Ævar Arnfjörð Bjarmason, Mar 21, 2019
  58. 04/10 gc docs: include the "gc.*" section from "config" in "gc"Ævar Arnfjörð Bjarmason, Mar 21, 2019
  59. 05/10 gc docs: re-flow the "gc.*" section in "config"Ævar Arnfjörð Bjarmason, Mar 21, 2019
  60. 06/10 gc docs: note how --aggressive impacts --window & --depthÆvar Arnfjörð Bjarmason, Mar 21, 2019
  61. 07/10 gc docs: downplay the usefulness of --aggressiveÆvar Arnfjörð Bjarmason, Mar 21, 2019
  62. 08/10 gc docs: note "gc --aggressive" in "fast-import"Ævar Arnfjörð Bjarmason, Mar 21, 2019
  63. 09/10 gc docs: clarify that "gc" doesn't throw away referenced objectsÆvar Arnfjörð Bjarmason, Mar 21, 2019
  64. 10/10 gc docs: remove incorrect reference to gc.auto=0Ævar Arnfjörð Bjarmason, Mar 21, 2019

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.