git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH 0/4] gc docs: modernize and fix the documentation

From
Jeff King <peff@peff.net>
Date
May 7, 2019, 07:51 UTC
Message-ID
<20190507075158.GG28060@sigill.intra.peff.net>
In-Reply-To
<878svjj4t5.fsf@evledraar.gmail.com>
On Mon, May 06, 2019 at 11:44:06AM +0200, Ævar Arnfjörð Bjarmason wrote:
Show 16 quoted lines
> Maybe there's some case I haven't thought of that makes this stupid, but
> I wonder if something like a "gc quarantine" might be a fix fo both of
> the the issues you noted above.
> 
> I.e. it seems to me that the main issue is that we conflate "mtime 2
> weeks old because it's unreferenced for 2 weeks" v.s. "mtime 2 weeks old
> because we haven't gotten around to a 'gc'".
> 
> So in such a "gc quarantine" mode when we discover an object/pack that's
> unreachable/purely made up of unreachable objects we'd move the relevant
> loose object/"loose" pack to such a quarantine, which would just be
> .git/unreferenced-objects/{??,pack}/ or whatever.
> 
> AFAICT both cases you mentioned above would be mitigated by this because
> we'd no longer conflate "haven't gc'd this yet and it's 2 weeks old"
> v.s. "hasn't been referenced in 2 weeks".

Michael Haggerty and I have (off-list) discussed variations on that, but it opens up a lot of new issues. Moving something into quarantine isn't atomic. So you've still corrupted the repo, but now it's recoverable by reaching into the quarantine. Who notices that the repo is corrupt, and how? When do we expire objects from quarantine?

I think the heart of the issue is really the lack of atomicity in the operations. You need some way to mark "I am using this now" in a way that cannot race with "looks like nobody is using this, so I'll delete it".

And ideally without traversing large bits of the graph on the writing side, and without requiring any stop-the-world locks during pruning.

Show 9 quoted lines
> I started looking at this initially because I was wondering if the
> --keep-unreachable mode you modified in e26a8c4721 ("repack: extend
> --keep-unreachable to loose objects", 2016-06-13) could be made to write
> out such "unreferenced" objects into their *own* pack, so we could
> delete them all at once as a batch, and wouldn't create the "ref
> explosions" mentioned in [1].
> 
> But of course without an accompanying quarantine described above doing
> that would just make this race condition worse.

I'm not sure it really makes it worse. The pack would have the same mtime as the loose objects would.

-Peff
Previous: Ævar Arnfjörð BjarmasonNext: Ævar Arnfjörð Bjarmason
Message 24 of 64 in “gc docs: modernize and fix the documentation”
  1. 0/4 gc docs: modernize and fix the documentationÆvar Arnfjörð Bjarmason, Mar 18, 2019
  2. 1/4 gc docs: modernize the advice for manually running "gc"Ævar Arnfjörð Bjarmason, Mar 18, 2019
  3. Jeff KingMar 18, 2019
  4. Ævar Arnfjörð BjarmasonMar 18, 2019
  5. 2/4 gc docs: include the "gc.*" section from "config" in "gc"Ævar Arnfjörð Bjarmason, Mar 18, 2019
  6. Jeff KingMar 18, 2019
  7. Andreas HeidukMar 21, 2019
  8. Duy NguyenMar 19, 2019
  9. 3/4 gc docs: de-duplicate "OPTIONS" and "CONFIGURATION"Ævar Arnfjörð Bjarmason, Mar 18, 2019
  10. Jeff KingMar 18, 2019
  11. Ævar Arnfjörð BjarmasonMar 18, 2019
  12. Jeff KingMar 18, 2019
  13. 4/4 gc docs: downplay the usefulness of --aggressiveÆvar Arnfjörð Bjarmason, Mar 18, 2019
  14. Jonathan NiederMar 18, 2019
  15. Jeff KingMar 18, 2019
  16. Ævar Arnfjörð BjarmasonMar 18, 2019
  17. Jeff KingMar 18, 2019
  18. Johannes SixtMar 19, 2019
  19. Ævar Arnfjörð BjarmasonMar 19, 2019
  20. Jeff KingMar 18, 2019
  21. Ævar Arnfjörð BjarmasonMar 18, 2019
  22. Jeff KingMar 19, 2019
  23. Ævar Arnfjörð BjarmasonMay 6, 2019
  24. Jeff KingMay 7, 2019
  25. Ævar Arnfjörð BjarmasonMay 9, 2019
  26. Jeff KingJul 31, 2019
  27. Ævar Arnfjörð BjarmasonJul 31, 2019
  28. 00/10 gc docs: modernize and fix the documentationÆvar Arnfjörð Bjarmason, Mar 21, 2019
  29. 00/11 gc docs: modernize the advice for manually running "gc"Ævar Arnfjörð Bjarmason, Mar 22, 2019
  30. 01/11 gc docs: modernize the advice for manually running "gc"Ævar Arnfjörð Bjarmason, Mar 22, 2019
  31. 02/11 gc docs: stop noting "repack" flagsÆvar Arnfjörð Bjarmason, Mar 22, 2019
  32. 03/11 gc docs: clean grammar for "gc.bigPackThreshold"Ævar Arnfjörð Bjarmason, Mar 22, 2019
  33. 06/11 gc docs: fix formatting for "gc.writeCommitGraph"Ævar Arnfjörð Bjarmason, Mar 22, 2019
  34. 05/11 gc docs: re-flow the "gc.*" section in "config"Ævar Arnfjörð Bjarmason, Mar 22, 2019
  35. 07/11 gc docs: note how --aggressive impacts --window & --depthÆvar Arnfjörð Bjarmason, Mar 22, 2019
  36. 04/11 gc docs: include the "gc.*" section from "config" in "gc"Ævar Arnfjörð Bjarmason, Mar 22, 2019
  37. Todd ZullingerMar 30, 2019
  38. 00/11 gc docs: modernize and fix the documentationÆvar Arnfjörð Bjarmason, Apr 7, 2019
  39. 01/11 gc docs: modernize the advice for manually running "gc"Ævar Arnfjörð Bjarmason, Apr 7, 2019
  40. 02/11 gc docs: stop noting "repack" flagsÆvar Arnfjörð Bjarmason, Apr 7, 2019
  41. 03/11 gc docs: clean grammar for "gc.bigPackThreshold"Ævar Arnfjörð Bjarmason, Apr 7, 2019
  42. 04/11 gc docs: include the "gc.*" section from "config" in "gc"Ævar Arnfjörð Bjarmason, Apr 7, 2019
  43. 05/11 gc docs: re-flow the "gc.*" section in "config"Ævar Arnfjörð Bjarmason, Apr 7, 2019
  44. 06/11 gc docs: fix formatting for "gc.writeCommitGraph"Ævar Arnfjörð Bjarmason, Apr 7, 2019
  45. 07/11 gc docs: note how --aggressive impacts --window & --depthÆvar Arnfjörð Bjarmason, Apr 7, 2019
  46. 09/11 gc docs: note "gc --aggressive" in "fast-import"Ævar Arnfjörð Bjarmason, Apr 7, 2019
  47. 10/11 gc docs: clarify that "gc" doesn't throw away referenced objectsÆvar Arnfjörð Bjarmason, Apr 7, 2019
  48. 11/11 gc docs: remove incorrect reference to gc.auto=0Ævar Arnfjörð Bjarmason, Apr 7, 2019
  49. 08/11 gc docs: downplay the usefulness of --aggressiveÆvar Arnfjörð Bjarmason, Apr 7, 2019
  50. 10/11 gc docs: clarify that "gc" doesn't throw away referenced objectsÆvar Arnfjörð Bjarmason, Mar 22, 2019
  51. 08/11 gc docs: downplay the usefulness of --aggressiveÆvar Arnfjörð Bjarmason, Mar 22, 2019
  52. 11/11 gc docs: remove incorrect reference to gc.auto=0Ævar Arnfjörð Bjarmason, Mar 22, 2019
  53. 09/11 gc docs: note "gc --aggressive" in "fast-import"Ævar Arnfjörð Bjarmason, Mar 22, 2019
  54. 02/10 gc docs: stop noting "repack" flagsÆvar Arnfjörð Bjarmason, Mar 21, 2019
  55. 01/10 gc docs: modernize the advice for manually running "gc"Ævar Arnfjörð Bjarmason, Mar 21, 2019
  56. Junio C HamanoMar 22, 2019
  57. 03/10 gc docs: clean grammar for "gc.bigPackThreshold"Ævar Arnfjörð Bjarmason, Mar 21, 2019
  58. 04/10 gc docs: include the "gc.*" section from "config" in "gc"Ævar Arnfjörð Bjarmason, Mar 21, 2019
  59. 05/10 gc docs: re-flow the "gc.*" section in "config"Ævar Arnfjörð Bjarmason, Mar 21, 2019
  60. 06/10 gc docs: note how --aggressive impacts --window & --depthÆvar Arnfjörð Bjarmason, Mar 21, 2019
  61. 07/10 gc docs: downplay the usefulness of --aggressiveÆvar Arnfjörð Bjarmason, Mar 21, 2019
  62. 08/10 gc docs: note "gc --aggressive" in "fast-import"Ævar Arnfjörð Bjarmason, Mar 21, 2019
  63. 09/10 gc docs: clarify that "gc" doesn't throw away referenced objectsÆvar Arnfjörð Bjarmason, Mar 21, 2019
  64. 10/10 gc docs: remove incorrect reference to gc.auto=0Ævar Arnfjörð Bjarmason, Mar 21, 2019

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.