git/list[1] front-page[2] threads[3] people[4] search[5] about
 

git gc --auto yelling at users where a repo legitimately has >6700 loose objects

From
Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Date
Jan 11, 2018, 21:33 UTC
Message-ID
<87inc89j38.fsf@evledraar.gmail.com>

I recently disabled gc.auto=0 and my nightly aggressive repack script on our big monorepo across our infra, relying instead on git gc --auto in the background to just do its thing.

I didn't want users to wait for git-gc, and I'd written this nightly cronjob before git-gc learned to detach to the background.

But now I have git-gc on some servers yelling at users on every pull command:

    warning: There are too many unreachable loose objects; run 'git prune' to remove them.

The reason is that I have all the values at git's default settings, and there legitimately are >~6700 loose objects that were created in the last 2 weeks.

For those rusty on git-gc's defaults, this is what it looks like in this scenario:

 1. User runs "git pull"
 2. git gc --auto is called, there are >6700 loose objects
 3. it forks into the background, tries to prune and repack, objects
    older than gc.pruneExpire (2.weeks.ago) are pruned.
 4. At the end of all this, we check *again* if we have >6700 objects,
    if we do we print "run 'git prune'" to .git/gc.log, and will just
    emit that error for the next day before trying again, at which point
    we unlink the gc.log and retry, see gc.logExpiry.

Right now I've just worked around this by setting gc.pruneExpire to a lower value (4.days.ago). But there's a larger issue to be addressed here, and I'm not sure how.

When the warning was added in [1] it didn't know to detach to the background yet, that came in [2], shortly after came gc.log in [3].

We could add another gc.auto-like limit, which could be set at some higher value than gc.auto. "Hey if I have more than 6700 loose objects, prune the <2wks old ones, but if at the end there's still >6700 I don't want to hear about it unless there's >6700*N".

I thought I'd just add that, but the details of how to pass that message around get nasty. With that solution we *also* don't want git gc to start churning in the background once we reach >6700 objects, so we need something like gc.logExpiry which defers the gc until the next day. We might need to create .git/gc-waitabit.marker, ew.

More generally, these hard limits seem contrary to what the user cares about. E.g. I suspect that most of these loose objects come from branches since deleted in upstream, whose objects could have a different retention policy.

Or we could say "I want 2 weeks of objects, but if that runs against the 6700 limit just keep the latest 6700/2".

1. a087cc9819 ("git-gc --auto: protect ourselves from accumulated
   cruft", 2007-09-17)
2. 9f673f9477 ("gc: config option for running --auto in background",
   2014-02-08)
3. 329e6e8794 ("gc: save log from daemonized gc --auto and print it next
   time", 2015-09-19)
Next: Duy Nguyen
Message 1 of 9 in “git gc --auto yelling at users where a repo legitimately has >6700 loose objects”
  1. Ævar Arnfjörð BjarmasonJan 11, 2018
  2. Duy NguyenJan 12, 2018
  3. Duy NguyenJan 12, 2018
  4. Ævar Arnfjörð BjarmasonJan 12, 2018
  5. Jeff KingJan 13, 2018
  6. Jeff KingJan 12, 2018
  7. Duy NguyenJan 12, 2018
  8. Jeff KingJan 13, 2018
  9. Ævar Arnfjörð BjarmasonFeb 8, 2018

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.