git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH] odb/files: be less aggressive with geometric repacking

From
Patrick Steinhardt <ps@pks.im>
Date
Aug 21, 2026, 11:40 UTC
Message-ID
<aog5Hwp5EQA0k500@pks.im>
In-Reply-To
<CABPp-BHgyVTHB_OGmCL4JprFFe6_MapOQNSjUOhJxu-+oWbErg@mail.gmail.com>
On Thu, Aug 20, 2026 at 11:40:46PM -0700, Elijah Newren wrote:
> On Tue, Aug 11, 2026 at 2:17 AM Patrick Steinhardt <ps@pks.im> wrote:
[snip]
Show 12 quoted lines
> > Being this aggressive is also causing problems as reported by our users.
> > When running lots of concurrent writers, those writes will constantly
> > end up spawning maintenance jobs that end up repacking objects. As we
> > also prune objects, a concurrently running process that tries to write
> > an object may see that the sharding directories get removed under their
> > feet. While we try re-creating such leading directories, we only do so a
> > single time, and it may happen that the directory vanishes again before
> > we had the chance to create the loose object. This is not a new problem,
> > but it is exacerbated by us running maintenance this aggressively.
> 
> Unrelated to this patch...but should git avoid pruning the loose
> object sharding directories?

I was wondering about that, too. There are two contradicting arguments to make here:

  - Pruning the sharding directories allows us to quickly determine that
    an empty shard cannot have an object.
  - Not pruning the sharding directories may avoid a lot of write churn.

The question is how large the impact of these two individual arguments is.

By gut feeling, I think that the first argument is somewhat weak. Not having empty directories means that looking up a loose object by its path will be slightly faster because we have to walk one less directory in the hierarchy. But this really only matters in the case where we look for a nonexistent object, which does not happen all that often because we prefer searching packfiles first.

Furthermore, iterating through all objects in the object database will be faster, as we don't have to open each of the directories only to find them empty. But again, that's not really something that we do all that frequently.

On the other hand, we _do_ have to recreate the loose object shards quite frequently as that's how we write data into a repository. And as we've seen, pruning those shards can easily cause races.

So in the end I think it could be a useful thing to explore. The only thing I wonder is whether there's a good reason for why we prune those that I miss.

Patrick
Previous: Elijah Newren
Message 8 of 8 in “odb/files: be less aggressive with geometric repacking”
  1. odb/files: be less aggressive with geometric repackingPatrick Steinhardt, Aug 11, 2026
  2. Justin ToblerAug 11, 2026
  3. Patrick SteinhardtAug 12, 2026
  4. Justin ToblerAug 18, 2026
  5. Patrick SteinhardtAug 19, 2026
  6. Patrick SteinhardtAug 21, 2026
  7. Elijah NewrenAug 21, 2026
  8. Patrick SteinhardtAug 21, 2026

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.