{"thread":{"id":"66152","subject":"[PATCH] odb/files: be less aggressive with geometric repacking","startedAt":"2026-08-11T09:05:07Z","lastAt":"2026-08-21T11:40:22Z","messageCount":8,"participants":["Patrick Steinhardt","Justin Tobler","Elijah Newren"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"550262","messageId":"20260811-pks-geometric-maintenance-reduce-frequency-v1-1-7a54c42355ac@pks.im","threadId":"66152","inReplyTo":null,"subject":"[PATCH] odb/files: be less aggressive with geometric repacking","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-08-11T09:04:48Z","receivedAt":"2026-08-11T09:05:07Z","isPatch":true,"body":"When performing auto-maintenance with geometric repacking we have two\nconditions that may trigger a repack:\n\n  - Either the geometric sequence of packfiles is invalidated.\n\n  - Or we have too many loose objects.\n\nThe first condition shouldn't trigger all that often: it may be hit when\nwe fetch a new packfile, but users tend to not do that all the time. The\nsecond condition is what typically triggers more regularly though, as\nevery command that ends up writing new objects may cause us to cross the\nthreshold of loose objects. It is thus preferable to not be too\naggressive here, as otherwise we may end up repacking objects quite\noften.\n\nFor the geometric-repacking strategy though we have a default of 100\nobjects, only. As we're approximating the count of objects by only\nreading the \"objects/17/\" shared, we'd only need 2 objects in there\nbefore we perform a repack by default, which is quite aggressive.\ngit-gc(1) on the other hand has a default of 6700, so it is quite a bit\nmore conservative here.\n\nBeing this aggressive is also causing problems as reported by our users.\nWhen running lots of concurrent writers, those writes will constantly\nend up spawning maintenance jobs that end up repacking objects. As we\nalso prune objects, a concurrently running process that tries to write\nan object may see that the sharding directories get removed under their\nfeet. While we try re-creating such leading directories, we only do so a\nsingle time, and it may happen that the directory vanishes again before\nwe had the chance to create the loose object. This is not a new problem,\nbut it is exacerbated by us running maintenance this aggressively.\n\nImprove the status quo by reducing the frequency at which we pack loose\nobjects to the same frequency that git-gc(1) uses.\n\nReported-by: Stefan Haller <lists@haller-berlin.de>\nSigned-off-by: Patrick Steinhardt <ps@pks.im>\n---\nHi,\n\nas reported by Stefan at [1]. Thanks!\n\nPatrick\n\n[1]: <4f6a96ac-d993-4872-b3c4-30d899f61ca9@haller-berlin.de>\n---\n Documentation/config/maintenance.adoc | 2 +-\n odb/source-files.c                    | 2 +-\n 2 files changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/Documentation/config/maintenance.adoc b/Documentation/config/maintenance.adoc\nindex b578856dde..da8be9f812 100644\n--- a/Documentation/config/maintenance.adoc\n+++ b/Documentation/config/maintenance.adoc\n@@ -101,7 +101,7 @@ maintenance.geometric-repack.auto::\n \tthere are packfiles that need to be merged together to retain the\n \tgeometric progression, or when there are at least this many loose\n \tobjects that would be written into a new packfile. The default value is\n-\t100.\n+\t6700.\n \n maintenance.geometric-repack.splitFactor::\n \tThis integer config option controls the factor used for the geometric\ndiff --git a/odb/source-files.c b/odb/source-files.c\nindex 5a68af7d84..555e466145 100644\n--- a/odb/source-files.c\n+++ b/odb/source-files.c\n@@ -521,7 +521,7 @@ bool odb_source_files_optimize_required(struct odb_source *source,\n \t\t};\n \t\tstruct existing_packs existing_packs = EXISTING_PACKS_INIT;\n \t\tstruct string_list kept_packs = STRING_LIST_INIT_DUP;\n-\t\tint auto_value = 100;\n+\t\tint auto_value = 6700;\n \t\tbool ret;\n \n \t\trepo_config_get_int(repo, \"maintenance.geometric-repack.auto\",\n\n---\nbase-commit: 010afd3166ddc64c9863b1506f12cbcdda0d4ea1\nchange-id: 20260810-pks-geometric-maintenance-reduce-frequency-5c1c9423ceb3\n\n"},{"id":"550327","messageId":"anuFzZluJEU21MB0@denethor","threadId":"66152","inReplyTo":"20260811-pks-geometric-maintenance-reduce-frequency-v1-1-7a54c42355ac@pks.im","subject":"Re: [PATCH] odb/files: be less aggressive with geometric repacking","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2026-08-11T20:44:12Z","receivedAt":"2026-08-11T20:44:16Z","isPatch":true,"body":"On 26/08/11 11:04AM, Patrick Steinhardt wrote:\n> When performing auto-maintenance with geometric repacking we have two\n> conditions that may trigger a repack:\n> \n>   - Either the geometric sequence of packfiles is invalidated.\n> \n>   - Or we have too many loose objects.\n> \n> The first condition shouldn't trigger all that often: it may be hit when\n> we fetch a new packfile, but users tend to not do that all the time. The\n> second condition is what typically triggers more regularly though, as\n> every command that ends up writing new objects may cause us to cross the\n> threshold of loose objects. It is thus preferable to not be too\n> aggressive here, as otherwise we may end up repacking objects quite\n> often.\n> \n> For the geometric-repacking strategy though we have a default of 100\n> objects, only. As we're approximating the count of objects by only\n> reading the \"objects/17/\" shared, we'd only need 2 objects in there\n> before we perform a repack by default, which is quite aggressive.\n> git-gc(1) on the other hand has a default of 6700, so it is quite a bit\n> more conservative here.\n\nOk IIUC, the reason two loose objects can potentially trigger repacking\nis because the heuristic used to estimate the number of loose objects\nonly counts objects in \"objects/17/\" and multiples it by 256 (the\nmaximum number of directories that are fanned-out). That makes sense and\nindeed seems like it could lead to repacking processes be spawned more\nfrequently than desired.\n\nMy first thought is whether the heuristic itself should be updated to\ncapture a more accurate estimate for the number of objects. That would\nof course require looking up more objects and thus be more expensive. If\nthe goal here is just for a very rough estimate anyways, maybe it\nwouldn't be worth it though.\n\nIncreasing the loose object threshold here to be more conservative seems\nlike a reasonable approach. I'm not sure exactly why 6700 was chosen\nhere. 6700 / 256 ~= 26.2 which means \"objects/17/\" would have to contain\nat least 27 objects before repacking is triggered. That is certainly\nmuch more conservative. I see that 6700 has also been chosen else where\nin the codebase as the threshold too. It might be nice to explain the\nreasoning a bit more in the commit message though.\n\n> Being this aggressive is also causing problems as reported by our users.\n> When running lots of concurrent writers, those writes will constantly\n> end up spawning maintenance jobs that end up repacking objects. As we\n> also prune objects, a concurrently running process that tries to write\n> an object may see that the sharding directories get removed under their\n> feet. While we try re-creating such leading directories, we only do so a\n> single time, and it may happen that the directory vanishes again before\n> we had the chance to create the loose object. This is not a new problem,\n> but it is exacerbated by us running maintenance this aggressively.\n> \n> Improve the status quo by reducing the frequency at which we pack loose\n> objects to the same frequency that git-gc(1) uses.\n\nMakes sense and the patch itself looks trivially correct.\n\n-Justin\n"},{"id":"550341","messageId":"anwIRuuaYG3AgG1m@pks.im","threadId":"66152","inReplyTo":"anuFzZluJEU21MB0@denethor","subject":"Re: [PATCH] odb/files: be less aggressive with geometric repacking","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-08-12T05:44:38Z","receivedAt":"2026-08-12T05:44:46Z","isPatch":true,"body":"On Tue, Aug 11, 2026 at 03:44:12PM -0500, Justin Tobler wrote:\n> On 26/08/11 11:04AM, Patrick Steinhardt wrote:\n> > When performing auto-maintenance with geometric repacking we have two\n> > conditions that may trigger a repack:\n> > \n> >   - Either the geometric sequence of packfiles is invalidated.\n> > \n> >   - Or we have too many loose objects.\n> > \n> > The first condition shouldn't trigger all that often: it may be hit when\n> > we fetch a new packfile, but users tend to not do that all the time. The\n> > second condition is what typically triggers more regularly though, as\n> > every command that ends up writing new objects may cause us to cross the\n> > threshold of loose objects. It is thus preferable to not be too\n> > aggressive here, as otherwise we may end up repacking objects quite\n> > often.\n> > \n> > For the geometric-repacking strategy though we have a default of 100\n> > objects, only. As we're approximating the count of objects by only\n> > reading the \"objects/17/\" shared, we'd only need 2 objects in there\n> > before we perform a repack by default, which is quite aggressive.\n> > git-gc(1) on the other hand has a default of 6700, so it is quite a bit\n> > more conservative here.\n> \n> Ok IIUC, the reason two loose objects can potentially trigger repacking\n> is because the heuristic used to estimate the number of loose objects\n> only counts objects in \"objects/17/\" and multiples it by 256 (the\n> maximum number of directories that are fanned-out). That makes sense and\n> indeed seems like it could lead to repacking processes be spawned more\n> frequently than desired.\n> \n> My first thought is whether the heuristic itself should be updated to\n> capture a more accurate estimate for the number of objects. That would\n> of course require looking up more objects and thus be more expensive. If\n> the goal here is just for a very rough estimate anyways, maybe it\n> wouldn't be worth it though.\n\nThat wouldn't really solve the problem though. The problem is not really\nthat the estimation can be wrong, it's rather that even if it was always\ncorrect we're still being too aggressive with packing the loose objects.\nBecause ultimately, a 100 objects is a comparatively small threshold,\nand leads to 67 times more repacking compared to git-gc(1).\n\n> Increasing the loose object threshold here to be more conservative seems\n> like a reasonable approach. I'm not sure exactly why 6700 was chosen\n> here. 6700 / 256 ~= 26.2 which means \"objects/17/\" would have to contain\n> at least 27 objects before repacking is triggered. That is certainly\n> much more conservative. I see that 6700 has also been chosen else where\n> in the codebase as the threshold too. It might be nice to explain the\n> reasoning a bit more in the commit message though.\n\nHmm, don't I already do that? In the paragraph you're responding to I'm\nsaying that git-gc(1) already had that default forever, so I'm adjusting\nour heuristic to match that.\n\nThanks!\n\nPatrick\n"},{"id":"550784","messageId":"aoTcxJSmKWNhnjZ9@denethor","threadId":"66152","inReplyTo":"anwIRuuaYG3AgG1m@pks.im","subject":"Re: [PATCH] odb/files: be less aggressive with geometric repacking","fromName":"Justin Tobler","fromEmail":"jltobler@gmail.com","sentAt":"2026-08-18T22:34:54Z","receivedAt":"2026-08-18T22:34:58Z","isPatch":true,"body":"On 26/08/12 07:44AM, Patrick Steinhardt wrote:\n> On Tue, Aug 11, 2026 at 03:44:12PM -0500, Justin Tobler wrote:\n> > On 26/08/11 11:04AM, Patrick Steinhardt wrote:\n> > > When performing auto-maintenance with geometric repacking we have two\n> > > conditions that may trigger a repack:\n> > > \n> > >   - Either the geometric sequence of packfiles is invalidated.\n> > > \n> > >   - Or we have too many loose objects.\n> > > \n> > > The first condition shouldn't trigger all that often: it may be hit when\n> > > we fetch a new packfile, but users tend to not do that all the time. The\n> > > second condition is what typically triggers more regularly though, as\n> > > every command that ends up writing new objects may cause us to cross the\n> > > threshold of loose objects. It is thus preferable to not be too\n> > > aggressive here, as otherwise we may end up repacking objects quite\n> > > often.\n> > > \n> > > For the geometric-repacking strategy though we have a default of 100\n> > > objects, only. As we're approximating the count of objects by only\n> > > reading the \"objects/17/\" shared, we'd only need 2 objects in there\n> > > before we perform a repack by default, which is quite aggressive.\n> > > git-gc(1) on the other hand has a default of 6700, so it is quite a bit\n> > > more conservative here.\n> > \n> > Ok IIUC, the reason two loose objects can potentially trigger repacking\n> > is because the heuristic used to estimate the number of loose objects\n> > only counts objects in \"objects/17/\" and multiples it by 256 (the\n> > maximum number of directories that are fanned-out). That makes sense and\n> > indeed seems like it could lead to repacking processes be spawned more\n> > frequently than desired.\n> > \n> > My first thought is whether the heuristic itself should be updated to\n> > capture a more accurate estimate for the number of objects. That would\n> > of course require looking up more objects and thus be more expensive. If\n> > the goal here is just for a very rough estimate anyways, maybe it\n> > wouldn't be worth it though.\n> \n> That wouldn't really solve the problem though. The problem is not really\n> that the estimation can be wrong, it's rather that even if it was always\n> correct we're still being too aggressive with packing the loose objects.\n> Because ultimately, a 100 objects is a comparatively small threshold,\n> and leads to 67 times more repacking compared to git-gc(1).\n\nOk, that makes sense.\n\n> > Increasing the loose object threshold here to be more conservative seems\n> > like a reasonable approach. I'm not sure exactly why 6700 was chosen\n> > here. 6700 / 256 ~= 26.2 which means \"objects/17/\" would have to contain\n> > at least 27 objects before repacking is triggered. That is certainly\n> > much more conservative. I see that 6700 has also been chosen else where\n> > in the codebase as the threshold too. It might be nice to explain the\n> > reasoning a bit more in the commit message though.\n> \n> Hmm, don't I already do that? In the paragraph you're responding to I'm\n> saying that git-gc(1) already had that default forever, so I'm adjusting\n> our heuristic to match that.\n\nI think I was just curious as to why 6700 was the chosen number for\ngit-gc(1) as well, but its probably just good to be consistent here. I\nthink this patch is fine as is.\n\n-Justin\n"},{"id":"550786","messageId":"aoU2iTmskL788erN@pks.im","threadId":"66152","inReplyTo":"aoTcxJSmKWNhnjZ9@denethor","subject":"Re: [PATCH] odb/files: be less aggressive with geometric repacking","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-08-19T04:52:25Z","receivedAt":"2026-08-19T04:52:33Z","isPatch":true,"body":"On Tue, Aug 18, 2026 at 05:34:54PM -0500, Justin Tobler wrote:\n> On 26/08/12 07:44AM, Patrick Steinhardt wrote:\n> > On Tue, Aug 11, 2026 at 03:44:12PM -0500, Justin Tobler wrote:\n> > > Increasing the loose object threshold here to be more conservative seems\n> > > like a reasonable approach. I'm not sure exactly why 6700 was chosen\n> > > here. 6700 / 256 ~= 26.2 which means \"objects/17/\" would have to contain\n> > > at least 27 objects before repacking is triggered. That is certainly\n> > > much more conservative. I see that 6700 has also been chosen else where\n> > > in the codebase as the threshold too. It might be nice to explain the\n> > > reasoning a bit more in the commit message though.\n> > \n> > Hmm, don't I already do that? In the paragraph you're responding to I'm\n> > saying that git-gc(1) already had that default forever, so I'm adjusting\n> > our heuristic to match that.\n> \n> I think I was just curious as to why 6700 was the chosen number for\n> git-gc(1) as well, but its probably just good to be consistent here. I\n> think this patch is fine as is.\n\nThat's a good question. It has been introduced all the way back in\n2c3c439947 (Implement git gc --auto, 2007-09-05), but that commit does\nnot mention any reasoning for the 6700 limit either.\n\nDigging in history a bit surfaces this nugget [1]. So the limit was\nchosen so that git-gc(1) would not trigger for a fully unpacked Git\nv0.99, would trigger for v1.0, but not triggering when doing an\nincremental gc after going from v0.99 to v1.0. This is of course quite\narbitrary, but as the mail points out, \"[t]he default threshold is\narbitrarily set by yours truly\" (Junio).\n\nPatrick\n\n[1]: https://lore.kernel.org/git/7vr6lcj2zi.fsf@gitster.siamese.dyndns.org/\n"},{"id":"550991","messageId":"aofxfh2czxv8ih0j@pks.im","threadId":"66152","inReplyTo":"20260811-pks-geometric-maintenance-reduce-frequency-v1-1-7a54c42355ac@pks.im","subject":"Re: [PATCH] odb/files: be less aggressive with geometric repacking","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-08-21T06:34:38Z","receivedAt":"2026-08-21T06:34:44Z","isPatch":true,"body":"Cc'ing Junio, as I haven't seen this topic in \"What's cooking\" yet. I\nassume it must've fallen through the cracks. Thanks!\n\nPatrick\n"},{"id":"550993","messageId":"CABPp-BHgyVTHB_OGmCL4JprFFe6_MapOQNSjUOhJxu-+oWbErg@mail.gmail.com","threadId":"66152","inReplyTo":"20260811-pks-geometric-maintenance-reduce-frequency-v1-1-7a54c42355ac@pks.im","subject":"Re: [PATCH] odb/files: be less aggressive with geometric repacking","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2026-08-21T06:40:46Z","receivedAt":"2026-08-21T06:40:59Z","isPatch":true,"body":"Heh, looks like a typed up a response and got distracted just before\nthe end and never came back and sent it.  Sending now due to Patrick's\nping about this not showing up in What's Cooking; maybe an extra\nreview will help.  :-)\n\nOn Tue, Aug 11, 2026 at 2:17 AM Patrick Steinhardt <ps@pks.im> wrote:\n>\n> When performing auto-maintenance with geometric repacking we have two\n> conditions that may trigger a repack:\n>\n>   - Either the geometric sequence of packfiles is invalidated.\n>\n>   - Or we have too many loose objects.\n>\n> The first condition shouldn't trigger all that often: it may be hit when\n> we fetch a new packfile, but users tend to not do that all the time. The\n> second condition is what typically triggers more regularly though, as\n> every command that ends up writing new objects may cause us to cross the\n> threshold of loose objects. It is thus preferable to not be too\n> aggressive here, as otherwise we may end up repacking objects quite\n> often.\n>\n> For the geometric-repacking strategy though we have a default of 100\n> objects, only. As we're approximating the count of objects by only\n> reading the \"objects/17/\" shared, we'd only need 2 objects in there\n> before we perform a repack by default, which is quite aggressive.\n> git-gc(1) on the other hand has a default of 6700, so it is quite a bit\n> more conservative here.\n\n2?  Wouldn't you only need 1 (or if you could have fractional numbers\nof objects, only 0.390625 of them)?  <looks around...>    Oh, huh:\n\n        /*\n         * This is weird, but stems from legacy behaviour: the GC auto\n         * threshold was always essentially interpreted as if it was rounded up\n         * to the next multiple 256 of, so we retain this behaviour for now.\n         */\n        return loose_count > (DIV_ROUND_UP(((unsigned long) limit), 256) * 256);\n\nSo, indeed, you need 2.\n\n> Being this aggressive is also causing problems as reported by our users.\n> When running lots of concurrent writers, those writes will constantly\n> end up spawning maintenance jobs that end up repacking objects. As we\n> also prune objects, a concurrently running process that tries to write\n> an object may see that the sharding directories get removed under their\n> feet. While we try re-creating such leading directories, we only do so a\n> single time, and it may happen that the directory vanishes again before\n> we had the chance to create the loose object. This is not a new problem,\n> but it is exacerbated by us running maintenance this aggressively.\n\nUnrelated to this patch...but should git avoid pruning the loose\nobject sharding directories?\n\n> Improve the status quo by reducing the frequency at which we pack loose\n> objects to the same frequency that git-gc(1) uses.\n\nMakes sense.\n\n> Reported-by: Stefan Haller <lists@haller-berlin.de>\n> Signed-off-by: Patrick Steinhardt <ps@pks.im>\n> ---\n> Hi,\n>\n> as reported by Stefan at [1]. Thanks!\n>\n> Patrick\n>\n> [1]: <4f6a96ac-d993-4872-b3c4-30d899f61ca9@haller-berlin.de>\n> ---\n>  Documentation/config/maintenance.adoc | 2 +-\n>  odb/source-files.c                    | 2 +-\n>  2 files changed, 2 insertions(+), 2 deletions(-)\n>\n> diff --git a/Documentation/config/maintenance.adoc b/Documentation/config/maintenance.adoc\n> index b578856dde..da8be9f812 100644\n> --- a/Documentation/config/maintenance.adoc\n> +++ b/Documentation/config/maintenance.adoc\n> @@ -101,7 +101,7 @@ maintenance.geometric-repack.auto::\n>         there are packfiles that need to be merged together to retain the\n>         geometric progression, or when there are at least this many loose\n>         objects that would be written into a new packfile. The default value is\n> -       100.\n> +       6700.\n>\n>  maintenance.geometric-repack.splitFactor::\n>         This integer config option controls the factor used for the geometric\n> diff --git a/odb/source-files.c b/odb/source-files.c\n> index 5a68af7d84..555e466145 100644\n> --- a/odb/source-files.c\n> +++ b/odb/source-files.c\n> @@ -521,7 +521,7 @@ bool odb_source_files_optimize_required(struct odb_source *source,\n>                 };\n>                 struct existing_packs existing_packs = EXISTING_PACKS_INIT;\n>                 struct string_list kept_packs = STRING_LIST_INIT_DUP;\n> -               int auto_value = 100;\n> +               int auto_value = 6700;\n>                 bool ret;\n>\n>                 repo_config_get_int(repo, \"maintenance.geometric-repack.auto\",\n>\n> ---\n> base-commit: 010afd3166ddc64c9863b1506f12cbcdda0d4ea1\n> change-id: 20260810-pks-geometric-maintenance-reduce-frequency-5c1c9423ceb3\n\nLooks good to me.\n"},{"id":"551001","messageId":"aog5Hwp5EQA0k500@pks.im","threadId":"66152","inReplyTo":"CABPp-BHgyVTHB_OGmCL4JprFFe6_MapOQNSjUOhJxu-+oWbErg@mail.gmail.com","subject":"Re: [PATCH] odb/files: be less aggressive with geometric repacking","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2026-08-21T11:40:15Z","receivedAt":"2026-08-21T11:40:22Z","isPatch":true,"body":"On Thu, Aug 20, 2026 at 11:40:46PM -0700, Elijah Newren wrote:\n> On Tue, Aug 11, 2026 at 2:17 AM Patrick Steinhardt <ps@pks.im> wrote:\n[snip]\n> > Being this aggressive is also causing problems as reported by our users.\n> > When running lots of concurrent writers, those writes will constantly\n> > end up spawning maintenance jobs that end up repacking objects. As we\n> > also prune objects, a concurrently running process that tries to write\n> > an object may see that the sharding directories get removed under their\n> > feet. While we try re-creating such leading directories, we only do so a\n> > single time, and it may happen that the directory vanishes again before\n> > we had the chance to create the loose object. This is not a new problem,\n> > but it is exacerbated by us running maintenance this aggressively.\n> \n> Unrelated to this patch...but should git avoid pruning the loose\n> object sharding directories?\n\nI was wondering about that, too. There are two contradicting arguments\nto make here:\n\n  - Pruning the sharding directories allows us to quickly determine that\n    an empty shard cannot have an object.\n\n  - Not pruning the sharding directories may avoid a lot of write churn.\n\nThe question is how large the impact of these two individual arguments\nis.\n\nBy gut feeling, I think that the first argument is somewhat weak. Not\nhaving empty directories means that looking up a loose object by its\npath will be slightly faster because we have to walk one less directory\nin the hierarchy. But this really only matters in the case where we look\nfor a nonexistent object, which does not happen all that often because\nwe prefer searching packfiles first.\n\nFurthermore, iterating through all objects in the object database will\nbe faster, as we don't have to open each of the directories only to find\nthem empty. But again, that's not really something that we do all that\nfrequently.\n\nOn the other hand, we _do_ have to recreate the loose object shards\nquite frequently as that's how we write data into a repository. And as\nwe've seen, pruning those shards can easily cause races.\n\nSo in the end I think it could be a useful thing to explore. The only\nthing I wonder is whether there's a good reason for why we prune those\nthat I miss.\n\nPatrick\n"}]}