{"thread":{"id":"62975","subject":"git keeps recreating packs, exploding backup increments","startedAt":"2025-02-19T09:47:23Z","lastAt":"2025-05-09T10:34:02Z","messageCount":6,"participants":["Pierre Ossman","Han Young","Patrick Steinhardt"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"512658","messageId":"1524b9a5-6f8b-4537-ba6b-bdfdd4b1bdcb@cendio.se","threadId":"62975","inReplyTo":null,"subject":"git keeps recreating packs, exploding backup increments","fromName":"Pierre Ossman","fromEmail":"ossman@cendio.se","sentAt":"2025-02-19T09:38:25Z","receivedAt":"2025-02-19T09:47:23Z","isPatch":false,"sender":{"key":"ossman@cendio.se","avatar":"https://gravatar.com/avatar/d72603ba4ea49f4dd970131b7944709d2c353664d7d389e2264caa6cc8eb618c?d=mp&s=160"},"body":"Hi,\n\nI'm trying to understand git's repacking behaviour, as the observed \nbehaviour doesn't match how I read the documentation or the code.\n\nThe problem we see is excessive backup increments for developer \ndirectories. The cause is that pack files keep getting regenerated for \nlarge repositories.\n\nUsers are not running 'git gc' manually, so the assumption is that this \nis caused by 'git gc --auto' being run implicitly.\n\n From what I can see in the code, and the documentation, it should only \npack up objects not already found in existing packs. Or at the very \nleast, not the objects found in the largest existing pack.\n\n(at least not until gc.autoPackLimit is hit)\n\nBut this isn't happening. Old packs are constantly being replaced by new \nones. Despite most of the objects being old and stable.\n\nWe tried gc.bigPackThreshold in the hope it would force it to reuse \npacks better. But all we got instead was duplication. It still creates \nnew packs with everything. It just stopped removing the old ones.\n\nSome guidance would be appreciated. I cannot find anything in the code \nor documentation that explains the current behaviour.\n\nRegads,\n-- \nPierre Ossman           Software Development\nCendio AB               https://cendio.com\nTeknikringen 8          https://twitter.com/ThinLinc\n583 30 Linköping        https://facebook.com/ThinLinc\nPhone: +46-13-214600\n\nA: Because it messes up the order in which people normally read text.\nQ: Why is top-posting such a bad thing?\n\n"},{"id":"512754","messageId":"CAG1j3zGmA30w545+-6qFV6x+3HvM+fueYH-rv-_gaSTpZStMHg@mail.gmail.com","threadId":"62975","inReplyTo":"1524b9a5-6f8b-4537-ba6b-bdfdd4b1bdcb@cendio.se","subject":"Re: [External] git keeps recreating packs, exploding backup increments","fromName":"Han Young","fromEmail":"hanyang.tony@bytedance.com","sentAt":"2025-02-20T03:03:01Z","receivedAt":"2025-02-20T03:03:14Z","isPatch":false,"sender":{"key":"hanyang.tony@bytedance.com","avatar":"https://avatars.githubusercontent.com/u/108711387?v=4"},"body":"On Wed, Feb 19, 2025 at 5:58 PM Pierre Ossman <ossman@cendio.se> wrote:\n> We tried gc.bigPackThreshold in the hope it would force it to reuse\n> packs better. But all we got instead was duplication. It still creates\n> new packs with everything. It just stopped removing the old ones.\n\nIs the repo partially cloned? git-repack will always pack promisor\npacks even if it's a keep pack. This patch would fix it\nhttps://lore.kernel.org/git/2728513.vuYhMxLoTh@mintaka.ncbr.muni.cz/\n\nThanks\n"},{"id":"512758","messageId":"ba212d4e-32c5-472a-8604-2a2653bde17c@cendio.se","threadId":"62975","inReplyTo":"CAG1j3zGmA30w545+-6qFV6x+3HvM+fueYH-rv-_gaSTpZStMHg@mail.gmail.com","subject":"Re: [External] git keeps recreating packs, exploding backup increments","fromName":"Pierre Ossman","fromEmail":"ossman@cendio.se","sentAt":"2025-02-20T08:26:38Z","receivedAt":"2025-02-20T08:26:41Z","isPatch":false,"sender":{"key":"ossman@cendio.se","avatar":"https://gravatar.com/avatar/d72603ba4ea49f4dd970131b7944709d2c353664d7d389e2264caa6cc8eb618c?d=mp&s=160"},"body":"On 20/02/2025 04:03, Han Young wrote:\n> On Wed, Feb 19, 2025 at 5:58 PM Pierre Ossman <ossman@cendio.se> wrote:\n>> We tried gc.bigPackThreshold in the hope it would force it to reuse\n>> packs better. But all we got instead was duplication. It still creates\n>> new packs with everything. It just stopped removing the old ones.\n> \n> Is the repo partially cloned? git-repack will always pack promisor\n> packs even if it's a keep pack. This patch would fix it\n> https://lore.kernel.org/git/2728513.vuYhMxLoTh@mintaka.ncbr.muni.cz/\n> \n\nYes, the big offender is often partially cloned. So that could be part \nof it, thanks.\n\nBut we're seeing it in other repositories as well. E.g. I have a \nlong-lived TigerVNC repository where the biggest pack file is just one \nweek old. In that case, it's merely 21 MiB, so it's not a practical \nissue. But it does show that git keeps replacing it.\n\nAnything I/we can do to shed more light on the issue?\n\nRegards,\n-- \nPierre Ossman           Software Development\nCendio AB               https://cendio.com\nTeknikringen 8          https://twitter.com/ThinLinc\n583 30 Linköping        https://facebook.com/ThinLinc\nPhone: +46-13-214600\n\nA: Because it messes up the order in which people normally read text.\nQ: Why is top-posting such a bad thing?\n"},{"id":"512807","messageId":"Z7g2aEpEboL5mvRa@pks.im","threadId":"62975","inReplyTo":"ba212d4e-32c5-472a-8604-2a2653bde17c@cendio.se","subject":"Re: [External] git keeps recreating packs, exploding backup increments","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2025-02-21T08:16:40Z","receivedAt":"2025-02-21T08:16:48Z","isPatch":false,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Thu, Feb 20, 2025 at 09:26:38AM +0100, Pierre Ossman wrote:\n> On 20/02/2025 04:03, Han Young wrote:\n> > On Wed, Feb 19, 2025 at 5:58 PM Pierre Ossman <ossman@cendio.se> wrote:\n> > > We tried gc.bigPackThreshold in the hope it would force it to reuse\n> > > packs better. But all we got instead was duplication. It still creates\n> > > new packs with everything. It just stopped removing the old ones.\n> > \n> > Is the repo partially cloned? git-repack will always pack promisor\n> > packs even if it's a keep pack. This patch would fix it\n> > https://lore.kernel.org/git/2728513.vuYhMxLoTh@mintaka.ncbr.muni.cz/\n> > \n> \n> Yes, the big offender is often partially cloned. So that could be part of\n> it, thanks.\n> \n> But we're seeing it in other repositories as well. E.g. I have a long-lived\n> TigerVNC repository where the biggest pack file is just one week old. In\n> that case, it's merely 21 MiB, so it's not a practical issue. But it does\n> show that git keeps replacing it.\n> \n> Anything I/we can do to shed more light on the issue?\n\nWell, one of the interesting things to learn would be how often you end\nup updating those repositories. You have discovered \"gc.autoPackLimit\"\nalready, which determines when exactly Git is going to repack existing\npackfiles into one, and mentioned that it doesn't seem to help you. But\nwhether it does or doesn't help really depends on how frequently you\ngain new packfiles in the impacted repositories.\n\nWhen you have fast-moving repositories and developers fetch several\ntimes per day, then it is quite likely that they accumulate multiple new\npackfiles per day. And thus, it's not all that unexpected that you will\nhave to repack the whole repository rather regularly. If so, this is\nworking as designed. You can tune the parameters for how often Git will\ndo an all-into-one repack, but also have to keep in mind that the more\npackfiles there are, the less efficient Git will in general be.\n\nThat being said, there is an alternative: Git nowadays doesn't use\ngit-gc(1) anymore to perform auto-maintenance, but instead it invokes\ngit-maintenance(1). And that command allows the user to pick what tasks\nshould be performed. By default it uses git-gc(1) under the hood indeed,\nbut you also ask it to not do so and instead use an alternative\nmechanism to pack your objects.\n\nThe alternative would be the \"incremental-repack\" task. This task does\nnot use git-gc(1) with its incremental/all-into-one repack split, but it\ninstead uses git-multi-pack-index(1). git-maintenance(1) tweaks the\n`--batch-size` parameter of `git multi-pack-index repack` so that it\ntypically doesn't have to repack the one large packfile, but combines at\nleast two smaller ones. I use a mechanism like that, which I've\nconfigured as follows:\n\n    [maintenance \"commit-graph\"]\n        enabled = true\n    [maintenance \"gc\"]\n        enabled = false\n    [maintenance \"incremental-repack\"]\n        enabled = true\n    [maintenance \"loose-objects\"]\n        enabled = true\n    [maintenance \"pack-refs\"]\n        enabled = true\n\nI think this strategy still isn't quite optimal, as nowadays we should\nprobably make use of `git repack --geometric` instead of manually\ncomputing batch sizes. This would ensure that the packfiles present in\nthe repository form a geometric sequence regarding their size, so you\nend up repacking the biggest packfile very infrequently. Such a task has\nnot been implemented yet, but it shouldn't be all that hard to do,\neither.\n\nPatrick\n"},{"id":"512886","messageId":"b0b1007c-34a6-4e3f-9982-cfe617affe1a@cendio.se","threadId":"62975","inReplyTo":"Z7g2aEpEboL5mvRa@pks.im","subject":"Re: [External] git keeps recreating packs, exploding backup increments","fromName":"Pierre Ossman","fromEmail":"ossman@cendio.se","sentAt":"2025-02-24T09:10:23Z","receivedAt":"2025-02-24T09:10:31Z","isPatch":false,"sender":{"key":"ossman@cendio.se","avatar":"https://gravatar.com/avatar/d72603ba4ea49f4dd970131b7944709d2c353664d7d389e2264caa6cc8eb618c?d=mp&s=160"},"body":"On 21/02/2025 09:16, Patrick Steinhardt wrote:\n>>\n>> Anything I/we can do to shed more light on the issue?\n> \n> Well, one of the interesting things to learn would be how often you end\n> up updating those repositories. You have discovered \"gc.autoPackLimit\"\n> already, which determines when exactly Git is going to repack existing\n> packfiles into one, and mentioned that it doesn't seem to help you. But\n> whether it does or doesn't help really depends on how frequently you\n> gain new packfiles in the impacted repositories.\n> \n> When you have fast-moving repositories and developers fetch several\n> times per day, then it is quite likely that they accumulate multiple new\n> packfiles per day. And thus, it's not all that unexpected that you will\n> have to repack the whole repository rather regularly. If so, this is\n> working as designed. You can tune the parameters for how often Git will\n> do an all-into-one repack, but also have to keep in mind that the more\n> packfiles there are, the less efficient Git will in general be.\n> \n\nI don't think the most problematic repo should be moving that fast. But \nI might be wrong. We've reverted all settings to default, and we'll try \nto keep an eye on what happens to the pack files to gain more understanding.\n\n> That being said, there is an alternative: Git nowadays doesn't use\n> git-gc(1) anymore to perform auto-maintenance, but instead it invokes\n> git-maintenance(1). And that command allows the user to pick what tasks\n> should be performed. By default it uses git-gc(1) under the hood indeed,\n> but you also ask it to not do so and instead use an alternative\n> mechanism to pack your objects.\n> \n\nThanks. This is definitely something we can try. We'll observe the \nsystem for now, to establish a new baseline. Then we'll try some of \nthese settings and see how it affect things.\n\nRegards,\n-- \nPierre Ossman           Software Development\nCendio AB               https://cendio.com\nTeknikringen 8          https://twitter.com/ThinLinc\n583 30 Linköping        https://facebook.com/ThinLinc\nPhone: +46-13-214600\n\nA: Because it messes up the order in which people normally read text.\nQ: Why is top-posting such a bad thing?\n"},{"id":"517651","messageId":"0f0a0dc6-2223-4d59-bc9e-5c8ebfbff09e@cendio.se","threadId":"62975","inReplyTo":"b0b1007c-34a6-4e3f-9982-cfe617affe1a@cendio.se","subject":"Re: [External] git keeps recreating packs, exploding backup increments","fromName":"Pierre Ossman","fromEmail":"ossman@cendio.se","sentAt":"2025-05-09T10:27:47Z","receivedAt":"2025-05-09T10:34:02Z","isPatch":false,"sender":{"key":"ossman@cendio.se","avatar":"https://gravatar.com/avatar/d72603ba4ea49f4dd970131b7944709d2c353664d7d389e2264caa6cc8eb618c?d=mp&s=160"},"body":"Following up on this old thread.\n\nI think the entire thing might be a false diagnostic on our part, and \ngit's default behaviour is working just fine for us.\n\nWe initially starting looking at this because the backups exploded in \nsize. But git is complex enough that we had a hard time examining things \nretroactively. Instead, we started monitoring things going forward to \nfind likely causes.\n\nWhat we did was to keep an eye on how new the files were in \n.git/objects/pack. And we kept seeing that the large packs were brand \nspanking new. Hence, the attempts at reconfiguring git, and the thread here.\n\nBut it seems we didn't look close enough. Although the timestamps \nsuggest that the packs are constantly being modified, the names and \ncontents actually stay the same.\n\nI've been keeping a closer eye on a couple of active repositories, and \nwe aren't actually seeing any excessive growth in size. But the largest \npack files always have a very current modification time.\n\nNo idea what caused that initial spike in backup storage. We'll have to \nrevisit that if it happens again.\n\nRegards,\n-- \nPierre Ossman           Software Development\nCendio AB               https://cendio.com\nTeknikringen 8          https://twitter.com/ThinLinc\n583 30 Linköping        https://facebook.com/ThinLinc\nPhone: +46-13-214600\n\nA: Because it messes up the order in which people normally read text.\nQ: Why is top-posting such a bad thing?\n"}]}