{"thread":{"id":"55444","subject":"There should have be git gc --repack-arguments","startedAt":"2021-04-07T12:10:50Z","lastAt":"2021-04-09T15:49:14Z","messageCount":10,"participants":["Bagas Sanjaya","Jeff King","Bryan Turner","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"421116","messageId":"b35a68a1-e693-5502-7a28-a1dd8222d3a0@gmail.com","threadId":"55444","inReplyTo":null,"subject":"There should have be git gc --repack-arguments","fromName":"Bagas Sanjaya","fromEmail":"bagasdotme@gmail.com","sentAt":"2021-04-07T12:10:43Z","receivedAt":"2021-04-07T12:10:50Z","isPatch":false,"sender":{"key":"bagasdotme@gmail.com","avatar":"https://avatars.githubusercontent.com/u/40219486?v=4"},"body":"Hi,\n\nI request that git gc should have --repack-arguments option. The value\nof this option should be passed to git repack.\n\nThe use case is when I have very large repos (such as GCC and Linux kernel)\non a server with small RAM (1-2 GB). When doing gc on such repo, the repack\nstep may hang because git-repack have to create single large packfile which\ncan be larger than available memory (RAM+swap), so it must be necessary to\ndo git repack --window-memory= --max-pack-size= to create split and smaller packs instead.\n\nThere should also git config item gc.repackArguments, which have the same\neffect as git gc --repack-arguments, with the option takes precedence over\nthe config.\n\n-- \nAn old man doll... just what I always wanted! - Clara\n"},{"id":"421138","messageId":"YG4J7vtTRVpGGLoo@coredump.intra.peff.net","threadId":"55444","inReplyTo":"b35a68a1-e693-5502-7a28-a1dd8222d3a0@gmail.com","subject":"Re: There should have be git gc --repack-arguments","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-04-07T19:37:18Z","receivedAt":"2021-04-07T19:37:22Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Apr 07, 2021 at 07:10:43PM +0700, Bagas Sanjaya wrote:\n\n> I request that git gc should have --repack-arguments option. The value\n> of this option should be passed to git repack.\n\nI think in general we prefer to make individual options configurable,\nrather than having a blanket \"pass along these options\" argument, for\ntwo reasons:\n\n - some options may cause the sub-program to behave unexpectedly. E.g.,\n if you put \"-a\" in the repack-arguments, that may be subverting\n git-gc's assumptions about how repack will behave\n\n - arguments are a list, not a string. So you have to provide some\n mechanism for splitting them (presumably on whitespace, but what if\n we need quoting)?\n\n> The use case is when I have very large repos (such as GCC and Linux kernel)\n> on a server with small RAM (1-2 GB). When doing gc on such repo, the repack\n> step may hang because git-repack have to create single large packfile which\n> can be larger than available memory (RAM+swap), so it must be necessary to\n> do git repack --window-memory= --max-pack-size= pack size> to create split and smaller packs instead.\n> \n> There should also git config item gc.repackArguments, which have the same\n> effect as git gc --repack-arguments, with the option takes precedence over\n> the config.\n\nYou can set pack.windowMemory in your config already, to solve the first\npart.\n\nYou can also set pack.packSizeLimit for the latter, though I do not\nrecommend it. It will not help with memory usage (neither while\nrepacking nor for later commands). We do mmap() the resulting packfiles,\nbut we rely on the operating system to manage the actual in-RAM working\nset (but that is also true with multiple packfiles; we are happy to map\nseveral of them at once). And it may make your on-disk size much larger.\nWe don't allow deltas between on-disk packs, which means some objects\nwhich could be stored as deltas won't be. That in turn hurts on a\nmemory-starved system because we'll need more block cache to perform the\nsame task. It also results in extra CPU when serving fetches or pushing,\nsince we'll try to find new deltas between the packs on the fly.\n\n-Peff\n"},{"id":"421139","messageId":"CAGyf7-GQ_1JV6X3Z0h4c3+Qy1eZ30RW-Mni=72p007md5NLKMg@mail.gmail.com","threadId":"55444","inReplyTo":"b35a68a1-e693-5502-7a28-a1dd8222d3a0@gmail.com","subject":"Re: There should have be git gc --repack-arguments","fromName":"Bryan Turner","fromEmail":"bturner@atlassian.com","sentAt":"2021-04-07T19:38:37Z","receivedAt":"2021-04-07T19:38:51Z","isPatch":false,"sender":{"key":"bturner@atlassian.com","avatar":"https://gravatar.com/avatar/16bcf3167981c1ef7c804e502642366d888a35b0d0b0a4ca01fdc442aa1acb1e?d=mp&s=160"},"body":"On Wed, Apr 7, 2021 at 5:10 AM Bagas Sanjaya wrote:\n>\n> Hi,\n>\n> I request that git gc should have --repack-arguments option. The value\n> of this option should be passed to git repack.\n>\n> The use case is when I have very large repos (such as GCC and Linux kernel)\n> on a server with small RAM (1-2 GB). When doing gc on such repo, the repack\n> step may hang because git-repack have to create single large packfile which\n> can be larger than available memory (RAM+swap), so it must be necessary to\n> do git repack --window-memory= --max-pack-size= pack size> to create split and smaller packs instead.\n\nI can't speak to the feature request, but since there are\nconfiguration knobs already for both of those, that implies you can\nuse git -c pack.windowMemory=... -c pack.packSizeLimit=... gc and\nthose configuration settings will be propagated to the git repack\nprocess that git gc runs.\n\n>\n> There should also git config item gc.repackArguments, which have the same\n> effect as git gc --repack-arguments, with the option takes precedence over\n> the config.\n\nPassing configuration settings as I show above would already take\nprecedence over any config file, since config from the command line is\nhigher priority.\n\nHope this helps!\nBryan\n"},{"id":"421149","messageId":"xmqq8s5tzv4f.fsf@gitster.g","threadId":"55444","inReplyTo":"YG4J7vtTRVpGGLoo@coredump.intra.peff.net","subject":"Re: There should have be git gc --repack-arguments","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-04-07T20:40:16Z","receivedAt":"2021-04-07T20:40:29Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King writes:\n\n>> ... git repack ... --max-pack-size= to create split and\n>> smaller packs instead.\n> ...\n> You can also set pack.packSizeLimit for the latter, though I do not\n> recommend it. It will not help with memory usage (neither while\n> repacking nor for later commands).\n\nIn other words, passing --max-pack-size, whether it is done with a\nnew --repack-arguments option or it is done with the existing\npack.packSizeLimit configuration, would make things worse.\n\nSo in conclusion:\n\n - attempting to repack everything into one pack on a memory starved\n box would be helped with reduced window memory size.\n\n - on a small box, it may make sense to avoid repacking everything\n into one in the first place, but we do not want the number of\n packs to grow unbounded.\n\nWould the new geometric repack feature help here, especially for the\nlatter?\n\n"},{"id":"421158","messageId":"YG4mImcQyTC1/S8X@coredump.intra.peff.net","threadId":"55444","inReplyTo":"xmqq8s5tzv4f.fsf@gitster.g","subject":"Re: There should have be git gc --repack-arguments","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-04-07T21:37:38Z","receivedAt":"2021-04-07T21:37:44Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Apr 07, 2021 at 01:40:16PM -0700, Junio C Hamano wrote:\n\n> Jeff King writes:\n> \n> >> ... git repack ... --max-pack-size= to create split and\n> >> smaller packs instead.\n> > ...\n> > You can also set pack.packSizeLimit for the latter, though I do not\n> > recommend it. It will not help with memory usage (neither while\n> > repacking nor for later commands).\n> \n> In other words, passing --max-pack-size, whether it is done with a\n> new --repack-arguments option or it is done with the existing\n> pack.packSizeLimit configuration, would make things worse.\n\nRight. I wish we didn't have --max-pack-size at all. I do not think it\nis ever a good idea, and it complicates the packing code quite a bit.\n\nThese days we have index v2 to let us address more than 4GB in a\npackfile. I suppose it's possible you could have a filesystem whose max\nfile size is smaller than your total packfile, but that seems pretty\nunlikely these days (even 32-bit systems tend to have large file\nsupport).\n\nBut that's all a tangent. :)\n\n> So in conclusion:\n> \n> - attempting to repack everything into one pack on a memory starved\n> box would be helped with reduced window memory size.\n\nYes, though less than you might think. It is only trying to keep the\nmemory used by delta compression at bay. The per-object book-keeping\ntends to be quite high by itself. If you are under memory pressure\nduring delta compression, you may also be better off reducing the number\nof threads (since each thread is simultaneously using windowMemory\nbytes).\n\n> - on a small box, it may make sense to avoid repacking everything\n> into one in the first place, but we do not want the number of\n> packs to grow unbounded.\n> \n> Would the new geometric repack feature help here, especially for the\n> latter?\n\nYes, I think it would. You'd perhaps want to generate a multi-pack-index\nfile, too, to avoid having to look for objects in multiple packs\nsequentially (we have a \"git repack --write-midx\" option on the way, as\nwell).\n\n-Peff\n"},{"id":"421162","messageId":"xmqqa6q9yc8c.fsf@gitster.g","threadId":"55444","inReplyTo":"YG4mImcQyTC1/S8X@coredump.intra.peff.net","subject":"Re: There should have be git gc --repack-arguments","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-04-07T22:13:39Z","receivedAt":"2021-04-07T22:13:46Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King writes:\n\n> On Wed, Apr 07, 2021 at 01:40:16PM -0700, Junio C Hamano wrote:\n>\n>> Jeff King writes:\n>> \n>> >> ... git repack ... --max-pack-size= to create split and\n>> >> smaller packs instead.\n>> > ...\n>> > You can also set pack.packSizeLimit for the latter, though I do not\n>> > recommend it. It will not help with memory usage (neither while\n>> > repacking nor for later commands).\n>> \n>> In other words, passing --max-pack-size, whether it is done with a\n>> new --repack-arguments option or it is done with the existing\n>> pack.packSizeLimit configuration, would make things worse.\n>\n> Right. I wish we didn't have --max-pack-size at all. I do not think it\n> is ever a good idea, and it complicates the packing code quite a bit.\n\nI suspect that the original motivation was sneaker-netting on\nmultiple floppy disks ;-)\n\n>> - on a small box, it may make sense to avoid repacking everything\n>> into one in the first place, but we do not want the number of\n>> packs to grow unbounded.\n>> \n>> Would the new geometric repack feature help here, especially for the\n>> latter?\n>\n> Yes, I think it would. You'd perhaps want to generate a multi-pack-index\n> file, too, to avoid having to look for objects in multiple packs\n> sequentially (we have a \"git repack --write-midx\" option on the way, as\n> well).\n\nThanks.\n"},{"id":"421165","messageId":"YG4ws7PiKKKjPUff@coredump.intra.peff.net","threadId":"55444","inReplyTo":"xmqqa6q9yc8c.fsf@gitster.g","subject":"Re: There should have be git gc --repack-arguments","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-04-07T22:22:43Z","receivedAt":"2021-04-07T22:22:46Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Apr 07, 2021 at 03:13:39PM -0700, Junio C Hamano wrote:\n\n> >> > You can also set pack.packSizeLimit for the latter, though I do not\n> >> > recommend it. It will not help with memory usage (neither while\n> >> > repacking nor for later commands).\n> >> \n> >> In other words, passing --max-pack-size, whether it is done with a\n> >> new --repack-arguments option or it is done with the existing\n> >> pack.packSizeLimit configuration, would make things worse.\n> >\n> > Right. I wish we didn't have --max-pack-size at all. I do not think it\n> > is ever a good idea, and it complicates the packing code quite a bit.\n> \n> I suspect that the original motivation was sneaker-netting on\n> multiple floppy disks ;-)\n\nThat had always been my impression, too. But when I looked in the\narchive while writing my earlier reply, most of the discussion near\n--max-pack-size had to do with the early index limitations.\n\nIf you are sneaker-netting, you are probably better off to just split\nthe pack at byte boundaries with an external tool anyway, for two\nreasons:\n\n - our max-pack-size is just a guideline. It only splits at object\n boundaries so if you have an object bigger than the max, we'll\n exceed it.\n\n - dedicated splitting tools often have useful extra features, like\n k-of-n error correction.\n\nBesides, if you are sneaker netting you'd want to use a bundle, and I\ndon't think bundles support max-pack-size. :)\n\nAnyway, all off-topic but an interesting diversion.\n\n-Peff\n"},{"id":"421210","messageId":"7c53b1e6-5801-7e96-2939-18abbd8d1e53@gmail.com","threadId":"55444","inReplyTo":"CAGyf7-GQ_1JV6X3Z0h4c3+Qy1eZ30RW-Mni=72p007md5NLKMg@mail.gmail.com","subject":"Re: There should have be git gc --repack-arguments","fromName":"Bagas Sanjaya","fromEmail":"bagasdotme@gmail.com","sentAt":"2021-04-08T13:31:13Z","receivedAt":"2021-04-08T13:31:26Z","isPatch":false,"sender":{"key":"bagasdotme@gmail.com","avatar":"https://avatars.githubusercontent.com/u/40219486?v=4"},"body":"On 08/04/21 02.38, Bryan Turner wrote:\n> I can't speak to the feature request, but since there are\n> configuration knobs already for both of those, that implies you can\n> use git -c pack.windowMemory=... -c pack.packSizeLimit=... gc and\n> those configuration settings will be propagated to the git repack\n> process that git gc runs.\n\nOops, I overlooked that. Thanks for reminding me!\n\n-- \nAn old man doll... just what I always wanted! - Clara\n"},{"id":"421367","messageId":"1edb799b-5e17-264d-e525-5355c52af36a@gmail.com","threadId":"55444","inReplyTo":"YG4ws7PiKKKjPUff@coredump.intra.peff.net","subject":"Re: There should have be git gc --repack-arguments","fromName":"Bagas Sanjaya","fromEmail":"bagasdotme@gmail.com","sentAt":"2021-04-09T09:58:32Z","receivedAt":"2021-04-09T10:00:20Z","isPatch":false,"sender":{"key":"bagasdotme@gmail.com","avatar":"https://avatars.githubusercontent.com/u/40219486?v=4"},"body":"On 08/04/21 05.22, Jeff King wrote:\n> If you are sneaker-netting, you are probably better off to just split\n> the pack at byte boundaries with an external tool anyway, for two\n> reasons:\n> \n> - our max-pack-size is just a guideline. It only splits at object\n> boundaries so if you have an object bigger than the max, we'll\n> exceed it.\n> \n> - dedicated splitting tools often have useful extra features, like\n> k-of-n error correction.\n> \nWhat external tools are for splitting packs? Can splitted packs\nby such tools still be usable by Git?\n\n-- \nAn old man doll... just what I always wanted! - Clara\n"},{"id":"421412","messageId":"YHB3d31eTACBd6pY@coredump.intra.peff.net","threadId":"55444","inReplyTo":"1edb799b-5e17-264d-e525-5355c52af36a@gmail.com","subject":"Re: There should have be git gc --repack-arguments","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-04-09T15:49:11Z","receivedAt":"2021-04-09T15:49:14Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Apr 09, 2021 at 04:58:32PM +0700, Bagas Sanjaya wrote:\n\n> On 08/04/21 05.22, Jeff King wrote:\n> > If you are sneaker-netting, you are probably better off to just split\n> > the pack at byte boundaries with an external tool anyway, for two\n> > reasons:\n> > \n> > - our max-pack-size is just a guideline. It only splits at object\n> > boundaries so if you have an object bigger than the max, we'll\n> > exceed it.\n> > \n> > - dedicated splitting tools often have useful extra features, like\n> > k-of-n error correction.\n> > \n> What external tools are for splitting packs? Can splitted packs\n> by such tools still be usable by Git?\n\nNo, but you can reassemble the parts at the destination before feeding\nthem to Git. On a system with normal posix tools, you can split like:\n\n git pack-objects --stdout --all