{"thread":{"id":"58882","subject":"Auto packing the repository - foreground or background in Windows?","startedAt":"2022-12-01T12:25:24Z","lastAt":"2022-12-09T15:12:21Z","messageCount":7,"participants":["Tao Klerks","Derrick Stolee","Ævar Arnfjörð Bjarmason","Jeff Hostetler"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"468311","messageId":"CAPMMpoiN=Rj_z9VEJZ4EFhb8gBeqb6H7JhTUBbn-b-t-jHRVnA@mail.gmail.com","threadId":"58882","inReplyTo":null,"subject":"Auto packing the repository - foreground or background in Windows?","fromName":"Tao Klerks","fromEmail":"tao@klerks.biz","sentAt":"2022-12-01T12:25:08Z","receivedAt":"2022-12-01T12:25:24Z","isPatch":false,"sender":{"key":"tao@klerks.biz","avatar":"https://avatars.githubusercontent.com/u/531704?v=4"},"body":"Hi folks,\n\nI came across a Windows user today whose \"fetch\" operations were\ntaking a long time, because their repository had passed some\npersistent maintenance-triggering threshold *and* the resulting\nauto-gc was running in the foreground (and not resolving the\nmaintenance-triggering condition automatically).\n\nThe user was seeing, at the end of their fetch, something like:\n\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nEnumerating objects: 311322, done.\nNothing new to pack.\nChecking connectivity: 1490123\n\nEventually, they noticed a subsequent recommendation to run \"git\nprune\", after the connectivity check completed, and after they did the\ngit prune, they started getting \"bad object\" errors on fetch - so\nthere was clearly something else going wrong somewhere...\n\nBut my *question* is: Does anyone know where I could/should look to\nunderstand why the GC was happening in the foreground, even though the\nmessage says it will run in the background?\n\nI don't know how to create the conditions for the auto-GC on demand\n(how to create lots of loose objects??), so I don't know how to verify\nwhether it ever runs in the background on Windows, or what that might\ndepend on. I saw some discussions in 2016, but I can't tell what the\nconclusion was; is it simply the case that git has been \"lying\" about\nrunning GC in the background, on windows, for all these years? Or is\nthere something specific going on in this user's environment?\n\nAny info welcome, thank you!\n\nTao\n"},{"id":"468631","messageId":"4dcad1f5-9ebd-d15a-b663-a3513ae1bcb8@github.com","threadId":"58882","inReplyTo":"CAPMMpoiN=Rj_z9VEJZ4EFhb8gBeqb6H7JhTUBbn-b-t-jHRVnA@mail.gmail.com","subject":"Re: Auto packing the repository - foreground or background in Windows?","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-12-06T18:03:35Z","receivedAt":"2022-12-06T18:03:41Z","isPatch":false,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 12/1/2022 7:25 AM, Tao Klerks wrote: \n> But my *question* is: Does anyone know where I could/should look to\n> understand why the GC was happening in the foreground, even though the\n> message says it will run in the background?\n\nOn Windows, Git's foreground process cannot complete without also\nkilling the background process. I'm not sure on the concrete details,\nbut the lack of a background \"git gc --auto\" here is deliberate for\nthat platform.\n \n> I don't know how to create the conditions for the auto-GC on demand\n> (how to create lots of loose objects??), so I don't know how to verify\n> whether it ever runs in the background on Windows, or what that might\n> depend on. I saw some discussions in 2016, but I can't tell what the\n> conclusion was; is it simply the case that git has been \"lying\" about\n> running GC in the background, on windows, for all these years? Or is\n> there something specific going on in this user's environment?\n\nInstead, the modern recommendation for repositories where \"git gc --auto\"\nwould be slow is to run \"git maintenance start\" which will schedule\nbackground maintenance jobs with the Windows scheduler. Those processes\nare built to do updates that are non-invasive to concurrent foreground\nprocesses. It also sets config to avoid \"git gc --auto\" commands at the\nend of foreground Git processes.\n\nSee [1] for more details.\n\n[1] https://git-scm.com/docs/git-maintenance\n\nThanks,\n-Stolee\n"},{"id":"468633","messageId":"221206.86o7sgbagq.gmgdl@evledraar.gmail.com","threadId":"58882","inReplyTo":"4dcad1f5-9ebd-d15a-b663-a3513ae1bcb8@github.com","subject":"Re: Auto packing the repository - foreground or background in Windows?","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2022-12-06T19:19:46Z","receivedAt":"2022-12-06T19:26:05Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Dec 06 2022, Derrick Stolee wrote:\n\n> On 12/1/2022 7:25 AM, Tao Klerks wrote: \n>> But my *question* is: Does anyone know where I could/should look to\n>> understand why the GC was happening in the foreground, even though the\n>> message says it will run in the background?\n>\n> On Windows, Git's foreground process cannot complete without also\n> killing the background process. I'm not sure on the concrete details,\n> but the lack of a background \"git gc --auto\" here is deliberate for\n> that platform.\n>  \n>> I don't know how to create the conditions for the auto-GC on demand\n>> (how to create lots of loose objects??), so I don't know how to verify\n>> whether it ever runs in the background on Windows, or what that might\n>> depend on. I saw some discussions in 2016, but I can't tell what the\n>> conclusion was; is it simply the case that git has been \"lying\" about\n>> running GC in the background, on windows, for all these years? Or is\n>> there something specific going on in this user's environment?\n>\n> Instead, the modern recommendation for repositories where \"git gc --auto\"\n> would be slow is to run \"git maintenance start\" which will schedule\n> background maintenance jobs with the Windows scheduler. Those processes\n> are built to do updates that are non-invasive to concurrent foreground\n> processes. It also sets config to avoid \"git gc --auto\" commands at the\n> end of foreground Git processes.\n>\n> See [1] for more details.\n>\n> [1] https://git-scm.com/docs/git-maintenance\n\nThat's good advice, but Tao is pointing out that the message we emit is\nbuggy here, which is a correct.\n\nThe problem is just that on Windows we always fail to daemonize(), but\ndidn't correct the bits that know that to the bits that emit the\nmessage.\n\nI think this should fix it:\n\t\n\tdiff --git a/builtin/gc.c b/builtin/gc.c\n\tindex 02455fdcd73..a5f599ebff0 100644\n\t--- a/builtin/gc.c\n\t+++ b/builtin/gc.c\n\t@@ -623,9 +623,11 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n\t \t\tif (!need_to_gc())\n\t \t\t\treturn 0;\n\t \t\tif (!quiet) {\n\t+#ifndef NO_POSIX_GOODIES\n\t \t\t\tif (detach_auto)\n\t \t\t\t\tfprintf(stderr, _(\"Auto packing the repository in background for optimum performance.\\n\"));\n\t \t\t\telse\n\t+#endif\n\t \t\t\t\tfprintf(stderr, _(\"Auto packing the repository for optimum performance.\\n\"));\n\t \t\t\tfprintf(stderr, _(\"See \\\"git help gc\\\" for manual housekeeping.\\n\"));\n\t \t\t}\nTao: If you're interested do you mind carrying that (or some other\nsimilar) patch forward?\n\nThe above is: Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>\n"},{"id":"468648","messageId":"091329e0-9af3-b853-0ed3-edd10d2a2b72@jeffhostetler.com","threadId":"58882","inReplyTo":"4dcad1f5-9ebd-d15a-b663-a3513ae1bcb8@github.com","subject":"Re: Auto packing the repository - foreground or background in Windows?","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2022-12-06T22:41:20Z","receivedAt":"2022-12-06T22:41:25Z","isPatch":false,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 12/6/22 1:03 PM, Derrick Stolee wrote:\n> On 12/1/2022 7:25 AM, Tao Klerks wrote:\n>> But my *question* is: Does anyone know where I could/should look to\n>> understand why the GC was happening in the foreground, even though the\n>> message says it will run in the background?\n> \n> On Windows, Git's foreground process cannot complete without also\n> killing the background process. I'm not sure on the concrete details,\n> but the lack of a background \"git gc --auto\" here is deliberate for\n> that platform.\n\nHere the GC code uses `daemonize()`.  On Posix this is a wrapper around\n`fork()` where the parent exits and the child continues the computation\n(without stdin/out).\n\nHowever, `daemonize()` just returns an ENOSYS on Windows, since Windows\ndoesn't have `fork()`.  The net result is that the foreground process\nfalls thru and does the actual work.\n\n\n[...]\n> Instead, the modern recommendation for repositories where \"git gc --auto\"\n> would be slow is to run \"git maintenance start\" which will schedule\n> background maintenance jobs with the Windows scheduler. Those processes\n> are built to do updates that are non-invasive to concurrent foreground\n> processes. It also sets config to avoid \"git gc --auto\" commands at the\n> end of foreground Git processes.\n> \n> See [1] for more details.\n> \n> [1] https://git-scm.com/docs/git-maintenance\n\nIt is possible to do the GC in the background on Windows (using other\ntechniques), but I don't think it is worth the bother now that we have\n`git maintenance` to handle it.\n\nAnd yes, AEvar's suggested fix for printing the correct error message\nlooks helpful.\n\n\nJeff\n"},{"id":"468786","messageId":"CAPMMpojW1mHmZ7_P=JpDZr2JjNxn-Rt_CP5VjtnMVJ6ZDFk36g@mail.gmail.com","threadId":"58882","inReplyTo":"091329e0-9af3-b853-0ed3-edd10d2a2b72@jeffhostetler.com","subject":"Re: Auto packing the repository - foreground or background in Windows?","fromName":"Tao Klerks","fromEmail":"tao@klerks.biz","sentAt":"2022-12-08T14:29:44Z","receivedAt":"2022-12-08T14:30:01Z","isPatch":false,"sender":{"key":"tao@klerks.biz","avatar":"https://avatars.githubusercontent.com/u/531704?v=4"},"body":"Thank you all. I will try to carry forward a patch with this change\nand a hint about the recommended maintenance approach for longer runs.\n\nOn Tue, Dec 6, 2022 at 11:41 PM Jeff Hostetler <git@jeffhostetler.com> wrote:\n>\n>\n>\n> On 12/6/22 1:03 PM, Derrick Stolee wrote:\n> > On 12/1/2022 7:25 AM, Tao Klerks wrote:\n> >> But my *question* is: Does anyone know where I could/should look to\n> >> understand why the GC was happening in the foreground, even though the\n> >> message says it will run in the background?\n> >\n> > On Windows, Git's foreground process cannot complete without also\n> > killing the background process. I'm not sure on the concrete details,\n> > but the lack of a background \"git gc --auto\" here is deliberate for\n> > that platform.\n>\n> Here the GC code uses `daemonize()`.  On Posix this is a wrapper around\n> `fork()` where the parent exits and the child continues the computation\n> (without stdin/out).\n>\n> However, `daemonize()` just returns an ENOSYS on Windows, since Windows\n> doesn't have `fork()`.  The net result is that the foreground process\n> falls thru and does the actual work.\n>\n>\n> [...]\n> > Instead, the modern recommendation for repositories where \"git gc --auto\"\n> > would be slow is to run \"git maintenance start\" which will schedule\n> > background maintenance jobs with the Windows scheduler. Those processes\n> > are built to do updates that are non-invasive to concurrent foreground\n> > processes. It also sets config to avoid \"git gc --auto\" commands at the\n> > end of foreground Git processes.\n> >\n> > See [1] for more details.\n> >\n> > [1] https://git-scm.com/docs/git-maintenance\n>\n> It is possible to do the GC in the background on Windows (using other\n> techniques), but I don't think it is worth the bother now that we have\n> `git maintenance` to handle it.\n>\n> And yes, AEvar's suggested fix for printing the correct error message\n> looks helpful.\n>\n>\n> Jeff\n"},{"id":"468788","messageId":"CAPMMpoii52KrR2MBpPdSEH8-jc7uMPzi4DH5g2bchwd-RPNTJw@mail.gmail.com","threadId":"58882","inReplyTo":"4dcad1f5-9ebd-d15a-b663-a3513ae1bcb8@github.com","subject":"Re: Auto packing the repository - foreground or background in Windows?","fromName":"Tao Klerks","fromEmail":"tao@klerks.biz","sentAt":"2022-12-08T14:52:07Z","receivedAt":"2022-12-08T14:52:29Z","isPatch":false,"sender":{"key":"tao@klerks.biz","avatar":"https://avatars.githubusercontent.com/u/531704?v=4"},"body":"On Tue, Dec 6, 2022 at 7:03 PM Derrick Stolee <derrickstolee@github.com> wrote:\n>\n> Instead, the modern recommendation for repositories where \"git gc --auto\"\n> would be slow is to run \"git maintenance start\" which will schedule\n> background maintenance jobs with the Windows scheduler. Those processes\n> are built to do updates that are non-invasive to concurrent foreground\n> processes. It also sets config to avoid \"git gc --auto\" commands at the\n> end of foreground Git processes.\n>\n> See [1] for more details.\n>\n> [1] https://git-scm.com/docs/git-maintenance\n>\n\nThanks Stolee, I've known about the existence of this system for a\nwhile, but I can't quite figure out what's recommended for who, when,\ngiven the doc at https://git-scm.com/docs/git-maintenance\n\nClearly on Windows, one reason to do \"git maintenance start\" is to\navoid foregrounded \"git gc --auto\" runs later. That's a clear enough\nbenefit to say \"frequent users of large repos on windows *should* run\n'git maintenance start' (or have some setup process or GUI do it for\nthem) on those large repos\".\n\nIs there a corresponding tangible benefit on MacOS and/or Linux, over\nsimply getting \"git gc --auto\" do its backgrounded thing when it feels\nlike it? Or is there an eventual plan to *switch* from the current\n\"git gc --auto\" spawning to a \"git maintenance start\" execution when\ntrigger conditions are met? Are there any *dis*advantages to running\n\"git maintenance start\" in general or on any given platform?\n\nFor \"my users\", I have something like Scalar that can start the\nmaintenance on the repo where it's needed - but it seems like there\nwill be lots of users out there in the world who clone things like the\nlinux repo, which looks like it is big enough to warrant these kinds\nof concerns, but it doesn't seem obvious that anyone will ever find\n\"https://git-scm.com/docs/git-maintenance\" and decide to run \"git\nmaintenance start\" on their own...\n\nAs I noted in another email, I propose to replace \"Auto packing the\nrepository for optimum performance\" with something like \"Auto packing\nthe repository for optimum performance; to run this kind of\nmaintenance in the background, see 'git maintenance' at\nhttps://git-scm.com/docs/git-maintenance.\" - but I imagine I'm missing\na bigger picture / a long-term plan for how these two mechanisms\nshould interact.\n\nMy apologies if I've missed one or many conversations about this on\nthe list, but maybe a pointer here can also help me add directional\nhints at https://git-scm.com/docs/git-maintenance for \"outside users\"?\n"},{"id":"468832","messageId":"1e8386ed-6acb-0deb-2e46-2e9dbd6e4ad5@github.com","threadId":"58882","inReplyTo":"CAPMMpoii52KrR2MBpPdSEH8-jc7uMPzi4DH5g2bchwd-RPNTJw@mail.gmail.com","subject":"Re: Auto packing the repository - foreground or background in Windows?","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2022-12-09T15:11:56Z","receivedAt":"2022-12-09T15:12:21Z","isPatch":false,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 12/8/2022 9:52 AM, Tao Klerks wrote:\n> On Tue, Dec 6, 2022 at 7:03 PM Derrick Stolee <derrickstolee@github.com> wrote:\n>>\n>> Instead, the modern recommendation for repositories where \"git gc --auto\"\n>> would be slow is to run \"git maintenance start\" which will schedule\n>> background maintenance jobs with the Windows scheduler. Those processes\n>> are built to do updates that are non-invasive to concurrent foreground\n>> processes. It also sets config to avoid \"git gc --auto\" commands at the\n>> end of foreground Git processes.\n>>\n>> See [1] for more details.\n>>\n>> [1] https://git-scm.com/docs/git-maintenance\n>>\n> \n> Thanks Stolee, I've known about the existence of this system for a\n> while, but I can't quite figure out what's recommended for who, when,\n> given the doc at https://git-scm.com/docs/git-maintenance\n\nThanks for the feedback that this document could use a clearer\nhigh-level description for recommended ways to use the command, and\n_when_.\n\nOne goal when creating the documentation was to _not_ recommend a\nspecific use pattern, instead focusing on the many ways a user could\ncustomize their maintenance patterns. Perhaps the feature has\nstabilized enough (and shown its benefits) that we could add a\nrecommended use section.\n \n> Clearly on Windows, one reason to do \"git maintenance start\" is to\n> avoid foregrounded \"git gc --auto\" runs later. That's a clear enough\n> benefit to say \"frequent users of large repos on windows *should* run\n> 'git maintenance start' (or have some setup process or GUI do it for\n> them) on those large repos\".\n> \n> Is there a corresponding tangible benefit on MacOS and/or Linux, over\n> simply getting \"git gc --auto\" do its backgrounded thing when it feels\n> like it? Or is there an eventual plan to *switch* from the current\n> \"git gc --auto\" spawning to a \"git maintenance start\" execution when\n> trigger conditions are met? Are there any *dis*advantages to running\n> \"git maintenance start\" in general or on any given platform?\n\nFor large repositories, the default 'git gc --auto' takes a lot of\nresources to rewrite all object data into a single pack-file. The\nbackground maintenance does smaller, incremental repacks. Here,\n\"large\" means \"more than 2GB of packed object data\", since that's\nthe default limit for the incremental repacks starting a new pack.\n\nThere's other benefits where it does hourly prefetches, getting\nobject data from remotes before the user requests a ref update\nthrough 'git fetch' or 'git pull'. Those foreground operations\nspeed up, as well.\n\n> For \"my users\", I have something like Scalar that can start the\n> maintenance on the repo where it's needed - but it seems like there\n> will be lots of users out there in the world who clone things like the\n> linux repo, which looks like it is big enough to warrant these kinds\n> of concerns, but it doesn't seem obvious that anyone will ever find\n> \"https://git-scm.com/docs/git-maintenance\" and decide to run \"git\n> maintenance start\" on their own...\n\nWe do what we can to advertise these kinds of features, but at some\npoint users need to self-discover things. But that's also a motivation\nfor the Scalar command: the user can relax some control to allow the\nScalar command to choose those recommended settings on behalf of the\nuser.\n\n> As I noted in another email, I propose to replace \"Auto packing the\n> repository for optimum performance\" with something like \"Auto packing\n> the repository for optimum performance; to run this kind of\n> maintenance in the background, see 'git maintenance' at\n> https://git-scm.com/docs/git-maintenance.\" - but I imagine I'm missing\n> a bigger picture / a long-term plan for how these two mechanisms\n> should interact.\n\nA message that points out 'git maintenance' like this might work best\nas part of the \"advice\" API, so those who don't want to see the\nmessage every time could disable it.\n\n> My apologies if I've missed one or many conversations about this on\n> the list, but maybe a pointer here can also help me add directional\n> hints at https://git-scm.com/docs/git-maintenance for \"outside users\"?\n\nI'm trying to think of a builtin whose documentation has such strong\n\"recommended use\" language.\n\nThe best I could think about are commands with substantial \"examples\"\nsections, such as 'git bundle'.\n\nA more radical approach would be to create a new doc type that\nprovides recommendations for how to manage large repositories. I\nimagine it would be sorted in order of increasing complexity,\nsomething like:\n\n 1. Use 'scalar' and see if it works for your needs.\n\n 2. Self-serve with 'git maintenance start', 'git sparse-checkout',\n    partial clone, and feature.manyFiles=true as needed.\n\n 3. Go deep on individual plumbing commands and config options\n    that provide knobs to tweak how Git manages information.\n\nI think starting with some examples or a \"recommended use\" section\nfor 'git maintenance' would be a better first step.\n\nThanks,\n-Stolee\n"}]}