{"thread":{"id":"53847","subject":"Re: [PATCH 00/21] Maintenance builtin, allowing 'gc --auto' customization","startedAt":"2020-07-13T06:19:06Z","lastAt":"2020-07-14T13:46:15Z","messageCount":2,"participants":["Son Luong Ngoc","Derrick Stolee"],"isPatch":true,"patchVersion":1,"patchTotal":21},"messages":[{"id":"401474","messageId":"CAL3xRKfTTr=Tkq2NY7zUwCSWGHXO2FAwEHKvV2-4AavX9Cs7ow@mail.gmail.com","threadId":"53847","inReplyTo":null,"subject":"Re: [PATCH 00/21] Maintenance builtin, allowing 'gc --auto' customization","fromName":"Son Luong Ngoc","fromEmail":"sluongng@gmail.com","sentAt":"2020-07-13T06:18:52Z","receivedAt":"2020-07-13T06:19:06Z","isPatch":true,"sender":{"key":"sluongng@gmail.com","avatar":"https://avatars.githubusercontent.com/u/26684313?v=4"},"body":"Hi Derrick,\n\n> This is a second attempt at redesigning Git's repository maintenance\n> patterns. The first attempt [1] included a way to run jobs in the background\n> using a long-lived process; that idea was rejected and is not included in\n> this series. A future series will use the OS to handle scheduling tasks.\n>\n> [1]\n> https://lore.kernel.org/git/pull.597.git.1585946894.gitgitgadget@gmail.com/\n>\n> As mentioned before, git gc already plays the role of maintaining Git\n> repositories. It has accumulated several smaller pieces in its long history,\n> including:\n>\n>  1. Repacking all reachable objects into one pack-file (and deleting\n>     unreachable objects).\n>  2. Packing refs.\n>  3. Expiring reflogs.\n>  4. Clearing rerere logs.\n>  5. Updating the commit-graph file.\n\nIt's worth mentioning 'git worktree prune' as well.\n\n>\n> While expiring reflogs, clearing rererelogs, and deleting unreachable\n> objects are suitable under the guise of \"garbage collection\", packing refs\n> and updating the commit-graph file are not as obviously fitting. Further,\n> these operations are \"all or nothing\" in that they rewrite almost all\n> repository data, which does not perform well at extremely large scales.\n> These operations can also be disruptive to foreground Git commands when git\n> gc --auto triggers during routine use.\n>\n> This series does not intend to change what git gc does, but instead create\n> new choices for automatic maintenance activities, of which git gc remains\n> the only one enabled by default.\n>\n> The new maintenance tasks are:\n>\n>  * 'commit-graph' : write and verify a single layer of an incremental\n>    commit-graph.\n>  * 'loose-objects' : prune packed loose objects, then create a new pack from\n>    a batch of loose objects.\n>  * 'pack-files' : expire redundant packs from the multi-pack-index, then\n>    repack using the multi-pack-index's incremental repack strategy.\n>  * 'fetch' : fetch from each remote, storing the refs in 'refs/hidden//'.\n\nAs some of the previous discussions [1] have raised, I think 'prefetch' would\ncommunicate the refs' purpose better than just 'hidden'.\nIn-fact, I would suggest naming the task 'prefetch' instead, just to avoid\npotential communication issue between 'git fetch' and 'git maintenance fetch'.\n\n[1]: https://lore.kernel.org/git/xmqqeet1y8wy.fsf@gitster.c.googlers.com/\n\n>\n> These tasks are all disabled by default, but can be enabled with config\n> options or run explicitly using \"git maintenance run --task=\". There are\n> additional config options to allow customizing the conditions for which the\n> tasks run during the '--auto' option. ('fetch' will never run with the\n> '--auto' option.)\n>\n>  Because 'gc' is implemented as a maintenance task, the most dramatic change\n> of this series is to convert the 'git gc --auto' calls into 'git maintenance\n> run --auto' calls at the end of some Git commands. By default, the only\n> change is that 'git gc --auto' will be run below an additional 'git\n> maintenance' process.\n>\n> The 'git maintenance' builtin has a 'run' subcommand so it can be extended\n> later with subcommands that manage background maintenance, such as 'start',\n> 'stop', 'pause', or 'schedule'. These are not the subject of this series, as\n> it is important to focus on the maintenance activities themselves.\n>\n> An expert user could set up scheduled background maintenance themselves with\n> the current series. I have the following crontab data set up to run\n> maintenance on an hourly basis:\n>\n> 0 * * * * git -C /<path-to-repo> maintenance run --no-quiet >>/<path-to-repo>/.git/maintenance.log\n\nPerhaps the logging should be included inside the maintenance command instead\nof relying on the append here?\nGiven that we have 'gc.log', I would imagine 'maintenance.log' is not\ntoo far-fetched?\n\n>\n> My config includes all tasks except the 'gc' task. The hourly run is\n> over-aggressive, but is sufficient for testing. I'll replace it with daily\n> when I feel satisfied.\n>\n> Hopefully this direction is seen as a positive one. My goal was to add more\n> options for expert users, along with the flexibility to create background\n> maintenance via the OS in a later series.\n>\n> OUTLINE\n> =======\n>\n> Patches 1-4 remove some references to the_repository in builtin/gc.c before\n> we start depending on code in that builtin.\n>\n> Patches 5-7 create the 'git maintenance run' builtin and subcommand as a\n> simple shim over 'git gc' and replaces calls to 'git gc --auto' from other\n> commands.\n>\n> Patches 8-15 create new maintenance tasks. These are the same tasks sent in\n> the previous RFC.\n>\n> Patches 16-21 create more customization through config and perform other\n> polish items.\n>\n> FUTURE WORK\n> ===========\n>\n>  * Add 'start', 'stop', and 'schedule' subcommands to initialize the\n>    commands run in the background.\n>\n>\n>  * Split the 'gc' builtin into smaller maintenance tasks that are enabled by\n>    default, but might have different '--auto' conditions and more config\n>    options.\n>\n>\n>  * Replace config like 'gc.writeCommitGraph' and 'fetch.writeCommitGraph'\n>    with use of the 'commit-graph' task.\n>\n>\n>\n> Thanks, -Stolee\n\nThanks,\nSon Luong.\n"},{"id":"401547","messageId":"f9b2d585-d579-235c-034e-ab7e2c953d16@gmail.com","threadId":"53847","inReplyTo":"CAL3xRKfTTr=Tkq2NY7zUwCSWGHXO2FAwEHKvV2-4AavX9Cs7ow@mail.gmail.com","subject":"Re: [PATCH 00/21] Maintenance builtin, allowing 'gc --auto' customization","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-07-14T13:46:10Z","receivedAt":"2020-07-14T13:46:15Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 7/13/2020 2:18 AM, Son Luong Ngoc wrote:\n> Hi Derrick,\n> \n>> This is a second attempt at redesigning Git's repository maintenance\n>> patterns. The first attempt [1] included a way to run jobs in the background\n>> using a long-lived process; that idea was rejected and is not included in\n>> this series. A future series will use the OS to handle scheduling tasks.\n>>\n>> [1]\n>> https://lore.kernel.org/git/pull.597.git.1585946894.gitgitgadget@gmail.com/\n>>\n>> As mentioned before, git gc already plays the role of maintaining Git\n>> repositories. It has accumulated several smaller pieces in its long history,\n>> including:\n>>\n>>  1. Repacking all reachable objects into one pack-file (and deleting\n>>     unreachable objects).\n>>  2. Packing refs.\n>>  3. Expiring reflogs.\n>>  4. Clearing rerere logs.\n>>  5. Updating the commit-graph file.\n> \n> It's worth mentioning 'git worktree prune' as well.\n\nGood point. I'll also say \"including, but not limited to:\"\n\n>> While expiring reflogs, clearing rererelogs, and deleting unreachable\n>> objects are suitable under the guise of \"garbage collection\", packing refs\n>> and updating the commit-graph file are not as obviously fitting. Further,\n>> these operations are \"all or nothing\" in that they rewrite almost all\n>> repository data, which does not perform well at extremely large scales.\n>> These operations can also be disruptive to foreground Git commands when git\n>> gc --auto triggers during routine use.\n>>\n>> This series does not intend to change what git gc does, but instead create\n>> new choices for automatic maintenance activities, of which git gc remains\n>> the only one enabled by default.\n>>\n>> The new maintenance tasks are:\n>>\n>>  * 'commit-graph' : write and verify a single layer of an incremental\n>>    commit-graph.\n>>  * 'loose-objects' : prune packed loose objects, then create a new pack from\n>>    a batch of loose objects.\n>>  * 'pack-files' : expire redundant packs from the multi-pack-index, then\n>>    repack using the multi-pack-index's incremental repack strategy.\n>>  * 'fetch' : fetch from each remote, storing the refs in 'refs/hidden//'.\n> \n> As some of the previous discussions [1] have raised, I think 'prefetch' would\n> communicate the refs' purpose better than just 'hidden'.\n> In-fact, I would suggest naming the task 'prefetch' instead, just to avoid\n> potential communication issue between 'git fetch' and 'git maintenance fetch'.\n> \n> [1]: https://lore.kernel.org/git/xmqqeet1y8wy.fsf@gitster.c.googlers.com/\n\nThanks for the reminder. I'll rename the task as you suggest.\n\n>> These tasks are all disabled by default, but can be enabled with config\n>> options or run explicitly using \"git maintenance run --task=\". There are\n>> additional config options to allow customizing the conditions for which the\n>> tasks run during the '--auto' option. ('fetch' will never run with the\n>> '--auto' option.)\n>>\n>>  Because 'gc' is implemented as a maintenance task, the most dramatic change\n>> of this series is to convert the 'git gc --auto' calls into 'git maintenance\n>> run --auto' calls at the end of some Git commands. By default, the only\n>> change is that 'git gc --auto' will be run below an additional 'git\n>> maintenance' process.\n>>\n>> The 'git maintenance' builtin has a 'run' subcommand so it can be extended\n>> later with subcommands that manage background maintenance, such as 'start',\n>> 'stop', 'pause', or 'schedule'. These are not the subject of this series, as\n>> it is important to focus on the maintenance activities themselves.\n>>\n>> An expert user could set up scheduled background maintenance themselves with\n>> the current series. I have the following crontab data set up to run\n>> maintenance on an hourly basis:\n>>\n>> 0 * * * * git -C /<path-to-repo> maintenance run --no-quiet >>/<path-to-repo>/.git/maintenance.log\n> \n> Perhaps the logging should be included inside the maintenance command instead\n> of relying on the append here?\n> Given that we have 'gc.log', I would imagine 'maintenance.log' is not\n> too far-fetched?\n\nI'll research gc.log and how that works.\n\n>> My config includes all tasks except the 'gc' task. The hourly run is\n>> over-aggressive, but is sufficient for testing. I'll replace it with daily\n>> when I feel satisfied.\n>>\n>> Hopefully this direction is seen as a positive one. My goal was to add more\n>> options for expert users, along with the flexibility to create background\n>> maintenance via the OS in a later series.\n>>\n>> OUTLINE\n>> =======\n>>\n>> Patches 1-4 remove some references to the_repository in builtin/gc.c before\n>> we start depending on code in that builtin.\n>>\n>> Patches 5-7 create the 'git maintenance run' builtin and subcommand as a\n>> simple shim over 'git gc' and replaces calls to 'git gc --auto' from other\n>> commands.\n>>\n>> Patches 8-15 create new maintenance tasks. These are the same tasks sent in\n>> the previous RFC.\n>>\n>> Patches 16-21 create more customization through config and perform other\n>> polish items.\n>>\n>> FUTURE WORK\n>> ===========\n>>\n>>  * Add 'start', 'stop', and 'schedule' subcommands to initialize the\n>>    commands run in the background.\n>>\n>>\n>>  * Split the 'gc' builtin into smaller maintenance tasks that are enabled by\n>>    default, but might have different '--auto' conditions and more config\n>>    options.\n>>\n>>\n>>  * Replace config like 'gc.writeCommitGraph' and 'fetch.writeCommitGraph'\n>>    with use of the 'commit-graph' task.\n>>\n>>\n>>\n>> Thanks, -Stolee\n> \n> Thanks,\n> Son Luong.\n\nThank you!\n-Stolee\n\n"}]}