{"thread":{"id":"54189","subject":"[PATCH 0/7] Maintenance III: Background maintenance","startedAt":"2020-09-04T15:42:40Z","lastAt":"2021-05-10T18:42:40Z","messageCount":62,"participants":["Derrick Stolee via GitGitGadget","SZEDER Gábor","Derrick Stolee","Junio C Hamano","Đoàn Trần Công Danh","Martin Ågren","Josh Steadmon","Ævar Arnfjörð Bjarmason","Andrzej Hunt","Eric Sunshine"],"isPatch":true,"patchVersion":1,"patchTotal":7},"messages":[{"id":"405046","messageId":"pull.724.git.1599234126.gitgitgadget@gmail.com","threadId":"54189","inReplyTo":null,"subject":"[PATCH 0/7] Maintenance III: Background maintenance","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-09-04T15:41:59Z","receivedAt":"2020-09-04T15:42:40Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"This is based on ds/maintenance-part-2 and replaces the RFC from [1].\n\n[1] \nhttps://lore.kernel.org/git/pull.680.v3.git.1598629517.gitgitgadget@gmail.com/\n\nThis series introduces background maintenance to Git, through an integration\nwith cron and crontab.\n\nSome preliminary work is done to allow a new --schedule option that tells\nthe command which tasks to run based on a maintenance..schedule config\noption. The timing is not enforced by Git, but instead is expected to be\nprovided as a hint from a cron schedule. The options are \"hourly\", \"daily\",\nand \"weekly\".\n\nA new for-each-repo builtin runs Git commands on every repo in a given list.\nCurrently, the list is stored as a config setting, allowing a new \nmaintenance.repos config list to store the repositories registered for\nbackground maintenance. Others may want to add a --file= option for\ntheir own workflows, but I focused on making this as simple as possible for\nnow.\n\nThe updates to the git maintenance builtin include new register/unregister \nsubcommands and start/stop subcommands. The register subcommand initializes\nthe config while the start subcommand does everything register does plus \nupdate the cron table. The unregister and stop commands reverse this\nprocess.\n\nA troubleshooting guide is added to Documentation/git-maintenance.txt to\nadvise expert users who choose to create custom cron schedules.\n\nThe very last patch is entirely optional. It sets a recommended schedule\nbased on my own experience with very large repositories. I'm open to other\nsuggestions, but these are ones that I think work well and don't cause a\n\"rewrite the world\" scenario like running nightly 'gc' would do.\n\nI've been testing this scenario on my macOS laptop and Linux desktop. I have\nmodified my cron task to provide logging via trace2 so I can see what's\nhappening. A future direction here would be to add some maintenance logs to\nthe repository so we can track what is happening and diagnose whether the\nmaintenance strategy is working on real repos.\n\nNote: git maintenance (start|stop) only works on machines with cron by\ndesign. The proper thing to do on Windows will come later. Perhaps this\ncommand should be marked as unavailable on Windows somehow, or at least a\nbetter error than \"cron may not be available on your system\". I did find\nthat that message is helpful sometimes: macOS worker agents for CI builds\ntypically do not have cron available.\n\nDerrick Stolee (7):\n maintenance: optionally skip --auto process\n maintenance: add --schedule option and config\n for-each-repo: run subcommands on configured repos\n maintenance: add [un]register subcommands\n maintenance: add start/stop subcommands\n maintenance: recommended schedule in register/start\n maintenance: add troubleshooting guide to docs\n\n .gitignore | 1 +\n Documentation/config/maintenance.txt | 10 +\n Documentation/git-for-each-repo.txt | 59 ++++++\n Documentation/git-maintenance.txt | 88 +++++++-\n Makefile | 2 +\n builtin.h | 1 +\n builtin/for-each-repo.c | 58 ++++++\n builtin/gc.c | 292 ++++++++++++++++++++++++++-\n command-list.txt | 1 +\n git.c | 1 +\n run-command.c | 6 +\n t/helper/test-crontab.c | 35 ++++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/t0068-for-each-repo.sh | 30 +++\n t/t7900-maintenance.sh | 114 ++++++++++-\n t/test-lib.sh | 6 +\n 17 files changed, 698 insertions(+), 8 deletions(-)\n create mode 100644 Documentation/git-for-each-repo.txt\n create mode 100644 builtin/for-each-repo.c\n create mode 100644 t/helper/test-crontab.c\n create mode 100755 t/t0068-for-each-repo.sh\n\n\nbase-commit: e576ac2c7c7f6c7aa5ac08a516baeb61bf723596\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-724%2Fderrickstolee%2Fmaintenance%2Fscheduled-v1\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-724/derrickstolee/maintenance/scheduled-v1\nPull-Request: https://github.com/gitgitgadget/git/pull/724\n-- \ngitgitgadget\n"},{"id":"405047","messageId":"b7775b3aaffb39b0479f6a0aa6e475853ca4ac98.1599234126.git.gitgitgadget@gmail.com","threadId":"54189","inReplyTo":"pull.724.git.1599234126.gitgitgadget@gmail.com","subject":"[PATCH 4/7] maintenance: add [un]register subcommands","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-09-04T15:42:03Z","receivedAt":"2020-09-04T15:42:50Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIn preparation for launching background maintenance from the 'git\nmaintenance' builtin, create register/unregister subcommands. These\ncommands update the new 'maintenance.repos' config option in the global\nconfig so the background maintenance job knows which repositories to\nmaintain.\n\nThese commands allow users to add a repository to the background\nmaintenance list without disrupting the actual maintenance mechanism.\n\nFor example, a user can run 'git maintenance register' when no\nbackground maintenance is running and it will not start the background\nmaintenance. A later update to start running background maintenance will\nthen pick up this repository automatically.\n\nThe opposite example is that a user can run 'git maintenance unregister'\nto remove the current repository from background maintenance without\nhalting maintenance for other repositories.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-maintenance.txt | 14 ++++++++\n builtin/gc.c | 55 ++++++++++++++++++++++++++++++-\n t/t7900-maintenance.sh | 17 +++++++++-\n 3 files changed, 84 insertions(+), 2 deletions(-)\n\ndiff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt\nindex 3af5907b01..78d0d8df91 100644\n--- a/Documentation/git-maintenance.txt\n+++ b/Documentation/git-maintenance.txt\n@@ -29,6 +29,15 @@ Git repository.\n SUBCOMMANDS\n -----------\n \n+register::\n+\tInitialize Git config values so any scheduled maintenance will\n+\tstart running on this repository. This adds the repository to the\n+\t`maintenance.repo` config variable in the current user's global\n+\tconfig and enables some recommended configuration values for\n+\t`maintenance..schedule`. The tasks that are enabled are safe\n+\tfor running in the background without disrupting foreground\n+\tprocesses.\n+\n run::\n \tRun one or more maintenance tasks. If one or more `--task` options\n \tare specified, then those tasks are run in that order. Otherwise,\n@@ -36,6 +45,11 @@ run::\n \tconfig options are true. By default, only `maintenance.gc.enabled`\n \tis true.\n \n+unregister::\n+\tRemove the current repository from background maintenance. This\n+\tonly removes the repository from the configured list. It does not\n+\tstop the background maintenance processes from running.\n+\n TASKS\n -----\n \ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 85a3370692..ec77e8d2fa 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -1407,7 +1407,56 @@ static int maintenance_run(int argc, const char **argv, const char *prefix)\n \treturn maintenance_run_tasks(&opts);\n }\n \n-static const char builtin_maintenance_usage[] = N_(\"git maintenance run []\");\n+static int maintenance_register(void)\n+{\n+\tstruct child_process config_set = CHILD_PROCESS_INIT;\n+\tstruct child_process config_get = CHILD_PROCESS_INIT;\n+\n+\t/* There is no current repository, so skip registering it */\n+\tif (!the_repository || !the_repository->gitdir)\n+\t\treturn 0;\n+\n+\tconfig_get.git_cmd = 1;\n+\tstrvec_pushl(&config_get.args, \"config\", \"--global\", \"--get\", \"maintenance.repo\",\n+\t\t the_repository->worktree ? the_repository->worktree\n+\t\t\t\t\t : the_repository->gitdir,\n+\t\t\t NULL);\n+\tconfig_get.out = -1;\n+\n+\tif (start_command(&config_get))\n+\t\treturn error(_(\"failed to run 'git config'\"));\n+\n+\t/* We already have this value in our config! */\n+\tif (!finish_command(&config_get))\n+\t\treturn 0;\n+\n+\tconfig_set.git_cmd = 1;\n+\tstrvec_pushl(&config_set.args, \"config\", \"--add\", \"--global\", \"maintenance.repo\",\n+\t\t the_repository->worktree ? the_repository->worktree\n+\t\t\t\t\t : the_repository->gitdir,\n+\t\t NULL);\n+\n+\treturn run_command(&config_set);\n+}\n+\n+static int maintenance_unregister(void)\n+{\n+\tstruct child_process config_unset = CHILD_PROCESS_INIT;\n+\n+\tif (!the_repository || !the_repository->gitdir)\n+\t\treturn error(_(\"no current repository to unregister\"));\n+\n+\tconfig_unset.git_cmd = 1;\n+\tstrvec_pushl(&config_unset.args, \"config\", \"--global\", \"--unset\",\n+\t\t \"maintenance.repo\",\n+\t\t the_repository->worktree ? the_repository->worktree\n+\t\t\t\t\t : the_repository->gitdir,\n+\t\t NULL);\n+\n+\treturn run_command(&config_unset);\n+}\n+\n+static const char builtin_maintenance_usage[] =\tN_(\"git maintenance []\");\n \n int cmd_maintenance(int argc, const char **argv, const char *prefix)\n {\n@@ -1416,6 +1465,10 @@ int cmd_maintenance(int argc, const char **argv, const char *prefix)\n \n \tif (!strcmp(argv[1], \"run\"))\n \t\treturn maintenance_run(argc - 1, argv + 1, prefix);\n+\tif (!strcmp(argv[1], \"register\"))\n+\t\treturn maintenance_register();\n+\tif (!strcmp(argv[1], \"unregister\"))\n+\t\treturn maintenance_unregister();\n \n \tdie(_(\"invalid subcommand: %s\"), argv[1]);\n }\ndiff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\nindex 328bbaa830..272d1605d2 100755\n--- a/t/t7900-maintenance.sh\n+++ b/t/t7900-maintenance.sh\n@@ -9,7 +9,7 @@ GIT_TEST_MULTI_PACK_INDEX=0\n \n test_expect_success 'help text' '\n \ttest_expect_code 129 git maintenance -h 2>err &&\n-\ttest_i18ngrep \"usage: git maintenance run\" err &&\n+\ttest_i18ngrep \"usage: git maintenance \" err &&\n \ttest_expect_code 128 git maintenance barf 2>err &&\n \ttest_i18ngrep \"invalid subcommand: barf\" err\n '\n@@ -304,4 +304,19 @@ test_expect_success '--schedule inheritance weekly -> daily -> hourly' '\n \ttest_subcommand git multi-pack-index write --no-progress before &&\n+\tgit maintenance register &&\n+\tgit config --global --get-all maintenance.repo >actual &&\n+\tcp before after &&\n+\tpwd >>after &&\n+\ttest_cmp after actual &&\n+\tgit maintenance unregister &&\n+\tgit config --global --get-all maintenance.repo >actual &&\n+\ttest_cmp before actual\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"405048","messageId":"6082d939eb6076303e68deb1c61d9313d62788ab.1599234126.git.gitgitgadget@gmail.com","threadId":"54189","inReplyTo":"pull.724.git.1599234126.gitgitgadget@gmail.com","subject":"[PATCH 3/7] for-each-repo: run subcommands on configured repos","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-09-04T15:42:02Z","receivedAt":"2020-09-04T15:42:51Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIt can be helpful to store a list of repositories in global or system\nconfig and then iterate Git commands on that list. Create a new builtin\nthat makes this process simple for experts. We will use this builtin to\nrun scheduled maintenance on all configured repositories in a future\nchange.\n\nThe test is very simple, but does highlight that the \"--\" argument is\noptional.\n\nSigned-off-by: Derrick Stolee \n---\n .gitignore | 1 +\n Documentation/git-for-each-repo.txt | 59 +++++++++++++++++++++++++++++\n Makefile | 1 +\n builtin.h | 1 +\n builtin/for-each-repo.c | 58 ++++++++++++++++++++++++++++\n command-list.txt | 1 +\n git.c | 1 +\n t/t0068-for-each-repo.sh | 30 +++++++++++++++\n 8 files changed, 152 insertions(+)\n create mode 100644 Documentation/git-for-each-repo.txt\n create mode 100644 builtin/for-each-repo.c\n create mode 100755 t/t0068-for-each-repo.sh\n\ndiff --git a/.gitignore b/.gitignore\nindex a5808fa30d..5eb2a2be71 100644\n--- a/.gitignore\n+++ b/.gitignore\n@@ -67,6 +67,7 @@\n /git-filter-branch\n /git-fmt-merge-msg\n /git-for-each-ref\n+/git-for-each-repo\n /git-format-patch\n /git-fsck\n /git-fsck-objects\ndiff --git a/Documentation/git-for-each-repo.txt b/Documentation/git-for-each-repo.txt\nnew file mode 100644\nindex 0000000000..94bd19da26\n--- /dev/null\n+++ b/Documentation/git-for-each-repo.txt\n@@ -0,0 +1,59 @@\n+git-for-each-repo(1)\n+====================\n+\n+NAME\n+----\n+git-for-each-repo - Run a Git command on a list of repositories\n+\n+\n+SYNOPSIS\n+--------\n+[verse]\n+'git for-each-repo' --config= [--] \n+\n+\n+DESCRIPTION\n+-----------\n+Run a Git command on a list of repositories. The arguments after the\n+known options or `--` indicator are used as the arguments for the Git\n+subprocess.\n+\n+THIS COMMAND IS EXPERIMENTAL. THE BEHAVIOR MAY CHANGE.\n+\n+For example, we could run maintenance on each of a list of repositories\n+stored in a `maintenance.repo` config variable using\n+\n+-------------\n+git for-each-repo --config=maintenance.repo maintenance run\n+-------------\n+\n+This will run `git -C maintenance run` for each value ``\n+in the multi-valued config variable `maintenance.repo`.\n+\n+\n+OPTIONS\n+-------\n+--config=::\n+\tUse the given config variable as a multi-valued list storing\n+\tabsolute path names. Iterate on that list of paths to run\n+\tthe given arguments.\n++\n+These config values are loaded from system, global, and local Git config,\n+as available. If `git for-each-repo` is run in a directory that is not a\n+Git repository, then only the system and global config is used.\n+\n+\n+SUBPROCESS BEHAVIOR\n+-------------------\n+\n+If any `git -C ` subprocess returns a non-zero exit code,\n+then the `git for-each-repo` process returns that exit code without running\n+more subprocesses.\n+\n+Each `git -C ` subprocess inherits the standard file\n+descriptors `stdin`, `stdout`, and `stderr`.\n+\n+\n+GIT\n+---\n+Part of the linkgit:git[1] suite\ndiff --git a/Makefile b/Makefile\nindex 65f8cfb236..7c588ff036 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1071,6 +1071,7 @@ BUILTIN_OBJS += builtin/fetch-pack.o\n BUILTIN_OBJS += builtin/fetch.o\n BUILTIN_OBJS += builtin/fmt-merge-msg.o\n BUILTIN_OBJS += builtin/for-each-ref.o\n+BUILTIN_OBJS += builtin/for-each-repo.o\n BUILTIN_OBJS += builtin/fsck.o\n BUILTIN_OBJS += builtin/gc.o\n BUILTIN_OBJS += builtin/get-tar-commit-id.o\ndiff --git a/builtin.h b/builtin.h\nindex 17c1c0ce49..ff7c6e5aa9 100644\n--- a/builtin.h\n+++ b/builtin.h\n@@ -150,6 +150,7 @@ int cmd_fetch(int argc, const char **argv, const char *prefix);\n int cmd_fetch_pack(int argc, const char **argv, const char *prefix);\n int cmd_fmt_merge_msg(int argc, const char **argv, const char *prefix);\n int cmd_for_each_ref(int argc, const char **argv, const char *prefix);\n+int cmd_for_each_repo(int argc, const char **argv, const char *prefix);\n int cmd_format_patch(int argc, const char **argv, const char *prefix);\n int cmd_fsck(int argc, const char **argv, const char *prefix);\n int cmd_gc(int argc, const char **argv, const char *prefix);\ndiff --git a/builtin/for-each-repo.c b/builtin/for-each-repo.c\nnew file mode 100644\nindex 0000000000..5bba623ff1\n--- /dev/null\n+++ b/builtin/for-each-repo.c\n@@ -0,0 +1,58 @@\n+#include \"cache.h\"\n+#include \"config.h\"\n+#include \"builtin.h\"\n+#include \"parse-options.h\"\n+#include \"run-command.h\"\n+#include \"string-list.h\"\n+\n+static const char * const for_each_repo_usage[] = {\n+\tN_(\"git for-each-repo --config= \"),\n+\tNULL\n+};\n+\n+static int run_command_on_repo(const char *path,\n+\t\t\t void *cbdata)\n+{\n+\tint i;\n+\tstruct child_process child = CHILD_PROCESS_INIT;\n+\tstruct strvec *args = (struct strvec *)cbdata;\n+\n+\tchild.git_cmd = 1;\n+\tstrvec_pushl(&child.args, \"-C\", path, NULL);\n+\n+\tfor (i = 0; i < args->nr; i++)\n+\t\tstrvec_push(&child.args, args->v[i]);\n+\n+\treturn run_command(&child);\n+}\n+\n+int cmd_for_each_repo(int argc, const char **argv, const char *prefix)\n+{\n+\tstatic const char *config_key = NULL;\n+\tint i, result = 0;\n+\tconst struct string_list *values;\n+\tstruct strvec args = STRVEC_INIT;\n+\n+\tconst struct option options[] = {\n+\t\tOPT_STRING(0, \"config\", &config_key, N_(\"config\"),\n+\t\t\t N_(\"config key storing a list of repository paths\")),\n+\t\tOPT_END()\n+\t};\n+\n+\targc = parse_options(argc, argv, prefix, options, for_each_repo_usage,\n+\t\t\t PARSE_OPT_STOP_AT_NON_OPTION);\n+\n+\tif (!config_key)\n+\t\tdie(_(\"missing --config=\"));\n+\n+\tfor (i = 0; i < argc; i++)\n+\t\tstrvec_push(&args, argv[i]);\n+\n+\tvalues = repo_config_get_value_multi(the_repository,\n+\t\t\t\t\t config_key);\n+\n+\tfor (i = 0; !result && i < values->nr; i++)\n+\t\tresult = run_command_on_repo(values->items[i].string, &args);\n+\n+\treturn result;\n+}\ndiff --git a/command-list.txt b/command-list.txt\nindex 0e3204e7d1..581499be82 100644\n--- a/command-list.txt\n+++ b/command-list.txt\n@@ -94,6 +94,7 @@ git-fetch-pack synchingrepositories\n git-filter-branch ancillarymanipulators\n git-fmt-merge-msg purehelpers\n git-for-each-ref plumbinginterrogators\n+git-for-each-repo plumbinginterrogators\n git-format-patch mainporcelain\n git-fsck ancillaryinterrogators complete\n git-gc mainporcelain\ndiff --git a/git.c b/git.c\nindex 24f250d29a..1cab64b5d1 100644\n--- a/git.c\n+++ b/git.c\n@@ -511,6 +511,7 @@ static struct cmd_struct commands[] = {\n \t{ \"fetch-pack\", cmd_fetch_pack, RUN_SETUP | NO_PARSEOPT },\n \t{ \"fmt-merge-msg\", cmd_fmt_merge_msg, RUN_SETUP },\n \t{ \"for-each-ref\", cmd_for_each_ref, RUN_SETUP },\n+\t{ \"for-each-repo\", cmd_for_each_repo, RUN_SETUP_GENTLY },\n \t{ \"format-patch\", cmd_format_patch, RUN_SETUP },\n \t{ \"fsck\", cmd_fsck, RUN_SETUP },\n \t{ \"fsck-objects\", cmd_fsck, RUN_SETUP },\ndiff --git a/t/t0068-for-each-repo.sh b/t/t0068-for-each-repo.sh\nnew file mode 100755\nindex 0000000000..136b4ec839\n--- /dev/null\n+++ b/t/t0068-for-each-repo.sh\n@@ -0,0 +1,30 @@\n+#!/bin/sh\n+\n+test_description='git for-each-repo builtin'\n+\n+. ./test-lib.sh\n+\n+test_expect_success 'run based on configured value' '\n+\tgit init one &&\n+\tgit init two &&\n+\tgit init three &&\n+\tgit -C two commit --allow-empty -m \"DID NOT RUN\" &&\n+\tgit config run.key \"$TRASH_DIRECTORY/one\" &&\n+\tgit config --add run.key \"$TRASH_DIRECTORY/three\" &&\n+\tgit for-each-repo --config=run.key commit --allow-empty -m \"ran\" &&\n+\tgit -C one log -1 --pretty=format:%s >message &&\n+\tgrep ran message &&\n+\tgit -C two log -1 --pretty=format:%s >message &&\n+\t! grep ran message &&\n+\tgit -C three log -1 --pretty=format:%s >message &&\n+\tgrep ran message &&\n+\tgit for-each-repo --config=run.key -- commit --allow-empty -m \"ran again\" &&\n+\tgit -C one log -1 --pretty=format:%s >message &&\n+\tgrep again message &&\n+\tgit -C two log -1 --pretty=format:%s >message &&\n+\t! grep again message &&\n+\tgit -C three log -1 --pretty=format:%s >message &&\n+\tgrep again message\n+'\n+\n+test_done\n-- \ngitgitgadget\n\n"},{"id":"405049","messageId":"bd957290094351f42939e3607453ff3882b659d8.1599234126.git.gitgitgadget@gmail.com","threadId":"54189","inReplyTo":"pull.724.git.1599234126.gitgitgadget@gmail.com","subject":"[PATCH 1/7] maintenance: optionally skip --auto process","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-09-04T15:42:00Z","receivedAt":"2020-09-04T15:42:52Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nSome commands run 'git maintenance run --auto --[no-]quiet' after doing\ntheir normal work, as a way to keep repositories clean as they are used.\nCurrently, users who do not want this maintenance to occur would set the\n'gc.auto' config option to 0 to avoid the 'gc' task from running.\nHowever, this does not stop the extra process invocation. On Windows,\nthis extra process invocation can be more expensive than necessary.\n\nAllow users to drop this extra process by setting 'maintenance.auto' to\n'false'.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/config/maintenance.txt | 5 +++++\n run-command.c | 6 ++++++\n t/t7900-maintenance.sh | 13 +++++++++++++\n 3 files changed, 24 insertions(+)\n\ndiff --git a/Documentation/config/maintenance.txt b/Documentation/config/maintenance.txt\nindex a0706d8f09..06db758172 100644\n--- a/Documentation/config/maintenance.txt\n+++ b/Documentation/config/maintenance.txt\n@@ -1,3 +1,8 @@\n+maintenance.auto::\n+\tThis boolean config option controls whether some commands run\n+\t`git maintenance run --auto` after doing their normal work. Defaults\n+\tto true.\n+\n maintenance..enabled::\n \tThis boolean config option controls whether the maintenance task\n \twith name `` is run when no `--task` option is specified to\ndiff --git a/run-command.c b/run-command.c\nindex 2ee59acdc8..ea4d0fb4b1 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -7,6 +7,7 @@\n #include \"strbuf.h\"\n #include \"string-list.h\"\n #include \"quote.h\"\n+#include \"config.h\"\n \n void child_process_init(struct child_process *child)\n {\n@@ -1868,8 +1869,13 @@ int run_processes_parallel_tr2(int n, get_next_task_fn get_next_task,\n \n int run_auto_maintenance(int quiet)\n {\n+\tint enabled;\n \tstruct child_process maint = CHILD_PROCESS_INIT;\n \n+\tif (!git_config_get_bool(\"maintenance.auto\", &enabled) &&\n+\t !enabled)\n+\t\treturn 0;\n+\n \tmaint.git_cmd = 1;\n \tstrvec_pushl(&maint.args, \"maintenance\", \"run\", \"--auto\", NULL);\n \tstrvec_push(&maint.args, quiet ? \"--quiet\" : \"--no-quiet\");\ndiff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\nindex 6f878b0141..e0ba19e1ff 100755\n--- a/t/t7900-maintenance.sh\n+++ b/t/t7900-maintenance.sh\n@@ -26,6 +26,19 @@ test_expect_success 'run [--auto|--quiet]' '\n \ttest_subcommand git gc --no-quiet .enabled' '\n \tgit config maintenance.gc.enabled false &&\n \tgit config maintenance.commit-graph.enabled true &&\n-- \ngitgitgadget\n\n"},{"id":"405050","messageId":"c00de53906efd914b32995c7b5be989586faeda2.1599234127.git.gitgitgadget@gmail.com","threadId":"54189","inReplyTo":"pull.724.git.1599234126.gitgitgadget@gmail.com","subject":"[PATCH 7/7] maintenance: add troubleshooting guide to docs","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-09-04T15:42:06Z","receivedAt":"2020-09-04T15:42:54Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe 'git maintenance run' subcommand takes a lock on the object database\nto prevent concurrent processes from competing for resources. This is an\nimportant safety measure to prevent possible repository corruption and\ndata loss.\n\nThis feature can lead to confusing behavior if a user is not aware of\nit. Add a TROUBLESHOOTING section to the 'git maintenance' builtin\ndocumentation that discusses these tradeoffs. The short version of this\nsection is that Git will not corrupt your repository, but if the list of\nscheduled tasks takes longer than an hour then some scheduled tasks may\nbe dropped due to this object database collision. For example, a\nlong-running \"daily\" task at midnight might prevent an \"hourly\" task\nfrom running at 1AM.\n\nThe opposite is also possible, but less likely as long as the \"hourly\"\ntasks are much faster than the \"daily\" and \"weekly\" tasks.\n\nHelped-by: Junio C Hamano \nSigned-off-by: Derrick Stolee \n---\n Documentation/git-maintenance.txt | 44 +++++++++++++++++++++++++++++++\n 1 file changed, 44 insertions(+)\n\ndiff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt\nindex 364b3e32bf..f58dd60e40 100644\n--- a/Documentation/git-maintenance.txt\n+++ b/Documentation/git-maintenance.txt\n@@ -161,6 +161,50 @@ OPTIONS\n \t`maintenance..enabled` configured as `true` are considered.\n \tSee the 'TASKS' section for the list of accepted `` values.\n \n+\n+TROUBLESHOOTING\n+---------------\n+The `git maintenance` command is designed to simplify the repository\n+maintenance patterns while minimizing user wait time during Git commands.\n+A variety of configuration options are available to allow customizing this\n+process. The default maintenance options focus on operations that complete\n+quickly, even on large repositories.\n+\n+Users may find some cases where scheduled maintenance tasks do not run as\n+frequently as intended. Each `git maintenance run` command takes a lock on\n+the repository's object database, and this prevents other concurrent\n+`git maintenance run` commands from running on the same repository. Without\n+this safeguard, competing processes could leave the repository in an\n+unpredictable state.\n+\n+The background maintenance schedule runs `git maintenance run` processes\n+on an hourly basis. Each run executes the \"hourly\" tasks. At midnight,\n+that process also executes the \"daily\" tasks. At midnight on the first day\n+of the week, that process also executes the \"weekly\" tasks. A single\n+process iterates over each registered repository, performing the scheduled\n+tasks for that frequency. Depending on the number of registered\n+repositories and their sizes, this process may take longer than an hour.\n+In this case, multiple `git maintenance run` commands may run on the same\n+repository at the same time, colliding on the object database lock. This\n+results in one of the two tasks not running.\n+\n+If you find that some maintenance windows are taking longer than one hour\n+to complete, then consider reducing the complexity of your maintenance\n+tasks. For example, the `gc` task is much slower than the\n+`incremental-repack` task. However, this comes at a cost of a slightly\n+larger object database. Consider moving more expensive tasks to be run\n+less frequently.\n+\n+Expert users may consider scheduling their own maintenance tasks using a\n+different schedule than is available through `git maintenance start` and\n+Git configuration options. These users should be aware of the object\n+database lock and how concurrent `git maintenance run` commands behave.\n+Further, the `git gc` command should not be combined with\n+`git maintenance run` commands. `git gc` modifies the object database\n+but does not take the lock in the same way as `git maintenance run`. If\n+possible, use `git maintenance run --task=gc` instead of `git gc`.\n+\n+\n GIT\n ---\n Part of the linkgit:git[1] suite\n-- \ngitgitgadget\n"},{"id":"405051","messageId":"e02641881d998d1e6a31e941b61eb6f89d0519f7.1599234127.git.gitgitgadget@gmail.com","threadId":"54189","inReplyTo":"pull.724.git.1599234126.gitgitgadget@gmail.com","subject":"[PATCH 5/7] maintenance: add start/stop subcommands","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-09-04T15:42:04Z","receivedAt":"2020-09-04T15:42:58Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nAdd new subcommands to 'git maintenance' that start or stop background\nmaintenance using 'cron', when available. This integration is as simple\nas I could make it, barring some implementation complications.\n\nThe schedule is laid out as follows:\n\n 0 1-23 * * * $cmd maintenance run --schedule=hourly\n 0 0 * * 1-6 $cmd maintenance run --schedule=daily\n 0 0 * * 0 $cmd maintenance run --schedule=weekly\n\nwhere $cmd is a properly-qualified 'git for-each-repo' execution:\n\n$cmd=$path/git --exec-path=$path for-each-repo --config=maintenance.repo\n\nwhere $path points to the location of the Git executable running 'git\nmaintenance start'. This is critical for systems with multiple versions\nof Git. Specifically, macOS has a system version at '/usr/bin/git' while\nthe version that users can install resides at '/usr/local/bin/git'\n(symlinked to '/usr/local/libexec/git-core/git'). This will also use\nyour locally-built version if you build and run this in your development\nenvironment without installing first.\n\nThis conditional schedule avoids having cron launch multiple 'git\nfor-each-repo' commands in parallel. Such parallel commands would likely\nlead to the 'hourly' and 'daily' tasks competing over the object\ndatabase lock. This could lead to to some tasks never being run! Since\nthe --schedule= argument will run all tasks with _at least_\nthe given frequency, the daily runs will also run the hourly tasks.\nSimilarly, the weekly runs will also run the daily and hourly tasks.\n\nThe GIT_TEST_CRONTAB environment variable is not intended for users to\nedit, but instead as a way to mock the 'crontab [-l]' command. This\nvariable is set in test-lib.sh to avoid a future test from accidentally\nrunning anything with the cron integration from modifying the user's\nschedule. We use GIT_TEST_CRONTAB='test-tool crontab ' in our\ntests to check how the schedule is modified in 'git maintenance\n(start|stop)' commands.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-maintenance.txt | 11 +++\n Makefile | 1 +\n builtin/gc.c | 124 ++++++++++++++++++++++++++++++\n t/helper/test-crontab.c | 35 +++++++++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/t7900-maintenance.sh | 28 +++++++\n t/test-lib.sh | 6 ++\n 8 files changed, 207 insertions(+)\n create mode 100644 t/helper/test-crontab.c\n\ndiff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt\nindex 78d0d8df91..7f8c279fe8 100644\n--- a/Documentation/git-maintenance.txt\n+++ b/Documentation/git-maintenance.txt\n@@ -45,6 +45,17 @@ run::\n \tconfig options are true. By default, only `maintenance.gc.enabled`\n \tis true.\n \n+start::\n+\tStart running maintenance on the current repository. This performs\n+\tthe same config updates as the `register` subcommand, then updates\n+\tthe background scheduler to run `git maintenance run --scheduled`\n+\ton an hourly basis.\n+\n+stop::\n+\tHalt the background maintenance schedule. The current repository\n+\tis not removed from the list of maintained repositories, in case\n+\tthe background maintenance is restarted later.\n+\n unregister::\n \tRemove the current repository from background maintenance. This\n \tonly removes the repository from the configured list. It does not\ndiff --git a/Makefile b/Makefile\nindex 7c588ff036..c39b39bd7d 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -690,6 +690,7 @@ TEST_BUILTINS_OBJS += test-advise.o\n TEST_BUILTINS_OBJS += test-bloom.o\n TEST_BUILTINS_OBJS += test-chmtime.o\n TEST_BUILTINS_OBJS += test-config.o\n+TEST_BUILTINS_OBJS += test-crontab.o\n TEST_BUILTINS_OBJS += test-ctype.o\n TEST_BUILTINS_OBJS += test-date.o\n TEST_BUILTINS_OBJS += test-delta.o\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex ec77e8d2fa..9914417e25 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -32,6 +32,7 @@\n #include \"remote.h\"\n #include \"midx.h\"\n #include \"object-store.h\"\n+#include \"exec-cmd.h\"\n \n #define FAILED_RUN \"failed to run %s\"\n \n@@ -1456,6 +1457,125 @@ static int maintenance_unregister(void)\n \treturn run_command(&config_unset);\n }\n \n+#define BEGIN_LINE \"# BEGIN GIT MAINTENANCE SCHEDULE\"\n+#define END_LINE \"# END GIT MAINTENANCE SCHEDULE\"\n+\n+static int update_background_schedule(int run_maintenance)\n+{\n+\tint result = 0;\n+\tint in_old_region = 0;\n+\tstruct child_process crontab_list = CHILD_PROCESS_INIT;\n+\tstruct child_process crontab_edit = CHILD_PROCESS_INIT;\n+\tFILE *cron_list, *cron_in;\n+\tconst char *crontab_name;\n+\tstruct strbuf line = STRBUF_INIT;\n+\tstruct lock_file lk;\n+\tchar *lock_path = xstrfmt(\"%s/schedule\", the_repository->objects->odb->path);\n+\n+\tif (hold_lock_file_for_update(&lk, lock_path, LOCK_NO_DEREF) < 0)\n+\t\treturn error(_(\"another process is scheduling background maintenance\"));\n+\n+\tcrontab_name = getenv(\"GIT_TEST_CRONTAB\");\n+\tif (!crontab_name)\n+\t\tcrontab_name = \"crontab\";\n+\n+\tstrvec_split(&crontab_list.args, crontab_name);\n+\tstrvec_push(&crontab_list.args, \"-l\");\n+\tcrontab_list.in = -1;\n+\tcrontab_list.out = dup(lk.tempfile->fd);\n+\tcrontab_list.git_cmd = 0;\n+\n+\tif (start_command(&crontab_list)) {\n+\t\tresult = error(_(\"failed to run 'crontab -l'; your system might not support 'cron'\"));\n+\t\tgoto cleanup;\n+\t}\n+\n+\t/* Ignore exit code, as an empty crontab will return error. */\n+\tfinish_command(&crontab_list);\n+\n+\t/*\n+\t * Read from the .lock file, filtering out the old\n+\t * schedule while appending the new schedule.\n+\t */\n+\tcron_list = fdopen(lk.tempfile->fd, \"r\");\n+\trewind(cron_list);\n+\n+\tstrvec_split(&crontab_edit.args, crontab_name);\n+\tcrontab_edit.in = -1;\n+\tcrontab_edit.git_cmd = 0;\n+\n+\tif (start_command(&crontab_edit)) {\n+\t\tresult = error(_(\"failed to run 'crontab'; your system might not support 'cron'\"));\n+\t\tgoto cleanup;\n+\t}\n+\n+\tcron_in = fdopen(crontab_edit.in, \"w\");\n+\tif (!cron_in) {\n+\t\tresult = error(_(\"failed to open stdin of 'crontab'\"));\n+\t\tgoto done_editing;\n+\t}\n+\n+\twhile (!strbuf_getline_lf(&line, cron_list)) {\n+\t\tif (!in_old_region && !strcmp(line.buf, BEGIN_LINE))\n+\t\t\tin_old_region = 1;\n+\t\tif (in_old_region)\n+\t\t\tcontinue;\n+\t\tfprintf(cron_in, \"%s\\n\", line.buf);\n+\t\tif (in_old_region && !strcmp(line.buf, END_LINE))\n+\t\t\tin_old_region = 0;\n+\t}\n+\n+\tif (run_maintenance) {\n+\t\tstruct strbuf line_format = STRBUF_INIT;\n+\t\tconst char *exec_path = git_exec_path();\n+\n+\t\tfprintf(cron_in, \"%s\\n\", BEGIN_LINE);\n+\t\tfprintf(cron_in,\n+\t\t\t\"# The following schedule was created by Git\\n\");\n+\t\tfprintf(cron_in, \"# Any edits made in this region might be\\n\");\n+\t\tfprintf(cron_in,\n+\t\t\t\"# replaced in the future by a Git command.\\n\\n\");\n+\n+\t\tstrbuf_addf(&line_format,\n+\t\t\t \"%%s %%s * * %%s \\\"%s/git\\\" --exec-path=\\\"%s\\\" for-each-repo --config=maintenance.repo maintenance run --schedule=%%s\\n\",\n+\t\t\t exec_path, exec_path);\n+\t\tfprintf(cron_in, line_format.buf, \"0\", \"1-23\", \"*\", \"hourly\");\n+\t\tfprintf(cron_in, line_format.buf, \"0\", \"0\", \"1-6\", \"daily\");\n+\t\tfprintf(cron_in, line_format.buf, \"0\", \"0\", \"0\", \"weekly\");\n+\t\tstrbuf_release(&line_format);\n+\n+\t\tfprintf(cron_in, \"\\n%s\\n\", END_LINE);\n+\t}\n+\n+\tfflush(cron_in);\n+\tfclose(cron_in);\n+\tclose(crontab_edit.in);\n+\n+done_editing:\n+\tif (finish_command(&crontab_edit)) {\n+\t\tresult = error(_(\"'crontab' died\"));\n+\t\tgoto cleanup;\n+\t}\n+\tfclose(cron_list);\n+\n+cleanup:\n+\trollback_lock_file(&lk);\n+\treturn result;\n+}\n+\n+static int maintenance_start(void)\n+{\n+\tif (maintenance_register())\n+\t\twarning(_(\"failed to add repo to global config\"));\n+\n+\treturn update_background_schedule(1);\n+}\n+\n+static int maintenance_stop(void)\n+{\n+\treturn update_background_schedule(0);\n+}\n+\n static const char builtin_maintenance_usage[] =\tN_(\"git maintenance []\");\n \n int cmd_maintenance(int argc, const char **argv, const char *prefix)\n@@ -1465,6 +1585,10 @@ int cmd_maintenance(int argc, const char **argv, const char *prefix)\n \n \tif (!strcmp(argv[1], \"run\"))\n \t\treturn maintenance_run(argc - 1, argv + 1, prefix);\n+\tif (!strcmp(argv[1], \"start\"))\n+\t\treturn maintenance_start();\n+\tif (!strcmp(argv[1], \"stop\"))\n+\t\treturn maintenance_stop();\n \tif (!strcmp(argv[1], \"register\"))\n \t\treturn maintenance_register();\n \tif (!strcmp(argv[1], \"unregister\"))\ndiff --git a/t/helper/test-crontab.c b/t/helper/test-crontab.c\nnew file mode 100644\nindex 0000000000..f5db6319c6\n--- /dev/null\n+++ b/t/helper/test-crontab.c\n@@ -0,0 +1,35 @@\n+#include \"test-tool.h\"\n+#include \"cache.h\"\n+\n+/*\n+ * Usage: test-tool cron [-l]\n+ *\n+ * If -l is specified, then write the contents of to stdou.\n+ * Otherwise, write from stdin into .\n+ */\n+int cmd__crontab(int argc, const char **argv)\n+{\n+\tchar a;\n+\tFILE *from, *to;\n+\n+\tif (argc == 3 && !strcmp(argv[2], \"-l\")) {\n+\t\tfrom = fopen(argv[1], \"r\");\n+\t\tif (!from)\n+\t\t\treturn 0;\n+\t\tto = stdout;\n+\t} else if (argc == 2) {\n+\t\tfrom = stdin;\n+\t\tto = fopen(argv[1], \"w\");\n+\t} else\n+\t\treturn error(\"unknown arguments\");\n+\n+\twhile ((a = fgetc(from)) != EOF)\n+\t\tfputc(a, to);\n+\n+\tif (argc == 3)\n+\t\tfclose(from);\n+\telse\n+\t\tfclose(to);\n+\n+\treturn 0;\n+}\ndiff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\nindex 590b2efca7..432b49d948 100644\n--- a/t/helper/test-tool.c\n+++ b/t/helper/test-tool.c\n@@ -18,6 +18,7 @@ static struct test_cmd cmds[] = {\n \t{ \"bloom\", cmd__bloom },\n \t{ \"chmtime\", cmd__chmtime },\n \t{ \"config\", cmd__config },\n+\t{ \"crontab\", cmd__crontab },\n \t{ \"ctype\", cmd__ctype },\n \t{ \"date\", cmd__date },\n \t{ \"delta\", cmd__delta },\ndiff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\nindex ddc8e990e9..7c3281e071 100644\n--- a/t/helper/test-tool.h\n+++ b/t/helper/test-tool.h\n@@ -8,6 +8,7 @@ int cmd__advise_if_enabled(int argc, const char **argv);\n int cmd__bloom(int argc, const char **argv);\n int cmd__chmtime(int argc, const char **argv);\n int cmd__config(int argc, const char **argv);\n+int cmd__crontab(int argc, const char **argv);\n int cmd__ctype(int argc, const char **argv);\n int cmd__date(int argc, const char **argv);\n int cmd__delta(int argc, const char **argv);\ndiff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\nindex 272d1605d2..8803fcf621 100755\n--- a/t/t7900-maintenance.sh\n+++ b/t/t7900-maintenance.sh\n@@ -319,4 +319,32 @@ test_expect_success 'register and unregister' '\n \ttest_cmp before actual\n '\n \n+test_expect_success 'start from empty cron table' '\n+\tGIT_TEST_CRONTAB=\"test-tool crontab cron.txt\" git maintenance start &&\n+\n+\t# start registers the repo\n+\tgit config --get --global maintenance.repo \"$(pwd)\" &&\n+\n+\tgrep \"for-each-repo --config=maintenance.repo maintenance run --schedule=daily\" cron.txt &&\n+\tgrep \"for-each-repo --config=maintenance.repo maintenance run --schedule=hourly\" cron.txt &&\n+\tgrep \"for-each-repo --config=maintenance.repo maintenance run --schedule=weekly\" cron.txt\n+'\n+\n+test_expect_success 'stop from existing schedule' '\n+\tGIT_TEST_CRONTAB=\"test-tool crontab cron.txt\" git maintenance stop &&\n+\n+\t# stop does not unregister the repo\n+\tgit config --get --global maintenance.repo \"$(pwd)\" &&\n+\n+\t# Operation is idempotent\n+\tGIT_TEST_CRONTAB=\"test-tool crontab cron.txt\" git maintenance stop &&\n+\ttest_must_be_empty cron.txt\n+'\n+\n+test_expect_success 'start preserves existing schedule' '\n+\techo \"Important information!\" >cron.txt &&\n+\tGIT_TEST_CRONTAB=\"test-tool crontab cron.txt\" git maintenance start &&\n+\tgrep \"Important information!\" cron.txt\n+'\n+\n test_done\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex ef31f40037..4a60d1ed76 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -1702,3 +1702,9 @@ test_lazy_prereq SHA1 '\n test_lazy_prereq REBASE_P '\n \ttest -z \"$GIT_TEST_SKIP_REBASE_P\"\n '\n+\n+# Ensure that no test accidentally triggers a Git command\n+# that runs 'crontab', affecting a user's cron schedule.\n+# Tests that verify the cron integration must set this locally\n+# to avoid errors.\n+GIT_TEST_CRONTAB=\"exit 1\"\n-- \ngitgitgadget\n\n"},{"id":"405052","messageId":"1783e80b8d3b8361d1d62947a49ba584685dacc4.1599234126.git.gitgitgadget@gmail.com","threadId":"54189","inReplyTo":"pull.724.git.1599234126.gitgitgadget@gmail.com","subject":"[PATCH 2/7] maintenance: add --schedule option and config","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-09-04T15:42:01Z","receivedAt":"2020-09-04T15:43:04Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nA user may want to run certain maintenance tasks based on frequency, not\nconditions given in the repository. For example, the user may want to\nperform a 'prefetch' task every hour, or 'gc' task every day. To assist,\nupdate the 'git maintenance run' command to include a\n'--schedule=' option. The allowed frequencies are 'hourly',\n'daily', and 'weekly'. These values are also allowed in a new config\nvalue 'maintenance..schedule'.\n\nThe 'git maintenance run --schedule=' checks the '*.schedule'\nconfig value for each enabled task to see if the configured frequency is\nat least as frequent as the frequency from the '--schedule' argument. We\nuse the following order, for full clarity:\n\n\t'hourly' > 'daily' > 'weekly'\n\nUse new 'enum schedule_priority' to track these values numerically.\n\nThe following cron table would run the scheduled tasks with the correct\nfrequencies:\n\n 0 1-23 * * * git -C maintenance run --scheduled=hourly\n 0 0 * * 1-6 git -C maintenance run --scheduled=daily\n 0 0 * * 0 git -C maintenance run --scheduled=weekly\n\nThis cron schedule will run --scheduled=hourly every hour except at\nmidnight. This avoids a concurrent run with the --scheduled=daily that\nruns at midnight every day except the first day of the week. This avoids\na concurrent run with the --scheduled=weekly that runs at midnight on\nthe first day of the week. Since --scheduled=daily also runs the\n'hourly' tasks and --scheduled=weekly runs the 'hourly' and 'daily'\ntasks, we will still see all tasks run with the proper frequencies.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/config/maintenance.txt | 5 +++\n Documentation/git-maintenance.txt | 13 +++++-\n builtin/gc.c | 67 +++++++++++++++++++++++++---\n t/t7900-maintenance.sh | 40 +++++++++++++++++\n 4 files changed, 119 insertions(+), 6 deletions(-)\n\ndiff --git a/Documentation/config/maintenance.txt b/Documentation/config/maintenance.txt\nindex 06db758172..70585564fa 100644\n--- a/Documentation/config/maintenance.txt\n+++ b/Documentation/config/maintenance.txt\n@@ -10,6 +10,11 @@ maintenance..enabled::\n \t`--task` option exists. By default, only `maintenance.gc.enabled`\n \tis true.\n \n+maintenance..schedule::\n+\tThis config option controls whether or not the given `` runs\n+\tduring a `git maintenance run --schedule=` command. The\n+\tvalue must be one of \"hourly\", \"daily\", or \"weekly\".\n+\n maintenance.commit-graph.auto::\n \tThis integer config option controls how often the `commit-graph` task\n \tshould be run as part of `git maintenance run --auto`. If zero, then\ndiff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt\nindex b44efb05a3..3af5907b01 100644\n--- a/Documentation/git-maintenance.txt\n+++ b/Documentation/git-maintenance.txt\n@@ -107,7 +107,18 @@ OPTIONS\n \tonly if certain thresholds are met. For example, the `gc` task\n \truns when the number of loose objects exceeds the number stored\n \tin the `gc.auto` config setting, or when the number of pack-files\n-\texceeds the `gc.autoPackLimit` config setting.\n+\texceeds the `gc.autoPackLimit` config setting. Not compatible with\n+\tthe `--schedule` option.\n+\n+--schedule::\n+\tWhen combined with the `run` subcommand, run maintenance tasks\n+\tonly if certain time conditions are met, as specified by the\n+\t`maintenance..schedule` config value for each ``.\n+\tThis config value specifies a number of seconds since the last\n+\ttime that task ran, according to the `maintenance..lastRun`\n+\tconfig value. The tasks that are tested are those provided by\n+\tthe `--task=` option(s) or those with\n+\t`maintenance..enabled` set to true.\n \n --quiet::\n \tDo not report progress or other information over `stderr`.\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex f8459df04c..85a3370692 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -704,14 +704,51 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \treturn 0;\n }\n \n-static const char * const builtin_maintenance_run_usage[] = {\n-\tN_(\"git maintenance run [--auto] [--[no-]quiet] [--task=]\"),\n+static const char *const builtin_maintenance_run_usage[] = {\n+\tN_(\"git maintenance run [--auto] [--[no-]quiet] [--task=] [--schedule]\"),\n \tNULL\n };\n \n+enum schedule_priority {\n+\tSCHEDULE_NONE = 0,\n+\tSCHEDULE_WEEKLY = 1,\n+\tSCHEDULE_DAILY = 2,\n+\tSCHEDULE_HOURLY = 3,\n+};\n+\n+static enum schedule_priority parse_schedule(const char *value)\n+{\n+\tif (!value)\n+\t\treturn SCHEDULE_NONE;\n+\tif (!strcasecmp(value, \"hourly\"))\n+\t\treturn SCHEDULE_HOURLY;\n+\tif (!strcasecmp(value, \"daily\"))\n+\t\treturn SCHEDULE_DAILY;\n+\tif (!strcasecmp(value, \"weekly\"))\n+\t\treturn SCHEDULE_WEEKLY;\n+\treturn SCHEDULE_NONE;\n+}\n+\n+static int maintenance_opt_schedule(const struct option *opt, const char *arg,\n+\t\t\t\t int unset)\n+{\n+\tenum schedule_priority *priority = opt->value;\n+\n+\tif (unset)\n+\t\tdie(_(\"--no-schedule is not allowed\"));\n+\n+\t*priority = parse_schedule(arg);\n+\n+\tif (!*priority)\n+\t\tdie(_(\"unrecognized --schedule argument '%s'\"), arg);\n+\n+\treturn 0;\n+}\n+\n struct maintenance_run_opts {\n \tint auto_flag;\n \tint quiet;\n+\tenum schedule_priority schedule;\n };\n \n /* Remember to update object flag allocation in object.h */\n@@ -1159,6 +1196,8 @@ struct maintenance_task {\n \tmaintenance_auto_fn *auto_condition;\n \tunsigned enabled:1;\n \n+\tenum schedule_priority schedule;\n+\n \t/* -1 if not selected. */\n \tint selected_order;\n };\n@@ -1250,8 +1289,10 @@ static int maintenance_run_tasks(struct maintenance_run_opts *opts)\n \t\t\tcontinue;\n \n \t\tif (opts->auto_flag &&\n-\t\t (!tasks[i].auto_condition ||\n-\t\t !tasks[i].auto_condition()))\n+\t\t (!tasks[i].auto_condition || !tasks[i].auto_condition()))\n+\t\t\tcontinue;\n+\n+\t\tif (opts->schedule && tasks[i].schedule < opts->schedule)\n \t\t\tcontinue;\n \n \t\ttrace2_region_enter(\"maintenance\", tasks[i].name, r);\n@@ -1274,13 +1315,23 @@ static void initialize_task_config(void)\n \n \tfor (i = 0; i < TASK__COUNT; i++) {\n \t\tint config_value;\n+\t\tchar *config_str;\n \n-\t\tstrbuf_setlen(&config_name, 0);\n+\t\tstrbuf_reset(&config_name);\n \t\tstrbuf_addf(&config_name, \"maintenance.%s.enabled\",\n \t\t\t tasks[i].name);\n \n \t\tif (!git_config_get_bool(config_name.buf, &config_value))\n \t\t\ttasks[i].enabled = config_value;\n+\n+\t\tstrbuf_reset(&config_name);\n+\t\tstrbuf_addf(&config_name, \"maintenance.%s.schedule\",\n+\t\t\t tasks[i].name);\n+\n+\t\tif (!git_config_get_string(config_name.buf, &config_str)) {\n+\t\t\ttasks[i].schedule = parse_schedule(config_str);\n+\t\t\tfree(config_str);\n+\t\t}\n \t}\n \n \tstrbuf_release(&config_name);\n@@ -1324,6 +1375,9 @@ static int maintenance_run(int argc, const char **argv, const char *prefix)\n \tstruct option builtin_maintenance_run_options[] = {\n \t\tOPT_BOOL(0, \"auto\", &opts.auto_flag,\n \t\t\t N_(\"run tasks based on the state of the repository\")),\n+\t\tOPT_CALLBACK(0, \"schedule\", &opts.schedule, N_(\"frequency\"),\n+\t\t\t N_(\"run tasks based on frequency\"),\n+\t\t\t maintenance_opt_schedule),\n \t\tOPT_BOOL(0, \"quiet\", &opts.quiet,\n \t\t\t N_(\"do not report progress or other information over stderr\")),\n \t\tOPT_CALLBACK_F(0, \"task\", NULL, N_(\"task\"),\n@@ -1344,6 +1398,9 @@ static int maintenance_run(int argc, const char **argv, const char *prefix)\n \t\t\t builtin_maintenance_run_usage,\n \t\t\t PARSE_OPT_STOP_AT_NON_OPTION);\n \n+\tif (opts.auto_flag && opts.schedule)\n+\t\tdie(_(\"use at most one of --auto and --schedule=\"));\n+\n \tif (argc != 0)\n \t\tusage_with_options(builtin_maintenance_run_usage,\n \t\t\t\t builtin_maintenance_run_options);\ndiff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\nindex e0ba19e1ff..328bbaa830 100755\n--- a/t/t7900-maintenance.sh\n+++ b/t/t7900-maintenance.sh\n@@ -264,4 +264,44 @@ test_expect_success 'maintenance.incremental-repack.auto' '\n \tdone\n '\n \n+test_expect_success '--auto and --schedule incompatible' '\n+\ttest_must_fail git maintenance run --auto --schedule=daily 2>err &&\n+\ttest_i18ngrep \"at most one\" err\n+'\n+\n+test_expect_success 'invalid --schedule value' '\n+\ttest_must_fail git maintenance run --schedule=annually 2>err &&\n+\ttest_i18ngrep \"unrecognized --schedule\" err\n+'\n+\n+test_expect_success '--schedule inheritance weekly -> daily -> hourly' '\n+\tgit config maintenance.loose-objects.enabled true &&\n+\tgit config maintenance.loose-objects.schedule hourly &&\n+\tgit config maintenance.commit-graph.enabled true &&\n+\tgit config maintenance.commit-graph.schedule daily &&\n+\tgit config maintenance.incremental-repack.enabled true &&\n+\tgit config maintenance.incremental-repack.schedule weekly &&\n+\n+\tGIT_TRACE2_EVENT=\"$(pwd)/hourly.txt\" \\\n+\t\tgit maintenance run --schedule=hourly 2>/dev/null &&\n+\ttest_subcommand git prune-packed --quiet /dev/null &&\n+\ttest_subcommand git prune-packed --quiet /dev/null &&\n+\ttest_subcommand git prune-packed --quiet \n\nThe 'git maintenance (register|start)' subcommands add the current\nrepository to the global Git config so maintenance will operate on that\nrepository. It does not specify what maintenance should occur or how\noften.\n\nIf a user sets any 'maintenance..schedule' config value, then\nthey have chosen a specific schedule for themselves and Git should\nrespect that.\n\nHowever, in an effort to recommend a good schedule for repositories of\nall sizes, set new config values for recommended tasks that are safe to\nrun in the background while users run foreground Git commands. These\ncommands are generally everything but the 'gc' task.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-maintenance.txt | 6 ++++\n builtin/gc.c | 46 +++++++++++++++++++++++++++++++\n t/t7900-maintenance.sh | 16 +++++++++++\n 3 files changed, 68 insertions(+)\n\ndiff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt\nindex 7f8c279fe8..364b3e32bf 100644\n--- a/Documentation/git-maintenance.txt\n+++ b/Documentation/git-maintenance.txt\n@@ -37,6 +37,12 @@ register::\n \t`maintenance..schedule`. The tasks that are enabled are safe\n \tfor running in the background without disrupting foreground\n \tprocesses.\n++\n+If your repository has no 'maintenance..schedule' configuration\n+values set, then Git will set configuration values to some recommended\n+settings. These settings disable foreground maintenance while performing\n+maintenance tasks in the background that will not interrupt foreground Git\n+operations.\n \n run::\n \tRun one or more maintenance tasks. If one or more `--task` options\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 9914417e25..5f253d3458 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -1408,6 +1408,49 @@ static int maintenance_run(int argc, const char **argv, const char *prefix)\n \treturn maintenance_run_tasks(&opts);\n }\n \n+static int has_schedule_config(void)\n+{\n+\tint i, found = 0;\n+\tstruct strbuf config_name = STRBUF_INIT;\n+\tsize_t prefix;\n+\n+\tstrbuf_addstr(&config_name, \"maintenance.\");\n+\tprefix = config_name.len;\n+\n+\tfor (i = 0; !found && i < TASK__COUNT; i++) {\n+\t\tchar *value;\n+\n+\t\tstrbuf_setlen(&config_name, prefix);\n+\t\tstrbuf_addf(&config_name, \"%s.schedule\", tasks[i].name);\n+\n+\t\tif (!git_config_get_string(config_name.buf, &value)) {\n+\t\t\tfound = 1;\n+\t\t\tFREE_AND_NULL(value);\n+\t\t}\n+\t}\n+\n+\tstrbuf_release(&config_name);\n+\treturn found;\n+}\n+\n+static void set_recommended_schedule(void)\n+{\n+\tgit_config_set(\"maintenance.auto\", \"false\");\n+\tgit_config_set(\"maintenance.gc.enabled\", \"false\");\n+\n+\tgit_config_set(\"maintenance.prefetch.enabled\", \"true\");\n+\tgit_config_set(\"maintenance.prefetch.schedule\", \"hourly\");\n+\n+\tgit_config_set(\"maintenance.commit-graph.enabled\", \"true\");\n+\tgit_config_set(\"maintenance.commit-graph.schedule\", \"hourly\");\n+\n+\tgit_config_set(\"maintenance.loose-objects.enabled\", \"true\");\n+\tgit_config_set(\"maintenance.loose-objects.schedule\", \"daily\");\n+\n+\tgit_config_set(\"maintenance.incremental-repack.enabled\", \"true\");\n+\tgit_config_set(\"maintenance.incremental-repack.schedule\", \"daily\");\n+}\n+\n static int maintenance_register(void)\n {\n \tstruct child_process config_set = CHILD_PROCESS_INIT;\n@@ -1417,6 +1460,9 @@ static int maintenance_register(void)\n \tif (!the_repository || !the_repository->gitdir)\n \t\treturn 0;\n \n+\tif (!has_schedule_config())\n+\t\tset_recommended_schedule();\n+\n \tconfig_get.git_cmd = 1;\n \tstrvec_pushl(&config_get.args, \"config\", \"--global\", \"--get\", \"maintenance.repo\",\n \t\t the_repository->worktree ? the_repository->worktree\ndiff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\nindex 8803fcf621..5a31f3925b 100755\n--- a/t/t7900-maintenance.sh\n+++ b/t/t7900-maintenance.sh\n@@ -309,7 +309,23 @@ test_expect_success 'register and unregister' '\n \tgit config --global --add maintenance.repo /existing1 &&\n \tgit config --global --add maintenance.repo /existing2 &&\n \tgit config --global --get-all maintenance.repo >before &&\n+\n+\t# We still have maintenance..schedule config set,\n+\t# so this does not update the local schedule\n+\tgit maintenance register &&\n+\ttest_must_fail git config maintenance.auto &&\n+\n+\t# Clear previous maintenance..schedule values\n+\tfor task in loose-objects commit-graph incremental-repack\n+\tdo\n+\t\tgit config --unset maintenance.$task.schedule || return 1\n+\tdone &&\n \tgit maintenance register &&\n+\ttest_cmp_config false maintenance.auto &&\n+\ttest_cmp_config false maintenance.gc.enabled &&\n+\ttest_cmp_config true maintenance.prefetch.enabled &&\n+\ttest_cmp_config hourly maintenance.commit-graph.schedule &&\n+\ttest_cmp_config daily maintenance.incremental-repack.schedule &&\n \tgit config --global --get-all maintenance.repo >actual &&\n \tcp before after &&\n \tpwd >>after &&\n-- \ngitgitgadget\n\n"},{"id":"405148","messageId":"20200908062914.GC6209@szeder.dev","threadId":"54189","inReplyTo":"e02641881d998d1e6a31e941b61eb6f89d0519f7.1599234127.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 5/7] maintenance: add start/stop subcommands","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2020-09-08T06:29:14Z","receivedAt":"2020-09-08T06:29:25Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Fri, Sep 04, 2020 at 03:42:04PM +0000, Derrick Stolee via GitGitGadget wrote:\n> From: Derrick Stolee \n> \n> Add new subcommands to 'git maintenance' that start or stop background\n> maintenance using 'cron', when available. This integration is as simple\n> as I could make it, barring some implementation complications.\n> \n> The schedule is laid out as follows:\n> \n> 0 1-23 * * * $cmd maintenance run --schedule=hourly\n> 0 0 * * 1-6 $cmd maintenance run --schedule=daily\n> 0 0 * * 0 $cmd maintenance run --schedule=weekly\n> \n> where $cmd is a properly-qualified 'git for-each-repo' execution:\n> \n> $cmd=$path/git --exec-path=$path for-each-repo --config=maintenance.repo\n> \n> where $path points to the location of the Git executable running 'git\n> maintenance start'. This is critical for systems with multiple versions\n> of Git. Specifically, macOS has a system version at '/usr/bin/git' while\n> the version that users can install resides at '/usr/local/bin/git'\n> (symlinked to '/usr/local/libexec/git-core/git'). This will also use\n> your locally-built version if you build and run this in your development\n> environment without installing first.\n> \n> This conditional schedule avoids having cron launch multiple 'git\n> for-each-repo' commands in parallel. Such parallel commands would likely\n> lead to the 'hourly' and 'daily' tasks competing over the object\n> database lock. This could lead to to some tasks never being run! Since\n> the --schedule= argument will run all tasks with _at least_\n> the given frequency, the daily runs will also run the hourly tasks.\n> Similarly, the weekly runs will also run the daily and hourly tasks.\n> \n> The GIT_TEST_CRONTAB environment variable is not intended for users to\n> edit, but instead as a way to mock the 'crontab [-l]' command. This\n> variable is set in test-lib.sh to avoid a future test from accidentally\n> running anything with the cron integration from modifying the user's\n> schedule. We use GIT_TEST_CRONTAB='test-tool crontab ' in our\n> tests to check how the schedule is modified in 'git maintenance\n> (start|stop)' commands.\n> \n> Signed-off-by: Derrick Stolee \n> ---\n\n\n> diff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\n> index 272d1605d2..8803fcf621 100755\n> --- a/t/t7900-maintenance.sh\n> +++ b/t/t7900-maintenance.sh\n> @@ -319,4 +319,32 @@ test_expect_success 'register and unregister' '\n> \ttest_cmp before actual\n> '\n> \n> +test_expect_success 'start from empty cron table' '\n> +\tGIT_TEST_CRONTAB=\"test-tool crontab cron.txt\" git maintenance start &&\n\nThis command hangs when run on Travis CI's s390x arch. Now, Travis\nCI's multi-arch support is labelled as an alpha feature and isn't\nexactly bug free, so Cc-ing Adrian and Todd, who reported and tested\nbig-endian issues and fixes in the past, in the hope that they can\nconfirm.\n\n> +\n> +\t# start registers the repo\n> +\tgit config --get --global maintenance.repo \"$(pwd)\" &&\n> +\n> +\tgrep \"for-each-repo --config=maintenance.repo maintenance run --schedule=daily\" cron.txt &&\n> +\tgrep \"for-each-repo --config=maintenance.repo maintenance run --schedule=hourly\" cron.txt &&\n> +\tgrep \"for-each-repo --config=maintenance.repo maintenance run --schedule=weekly\" cron.txt\n> +'\n> +\n> +test_expect_success 'stop from existing schedule' '\n> +\tGIT_TEST_CRONTAB=\"test-tool crontab cron.txt\" git maintenance stop &&\n> +\n> +\t# stop does not unregister the repo\n> +\tgit config --get --global maintenance.repo \"$(pwd)\" &&\n> +\n> +\t# Operation is idempotent\n> +\tGIT_TEST_CRONTAB=\"test-tool crontab cron.txt\" git maintenance stop &&\n> +\ttest_must_be_empty cron.txt\n> +'\n> +\n> +test_expect_success 'start preserves existing schedule' '\n> +\techo \"Important information!\" >cron.txt &&\n> +\tGIT_TEST_CRONTAB=\"test-tool crontab cron.txt\" git maintenance start &&\n> +\tgrep \"Important information!\" cron.txt\n> +'\n> +\n> test_done\n\n\n> diff --git a/t/helper/test-crontab.c b/t/helper/test-crontab.c\n> new file mode 100644\n> index 0000000000..f5db6319c6\n> --- /dev/null\n> +++ b/t/helper/test-crontab.c\n> @@ -0,0 +1,35 @@\n> +#include \"test-tool.h\"\n> +#include \"cache.h\"\n> +\n> +/*\n> + * Usage: test-tool cron [-l]\n> + *\n> + * If -l is specified, then write the contents of to stdou.\n\ns/stdou/stdout/\n\n> + * Otherwise, write from stdin into .\n> + */\n> +int cmd__crontab(int argc, const char **argv)\n> +{\n> +\tchar a;\n\nSo 'a' is a char...\n\n> +\tFILE *from, *to;\n> +\n> +\tif (argc == 3 && !strcmp(argv[2], \"-l\")) {\n> +\t\tfrom = fopen(argv[1], \"r\");\n> +\t\tif (!from)\n> +\t\t\treturn 0;\n> +\t\tto = stdout;\n> +\t} else if (argc == 2) {\n> +\t\tfrom = stdin;\n> +\t\tto = fopen(argv[1], \"w\");\n> +\t} else\n> +\t\treturn error(\"unknown arguments\");\n> +\n> +\twhile ((a = fgetc(from)) != EOF)\n\nfgetc() returns an int, which is assigned to a char, which is then\ncompared to whatever EOF might be on the platform. Apparently this\ncasting and comparison doesn't work as expected on s390x (I haven't\neven tried to think it through...), and instead of detecting EOF and\nexiting we end up in an endless loop writing 0xff bytes to 'cron.txt',\nwhile 'git maintenance start' in vain waits for 'test-crontab' to\nexit.\n\nChanging the type of 'a' to int fixes this issue, and all these tests\npass.\n\n> +\t\tfputc(a, to);\n> +\n> +\tif (argc == 3)\n> +\t\tfclose(from);\n> +\telse\n> +\t\tfclose(to);\n> +\n> +\treturn 0;\n> +}\n"},{"id":"405159","messageId":"50ae1298-b551-8a49-6821-0659012119ee@gmail.com","threadId":"54189","inReplyTo":"20200908062914.GC6209@szeder.dev","subject":"Re: [PATCH 5/7] maintenance: add start/stop subcommands","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-09-08T12:43:30Z","receivedAt":"2020-09-08T16:41:12Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 9/8/2020 2:29 AM, SZEDER Gábor wrote:\n> On Fri, Sep 04, 2020 at 03:42:04PM +0000, Derrick Stolee via GitGitGadget wrote:\n>> +test_expect_success 'start from empty cron table' '\n>> +\tGIT_TEST_CRONTAB=\"test-tool crontab cron.txt\" git maintenance start &&\n> \n> This command hangs when run on Travis CI's s390x arch. Now, Travis\n> CI's multi-arch support is labelled as an alpha feature and isn't\n> exactly bug free, so Cc-ing Adrian and Todd, who reported and tested\n> big-endian issues and fixes in the past, in the hope that they can\n> confirm.\n\nSounds like you have found the issue below.\n\n>> diff --git a/t/helper/test-crontab.c b/t/helper/test-crontab.c\n>> new file mode 100644\n>> index 0000000000..f5db6319c6\n>> --- /dev/null\n>> +++ b/t/helper/test-crontab.c\n>> @@ -0,0 +1,35 @@\n>> +#include \"test-tool.h\"\n>> +#include \"cache.h\"\n>> +\n>> +/*\n>> + * Usage: test-tool cron [-l]\n>> + *\n>> + * If -l is specified, then write the contents of to stdou.\n> \n> s/stdou/stdout/\n\nThanks.\n\n>> + * Otherwise, write from stdin into .\n>> + */\n>> +int cmd__crontab(int argc, const char **argv)\n>> +{\n>> +\tchar a;\n> \n> So 'a' is a char...\n> \n>> +\tFILE *from, *to;\n>> +\n>> +\tif (argc == 3 && !strcmp(argv[2], \"-l\")) {\n>> +\t\tfrom = fopen(argv[1], \"r\");\n>> +\t\tif (!from)\n>> +\t\t\treturn 0;\n>> +\t\tto = stdout;\n>> +\t} else if (argc == 2) {\n>> +\t\tfrom = stdin;\n>> +\t\tto = fopen(argv[1], \"w\");\n>> +\t} else\n>> +\t\treturn error(\"unknown arguments\");\n>> +\n>> +\twhile ((a = fgetc(from)) != EOF)\n> \n> fgetc() returns an int, which is assigned to a char, which is then\n> compared to whatever EOF might be on the platform. Apparently this\n> casting and comparison doesn't work as expected on s390x (I haven't\n> even tried to think it through...), and instead of detecting EOF and\n> exiting we end up in an endless loop writing 0xff bytes to 'cron.txt',\n> while 'git maintenance start' in vain waits for 'test-crontab' to\n> exit.\n> \n> Changing the type of 'a' to int fixes this issue, and all these tests\n> pass.\n\nThanks for the help here. I'll fix this in the next version.\n\n-Stolee\n"},{"id":"405176","messageId":"xmqqv9goxdyc.fsf@gitster.c.googlers.com","threadId":"54189","inReplyTo":"20200908062914.GC6209@szeder.dev","subject":"Re: [PATCH 5/7] maintenance: add start/stop subcommands","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-09-08T19:31:39Z","receivedAt":"2020-09-08T19:32:52Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"SZEDER Gábor writes:\n\n>> +int cmd__crontab(int argc, const char **argv)\n>> +{\n>> +\tchar a;\n>\n> So 'a' is a char...\n>\n>> +\tFILE *from, *to;\n>> +\n>> +\tif (argc == 3 && !strcmp(argv[2], \"-l\")) {\n>> +\t\tfrom = fopen(argv[1], \"r\");\n>> +\t\tif (!from)\n>> +\t\t\treturn 0;\n>> +\t\tto = stdout;\n>> +\t} else if (argc == 2) {\n>> +\t\tfrom = stdin;\n>> +\t\tto = fopen(argv[1], \"w\");\n>> +\t} else\n>> +\t\treturn error(\"unknown arguments\");\n>> +\n>> +\twhile ((a = fgetc(from)) != EOF)\n>\n> fgetc() returns an int, which is assigned to a char, which is then\n> compared to whatever EOF might be on the platform. Apparently this\n> casting and comparison doesn't work as expected on s390x (I haven't\n> even tried to think it through...), and instead of detecting EOF and\n> exiting we end up in an endless loop writing 0xff bytes to 'cron.txt',\n> while 'git maintenance start' in vain waits for 'test-crontab' to\n> exit.\n\nAh, is this fun with unsigned char never comparing equal to -1?\n"},{"id":"405178","messageId":"20200908130738.GB25593@danh.dev","threadId":"54189","inReplyTo":"1783e80b8d3b8361d1d62947a49ba584685dacc4.1599234126.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 2/7] maintenance: add --schedule option and config","fromName":"Đoàn Trần Công Danh","fromEmail":"congdanhqx@gmail.com","sentAt":"2020-09-08T13:07:49Z","receivedAt":"2020-09-08T19:39:26Z","isPatch":true,"sender":{"key":"congdanhqx@gmail.com","avatar":"https://avatars.githubusercontent.com/u/42673067?v=4"},"body":"On 2020-09-04 15:42:01+0000, Derrick Stolee via GitGitGadget wrote:\n> From: Derrick Stolee \n> \n> A user may want to run certain maintenance tasks based on frequency, not\n> conditions given in the repository. For example, the user may want to\n\nHm, sorry but I couldn't decipher \"not conditions\" here. :|\n\n> perform a 'prefetch' task every hour, or 'gc' task every day. To assist,\n\nI think it's better to say: \"To assist those users\", at least it's\neasier to read for non-native English like me.\n\n> update the 'git maintenance run' command to include a\n> '--schedule=' option. The allowed frequencies are 'hourly',\n\nSo, we have \"--schedule=\" here, ...\n\n> 'daily', and 'weekly'. These values are also allowed in a new config\n> value 'maintenance..schedule'.\n> \n> The 'git maintenance run --schedule=' checks the '*.schedule'\n\nand here, ...\n\n> config value for each enabled task to see if the configured frequency is\n> at least as frequent as the frequency from the '--schedule' argument. We\n> use the following order, for full clarity:\n> \n> \t'hourly' > 'daily' > 'weekly'\n> \n> Use new 'enum schedule_priority' to track these values numerically.\n> \n> The following cron table would run the scheduled tasks with the correct\n> frequencies:\n> \n> 0 1-23 * * * git -C maintenance run --scheduled=hourly\n> 0 0 * * 1-6 git -C maintenance run --scheduled=daily\n> 0 0 * * 0 git -C maintenance run --scheduled=weekly\n\nbut it's spelt with \"--scheduled=\", here and below, mispell, I guess.\n\nReading the patch, it looks like \"--scheduled=\" is mispelt.\n\n> This cron schedule will run --scheduled=hourly every hour except at\n> midnight. This avoids a concurrent run with the --scheduled=daily that\n> runs at midnight every day except the first day of the week. This avoids\n> a concurrent run with the --scheduled=weekly that runs at midnight on\n> the first day of the week. Since --scheduled=daily also runs the\n> 'hourly' tasks and --scheduled=weekly runs the 'hourly' and 'daily'\n> tasks, we will still see all tasks run with the proper frequencies.\n> \n> Signed-off-by: Derrick Stolee \n> ---\n> Documentation/config/maintenance.txt | 5 +++\n> Documentation/git-maintenance.txt | 13 +++++-\n> builtin/gc.c | 67 +++++++++++++++++++++++++---\n> t/t7900-maintenance.sh | 40 +++++++++++++++++\n> 4 files changed, 119 insertions(+), 6 deletions(-)\n> \n> diff --git a/Documentation/config/maintenance.txt b/Documentation/config/maintenance.txt\n> index 06db758172..70585564fa 100644\n> --- a/Documentation/config/maintenance.txt\n> +++ b/Documentation/config/maintenance.txt\n> @@ -10,6 +10,11 @@ maintenance..enabled::\n> \t`--task` option exists. By default, only `maintenance.gc.enabled`\n> \tis true.\n> \n> +maintenance..schedule::\n> +\tThis config option controls whether or not the given `` runs\n> +\tduring a `git maintenance run --schedule=` command. The\n> +\tvalue must be one of \"hourly\", \"daily\", or \"weekly\".\n> +\n> maintenance.commit-graph.auto::\n> \tThis integer config option controls how often the `commit-graph` task\n> \tshould be run as part of `git maintenance run --auto`. If zero, then\n> diff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt\n> index b44efb05a3..3af5907b01 100644\n> --- a/Documentation/git-maintenance.txt\n> +++ b/Documentation/git-maintenance.txt\n> @@ -107,7 +107,18 @@ OPTIONS\n> \tonly if certain thresholds are met. For example, the `gc` task\n> \truns when the number of loose objects exceeds the number stored\n> \tin the `gc.auto` config setting, or when the number of pack-files\n> -\texceeds the `gc.autoPackLimit` config setting.\n> +\texceeds the `gc.autoPackLimit` config setting. Not compatible with\n> +\tthe `--schedule` option.\n> +\n> +--schedule::\n> +\tWhen combined with the `run` subcommand, run maintenance tasks\n> +\tonly if certain time conditions are met, as specified by the\n> +\t`maintenance..schedule` config value for each ``.\n> +\tThis config value specifies a number of seconds since the last\n> +\ttime that task ran, according to the `maintenance..lastRun`\n> +\tconfig value. The tasks that are tested are those provided by\n> +\tthe `--task=` option(s) or those with\n> +\t`maintenance..enabled` set to true.\n> \n> --quiet::\n> \tDo not report progress or other information over `stderr`.\n> diff --git a/builtin/gc.c b/builtin/gc.c\n> index f8459df04c..85a3370692 100644\n> --- a/builtin/gc.c\n> +++ b/builtin/gc.c\n> @@ -704,14 +704,51 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n> \treturn 0;\n> }\n> \n> -static const char * const builtin_maintenance_run_usage[] = {\n> -\tN_(\"git maintenance run [--auto] [--[no-]quiet] [--task=]\"),\n> +static const char *const builtin_maintenance_run_usage[] = {\n> +\tN_(\"git maintenance run [--auto] [--[no-]quiet] [--task=] [--schedule]\"),\n> \tNULL\n> };\n> \n> +enum schedule_priority {\n> +\tSCHEDULE_NONE = 0,\n> +\tSCHEDULE_WEEKLY = 1,\n> +\tSCHEDULE_DAILY = 2,\n> +\tSCHEDULE_HOURLY = 3,\n> +};\n> +\n> +static enum schedule_priority parse_schedule(const char *value)\n> +{\n> +\tif (!value)\n> +\t\treturn SCHEDULE_NONE;\n> +\tif (!strcasecmp(value, \"hourly\"))\n> +\t\treturn SCHEDULE_HOURLY;\n> +\tif (!strcasecmp(value, \"daily\"))\n> +\t\treturn SCHEDULE_DAILY;\n> +\tif (!strcasecmp(value, \"weekly\"))\n> +\t\treturn SCHEDULE_WEEKLY;\n> +\treturn SCHEDULE_NONE;\n> +}\n> +\n> +static int maintenance_opt_schedule(const struct option *opt, const char *arg,\n> +\t\t\t\t int unset)\n> +{\n> +\tenum schedule_priority *priority = opt->value;\n> +\n> +\tif (unset)\n> +\t\tdie(_(\"--no-schedule is not allowed\"));\n> +\n> +\t*priority = parse_schedule(arg);\n> +\n> +\tif (!*priority)\n> +\t\tdie(_(\"unrecognized --schedule argument '%s'\"), arg);\n> +\n> +\treturn 0;\n> +}\n> +\n> struct maintenance_run_opts {\n> \tint auto_flag;\n> \tint quiet;\n> +\tenum schedule_priority schedule;\n> };\n> \n> /* Remember to update object flag allocation in object.h */\n> @@ -1159,6 +1196,8 @@ struct maintenance_task {\n> \tmaintenance_auto_fn *auto_condition;\n> \tunsigned enabled:1;\n> \n> +\tenum schedule_priority schedule;\n> +\n> \t/* -1 if not selected. */\n> \tint selected_order;\n> };\n> @@ -1250,8 +1289,10 @@ static int maintenance_run_tasks(struct maintenance_run_opts *opts)\n> \t\t\tcontinue;\n> \n> \t\tif (opts->auto_flag &&\n> -\t\t (!tasks[i].auto_condition ||\n> -\t\t !tasks[i].auto_condition()))\n> +\t\t (!tasks[i].auto_condition || !tasks[i].auto_condition()))\n> +\t\t\tcontinue;\n\nThis line only add unnecessary noise to this patch.\n\n-- \nDanh\n> +\n> +\t\tif (opts->schedule && tasks[i].schedule < opts->schedule)\n> \t\t\tcontinue;\n> \n> \t\ttrace2_region_enter(\"maintenance\", tasks[i].name, r);\n> @@ -1274,13 +1315,23 @@ static void initialize_task_config(void)\n> \n> \tfor (i = 0; i < TASK__COUNT; i++) {\n> \t\tint config_value;\n> +\t\tchar *config_str;\n> \n> -\t\tstrbuf_setlen(&config_name, 0);\n> +\t\tstrbuf_reset(&config_name);\n> \t\tstrbuf_addf(&config_name, \"maintenance.%s.enabled\",\n> \t\t\t tasks[i].name);\n> \n> \t\tif (!git_config_get_bool(config_name.buf, &config_value))\n> \t\t\ttasks[i].enabled = config_value;\n> +\n> +\t\tstrbuf_reset(&config_name);\n> +\t\tstrbuf_addf(&config_name, \"maintenance.%s.schedule\",\n> +\t\t\t tasks[i].name);\n> +\n> +\t\tif (!git_config_get_string(config_name.buf, &config_str)) {\n> +\t\t\ttasks[i].schedule = parse_schedule(config_str);\n> +\t\t\tfree(config_str);\n> +\t\t}\n> \t}\n> \n> \tstrbuf_release(&config_name);\n> @@ -1324,6 +1375,9 @@ static int maintenance_run(int argc, const char **argv, const char *prefix)\n> \tstruct option builtin_maintenance_run_options[] = {\n> \t\tOPT_BOOL(0, \"auto\", &opts.auto_flag,\n> \t\t\t N_(\"run tasks based on the state of the repository\")),\n> +\t\tOPT_CALLBACK(0, \"schedule\", &opts.schedule, N_(\"frequency\"),\n> +\t\t\t N_(\"run tasks based on frequency\"),\n> +\t\t\t maintenance_opt_schedule),\n> \t\tOPT_BOOL(0, \"quiet\", &opts.quiet,\n> \t\t\t N_(\"do not report progress or other information over stderr\")),\n> \t\tOPT_CALLBACK_F(0, \"task\", NULL, N_(\"task\"),\n> @@ -1344,6 +1398,9 @@ static int maintenance_run(int argc, const char **argv, const char *prefix)\n> \t\t\t builtin_maintenance_run_usage,\n> \t\t\t PARSE_OPT_STOP_AT_NON_OPTION);\n> \n> +\tif (opts.auto_flag && opts.schedule)\n> +\t\tdie(_(\"use at most one of --auto and --schedule=\"));\n> +\n> \tif (argc != 0)\n> \t\tusage_with_options(builtin_maintenance_run_usage,\n> \t\t\t\t builtin_maintenance_run_options);\n> diff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\n> index e0ba19e1ff..328bbaa830 100755\n> --- a/t/t7900-maintenance.sh\n> +++ b/t/t7900-maintenance.sh\n> @@ -264,4 +264,44 @@ test_expect_success 'maintenance.incremental-repack.auto' '\n> \tdone\n> '\n> \n> +test_expect_success '--auto and --schedule incompatible' '\n> +\ttest_must_fail git maintenance run --auto --schedule=daily 2>err &&\n> +\ttest_i18ngrep \"at most one\" err\n> +'\n> +\n> +test_expect_success 'invalid --schedule value' '\n> +\ttest_must_fail git maintenance run --schedule=annually 2>err &&\n> +\ttest_i18ngrep \"unrecognized --schedule\" err\n> +'\n> +\n> +test_expect_success '--schedule inheritance weekly -> daily -> hourly' '\n> +\tgit config maintenance.loose-objects.enabled true &&\n> +\tgit config maintenance.loose-objects.schedule hourly &&\n> +\tgit config maintenance.commit-graph.enabled true &&\n> +\tgit config maintenance.commit-graph.schedule daily &&\n> +\tgit config maintenance.incremental-repack.enabled true &&\n> +\tgit config maintenance.incremental-repack.schedule weekly &&\n> +\n> +\tGIT_TRACE2_EVENT=\"$(pwd)/hourly.txt\" \\\n> +\t\tgit maintenance run --schedule=hourly 2>/dev/null &&\n> +\ttest_subcommand git prune-packed --quiet +\ttest_subcommand ! git commit-graph write --split --reachable \\\n> +\t\t--no-progress +\ttest_subcommand ! git multi-pack-index write --no-progress +\n> +\tGIT_TRACE2_EVENT=\"$(pwd)/daily.txt\" \\\n> +\t\tgit maintenance run --schedule=daily 2>/dev/null &&\n> +\ttest_subcommand git prune-packed --quiet +\ttest_subcommand git commit-graph write --split --reachable \\\n> +\t\t--no-progress +\ttest_subcommand ! git multi-pack-index write --no-progress +\n> +\tGIT_TRACE2_EVENT=\"$(pwd)/weekly.txt\" \\\n> +\t\tgit maintenance run --schedule=weekly 2>/dev/null &&\n> +\ttest_subcommand git prune-packed --quiet +\ttest_subcommand git commit-graph write --split --reachable \\\n> +\t\t--no-progress +\ttest_subcommand git multi-pack-index write --no-progress +'\n> +\n> test_done\n> -- \n> gitgitgadget\n> \n\n-- \nDanh\n"},{"id":"405247","messageId":"f1d8e155-2f9e-b0ea-6845-4c0e0d7c94d6@gmail.com","threadId":"54189","inReplyTo":"20200908130738.GB25593@danh.dev","subject":"Re: [PATCH 2/7] maintenance: add --schedule option and config","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-09-09T12:14:54Z","receivedAt":"2020-09-09T15:00:13Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 9/8/2020 9:07 AM, Đoàn Trần Công Danh wrote:\n> On 2020-09-04 15:42:01+0000, Derrick Stolee via GitGitGadget wrote:\n>> From: Derrick Stolee \n>>\n>> A user may want to run certain maintenance tasks based on frequency, not\n>> conditions given in the repository. For example, the user may want to\n> \n> Hm, sorry but I couldn't decipher \"not conditions\" here. :|\n\nAwkward, yes. I intended to contrast frequency-based maintenance with\nthreshold-based maintenance (git gc --auto).\n\n>> perform a 'prefetch' task every hour, or 'gc' task every day. To assist,\n> \n> I think it's better to say: \"To assist those users\", at least it's\n> easier to read for non-native English like me.\n\nThanks.\n\n>> update the 'git maintenance run' command to include a\n>> '--schedule=' option. The allowed frequencies are 'hourly',\n> \n> So, we have \"--schedule=\" here, ...\n> \n>> 'daily', and 'weekly'. These values are also allowed in a new config\n>> value 'maintenance..schedule'.\n>>\n>> The 'git maintenance run --schedule=' checks the '*.schedule'\n> \n> and here, ...\n> \n>> config value for each enabled task to see if the configured frequency is\n>> at least as frequent as the frequency from the '--schedule' argument. We\n>> use the following order, for full clarity:\n>>\n>> \t'hourly' > 'daily' > 'weekly'\n>>\n>> Use new 'enum schedule_priority' to track these values numerically.\n>>\n>> The following cron table would run the scheduled tasks with the correct\n>> frequencies:\n>>\n>> 0 1-23 * * * git -C maintenance run --scheduled=hourly\n>> 0 0 * * 1-6 git -C maintenance run --scheduled=daily\n>> 0 0 * * 0 git -C maintenance run --scheduled=weekly\n> \n> but it's spelt with \"--scheduled=\", here and below, mispell, I guess.> \n> Reading the patch, it looks like \"--scheduled=\" is mispelt.\n\nYes, a previous version used \"scheduled\" and I didn't fix it here.\n\n>> @@ -1250,8 +1289,10 @@ static int maintenance_run_tasks(struct maintenance_run_opts *opts)\n>> \t\t\tcontinue;\n>> \n>> \t\tif (opts->auto_flag &&\n>> -\t\t (!tasks[i].auto_condition ||\n>> -\t\t !tasks[i].auto_condition()))\n>> +\t\t (!tasks[i].auto_condition || !tasks[i].auto_condition()))\n>> +\t\t\tcontinue;\n> \n> This line only add unnecessary noise to this patch.\n\nThanks,\n-Stolee\n \n"},{"id":"405419","messageId":"pull.724.v2.git.1599846560.gitgitgadget@gmail.com","threadId":"54189","inReplyTo":"pull.724.git.1599234126.gitgitgadget@gmail.com","subject":"[PATCH v2 0/7] Maintenance III: Background maintenance","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-09-11T17:49:13Z","receivedAt":"2020-09-11T17:49:35Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"This is based on ds/maintenance-part-2 and replaces the RFC from [1].\n\n[1] \nhttps://lore.kernel.org/git/pull.680.v3.git.1598629517.gitgitgadget@gmail.com/\n\nThis series introduces background maintenance to Git, through an integration\nwith cron and crontab.\n\nSome preliminary work is done to allow a new --schedule option that tells\nthe command which tasks to run based on a maintenance..schedule config\noption. The timing is not enforced by Git, but instead is expected to be\nprovided as a hint from a cron schedule. The options are \"hourly\", \"daily\",\nand \"weekly\".\n\nA new for-each-repo builtin runs Git commands on every repo in a given list.\nCurrently, the list is stored as a config setting, allowing a new \nmaintenance.repos config list to store the repositories registered for\nbackground maintenance. Others may want to add a --file= option for\ntheir own workflows, but I focused on making this as simple as possible for\nnow.\n\nThe updates to the git maintenance builtin include new register/unregister \nsubcommands and start/stop subcommands. The register subcommand initializes\nthe config while the start subcommand does everything register does plus \nupdate the cron table. The unregister and stop commands reverse this\nprocess.\n\nA troubleshooting guide is added to Documentation/git-maintenance.txt to\nadvise expert users who choose to create custom cron schedules.\n\nThe very last patch is entirely optional. It sets a recommended schedule\nbased on my own experience with very large repositories. I'm open to other\nsuggestions, but these are ones that I think work well and don't cause a\n\"rewrite the world\" scenario like running nightly 'gc' would do.\n\nI've been testing this scenario on my macOS laptop and Linux desktop. I have\nmodified my cron task to provide logging via trace2 so I can see what's\nhappening. A future direction here would be to add some maintenance logs to\nthe repository so we can track what is happening and diagnose whether the\nmaintenance strategy is working on real repos.\n\nNote: git maintenance (start|stop) only works on machines with cron by\ndesign. The proper thing to do on Windows will come later. Perhaps this\ncommand should be marked as unavailable on Windows somehow, or at least a\nbetter error than \"cron may not be available on your system\". I did find\nthat that message is helpful sometimes: macOS worker agents for CI builds\ntypically do not have cron available.\n\nUpdates in v2:\n\n * Fixed the char/int issue in test-tool crontab, and a typo.\n * Updated commit message and patch noise in PATCH 2\n * This should fix the test failures, allowing this to be picked up in\n 'seen'.\n\nDerrick Stolee (7):\n maintenance: optionally skip --auto process\n maintenance: add --schedule option and config\n for-each-repo: run subcommands on configured repos\n maintenance: add [un]register subcommands\n maintenance: add start/stop subcommands\n maintenance: recommended schedule in register/start\n maintenance: add troubleshooting guide to docs\n\n .gitignore | 1 +\n Documentation/config/maintenance.txt | 10 +\n Documentation/git-for-each-repo.txt | 59 ++++++\n Documentation/git-maintenance.txt | 88 +++++++-\n Makefile | 2 +\n builtin.h | 1 +\n builtin/for-each-repo.c | 58 ++++++\n builtin/gc.c | 289 ++++++++++++++++++++++++++-\n command-list.txt | 1 +\n git.c | 1 +\n run-command.c | 6 +\n t/helper/test-crontab.c | 35 ++++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/t0068-for-each-repo.sh | 30 +++\n t/t7900-maintenance.sh | 114 ++++++++++-\n t/test-lib.sh | 6 +\n 17 files changed, 697 insertions(+), 6 deletions(-)\n create mode 100644 Documentation/git-for-each-repo.txt\n create mode 100644 builtin/for-each-repo.c\n create mode 100644 t/helper/test-crontab.c\n create mode 100755 t/t0068-for-each-repo.sh\n\n\nbase-commit: 6f11fba53777584b94dd9ed32976c2079d645fa2\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-724%2Fderrickstolee%2Fmaintenance%2Fscheduled-v2\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-724/derrickstolee/maintenance/scheduled-v2\nPull-Request: https://github.com/gitgitgadget/git/pull/724\n\nRange-diff vs v1:\n\n 1: bd95729009 = 1: b21cd68c90 maintenance: optionally skip --auto process\n 2: 1783e80b8d ! 2: e2d14d66d4 maintenance: add --schedule option and config\n @@ Metadata\n ## Commit message ##\n maintenance: add --schedule option and config\n \n - A user may want to run certain maintenance tasks based on frequency, not\n - conditions given in the repository. For example, the user may want to\n - perform a 'prefetch' task every hour, or 'gc' task every day. To assist,\n - update the 'git maintenance run' command to include a\n - '--schedule=' option. The allowed frequencies are 'hourly',\n - 'daily', and 'weekly'. These values are also allowed in a new config\n - value 'maintenance..schedule'.\n + Maintenance currently triggers when certain data-size thresholds are\n + met, such as number of pack-files or loose objects. Users may want to\n + run certain maintenance tasks based on frequency instead. For example,\n + a user may want to perform a 'prefetch' task every hour, or 'gc' task\n + every day. To help these users, update the 'git maintenance run' command\n + to include a '--schedule=' option. The allowed frequencies\n + are 'hourly', 'daily', and 'weekly'. These values are also allowed in a\n + new config value 'maintenance..schedule'.\n \n The 'git maintenance run --schedule=' checks the '*.schedule'\n config value for each enabled task to see if the configured frequency is\n @@ Commit message\n The following cron table would run the scheduled tasks with the correct\n frequencies:\n \n - 0 1-23 * * * git -C maintenance run --scheduled=hourly\n - 0 0 * * 1-6 git -C maintenance run --scheduled=daily\n - 0 0 * * 0 git -C maintenance run --scheduled=weekly\n + 0 1-23 * * * git -C maintenance run --schedule=hourly\n + 0 0 * * 1-6 git -C maintenance run --schedule=daily\n + 0 0 * * 0 git -C maintenance run --schedule=weekly\n \n - This cron schedule will run --scheduled=hourly every hour except at\n - midnight. This avoids a concurrent run with the --scheduled=daily that\n + This cron schedule will run --schedule=hourly every hour except at\n + midnight. This avoids a concurrent run with the --schedule=daily that\n runs at midnight every day except the first day of the week. This avoids\n - a concurrent run with the --scheduled=weekly that runs at midnight on\n - the first day of the week. Since --scheduled=daily also runs the\n - 'hourly' tasks and --scheduled=weekly runs the 'hourly' and 'daily'\n + a concurrent run with the --schedule=weekly that runs at midnight on\n + the first day of the week. Since --schedule=daily also runs the\n + 'hourly' tasks and --schedule=weekly runs the 'hourly' and 'daily'\n tasks, we will still see all tasks run with the proper frequencies.\n \n Signed-off-by: Derrick Stolee \n @@ builtin/gc.c: struct maintenance_task {\n \tint selected_order;\n };\n @@ builtin/gc.c: static int maintenance_run_tasks(struct maintenance_run_opts *opts)\n + \t\t !tasks[i].auto_condition()))\n \t\t\tcontinue;\n \n - \t\tif (opts->auto_flag &&\n --\t\t (!tasks[i].auto_condition ||\n --\t\t !tasks[i].auto_condition()))\n -+\t\t (!tasks[i].auto_condition || !tasks[i].auto_condition()))\n ++\t\tif (opts->schedule && tasks[i].schedule < opts->schedule)\n +\t\t\tcontinue;\n +\n -+\t\tif (opts->schedule && tasks[i].schedule < opts->schedule)\n - \t\t\tcontinue;\n - \n \t\ttrace2_region_enter(\"maintenance\", tasks[i].name, r);\n + \t\tif (tasks[i].fn(opts)) {\n + \t\t\terror(_(\"task '%s' failed\"), tasks[i].name);\n @@ builtin/gc.c: static void initialize_task_config(void)\n \n \tfor (i = 0; i < TASK__COUNT; i++) {\n 3: 6082d939eb = 3: 41a346dfbb for-each-repo: run subcommands on configured repos\n 4: b7775b3aaf = 4: 1f49cda18e maintenance: add [un]register subcommands\n 5: e02641881d ! 5: e9b2a39c1d maintenance: add start/stop subcommands\n @@ t/helper/test-crontab.c (new)\n +/*\n + * Usage: test-tool cron [-l]\n + *\n -+ * If -l is specified, then write the contents of to stdou.\n ++ * If -l is specified, then write the contents of to stdout.\n + * Otherwise, write from stdin into .\n + */\n +int cmd__crontab(int argc, const char **argv)\n +{\n -+\tchar a;\n ++\tint a;\n +\tFILE *from, *to;\n +\n +\tif (argc == 3 && !strcmp(argv[2], \"-l\")) {\n 6: 8a285e00e6 = 6: f609c1bde2 maintenance: recommended schedule in register/start\n 7: c00de53906 = 7: 2344eff4ba maintenance: add troubleshooting guide to docs\n\n-- \ngitgitgadget\n"},{"id":"405420","messageId":"b21cd68c905904914224960145999ab2017235b9.1599846560.git.gitgitgadget@gmail.com","threadId":"54189","inReplyTo":"pull.724.v2.git.1599846560.gitgitgadget@gmail.com","subject":"[PATCH v2 1/7] maintenance: optionally skip --auto process","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-09-11T17:49:14Z","receivedAt":"2020-09-11T17:49:40Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nSome commands run 'git maintenance run --auto --[no-]quiet' after doing\ntheir normal work, as a way to keep repositories clean as they are used.\nCurrently, users who do not want this maintenance to occur would set the\n'gc.auto' config option to 0 to avoid the 'gc' task from running.\nHowever, this does not stop the extra process invocation. On Windows,\nthis extra process invocation can be more expensive than necessary.\n\nAllow users to drop this extra process by setting 'maintenance.auto' to\n'false'.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/config/maintenance.txt | 5 +++++\n run-command.c | 6 ++++++\n t/t7900-maintenance.sh | 13 +++++++++++++\n 3 files changed, 24 insertions(+)\n\ndiff --git a/Documentation/config/maintenance.txt b/Documentation/config/maintenance.txt\nindex a0706d8f09..06db758172 100644\n--- a/Documentation/config/maintenance.txt\n+++ b/Documentation/config/maintenance.txt\n@@ -1,3 +1,8 @@\n+maintenance.auto::\n+\tThis boolean config option controls whether some commands run\n+\t`git maintenance run --auto` after doing their normal work. Defaults\n+\tto true.\n+\n maintenance..enabled::\n \tThis boolean config option controls whether the maintenance task\n \twith name `` is run when no `--task` option is specified to\ndiff --git a/run-command.c b/run-command.c\nindex 2ee59acdc8..ea4d0fb4b1 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -7,6 +7,7 @@\n #include \"strbuf.h\"\n #include \"string-list.h\"\n #include \"quote.h\"\n+#include \"config.h\"\n \n void child_process_init(struct child_process *child)\n {\n@@ -1868,8 +1869,13 @@ int run_processes_parallel_tr2(int n, get_next_task_fn get_next_task,\n \n int run_auto_maintenance(int quiet)\n {\n+\tint enabled;\n \tstruct child_process maint = CHILD_PROCESS_INIT;\n \n+\tif (!git_config_get_bool(\"maintenance.auto\", &enabled) &&\n+\t !enabled)\n+\t\treturn 0;\n+\n \tmaint.git_cmd = 1;\n \tstrvec_pushl(&maint.args, \"maintenance\", \"run\", \"--auto\", NULL);\n \tstrvec_push(&maint.args, quiet ? \"--quiet\" : \"--no-quiet\");\ndiff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\nindex 6f878b0141..e0ba19e1ff 100755\n--- a/t/t7900-maintenance.sh\n+++ b/t/t7900-maintenance.sh\n@@ -26,6 +26,19 @@ test_expect_success 'run [--auto|--quiet]' '\n \ttest_subcommand git gc --no-quiet .enabled' '\n \tgit config maintenance.gc.enabled false &&\n \tgit config maintenance.commit-graph.enabled true &&\n-- \ngitgitgadget\n\n"},{"id":"405421","messageId":"e2d14d66d46cdbb22a896e42cdacfb044449b51a.1599846561.git.gitgitgadget@gmail.com","threadId":"54189","inReplyTo":"pull.724.v2.git.1599846560.gitgitgadget@gmail.com","subject":"[PATCH v2 2/7] maintenance: add --schedule option and config","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-09-11T17:49:15Z","receivedAt":"2020-09-11T17:49:45Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nMaintenance currently triggers when certain data-size thresholds are\nmet, such as number of pack-files or loose objects. Users may want to\nrun certain maintenance tasks based on frequency instead. For example,\na user may want to perform a 'prefetch' task every hour, or 'gc' task\nevery day. To help these users, update the 'git maintenance run' command\nto include a '--schedule=' option. The allowed frequencies\nare 'hourly', 'daily', and 'weekly'. These values are also allowed in a\nnew config value 'maintenance..schedule'.\n\nThe 'git maintenance run --schedule=' checks the '*.schedule'\nconfig value for each enabled task to see if the configured frequency is\nat least as frequent as the frequency from the '--schedule' argument. We\nuse the following order, for full clarity:\n\n\t'hourly' > 'daily' > 'weekly'\n\nUse new 'enum schedule_priority' to track these values numerically.\n\nThe following cron table would run the scheduled tasks with the correct\nfrequencies:\n\n 0 1-23 * * * git -C maintenance run --schedule=hourly\n 0 0 * * 1-6 git -C maintenance run --schedule=daily\n 0 0 * * 0 git -C maintenance run --schedule=weekly\n\nThis cron schedule will run --schedule=hourly every hour except at\nmidnight. This avoids a concurrent run with the --schedule=daily that\nruns at midnight every day except the first day of the week. This avoids\na concurrent run with the --schedule=weekly that runs at midnight on\nthe first day of the week. Since --schedule=daily also runs the\n'hourly' tasks and --schedule=weekly runs the 'hourly' and 'daily'\ntasks, we will still see all tasks run with the proper frequencies.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/config/maintenance.txt | 5 +++\n Documentation/git-maintenance.txt | 13 +++++-\n builtin/gc.c | 64 ++++++++++++++++++++++++++--\n t/t7900-maintenance.sh | 40 +++++++++++++++++\n 4 files changed, 118 insertions(+), 4 deletions(-)\n\ndiff --git a/Documentation/config/maintenance.txt b/Documentation/config/maintenance.txt\nindex 06db758172..70585564fa 100644\n--- a/Documentation/config/maintenance.txt\n+++ b/Documentation/config/maintenance.txt\n@@ -10,6 +10,11 @@ maintenance..enabled::\n \t`--task` option exists. By default, only `maintenance.gc.enabled`\n \tis true.\n \n+maintenance..schedule::\n+\tThis config option controls whether or not the given `` runs\n+\tduring a `git maintenance run --schedule=` command. The\n+\tvalue must be one of \"hourly\", \"daily\", or \"weekly\".\n+\n maintenance.commit-graph.auto::\n \tThis integer config option controls how often the `commit-graph` task\n \tshould be run as part of `git maintenance run --auto`. If zero, then\ndiff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt\nindex b44efb05a3..3af5907b01 100644\n--- a/Documentation/git-maintenance.txt\n+++ b/Documentation/git-maintenance.txt\n@@ -107,7 +107,18 @@ OPTIONS\n \tonly if certain thresholds are met. For example, the `gc` task\n \truns when the number of loose objects exceeds the number stored\n \tin the `gc.auto` config setting, or when the number of pack-files\n-\texceeds the `gc.autoPackLimit` config setting.\n+\texceeds the `gc.autoPackLimit` config setting. Not compatible with\n+\tthe `--schedule` option.\n+\n+--schedule::\n+\tWhen combined with the `run` subcommand, run maintenance tasks\n+\tonly if certain time conditions are met, as specified by the\n+\t`maintenance..schedule` config value for each ``.\n+\tThis config value specifies a number of seconds since the last\n+\ttime that task ran, according to the `maintenance..lastRun`\n+\tconfig value. The tasks that are tested are those provided by\n+\tthe `--task=` option(s) or those with\n+\t`maintenance..enabled` set to true.\n \n --quiet::\n \tDo not report progress or other information over `stderr`.\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex f8459df04c..e28561b6c5 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -704,14 +704,51 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \treturn 0;\n }\n \n-static const char * const builtin_maintenance_run_usage[] = {\n-\tN_(\"git maintenance run [--auto] [--[no-]quiet] [--task=]\"),\n+static const char *const builtin_maintenance_run_usage[] = {\n+\tN_(\"git maintenance run [--auto] [--[no-]quiet] [--task=] [--schedule]\"),\n \tNULL\n };\n \n+enum schedule_priority {\n+\tSCHEDULE_NONE = 0,\n+\tSCHEDULE_WEEKLY = 1,\n+\tSCHEDULE_DAILY = 2,\n+\tSCHEDULE_HOURLY = 3,\n+};\n+\n+static enum schedule_priority parse_schedule(const char *value)\n+{\n+\tif (!value)\n+\t\treturn SCHEDULE_NONE;\n+\tif (!strcasecmp(value, \"hourly\"))\n+\t\treturn SCHEDULE_HOURLY;\n+\tif (!strcasecmp(value, \"daily\"))\n+\t\treturn SCHEDULE_DAILY;\n+\tif (!strcasecmp(value, \"weekly\"))\n+\t\treturn SCHEDULE_WEEKLY;\n+\treturn SCHEDULE_NONE;\n+}\n+\n+static int maintenance_opt_schedule(const struct option *opt, const char *arg,\n+\t\t\t\t int unset)\n+{\n+\tenum schedule_priority *priority = opt->value;\n+\n+\tif (unset)\n+\t\tdie(_(\"--no-schedule is not allowed\"));\n+\n+\t*priority = parse_schedule(arg);\n+\n+\tif (!*priority)\n+\t\tdie(_(\"unrecognized --schedule argument '%s'\"), arg);\n+\n+\treturn 0;\n+}\n+\n struct maintenance_run_opts {\n \tint auto_flag;\n \tint quiet;\n+\tenum schedule_priority schedule;\n };\n \n /* Remember to update object flag allocation in object.h */\n@@ -1159,6 +1196,8 @@ struct maintenance_task {\n \tmaintenance_auto_fn *auto_condition;\n \tunsigned enabled:1;\n \n+\tenum schedule_priority schedule;\n+\n \t/* -1 if not selected. */\n \tint selected_order;\n };\n@@ -1254,6 +1293,9 @@ static int maintenance_run_tasks(struct maintenance_run_opts *opts)\n \t\t !tasks[i].auto_condition()))\n \t\t\tcontinue;\n \n+\t\tif (opts->schedule && tasks[i].schedule < opts->schedule)\n+\t\t\tcontinue;\n+\n \t\ttrace2_region_enter(\"maintenance\", tasks[i].name, r);\n \t\tif (tasks[i].fn(opts)) {\n \t\t\terror(_(\"task '%s' failed\"), tasks[i].name);\n@@ -1274,13 +1316,23 @@ static void initialize_task_config(void)\n \n \tfor (i = 0; i < TASK__COUNT; i++) {\n \t\tint config_value;\n+\t\tchar *config_str;\n \n-\t\tstrbuf_setlen(&config_name, 0);\n+\t\tstrbuf_reset(&config_name);\n \t\tstrbuf_addf(&config_name, \"maintenance.%s.enabled\",\n \t\t\t tasks[i].name);\n \n \t\tif (!git_config_get_bool(config_name.buf, &config_value))\n \t\t\ttasks[i].enabled = config_value;\n+\n+\t\tstrbuf_reset(&config_name);\n+\t\tstrbuf_addf(&config_name, \"maintenance.%s.schedule\",\n+\t\t\t tasks[i].name);\n+\n+\t\tif (!git_config_get_string(config_name.buf, &config_str)) {\n+\t\t\ttasks[i].schedule = parse_schedule(config_str);\n+\t\t\tfree(config_str);\n+\t\t}\n \t}\n \n \tstrbuf_release(&config_name);\n@@ -1324,6 +1376,9 @@ static int maintenance_run(int argc, const char **argv, const char *prefix)\n \tstruct option builtin_maintenance_run_options[] = {\n \t\tOPT_BOOL(0, \"auto\", &opts.auto_flag,\n \t\t\t N_(\"run tasks based on the state of the repository\")),\n+\t\tOPT_CALLBACK(0, \"schedule\", &opts.schedule, N_(\"frequency\"),\n+\t\t\t N_(\"run tasks based on frequency\"),\n+\t\t\t maintenance_opt_schedule),\n \t\tOPT_BOOL(0, \"quiet\", &opts.quiet,\n \t\t\t N_(\"do not report progress or other information over stderr\")),\n \t\tOPT_CALLBACK_F(0, \"task\", NULL, N_(\"task\"),\n@@ -1344,6 +1399,9 @@ static int maintenance_run(int argc, const char **argv, const char *prefix)\n \t\t\t builtin_maintenance_run_usage,\n \t\t\t PARSE_OPT_STOP_AT_NON_OPTION);\n \n+\tif (opts.auto_flag && opts.schedule)\n+\t\tdie(_(\"use at most one of --auto and --schedule=\"));\n+\n \tif (argc != 0)\n \t\tusage_with_options(builtin_maintenance_run_usage,\n \t\t\t\t builtin_maintenance_run_options);\ndiff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\nindex e0ba19e1ff..328bbaa830 100755\n--- a/t/t7900-maintenance.sh\n+++ b/t/t7900-maintenance.sh\n@@ -264,4 +264,44 @@ test_expect_success 'maintenance.incremental-repack.auto' '\n \tdone\n '\n \n+test_expect_success '--auto and --schedule incompatible' '\n+\ttest_must_fail git maintenance run --auto --schedule=daily 2>err &&\n+\ttest_i18ngrep \"at most one\" err\n+'\n+\n+test_expect_success 'invalid --schedule value' '\n+\ttest_must_fail git maintenance run --schedule=annually 2>err &&\n+\ttest_i18ngrep \"unrecognized --schedule\" err\n+'\n+\n+test_expect_success '--schedule inheritance weekly -> daily -> hourly' '\n+\tgit config maintenance.loose-objects.enabled true &&\n+\tgit config maintenance.loose-objects.schedule hourly &&\n+\tgit config maintenance.commit-graph.enabled true &&\n+\tgit config maintenance.commit-graph.schedule daily &&\n+\tgit config maintenance.incremental-repack.enabled true &&\n+\tgit config maintenance.incremental-repack.schedule weekly &&\n+\n+\tGIT_TRACE2_EVENT=\"$(pwd)/hourly.txt\" \\\n+\t\tgit maintenance run --schedule=hourly 2>/dev/null &&\n+\ttest_subcommand git prune-packed --quiet /dev/null &&\n+\ttest_subcommand git prune-packed --quiet /dev/null &&\n+\ttest_subcommand git prune-packed --quiet \n\nIt can be helpful to store a list of repositories in global or system\nconfig and then iterate Git commands on that list. Create a new builtin\nthat makes this process simple for experts. We will use this builtin to\nrun scheduled maintenance on all configured repositories in a future\nchange.\n\nThe test is very simple, but does highlight that the \"--\" argument is\noptional.\n\nSigned-off-by: Derrick Stolee \n---\n .gitignore | 1 +\n Documentation/git-for-each-repo.txt | 59 +++++++++++++++++++++++++++++\n Makefile | 1 +\n builtin.h | 1 +\n builtin/for-each-repo.c | 58 ++++++++++++++++++++++++++++\n command-list.txt | 1 +\n git.c | 1 +\n t/t0068-for-each-repo.sh | 30 +++++++++++++++\n 8 files changed, 152 insertions(+)\n create mode 100644 Documentation/git-for-each-repo.txt\n create mode 100644 builtin/for-each-repo.c\n create mode 100755 t/t0068-for-each-repo.sh\n\ndiff --git a/.gitignore b/.gitignore\nindex a5808fa30d..5eb2a2be71 100644\n--- a/.gitignore\n+++ b/.gitignore\n@@ -67,6 +67,7 @@\n /git-filter-branch\n /git-fmt-merge-msg\n /git-for-each-ref\n+/git-for-each-repo\n /git-format-patch\n /git-fsck\n /git-fsck-objects\ndiff --git a/Documentation/git-for-each-repo.txt b/Documentation/git-for-each-repo.txt\nnew file mode 100644\nindex 0000000000..94bd19da26\n--- /dev/null\n+++ b/Documentation/git-for-each-repo.txt\n@@ -0,0 +1,59 @@\n+git-for-each-repo(1)\n+====================\n+\n+NAME\n+----\n+git-for-each-repo - Run a Git command on a list of repositories\n+\n+\n+SYNOPSIS\n+--------\n+[verse]\n+'git for-each-repo' --config= [--] \n+\n+\n+DESCRIPTION\n+-----------\n+Run a Git command on a list of repositories. The arguments after the\n+known options or `--` indicator are used as the arguments for the Git\n+subprocess.\n+\n+THIS COMMAND IS EXPERIMENTAL. THE BEHAVIOR MAY CHANGE.\n+\n+For example, we could run maintenance on each of a list of repositories\n+stored in a `maintenance.repo` config variable using\n+\n+-------------\n+git for-each-repo --config=maintenance.repo maintenance run\n+-------------\n+\n+This will run `git -C maintenance run` for each value ``\n+in the multi-valued config variable `maintenance.repo`.\n+\n+\n+OPTIONS\n+-------\n+--config=::\n+\tUse the given config variable as a multi-valued list storing\n+\tabsolute path names. Iterate on that list of paths to run\n+\tthe given arguments.\n++\n+These config values are loaded from system, global, and local Git config,\n+as available. If `git for-each-repo` is run in a directory that is not a\n+Git repository, then only the system and global config is used.\n+\n+\n+SUBPROCESS BEHAVIOR\n+-------------------\n+\n+If any `git -C ` subprocess returns a non-zero exit code,\n+then the `git for-each-repo` process returns that exit code without running\n+more subprocesses.\n+\n+Each `git -C ` subprocess inherits the standard file\n+descriptors `stdin`, `stdout`, and `stderr`.\n+\n+\n+GIT\n+---\n+Part of the linkgit:git[1] suite\ndiff --git a/Makefile b/Makefile\nindex 65f8cfb236..7c588ff036 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1071,6 +1071,7 @@ BUILTIN_OBJS += builtin/fetch-pack.o\n BUILTIN_OBJS += builtin/fetch.o\n BUILTIN_OBJS += builtin/fmt-merge-msg.o\n BUILTIN_OBJS += builtin/for-each-ref.o\n+BUILTIN_OBJS += builtin/for-each-repo.o\n BUILTIN_OBJS += builtin/fsck.o\n BUILTIN_OBJS += builtin/gc.o\n BUILTIN_OBJS += builtin/get-tar-commit-id.o\ndiff --git a/builtin.h b/builtin.h\nindex 17c1c0ce49..ff7c6e5aa9 100644\n--- a/builtin.h\n+++ b/builtin.h\n@@ -150,6 +150,7 @@ int cmd_fetch(int argc, const char **argv, const char *prefix);\n int cmd_fetch_pack(int argc, const char **argv, const char *prefix);\n int cmd_fmt_merge_msg(int argc, const char **argv, const char *prefix);\n int cmd_for_each_ref(int argc, const char **argv, const char *prefix);\n+int cmd_for_each_repo(int argc, const char **argv, const char *prefix);\n int cmd_format_patch(int argc, const char **argv, const char *prefix);\n int cmd_fsck(int argc, const char **argv, const char *prefix);\n int cmd_gc(int argc, const char **argv, const char *prefix);\ndiff --git a/builtin/for-each-repo.c b/builtin/for-each-repo.c\nnew file mode 100644\nindex 0000000000..5bba623ff1\n--- /dev/null\n+++ b/builtin/for-each-repo.c\n@@ -0,0 +1,58 @@\n+#include \"cache.h\"\n+#include \"config.h\"\n+#include \"builtin.h\"\n+#include \"parse-options.h\"\n+#include \"run-command.h\"\n+#include \"string-list.h\"\n+\n+static const char * const for_each_repo_usage[] = {\n+\tN_(\"git for-each-repo --config= \"),\n+\tNULL\n+};\n+\n+static int run_command_on_repo(const char *path,\n+\t\t\t void *cbdata)\n+{\n+\tint i;\n+\tstruct child_process child = CHILD_PROCESS_INIT;\n+\tstruct strvec *args = (struct strvec *)cbdata;\n+\n+\tchild.git_cmd = 1;\n+\tstrvec_pushl(&child.args, \"-C\", path, NULL);\n+\n+\tfor (i = 0; i < args->nr; i++)\n+\t\tstrvec_push(&child.args, args->v[i]);\n+\n+\treturn run_command(&child);\n+}\n+\n+int cmd_for_each_repo(int argc, const char **argv, const char *prefix)\n+{\n+\tstatic const char *config_key = NULL;\n+\tint i, result = 0;\n+\tconst struct string_list *values;\n+\tstruct strvec args = STRVEC_INIT;\n+\n+\tconst struct option options[] = {\n+\t\tOPT_STRING(0, \"config\", &config_key, N_(\"config\"),\n+\t\t\t N_(\"config key storing a list of repository paths\")),\n+\t\tOPT_END()\n+\t};\n+\n+\targc = parse_options(argc, argv, prefix, options, for_each_repo_usage,\n+\t\t\t PARSE_OPT_STOP_AT_NON_OPTION);\n+\n+\tif (!config_key)\n+\t\tdie(_(\"missing --config=\"));\n+\n+\tfor (i = 0; i < argc; i++)\n+\t\tstrvec_push(&args, argv[i]);\n+\n+\tvalues = repo_config_get_value_multi(the_repository,\n+\t\t\t\t\t config_key);\n+\n+\tfor (i = 0; !result && i < values->nr; i++)\n+\t\tresult = run_command_on_repo(values->items[i].string, &args);\n+\n+\treturn result;\n+}\ndiff --git a/command-list.txt b/command-list.txt\nindex 0e3204e7d1..581499be82 100644\n--- a/command-list.txt\n+++ b/command-list.txt\n@@ -94,6 +94,7 @@ git-fetch-pack synchingrepositories\n git-filter-branch ancillarymanipulators\n git-fmt-merge-msg purehelpers\n git-for-each-ref plumbinginterrogators\n+git-for-each-repo plumbinginterrogators\n git-format-patch mainporcelain\n git-fsck ancillaryinterrogators complete\n git-gc mainporcelain\ndiff --git a/git.c b/git.c\nindex 24f250d29a..1cab64b5d1 100644\n--- a/git.c\n+++ b/git.c\n@@ -511,6 +511,7 @@ static struct cmd_struct commands[] = {\n \t{ \"fetch-pack\", cmd_fetch_pack, RUN_SETUP | NO_PARSEOPT },\n \t{ \"fmt-merge-msg\", cmd_fmt_merge_msg, RUN_SETUP },\n \t{ \"for-each-ref\", cmd_for_each_ref, RUN_SETUP },\n+\t{ \"for-each-repo\", cmd_for_each_repo, RUN_SETUP_GENTLY },\n \t{ \"format-patch\", cmd_format_patch, RUN_SETUP },\n \t{ \"fsck\", cmd_fsck, RUN_SETUP },\n \t{ \"fsck-objects\", cmd_fsck, RUN_SETUP },\ndiff --git a/t/t0068-for-each-repo.sh b/t/t0068-for-each-repo.sh\nnew file mode 100755\nindex 0000000000..136b4ec839\n--- /dev/null\n+++ b/t/t0068-for-each-repo.sh\n@@ -0,0 +1,30 @@\n+#!/bin/sh\n+\n+test_description='git for-each-repo builtin'\n+\n+. ./test-lib.sh\n+\n+test_expect_success 'run based on configured value' '\n+\tgit init one &&\n+\tgit init two &&\n+\tgit init three &&\n+\tgit -C two commit --allow-empty -m \"DID NOT RUN\" &&\n+\tgit config run.key \"$TRASH_DIRECTORY/one\" &&\n+\tgit config --add run.key \"$TRASH_DIRECTORY/three\" &&\n+\tgit for-each-repo --config=run.key commit --allow-empty -m \"ran\" &&\n+\tgit -C one log -1 --pretty=format:%s >message &&\n+\tgrep ran message &&\n+\tgit -C two log -1 --pretty=format:%s >message &&\n+\t! grep ran message &&\n+\tgit -C three log -1 --pretty=format:%s >message &&\n+\tgrep ran message &&\n+\tgit for-each-repo --config=run.key -- commit --allow-empty -m \"ran again\" &&\n+\tgit -C one log -1 --pretty=format:%s >message &&\n+\tgrep again message &&\n+\tgit -C two log -1 --pretty=format:%s >message &&\n+\t! grep again message &&\n+\tgit -C three log -1 --pretty=format:%s >message &&\n+\tgrep again message\n+'\n+\n+test_done\n-- \ngitgitgadget\n\n"},{"id":"405423","messageId":"1f49cda18ee8d6c585807506cf89c365de642ea7.1599846561.git.gitgitgadget@gmail.com","threadId":"54189","inReplyTo":"pull.724.v2.git.1599846560.gitgitgadget@gmail.com","subject":"[PATCH v2 4/7] maintenance: add [un]register subcommands","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-09-11T17:49:17Z","receivedAt":"2020-09-11T17:49:50Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIn preparation for launching background maintenance from the 'git\nmaintenance' builtin, create register/unregister subcommands. These\ncommands update the new 'maintenance.repos' config option in the global\nconfig so the background maintenance job knows which repositories to\nmaintain.\n\nThese commands allow users to add a repository to the background\nmaintenance list without disrupting the actual maintenance mechanism.\n\nFor example, a user can run 'git maintenance register' when no\nbackground maintenance is running and it will not start the background\nmaintenance. A later update to start running background maintenance will\nthen pick up this repository automatically.\n\nThe opposite example is that a user can run 'git maintenance unregister'\nto remove the current repository from background maintenance without\nhalting maintenance for other repositories.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-maintenance.txt | 14 ++++++++\n builtin/gc.c | 55 ++++++++++++++++++++++++++++++-\n t/t7900-maintenance.sh | 17 +++++++++-\n 3 files changed, 84 insertions(+), 2 deletions(-)\n\ndiff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt\nindex 3af5907b01..78d0d8df91 100644\n--- a/Documentation/git-maintenance.txt\n+++ b/Documentation/git-maintenance.txt\n@@ -29,6 +29,15 @@ Git repository.\n SUBCOMMANDS\n -----------\n \n+register::\n+\tInitialize Git config values so any scheduled maintenance will\n+\tstart running on this repository. This adds the repository to the\n+\t`maintenance.repo` config variable in the current user's global\n+\tconfig and enables some recommended configuration values for\n+\t`maintenance..schedule`. The tasks that are enabled are safe\n+\tfor running in the background without disrupting foreground\n+\tprocesses.\n+\n run::\n \tRun one or more maintenance tasks. If one or more `--task` options\n \tare specified, then those tasks are run in that order. Otherwise,\n@@ -36,6 +45,11 @@ run::\n \tconfig options are true. By default, only `maintenance.gc.enabled`\n \tis true.\n \n+unregister::\n+\tRemove the current repository from background maintenance. This\n+\tonly removes the repository from the configured list. It does not\n+\tstop the background maintenance processes from running.\n+\n TASKS\n -----\n \ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex e28561b6c5..0290b249c9 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -1408,7 +1408,56 @@ static int maintenance_run(int argc, const char **argv, const char *prefix)\n \treturn maintenance_run_tasks(&opts);\n }\n \n-static const char builtin_maintenance_usage[] = N_(\"git maintenance run []\");\n+static int maintenance_register(void)\n+{\n+\tstruct child_process config_set = CHILD_PROCESS_INIT;\n+\tstruct child_process config_get = CHILD_PROCESS_INIT;\n+\n+\t/* There is no current repository, so skip registering it */\n+\tif (!the_repository || !the_repository->gitdir)\n+\t\treturn 0;\n+\n+\tconfig_get.git_cmd = 1;\n+\tstrvec_pushl(&config_get.args, \"config\", \"--global\", \"--get\", \"maintenance.repo\",\n+\t\t the_repository->worktree ? the_repository->worktree\n+\t\t\t\t\t : the_repository->gitdir,\n+\t\t\t NULL);\n+\tconfig_get.out = -1;\n+\n+\tif (start_command(&config_get))\n+\t\treturn error(_(\"failed to run 'git config'\"));\n+\n+\t/* We already have this value in our config! */\n+\tif (!finish_command(&config_get))\n+\t\treturn 0;\n+\n+\tconfig_set.git_cmd = 1;\n+\tstrvec_pushl(&config_set.args, \"config\", \"--add\", \"--global\", \"maintenance.repo\",\n+\t\t the_repository->worktree ? the_repository->worktree\n+\t\t\t\t\t : the_repository->gitdir,\n+\t\t NULL);\n+\n+\treturn run_command(&config_set);\n+}\n+\n+static int maintenance_unregister(void)\n+{\n+\tstruct child_process config_unset = CHILD_PROCESS_INIT;\n+\n+\tif (!the_repository || !the_repository->gitdir)\n+\t\treturn error(_(\"no current repository to unregister\"));\n+\n+\tconfig_unset.git_cmd = 1;\n+\tstrvec_pushl(&config_unset.args, \"config\", \"--global\", \"--unset\",\n+\t\t \"maintenance.repo\",\n+\t\t the_repository->worktree ? the_repository->worktree\n+\t\t\t\t\t : the_repository->gitdir,\n+\t\t NULL);\n+\n+\treturn run_command(&config_unset);\n+}\n+\n+static const char builtin_maintenance_usage[] =\tN_(\"git maintenance []\");\n \n int cmd_maintenance(int argc, const char **argv, const char *prefix)\n {\n@@ -1417,6 +1466,10 @@ int cmd_maintenance(int argc, const char **argv, const char *prefix)\n \n \tif (!strcmp(argv[1], \"run\"))\n \t\treturn maintenance_run(argc - 1, argv + 1, prefix);\n+\tif (!strcmp(argv[1], \"register\"))\n+\t\treturn maintenance_register();\n+\tif (!strcmp(argv[1], \"unregister\"))\n+\t\treturn maintenance_unregister();\n \n \tdie(_(\"invalid subcommand: %s\"), argv[1]);\n }\ndiff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\nindex 328bbaa830..272d1605d2 100755\n--- a/t/t7900-maintenance.sh\n+++ b/t/t7900-maintenance.sh\n@@ -9,7 +9,7 @@ GIT_TEST_MULTI_PACK_INDEX=0\n \n test_expect_success 'help text' '\n \ttest_expect_code 129 git maintenance -h 2>err &&\n-\ttest_i18ngrep \"usage: git maintenance run\" err &&\n+\ttest_i18ngrep \"usage: git maintenance \" err &&\n \ttest_expect_code 128 git maintenance barf 2>err &&\n \ttest_i18ngrep \"invalid subcommand: barf\" err\n '\n@@ -304,4 +304,19 @@ test_expect_success '--schedule inheritance weekly -> daily -> hourly' '\n \ttest_subcommand git multi-pack-index write --no-progress before &&\n+\tgit maintenance register &&\n+\tgit config --global --get-all maintenance.repo >actual &&\n+\tcp before after &&\n+\tpwd >>after &&\n+\ttest_cmp after actual &&\n+\tgit maintenance unregister &&\n+\tgit config --global --get-all maintenance.repo >actual &&\n+\ttest_cmp before actual\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"405424","messageId":"e9b2a39c1da141a8a124790b9a59c7abf891d5db.1599846561.git.gitgitgadget@gmail.com","threadId":"54189","inReplyTo":"pull.724.v2.git.1599846560.gitgitgadget@gmail.com","subject":"[PATCH v2 5/7] maintenance: add start/stop subcommands","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-09-11T17:49:18Z","receivedAt":"2020-09-11T17:49:52Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nAdd new subcommands to 'git maintenance' that start or stop background\nmaintenance using 'cron', when available. This integration is as simple\nas I could make it, barring some implementation complications.\n\nThe schedule is laid out as follows:\n\n 0 1-23 * * * $cmd maintenance run --schedule=hourly\n 0 0 * * 1-6 $cmd maintenance run --schedule=daily\n 0 0 * * 0 $cmd maintenance run --schedule=weekly\n\nwhere $cmd is a properly-qualified 'git for-each-repo' execution:\n\n$cmd=$path/git --exec-path=$path for-each-repo --config=maintenance.repo\n\nwhere $path points to the location of the Git executable running 'git\nmaintenance start'. This is critical for systems with multiple versions\nof Git. Specifically, macOS has a system version at '/usr/bin/git' while\nthe version that users can install resides at '/usr/local/bin/git'\n(symlinked to '/usr/local/libexec/git-core/git'). This will also use\nyour locally-built version if you build and run this in your development\nenvironment without installing first.\n\nThis conditional schedule avoids having cron launch multiple 'git\nfor-each-repo' commands in parallel. Such parallel commands would likely\nlead to the 'hourly' and 'daily' tasks competing over the object\ndatabase lock. This could lead to to some tasks never being run! Since\nthe --schedule= argument will run all tasks with _at least_\nthe given frequency, the daily runs will also run the hourly tasks.\nSimilarly, the weekly runs will also run the daily and hourly tasks.\n\nThe GIT_TEST_CRONTAB environment variable is not intended for users to\nedit, but instead as a way to mock the 'crontab [-l]' command. This\nvariable is set in test-lib.sh to avoid a future test from accidentally\nrunning anything with the cron integration from modifying the user's\nschedule. We use GIT_TEST_CRONTAB='test-tool crontab ' in our\ntests to check how the schedule is modified in 'git maintenance\n(start|stop)' commands.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-maintenance.txt | 11 +++\n Makefile | 1 +\n builtin/gc.c | 124 ++++++++++++++++++++++++++++++\n t/helper/test-crontab.c | 35 +++++++++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/t7900-maintenance.sh | 28 +++++++\n t/test-lib.sh | 6 ++\n 8 files changed, 207 insertions(+)\n create mode 100644 t/helper/test-crontab.c\n\ndiff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt\nindex 78d0d8df91..7f8c279fe8 100644\n--- a/Documentation/git-maintenance.txt\n+++ b/Documentation/git-maintenance.txt\n@@ -45,6 +45,17 @@ run::\n \tconfig options are true. By default, only `maintenance.gc.enabled`\n \tis true.\n \n+start::\n+\tStart running maintenance on the current repository. This performs\n+\tthe same config updates as the `register` subcommand, then updates\n+\tthe background scheduler to run `git maintenance run --scheduled`\n+\ton an hourly basis.\n+\n+stop::\n+\tHalt the background maintenance schedule. The current repository\n+\tis not removed from the list of maintained repositories, in case\n+\tthe background maintenance is restarted later.\n+\n unregister::\n \tRemove the current repository from background maintenance. This\n \tonly removes the repository from the configured list. It does not\ndiff --git a/Makefile b/Makefile\nindex 7c588ff036..c39b39bd7d 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -690,6 +690,7 @@ TEST_BUILTINS_OBJS += test-advise.o\n TEST_BUILTINS_OBJS += test-bloom.o\n TEST_BUILTINS_OBJS += test-chmtime.o\n TEST_BUILTINS_OBJS += test-config.o\n+TEST_BUILTINS_OBJS += test-crontab.o\n TEST_BUILTINS_OBJS += test-ctype.o\n TEST_BUILTINS_OBJS += test-date.o\n TEST_BUILTINS_OBJS += test-delta.o\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 0290b249c9..4f2d17c0c0 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -32,6 +32,7 @@\n #include \"remote.h\"\n #include \"midx.h\"\n #include \"object-store.h\"\n+#include \"exec-cmd.h\"\n \n #define FAILED_RUN \"failed to run %s\"\n \n@@ -1457,6 +1458,125 @@ static int maintenance_unregister(void)\n \treturn run_command(&config_unset);\n }\n \n+#define BEGIN_LINE \"# BEGIN GIT MAINTENANCE SCHEDULE\"\n+#define END_LINE \"# END GIT MAINTENANCE SCHEDULE\"\n+\n+static int update_background_schedule(int run_maintenance)\n+{\n+\tint result = 0;\n+\tint in_old_region = 0;\n+\tstruct child_process crontab_list = CHILD_PROCESS_INIT;\n+\tstruct child_process crontab_edit = CHILD_PROCESS_INIT;\n+\tFILE *cron_list, *cron_in;\n+\tconst char *crontab_name;\n+\tstruct strbuf line = STRBUF_INIT;\n+\tstruct lock_file lk;\n+\tchar *lock_path = xstrfmt(\"%s/schedule\", the_repository->objects->odb->path);\n+\n+\tif (hold_lock_file_for_update(&lk, lock_path, LOCK_NO_DEREF) < 0)\n+\t\treturn error(_(\"another process is scheduling background maintenance\"));\n+\n+\tcrontab_name = getenv(\"GIT_TEST_CRONTAB\");\n+\tif (!crontab_name)\n+\t\tcrontab_name = \"crontab\";\n+\n+\tstrvec_split(&crontab_list.args, crontab_name);\n+\tstrvec_push(&crontab_list.args, \"-l\");\n+\tcrontab_list.in = -1;\n+\tcrontab_list.out = dup(lk.tempfile->fd);\n+\tcrontab_list.git_cmd = 0;\n+\n+\tif (start_command(&crontab_list)) {\n+\t\tresult = error(_(\"failed to run 'crontab -l'; your system might not support 'cron'\"));\n+\t\tgoto cleanup;\n+\t}\n+\n+\t/* Ignore exit code, as an empty crontab will return error. */\n+\tfinish_command(&crontab_list);\n+\n+\t/*\n+\t * Read from the .lock file, filtering out the old\n+\t * schedule while appending the new schedule.\n+\t */\n+\tcron_list = fdopen(lk.tempfile->fd, \"r\");\n+\trewind(cron_list);\n+\n+\tstrvec_split(&crontab_edit.args, crontab_name);\n+\tcrontab_edit.in = -1;\n+\tcrontab_edit.git_cmd = 0;\n+\n+\tif (start_command(&crontab_edit)) {\n+\t\tresult = error(_(\"failed to run 'crontab'; your system might not support 'cron'\"));\n+\t\tgoto cleanup;\n+\t}\n+\n+\tcron_in = fdopen(crontab_edit.in, \"w\");\n+\tif (!cron_in) {\n+\t\tresult = error(_(\"failed to open stdin of 'crontab'\"));\n+\t\tgoto done_editing;\n+\t}\n+\n+\twhile (!strbuf_getline_lf(&line, cron_list)) {\n+\t\tif (!in_old_region && !strcmp(line.buf, BEGIN_LINE))\n+\t\t\tin_old_region = 1;\n+\t\tif (in_old_region)\n+\t\t\tcontinue;\n+\t\tfprintf(cron_in, \"%s\\n\", line.buf);\n+\t\tif (in_old_region && !strcmp(line.buf, END_LINE))\n+\t\t\tin_old_region = 0;\n+\t}\n+\n+\tif (run_maintenance) {\n+\t\tstruct strbuf line_format = STRBUF_INIT;\n+\t\tconst char *exec_path = git_exec_path();\n+\n+\t\tfprintf(cron_in, \"%s\\n\", BEGIN_LINE);\n+\t\tfprintf(cron_in,\n+\t\t\t\"# The following schedule was created by Git\\n\");\n+\t\tfprintf(cron_in, \"# Any edits made in this region might be\\n\");\n+\t\tfprintf(cron_in,\n+\t\t\t\"# replaced in the future by a Git command.\\n\\n\");\n+\n+\t\tstrbuf_addf(&line_format,\n+\t\t\t \"%%s %%s * * %%s \\\"%s/git\\\" --exec-path=\\\"%s\\\" for-each-repo --config=maintenance.repo maintenance run --schedule=%%s\\n\",\n+\t\t\t exec_path, exec_path);\n+\t\tfprintf(cron_in, line_format.buf, \"0\", \"1-23\", \"*\", \"hourly\");\n+\t\tfprintf(cron_in, line_format.buf, \"0\", \"0\", \"1-6\", \"daily\");\n+\t\tfprintf(cron_in, line_format.buf, \"0\", \"0\", \"0\", \"weekly\");\n+\t\tstrbuf_release(&line_format);\n+\n+\t\tfprintf(cron_in, \"\\n%s\\n\", END_LINE);\n+\t}\n+\n+\tfflush(cron_in);\n+\tfclose(cron_in);\n+\tclose(crontab_edit.in);\n+\n+done_editing:\n+\tif (finish_command(&crontab_edit)) {\n+\t\tresult = error(_(\"'crontab' died\"));\n+\t\tgoto cleanup;\n+\t}\n+\tfclose(cron_list);\n+\n+cleanup:\n+\trollback_lock_file(&lk);\n+\treturn result;\n+}\n+\n+static int maintenance_start(void)\n+{\n+\tif (maintenance_register())\n+\t\twarning(_(\"failed to add repo to global config\"));\n+\n+\treturn update_background_schedule(1);\n+}\n+\n+static int maintenance_stop(void)\n+{\n+\treturn update_background_schedule(0);\n+}\n+\n static const char builtin_maintenance_usage[] =\tN_(\"git maintenance []\");\n \n int cmd_maintenance(int argc, const char **argv, const char *prefix)\n@@ -1466,6 +1586,10 @@ int cmd_maintenance(int argc, const char **argv, const char *prefix)\n \n \tif (!strcmp(argv[1], \"run\"))\n \t\treturn maintenance_run(argc - 1, argv + 1, prefix);\n+\tif (!strcmp(argv[1], \"start\"))\n+\t\treturn maintenance_start();\n+\tif (!strcmp(argv[1], \"stop\"))\n+\t\treturn maintenance_stop();\n \tif (!strcmp(argv[1], \"register\"))\n \t\treturn maintenance_register();\n \tif (!strcmp(argv[1], \"unregister\"))\ndiff --git a/t/helper/test-crontab.c b/t/helper/test-crontab.c\nnew file mode 100644\nindex 0000000000..e7c0137a47\n--- /dev/null\n+++ b/t/helper/test-crontab.c\n@@ -0,0 +1,35 @@\n+#include \"test-tool.h\"\n+#include \"cache.h\"\n+\n+/*\n+ * Usage: test-tool cron [-l]\n+ *\n+ * If -l is specified, then write the contents of to stdout.\n+ * Otherwise, write from stdin into .\n+ */\n+int cmd__crontab(int argc, const char **argv)\n+{\n+\tint a;\n+\tFILE *from, *to;\n+\n+\tif (argc == 3 && !strcmp(argv[2], \"-l\")) {\n+\t\tfrom = fopen(argv[1], \"r\");\n+\t\tif (!from)\n+\t\t\treturn 0;\n+\t\tto = stdout;\n+\t} else if (argc == 2) {\n+\t\tfrom = stdin;\n+\t\tto = fopen(argv[1], \"w\");\n+\t} else\n+\t\treturn error(\"unknown arguments\");\n+\n+\twhile ((a = fgetc(from)) != EOF)\n+\t\tfputc(a, to);\n+\n+\tif (argc == 3)\n+\t\tfclose(from);\n+\telse\n+\t\tfclose(to);\n+\n+\treturn 0;\n+}\ndiff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\nindex 590b2efca7..432b49d948 100644\n--- a/t/helper/test-tool.c\n+++ b/t/helper/test-tool.c\n@@ -18,6 +18,7 @@ static struct test_cmd cmds[] = {\n \t{ \"bloom\", cmd__bloom },\n \t{ \"chmtime\", cmd__chmtime },\n \t{ \"config\", cmd__config },\n+\t{ \"crontab\", cmd__crontab },\n \t{ \"ctype\", cmd__ctype },\n \t{ \"date\", cmd__date },\n \t{ \"delta\", cmd__delta },\ndiff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\nindex ddc8e990e9..7c3281e071 100644\n--- a/t/helper/test-tool.h\n+++ b/t/helper/test-tool.h\n@@ -8,6 +8,7 @@ int cmd__advise_if_enabled(int argc, const char **argv);\n int cmd__bloom(int argc, const char **argv);\n int cmd__chmtime(int argc, const char **argv);\n int cmd__config(int argc, const char **argv);\n+int cmd__crontab(int argc, const char **argv);\n int cmd__ctype(int argc, const char **argv);\n int cmd__date(int argc, const char **argv);\n int cmd__delta(int argc, const char **argv);\ndiff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\nindex 272d1605d2..8803fcf621 100755\n--- a/t/t7900-maintenance.sh\n+++ b/t/t7900-maintenance.sh\n@@ -319,4 +319,32 @@ test_expect_success 'register and unregister' '\n \ttest_cmp before actual\n '\n \n+test_expect_success 'start from empty cron table' '\n+\tGIT_TEST_CRONTAB=\"test-tool crontab cron.txt\" git maintenance start &&\n+\n+\t# start registers the repo\n+\tgit config --get --global maintenance.repo \"$(pwd)\" &&\n+\n+\tgrep \"for-each-repo --config=maintenance.repo maintenance run --schedule=daily\" cron.txt &&\n+\tgrep \"for-each-repo --config=maintenance.repo maintenance run --schedule=hourly\" cron.txt &&\n+\tgrep \"for-each-repo --config=maintenance.repo maintenance run --schedule=weekly\" cron.txt\n+'\n+\n+test_expect_success 'stop from existing schedule' '\n+\tGIT_TEST_CRONTAB=\"test-tool crontab cron.txt\" git maintenance stop &&\n+\n+\t# stop does not unregister the repo\n+\tgit config --get --global maintenance.repo \"$(pwd)\" &&\n+\n+\t# Operation is idempotent\n+\tGIT_TEST_CRONTAB=\"test-tool crontab cron.txt\" git maintenance stop &&\n+\ttest_must_be_empty cron.txt\n+'\n+\n+test_expect_success 'start preserves existing schedule' '\n+\techo \"Important information!\" >cron.txt &&\n+\tGIT_TEST_CRONTAB=\"test-tool crontab cron.txt\" git maintenance start &&\n+\tgrep \"Important information!\" cron.txt\n+'\n+\n test_done\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex ef31f40037..4a60d1ed76 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -1702,3 +1702,9 @@ test_lazy_prereq SHA1 '\n test_lazy_prereq REBASE_P '\n \ttest -z \"$GIT_TEST_SKIP_REBASE_P\"\n '\n+\n+# Ensure that no test accidentally triggers a Git command\n+# that runs 'crontab', affecting a user's cron schedule.\n+# Tests that verify the cron integration must set this locally\n+# to avoid errors.\n+GIT_TEST_CRONTAB=\"exit 1\"\n-- \ngitgitgadget\n\n"},{"id":"405425","messageId":"2344eff4ba3f4e1ae1532c1521748c743732f236.1599846561.git.gitgitgadget@gmail.com","threadId":"54189","inReplyTo":"pull.724.v2.git.1599846560.gitgitgadget@gmail.com","subject":"[PATCH v2 7/7] maintenance: add troubleshooting guide to docs","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-09-11T17:49:20Z","receivedAt":"2020-09-11T17:49:52Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe 'git maintenance run' subcommand takes a lock on the object database\nto prevent concurrent processes from competing for resources. This is an\nimportant safety measure to prevent possible repository corruption and\ndata loss.\n\nThis feature can lead to confusing behavior if a user is not aware of\nit. Add a TROUBLESHOOTING section to the 'git maintenance' builtin\ndocumentation that discusses these tradeoffs. The short version of this\nsection is that Git will not corrupt your repository, but if the list of\nscheduled tasks takes longer than an hour then some scheduled tasks may\nbe dropped due to this object database collision. For example, a\nlong-running \"daily\" task at midnight might prevent an \"hourly\" task\nfrom running at 1AM.\n\nThe opposite is also possible, but less likely as long as the \"hourly\"\ntasks are much faster than the \"daily\" and \"weekly\" tasks.\n\nHelped-by: Junio C Hamano \nSigned-off-by: Derrick Stolee \n---\n Documentation/git-maintenance.txt | 44 +++++++++++++++++++++++++++++++\n 1 file changed, 44 insertions(+)\n\ndiff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt\nindex 364b3e32bf..f58dd60e40 100644\n--- a/Documentation/git-maintenance.txt\n+++ b/Documentation/git-maintenance.txt\n@@ -161,6 +161,50 @@ OPTIONS\n \t`maintenance..enabled` configured as `true` are considered.\n \tSee the 'TASKS' section for the list of accepted `` values.\n \n+\n+TROUBLESHOOTING\n+---------------\n+The `git maintenance` command is designed to simplify the repository\n+maintenance patterns while minimizing user wait time during Git commands.\n+A variety of configuration options are available to allow customizing this\n+process. The default maintenance options focus on operations that complete\n+quickly, even on large repositories.\n+\n+Users may find some cases where scheduled maintenance tasks do not run as\n+frequently as intended. Each `git maintenance run` command takes a lock on\n+the repository's object database, and this prevents other concurrent\n+`git maintenance run` commands from running on the same repository. Without\n+this safeguard, competing processes could leave the repository in an\n+unpredictable state.\n+\n+The background maintenance schedule runs `git maintenance run` processes\n+on an hourly basis. Each run executes the \"hourly\" tasks. At midnight,\n+that process also executes the \"daily\" tasks. At midnight on the first day\n+of the week, that process also executes the \"weekly\" tasks. A single\n+process iterates over each registered repository, performing the scheduled\n+tasks for that frequency. Depending on the number of registered\n+repositories and their sizes, this process may take longer than an hour.\n+In this case, multiple `git maintenance run` commands may run on the same\n+repository at the same time, colliding on the object database lock. This\n+results in one of the two tasks not running.\n+\n+If you find that some maintenance windows are taking longer than one hour\n+to complete, then consider reducing the complexity of your maintenance\n+tasks. For example, the `gc` task is much slower than the\n+`incremental-repack` task. However, this comes at a cost of a slightly\n+larger object database. Consider moving more expensive tasks to be run\n+less frequently.\n+\n+Expert users may consider scheduling their own maintenance tasks using a\n+different schedule than is available through `git maintenance start` and\n+Git configuration options. These users should be aware of the object\n+database lock and how concurrent `git maintenance run` commands behave.\n+Further, the `git gc` command should not be combined with\n+`git maintenance run` commands. `git gc` modifies the object database\n+but does not take the lock in the same way as `git maintenance run`. If\n+possible, use `git maintenance run --task=gc` instead of `git gc`.\n+\n+\n GIT\n ---\n Part of the linkgit:git[1] suite\n-- \ngitgitgadget\n"},{"id":"405426","messageId":"f609c1bde27558db2a9601ac34f8a51ce86f0e2c.1599846561.git.gitgitgadget@gmail.com","threadId":"54189","inReplyTo":"pull.724.v2.git.1599846560.gitgitgadget@gmail.com","subject":"[PATCH v2 6/7] maintenance: recommended schedule in register/start","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-09-11T17:49:19Z","receivedAt":"2020-09-11T17:50:00Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe 'git maintenance (register|start)' subcommands add the current\nrepository to the global Git config so maintenance will operate on that\nrepository. It does not specify what maintenance should occur or how\noften.\n\nIf a user sets any 'maintenance..schedule' config value, then\nthey have chosen a specific schedule for themselves and Git should\nrespect that.\n\nHowever, in an effort to recommend a good schedule for repositories of\nall sizes, set new config values for recommended tasks that are safe to\nrun in the background while users run foreground Git commands. These\ncommands are generally everything but the 'gc' task.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-maintenance.txt | 6 ++++\n builtin/gc.c | 46 +++++++++++++++++++++++++++++++\n t/t7900-maintenance.sh | 16 +++++++++++\n 3 files changed, 68 insertions(+)\n\ndiff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt\nindex 7f8c279fe8..364b3e32bf 100644\n--- a/Documentation/git-maintenance.txt\n+++ b/Documentation/git-maintenance.txt\n@@ -37,6 +37,12 @@ register::\n \t`maintenance..schedule`. The tasks that are enabled are safe\n \tfor running in the background without disrupting foreground\n \tprocesses.\n++\n+If your repository has no 'maintenance..schedule' configuration\n+values set, then Git will set configuration values to some recommended\n+settings. These settings disable foreground maintenance while performing\n+maintenance tasks in the background that will not interrupt foreground Git\n+operations.\n \n run::\n \tRun one or more maintenance tasks. If one or more `--task` options\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 4f2d17c0c0..2ef4c0960c 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -1409,6 +1409,49 @@ static int maintenance_run(int argc, const char **argv, const char *prefix)\n \treturn maintenance_run_tasks(&opts);\n }\n \n+static int has_schedule_config(void)\n+{\n+\tint i, found = 0;\n+\tstruct strbuf config_name = STRBUF_INIT;\n+\tsize_t prefix;\n+\n+\tstrbuf_addstr(&config_name, \"maintenance.\");\n+\tprefix = config_name.len;\n+\n+\tfor (i = 0; !found && i < TASK__COUNT; i++) {\n+\t\tchar *value;\n+\n+\t\tstrbuf_setlen(&config_name, prefix);\n+\t\tstrbuf_addf(&config_name, \"%s.schedule\", tasks[i].name);\n+\n+\t\tif (!git_config_get_string(config_name.buf, &value)) {\n+\t\t\tfound = 1;\n+\t\t\tFREE_AND_NULL(value);\n+\t\t}\n+\t}\n+\n+\tstrbuf_release(&config_name);\n+\treturn found;\n+}\n+\n+static void set_recommended_schedule(void)\n+{\n+\tgit_config_set(\"maintenance.auto\", \"false\");\n+\tgit_config_set(\"maintenance.gc.enabled\", \"false\");\n+\n+\tgit_config_set(\"maintenance.prefetch.enabled\", \"true\");\n+\tgit_config_set(\"maintenance.prefetch.schedule\", \"hourly\");\n+\n+\tgit_config_set(\"maintenance.commit-graph.enabled\", \"true\");\n+\tgit_config_set(\"maintenance.commit-graph.schedule\", \"hourly\");\n+\n+\tgit_config_set(\"maintenance.loose-objects.enabled\", \"true\");\n+\tgit_config_set(\"maintenance.loose-objects.schedule\", \"daily\");\n+\n+\tgit_config_set(\"maintenance.incremental-repack.enabled\", \"true\");\n+\tgit_config_set(\"maintenance.incremental-repack.schedule\", \"daily\");\n+}\n+\n static int maintenance_register(void)\n {\n \tstruct child_process config_set = CHILD_PROCESS_INIT;\n@@ -1418,6 +1461,9 @@ static int maintenance_register(void)\n \tif (!the_repository || !the_repository->gitdir)\n \t\treturn 0;\n \n+\tif (!has_schedule_config())\n+\t\tset_recommended_schedule();\n+\n \tconfig_get.git_cmd = 1;\n \tstrvec_pushl(&config_get.args, \"config\", \"--global\", \"--get\", \"maintenance.repo\",\n \t\t the_repository->worktree ? the_repository->worktree\ndiff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\nindex 8803fcf621..5a31f3925b 100755\n--- a/t/t7900-maintenance.sh\n+++ b/t/t7900-maintenance.sh\n@@ -309,7 +309,23 @@ test_expect_success 'register and unregister' '\n \tgit config --global --add maintenance.repo /existing1 &&\n \tgit config --global --add maintenance.repo /existing2 &&\n \tgit config --global --get-all maintenance.repo >before &&\n+\n+\t# We still have maintenance..schedule config set,\n+\t# so this does not update the local schedule\n+\tgit maintenance register &&\n+\ttest_must_fail git config maintenance.auto &&\n+\n+\t# Clear previous maintenance..schedule values\n+\tfor task in loose-objects commit-graph incremental-repack\n+\tdo\n+\t\tgit config --unset maintenance.$task.schedule || return 1\n+\tdone &&\n \tgit maintenance register &&\n+\ttest_cmp_config false maintenance.auto &&\n+\ttest_cmp_config false maintenance.gc.enabled &&\n+\ttest_cmp_config true maintenance.prefetch.enabled &&\n+\ttest_cmp_config hourly maintenance.commit-graph.schedule &&\n+\ttest_cmp_config daily maintenance.incremental-repack.schedule &&\n \tgit config --global --get-all maintenance.repo >actual &&\n \tcp before after &&\n \tpwd >>after &&\n-- \ngitgitgadget\n\n"},{"id":"405752","messageId":"20200917140543.GC8664@danh.dev","threadId":"54189","inReplyTo":"1f49cda18ee8d6c585807506cf89c365de642ea7.1599846561.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 4/7] maintenance: add [un]register subcommands","fromName":"Đoàn Trần Công Danh","fromEmail":"congdanhqx@gmail.com","sentAt":"2020-09-17T14:05:43Z","receivedAt":"2020-09-17T14:30:34Z","isPatch":true,"sender":{"key":"congdanhqx@gmail.com","avatar":"https://avatars.githubusercontent.com/u/42673067?v=4"},"body":"\nHi Stolee,\n\nSorry for reply this late.\n\nOn 2020-09-11 17:49:17+0000, Derrick Stolee via GitGitGadget wrote:\n> From: Derrick Stolee \n> \n> In preparation for launching background maintenance from the 'git\n> maintenance' builtin, create register/unregister subcommands. These\n> commands update the new 'maintenance.repos' config option in the global\n\nAnd also not spot this earlier.\n\nI think you meant 'maintenance.repo' (without s) here\nsince it's the one that was mentioned in the patch itself.\n\nOther than that, this series looks sane to me.\n\nThanks\nDanh\n\n> config so the background maintenance job knows which repositories to\n> maintain.\n> \n> These commands allow users to add a repository to the background\n> maintenance list without disrupting the actual maintenance mechanism.\n> \n> For example, a user can run 'git maintenance register' when no\n> background maintenance is running and it will not start the background\n> maintenance. A later update to start running background maintenance will\n> then pick up this repository automatically.\n> \n> The opposite example is that a user can run 'git maintenance unregister'\n> to remove the current repository from background maintenance without\n> halting maintenance for other repositories.\n> \n> Signed-off-by: Derrick Stolee \n> ---\n> Documentation/git-maintenance.txt | 14 ++++++++\n> builtin/gc.c | 55 ++++++++++++++++++++++++++++++-\n> t/t7900-maintenance.sh | 17 +++++++++-\n> 3 files changed, 84 insertions(+), 2 deletions(-)\n> \n> diff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt\n> index 3af5907b01..78d0d8df91 100644\n> --- a/Documentation/git-maintenance.txt\n> +++ b/Documentation/git-maintenance.txt\n> @@ -29,6 +29,15 @@ Git repository.\n> SUBCOMMANDS\n> -----------\n> \n> +register::\n> +\tInitialize Git config values so any scheduled maintenance will\n> +\tstart running on this repository. This adds the repository to the\n> +\t`maintenance.repo` config variable in the current user's global\n> +\tconfig and enables some recommended configuration values for\n> +\t`maintenance..schedule`. The tasks that are enabled are safe\n> +\tfor running in the background without disrupting foreground\n> +\tprocesses.\n> +\n> run::\n> \tRun one or more maintenance tasks. If one or more `--task` options\n> \tare specified, then those tasks are run in that order. Otherwise,\n> @@ -36,6 +45,11 @@ run::\n> \tconfig options are true. By default, only `maintenance.gc.enabled`\n> \tis true.\n> \n> +unregister::\n> +\tRemove the current repository from background maintenance. This\n> +\tonly removes the repository from the configured list. It does not\n> +\tstop the background maintenance processes from running.\n> +\n> TASKS\n> -----\n> \n> diff --git a/builtin/gc.c b/builtin/gc.c\n> index e28561b6c5..0290b249c9 100644\n> --- a/builtin/gc.c\n> +++ b/builtin/gc.c\n> @@ -1408,7 +1408,56 @@ static int maintenance_run(int argc, const char **argv, const char *prefix)\n> \treturn maintenance_run_tasks(&opts);\n> }\n> \n> -static const char builtin_maintenance_usage[] = N_(\"git maintenance run []\");\n> +static int maintenance_register(void)\n> +{\n> +\tstruct child_process config_set = CHILD_PROCESS_INIT;\n> +\tstruct child_process config_get = CHILD_PROCESS_INIT;\n> +\n> +\t/* There is no current repository, so skip registering it */\n> +\tif (!the_repository || !the_repository->gitdir)\n> +\t\treturn 0;\n> +\n> +\tconfig_get.git_cmd = 1;\n> +\tstrvec_pushl(&config_get.args, \"config\", \"--global\", \"--get\", \"maintenance.repo\",\n> +\t\t the_repository->worktree ? the_repository->worktree\n> +\t\t\t\t\t : the_repository->gitdir,\n> +\t\t\t NULL);\n> +\tconfig_get.out = -1;\n> +\n> +\tif (start_command(&config_get))\n> +\t\treturn error(_(\"failed to run 'git config'\"));\n> +\n> +\t/* We already have this value in our config! */\n> +\tif (!finish_command(&config_get))\n> +\t\treturn 0;\n> +\n> +\tconfig_set.git_cmd = 1;\n> +\tstrvec_pushl(&config_set.args, \"config\", \"--add\", \"--global\", \"maintenance.repo\",\n> +\t\t the_repository->worktree ? the_repository->worktree\n> +\t\t\t\t\t : the_repository->gitdir,\n> +\t\t NULL);\n> +\n> +\treturn run_command(&config_set);\n> +}\n> +\n> +static int maintenance_unregister(void)\n> +{\n> +\tstruct child_process config_unset = CHILD_PROCESS_INIT;\n> +\n> +\tif (!the_repository || !the_repository->gitdir)\n> +\t\treturn error(_(\"no current repository to unregister\"));\n> +\n> +\tconfig_unset.git_cmd = 1;\n> +\tstrvec_pushl(&config_unset.args, \"config\", \"--global\", \"--unset\",\n> +\t\t \"maintenance.repo\",\n> +\t\t the_repository->worktree ? the_repository->worktree\n> +\t\t\t\t\t : the_repository->gitdir,\n> +\t\t NULL);\n> +\n> +\treturn run_command(&config_unset);\n> +}\n> +\n> +static const char builtin_maintenance_usage[] =\tN_(\"git maintenance []\");\n> \n> int cmd_maintenance(int argc, const char **argv, const char *prefix)\n> {\n> @@ -1417,6 +1466,10 @@ int cmd_maintenance(int argc, const char **argv, const char *prefix)\n> \n> \tif (!strcmp(argv[1], \"run\"))\n> \t\treturn maintenance_run(argc - 1, argv + 1, prefix);\n> +\tif (!strcmp(argv[1], \"register\"))\n> +\t\treturn maintenance_register();\n> +\tif (!strcmp(argv[1], \"unregister\"))\n> +\t\treturn maintenance_unregister();\n> \n> \tdie(_(\"invalid subcommand: %s\"), argv[1]);\n> }\n> diff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\n> index 328bbaa830..272d1605d2 100755\n> --- a/t/t7900-maintenance.sh\n> +++ b/t/t7900-maintenance.sh\n> @@ -9,7 +9,7 @@ GIT_TEST_MULTI_PACK_INDEX=0\n> \n> test_expect_success 'help text' '\n> \ttest_expect_code 129 git maintenance -h 2>err &&\n> -\ttest_i18ngrep \"usage: git maintenance run\" err &&\n> +\ttest_i18ngrep \"usage: git maintenance \" err &&\n> \ttest_expect_code 128 git maintenance barf 2>err &&\n> \ttest_i18ngrep \"invalid subcommand: barf\" err\n> '\n> @@ -304,4 +304,19 @@ test_expect_success '--schedule inheritance weekly -> daily -> hourly' '\n> \ttest_subcommand git multi-pack-index write --no-progress '\n> \n> +test_expect_success 'register and unregister' '\n> +\ttest_when_finished git config --global --unset-all maintenance.repo &&\n> +\tgit config --global --add maintenance.repo /existing1 &&\n> +\tgit config --global --add maintenance.repo /existing2 &&\n> +\tgit config --global --get-all maintenance.repo >before &&\n> +\tgit maintenance register &&\n> +\tgit config --global --get-all maintenance.repo >actual &&\n> +\tcp before after &&\n> +\tpwd >>after &&\n> +\ttest_cmp after actual &&\n> +\tgit maintenance unregister &&\n> +\tgit config --global --get-all maintenance.repo >actual &&\n> +\ttest_cmp before actual\n> +'\n> +\n> test_done\n> -- \n> gitgitgadget\n> \n\n-- \nDanh\n"},{"id":"406639","messageId":"CAN0heSqkJoqXKP5ccaGMA1_ppd0bcQ7G0ozUH+H7tBMonhcrjQ@mail.gmail.com","threadId":"54189","inReplyTo":"f609c1bde27558db2a9601ac34f8a51ce86f0e2c.1599846561.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 6/7] maintenance: recommended schedule in register/start","fromName":"Martin Ågren","fromEmail":"martin.agren@gmail.com","sentAt":"2020-09-29T19:48:54Z","receivedAt":"2020-09-29T19:49:09Z","isPatch":true,"sender":{"key":"martin.agren@gmail.com","avatar":null},"body":"Hi Stolee,\n\nOn Fri, 11 Sep 2020 at 19:53, Derrick Stolee via GitGitGadget\n wrote:\n> If a user sets any 'maintenance..schedule' config value, then\n> they have chosen a specific schedule for themselves and Git should\n> respect that.\n>\n> However, in an effort to recommend a good schedule for repositories of\n> all sizes, set new config values for recommended tasks that are safe to\n> run in the background while users run foreground Git commands. These\n> commands are generally everything but the 'gc' task.\n\nIf there aren't any \"schedule\" configurations, we'll go ahead and\nsprinkle in quite a few of them. I suppose that another approach would\nbe that later, much later, when we go look for these configuration\nitems, we could go \"there is not a single one set, let's act as if\n*these* were configured\".\n\nThe advantage there would be that we can tweak those defaults over time.\nWhereas with the approach of this patch, v2.29.0 will give the user a\nsnapshot of 2020's best practices. If they want to catch up, they will\nneed to drop all their \"schedule\" config and re-\"register\", or use a\nfuture `git maintenance reregister`. ;-)\n\nAnyway, this is a convenience thing. There's a chance that \"convenience\"\ninterferes with \"perfect\" and \"optimal\". I guess that's to be expected.\n\n> +If your repository has no 'maintenance..schedule' configuration\n\nThank you for going above and beyond with marking config items et cetera\nfor rendering in `monospace`. I just noticed that this is slightly\nmis-marked-upped. If you end up rerolling this patch series for some\nreason, you might want to switch from 'single quotes' to `backticks` in\nthis particular instance.\n\nWhile I'm commenting anyway...\n\n> +static int has_schedule_config(void)\n> +{\n> + int i, found = 0;\n> + struct strbuf config_name = STRBUF_INIT;\n> + size_t prefix;\n> +\n> + strbuf_addstr(&config_name, \"maintenance.\");\n> + prefix = config_name.len;\n> +\n> + for (i = 0; !found && i < TASK__COUNT; i++) {\n> + char *value;\n> +\n> + strbuf_setlen(&config_name, prefix);\n> + strbuf_addf(&config_name, \"%s.schedule\", tasks[i].name);\n> +\n> + if (!git_config_get_string(config_name.buf, &value)) {\n> + found = 1;\n> + FREE_AND_NULL(value);\n> + }\n> + }\n> +\n> + strbuf_release(&config_name);\n> + return found;\n> +}\n\nThat `FREE_AND_NULL()` caught me off-guard. The pointer is on the stack.\nI suppose it doesn't *hurt*, but being careful to set it to NULL made me\ngo \"huh\".\n\nI suppose you could drop the `!found` check in favour of `break`-ing\nprecisely when you get a hit.\n\nAnd I do wonder how much the reuse of the \"maintenance.\" part of the\nbuffer helps performance.\n\nIn the end, you could use something like the following (not compiled):\n\n static int has_schedule_config(void)\n {\n int i, found = 0;\n struct strbuf config_name = STRBUF_INIT;\n\n for (i = 0; i < TASK__COUNT; i++) {\n const char *value;\n\n strbuf_reset(&config_name);\n strbuf_addf(&config_name, \"maintenance.%s.schedule\",\ntasks[i].name);\n\n if (!git_config_get_value(config_name.buf, &value)) {\n found = 1;\n break;\n }\n }\n\n strbuf_release(&config_name);\n return found;\n }\n\nAnyway, that's just microniting, obviously, but maybe in the sum it has\nsome value.\n\n\nMartin\n"},{"id":"406703","messageId":"bb9cd08f-1e59-ae19-b184-545688451203@gmail.com","threadId":"54189","inReplyTo":"CAN0heSqkJoqXKP5ccaGMA1_ppd0bcQ7G0ozUH+H7tBMonhcrjQ@mail.gmail.com","subject":"Re: [PATCH v2 6/7] maintenance: recommended schedule in register/start","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-09-30T20:11:47Z","receivedAt":"2020-09-30T20:11:57Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 9/29/2020 3:48 PM, Martin Ågren wrote:\n> Hi Stolee,\n> \n> On Fri, 11 Sep 2020 at 19:53, Derrick Stolee via GitGitGadget\n> wrote:\n>> If a user sets any 'maintenance..schedule' config value, then\n>> they have chosen a specific schedule for themselves and Git should\n>> respect that.\n>>\n>> However, in an effort to recommend a good schedule for repositories of\n>> all sizes, set new config values for recommended tasks that are safe to\n>> run in the background while users run foreground Git commands. These\n>> commands are generally everything but the 'gc' task.\n> \n> If there aren't any \"schedule\" configurations, we'll go ahead and\n> sprinkle in quite a few of them. I suppose that another approach would\n> be that later, much later, when we go look for these configuration\n> items, we could go \"there is not a single one set, let's act as if\n> *these* were configured\".\n\nI do like this alternative.\n\n> The advantage there would be that we can tweak those defaults over time.\n> Whereas with the approach of this patch, v2.29.0 will give the user a\n> snapshot of 2020's best practices. If they want to catch up, they will\n> need to drop all their \"schedule\" config and re-\"register\", or use a\n> future `git maintenance reregister`. ;-)\n\nThis is a significant advantage! Great idea.\n\nIt might be a bit difficult to slide this in, but I bet it would work\nout OK if we have a \"initialize_schedule()\" option that is only run\nwhen the \"--schedule=<...>\" option is given? The trickiest part is\nactually setting the \".enabled\" configs to \"true\" as well. The condition\nfor using the \"default\" schedule might get a bit complicated. I do think\nit is worth some effort to do, as adjusting defaults in code is certainly\neasier than modifying config values.\n\n> Anyway, this is a convenience thing. There's a chance that \"convenience\"\n> interferes with \"perfect\" and \"optimal\". I guess that's to be expected.\n> \n>> +If your repository has no 'maintenance..schedule' configuration\n> \n> Thank you for going above and beyond with marking config items et cetera\n> for rendering in `monospace`. I just noticed that this is slightly\n> mis-marked-upped. If you end up rerolling this patch series for some\n> reason, you might want to switch from 'single quotes' to `backticks` in\n> this particular instance.\n\nThanks! Yeah that was a mis-type.\n\n> While I'm commenting anyway...\n> \n>> +static int has_schedule_config(void)\n>> +{\n>> + int i, found = 0;\n>> + struct strbuf config_name = STRBUF_INIT;\n>> + size_t prefix;\n>> +\n>> + strbuf_addstr(&config_name, \"maintenance.\");\n>> + prefix = config_name.len;\n>> +\n>> + for (i = 0; !found && i < TASK__COUNT; i++) {\n>> + char *value;\n>> +\n>> + strbuf_setlen(&config_name, prefix);\n>> + strbuf_addf(&config_name, \"%s.schedule\", tasks[i].name);\n>> +\n>> + if (!git_config_get_string(config_name.buf, &value)) {\n>> + found = 1;\n>> + FREE_AND_NULL(value);\n>> + }\n>> + }\n>> +\n>> + strbuf_release(&config_name);\n>> + return found;\n>> +}\n> \n> That `FREE_AND_NULL()` caught me off-guard. The pointer is on the stack.\n> I suppose it doesn't *hurt*, but being careful to set it to NULL made me\n> go \"huh\".\n> \n> I suppose you could drop the `!found` check in favour of `break`-ing\n> precisely when you get a hit.\n> \n> And I do wonder how much the reuse of the \"maintenance.\" part of the\n> buffer helps performance.\n\nAll valid points.\n\n> In the end, you could use something like the following (not compiled):\n> \n> static int has_schedule_config(void)\n> {\n> int i, found = 0;\n> struct strbuf config_name = STRBUF_INIT;\n> \n> for (i = 0; i < TASK__COUNT; i++) {\n> const char *value;\n> \n> strbuf_reset(&config_name);\n> strbuf_addf(&config_name, \"maintenance.%s.schedule\",\n> tasks[i].name);\n> \n> if (!git_config_get_value(config_name.buf, &value)) {\n> found = 1;\n> break;\n> }\n> }\n> \n> strbuf_release(&config_name);\n> return found;\n> }\n> \n> Anyway, that's just microniting, obviously, but maybe in the sum it has\n> some value.\n\nSounds good to me. I'll work on a new version that makes your\nrecommendations.\n\nThanks,\n-Stolee\n\n\n\n"},{"id":"406790","messageId":"3e18199a-1dcb-057d-4753-03d5acfcc036@gmail.com","threadId":"54189","inReplyTo":"bb9cd08f-1e59-ae19-b184-545688451203@gmail.com","subject":"Re: [PATCH v2 6/7] maintenance: recommended schedule in register/start","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-10-01T20:38:48Z","receivedAt":"2020-10-01T20:38:53Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 9/30/2020 4:11 PM, Derrick Stolee wrote:\n> On 9/29/2020 3:48 PM, Martin Ågren wrote:\n>> If there aren't any \"schedule\" configurations, we'll go ahead and\n>> sprinkle in quite a few of them. I suppose that another approach would\n>> be that later, much later, when we go look for these configuration\n>> items, we could go \"there is not a single one set, let's act as if\n>> *these* were configured\".\n> \n> I do like this alternative.\n> \n>> The advantage there would be that we can tweak those defaults over time.\n>> Whereas with the approach of this patch, v2.29.0 will give the user a\n>> snapshot of 2020's best practices. If they want to catch up, they will\n>> need to drop all their \"schedule\" config and re-\"register\", or use a\n>> future `git maintenance reregister`. ;-)\n> \n> This is a significant advantage! Great idea.\n\nThank you for giving me the idea to pursue this direction. The\nreplacement patch isn't too bad, and I think this is a much better\napproach to satisfy the situation.\n\nWhat do you think?\n\nThanks,\n-Stolee\n\n-- >8 --\n\nFrom 7c7698da17327d17485ba1b23a16a0a8d54efaad Mon Sep 17 00:00:00 2001\nFrom: Derrick Stolee \nDate: Tue, 18 Aug 2020 15:15:02 -0400\nSubject: [PATCH] maintenance: recommended schedule in register/start\nMIME-Version: 1.0\nContent-Type: text/plain; charset=UTF-8\nContent-Transfer-Encoding: 8bit\n\nThe 'git maintenance (register|start)' subcommands add the current\nrepository to the global Git config so maintenance will operate on that\nrepository. It does not specify what maintenance should occur or how\noften.\n\nIf a user sets any 'maintenance..schedule' config value, then\nthey have chosen a specific schedule for themselves and Git should\nrespect that.\n\nTo make this process extremely simple for users, assume a default\nschedule when no 'maintenance..schedule' or '...enabled' config\nsettings are concretely set. This is only an in-process assumption, so\nfuture versions of Git could adjust this expected schedule.\n\nHelped-by: Martin Ågren \nSigned-off-by: Derrick Stolee \n---\n Documentation/git-maintenance.txt | 15 ++++++++\n builtin/gc.c | 58 +++++++++++++++++++++++++++++++\n t/t7900-maintenance.sh | 11 +++---\n 3 files changed, 80 insertions(+), 4 deletions(-)\n\ndiff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt\nindex 7628a6d157..52fff86844 100644\n--- a/Documentation/git-maintenance.txt\n+++ b/Documentation/git-maintenance.txt\n@@ -37,6 +37,21 @@ register::\n \t`maintenance..schedule`. The tasks that are enabled are safe\n \tfor running in the background without disrupting foreground\n \tprocesses.\n++\n+If your repository has no `maintenance..schedule` configuration\n+values set, then Git will use a recommended default schedule that performs\n+background maintenance that will not interrupt foreground commands. The\n+default schedule is as follows:\n++\n+* `gc`: disabled.\n+* `commit-graph`: hourly.\n+* `prefetch`: hourly.\n+* `loose-objects`: daily.\n+* `incremental-repack`: daily.\n++\n+`git maintenance register` will also disable foreground maintenance by\n+setting `maintenance.auto = false` in the current repository. This config\n+setting will remain after a `git maintenance unregister` command.\n \n run::\n \tRun one or more maintenance tasks. If one or more `--task` options\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex a387f46585..965690704b 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -1251,6 +1251,59 @@ static int compare_tasks_by_selection(const void *a_, const void *b_)\n \treturn b->selected_order - a->selected_order;\n }\n \n+static int has_schedule_config(void)\n+{\n+\tint i, found = 0;\n+\tstruct strbuf config_name = STRBUF_INIT;\n+\tsize_t prefix;\n+\n+\tstrbuf_addstr(&config_name, \"maintenance.\");\n+\tprefix = config_name.len;\n+\n+\tfor (i = 0; !found && i < TASK__COUNT; i++) {\n+\t\tchar *value;\n+\n+\t\tstrbuf_setlen(&config_name, prefix);\n+\t\tstrbuf_addf(&config_name, \"%s.schedule\", tasks[i].name);\n+\n+\t\tif (!git_config_get_string(config_name.buf, &value)) {\n+\t\t\tfound = 1;\n+\t\t\tFREE_AND_NULL(value);\n+\t\t}\n+\n+\t\tstrbuf_setlen(&config_name, prefix);\n+\t\tstrbuf_addf(&config_name, \"%s.enabled\", tasks[i].name);\n+\n+\t\tif (!git_config_get_string(config_name.buf, &value)) {\n+\t\t\tfound = 1;\n+\t\t\tFREE_AND_NULL(value);\n+\t\t}\n+\t}\n+\n+\tstrbuf_release(&config_name);\n+\treturn found;\n+}\n+\n+static void set_recommended_schedule(void)\n+{\n+\tif (has_schedule_config())\n+\t\treturn;\n+\n+\ttasks[TASK_GC].enabled = 0;\n+\n+\ttasks[TASK_PREFETCH].enabled = 1;\n+\ttasks[TASK_PREFETCH].schedule = SCHEDULE_HOURLY;\n+\n+\ttasks[TASK_COMMIT_GRAPH].enabled = 1;\n+\ttasks[TASK_COMMIT_GRAPH].schedule = SCHEDULE_HOURLY;\n+\n+\ttasks[TASK_LOOSE_OBJECTS].enabled = 1;\n+\ttasks[TASK_LOOSE_OBJECTS].schedule = SCHEDULE_DAILY;\n+\n+\ttasks[TASK_INCREMENTAL_REPACK].enabled = 1;\n+\ttasks[TASK_INCREMENTAL_REPACK].schedule = SCHEDULE_DAILY;\n+}\n+\n static int maintenance_run_tasks(struct maintenance_run_opts *opts)\n {\n \tint i, found_selected = 0;\n@@ -1280,6 +1333,8 @@ static int maintenance_run_tasks(struct maintenance_run_opts *opts)\n \n \tif (found_selected)\n \t\tQSORT(tasks, TASK__COUNT, compare_tasks_by_selection);\n+\telse if (opts->schedule != SCHEDULE_NONE)\n+\t\tset_recommended_schedule();\n \n \tfor (i = 0; i < TASK__COUNT; i++) {\n \t\tif (found_selected && tasks[i].selected_order < 0)\n@@ -1417,6 +1472,9 @@ static int maintenance_register(void)\n \tif (!the_repository || !the_repository->gitdir)\n \t\treturn 0;\n \n+\t/* Disable foreground maintenance */\n+\tgit_config_set(\"maintenance.auto\", \"false\");\n+\n \tconfig_get.git_cmd = 1;\n \tstrvec_pushl(&config_get.args, \"config\", \"--global\", \"--get\", \"maintenance.repo\",\n \t\t the_repository->worktree ? the_repository->worktree\ndiff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\nindex c8d7e65d3d..fae2ef81bd 100755\n--- a/t/t7900-maintenance.sh\n+++ b/t/t7900-maintenance.sh\n@@ -305,11 +305,14 @@ test_expect_success 'register and unregister' '\n \tgit config --global --add maintenance.repo /existing1 &&\n \tgit config --global --add maintenance.repo /existing2 &&\n \tgit config --global --get-all maintenance.repo >before &&\n+\n \tgit maintenance register &&\n-\tgit config --global --get-all maintenance.repo >actual &&\n-\tcp before after &&\n-\tpwd >>after &&\n-\ttest_cmp after actual &&\n+\ttest_cmp_config false maintenance.auto &&\n+\tgit config --global --get-all maintenance.repo >between &&\n+\tcp before expect &&\n+\tpwd >>expect &&\n+\ttest_cmp expect between &&\n+\n \tgit maintenance unregister &&\n \tgit config --global --get-all maintenance.repo >actual &&\n \ttest_cmp before actual\n-- \n2.28.0.284.gc2951c3dd58.dirty\n\n\n\n"},{"id":"406795","messageId":"20201002003844.GA28643@danh.dev","threadId":"54189","inReplyTo":"3e18199a-1dcb-057d-4753-03d5acfcc036@gmail.com","subject":"Re: [PATCH v2 6/7] maintenance: recommended schedule in register/start","fromName":"Đoàn Trần Công Danh","fromEmail":"congdanhqx@gmail.com","sentAt":"2020-10-02T00:38:44Z","receivedAt":"2020-10-02T00:38:49Z","isPatch":true,"sender":{"key":"congdanhqx@gmail.com","avatar":"https://avatars.githubusercontent.com/u/42673067?v=4"},"body":"On 2020-10-01 16:38:48-0400, Derrick Stolee wrote:\n> diff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt\n> index 7628a6d157..52fff86844 100644\n> --- a/Documentation/git-maintenance.txt\n> +++ b/Documentation/git-maintenance.txt\n> @@ -37,6 +37,21 @@ register::\n> \t`maintenance..schedule`. The tasks that are enabled are safe\n> \tfor running in the background without disrupting foreground\n> \tprocesses.\n> ++\n> +If your repository has no `maintenance..schedule` configuration\n> +values set, then Git will use a recommended default schedule that performs\n> +background maintenance that will not interrupt foreground commands. The\n> +default schedule is as follows:\n\nI don't mind about using a default schedule (but someone else might).\nI think some distributions will be paranoia with this change and shiped\nwith disable by default in system config.\n\n> ++\n> +* `gc`: disabled.\n> +* `commit-graph`: hourly.\n> +* `prefetch`: hourly.\n\nHowever, no `prefetch` in default schedule, please.\nIIUC, this is a network operation, if someone is on the go and paying\ntheir internet based on their traffic, this will be a disaster.\n\n\n> +* `loose-objects`: daily.\n> +* `incremental-repack`: daily.\n\nAnd I would say no incremental-repack, too.\nUsers don't want to a large operation of IO on some random time of the day,\nbe it when they open their PC in the morning, or when they want to close\ntheir laptop to go home.\n\n----------(Windows rant ahead)\nI still remember those days that Windows 8 was introduced,\nBack in that days, my computer still uses the old 7200rpm HDD.\nI was super-angry that whenever Windows is started, it starts some IO\ndisk-caching, indexing that hung my computer for a good 10 minutes.\nWhile that same computer can run Windows 7 and other OS fine.\nI don't particularly care how much my computer is faster after that.\nI want my computer usable at that time, instead of wasting a good 10\nminutes on nothing.\n---------(Windows rant end)\n\nEither the users know what are they doing, or we don't do anything at\nall. Let's them do it on their free time.\n\nIOW, Please let users opt in instead of opt out of this features.\n\n-- \n\nDanh\n"},{"id":"406796","messageId":"2198ba04-e73a-cfd5-960c-909a4f3ef0d2@gmail.com","threadId":"54189","inReplyTo":"20201002003844.GA28643@danh.dev","subject":"Re: [PATCH v2 6/7] maintenance: recommended schedule in register/start","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-10-02T01:55:40Z","receivedAt":"2020-10-02T01:55:47Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 10/1/2020 8:38 PM, Đoàn Trần Công Danh wrote:\n> On 2020-10-01 16:38:48-0400, Derrick Stolee wrote:\n>> diff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt\n>> index 7628a6d157..52fff86844 100644\n>> --- a/Documentation/git-maintenance.txt\n>> +++ b/Documentation/git-maintenance.txt\n>> @@ -37,6 +37,21 @@ register::\n>> \t`maintenance..schedule`. The tasks that are enabled are safe\n>> \tfor running in the background without disrupting foreground\n>> \tprocesses.\n>> ++\n>> +If your repository has no `maintenance..schedule` configuration\n>> +values set, then Git will use a recommended default schedule that performs\n>> +background maintenance that will not interrupt foreground commands. The\n>> +default schedule is as follows:\n> \n> I don't mind about using a default schedule (but someone else might).\n> I think some distributions will be paranoia with this change and shiped\n> with disable by default in system config.\n\nIf a user wants to prevent this schedule, then they can simply change\nany one of the `.schedule` or `.enabled` configs in their --global config\nand these defaults will not be used.\n\nOf course, perhaps you are missing the fact that \"git maintenance run\n--schedule=\" is only run as a cron job if a user chose to\nstart background maintenance using \"git maintenance start\" (or \"git\nmaintenance register\" after running the 'start' subcommand in another\nrepo). So this is _not_ starting by default without some amount of\nchoosing to opt in.\n\n>> ++\n>> +* `gc`: disabled.\n>> +* `commit-graph`: hourly.\n>> +* `prefetch`: hourly.\n> \n> However, no `prefetch` in default schedule, please.\n> IIUC, this is a network operation, if someone is on the go and paying\n> their internet based on their traffic, this will be a disaster.\n\nIt _is_ a network operation. You're right that we should make it clear\nthat network operations are being run in the background if the defaults\nare being used.\n\nOf course, this is why \"git maintenance stop\" exists. Background\nmaintenance can be halted while not being in a mode where this maintenance\nis acceptable.\n\nAnd further: these defaults are optimized for desktop machines that are\nexpected to always be on and connected to a non-metered network. Laptops\nare not always on, not always connected, and sometimes are metered. Perhaps\na user should decide that they don't want to have background maintenance,\nand then they can choose to not opt-in.\n\nIf this scenario is common enough, then we could extend the \"prefetch\"\ntask to somehow detect (on some platforms) that the network connection is\nmetered, and then not do any fetches.\n\n>> +* `loose-objects`: daily.\n>> +* `incremental-repack`: daily.\n> \n> And I would say no incremental-repack, too.\n> Users don't want to a large operation of IO on some random time of the day,\n> be it when they open their PC in the morning, or when they want to close\n> their laptop to go home.\n\nBut that's exactly why incremental-repack is an improvement over\na \"random\" instance of 'git gc --auto' going over an invisible threshold.\nThe \"incremental\" nature is intended to only do a reasonable amount of work \ninstead of rewriting everything.\n \n> ----------(Windows rant ahead)\n> I still remember those days that Windows 8 was introduced,\n> Back in that days, my computer still uses the old 7200rpm HDD.\n> I was super-angry that whenever Windows is started, it starts some IO\n> disk-caching, indexing that hung my computer for a good 10 minutes.\n> While that same computer can run Windows 7 and other OS fine.\n> I don't particularly care how much my computer is faster after that.\n> I want my computer usable at that time, instead of wasting a good 10\n> minutes on nothing.\n> ---------(Windows rant end)\n> \n> Either the users know what are they doing, or we don't do anything at\n> all. Let's them do it on their free time.\n> \n> IOW, Please let users opt in instead of opt out of this features.\n\nBut users _do_ opt-in to this feature. They need to start the maintenance\nat all to have this run. We just need to give them something that actually\n\"maintains\" their repository without being incredibly expensive or\npossibly leading to data loss. That is exactly why \"incremental-repack\" is\nchosen over the \"gc\" task. If there isn't enough work to be done, then\nthis task is very cheap to do.\n\nPerhaps all of your concerns are satisfied with this reassurance that\nbackground maintenance is completely opt-in and will not be set up\nwithout a user explicitly enabling it.\n\nThanks,\n-Stolee\n\n"},{"id":"406952","messageId":"dd9237927395ef69663ab376a2da74da4654c495.1601902635.git.gitgitgadget@gmail.com","threadId":"54189","inReplyTo":"pull.724.v3.git.1601902635.gitgitgadget@gmail.com","subject":"[PATCH v3 3/7] for-each-repo: run subcommands on configured repos","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-10-05T12:57:10Z","receivedAt":"2020-10-05T13:04:12Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIt can be helpful to store a list of repositories in global or system\nconfig and then iterate Git commands on that list. Create a new builtin\nthat makes this process simple for experts. We will use this builtin to\nrun scheduled maintenance on all configured repositories in a future\nchange.\n\nThe test is very simple, but does highlight that the \"--\" argument is\noptional.\n\nSigned-off-by: Derrick Stolee \n---\n .gitignore | 1 +\n Documentation/git-for-each-repo.txt | 59 +++++++++++++++++++++++++++++\n Makefile | 1 +\n builtin.h | 1 +\n builtin/for-each-repo.c | 58 ++++++++++++++++++++++++++++\n command-list.txt | 1 +\n git.c | 1 +\n t/t0068-for-each-repo.sh | 30 +++++++++++++++\n 8 files changed, 152 insertions(+)\n create mode 100644 Documentation/git-for-each-repo.txt\n create mode 100644 builtin/for-each-repo.c\n create mode 100755 t/t0068-for-each-repo.sh\n\ndiff --git a/.gitignore b/.gitignore\nindex a5808fa30d..5eb2a2be71 100644\n--- a/.gitignore\n+++ b/.gitignore\n@@ -67,6 +67,7 @@\n /git-filter-branch\n /git-fmt-merge-msg\n /git-for-each-ref\n+/git-for-each-repo\n /git-format-patch\n /git-fsck\n /git-fsck-objects\ndiff --git a/Documentation/git-for-each-repo.txt b/Documentation/git-for-each-repo.txt\nnew file mode 100644\nindex 0000000000..94bd19da26\n--- /dev/null\n+++ b/Documentation/git-for-each-repo.txt\n@@ -0,0 +1,59 @@\n+git-for-each-repo(1)\n+====================\n+\n+NAME\n+----\n+git-for-each-repo - Run a Git command on a list of repositories\n+\n+\n+SYNOPSIS\n+--------\n+[verse]\n+'git for-each-repo' --config= [--] \n+\n+\n+DESCRIPTION\n+-----------\n+Run a Git command on a list of repositories. The arguments after the\n+known options or `--` indicator are used as the arguments for the Git\n+subprocess.\n+\n+THIS COMMAND IS EXPERIMENTAL. THE BEHAVIOR MAY CHANGE.\n+\n+For example, we could run maintenance on each of a list of repositories\n+stored in a `maintenance.repo` config variable using\n+\n+-------------\n+git for-each-repo --config=maintenance.repo maintenance run\n+-------------\n+\n+This will run `git -C maintenance run` for each value ``\n+in the multi-valued config variable `maintenance.repo`.\n+\n+\n+OPTIONS\n+-------\n+--config=::\n+\tUse the given config variable as a multi-valued list storing\n+\tabsolute path names. Iterate on that list of paths to run\n+\tthe given arguments.\n++\n+These config values are loaded from system, global, and local Git config,\n+as available. If `git for-each-repo` is run in a directory that is not a\n+Git repository, then only the system and global config is used.\n+\n+\n+SUBPROCESS BEHAVIOR\n+-------------------\n+\n+If any `git -C ` subprocess returns a non-zero exit code,\n+then the `git for-each-repo` process returns that exit code without running\n+more subprocesses.\n+\n+Each `git -C ` subprocess inherits the standard file\n+descriptors `stdin`, `stdout`, and `stderr`.\n+\n+\n+GIT\n+---\n+Part of the linkgit:git[1] suite\ndiff --git a/Makefile b/Makefile\nindex 65f8cfb236..7c588ff036 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1071,6 +1071,7 @@ BUILTIN_OBJS += builtin/fetch-pack.o\n BUILTIN_OBJS += builtin/fetch.o\n BUILTIN_OBJS += builtin/fmt-merge-msg.o\n BUILTIN_OBJS += builtin/for-each-ref.o\n+BUILTIN_OBJS += builtin/for-each-repo.o\n BUILTIN_OBJS += builtin/fsck.o\n BUILTIN_OBJS += builtin/gc.o\n BUILTIN_OBJS += builtin/get-tar-commit-id.o\ndiff --git a/builtin.h b/builtin.h\nindex 17c1c0ce49..ff7c6e5aa9 100644\n--- a/builtin.h\n+++ b/builtin.h\n@@ -150,6 +150,7 @@ int cmd_fetch(int argc, const char **argv, const char *prefix);\n int cmd_fetch_pack(int argc, const char **argv, const char *prefix);\n int cmd_fmt_merge_msg(int argc, const char **argv, const char *prefix);\n int cmd_for_each_ref(int argc, const char **argv, const char *prefix);\n+int cmd_for_each_repo(int argc, const char **argv, const char *prefix);\n int cmd_format_patch(int argc, const char **argv, const char *prefix);\n int cmd_fsck(int argc, const char **argv, const char *prefix);\n int cmd_gc(int argc, const char **argv, const char *prefix);\ndiff --git a/builtin/for-each-repo.c b/builtin/for-each-repo.c\nnew file mode 100644\nindex 0000000000..5bba623ff1\n--- /dev/null\n+++ b/builtin/for-each-repo.c\n@@ -0,0 +1,58 @@\n+#include \"cache.h\"\n+#include \"config.h\"\n+#include \"builtin.h\"\n+#include \"parse-options.h\"\n+#include \"run-command.h\"\n+#include \"string-list.h\"\n+\n+static const char * const for_each_repo_usage[] = {\n+\tN_(\"git for-each-repo --config= \"),\n+\tNULL\n+};\n+\n+static int run_command_on_repo(const char *path,\n+\t\t\t void *cbdata)\n+{\n+\tint i;\n+\tstruct child_process child = CHILD_PROCESS_INIT;\n+\tstruct strvec *args = (struct strvec *)cbdata;\n+\n+\tchild.git_cmd = 1;\n+\tstrvec_pushl(&child.args, \"-C\", path, NULL);\n+\n+\tfor (i = 0; i < args->nr; i++)\n+\t\tstrvec_push(&child.args, args->v[i]);\n+\n+\treturn run_command(&child);\n+}\n+\n+int cmd_for_each_repo(int argc, const char **argv, const char *prefix)\n+{\n+\tstatic const char *config_key = NULL;\n+\tint i, result = 0;\n+\tconst struct string_list *values;\n+\tstruct strvec args = STRVEC_INIT;\n+\n+\tconst struct option options[] = {\n+\t\tOPT_STRING(0, \"config\", &config_key, N_(\"config\"),\n+\t\t\t N_(\"config key storing a list of repository paths\")),\n+\t\tOPT_END()\n+\t};\n+\n+\targc = parse_options(argc, argv, prefix, options, for_each_repo_usage,\n+\t\t\t PARSE_OPT_STOP_AT_NON_OPTION);\n+\n+\tif (!config_key)\n+\t\tdie(_(\"missing --config=\"));\n+\n+\tfor (i = 0; i < argc; i++)\n+\t\tstrvec_push(&args, argv[i]);\n+\n+\tvalues = repo_config_get_value_multi(the_repository,\n+\t\t\t\t\t config_key);\n+\n+\tfor (i = 0; !result && i < values->nr; i++)\n+\t\tresult = run_command_on_repo(values->items[i].string, &args);\n+\n+\treturn result;\n+}\ndiff --git a/command-list.txt b/command-list.txt\nindex 0e3204e7d1..581499be82 100644\n--- a/command-list.txt\n+++ b/command-list.txt\n@@ -94,6 +94,7 @@ git-fetch-pack synchingrepositories\n git-filter-branch ancillarymanipulators\n git-fmt-merge-msg purehelpers\n git-for-each-ref plumbinginterrogators\n+git-for-each-repo plumbinginterrogators\n git-format-patch mainporcelain\n git-fsck ancillaryinterrogators complete\n git-gc mainporcelain\ndiff --git a/git.c b/git.c\nindex 24f250d29a..1cab64b5d1 100644\n--- a/git.c\n+++ b/git.c\n@@ -511,6 +511,7 @@ static struct cmd_struct commands[] = {\n \t{ \"fetch-pack\", cmd_fetch_pack, RUN_SETUP | NO_PARSEOPT },\n \t{ \"fmt-merge-msg\", cmd_fmt_merge_msg, RUN_SETUP },\n \t{ \"for-each-ref\", cmd_for_each_ref, RUN_SETUP },\n+\t{ \"for-each-repo\", cmd_for_each_repo, RUN_SETUP_GENTLY },\n \t{ \"format-patch\", cmd_format_patch, RUN_SETUP },\n \t{ \"fsck\", cmd_fsck, RUN_SETUP },\n \t{ \"fsck-objects\", cmd_fsck, RUN_SETUP },\ndiff --git a/t/t0068-for-each-repo.sh b/t/t0068-for-each-repo.sh\nnew file mode 100755\nindex 0000000000..136b4ec839\n--- /dev/null\n+++ b/t/t0068-for-each-repo.sh\n@@ -0,0 +1,30 @@\n+#!/bin/sh\n+\n+test_description='git for-each-repo builtin'\n+\n+. ./test-lib.sh\n+\n+test_expect_success 'run based on configured value' '\n+\tgit init one &&\n+\tgit init two &&\n+\tgit init three &&\n+\tgit -C two commit --allow-empty -m \"DID NOT RUN\" &&\n+\tgit config run.key \"$TRASH_DIRECTORY/one\" &&\n+\tgit config --add run.key \"$TRASH_DIRECTORY/three\" &&\n+\tgit for-each-repo --config=run.key commit --allow-empty -m \"ran\" &&\n+\tgit -C one log -1 --pretty=format:%s >message &&\n+\tgrep ran message &&\n+\tgit -C two log -1 --pretty=format:%s >message &&\n+\t! grep ran message &&\n+\tgit -C three log -1 --pretty=format:%s >message &&\n+\tgrep ran message &&\n+\tgit for-each-repo --config=run.key -- commit --allow-empty -m \"ran again\" &&\n+\tgit -C one log -1 --pretty=format:%s >message &&\n+\tgrep again message &&\n+\tgit -C two log -1 --pretty=format:%s >message &&\n+\t! grep again message &&\n+\tgit -C three log -1 --pretty=format:%s >message &&\n+\tgrep again message\n+'\n+\n+test_done\n-- \ngitgitgadget\n\n"},{"id":"406951","messageId":"dae8c04bb5523c9b63c770862a1104a0ff4aa6c4.1601902635.git.gitgitgadget@gmail.com","threadId":"54189","inReplyTo":"pull.724.v3.git.1601902635.gitgitgadget@gmail.com","subject":"[PATCH v3 2/7] maintenance: add --schedule option and config","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-10-05T12:57:09Z","receivedAt":"2020-10-05T13:04:13Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nMaintenance currently triggers when certain data-size thresholds are\nmet, such as number of pack-files or loose objects. Users may want to\nrun certain maintenance tasks based on frequency instead. For example,\na user may want to perform a 'prefetch' task every hour, or 'gc' task\nevery day. To help these users, update the 'git maintenance run' command\nto include a '--schedule=' option. The allowed frequencies\nare 'hourly', 'daily', and 'weekly'. These values are also allowed in a\nnew config value 'maintenance..schedule'.\n\nThe 'git maintenance run --schedule=' checks the '*.schedule'\nconfig value for each enabled task to see if the configured frequency is\nat least as frequent as the frequency from the '--schedule' argument. We\nuse the following order, for full clarity:\n\n\t'hourly' > 'daily' > 'weekly'\n\nUse new 'enum schedule_priority' to track these values numerically.\n\nThe following cron table would run the scheduled tasks with the correct\nfrequencies:\n\n 0 1-23 * * * git -C maintenance run --schedule=hourly\n 0 0 * * 1-6 git -C maintenance run --schedule=daily\n 0 0 * * 0 git -C maintenance run --schedule=weekly\n\nThis cron schedule will run --schedule=hourly every hour except at\nmidnight. This avoids a concurrent run with the --schedule=daily that\nruns at midnight every day except the first day of the week. This avoids\na concurrent run with the --schedule=weekly that runs at midnight on\nthe first day of the week. Since --schedule=daily also runs the\n'hourly' tasks and --schedule=weekly runs the 'hourly' and 'daily'\ntasks, we will still see all tasks run with the proper frequencies.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/config/maintenance.txt | 5 +++\n Documentation/git-maintenance.txt | 13 +++++-\n builtin/gc.c | 64 ++++++++++++++++++++++++++--\n t/t7900-maintenance.sh | 40 +++++++++++++++++\n 4 files changed, 118 insertions(+), 4 deletions(-)\n\ndiff --git a/Documentation/config/maintenance.txt b/Documentation/config/maintenance.txt\nindex 06db758172..70585564fa 100644\n--- a/Documentation/config/maintenance.txt\n+++ b/Documentation/config/maintenance.txt\n@@ -10,6 +10,11 @@ maintenance..enabled::\n \t`--task` option exists. By default, only `maintenance.gc.enabled`\n \tis true.\n \n+maintenance..schedule::\n+\tThis config option controls whether or not the given `` runs\n+\tduring a `git maintenance run --schedule=` command. The\n+\tvalue must be one of \"hourly\", \"daily\", or \"weekly\".\n+\n maintenance.commit-graph.auto::\n \tThis integer config option controls how often the `commit-graph` task\n \tshould be run as part of `git maintenance run --auto`. If zero, then\ndiff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt\nindex 3f5d8946b4..ed94f66e36 100644\n--- a/Documentation/git-maintenance.txt\n+++ b/Documentation/git-maintenance.txt\n@@ -110,7 +110,18 @@ OPTIONS\n \tonly if certain thresholds are met. For example, the `gc` task\n \truns when the number of loose objects exceeds the number stored\n \tin the `gc.auto` config setting, or when the number of pack-files\n-\texceeds the `gc.autoPackLimit` config setting.\n+\texceeds the `gc.autoPackLimit` config setting. Not compatible with\n+\tthe `--schedule` option.\n+\n+--schedule::\n+\tWhen combined with the `run` subcommand, run maintenance tasks\n+\tonly if certain time conditions are met, as specified by the\n+\t`maintenance..schedule` config value for each ``.\n+\tThis config value specifies a number of seconds since the last\n+\ttime that task ran, according to the `maintenance..lastRun`\n+\tconfig value. The tasks that are tested are those provided by\n+\tthe `--task=` option(s) or those with\n+\t`maintenance..enabled` set to true.\n \n --quiet::\n \tDo not report progress or other information over `stderr`.\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 2b99596ec8..03b24ea0db 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -703,14 +703,51 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \treturn 0;\n }\n \n-static const char * const builtin_maintenance_run_usage[] = {\n-\tN_(\"git maintenance run [--auto] [--[no-]quiet] [--task=]\"),\n+static const char *const builtin_maintenance_run_usage[] = {\n+\tN_(\"git maintenance run [--auto] [--[no-]quiet] [--task=] [--schedule]\"),\n \tNULL\n };\n \n+enum schedule_priority {\n+\tSCHEDULE_NONE = 0,\n+\tSCHEDULE_WEEKLY = 1,\n+\tSCHEDULE_DAILY = 2,\n+\tSCHEDULE_HOURLY = 3,\n+};\n+\n+static enum schedule_priority parse_schedule(const char *value)\n+{\n+\tif (!value)\n+\t\treturn SCHEDULE_NONE;\n+\tif (!strcasecmp(value, \"hourly\"))\n+\t\treturn SCHEDULE_HOURLY;\n+\tif (!strcasecmp(value, \"daily\"))\n+\t\treturn SCHEDULE_DAILY;\n+\tif (!strcasecmp(value, \"weekly\"))\n+\t\treturn SCHEDULE_WEEKLY;\n+\treturn SCHEDULE_NONE;\n+}\n+\n+static int maintenance_opt_schedule(const struct option *opt, const char *arg,\n+\t\t\t\t int unset)\n+{\n+\tenum schedule_priority *priority = opt->value;\n+\n+\tif (unset)\n+\t\tdie(_(\"--no-schedule is not allowed\"));\n+\n+\t*priority = parse_schedule(arg);\n+\n+\tif (!*priority)\n+\t\tdie(_(\"unrecognized --schedule argument '%s'\"), arg);\n+\n+\treturn 0;\n+}\n+\n struct maintenance_run_opts {\n \tint auto_flag;\n \tint quiet;\n+\tenum schedule_priority schedule;\n };\n \n /* Remember to update object flag allocation in object.h */\n@@ -1158,6 +1195,8 @@ struct maintenance_task {\n \tmaintenance_auto_fn *auto_condition;\n \tunsigned enabled:1;\n \n+\tenum schedule_priority schedule;\n+\n \t/* -1 if not selected. */\n \tint selected_order;\n };\n@@ -1253,6 +1292,9 @@ static int maintenance_run_tasks(struct maintenance_run_opts *opts)\n \t\t !tasks[i].auto_condition()))\n \t\t\tcontinue;\n \n+\t\tif (opts->schedule && tasks[i].schedule < opts->schedule)\n+\t\t\tcontinue;\n+\n \t\ttrace2_region_enter(\"maintenance\", tasks[i].name, r);\n \t\tif (tasks[i].fn(opts)) {\n \t\t\terror(_(\"task '%s' failed\"), tasks[i].name);\n@@ -1273,13 +1315,23 @@ static void initialize_task_config(void)\n \n \tfor (i = 0; i < TASK__COUNT; i++) {\n \t\tint config_value;\n+\t\tchar *config_str;\n \n-\t\tstrbuf_setlen(&config_name, 0);\n+\t\tstrbuf_reset(&config_name);\n \t\tstrbuf_addf(&config_name, \"maintenance.%s.enabled\",\n \t\t\t tasks[i].name);\n \n \t\tif (!git_config_get_bool(config_name.buf, &config_value))\n \t\t\ttasks[i].enabled = config_value;\n+\n+\t\tstrbuf_reset(&config_name);\n+\t\tstrbuf_addf(&config_name, \"maintenance.%s.schedule\",\n+\t\t\t tasks[i].name);\n+\n+\t\tif (!git_config_get_string(config_name.buf, &config_str)) {\n+\t\t\ttasks[i].schedule = parse_schedule(config_str);\n+\t\t\tfree(config_str);\n+\t\t}\n \t}\n \n \tstrbuf_release(&config_name);\n@@ -1323,6 +1375,9 @@ static int maintenance_run(int argc, const char **argv, const char *prefix)\n \tstruct option builtin_maintenance_run_options[] = {\n \t\tOPT_BOOL(0, \"auto\", &opts.auto_flag,\n \t\t\t N_(\"run tasks based on the state of the repository\")),\n+\t\tOPT_CALLBACK(0, \"schedule\", &opts.schedule, N_(\"frequency\"),\n+\t\t\t N_(\"run tasks based on frequency\"),\n+\t\t\t maintenance_opt_schedule),\n \t\tOPT_BOOL(0, \"quiet\", &opts.quiet,\n \t\t\t N_(\"do not report progress or other information over stderr\")),\n \t\tOPT_CALLBACK_F(0, \"task\", NULL, N_(\"task\"),\n@@ -1343,6 +1398,9 @@ static int maintenance_run(int argc, const char **argv, const char *prefix)\n \t\t\t builtin_maintenance_run_usage,\n \t\t\t PARSE_OPT_STOP_AT_NON_OPTION);\n \n+\tif (opts.auto_flag && opts.schedule)\n+\t\tdie(_(\"use at most one of --auto and --schedule=\"));\n+\n \tif (argc != 0)\n \t\tusage_with_options(builtin_maintenance_run_usage,\n \t\t\t\t builtin_maintenance_run_options);\ndiff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\nindex c7caaa7a55..33d73cd01c 100755\n--- a/t/t7900-maintenance.sh\n+++ b/t/t7900-maintenance.sh\n@@ -260,4 +260,44 @@ test_expect_success 'maintenance.incremental-repack.auto' '\n \ttest_subcommand git multi-pack-index write --no-progress err &&\n+\ttest_i18ngrep \"at most one\" err\n+'\n+\n+test_expect_success 'invalid --schedule value' '\n+\ttest_must_fail git maintenance run --schedule=annually 2>err &&\n+\ttest_i18ngrep \"unrecognized --schedule\" err\n+'\n+\n+test_expect_success '--schedule inheritance weekly -> daily -> hourly' '\n+\tgit config maintenance.loose-objects.enabled true &&\n+\tgit config maintenance.loose-objects.schedule hourly &&\n+\tgit config maintenance.commit-graph.enabled true &&\n+\tgit config maintenance.commit-graph.schedule daily &&\n+\tgit config maintenance.incremental-repack.enabled true &&\n+\tgit config maintenance.incremental-repack.schedule weekly &&\n+\n+\tGIT_TRACE2_EVENT=\"$(pwd)/hourly.txt\" \\\n+\t\tgit maintenance run --schedule=hourly 2>/dev/null &&\n+\ttest_subcommand git prune-packed --quiet /dev/null &&\n+\ttest_subcommand git prune-packed --quiet /dev/null &&\n+\ttest_subcommand git prune-packed --quiet \n\nThe 'git maintenance (register|start)' subcommands add the current\nrepository to the global Git config so maintenance will operate on that\nrepository. It does not specify what maintenance should occur or how\noften.\n\nIf a user sets any 'maintenance..schedule' config value, then\nthey have chosen a specific schedule for themselves and Git should\nrespect that when running 'git maintenance run --schedule='.\n\nTo make this process extremely simple for users, assume a default\nschedule when no 'maintenance..schedule' or '...enabled' config\nsettings are concretely set. This is only an in-process assumption, so\nfuture versions of Git could adjust this expected schedule.\n\nHelped-by: Martin Ågren \nSigned-off-by: Derrick Stolee \n---\n Documentation/git-maintenance.txt | 15 ++++++++\n builtin/gc.c | 58 +++++++++++++++++++++++++++++++\n t/t7900-maintenance.sh | 11 +++---\n 3 files changed, 80 insertions(+), 4 deletions(-)\n\ndiff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt\nindex 7628a6d157..52fff86844 100644\n--- a/Documentation/git-maintenance.txt\n+++ b/Documentation/git-maintenance.txt\n@@ -37,6 +37,21 @@ register::\n \t`maintenance..schedule`. The tasks that are enabled are safe\n \tfor running in the background without disrupting foreground\n \tprocesses.\n++\n+If your repository has no `maintenance..schedule` configuration\n+values set, then Git will use a recommended default schedule that performs\n+background maintenance that will not interrupt foreground commands. The\n+default schedule is as follows:\n++\n+* `gc`: disabled.\n+* `commit-graph`: hourly.\n+* `prefetch`: hourly.\n+* `loose-objects`: daily.\n+* `incremental-repack`: daily.\n++\n+`git maintenance register` will also disable foreground maintenance by\n+setting `maintenance.auto = false` in the current repository. This config\n+setting will remain after a `git maintenance unregister` command.\n \n run::\n \tRun one or more maintenance tasks. If one or more `--task` options\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex a387f46585..965690704b 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -1251,6 +1251,59 @@ static int compare_tasks_by_selection(const void *a_, const void *b_)\n \treturn b->selected_order - a->selected_order;\n }\n \n+static int has_schedule_config(void)\n+{\n+\tint i, found = 0;\n+\tstruct strbuf config_name = STRBUF_INIT;\n+\tsize_t prefix;\n+\n+\tstrbuf_addstr(&config_name, \"maintenance.\");\n+\tprefix = config_name.len;\n+\n+\tfor (i = 0; !found && i < TASK__COUNT; i++) {\n+\t\tchar *value;\n+\n+\t\tstrbuf_setlen(&config_name, prefix);\n+\t\tstrbuf_addf(&config_name, \"%s.schedule\", tasks[i].name);\n+\n+\t\tif (!git_config_get_string(config_name.buf, &value)) {\n+\t\t\tfound = 1;\n+\t\t\tFREE_AND_NULL(value);\n+\t\t}\n+\n+\t\tstrbuf_setlen(&config_name, prefix);\n+\t\tstrbuf_addf(&config_name, \"%s.enabled\", tasks[i].name);\n+\n+\t\tif (!git_config_get_string(config_name.buf, &value)) {\n+\t\t\tfound = 1;\n+\t\t\tFREE_AND_NULL(value);\n+\t\t}\n+\t}\n+\n+\tstrbuf_release(&config_name);\n+\treturn found;\n+}\n+\n+static void set_recommended_schedule(void)\n+{\n+\tif (has_schedule_config())\n+\t\treturn;\n+\n+\ttasks[TASK_GC].enabled = 0;\n+\n+\ttasks[TASK_PREFETCH].enabled = 1;\n+\ttasks[TASK_PREFETCH].schedule = SCHEDULE_HOURLY;\n+\n+\ttasks[TASK_COMMIT_GRAPH].enabled = 1;\n+\ttasks[TASK_COMMIT_GRAPH].schedule = SCHEDULE_HOURLY;\n+\n+\ttasks[TASK_LOOSE_OBJECTS].enabled = 1;\n+\ttasks[TASK_LOOSE_OBJECTS].schedule = SCHEDULE_DAILY;\n+\n+\ttasks[TASK_INCREMENTAL_REPACK].enabled = 1;\n+\ttasks[TASK_INCREMENTAL_REPACK].schedule = SCHEDULE_DAILY;\n+}\n+\n static int maintenance_run_tasks(struct maintenance_run_opts *opts)\n {\n \tint i, found_selected = 0;\n@@ -1280,6 +1333,8 @@ static int maintenance_run_tasks(struct maintenance_run_opts *opts)\n \n \tif (found_selected)\n \t\tQSORT(tasks, TASK__COUNT, compare_tasks_by_selection);\n+\telse if (opts->schedule != SCHEDULE_NONE)\n+\t\tset_recommended_schedule();\n \n \tfor (i = 0; i < TASK__COUNT; i++) {\n \t\tif (found_selected && tasks[i].selected_order < 0)\n@@ -1417,6 +1472,9 @@ static int maintenance_register(void)\n \tif (!the_repository || !the_repository->gitdir)\n \t\treturn 0;\n \n+\t/* Disable foreground maintenance */\n+\tgit_config_set(\"maintenance.auto\", \"false\");\n+\n \tconfig_get.git_cmd = 1;\n \tstrvec_pushl(&config_get.args, \"config\", \"--global\", \"--get\", \"maintenance.repo\",\n \t\t the_repository->worktree ? the_repository->worktree\ndiff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\nindex 7715e40391..7154987fd2 100755\n--- a/t/t7900-maintenance.sh\n+++ b/t/t7900-maintenance.sh\n@@ -305,11 +305,14 @@ test_expect_success 'register and unregister' '\n \tgit config --global --add maintenance.repo /existing1 &&\n \tgit config --global --add maintenance.repo /existing2 &&\n \tgit config --global --get-all maintenance.repo >before &&\n+\n \tgit maintenance register &&\n-\tgit config --global --get-all maintenance.repo >actual &&\n-\tcp before after &&\n-\tpwd >>after &&\n-\ttest_cmp after actual &&\n+\ttest_cmp_config false maintenance.auto &&\n+\tgit config --global --get-all maintenance.repo >between &&\n+\tcp before expect &&\n+\tpwd >>expect &&\n+\ttest_cmp expect between &&\n+\n \tgit maintenance unregister &&\n \tgit config --global --get-all maintenance.repo >actual &&\n \ttest_cmp before actual\n-- \ngitgitgadget\n\n"},{"id":"406954","messageId":"5194f6b1facbd14cc17eea0337c0cc397a2a51fc.1601902635.git.gitgitgadget@gmail.com","threadId":"54189","inReplyTo":"pull.724.v3.git.1601902635.gitgitgadget@gmail.com","subject":"[PATCH v3 5/7] maintenance: add start/stop subcommands","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-10-05T12:57:12Z","receivedAt":"2020-10-05T13:04:18Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nAdd new subcommands to 'git maintenance' that start or stop background\nmaintenance using 'cron', when available. This integration is as simple\nas I could make it, barring some implementation complications.\n\nThe schedule is laid out as follows:\n\n 0 1-23 * * * $cmd maintenance run --schedule=hourly\n 0 0 * * 1-6 $cmd maintenance run --schedule=daily\n 0 0 * * 0 $cmd maintenance run --schedule=weekly\n\nwhere $cmd is a properly-qualified 'git for-each-repo' execution:\n\n$cmd=$path/git --exec-path=$path for-each-repo --config=maintenance.repo\n\nwhere $path points to the location of the Git executable running 'git\nmaintenance start'. This is critical for systems with multiple versions\nof Git. Specifically, macOS has a system version at '/usr/bin/git' while\nthe version that users can install resides at '/usr/local/bin/git'\n(symlinked to '/usr/local/libexec/git-core/git'). This will also use\nyour locally-built version if you build and run this in your development\nenvironment without installing first.\n\nThis conditional schedule avoids having cron launch multiple 'git\nfor-each-repo' commands in parallel. Such parallel commands would likely\nlead to the 'hourly' and 'daily' tasks competing over the object\ndatabase lock. This could lead to to some tasks never being run! Since\nthe --schedule= argument will run all tasks with _at least_\nthe given frequency, the daily runs will also run the hourly tasks.\nSimilarly, the weekly runs will also run the daily and hourly tasks.\n\nThe GIT_TEST_CRONTAB environment variable is not intended for users to\nedit, but instead as a way to mock the 'crontab [-l]' command. This\nvariable is set in test-lib.sh to avoid a future test from accidentally\nrunning anything with the cron integration from modifying the user's\nschedule. We use GIT_TEST_CRONTAB='test-tool crontab ' in our\ntests to check how the schedule is modified in 'git maintenance\n(start|stop)' commands.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-maintenance.txt | 11 +++\n Makefile | 1 +\n builtin/gc.c | 124 ++++++++++++++++++++++++++++++\n t/helper/test-crontab.c | 35 +++++++++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/t7900-maintenance.sh | 28 +++++++\n t/test-lib.sh | 6 ++\n 8 files changed, 207 insertions(+)\n create mode 100644 t/helper/test-crontab.c\n\ndiff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt\nindex 1c59fd0cb5..7628a6d157 100644\n--- a/Documentation/git-maintenance.txt\n+++ b/Documentation/git-maintenance.txt\n@@ -45,6 +45,17 @@ run::\n \tconfig options are true. By default, only `maintenance.gc.enabled`\n \tis true.\n \n+start::\n+\tStart running maintenance on the current repository. This performs\n+\tthe same config updates as the `register` subcommand, then updates\n+\tthe background scheduler to run `git maintenance run --scheduled`\n+\ton an hourly basis.\n+\n+stop::\n+\tHalt the background maintenance schedule. The current repository\n+\tis not removed from the list of maintained repositories, in case\n+\tthe background maintenance is restarted later.\n+\n unregister::\n \tRemove the current repository from background maintenance. This\n \tonly removes the repository from the configured list. It does not\ndiff --git a/Makefile b/Makefile\nindex 7c588ff036..c39b39bd7d 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -690,6 +690,7 @@ TEST_BUILTINS_OBJS += test-advise.o\n TEST_BUILTINS_OBJS += test-bloom.o\n TEST_BUILTINS_OBJS += test-chmtime.o\n TEST_BUILTINS_OBJS += test-config.o\n+TEST_BUILTINS_OBJS += test-crontab.o\n TEST_BUILTINS_OBJS += test-ctype.o\n TEST_BUILTINS_OBJS += test-date.o\n TEST_BUILTINS_OBJS += test-delta.o\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex edf1d35ce5..a387f46585 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -31,6 +31,7 @@\n #include \"refs.h\"\n #include \"remote.h\"\n #include \"object-store.h\"\n+#include \"exec-cmd.h\"\n \n #define FAILED_RUN \"failed to run %s\"\n \n@@ -1456,6 +1457,125 @@ static int maintenance_unregister(void)\n \treturn run_command(&config_unset);\n }\n \n+#define BEGIN_LINE \"# BEGIN GIT MAINTENANCE SCHEDULE\"\n+#define END_LINE \"# END GIT MAINTENANCE SCHEDULE\"\n+\n+static int update_background_schedule(int run_maintenance)\n+{\n+\tint result = 0;\n+\tint in_old_region = 0;\n+\tstruct child_process crontab_list = CHILD_PROCESS_INIT;\n+\tstruct child_process crontab_edit = CHILD_PROCESS_INIT;\n+\tFILE *cron_list, *cron_in;\n+\tconst char *crontab_name;\n+\tstruct strbuf line = STRBUF_INIT;\n+\tstruct lock_file lk;\n+\tchar *lock_path = xstrfmt(\"%s/schedule\", the_repository->objects->odb->path);\n+\n+\tif (hold_lock_file_for_update(&lk, lock_path, LOCK_NO_DEREF) < 0)\n+\t\treturn error(_(\"another process is scheduling background maintenance\"));\n+\n+\tcrontab_name = getenv(\"GIT_TEST_CRONTAB\");\n+\tif (!crontab_name)\n+\t\tcrontab_name = \"crontab\";\n+\n+\tstrvec_split(&crontab_list.args, crontab_name);\n+\tstrvec_push(&crontab_list.args, \"-l\");\n+\tcrontab_list.in = -1;\n+\tcrontab_list.out = dup(lk.tempfile->fd);\n+\tcrontab_list.git_cmd = 0;\n+\n+\tif (start_command(&crontab_list)) {\n+\t\tresult = error(_(\"failed to run 'crontab -l'; your system might not support 'cron'\"));\n+\t\tgoto cleanup;\n+\t}\n+\n+\t/* Ignore exit code, as an empty crontab will return error. */\n+\tfinish_command(&crontab_list);\n+\n+\t/*\n+\t * Read from the .lock file, filtering out the old\n+\t * schedule while appending the new schedule.\n+\t */\n+\tcron_list = fdopen(lk.tempfile->fd, \"r\");\n+\trewind(cron_list);\n+\n+\tstrvec_split(&crontab_edit.args, crontab_name);\n+\tcrontab_edit.in = -1;\n+\tcrontab_edit.git_cmd = 0;\n+\n+\tif (start_command(&crontab_edit)) {\n+\t\tresult = error(_(\"failed to run 'crontab'; your system might not support 'cron'\"));\n+\t\tgoto cleanup;\n+\t}\n+\n+\tcron_in = fdopen(crontab_edit.in, \"w\");\n+\tif (!cron_in) {\n+\t\tresult = error(_(\"failed to open stdin of 'crontab'\"));\n+\t\tgoto done_editing;\n+\t}\n+\n+\twhile (!strbuf_getline_lf(&line, cron_list)) {\n+\t\tif (!in_old_region && !strcmp(line.buf, BEGIN_LINE))\n+\t\t\tin_old_region = 1;\n+\t\tif (in_old_region)\n+\t\t\tcontinue;\n+\t\tfprintf(cron_in, \"%s\\n\", line.buf);\n+\t\tif (in_old_region && !strcmp(line.buf, END_LINE))\n+\t\t\tin_old_region = 0;\n+\t}\n+\n+\tif (run_maintenance) {\n+\t\tstruct strbuf line_format = STRBUF_INIT;\n+\t\tconst char *exec_path = git_exec_path();\n+\n+\t\tfprintf(cron_in, \"%s\\n\", BEGIN_LINE);\n+\t\tfprintf(cron_in,\n+\t\t\t\"# The following schedule was created by Git\\n\");\n+\t\tfprintf(cron_in, \"# Any edits made in this region might be\\n\");\n+\t\tfprintf(cron_in,\n+\t\t\t\"# replaced in the future by a Git command.\\n\\n\");\n+\n+\t\tstrbuf_addf(&line_format,\n+\t\t\t \"%%s %%s * * %%s \\\"%s/git\\\" --exec-path=\\\"%s\\\" for-each-repo --config=maintenance.repo maintenance run --schedule=%%s\\n\",\n+\t\t\t exec_path, exec_path);\n+\t\tfprintf(cron_in, line_format.buf, \"0\", \"1-23\", \"*\", \"hourly\");\n+\t\tfprintf(cron_in, line_format.buf, \"0\", \"0\", \"1-6\", \"daily\");\n+\t\tfprintf(cron_in, line_format.buf, \"0\", \"0\", \"0\", \"weekly\");\n+\t\tstrbuf_release(&line_format);\n+\n+\t\tfprintf(cron_in, \"\\n%s\\n\", END_LINE);\n+\t}\n+\n+\tfflush(cron_in);\n+\tfclose(cron_in);\n+\tclose(crontab_edit.in);\n+\n+done_editing:\n+\tif (finish_command(&crontab_edit)) {\n+\t\tresult = error(_(\"'crontab' died\"));\n+\t\tgoto cleanup;\n+\t}\n+\tfclose(cron_list);\n+\n+cleanup:\n+\trollback_lock_file(&lk);\n+\treturn result;\n+}\n+\n+static int maintenance_start(void)\n+{\n+\tif (maintenance_register())\n+\t\twarning(_(\"failed to add repo to global config\"));\n+\n+\treturn update_background_schedule(1);\n+}\n+\n+static int maintenance_stop(void)\n+{\n+\treturn update_background_schedule(0);\n+}\n+\n static const char builtin_maintenance_usage[] =\tN_(\"git maintenance []\");\n \n int cmd_maintenance(int argc, const char **argv, const char *prefix)\n@@ -1466,6 +1586,10 @@ int cmd_maintenance(int argc, const char **argv, const char *prefix)\n \n \tif (!strcmp(argv[1], \"run\"))\n \t\treturn maintenance_run(argc - 1, argv + 1, prefix);\n+\tif (!strcmp(argv[1], \"start\"))\n+\t\treturn maintenance_start();\n+\tif (!strcmp(argv[1], \"stop\"))\n+\t\treturn maintenance_stop();\n \tif (!strcmp(argv[1], \"register\"))\n \t\treturn maintenance_register();\n \tif (!strcmp(argv[1], \"unregister\"))\ndiff --git a/t/helper/test-crontab.c b/t/helper/test-crontab.c\nnew file mode 100644\nindex 0000000000..e7c0137a47\n--- /dev/null\n+++ b/t/helper/test-crontab.c\n@@ -0,0 +1,35 @@\n+#include \"test-tool.h\"\n+#include \"cache.h\"\n+\n+/*\n+ * Usage: test-tool cron [-l]\n+ *\n+ * If -l is specified, then write the contents of to stdout.\n+ * Otherwise, write from stdin into .\n+ */\n+int cmd__crontab(int argc, const char **argv)\n+{\n+\tint a;\n+\tFILE *from, *to;\n+\n+\tif (argc == 3 && !strcmp(argv[2], \"-l\")) {\n+\t\tfrom = fopen(argv[1], \"r\");\n+\t\tif (!from)\n+\t\t\treturn 0;\n+\t\tto = stdout;\n+\t} else if (argc == 2) {\n+\t\tfrom = stdin;\n+\t\tto = fopen(argv[1], \"w\");\n+\t} else\n+\t\treturn error(\"unknown arguments\");\n+\n+\twhile ((a = fgetc(from)) != EOF)\n+\t\tfputc(a, to);\n+\n+\tif (argc == 3)\n+\t\tfclose(from);\n+\telse\n+\t\tfclose(to);\n+\n+\treturn 0;\n+}\ndiff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\nindex 590b2efca7..432b49d948 100644\n--- a/t/helper/test-tool.c\n+++ b/t/helper/test-tool.c\n@@ -18,6 +18,7 @@ static struct test_cmd cmds[] = {\n \t{ \"bloom\", cmd__bloom },\n \t{ \"chmtime\", cmd__chmtime },\n \t{ \"config\", cmd__config },\n+\t{ \"crontab\", cmd__crontab },\n \t{ \"ctype\", cmd__ctype },\n \t{ \"date\", cmd__date },\n \t{ \"delta\", cmd__delta },\ndiff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\nindex ddc8e990e9..7c3281e071 100644\n--- a/t/helper/test-tool.h\n+++ b/t/helper/test-tool.h\n@@ -8,6 +8,7 @@ int cmd__advise_if_enabled(int argc, const char **argv);\n int cmd__bloom(int argc, const char **argv);\n int cmd__chmtime(int argc, const char **argv);\n int cmd__config(int argc, const char **argv);\n+int cmd__crontab(int argc, const char **argv);\n int cmd__ctype(int argc, const char **argv);\n int cmd__date(int argc, const char **argv);\n int cmd__delta(int argc, const char **argv);\ndiff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\nindex 8f383d01d9..7715e40391 100755\n--- a/t/t7900-maintenance.sh\n+++ b/t/t7900-maintenance.sh\n@@ -315,4 +315,32 @@ test_expect_success 'register and unregister' '\n \ttest_cmp before actual\n '\n \n+test_expect_success 'start from empty cron table' '\n+\tGIT_TEST_CRONTAB=\"test-tool crontab cron.txt\" git maintenance start &&\n+\n+\t# start registers the repo\n+\tgit config --get --global maintenance.repo \"$(pwd)\" &&\n+\n+\tgrep \"for-each-repo --config=maintenance.repo maintenance run --schedule=daily\" cron.txt &&\n+\tgrep \"for-each-repo --config=maintenance.repo maintenance run --schedule=hourly\" cron.txt &&\n+\tgrep \"for-each-repo --config=maintenance.repo maintenance run --schedule=weekly\" cron.txt\n+'\n+\n+test_expect_success 'stop from existing schedule' '\n+\tGIT_TEST_CRONTAB=\"test-tool crontab cron.txt\" git maintenance stop &&\n+\n+\t# stop does not unregister the repo\n+\tgit config --get --global maintenance.repo \"$(pwd)\" &&\n+\n+\t# Operation is idempotent\n+\tGIT_TEST_CRONTAB=\"test-tool crontab cron.txt\" git maintenance stop &&\n+\ttest_must_be_empty cron.txt\n+'\n+\n+test_expect_success 'start preserves existing schedule' '\n+\techo \"Important information!\" >cron.txt &&\n+\tGIT_TEST_CRONTAB=\"test-tool crontab cron.txt\" git maintenance start &&\n+\tgrep \"Important information!\" cron.txt\n+'\n+\n test_done\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex ef31f40037..4a60d1ed76 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -1702,3 +1702,9 @@ test_lazy_prereq SHA1 '\n test_lazy_prereq REBASE_P '\n \ttest -z \"$GIT_TEST_SKIP_REBASE_P\"\n '\n+\n+# Ensure that no test accidentally triggers a Git command\n+# that runs 'crontab', affecting a user's cron schedule.\n+# Tests that verify the cron integration must set this locally\n+# to avoid errors.\n+GIT_TEST_CRONTAB=\"exit 1\"\n-- \ngitgitgadget\n\n"},{"id":"406955","messageId":"pull.724.v3.git.1601902635.gitgitgadget@gmail.com","threadId":"54189","inReplyTo":"pull.724.v2.git.1599846560.gitgitgadget@gmail.com","subject":"[PATCH v3 0/7] Maintenance III: Background maintenance","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-10-05T12:57:07Z","receivedAt":"2020-10-05T13:04:24Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"This is based on ds/maintenance-part-2 and replaces the RFC from [1].\n\n[1] \nhttps://lore.kernel.org/git/pull.680.v3.git.1598629517.gitgitgadget@gmail.com/\n\nThis series introduces background maintenance to Git, through an integration\nwith cron and crontab.\n\nSome preliminary work is done to allow a new --schedule option that tells\nthe command which tasks to run based on a maintenance..schedule config\noption. The timing is not enforced by Git, but instead is expected to be\nprovided as a hint from a cron schedule. The options are \"hourly\", \"daily\",\nand \"weekly\".\n\nA new for-each-repo builtin runs Git commands on every repo in a given list.\nCurrently, the list is stored as a config setting, allowing a new \nmaintenance.repos config list to store the repositories registered for\nbackground maintenance. Others may want to add a --file= option for\ntheir own workflows, but I focused on making this as simple as possible for\nnow.\n\nThe updates to the git maintenance builtin include new register/unregister \nsubcommands and start/stop subcommands. The register subcommand initializes\nthe config while the start subcommand does everything register does plus \nupdate the cron table. The unregister and stop commands reverse this\nprocess.\n\nA troubleshooting guide is added to Documentation/git-maintenance.txt to\nadvise expert users who choose to create custom cron schedules.\n\nThe very last patch is entirely optional. It sets a recommended schedule\nbased on my own experience with very large repositories. I'm open to other\nsuggestions, but these are ones that I think work well and don't cause a\n\"rewrite the world\" scenario like running nightly 'gc' would do.\n\nI've been testing this scenario on my macOS laptop and Linux desktop. I have\nmodified my cron task to provide logging via trace2 so I can see what's\nhappening. A future direction here would be to add some maintenance logs to\nthe repository so we can track what is happening and diagnose whether the\nmaintenance strategy is working on real repos.\n\nNote: git maintenance (start|stop) only works on machines with cron by\ndesign. The proper thing to do on Windows will come later. Perhaps this\ncommand should be marked as unavailable on Windows somehow, or at least a\nbetter error than \"cron may not be available on your system\". I did find\nthat that message is helpful sometimes: macOS worker agents for CI builds\ntypically do not have cron available.\n\nUpdates in v3:\n\n * Instead of writing config upon \"register\" or \"start\", simply create an\n in-memory default schedule when no .schedule or .enabled configs are\n present. Thanks, Martin! This causes patch 6 to look so different that\n the range-diff considers it a dropped-and-added patch instead of showing\n a diff.\n * There are some context lines that changed because this is rebased onto a\n recent version of ds/maintenance-part-2.\n\nUpdates in v2:\n\n * Fixed the char/int issue in test-tool crontab, and a typo.\n * Updated commit message and patch noise in PATCH 2\n * This should fix the test failures, allowing this to be picked up in\n 'seen'.\n\nDerrick Stolee (7):\n maintenance: optionally skip --auto process\n maintenance: add --schedule option and config\n for-each-repo: run subcommands on configured repos\n maintenance: add [un]register subcommands\n maintenance: add start/stop subcommands\n maintenance: use default schedule if not configured\n maintenance: add troubleshooting guide to docs\n\n .gitignore | 1 +\n Documentation/config/maintenance.txt | 10 +\n Documentation/git-for-each-repo.txt | 59 ++++++\n Documentation/git-maintenance.txt | 97 ++++++++-\n Makefile | 2 +\n builtin.h | 1 +\n builtin/for-each-repo.c | 58 ++++++\n builtin/gc.c | 301 ++++++++++++++++++++++++++-\n command-list.txt | 1 +\n git.c | 1 +\n run-command.c | 6 +\n t/helper/test-crontab.c | 35 ++++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/t0068-for-each-repo.sh | 30 +++\n t/t7900-maintenance.sh | 101 ++++++++-\n t/test-lib.sh | 6 +\n 17 files changed, 705 insertions(+), 6 deletions(-)\n create mode 100644 Documentation/git-for-each-repo.txt\n create mode 100644 builtin/for-each-repo.c\n create mode 100644 t/helper/test-crontab.c\n create mode 100755 t/t0068-for-each-repo.sh\n\n\nbase-commit: e841a79a131d8ce491cf04d0ca3e24f139a10b82\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-724%2Fderrickstolee%2Fmaintenance%2Fscheduled-v3\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-724/derrickstolee/maintenance/scheduled-v3\nPull-Request: https://github.com/gitgitgadget/git/pull/724\n\nRange-diff vs v2:\n\n 1: b21cd68c90 = 1: 02e7286dba maintenance: optionally skip --auto process\n 2: e2d14d66d4 ! 2: dae8c04bb5 maintenance: add --schedule option and config\n @@ builtin/gc.c: static int maintenance_run(int argc, const char **argv, const char\n \n ## t/t7900-maintenance.sh ##\n @@ t/t7900-maintenance.sh: test_expect_success 'maintenance.incremental-repack.auto' '\n - \tdone\n + \ttest_subcommand git multi-pack-index write --no-progress \" err &&\n \ttest_expect_code 128 git maintenance barf 2>err &&\n - \ttest_i18ngrep \"invalid subcommand: barf\" err\n - '\n + \ttest_i18ngrep \"invalid subcommand: barf\" err &&\n + \ttest_expect_code 129 git maintenance 2>err &&\n @@ t/t7900-maintenance.sh: test_expect_success '--schedule inheritance weekly -> daily -> hourly' '\n \ttest_subcommand git multi-pack-index write --no-progress \n \n ## Commit message ##\n - maintenance: recommended schedule in register/start\n + maintenance: use default schedule if not configured\n \n The 'git maintenance (register|start)' subcommands add the current\n repository to the global Git config so maintenance will operate on that\n @@ Commit message\n \n If a user sets any 'maintenance..schedule' config value, then\n they have chosen a specific schedule for themselves and Git should\n - respect that.\n + respect that when running 'git maintenance run --schedule='.\n \n - However, in an effort to recommend a good schedule for repositories of\n - all sizes, set new config values for recommended tasks that are safe to\n - run in the background while users run foreground Git commands. These\n - commands are generally everything but the 'gc' task.\n + To make this process extremely simple for users, assume a default\n + schedule when no 'maintenance..schedule' or '...enabled' config\n + settings are concretely set. This is only an in-process assumption, so\n + future versions of Git could adjust this expected schedule.\n \n + Helped-by: Martin Ågren \n Signed-off-by: Derrick Stolee \n \n ## Documentation/git-maintenance.txt ##\n @@ Documentation/git-maintenance.txt: register::\n \tfor running in the background without disrupting foreground\n \tprocesses.\n ++\n -+If your repository has no 'maintenance..schedule' configuration\n -+values set, then Git will set configuration values to some recommended\n -+settings. These settings disable foreground maintenance while performing\n -+maintenance tasks in the background that will not interrupt foreground Git\n -+operations.\n ++If your repository has no `maintenance..schedule` configuration\n ++values set, then Git will use a recommended default schedule that performs\n ++background maintenance that will not interrupt foreground commands. The\n ++default schedule is as follows:\n +++\n ++* `gc`: disabled.\n ++* `commit-graph`: hourly.\n ++* `prefetch`: hourly.\n ++* `loose-objects`: daily.\n ++* `incremental-repack`: daily.\n +++\n ++`git maintenance register` will also disable foreground maintenance by\n ++setting `maintenance.auto = false` in the current repository. This config\n ++setting will remain after a `git maintenance unregister` command.\n \n run::\n \tRun one or more maintenance tasks. If one or more `--task` options\n \n ## builtin/gc.c ##\n -@@ builtin/gc.c: static int maintenance_run(int argc, const char **argv, const char *prefix)\n - \treturn maintenance_run_tasks(&opts);\n +@@ builtin/gc.c: static int compare_tasks_by_selection(const void *a_, const void *b_)\n + \treturn b->selected_order - a->selected_order;\n }\n \n +static int has_schedule_config(void)\n @@ builtin/gc.c: static int maintenance_run(int argc, const char **argv, const char\n +\t\t\tfound = 1;\n +\t\t\tFREE_AND_NULL(value);\n +\t\t}\n ++\n ++\t\tstrbuf_setlen(&config_name, prefix);\n ++\t\tstrbuf_addf(&config_name, \"%s.enabled\", tasks[i].name);\n ++\n ++\t\tif (!git_config_get_string(config_name.buf, &value)) {\n ++\t\t\tfound = 1;\n ++\t\t\tFREE_AND_NULL(value);\n ++\t\t}\n +\t}\n +\n +\tstrbuf_release(&config_name);\n @@ builtin/gc.c: static int maintenance_run(int argc, const char **argv, const char\n +\n +static void set_recommended_schedule(void)\n +{\n -+\tgit_config_set(\"maintenance.auto\", \"false\");\n -+\tgit_config_set(\"maintenance.gc.enabled\", \"false\");\n ++\tif (has_schedule_config())\n ++\t\treturn;\n ++\n ++\ttasks[TASK_GC].enabled = 0;\n +\n -+\tgit_config_set(\"maintenance.prefetch.enabled\", \"true\");\n -+\tgit_config_set(\"maintenance.prefetch.schedule\", \"hourly\");\n ++\ttasks[TASK_PREFETCH].enabled = 1;\n ++\ttasks[TASK_PREFETCH].schedule = SCHEDULE_HOURLY;\n +\n -+\tgit_config_set(\"maintenance.commit-graph.enabled\", \"true\");\n -+\tgit_config_set(\"maintenance.commit-graph.schedule\", \"hourly\");\n ++\ttasks[TASK_COMMIT_GRAPH].enabled = 1;\n ++\ttasks[TASK_COMMIT_GRAPH].schedule = SCHEDULE_HOURLY;\n +\n -+\tgit_config_set(\"maintenance.loose-objects.enabled\", \"true\");\n -+\tgit_config_set(\"maintenance.loose-objects.schedule\", \"daily\");\n ++\ttasks[TASK_LOOSE_OBJECTS].enabled = 1;\n ++\ttasks[TASK_LOOSE_OBJECTS].schedule = SCHEDULE_DAILY;\n +\n -+\tgit_config_set(\"maintenance.incremental-repack.enabled\", \"true\");\n -+\tgit_config_set(\"maintenance.incremental-repack.schedule\", \"daily\");\n ++\ttasks[TASK_INCREMENTAL_REPACK].enabled = 1;\n ++\ttasks[TASK_INCREMENTAL_REPACK].schedule = SCHEDULE_DAILY;\n +}\n +\n - static int maintenance_register(void)\n + static int maintenance_run_tasks(struct maintenance_run_opts *opts)\n {\n - \tstruct child_process config_set = CHILD_PROCESS_INIT;\n + \tint i, found_selected = 0;\n +@@ builtin/gc.c: static int maintenance_run_tasks(struct maintenance_run_opts *opts)\n + \n + \tif (found_selected)\n + \t\tQSORT(tasks, TASK__COUNT, compare_tasks_by_selection);\n ++\telse if (opts->schedule != SCHEDULE_NONE)\n ++\t\tset_recommended_schedule();\n + \n + \tfor (i = 0; i < TASK__COUNT; i++) {\n + \t\tif (found_selected && tasks[i].selected_order < 0)\n @@ builtin/gc.c: static int maintenance_register(void)\n \tif (!the_repository || !the_repository->gitdir)\n \t\treturn 0;\n \n -+\tif (!has_schedule_config())\n -+\t\tset_recommended_schedule();\n ++\t/* Disable foreground maintenance */\n ++\tgit_config_set(\"maintenance.auto\", \"false\");\n +\n \tconfig_get.git_cmd = 1;\n \tstrvec_pushl(&config_get.args, \"config\", \"--global\", \"--get\", \"maintenance.repo\",\n @@ t/t7900-maintenance.sh: test_expect_success 'register and unregister' '\n \tgit config --global --add maintenance.repo /existing2 &&\n \tgit config --global --get-all maintenance.repo >before &&\n +\n -+\t# We still have maintenance..schedule config set,\n -+\t# so this does not update the local schedule\n -+\tgit maintenance register &&\n -+\ttest_must_fail git config maintenance.auto &&\n -+\n -+\t# Clear previous maintenance..schedule values\n -+\tfor task in loose-objects commit-graph incremental-repack\n -+\tdo\n -+\t\tgit config --unset maintenance.$task.schedule || return 1\n -+\tdone &&\n \tgit maintenance register &&\n +-\tgit config --global --get-all maintenance.repo >actual &&\n +-\tcp before after &&\n +-\tpwd >>after &&\n +-\ttest_cmp after actual &&\n +\ttest_cmp_config false maintenance.auto &&\n -+\ttest_cmp_config false maintenance.gc.enabled &&\n -+\ttest_cmp_config true maintenance.prefetch.enabled &&\n -+\ttest_cmp_config hourly maintenance.commit-graph.schedule &&\n -+\ttest_cmp_config daily maintenance.incremental-repack.schedule &&\n ++\tgit config --global --get-all maintenance.repo >between &&\n ++\tcp before expect &&\n ++\tpwd >>expect &&\n ++\ttest_cmp expect between &&\n ++\n + \tgit maintenance unregister &&\n \tgit config --global --get-all maintenance.repo >actual &&\n - \tcp before after &&\n - \tpwd >>after &&\n + \ttest_cmp before actual\n 7: 2344eff4ba = 7: 8e42ff44ce maintenance: add troubleshooting guide to docs\n\n-- \ngitgitgadget\n"},{"id":"406956","messageId":"922b984c8aa4217a2f60307513dfa43f16475a00.1601902635.git.gitgitgadget@gmail.com","threadId":"54189","inReplyTo":"pull.724.v3.git.1601902635.gitgitgadget@gmail.com","subject":"[PATCH v3 4/7] maintenance: add [un]register subcommands","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-10-05T12:57:11Z","receivedAt":"2020-10-05T13:04:28Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIn preparation for launching background maintenance from the 'git\nmaintenance' builtin, create register/unregister subcommands. These\ncommands update the new 'maintenance.repos' config option in the global\nconfig so the background maintenance job knows which repositories to\nmaintain.\n\nThese commands allow users to add a repository to the background\nmaintenance list without disrupting the actual maintenance mechanism.\n\nFor example, a user can run 'git maintenance register' when no\nbackground maintenance is running and it will not start the background\nmaintenance. A later update to start running background maintenance will\nthen pick up this repository automatically.\n\nThe opposite example is that a user can run 'git maintenance unregister'\nto remove the current repository from background maintenance without\nhalting maintenance for other repositories.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-maintenance.txt | 14 ++++++++\n builtin/gc.c | 55 ++++++++++++++++++++++++++++++-\n t/t7900-maintenance.sh | 17 +++++++++-\n 3 files changed, 84 insertions(+), 2 deletions(-)\n\ndiff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt\nindex ed94f66e36..1c59fd0cb5 100644\n--- a/Documentation/git-maintenance.txt\n+++ b/Documentation/git-maintenance.txt\n@@ -29,6 +29,15 @@ Git repository.\n SUBCOMMANDS\n -----------\n \n+register::\n+\tInitialize Git config values so any scheduled maintenance will\n+\tstart running on this repository. This adds the repository to the\n+\t`maintenance.repo` config variable in the current user's global\n+\tconfig and enables some recommended configuration values for\n+\t`maintenance..schedule`. The tasks that are enabled are safe\n+\tfor running in the background without disrupting foreground\n+\tprocesses.\n+\n run::\n \tRun one or more maintenance tasks. If one or more `--task` options\n \tare specified, then those tasks are run in that order. Otherwise,\n@@ -36,6 +45,11 @@ run::\n \tconfig options are true. By default, only `maintenance.gc.enabled`\n \tis true.\n \n+unregister::\n+\tRemove the current repository from background maintenance. This\n+\tonly removes the repository from the configured list. It does not\n+\tstop the background maintenance processes from running.\n+\n TASKS\n -----\n \ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 03b24ea0db..edf1d35ce5 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -1407,7 +1407,56 @@ static int maintenance_run(int argc, const char **argv, const char *prefix)\n \treturn maintenance_run_tasks(&opts);\n }\n \n-static const char builtin_maintenance_usage[] = N_(\"git maintenance run []\");\n+static int maintenance_register(void)\n+{\n+\tstruct child_process config_set = CHILD_PROCESS_INIT;\n+\tstruct child_process config_get = CHILD_PROCESS_INIT;\n+\n+\t/* There is no current repository, so skip registering it */\n+\tif (!the_repository || !the_repository->gitdir)\n+\t\treturn 0;\n+\n+\tconfig_get.git_cmd = 1;\n+\tstrvec_pushl(&config_get.args, \"config\", \"--global\", \"--get\", \"maintenance.repo\",\n+\t\t the_repository->worktree ? the_repository->worktree\n+\t\t\t\t\t : the_repository->gitdir,\n+\t\t\t NULL);\n+\tconfig_get.out = -1;\n+\n+\tif (start_command(&config_get))\n+\t\treturn error(_(\"failed to run 'git config'\"));\n+\n+\t/* We already have this value in our config! */\n+\tif (!finish_command(&config_get))\n+\t\treturn 0;\n+\n+\tconfig_set.git_cmd = 1;\n+\tstrvec_pushl(&config_set.args, \"config\", \"--add\", \"--global\", \"maintenance.repo\",\n+\t\t the_repository->worktree ? the_repository->worktree\n+\t\t\t\t\t : the_repository->gitdir,\n+\t\t NULL);\n+\n+\treturn run_command(&config_set);\n+}\n+\n+static int maintenance_unregister(void)\n+{\n+\tstruct child_process config_unset = CHILD_PROCESS_INIT;\n+\n+\tif (!the_repository || !the_repository->gitdir)\n+\t\treturn error(_(\"no current repository to unregister\"));\n+\n+\tconfig_unset.git_cmd = 1;\n+\tstrvec_pushl(&config_unset.args, \"config\", \"--global\", \"--unset\",\n+\t\t \"maintenance.repo\",\n+\t\t the_repository->worktree ? the_repository->worktree\n+\t\t\t\t\t : the_repository->gitdir,\n+\t\t NULL);\n+\n+\treturn run_command(&config_unset);\n+}\n+\n+static const char builtin_maintenance_usage[] =\tN_(\"git maintenance []\");\n \n int cmd_maintenance(int argc, const char **argv, const char *prefix)\n {\n@@ -1417,6 +1466,10 @@ int cmd_maintenance(int argc, const char **argv, const char *prefix)\n \n \tif (!strcmp(argv[1], \"run\"))\n \t\treturn maintenance_run(argc - 1, argv + 1, prefix);\n+\tif (!strcmp(argv[1], \"register\"))\n+\t\treturn maintenance_register();\n+\tif (!strcmp(argv[1], \"unregister\"))\n+\t\treturn maintenance_unregister();\n \n \tdie(_(\"invalid subcommand: %s\"), argv[1]);\n }\ndiff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\nindex 33d73cd01c..8f383d01d9 100755\n--- a/t/t7900-maintenance.sh\n+++ b/t/t7900-maintenance.sh\n@@ -9,7 +9,7 @@ GIT_TEST_MULTI_PACK_INDEX=0\n \n test_expect_success 'help text' '\n \ttest_expect_code 129 git maintenance -h 2>err &&\n-\ttest_i18ngrep \"usage: git maintenance run\" err &&\n+\ttest_i18ngrep \"usage: git maintenance \" err &&\n \ttest_expect_code 128 git maintenance barf 2>err &&\n \ttest_i18ngrep \"invalid subcommand: barf\" err &&\n \ttest_expect_code 129 git maintenance 2>err &&\n@@ -300,4 +300,19 @@ test_expect_success '--schedule inheritance weekly -> daily -> hourly' '\n \ttest_subcommand git multi-pack-index write --no-progress before &&\n+\tgit maintenance register &&\n+\tgit config --global --get-all maintenance.repo >actual &&\n+\tcp before after &&\n+\tpwd >>after &&\n+\ttest_cmp after actual &&\n+\tgit maintenance unregister &&\n+\tgit config --global --get-all maintenance.repo >actual &&\n+\ttest_cmp before actual\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"406957","messageId":"8e42ff44ce9b28a070648cf760f94e2543cf0812.1601902635.git.gitgitgadget@gmail.com","threadId":"54189","inReplyTo":"pull.724.v3.git.1601902635.gitgitgadget@gmail.com","subject":"[PATCH v3 7/7] maintenance: add troubleshooting guide to docs","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-10-05T12:57:14Z","receivedAt":"2020-10-05T13:04:33Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe 'git maintenance run' subcommand takes a lock on the object database\nto prevent concurrent processes from competing for resources. This is an\nimportant safety measure to prevent possible repository corruption and\ndata loss.\n\nThis feature can lead to confusing behavior if a user is not aware of\nit. Add a TROUBLESHOOTING section to the 'git maintenance' builtin\ndocumentation that discusses these tradeoffs. The short version of this\nsection is that Git will not corrupt your repository, but if the list of\nscheduled tasks takes longer than an hour then some scheduled tasks may\nbe dropped due to this object database collision. For example, a\nlong-running \"daily\" task at midnight might prevent an \"hourly\" task\nfrom running at 1AM.\n\nThe opposite is also possible, but less likely as long as the \"hourly\"\ntasks are much faster than the \"daily\" and \"weekly\" tasks.\n\nHelped-by: Junio C Hamano \nSigned-off-by: Derrick Stolee \n---\n Documentation/git-maintenance.txt | 44 +++++++++++++++++++++++++++++++\n 1 file changed, 44 insertions(+)\n\ndiff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt\nindex 52fff86844..738a4c7ebd 100644\n--- a/Documentation/git-maintenance.txt\n+++ b/Documentation/git-maintenance.txt\n@@ -173,6 +173,50 @@ OPTIONS\n \t`maintenance..enabled` configured as `true` are considered.\n \tSee the 'TASKS' section for the list of accepted `` values.\n \n+\n+TROUBLESHOOTING\n+---------------\n+The `git maintenance` command is designed to simplify the repository\n+maintenance patterns while minimizing user wait time during Git commands.\n+A variety of configuration options are available to allow customizing this\n+process. The default maintenance options focus on operations that complete\n+quickly, even on large repositories.\n+\n+Users may find some cases where scheduled maintenance tasks do not run as\n+frequently as intended. Each `git maintenance run` command takes a lock on\n+the repository's object database, and this prevents other concurrent\n+`git maintenance run` commands from running on the same repository. Without\n+this safeguard, competing processes could leave the repository in an\n+unpredictable state.\n+\n+The background maintenance schedule runs `git maintenance run` processes\n+on an hourly basis. Each run executes the \"hourly\" tasks. At midnight,\n+that process also executes the \"daily\" tasks. At midnight on the first day\n+of the week, that process also executes the \"weekly\" tasks. A single\n+process iterates over each registered repository, performing the scheduled\n+tasks for that frequency. Depending on the number of registered\n+repositories and their sizes, this process may take longer than an hour.\n+In this case, multiple `git maintenance run` commands may run on the same\n+repository at the same time, colliding on the object database lock. This\n+results in one of the two tasks not running.\n+\n+If you find that some maintenance windows are taking longer than one hour\n+to complete, then consider reducing the complexity of your maintenance\n+tasks. For example, the `gc` task is much slower than the\n+`incremental-repack` task. However, this comes at a cost of a slightly\n+larger object database. Consider moving more expensive tasks to be run\n+less frequently.\n+\n+Expert users may consider scheduling their own maintenance tasks using a\n+different schedule than is available through `git maintenance start` and\n+Git configuration options. These users should be aware of the object\n+database lock and how concurrent `git maintenance run` commands behave.\n+Further, the `git gc` command should not be combined with\n+`git maintenance run` commands. `git gc` modifies the object database\n+but does not take the lock in the same way as `git maintenance run`. If\n+possible, use `git maintenance run --task=gc` instead of `git gc`.\n+\n+\n GIT\n ---\n Part of the linkgit:git[1] suite\n-- \ngitgitgadget\n"},{"id":"406958","messageId":"02e7286dba906f6cd729537d1f22e98ba4154c55.1601902635.git.gitgitgadget@gmail.com","threadId":"54189","inReplyTo":"pull.724.v3.git.1601902635.gitgitgadget@gmail.com","subject":"[PATCH v3 1/7] maintenance: optionally skip --auto process","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-10-05T12:57:08Z","receivedAt":"2020-10-05T13:04:34Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nSome commands run 'git maintenance run --auto --[no-]quiet' after doing\ntheir normal work, as a way to keep repositories clean as they are used.\nCurrently, users who do not want this maintenance to occur would set the\n'gc.auto' config option to 0 to avoid the 'gc' task from running.\nHowever, this does not stop the extra process invocation. On Windows,\nthis extra process invocation can be more expensive than necessary.\n\nAllow users to drop this extra process by setting 'maintenance.auto' to\n'false'.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/config/maintenance.txt | 5 +++++\n run-command.c | 6 ++++++\n t/t7900-maintenance.sh | 13 +++++++++++++\n 3 files changed, 24 insertions(+)\n\ndiff --git a/Documentation/config/maintenance.txt b/Documentation/config/maintenance.txt\nindex a0706d8f09..06db758172 100644\n--- a/Documentation/config/maintenance.txt\n+++ b/Documentation/config/maintenance.txt\n@@ -1,3 +1,8 @@\n+maintenance.auto::\n+\tThis boolean config option controls whether some commands run\n+\t`git maintenance run --auto` after doing their normal work. Defaults\n+\tto true.\n+\n maintenance..enabled::\n \tThis boolean config option controls whether the maintenance task\n \twith name `` is run when no `--task` option is specified to\ndiff --git a/run-command.c b/run-command.c\nindex 2ee59acdc8..ea4d0fb4b1 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -7,6 +7,7 @@\n #include \"strbuf.h\"\n #include \"string-list.h\"\n #include \"quote.h\"\n+#include \"config.h\"\n \n void child_process_init(struct child_process *child)\n {\n@@ -1868,8 +1869,13 @@ int run_processes_parallel_tr2(int n, get_next_task_fn get_next_task,\n \n int run_auto_maintenance(int quiet)\n {\n+\tint enabled;\n \tstruct child_process maint = CHILD_PROCESS_INIT;\n \n+\tif (!git_config_get_bool(\"maintenance.auto\", &enabled) &&\n+\t !enabled)\n+\t\treturn 0;\n+\n \tmaint.git_cmd = 1;\n \tstrvec_pushl(&maint.args, \"maintenance\", \"run\", \"--auto\", NULL);\n \tstrvec_push(&maint.args, quiet ? \"--quiet\" : \"--no-quiet\");\ndiff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\nindex 55116c2f04..c7caaa7a55 100755\n--- a/t/t7900-maintenance.sh\n+++ b/t/t7900-maintenance.sh\n@@ -28,6 +28,19 @@ test_expect_success 'run [--auto|--quiet]' '\n \ttest_subcommand git gc --no-quiet .enabled' '\n \tgit config maintenance.gc.enabled false &&\n \tgit config maintenance.commit-graph.enabled true &&\n-- \ngitgitgadget\n\n"},{"id":"406961","messageId":"20201005131647.GA20195@danh.dev","threadId":"54189","inReplyTo":"2198ba04-e73a-cfd5-960c-909a4f3ef0d2@gmail.com","subject":"Re: [PATCH v2 6/7] maintenance: recommended schedule in register/start","fromName":"Đoàn Trần Công Danh","fromEmail":"congdanhqx@gmail.com","sentAt":"2020-10-05T13:16:47Z","receivedAt":"2020-10-05T13:16:52Z","isPatch":true,"sender":{"key":"congdanhqx@gmail.com","avatar":"https://avatars.githubusercontent.com/u/42673067?v=4"},"body":"On 2020-10-01 21:55:40-0400, Derrick Stolee wrote:\n> On 10/1/2020 8:38 PM, Đoàn Trần Công Danh wrote:\n> > On 2020-10-01 16:38:48-0400, Derrick Stolee wrote:\n> >> diff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt\n> >> index 7628a6d157..52fff86844 100644\n> >> --- a/Documentation/git-maintenance.txt\n> >> +++ b/Documentation/git-maintenance.txt\n> >> @@ -37,6 +37,21 @@ register::\n> >> \t`maintenance..schedule`. The tasks that are enabled are safe\n> >> \tfor running in the background without disrupting foreground\n> >> \tprocesses.\n> >> ++\n> >> +If your repository has no `maintenance..schedule` configuration\n> >> +values set, then Git will use a recommended default schedule that performs\n> >> +background maintenance that will not interrupt foreground commands. The\n> >> +default schedule is as follows:\n> > \n> > I don't mind about using a default schedule (but someone else might).\n> > I think some distributions will be paranoia with this change and shiped\n> > with disable by default in system config.\n> \n> If a user wants to prevent this schedule, then they can simply change\n> any one of the `.schedule` or `.enabled` configs in their --global config\n> and these defaults will not be used.\n> \n> Of course, perhaps you are missing the fact that \"git maintenance run\n> --schedule=\" is only run as a cron job if a user chose to\n> start background maintenance using \"git maintenance start\" (or \"git\n> maintenance register\" after running the 'start' subcommand in another\n> repo). So this is _not_ starting by default without some amount of\n> choosing to opt in.\n\nYes, I missed that fact. Sorry for the noise I generated.\n\nThanks,\n-- Danh\n"},{"id":"406965","messageId":"a47fafec-345f-f675-18d1-98cefc455ecb@gmail.com","threadId":"54189","inReplyTo":"20201005131647.GA20195@danh.dev","subject":"Re: [PATCH v2 6/7] maintenance: recommended schedule in register/start","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-10-05T18:17:14Z","receivedAt":"2020-10-05T18:17:19Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 10/5/2020 9:16 AM, Đoàn Trần Công Danh wrote:\n> On 2020-10-01 21:55:40-0400, Derrick Stolee wrote:\n>> On 10/1/2020 8:38 PM, Đoàn Trần Công Danh wrote:\n>>> I don't mind about using a default schedule (but someone else might).\n>>> I think some distributions will be paranoia with this change and shiped\n>>> with disable by default in system config.\n>>\n>> If a user wants to prevent this schedule, then they can simply change\n>> any one of the `.schedule` or `.enabled` configs in their --global config\n>> and these defaults will not be used.\n>>\n>> Of course, perhaps you are missing the fact that \"git maintenance run\n>> --schedule=\" is only run as a cron job if a user chose to\n>> start background maintenance using \"git maintenance start\" (or \"git\n>> maintenance register\" after running the 'start' subcommand in another\n>> repo). So this is _not_ starting by default without some amount of\n>> choosing to opt in.\n> \n> Yes, I missed that fact. Sorry for the noise I generated.\n\nThanks for confirming. And thank you for the interest in the\nfeature! It's my fault for not including the context properly.\n\nThanks,\n-Stolee\n"},{"id":"406966","messageId":"20201005195701.14268-1-martin.agren@gmail.com","threadId":"54189","inReplyTo":"d833fffe89c94ecf3551c075960ba6d7607e9b17.1601902635.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 6/7] maintenance: use default schedule if not configured","fromName":"Martin Ågren","fromEmail":"martin.agren@gmail.com","sentAt":"2020-10-05T19:57:01Z","receivedAt":"2020-10-05T19:57:25Z","isPatch":true,"sender":{"key":"martin.agren@gmail.com","avatar":null},"body":"Hi Stolee,\n\nOn Mon, 5 Oct 2020 at 15:07, Derrick Stolee via GitGitGadget wrote:\n> The 'git maintenance (register|start)' subcommands add the current\n> repository to the global Git config so maintenance will operate on that\n> repository. It does not specify what maintenance should occur or how\n> often.\n\nI see that you posted a \"how about this?\" some days ago. I was offline\nfor the weekend with some margin on both sides, so I didn't see it until\nnow. Good that you just went ahead and posted the whole series anyway.\n\n> If a user sets any 'maintenance..schedule' config value, then\n> they have chosen a specific schedule for themselves and Git should\n> respect that when running 'git maintenance run --schedule='.\n>\n> To make this process extremely simple for users, assume a default\n> schedule when no 'maintenance..schedule' or '...enabled' config\n> settings are concretely set. This is only an in-process assumption, so\n> future versions of Git could adjust this expected schedule.\n\nThis obviously makes sense to me. ;-) One thing it does mean though is\nthat something like this:\n\n $ git maintenance register\n # Time goes by...\n # Someone said to try this:\n $ git config --add maintenance.commit-graph.schedule hourly\n $ git config --add maintenance.commit-graph.enable true\n # That could have been a no-op, since we were already on\n # such an hourly schedule, but it will effectively turn off\n # all other scheduled tasks. So some time later:\n # -- Why are my fetches so slow all of a sudden? :-(\n\nThat could be different if `git maintenance register` would turn on,\nsay, `maintenance.baseSchedule = standard` where setting those\n`maintenance.commit-graph.*` would tweak that \"standard\" \"base\nschedule\" (in a no-op way as it happens).\n\n> --- a/Documentation/git-maintenance.txt\n> +++ b/Documentation/git-maintenance.txt\n> @@ -37,6 +37,21 @@ register::\n\nAdding some more context manually:\n\nregister::\n\tInitialize Git config values so any scheduled maintenance will\n\tstart running on this repository. This adds the repository to the\n\t`maintenance.repo` config variable in the current user's global\n\tconfig and enables some recommended configuration values for\n> \t`maintenance..schedule`. The tasks that are enabled are safe\n> \tfor running in the background without disrupting foreground\n> \tprocesses.\n\nThe part about \"and enables some recommended configuration values\"\nshould probably be in this patch, not an earlier one, and maybe it\nshouldn't even be here. With the new approach of this version, this\ndoesn't really enable some recommended configuration values. Or maybe\nit does, I can't make up my mind, nor can I come up with an alternative\nformulation.\n\n> ++\n> +If your repository has no `maintenance..schedule` configuration\n> +values set, then Git will use a recommended default schedule that performs\n> +background maintenance that will not interrupt foreground commands. The\n> +default schedule is as follows:\n> ++\n\nIf you add a line of \"--\" here...\n\n> +* `gc`: disabled.\n> +* `commit-graph`: hourly.\n> +* `prefetch`: hourly.\n> +* `loose-objects`: daily.\n> +* `incremental-repack`: daily.\n\n... and one here, you'll drop some indentation at this point so that\nthe next paragraph doesn't align with the list above. (See patch below.)\n\n> ++\n> +`git maintenance register` will also disable foreground maintenance by\n> +setting `maintenance.auto = false` in the current repository. This config\n> +setting will remain after a `git maintenance unregister` command.\n\nThat last paragraph does belong here. The part about the different\ntasks ... maybe. With this new approach of not actually setting any\n`schedule`/`enabled` configuration variables, that list doesn't\nobviously have its natural home here. Maybe under `--schedule`, which is\nwhere the detection actually happens and the default defaults are\nimposed? Or maybe in a separate \"CONFIGURATION\" section. It could\ninclude config/maintenance.txt, then go on to define the whole fallback\nmechanism without having to worry about breaking the reader's flow. (The\nway it is now, this `register` command is fairly heavy compared to the\nsurrounding parts.)\n\n> +static int has_schedule_config(void)\n> +{\n> + int i, found = 0;\n> + struct strbuf config_name = STRBUF_INIT;\n> + size_t prefix;\n> +\n> + strbuf_addstr(&config_name, \"maintenance.\");\n> + prefix = config_name.len;\n> +\n> + for (i = 0; !found && i < TASK__COUNT; i++) {\n> + char *value;\n> +\n> + strbuf_setlen(&config_name, prefix);\n> + strbuf_addf(&config_name, \"%s.schedule\", tasks[i].name);\n> +\n> + if (!git_config_get_string(config_name.buf, &value)) {\n> + found = 1;\n> + FREE_AND_NULL(value);\n> + }\n> +\n> + strbuf_setlen(&config_name, prefix);\n> + strbuf_addf(&config_name, \"%s.enabled\", tasks[i].name);\n> +\n> + if (!git_config_get_string(config_name.buf, &value)) {\n> + found = 1;\n> + FREE_AND_NULL(value);\n> + }\n> + }\n> +\n> + strbuf_release(&config_name);\n> + return found;\n> +}\n\nI had the same reaction to `FREE_AND_NULL()` as on my previous reading.\nIf you have $reasons for doing it this way, not a big deal. I offer a\nsuggestion in patch form below anyway. Feel free to squash, adapt or\nignore as you see fit.\n\n> +\n> +static void set_recommended_schedule(void)\n> +{\n> + if (has_schedule_config())\n> + return;\n> +\n> + tasks[TASK_GC].enabled = 0;\n> +\n> + tasks[TASK_PREFETCH].enabled = 1;\n> + tasks[TASK_PREFETCH].schedule = SCHEDULE_HOURLY;\n> +\n> + tasks[TASK_COMMIT_GRAPH].enabled = 1;\n> + tasks[TASK_COMMIT_GRAPH].schedule = SCHEDULE_HOURLY;\n> +\n> + tasks[TASK_LOOSE_OBJECTS].enabled = 1;\n> + tasks[TASK_LOOSE_OBJECTS].schedule = SCHEDULE_DAILY;\n> +\n> + tasks[TASK_INCREMENTAL_REPACK].enabled = 1;\n> + tasks[TASK_INCREMENTAL_REPACK].schedule = SCHEDULE_DAILY;\n> +}\n> +\n\nOne thing I can't make up my mind about is how these `enabled` are used\nfor two purposes: Deciding what to do on `git maintenance run` without\nany `--task`, and deciding what to do on `git maintenance run\n--scheduled`.\n\n> static int maintenance_run_tasks(struct maintenance_run_opts *opts)\n> {\n> int i, found_selected = 0;\n> @@ -1280,6 +1333,8 @@ static int maintenance_run_tasks(struct maintenance_run_opts *opts)\n>\n> if (found_selected)\n> QSORT(tasks, TASK__COUNT, compare_tasks_by_selection);\n> + else if (opts->schedule != SCHEDULE_NONE)\n> + set_recommended_schedule();\n\n... And especially how we only impose the magic\n`maintenance..enabled` values when we are running with\n`--schedule`. So the answer to \"what is the default value of\n`maintenance.commit-graph.enabled`?\" is \"it depends on several factors\".\n\nSort of related: The presence of a `maintenance..schedule` is not\nsufficient to schedule the task. This looks like something that one\ncould easily trip on. Maybe you have already considered letting a zero\nvalue for `maintenance..schedule` mean \"disabled\" and ignoring the\n`enabled` config item for the scheduled runs, but rejected that for good\nreasons?\n\n> diff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\n> index 7715e40391..7154987fd2 100755\n> --- a/t/t7900-maintenance.sh\n> +++ b/t/t7900-maintenance.sh\n> @@ -305,11 +305,14 @@ test_expect_success 'register and unregister' '\n> git config --global --add maintenance.repo /existing1 &&\n> git config --global --add maintenance.repo /existing2 &&\n> git config --global --get-all maintenance.repo >before &&\n> +\n> git maintenance register &&\n> - git config --global --get-all maintenance.repo >actual &&\n> - cp before after &&\n> - pwd >>after &&\n> - test_cmp after actual &&\n> + test_cmp_config false maintenance.auto &&\n> + git config --global --get-all maintenance.repo >between &&\n> + cp before expect &&\n> + pwd >>expect &&\n> + test_cmp expect between &&\n> +\n> git maintenance unregister &&\n> git config --global --get-all maintenance.repo >actual &&\n> test_cmp before actual\n\nThis tests the one-time config tweaking. But we don't test any of the\n\"detect no config and impose a default\" logic. Neither that it kicks in\nat all, nor that it doesn't when it shouldn't.\n\nAs mentioned above, I end with some minor suggestions.\n\nMartin\n\ndiff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt\nindex 738a4c7ebd..2085b53dc5 100644\n--- a/Documentation/git-maintenance.txt\n+++ b/Documentation/git-maintenance.txt\n@@ -43,11 +43,13 @@ values set, then Git will use a recommended default schedule that performs\n background maintenance that will not interrupt foreground commands. The\n default schedule is as follows:\n +\n+--\n * `gc`: disabled.\n * `commit-graph`: hourly.\n * `prefetch`: hourly.\n * `loose-objects`: daily.\n * `incremental-repack`: daily.\n+--\n +\n `git maintenance register` will also disable foreground maintenance by\n setting `maintenance.auto = false` in the current repository. This config\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 965690704b..63f4c102b1 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -1253,35 +1253,31 @@ static int compare_tasks_by_selection(const void *a_, const void *b_)\n \n static int has_schedule_config(void)\n {\n-\tint i, found = 0;\n+\tint i;\n \tstruct strbuf config_name = STRBUF_INIT;\n \tsize_t prefix;\n+\tchar *value = NULL;\n \n \tstrbuf_addstr(&config_name, \"maintenance.\");\n \tprefix = config_name.len;\n \n-\tfor (i = 0; !found && i < TASK__COUNT; i++) {\n-\t\tchar *value;\n-\n+\tfor (i = 0; i < TASK__COUNT; i++) {\n \t\tstrbuf_setlen(&config_name, prefix);\n \t\tstrbuf_addf(&config_name, \"%s.schedule\", tasks[i].name);\n \n-\t\tif (!git_config_get_string(config_name.buf, &value)) {\n-\t\t\tfound = 1;\n-\t\t\tFREE_AND_NULL(value);\n-\t\t}\n+\t\tif (!git_config_get_string(config_name.buf, &value))\n+\t\t\tbreak;\n \n \t\tstrbuf_setlen(&config_name, prefix);\n \t\tstrbuf_addf(&config_name, \"%s.enabled\", tasks[i].name);\n \n-\t\tif (!git_config_get_string(config_name.buf, &value)) {\n-\t\t\tfound = 1;\n-\t\t\tFREE_AND_NULL(value);\n-\t\t}\n+\t\tif (!git_config_get_string(config_name.buf, &value))\n+\t\t\tbreak;\n \t}\n \n \tstrbuf_release(&config_name);\n-\treturn found;\n+\tfree(value);\n+\treturn i < TASK__COUNT;\n }\n \n static void set_recommended_schedule(void)\n-- \n2.28.0.297.g1956fa8f8d\n\n"},{"id":"407148","messageId":"b9791e64-57c5-465d-7175-efe6fcee8aac@gmail.com","threadId":"54189","inReplyTo":"20201005195701.14268-1-martin.agren@gmail.com","subject":"Re: [PATCH v3 6/7] maintenance: use default schedule if not configured","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-10-08T13:32:00Z","receivedAt":"2020-10-08T13:34:44Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 10/5/2020 3:57 PM, Martin Ågren wrote:\n> On Mon, 5 Oct 2020 at 15:07, Derrick Stolee via GitGitGadget wrote:\n>> To make this process extremely simple for users, assume a default\n>> schedule when no 'maintenance..schedule' or '...enabled' config\n>> settings are concretely set. This is only an in-process assumption, so\n>> future versions of Git could adjust this expected schedule.\n> \n> This obviously makes sense to me. ;-) One thing it does mean though is\n> that something like this:\n> \n> $ git maintenance register\n> # Time goes by...\n> # Someone said to try this:\n> $ git config --add maintenance.commit-graph.schedule hourly\n> $ git config --add maintenance.commit-graph.enable true\n> # That could have been a no-op, since we were already on\n> # such an hourly schedule, but it will effectively turn off\n> # all other scheduled tasks. So some time later:\n> # -- Why are my fetches so slow all of a sudden? :-(\n> \n> That could be different if `git maintenance register` would turn on,\n> say, `maintenance.baseSchedule = standard` where setting those\n> `maintenance.commit-graph.*` would tweak that \"standard\" \"base\n> schedule\" (in a no-op way as it happens).\n\nThank you so much for your detailed feedback! This is an excellent\npoint and I will be sure to account for it when I have time to\ncarefully examine the options and these kind of workflows.\n\nRight now, I'm a bit underwater getting ready for the v2.29.0\nrelease (in microsoft/git, Scalar, and VFS for Git) but I will\nrevisit this as my main focus after this release cycle. I have\nnot forgotten about this topic!!!\n\n> As mentioned above, I end with some minor suggestions.\n\nI really appreciate the effort you put in to this fixup.\nThanks,\n-Stolee\n\n"},{"id":"407633","messageId":"02e7286dba906f6cd729537d1f22e98ba4154c55.1602782524.git.gitgitgadget@gmail.com","threadId":"54189","inReplyTo":"pull.724.v4.git.1602782524.gitgitgadget@gmail.com","subject":"[PATCH v4 1/8] maintenance: optionally skip --auto process","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-10-15T17:21:57Z","receivedAt":"2020-10-15T17:22:09Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nSome commands run 'git maintenance run --auto --[no-]quiet' after doing\ntheir normal work, as a way to keep repositories clean as they are used.\nCurrently, users who do not want this maintenance to occur would set the\n'gc.auto' config option to 0 to avoid the 'gc' task from running.\nHowever, this does not stop the extra process invocation. On Windows,\nthis extra process invocation can be more expensive than necessary.\n\nAllow users to drop this extra process by setting 'maintenance.auto' to\n'false'.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/config/maintenance.txt | 5 +++++\n run-command.c | 6 ++++++\n t/t7900-maintenance.sh | 13 +++++++++++++\n 3 files changed, 24 insertions(+)\n\ndiff --git a/Documentation/config/maintenance.txt b/Documentation/config/maintenance.txt\nindex a0706d8f09..06db758172 100644\n--- a/Documentation/config/maintenance.txt\n+++ b/Documentation/config/maintenance.txt\n@@ -1,3 +1,8 @@\n+maintenance.auto::\n+\tThis boolean config option controls whether some commands run\n+\t`git maintenance run --auto` after doing their normal work. Defaults\n+\tto true.\n+\n maintenance..enabled::\n \tThis boolean config option controls whether the maintenance task\n \twith name `` is run when no `--task` option is specified to\ndiff --git a/run-command.c b/run-command.c\nindex 2ee59acdc8..ea4d0fb4b1 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -7,6 +7,7 @@\n #include \"strbuf.h\"\n #include \"string-list.h\"\n #include \"quote.h\"\n+#include \"config.h\"\n \n void child_process_init(struct child_process *child)\n {\n@@ -1868,8 +1869,13 @@ int run_processes_parallel_tr2(int n, get_next_task_fn get_next_task,\n \n int run_auto_maintenance(int quiet)\n {\n+\tint enabled;\n \tstruct child_process maint = CHILD_PROCESS_INIT;\n \n+\tif (!git_config_get_bool(\"maintenance.auto\", &enabled) &&\n+\t !enabled)\n+\t\treturn 0;\n+\n \tmaint.git_cmd = 1;\n \tstrvec_pushl(&maint.args, \"maintenance\", \"run\", \"--auto\", NULL);\n \tstrvec_push(&maint.args, quiet ? \"--quiet\" : \"--no-quiet\");\ndiff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\nindex 55116c2f04..c7caaa7a55 100755\n--- a/t/t7900-maintenance.sh\n+++ b/t/t7900-maintenance.sh\n@@ -28,6 +28,19 @@ test_expect_success 'run [--auto|--quiet]' '\n \ttest_subcommand git gc --no-quiet .enabled' '\n \tgit config maintenance.gc.enabled false &&\n \tgit config maintenance.commit-graph.enabled true &&\n-- \ngitgitgadget\n\n"},{"id":"407634","messageId":"pull.724.v4.git.1602782524.gitgitgadget@gmail.com","threadId":"54189","inReplyTo":"pull.724.v3.git.1601902635.gitgitgadget@gmail.com","subject":"[PATCH v4 0/8] Maintenance III: Background maintenance","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-10-15T17:21:56Z","receivedAt":"2020-10-15T17:22:10Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"This is based on ds/maintenance-part-2 and replaces the RFC from [1].\n\n[1] \nhttps://lore.kernel.org/git/pull.680.v3.git.1598629517.gitgitgadget@gmail.com/\n\nThis series introduces background maintenance to Git, through an integration\nwith cron and crontab.\n\nSome preliminary work is done to allow a new --schedule option that tells\nthe command which tasks to run based on a maintenance..schedule config\noption. The timing is not enforced by Git, but instead is expected to be\nprovided as a hint from a cron schedule. The options are \"hourly\", \"daily\",\nand \"weekly\".\n\nA new for-each-repo builtin runs Git commands on every repo in a given list.\nCurrently, the list is stored as a config setting, allowing a new \nmaintenance.repos config list to store the repositories registered for\nbackground maintenance. Others may want to add a --file= option for\ntheir own workflows, but I focused on making this as simple as possible for\nnow.\n\nThe updates to the git maintenance builtin include new register/unregister \nsubcommands and start/stop subcommands. The register subcommand initializes\nthe config while the start subcommand does everything register does plus \nupdate the cron table. The unregister and stop commands reverse this\nprocess.\n\nA troubleshooting guide is added to Documentation/git-maintenance.txt to\nadvise expert users who choose to create custom cron schedules.\n\nA new config option \"maintenance.strategy\" allows users to pick from one of\na potentially-expanding number of recommended schedules. Currently, the only\none baked-in is \"incremental\" although the documentation specifies that\n\"none\" will prevent any tasks from running by default. Using that value\nprevents the config from being overridden in \"git maintenance\n(start|register)\". Users can augment the incremental strategy by assigning\nspecific \"maintenance..schedule\" config options.\n\nI've been testing this scenario on my macOS laptop and Linux desktop. I have\nmodified my cron task to provide logging via trace2 so I can see what's\nhappening. A future direction here would be to add some maintenance logs to\nthe repository so we can track what is happening and diagnose whether the\nmaintenance strategy is working on real repos.\n\nNote: git maintenance (start|stop) only works on machines with cron by\ndesign. The proper thing to do on Windows will come later. Perhaps this\ncommand should be marked as unavailable on Windows somehow, or at least a\nbetter error than \"cron may not be available on your system\". I did find\nthat that message is helpful sometimes: macOS worker agents for CI builds\ntypically do not have cron available.\n\nUpdates in v4:\n\n * Thanks, Martin for pointing out a usability concern with how I was\n previously assigning a default schedule.\n * The new logic is to create a new \"maintenance.strategy\" config option,\n which can be explicitly disabled or augmented.\n * This approach should be a suitable blend of the two previous options, and\n gives us flexibility to adjust the schedule or add more strategies in the\n future!\n\nUpdates in v3:\n\n * Instead of writing config upon \"register\" or \"start\", simply create an\n in-memory default schedule when no .schedule or .enabled configs are\n present. Thanks, Martin! This causes patch 6 to look so different that\n the range-diff considers it a dropped-and-added patch instead of showing\n a diff.\n * There are some context lines that changed because this is rebased onto a\n recent version of ds/maintenance-part-2.\n\nUpdates in v2:\n\n * Fixed the char/int issue in test-tool crontab, and a typo.\n * Updated commit message and patch noise in PATCH 2\n * This should fix the test failures, allowing this to be picked up in\n 'seen'.\n\nDerrick Stolee (8):\n maintenance: optionally skip --auto process\n maintenance: add --schedule option and config\n for-each-repo: run subcommands on configured repos\n maintenance: add [un]register subcommands\n maintenance: add start/stop subcommands\n maintenance: create maintenance.strategy config\n maintenance: use 'incremental' strategy by default\n maintenance: add troubleshooting guide to docs\n\n .gitignore | 1 +\n Documentation/config/maintenance.txt | 25 +++\n Documentation/git-for-each-repo.txt | 59 ++++++\n Documentation/git-maintenance.txt | 99 +++++++++-\n Makefile | 2 +\n builtin.h | 1 +\n builtin/for-each-repo.c | 58 ++++++\n builtin/gc.c | 281 ++++++++++++++++++++++++++-\n command-list.txt | 1 +\n git.c | 1 +\n run-command.c | 6 +\n t/helper/test-crontab.c | 35 ++++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/t0068-for-each-repo.sh | 30 +++\n t/t7900-maintenance.sh | 159 ++++++++++++++-\n t/test-lib.sh | 6 +\n 17 files changed, 758 insertions(+), 8 deletions(-)\n create mode 100644 Documentation/git-for-each-repo.txt\n create mode 100644 builtin/for-each-repo.c\n create mode 100644 t/helper/test-crontab.c\n create mode 100755 t/t0068-for-each-repo.sh\n\n\nbase-commit: e841a79a131d8ce491cf04d0ca3e24f139a10b82\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-724%2Fderrickstolee%2Fmaintenance%2Fscheduled-v4\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-724/derrickstolee/maintenance/scheduled-v4\nPull-Request: https://github.com/gitgitgadget/git/pull/724\n\nRange-diff vs v3:\n\n 1: 02e7286dba = 1: 02e7286dba maintenance: optionally skip --auto process\n 2: dae8c04bb5 = 2: dae8c04bb5 maintenance: add --schedule option and config\n 3: dd92379273 = 3: dd92379273 for-each-repo: run subcommands on configured repos\n 4: 922b984c8a = 4: 922b984c8a maintenance: add [un]register subcommands\n 5: 5194f6b1fa = 5: 5194f6b1fa maintenance: add start/stop subcommands\n -: ---------- > 6: d696848b37 maintenance: create maintenance.strategy config\n 6: d833fffe89 ! 7: 145c63ed8c maintenance: use default schedule if not configured\n @@ Metadata\n Author: Derrick Stolee \n \n ## Commit message ##\n - maintenance: use default schedule if not configured\n + maintenance: use 'incremental' strategy by default\n \n The 'git maintenance (register|start)' subcommands add the current\n repository to the global Git config so maintenance will operate on that\n repository. It does not specify what maintenance should occur or how\n often.\n \n - If a user sets any 'maintenance..schedule' config value, then\n - they have chosen a specific schedule for themselves and Git should\n - respect that when running 'git maintenance run --schedule='.\n -\n - To make this process extremely simple for users, assume a default\n - schedule when no 'maintenance..schedule' or '...enabled' config\n - settings are concretely set. This is only an in-process assumption, so\n - future versions of Git could adjust this expected schedule.\n + To make it simple for users to start background maintenance with a\n + recommended schedlue, update the 'maintenance.strategy' config option in\n + both the 'register' and 'start' subcommands. This allows users to\n + customize beyond the defaults using individual\n + 'maintenance..schedule' options, but also the user can opt-out of\n + this strategy using 'maintenance.strategy=none'.\n \n Helped-by: Martin Ågren \n Signed-off-by: Derrick Stolee \n @@ Documentation/git-maintenance.txt: register::\n \tfor running in the background without disrupting foreground\n \tprocesses.\n ++\n -+If your repository has no `maintenance..schedule` configuration\n -+values set, then Git will use a recommended default schedule that performs\n -+background maintenance that will not interrupt foreground commands. The\n -+default schedule is as follows:\n ++The `register` subcomand will also set the `maintenance.strategy` config\n ++value to `incremental`, if this value is not previously set. The\n ++`incremental` strategy uses the following schedule for each maintenance\n ++task:\n ++\n ++--\n +* `gc`: disabled.\n +* `commit-graph`: hourly.\n +* `prefetch`: hourly.\n +* `loose-objects`: daily.\n +* `incremental-repack`: daily.\n ++--\n ++\n +`git maintenance register` will also disable foreground maintenance by\n +setting `maintenance.auto = false` in the current repository. This config\n @@ Documentation/git-maintenance.txt: register::\n \tRun one or more maintenance tasks. If one or more `--task` options\n \n ## builtin/gc.c ##\n -@@ builtin/gc.c: static int compare_tasks_by_selection(const void *a_, const void *b_)\n - \treturn b->selected_order - a->selected_order;\n - }\n +@@ builtin/gc.c: static int maintenance_run(int argc, const char **argv, const char *prefix)\n \n -+static int has_schedule_config(void)\n -+{\n -+\tint i, found = 0;\n -+\tstruct strbuf config_name = STRBUF_INIT;\n -+\tsize_t prefix;\n -+\n -+\tstrbuf_addstr(&config_name, \"maintenance.\");\n -+\tprefix = config_name.len;\n -+\n -+\tfor (i = 0; !found && i < TASK__COUNT; i++) {\n -+\t\tchar *value;\n -+\n -+\t\tstrbuf_setlen(&config_name, prefix);\n -+\t\tstrbuf_addf(&config_name, \"%s.schedule\", tasks[i].name);\n -+\n -+\t\tif (!git_config_get_string(config_name.buf, &value)) {\n -+\t\t\tfound = 1;\n -+\t\t\tFREE_AND_NULL(value);\n -+\t\t}\n -+\n -+\t\tstrbuf_setlen(&config_name, prefix);\n -+\t\tstrbuf_addf(&config_name, \"%s.enabled\", tasks[i].name);\n -+\n -+\t\tif (!git_config_get_string(config_name.buf, &value)) {\n -+\t\t\tfound = 1;\n -+\t\t\tFREE_AND_NULL(value);\n -+\t\t}\n -+\t}\n -+\n -+\tstrbuf_release(&config_name);\n -+\treturn found;\n -+}\n -+\n -+static void set_recommended_schedule(void)\n -+{\n -+\tif (has_schedule_config())\n -+\t\treturn;\n -+\n -+\ttasks[TASK_GC].enabled = 0;\n -+\n -+\ttasks[TASK_PREFETCH].enabled = 1;\n -+\ttasks[TASK_PREFETCH].schedule = SCHEDULE_HOURLY;\n -+\n -+\ttasks[TASK_COMMIT_GRAPH].enabled = 1;\n -+\ttasks[TASK_COMMIT_GRAPH].schedule = SCHEDULE_HOURLY;\n -+\n -+\ttasks[TASK_LOOSE_OBJECTS].enabled = 1;\n -+\ttasks[TASK_LOOSE_OBJECTS].schedule = SCHEDULE_DAILY;\n -+\n -+\ttasks[TASK_INCREMENTAL_REPACK].enabled = 1;\n -+\ttasks[TASK_INCREMENTAL_REPACK].schedule = SCHEDULE_DAILY;\n -+}\n -+\n - static int maintenance_run_tasks(struct maintenance_run_opts *opts)\n + static int maintenance_register(void)\n {\n - \tint i, found_selected = 0;\n -@@ builtin/gc.c: static int maintenance_run_tasks(struct maintenance_run_opts *opts)\n - \n - \tif (found_selected)\n - \t\tQSORT(tasks, TASK__COUNT, compare_tasks_by_selection);\n -+\telse if (opts->schedule != SCHEDULE_NONE)\n -+\t\tset_recommended_schedule();\n ++\tchar *config_value;\n + \tstruct child_process config_set = CHILD_PROCESS_INIT;\n + \tstruct child_process config_get = CHILD_PROCESS_INIT;\n \n - \tfor (i = 0; i < TASK__COUNT; i++) {\n - \t\tif (found_selected && tasks[i].selected_order < 0)\n @@ builtin/gc.c: static int maintenance_register(void)\n \tif (!the_repository || !the_repository->gitdir)\n \t\treturn 0;\n \n +\t/* Disable foreground maintenance */\n +\tgit_config_set(\"maintenance.auto\", \"false\");\n ++\n ++\t/* Set maintenance strategy, if unset */\n ++\tif (!git_config_get_string(\"maintenance.strategy\", &config_value))\n ++\t\tfree(config_value);\n ++\telse\n ++\t\tgit_config_set(\"maintenance.strategy\", \"incremental\");\n +\n \tconfig_get.git_cmd = 1;\n \tstrvec_pushl(&config_get.args, \"config\", \"--global\", \"--get\", \"maintenance.repo\",\n @@ t/t7900-maintenance.sh: test_expect_success 'register and unregister' '\n \tgit maintenance unregister &&\n \tgit config --global --get-all maintenance.repo >actual &&\n \ttest_cmp before actual\n +@@ t/t7900-maintenance.sh: test_expect_success 'start preserves existing schedule' '\n + \tgrep \"Important information!\" cron.txt\n + '\n + \n ++test_expect_success 'register preserves existing strategy' '\n ++\tgit config maintenance.strategy none &&\n ++\tgit maintenance register &&\n ++\ttest_config maintenance.strategy none &&\n ++\tgit config --unset maintenance.strategy &&\n ++\tgit maintenance register &&\n ++\ttest_config maintenance.strategy incremental\n ++'\n ++\n + test_done\n 7: 8e42ff44ce = 8: ce0ced705f maintenance: add troubleshooting guide to docs\n\n-- \ngitgitgadget\n"},{"id":"407635","messageId":"dd9237927395ef69663ab376a2da74da4654c495.1602782524.git.gitgitgadget@gmail.com","threadId":"54189","inReplyTo":"pull.724.v4.git.1602782524.gitgitgadget@gmail.com","subject":"[PATCH v4 3/8] for-each-repo: run subcommands on configured repos","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-10-15T17:21:59Z","receivedAt":"2020-10-15T17:22:14Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIt can be helpful to store a list of repositories in global or system\nconfig and then iterate Git commands on that list. Create a new builtin\nthat makes this process simple for experts. We will use this builtin to\nrun scheduled maintenance on all configured repositories in a future\nchange.\n\nThe test is very simple, but does highlight that the \"--\" argument is\noptional.\n\nSigned-off-by: Derrick Stolee \n---\n .gitignore | 1 +\n Documentation/git-for-each-repo.txt | 59 +++++++++++++++++++++++++++++\n Makefile | 1 +\n builtin.h | 1 +\n builtin/for-each-repo.c | 58 ++++++++++++++++++++++++++++\n command-list.txt | 1 +\n git.c | 1 +\n t/t0068-for-each-repo.sh | 30 +++++++++++++++\n 8 files changed, 152 insertions(+)\n create mode 100644 Documentation/git-for-each-repo.txt\n create mode 100644 builtin/for-each-repo.c\n create mode 100755 t/t0068-for-each-repo.sh\n\ndiff --git a/.gitignore b/.gitignore\nindex a5808fa30d..5eb2a2be71 100644\n--- a/.gitignore\n+++ b/.gitignore\n@@ -67,6 +67,7 @@\n /git-filter-branch\n /git-fmt-merge-msg\n /git-for-each-ref\n+/git-for-each-repo\n /git-format-patch\n /git-fsck\n /git-fsck-objects\ndiff --git a/Documentation/git-for-each-repo.txt b/Documentation/git-for-each-repo.txt\nnew file mode 100644\nindex 0000000000..94bd19da26\n--- /dev/null\n+++ b/Documentation/git-for-each-repo.txt\n@@ -0,0 +1,59 @@\n+git-for-each-repo(1)\n+====================\n+\n+NAME\n+----\n+git-for-each-repo - Run a Git command on a list of repositories\n+\n+\n+SYNOPSIS\n+--------\n+[verse]\n+'git for-each-repo' --config= [--] \n+\n+\n+DESCRIPTION\n+-----------\n+Run a Git command on a list of repositories. The arguments after the\n+known options or `--` indicator are used as the arguments for the Git\n+subprocess.\n+\n+THIS COMMAND IS EXPERIMENTAL. THE BEHAVIOR MAY CHANGE.\n+\n+For example, we could run maintenance on each of a list of repositories\n+stored in a `maintenance.repo` config variable using\n+\n+-------------\n+git for-each-repo --config=maintenance.repo maintenance run\n+-------------\n+\n+This will run `git -C maintenance run` for each value ``\n+in the multi-valued config variable `maintenance.repo`.\n+\n+\n+OPTIONS\n+-------\n+--config=::\n+\tUse the given config variable as a multi-valued list storing\n+\tabsolute path names. Iterate on that list of paths to run\n+\tthe given arguments.\n++\n+These config values are loaded from system, global, and local Git config,\n+as available. If `git for-each-repo` is run in a directory that is not a\n+Git repository, then only the system and global config is used.\n+\n+\n+SUBPROCESS BEHAVIOR\n+-------------------\n+\n+If any `git -C ` subprocess returns a non-zero exit code,\n+then the `git for-each-repo` process returns that exit code without running\n+more subprocesses.\n+\n+Each `git -C ` subprocess inherits the standard file\n+descriptors `stdin`, `stdout`, and `stderr`.\n+\n+\n+GIT\n+---\n+Part of the linkgit:git[1] suite\ndiff --git a/Makefile b/Makefile\nindex 65f8cfb236..7c588ff036 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1071,6 +1071,7 @@ BUILTIN_OBJS += builtin/fetch-pack.o\n BUILTIN_OBJS += builtin/fetch.o\n BUILTIN_OBJS += builtin/fmt-merge-msg.o\n BUILTIN_OBJS += builtin/for-each-ref.o\n+BUILTIN_OBJS += builtin/for-each-repo.o\n BUILTIN_OBJS += builtin/fsck.o\n BUILTIN_OBJS += builtin/gc.o\n BUILTIN_OBJS += builtin/get-tar-commit-id.o\ndiff --git a/builtin.h b/builtin.h\nindex 17c1c0ce49..ff7c6e5aa9 100644\n--- a/builtin.h\n+++ b/builtin.h\n@@ -150,6 +150,7 @@ int cmd_fetch(int argc, const char **argv, const char *prefix);\n int cmd_fetch_pack(int argc, const char **argv, const char *prefix);\n int cmd_fmt_merge_msg(int argc, const char **argv, const char *prefix);\n int cmd_for_each_ref(int argc, const char **argv, const char *prefix);\n+int cmd_for_each_repo(int argc, const char **argv, const char *prefix);\n int cmd_format_patch(int argc, const char **argv, const char *prefix);\n int cmd_fsck(int argc, const char **argv, const char *prefix);\n int cmd_gc(int argc, const char **argv, const char *prefix);\ndiff --git a/builtin/for-each-repo.c b/builtin/for-each-repo.c\nnew file mode 100644\nindex 0000000000..5bba623ff1\n--- /dev/null\n+++ b/builtin/for-each-repo.c\n@@ -0,0 +1,58 @@\n+#include \"cache.h\"\n+#include \"config.h\"\n+#include \"builtin.h\"\n+#include \"parse-options.h\"\n+#include \"run-command.h\"\n+#include \"string-list.h\"\n+\n+static const char * const for_each_repo_usage[] = {\n+\tN_(\"git for-each-repo --config= \"),\n+\tNULL\n+};\n+\n+static int run_command_on_repo(const char *path,\n+\t\t\t void *cbdata)\n+{\n+\tint i;\n+\tstruct child_process child = CHILD_PROCESS_INIT;\n+\tstruct strvec *args = (struct strvec *)cbdata;\n+\n+\tchild.git_cmd = 1;\n+\tstrvec_pushl(&child.args, \"-C\", path, NULL);\n+\n+\tfor (i = 0; i < args->nr; i++)\n+\t\tstrvec_push(&child.args, args->v[i]);\n+\n+\treturn run_command(&child);\n+}\n+\n+int cmd_for_each_repo(int argc, const char **argv, const char *prefix)\n+{\n+\tstatic const char *config_key = NULL;\n+\tint i, result = 0;\n+\tconst struct string_list *values;\n+\tstruct strvec args = STRVEC_INIT;\n+\n+\tconst struct option options[] = {\n+\t\tOPT_STRING(0, \"config\", &config_key, N_(\"config\"),\n+\t\t\t N_(\"config key storing a list of repository paths\")),\n+\t\tOPT_END()\n+\t};\n+\n+\targc = parse_options(argc, argv, prefix, options, for_each_repo_usage,\n+\t\t\t PARSE_OPT_STOP_AT_NON_OPTION);\n+\n+\tif (!config_key)\n+\t\tdie(_(\"missing --config=\"));\n+\n+\tfor (i = 0; i < argc; i++)\n+\t\tstrvec_push(&args, argv[i]);\n+\n+\tvalues = repo_config_get_value_multi(the_repository,\n+\t\t\t\t\t config_key);\n+\n+\tfor (i = 0; !result && i < values->nr; i++)\n+\t\tresult = run_command_on_repo(values->items[i].string, &args);\n+\n+\treturn result;\n+}\ndiff --git a/command-list.txt b/command-list.txt\nindex 0e3204e7d1..581499be82 100644\n--- a/command-list.txt\n+++ b/command-list.txt\n@@ -94,6 +94,7 @@ git-fetch-pack synchingrepositories\n git-filter-branch ancillarymanipulators\n git-fmt-merge-msg purehelpers\n git-for-each-ref plumbinginterrogators\n+git-for-each-repo plumbinginterrogators\n git-format-patch mainporcelain\n git-fsck ancillaryinterrogators complete\n git-gc mainporcelain\ndiff --git a/git.c b/git.c\nindex 24f250d29a..1cab64b5d1 100644\n--- a/git.c\n+++ b/git.c\n@@ -511,6 +511,7 @@ static struct cmd_struct commands[] = {\n \t{ \"fetch-pack\", cmd_fetch_pack, RUN_SETUP | NO_PARSEOPT },\n \t{ \"fmt-merge-msg\", cmd_fmt_merge_msg, RUN_SETUP },\n \t{ \"for-each-ref\", cmd_for_each_ref, RUN_SETUP },\n+\t{ \"for-each-repo\", cmd_for_each_repo, RUN_SETUP_GENTLY },\n \t{ \"format-patch\", cmd_format_patch, RUN_SETUP },\n \t{ \"fsck\", cmd_fsck, RUN_SETUP },\n \t{ \"fsck-objects\", cmd_fsck, RUN_SETUP },\ndiff --git a/t/t0068-for-each-repo.sh b/t/t0068-for-each-repo.sh\nnew file mode 100755\nindex 0000000000..136b4ec839\n--- /dev/null\n+++ b/t/t0068-for-each-repo.sh\n@@ -0,0 +1,30 @@\n+#!/bin/sh\n+\n+test_description='git for-each-repo builtin'\n+\n+. ./test-lib.sh\n+\n+test_expect_success 'run based on configured value' '\n+\tgit init one &&\n+\tgit init two &&\n+\tgit init three &&\n+\tgit -C two commit --allow-empty -m \"DID NOT RUN\" &&\n+\tgit config run.key \"$TRASH_DIRECTORY/one\" &&\n+\tgit config --add run.key \"$TRASH_DIRECTORY/three\" &&\n+\tgit for-each-repo --config=run.key commit --allow-empty -m \"ran\" &&\n+\tgit -C one log -1 --pretty=format:%s >message &&\n+\tgrep ran message &&\n+\tgit -C two log -1 --pretty=format:%s >message &&\n+\t! grep ran message &&\n+\tgit -C three log -1 --pretty=format:%s >message &&\n+\tgrep ran message &&\n+\tgit for-each-repo --config=run.key -- commit --allow-empty -m \"ran again\" &&\n+\tgit -C one log -1 --pretty=format:%s >message &&\n+\tgrep again message &&\n+\tgit -C two log -1 --pretty=format:%s >message &&\n+\t! grep again message &&\n+\tgit -C three log -1 --pretty=format:%s >message &&\n+\tgrep again message\n+'\n+\n+test_done\n-- \ngitgitgadget\n\n"},{"id":"407636","messageId":"5194f6b1facbd14cc17eea0337c0cc397a2a51fc.1602782524.git.gitgitgadget@gmail.com","threadId":"54189","inReplyTo":"pull.724.v4.git.1602782524.gitgitgadget@gmail.com","subject":"[PATCH v4 5/8] maintenance: add start/stop subcommands","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-10-15T17:22:01Z","receivedAt":"2020-10-15T17:22:16Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nAdd new subcommands to 'git maintenance' that start or stop background\nmaintenance using 'cron', when available. This integration is as simple\nas I could make it, barring some implementation complications.\n\nThe schedule is laid out as follows:\n\n 0 1-23 * * * $cmd maintenance run --schedule=hourly\n 0 0 * * 1-6 $cmd maintenance run --schedule=daily\n 0 0 * * 0 $cmd maintenance run --schedule=weekly\n\nwhere $cmd is a properly-qualified 'git for-each-repo' execution:\n\n$cmd=$path/git --exec-path=$path for-each-repo --config=maintenance.repo\n\nwhere $path points to the location of the Git executable running 'git\nmaintenance start'. This is critical for systems with multiple versions\nof Git. Specifically, macOS has a system version at '/usr/bin/git' while\nthe version that users can install resides at '/usr/local/bin/git'\n(symlinked to '/usr/local/libexec/git-core/git'). This will also use\nyour locally-built version if you build and run this in your development\nenvironment without installing first.\n\nThis conditional schedule avoids having cron launch multiple 'git\nfor-each-repo' commands in parallel. Such parallel commands would likely\nlead to the 'hourly' and 'daily' tasks competing over the object\ndatabase lock. This could lead to to some tasks never being run! Since\nthe --schedule= argument will run all tasks with _at least_\nthe given frequency, the daily runs will also run the hourly tasks.\nSimilarly, the weekly runs will also run the daily and hourly tasks.\n\nThe GIT_TEST_CRONTAB environment variable is not intended for users to\nedit, but instead as a way to mock the 'crontab [-l]' command. This\nvariable is set in test-lib.sh to avoid a future test from accidentally\nrunning anything with the cron integration from modifying the user's\nschedule. We use GIT_TEST_CRONTAB='test-tool crontab ' in our\ntests to check how the schedule is modified in 'git maintenance\n(start|stop)' commands.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-maintenance.txt | 11 +++\n Makefile | 1 +\n builtin/gc.c | 124 ++++++++++++++++++++++++++++++\n t/helper/test-crontab.c | 35 +++++++++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/t7900-maintenance.sh | 28 +++++++\n t/test-lib.sh | 6 ++\n 8 files changed, 207 insertions(+)\n create mode 100644 t/helper/test-crontab.c\n\ndiff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt\nindex 1c59fd0cb5..7628a6d157 100644\n--- a/Documentation/git-maintenance.txt\n+++ b/Documentation/git-maintenance.txt\n@@ -45,6 +45,17 @@ run::\n \tconfig options are true. By default, only `maintenance.gc.enabled`\n \tis true.\n \n+start::\n+\tStart running maintenance on the current repository. This performs\n+\tthe same config updates as the `register` subcommand, then updates\n+\tthe background scheduler to run `git maintenance run --scheduled`\n+\ton an hourly basis.\n+\n+stop::\n+\tHalt the background maintenance schedule. The current repository\n+\tis not removed from the list of maintained repositories, in case\n+\tthe background maintenance is restarted later.\n+\n unregister::\n \tRemove the current repository from background maintenance. This\n \tonly removes the repository from the configured list. It does not\ndiff --git a/Makefile b/Makefile\nindex 7c588ff036..c39b39bd7d 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -690,6 +690,7 @@ TEST_BUILTINS_OBJS += test-advise.o\n TEST_BUILTINS_OBJS += test-bloom.o\n TEST_BUILTINS_OBJS += test-chmtime.o\n TEST_BUILTINS_OBJS += test-config.o\n+TEST_BUILTINS_OBJS += test-crontab.o\n TEST_BUILTINS_OBJS += test-ctype.o\n TEST_BUILTINS_OBJS += test-date.o\n TEST_BUILTINS_OBJS += test-delta.o\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex edf1d35ce5..a387f46585 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -31,6 +31,7 @@\n #include \"refs.h\"\n #include \"remote.h\"\n #include \"object-store.h\"\n+#include \"exec-cmd.h\"\n \n #define FAILED_RUN \"failed to run %s\"\n \n@@ -1456,6 +1457,125 @@ static int maintenance_unregister(void)\n \treturn run_command(&config_unset);\n }\n \n+#define BEGIN_LINE \"# BEGIN GIT MAINTENANCE SCHEDULE\"\n+#define END_LINE \"# END GIT MAINTENANCE SCHEDULE\"\n+\n+static int update_background_schedule(int run_maintenance)\n+{\n+\tint result = 0;\n+\tint in_old_region = 0;\n+\tstruct child_process crontab_list = CHILD_PROCESS_INIT;\n+\tstruct child_process crontab_edit = CHILD_PROCESS_INIT;\n+\tFILE *cron_list, *cron_in;\n+\tconst char *crontab_name;\n+\tstruct strbuf line = STRBUF_INIT;\n+\tstruct lock_file lk;\n+\tchar *lock_path = xstrfmt(\"%s/schedule\", the_repository->objects->odb->path);\n+\n+\tif (hold_lock_file_for_update(&lk, lock_path, LOCK_NO_DEREF) < 0)\n+\t\treturn error(_(\"another process is scheduling background maintenance\"));\n+\n+\tcrontab_name = getenv(\"GIT_TEST_CRONTAB\");\n+\tif (!crontab_name)\n+\t\tcrontab_name = \"crontab\";\n+\n+\tstrvec_split(&crontab_list.args, crontab_name);\n+\tstrvec_push(&crontab_list.args, \"-l\");\n+\tcrontab_list.in = -1;\n+\tcrontab_list.out = dup(lk.tempfile->fd);\n+\tcrontab_list.git_cmd = 0;\n+\n+\tif (start_command(&crontab_list)) {\n+\t\tresult = error(_(\"failed to run 'crontab -l'; your system might not support 'cron'\"));\n+\t\tgoto cleanup;\n+\t}\n+\n+\t/* Ignore exit code, as an empty crontab will return error. */\n+\tfinish_command(&crontab_list);\n+\n+\t/*\n+\t * Read from the .lock file, filtering out the old\n+\t * schedule while appending the new schedule.\n+\t */\n+\tcron_list = fdopen(lk.tempfile->fd, \"r\");\n+\trewind(cron_list);\n+\n+\tstrvec_split(&crontab_edit.args, crontab_name);\n+\tcrontab_edit.in = -1;\n+\tcrontab_edit.git_cmd = 0;\n+\n+\tif (start_command(&crontab_edit)) {\n+\t\tresult = error(_(\"failed to run 'crontab'; your system might not support 'cron'\"));\n+\t\tgoto cleanup;\n+\t}\n+\n+\tcron_in = fdopen(crontab_edit.in, \"w\");\n+\tif (!cron_in) {\n+\t\tresult = error(_(\"failed to open stdin of 'crontab'\"));\n+\t\tgoto done_editing;\n+\t}\n+\n+\twhile (!strbuf_getline_lf(&line, cron_list)) {\n+\t\tif (!in_old_region && !strcmp(line.buf, BEGIN_LINE))\n+\t\t\tin_old_region = 1;\n+\t\tif (in_old_region)\n+\t\t\tcontinue;\n+\t\tfprintf(cron_in, \"%s\\n\", line.buf);\n+\t\tif (in_old_region && !strcmp(line.buf, END_LINE))\n+\t\t\tin_old_region = 0;\n+\t}\n+\n+\tif (run_maintenance) {\n+\t\tstruct strbuf line_format = STRBUF_INIT;\n+\t\tconst char *exec_path = git_exec_path();\n+\n+\t\tfprintf(cron_in, \"%s\\n\", BEGIN_LINE);\n+\t\tfprintf(cron_in,\n+\t\t\t\"# The following schedule was created by Git\\n\");\n+\t\tfprintf(cron_in, \"# Any edits made in this region might be\\n\");\n+\t\tfprintf(cron_in,\n+\t\t\t\"# replaced in the future by a Git command.\\n\\n\");\n+\n+\t\tstrbuf_addf(&line_format,\n+\t\t\t \"%%s %%s * * %%s \\\"%s/git\\\" --exec-path=\\\"%s\\\" for-each-repo --config=maintenance.repo maintenance run --schedule=%%s\\n\",\n+\t\t\t exec_path, exec_path);\n+\t\tfprintf(cron_in, line_format.buf, \"0\", \"1-23\", \"*\", \"hourly\");\n+\t\tfprintf(cron_in, line_format.buf, \"0\", \"0\", \"1-6\", \"daily\");\n+\t\tfprintf(cron_in, line_format.buf, \"0\", \"0\", \"0\", \"weekly\");\n+\t\tstrbuf_release(&line_format);\n+\n+\t\tfprintf(cron_in, \"\\n%s\\n\", END_LINE);\n+\t}\n+\n+\tfflush(cron_in);\n+\tfclose(cron_in);\n+\tclose(crontab_edit.in);\n+\n+done_editing:\n+\tif (finish_command(&crontab_edit)) {\n+\t\tresult = error(_(\"'crontab' died\"));\n+\t\tgoto cleanup;\n+\t}\n+\tfclose(cron_list);\n+\n+cleanup:\n+\trollback_lock_file(&lk);\n+\treturn result;\n+}\n+\n+static int maintenance_start(void)\n+{\n+\tif (maintenance_register())\n+\t\twarning(_(\"failed to add repo to global config\"));\n+\n+\treturn update_background_schedule(1);\n+}\n+\n+static int maintenance_stop(void)\n+{\n+\treturn update_background_schedule(0);\n+}\n+\n static const char builtin_maintenance_usage[] =\tN_(\"git maintenance []\");\n \n int cmd_maintenance(int argc, const char **argv, const char *prefix)\n@@ -1466,6 +1586,10 @@ int cmd_maintenance(int argc, const char **argv, const char *prefix)\n \n \tif (!strcmp(argv[1], \"run\"))\n \t\treturn maintenance_run(argc - 1, argv + 1, prefix);\n+\tif (!strcmp(argv[1], \"start\"))\n+\t\treturn maintenance_start();\n+\tif (!strcmp(argv[1], \"stop\"))\n+\t\treturn maintenance_stop();\n \tif (!strcmp(argv[1], \"register\"))\n \t\treturn maintenance_register();\n \tif (!strcmp(argv[1], \"unregister\"))\ndiff --git a/t/helper/test-crontab.c b/t/helper/test-crontab.c\nnew file mode 100644\nindex 0000000000..e7c0137a47\n--- /dev/null\n+++ b/t/helper/test-crontab.c\n@@ -0,0 +1,35 @@\n+#include \"test-tool.h\"\n+#include \"cache.h\"\n+\n+/*\n+ * Usage: test-tool cron [-l]\n+ *\n+ * If -l is specified, then write the contents of to stdout.\n+ * Otherwise, write from stdin into .\n+ */\n+int cmd__crontab(int argc, const char **argv)\n+{\n+\tint a;\n+\tFILE *from, *to;\n+\n+\tif (argc == 3 && !strcmp(argv[2], \"-l\")) {\n+\t\tfrom = fopen(argv[1], \"r\");\n+\t\tif (!from)\n+\t\t\treturn 0;\n+\t\tto = stdout;\n+\t} else if (argc == 2) {\n+\t\tfrom = stdin;\n+\t\tto = fopen(argv[1], \"w\");\n+\t} else\n+\t\treturn error(\"unknown arguments\");\n+\n+\twhile ((a = fgetc(from)) != EOF)\n+\t\tfputc(a, to);\n+\n+\tif (argc == 3)\n+\t\tfclose(from);\n+\telse\n+\t\tfclose(to);\n+\n+\treturn 0;\n+}\ndiff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\nindex 590b2efca7..432b49d948 100644\n--- a/t/helper/test-tool.c\n+++ b/t/helper/test-tool.c\n@@ -18,6 +18,7 @@ static struct test_cmd cmds[] = {\n \t{ \"bloom\", cmd__bloom },\n \t{ \"chmtime\", cmd__chmtime },\n \t{ \"config\", cmd__config },\n+\t{ \"crontab\", cmd__crontab },\n \t{ \"ctype\", cmd__ctype },\n \t{ \"date\", cmd__date },\n \t{ \"delta\", cmd__delta },\ndiff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\nindex ddc8e990e9..7c3281e071 100644\n--- a/t/helper/test-tool.h\n+++ b/t/helper/test-tool.h\n@@ -8,6 +8,7 @@ int cmd__advise_if_enabled(int argc, const char **argv);\n int cmd__bloom(int argc, const char **argv);\n int cmd__chmtime(int argc, const char **argv);\n int cmd__config(int argc, const char **argv);\n+int cmd__crontab(int argc, const char **argv);\n int cmd__ctype(int argc, const char **argv);\n int cmd__date(int argc, const char **argv);\n int cmd__delta(int argc, const char **argv);\ndiff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\nindex 8f383d01d9..7715e40391 100755\n--- a/t/t7900-maintenance.sh\n+++ b/t/t7900-maintenance.sh\n@@ -315,4 +315,32 @@ test_expect_success 'register and unregister' '\n \ttest_cmp before actual\n '\n \n+test_expect_success 'start from empty cron table' '\n+\tGIT_TEST_CRONTAB=\"test-tool crontab cron.txt\" git maintenance start &&\n+\n+\t# start registers the repo\n+\tgit config --get --global maintenance.repo \"$(pwd)\" &&\n+\n+\tgrep \"for-each-repo --config=maintenance.repo maintenance run --schedule=daily\" cron.txt &&\n+\tgrep \"for-each-repo --config=maintenance.repo maintenance run --schedule=hourly\" cron.txt &&\n+\tgrep \"for-each-repo --config=maintenance.repo maintenance run --schedule=weekly\" cron.txt\n+'\n+\n+test_expect_success 'stop from existing schedule' '\n+\tGIT_TEST_CRONTAB=\"test-tool crontab cron.txt\" git maintenance stop &&\n+\n+\t# stop does not unregister the repo\n+\tgit config --get --global maintenance.repo \"$(pwd)\" &&\n+\n+\t# Operation is idempotent\n+\tGIT_TEST_CRONTAB=\"test-tool crontab cron.txt\" git maintenance stop &&\n+\ttest_must_be_empty cron.txt\n+'\n+\n+test_expect_success 'start preserves existing schedule' '\n+\techo \"Important information!\" >cron.txt &&\n+\tGIT_TEST_CRONTAB=\"test-tool crontab cron.txt\" git maintenance start &&\n+\tgrep \"Important information!\" cron.txt\n+'\n+\n test_done\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex ef31f40037..4a60d1ed76 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -1702,3 +1702,9 @@ test_lazy_prereq SHA1 '\n test_lazy_prereq REBASE_P '\n \ttest -z \"$GIT_TEST_SKIP_REBASE_P\"\n '\n+\n+# Ensure that no test accidentally triggers a Git command\n+# that runs 'crontab', affecting a user's cron schedule.\n+# Tests that verify the cron integration must set this locally\n+# to avoid errors.\n+GIT_TEST_CRONTAB=\"exit 1\"\n-- \ngitgitgadget\n\n"},{"id":"407637","messageId":"145c63ed8c61126dd19dbcdbe3fed81c32862f0a.1602782524.git.gitgitgadget@gmail.com","threadId":"54189","inReplyTo":"pull.724.v4.git.1602782524.gitgitgadget@gmail.com","subject":"[PATCH v4 7/8] maintenance: use 'incremental' strategy by default","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-10-15T17:22:03Z","receivedAt":"2020-10-15T17:22:17Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe 'git maintenance (register|start)' subcommands add the current\nrepository to the global Git config so maintenance will operate on that\nrepository. It does not specify what maintenance should occur or how\noften.\n\nTo make it simple for users to start background maintenance with a\nrecommended schedlue, update the 'maintenance.strategy' config option in\nboth the 'register' and 'start' subcommands. This allows users to\ncustomize beyond the defaults using individual\n'maintenance..schedule' options, but also the user can opt-out of\nthis strategy using 'maintenance.strategy=none'.\n\nHelped-by: Martin Ågren \nSigned-off-by: Derrick Stolee \n---\n Documentation/git-maintenance.txt | 17 +++++++++++++++++\n builtin/gc.c | 10 ++++++++++\n t/t7900-maintenance.sh | 20 ++++++++++++++++----\n 3 files changed, 43 insertions(+), 4 deletions(-)\n\ndiff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt\nindex 7628a6d157..b5944b4c51 100644\n--- a/Documentation/git-maintenance.txt\n+++ b/Documentation/git-maintenance.txt\n@@ -37,6 +37,23 @@ register::\n \t`maintenance..schedule`. The tasks that are enabled are safe\n \tfor running in the background without disrupting foreground\n \tprocesses.\n++\n+The `register` subcomand will also set the `maintenance.strategy` config\n+value to `incremental`, if this value is not previously set. The\n+`incremental` strategy uses the following schedule for each maintenance\n+task:\n++\n+--\n+* `gc`: disabled.\n+* `commit-graph`: hourly.\n+* `prefetch`: hourly.\n+* `loose-objects`: daily.\n+* `incremental-repack`: daily.\n+--\n++\n+`git maintenance register` will also disable foreground maintenance by\n+setting `maintenance.auto = false` in the current repository. This config\n+setting will remain after a `git maintenance unregister` command.\n \n run::\n \tRun one or more maintenance tasks. If one or more `--task` options\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex a8248e7a45..e3098ef6a1 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -1434,6 +1434,7 @@ static int maintenance_run(int argc, const char **argv, const char *prefix)\n \n static int maintenance_register(void)\n {\n+\tchar *config_value;\n \tstruct child_process config_set = CHILD_PROCESS_INIT;\n \tstruct child_process config_get = CHILD_PROCESS_INIT;\n \n@@ -1441,6 +1442,15 @@ static int maintenance_register(void)\n \tif (!the_repository || !the_repository->gitdir)\n \t\treturn 0;\n \n+\t/* Disable foreground maintenance */\n+\tgit_config_set(\"maintenance.auto\", \"false\");\n+\n+\t/* Set maintenance strategy, if unset */\n+\tif (!git_config_get_string(\"maintenance.strategy\", &config_value))\n+\t\tfree(config_value);\n+\telse\n+\t\tgit_config_set(\"maintenance.strategy\", \"incremental\");\n+\n \tconfig_get.git_cmd = 1;\n \tstrvec_pushl(&config_get.args, \"config\", \"--global\", \"--get\", \"maintenance.repo\",\n \t\t the_repository->worktree ? the_repository->worktree\ndiff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\nindex 7440a0ea19..20184e96e1 100755\n--- a/t/t7900-maintenance.sh\n+++ b/t/t7900-maintenance.sh\n@@ -354,11 +354,14 @@ test_expect_success 'register and unregister' '\n \tgit config --global --add maintenance.repo /existing1 &&\n \tgit config --global --add maintenance.repo /existing2 &&\n \tgit config --global --get-all maintenance.repo >before &&\n+\n \tgit maintenance register &&\n-\tgit config --global --get-all maintenance.repo >actual &&\n-\tcp before after &&\n-\tpwd >>after &&\n-\ttest_cmp after actual &&\n+\ttest_cmp_config false maintenance.auto &&\n+\tgit config --global --get-all maintenance.repo >between &&\n+\tcp before expect &&\n+\tpwd >>expect &&\n+\ttest_cmp expect between &&\n+\n \tgit maintenance unregister &&\n \tgit config --global --get-all maintenance.repo >actual &&\n \ttest_cmp before actual\n@@ -392,4 +395,13 @@ test_expect_success 'start preserves existing schedule' '\n \tgrep \"Important information!\" cron.txt\n '\n \n+test_expect_success 'register preserves existing strategy' '\n+\tgit config maintenance.strategy none &&\n+\tgit maintenance register &&\n+\ttest_config maintenance.strategy none &&\n+\tgit config --unset maintenance.strategy &&\n+\tgit maintenance register &&\n+\ttest_config maintenance.strategy incremental\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"407638","messageId":"dae8c04bb5523c9b63c770862a1104a0ff4aa6c4.1602782524.git.gitgitgadget@gmail.com","threadId":"54189","inReplyTo":"pull.724.v4.git.1602782524.gitgitgadget@gmail.com","subject":"[PATCH v4 2/8] maintenance: add --schedule option and config","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-10-15T17:21:58Z","receivedAt":"2020-10-15T17:22:17Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nMaintenance currently triggers when certain data-size thresholds are\nmet, such as number of pack-files or loose objects. Users may want to\nrun certain maintenance tasks based on frequency instead. For example,\na user may want to perform a 'prefetch' task every hour, or 'gc' task\nevery day. To help these users, update the 'git maintenance run' command\nto include a '--schedule=' option. The allowed frequencies\nare 'hourly', 'daily', and 'weekly'. These values are also allowed in a\nnew config value 'maintenance..schedule'.\n\nThe 'git maintenance run --schedule=' checks the '*.schedule'\nconfig value for each enabled task to see if the configured frequency is\nat least as frequent as the frequency from the '--schedule' argument. We\nuse the following order, for full clarity:\n\n\t'hourly' > 'daily' > 'weekly'\n\nUse new 'enum schedule_priority' to track these values numerically.\n\nThe following cron table would run the scheduled tasks with the correct\nfrequencies:\n\n 0 1-23 * * * git -C maintenance run --schedule=hourly\n 0 0 * * 1-6 git -C maintenance run --schedule=daily\n 0 0 * * 0 git -C maintenance run --schedule=weekly\n\nThis cron schedule will run --schedule=hourly every hour except at\nmidnight. This avoids a concurrent run with the --schedule=daily that\nruns at midnight every day except the first day of the week. This avoids\na concurrent run with the --schedule=weekly that runs at midnight on\nthe first day of the week. Since --schedule=daily also runs the\n'hourly' tasks and --schedule=weekly runs the 'hourly' and 'daily'\ntasks, we will still see all tasks run with the proper frequencies.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/config/maintenance.txt | 5 +++\n Documentation/git-maintenance.txt | 13 +++++-\n builtin/gc.c | 64 ++++++++++++++++++++++++++--\n t/t7900-maintenance.sh | 40 +++++++++++++++++\n 4 files changed, 118 insertions(+), 4 deletions(-)\n\ndiff --git a/Documentation/config/maintenance.txt b/Documentation/config/maintenance.txt\nindex 06db758172..70585564fa 100644\n--- a/Documentation/config/maintenance.txt\n+++ b/Documentation/config/maintenance.txt\n@@ -10,6 +10,11 @@ maintenance..enabled::\n \t`--task` option exists. By default, only `maintenance.gc.enabled`\n \tis true.\n \n+maintenance..schedule::\n+\tThis config option controls whether or not the given `` runs\n+\tduring a `git maintenance run --schedule=` command. The\n+\tvalue must be one of \"hourly\", \"daily\", or \"weekly\".\n+\n maintenance.commit-graph.auto::\n \tThis integer config option controls how often the `commit-graph` task\n \tshould be run as part of `git maintenance run --auto`. If zero, then\ndiff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt\nindex 3f5d8946b4..ed94f66e36 100644\n--- a/Documentation/git-maintenance.txt\n+++ b/Documentation/git-maintenance.txt\n@@ -110,7 +110,18 @@ OPTIONS\n \tonly if certain thresholds are met. For example, the `gc` task\n \truns when the number of loose objects exceeds the number stored\n \tin the `gc.auto` config setting, or when the number of pack-files\n-\texceeds the `gc.autoPackLimit` config setting.\n+\texceeds the `gc.autoPackLimit` config setting. Not compatible with\n+\tthe `--schedule` option.\n+\n+--schedule::\n+\tWhen combined with the `run` subcommand, run maintenance tasks\n+\tonly if certain time conditions are met, as specified by the\n+\t`maintenance..schedule` config value for each ``.\n+\tThis config value specifies a number of seconds since the last\n+\ttime that task ran, according to the `maintenance..lastRun`\n+\tconfig value. The tasks that are tested are those provided by\n+\tthe `--task=` option(s) or those with\n+\t`maintenance..enabled` set to true.\n \n --quiet::\n \tDo not report progress or other information over `stderr`.\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 2b99596ec8..03b24ea0db 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -703,14 +703,51 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \treturn 0;\n }\n \n-static const char * const builtin_maintenance_run_usage[] = {\n-\tN_(\"git maintenance run [--auto] [--[no-]quiet] [--task=]\"),\n+static const char *const builtin_maintenance_run_usage[] = {\n+\tN_(\"git maintenance run [--auto] [--[no-]quiet] [--task=] [--schedule]\"),\n \tNULL\n };\n \n+enum schedule_priority {\n+\tSCHEDULE_NONE = 0,\n+\tSCHEDULE_WEEKLY = 1,\n+\tSCHEDULE_DAILY = 2,\n+\tSCHEDULE_HOURLY = 3,\n+};\n+\n+static enum schedule_priority parse_schedule(const char *value)\n+{\n+\tif (!value)\n+\t\treturn SCHEDULE_NONE;\n+\tif (!strcasecmp(value, \"hourly\"))\n+\t\treturn SCHEDULE_HOURLY;\n+\tif (!strcasecmp(value, \"daily\"))\n+\t\treturn SCHEDULE_DAILY;\n+\tif (!strcasecmp(value, \"weekly\"))\n+\t\treturn SCHEDULE_WEEKLY;\n+\treturn SCHEDULE_NONE;\n+}\n+\n+static int maintenance_opt_schedule(const struct option *opt, const char *arg,\n+\t\t\t\t int unset)\n+{\n+\tenum schedule_priority *priority = opt->value;\n+\n+\tif (unset)\n+\t\tdie(_(\"--no-schedule is not allowed\"));\n+\n+\t*priority = parse_schedule(arg);\n+\n+\tif (!*priority)\n+\t\tdie(_(\"unrecognized --schedule argument '%s'\"), arg);\n+\n+\treturn 0;\n+}\n+\n struct maintenance_run_opts {\n \tint auto_flag;\n \tint quiet;\n+\tenum schedule_priority schedule;\n };\n \n /* Remember to update object flag allocation in object.h */\n@@ -1158,6 +1195,8 @@ struct maintenance_task {\n \tmaintenance_auto_fn *auto_condition;\n \tunsigned enabled:1;\n \n+\tenum schedule_priority schedule;\n+\n \t/* -1 if not selected. */\n \tint selected_order;\n };\n@@ -1253,6 +1292,9 @@ static int maintenance_run_tasks(struct maintenance_run_opts *opts)\n \t\t !tasks[i].auto_condition()))\n \t\t\tcontinue;\n \n+\t\tif (opts->schedule && tasks[i].schedule < opts->schedule)\n+\t\t\tcontinue;\n+\n \t\ttrace2_region_enter(\"maintenance\", tasks[i].name, r);\n \t\tif (tasks[i].fn(opts)) {\n \t\t\terror(_(\"task '%s' failed\"), tasks[i].name);\n@@ -1273,13 +1315,23 @@ static void initialize_task_config(void)\n \n \tfor (i = 0; i < TASK__COUNT; i++) {\n \t\tint config_value;\n+\t\tchar *config_str;\n \n-\t\tstrbuf_setlen(&config_name, 0);\n+\t\tstrbuf_reset(&config_name);\n \t\tstrbuf_addf(&config_name, \"maintenance.%s.enabled\",\n \t\t\t tasks[i].name);\n \n \t\tif (!git_config_get_bool(config_name.buf, &config_value))\n \t\t\ttasks[i].enabled = config_value;\n+\n+\t\tstrbuf_reset(&config_name);\n+\t\tstrbuf_addf(&config_name, \"maintenance.%s.schedule\",\n+\t\t\t tasks[i].name);\n+\n+\t\tif (!git_config_get_string(config_name.buf, &config_str)) {\n+\t\t\ttasks[i].schedule = parse_schedule(config_str);\n+\t\t\tfree(config_str);\n+\t\t}\n \t}\n \n \tstrbuf_release(&config_name);\n@@ -1323,6 +1375,9 @@ static int maintenance_run(int argc, const char **argv, const char *prefix)\n \tstruct option builtin_maintenance_run_options[] = {\n \t\tOPT_BOOL(0, \"auto\", &opts.auto_flag,\n \t\t\t N_(\"run tasks based on the state of the repository\")),\n+\t\tOPT_CALLBACK(0, \"schedule\", &opts.schedule, N_(\"frequency\"),\n+\t\t\t N_(\"run tasks based on frequency\"),\n+\t\t\t maintenance_opt_schedule),\n \t\tOPT_BOOL(0, \"quiet\", &opts.quiet,\n \t\t\t N_(\"do not report progress or other information over stderr\")),\n \t\tOPT_CALLBACK_F(0, \"task\", NULL, N_(\"task\"),\n@@ -1343,6 +1398,9 @@ static int maintenance_run(int argc, const char **argv, const char *prefix)\n \t\t\t builtin_maintenance_run_usage,\n \t\t\t PARSE_OPT_STOP_AT_NON_OPTION);\n \n+\tif (opts.auto_flag && opts.schedule)\n+\t\tdie(_(\"use at most one of --auto and --schedule=\"));\n+\n \tif (argc != 0)\n \t\tusage_with_options(builtin_maintenance_run_usage,\n \t\t\t\t builtin_maintenance_run_options);\ndiff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\nindex c7caaa7a55..33d73cd01c 100755\n--- a/t/t7900-maintenance.sh\n+++ b/t/t7900-maintenance.sh\n@@ -260,4 +260,44 @@ test_expect_success 'maintenance.incremental-repack.auto' '\n \ttest_subcommand git multi-pack-index write --no-progress err &&\n+\ttest_i18ngrep \"at most one\" err\n+'\n+\n+test_expect_success 'invalid --schedule value' '\n+\ttest_must_fail git maintenance run --schedule=annually 2>err &&\n+\ttest_i18ngrep \"unrecognized --schedule\" err\n+'\n+\n+test_expect_success '--schedule inheritance weekly -> daily -> hourly' '\n+\tgit config maintenance.loose-objects.enabled true &&\n+\tgit config maintenance.loose-objects.schedule hourly &&\n+\tgit config maintenance.commit-graph.enabled true &&\n+\tgit config maintenance.commit-graph.schedule daily &&\n+\tgit config maintenance.incremental-repack.enabled true &&\n+\tgit config maintenance.incremental-repack.schedule weekly &&\n+\n+\tGIT_TRACE2_EVENT=\"$(pwd)/hourly.txt\" \\\n+\t\tgit maintenance run --schedule=hourly 2>/dev/null &&\n+\ttest_subcommand git prune-packed --quiet /dev/null &&\n+\ttest_subcommand git prune-packed --quiet /dev/null &&\n+\ttest_subcommand git prune-packed --quiet \n\nTo provide an on-ramp for users to use background maintenance without\nseveral 'git config' commands, create a 'maintenance.strategy' config\noption. Currently, the only important value is 'incremental' which\nassigns the following schedule:\n\n* gc: never\n* prefetch: hourly\n* commit-graph: hourly\n* loose-objects: daily\n* incremental-repack: daily\n\nThese tasks are chosen to minimize disruptions to foreground Git\ncommands and use few compute resources.\n\nThe 'maintenance.strategy' is intended as a baseline that can be\ncustomzied further by manually assigning 'maintenance..enabled'\nand 'maintenance..schedule' config options, which will override\nany recommendation from 'maintenance.strategy'. This operates similarly\nto config options like 'feature.experimental' which operate as \"meta\"\nconfig options that change default config values.\n\nThis presents a way forward for updating the 'incremental' strategy in\nthe future or adding new strategies. For example, a potential strategy\ncould be to include a 'full' strategy that runs the 'gc' task weekly\nand no other tasks by default.\n\nHelped-by: Martin Ågren \nSigned-off-by: Derrick Stolee \n---\n Documentation/config/maintenance.txt | 15 +++++++++\n builtin/gc.c | 28 ++++++++++++++--\n t/t7900-maintenance.sh | 49 ++++++++++++++++++++++++++++\n 3 files changed, 90 insertions(+), 2 deletions(-)\n\ndiff --git a/Documentation/config/maintenance.txt b/Documentation/config/maintenance.txt\nindex 70585564fa..a5ead09e4b 100644\n--- a/Documentation/config/maintenance.txt\n+++ b/Documentation/config/maintenance.txt\n@@ -3,6 +3,21 @@ maintenance.auto::\n \t`git maintenance run --auto` after doing their normal work. Defaults\n \tto true.\n \n+maintenance.strategy::\n+\tThis string config option provides a way to specify one of a few\n+\trecommended schedules for background maintenance. This only affects\n+\twhich tasks are run during `git maintenance run --schedule=X`\n+\tcommands, provided no `--task=` arguments are provided.\n+\tFurther, if a `maintenance..schedule` config value is set,\n+\tthen that value is used instead of the one provided by\n+\t`maintenance.strategy`. The possible strategy strings are:\n++\n+* `none`: This default setting implies no task are run at any schedule.\n+* `incremental`: This setting optimizes for performing small maintenance\n+ activities that do not delete any data. This does not schedule the `gc`\n+ task, but runs the `prefetch` and `commit-graph` tasks hourly and the\n+ `loose-objects` and `incremental-repack` tasks daily.\n+\n maintenance..enabled::\n \tThis boolean config option controls whether the maintenance task\n \twith name `` is run when no `--task` option is specified to\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex a387f46585..a8248e7a45 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -1308,12 +1308,35 @@ static int maintenance_run_tasks(struct maintenance_run_opts *opts)\n \treturn result;\n }\n \n-static void initialize_task_config(void)\n+static void initialize_maintenance_strategy(void)\n+{\n+\tchar *config_str;\n+\n+\tif (git_config_get_string(\"maintenance.strategy\", &config_str))\n+\t\treturn;\n+\n+\tif (!strcasecmp(config_str, \"incremental\")) {\n+\t\ttasks[TASK_GC].schedule = SCHEDULE_NONE;\n+\t\ttasks[TASK_COMMIT_GRAPH].enabled = 1;\n+\t\ttasks[TASK_COMMIT_GRAPH].schedule = SCHEDULE_HOURLY;\n+\t\ttasks[TASK_PREFETCH].enabled = 1;\n+\t\ttasks[TASK_PREFETCH].schedule = SCHEDULE_HOURLY;\n+\t\ttasks[TASK_INCREMENTAL_REPACK].enabled = 1;\n+\t\ttasks[TASK_INCREMENTAL_REPACK].schedule = SCHEDULE_DAILY;\n+\t\ttasks[TASK_LOOSE_OBJECTS].enabled = 1;\n+\t\ttasks[TASK_LOOSE_OBJECTS].schedule = SCHEDULE_DAILY;\n+\t}\n+}\n+\n+static void initialize_task_config(int schedule)\n {\n \tint i;\n \tstruct strbuf config_name = STRBUF_INIT;\n \tgc_config();\n \n+\tif (schedule)\n+\t\tinitialize_maintenance_strategy();\n+\n \tfor (i = 0; i < TASK__COUNT; i++) {\n \t\tint config_value;\n \t\tchar *config_str;\n@@ -1389,7 +1412,6 @@ static int maintenance_run(int argc, const char **argv, const char *prefix)\n \tmemset(&opts, 0, sizeof(opts));\n \n \topts.quiet = !isatty(2);\n-\tinitialize_task_config();\n \n \tfor (i = 0; i < TASK__COUNT; i++)\n \t\ttasks[i].selected_order = -1;\n@@ -1402,6 +1424,8 @@ static int maintenance_run(int argc, const char **argv, const char *prefix)\n \tif (opts.auto_flag && opts.schedule)\n \t\tdie(_(\"use at most one of --auto and --schedule=\"));\n \n+\tinitialize_task_config(opts.schedule);\n+\n \tif (argc != 0)\n \t\tusage_with_options(builtin_maintenance_run_usage,\n \t\t\t\t builtin_maintenance_run_options);\ndiff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\nindex 7715e40391..7440a0ea19 100755\n--- a/t/t7900-maintenance.sh\n+++ b/t/t7900-maintenance.sh\n@@ -300,6 +300,55 @@ test_expect_success '--schedule inheritance weekly -> daily -> hourly' '\n \ttest_subcommand git multi-pack-index write --no-progress \n\nIn preparation for launching background maintenance from the 'git\nmaintenance' builtin, create register/unregister subcommands. These\ncommands update the new 'maintenance.repos' config option in the global\nconfig so the background maintenance job knows which repositories to\nmaintain.\n\nThese commands allow users to add a repository to the background\nmaintenance list without disrupting the actual maintenance mechanism.\n\nFor example, a user can run 'git maintenance register' when no\nbackground maintenance is running and it will not start the background\nmaintenance. A later update to start running background maintenance will\nthen pick up this repository automatically.\n\nThe opposite example is that a user can run 'git maintenance unregister'\nto remove the current repository from background maintenance without\nhalting maintenance for other repositories.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-maintenance.txt | 14 ++++++++\n builtin/gc.c | 55 ++++++++++++++++++++++++++++++-\n t/t7900-maintenance.sh | 17 +++++++++-\n 3 files changed, 84 insertions(+), 2 deletions(-)\n\ndiff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt\nindex ed94f66e36..1c59fd0cb5 100644\n--- a/Documentation/git-maintenance.txt\n+++ b/Documentation/git-maintenance.txt\n@@ -29,6 +29,15 @@ Git repository.\n SUBCOMMANDS\n -----------\n \n+register::\n+\tInitialize Git config values so any scheduled maintenance will\n+\tstart running on this repository. This adds the repository to the\n+\t`maintenance.repo` config variable in the current user's global\n+\tconfig and enables some recommended configuration values for\n+\t`maintenance..schedule`. The tasks that are enabled are safe\n+\tfor running in the background without disrupting foreground\n+\tprocesses.\n+\n run::\n \tRun one or more maintenance tasks. If one or more `--task` options\n \tare specified, then those tasks are run in that order. Otherwise,\n@@ -36,6 +45,11 @@ run::\n \tconfig options are true. By default, only `maintenance.gc.enabled`\n \tis true.\n \n+unregister::\n+\tRemove the current repository from background maintenance. This\n+\tonly removes the repository from the configured list. It does not\n+\tstop the background maintenance processes from running.\n+\n TASKS\n -----\n \ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 03b24ea0db..edf1d35ce5 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -1407,7 +1407,56 @@ static int maintenance_run(int argc, const char **argv, const char *prefix)\n \treturn maintenance_run_tasks(&opts);\n }\n \n-static const char builtin_maintenance_usage[] = N_(\"git maintenance run []\");\n+static int maintenance_register(void)\n+{\n+\tstruct child_process config_set = CHILD_PROCESS_INIT;\n+\tstruct child_process config_get = CHILD_PROCESS_INIT;\n+\n+\t/* There is no current repository, so skip registering it */\n+\tif (!the_repository || !the_repository->gitdir)\n+\t\treturn 0;\n+\n+\tconfig_get.git_cmd = 1;\n+\tstrvec_pushl(&config_get.args, \"config\", \"--global\", \"--get\", \"maintenance.repo\",\n+\t\t the_repository->worktree ? the_repository->worktree\n+\t\t\t\t\t : the_repository->gitdir,\n+\t\t\t NULL);\n+\tconfig_get.out = -1;\n+\n+\tif (start_command(&config_get))\n+\t\treturn error(_(\"failed to run 'git config'\"));\n+\n+\t/* We already have this value in our config! */\n+\tif (!finish_command(&config_get))\n+\t\treturn 0;\n+\n+\tconfig_set.git_cmd = 1;\n+\tstrvec_pushl(&config_set.args, \"config\", \"--add\", \"--global\", \"maintenance.repo\",\n+\t\t the_repository->worktree ? the_repository->worktree\n+\t\t\t\t\t : the_repository->gitdir,\n+\t\t NULL);\n+\n+\treturn run_command(&config_set);\n+}\n+\n+static int maintenance_unregister(void)\n+{\n+\tstruct child_process config_unset = CHILD_PROCESS_INIT;\n+\n+\tif (!the_repository || !the_repository->gitdir)\n+\t\treturn error(_(\"no current repository to unregister\"));\n+\n+\tconfig_unset.git_cmd = 1;\n+\tstrvec_pushl(&config_unset.args, \"config\", \"--global\", \"--unset\",\n+\t\t \"maintenance.repo\",\n+\t\t the_repository->worktree ? the_repository->worktree\n+\t\t\t\t\t : the_repository->gitdir,\n+\t\t NULL);\n+\n+\treturn run_command(&config_unset);\n+}\n+\n+static const char builtin_maintenance_usage[] =\tN_(\"git maintenance []\");\n \n int cmd_maintenance(int argc, const char **argv, const char *prefix)\n {\n@@ -1417,6 +1466,10 @@ int cmd_maintenance(int argc, const char **argv, const char *prefix)\n \n \tif (!strcmp(argv[1], \"run\"))\n \t\treturn maintenance_run(argc - 1, argv + 1, prefix);\n+\tif (!strcmp(argv[1], \"register\"))\n+\t\treturn maintenance_register();\n+\tif (!strcmp(argv[1], \"unregister\"))\n+\t\treturn maintenance_unregister();\n \n \tdie(_(\"invalid subcommand: %s\"), argv[1]);\n }\ndiff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\nindex 33d73cd01c..8f383d01d9 100755\n--- a/t/t7900-maintenance.sh\n+++ b/t/t7900-maintenance.sh\n@@ -9,7 +9,7 @@ GIT_TEST_MULTI_PACK_INDEX=0\n \n test_expect_success 'help text' '\n \ttest_expect_code 129 git maintenance -h 2>err &&\n-\ttest_i18ngrep \"usage: git maintenance run\" err &&\n+\ttest_i18ngrep \"usage: git maintenance \" err &&\n \ttest_expect_code 128 git maintenance barf 2>err &&\n \ttest_i18ngrep \"invalid subcommand: barf\" err &&\n \ttest_expect_code 129 git maintenance 2>err &&\n@@ -300,4 +300,19 @@ test_expect_success '--schedule inheritance weekly -> daily -> hourly' '\n \ttest_subcommand git multi-pack-index write --no-progress before &&\n+\tgit maintenance register &&\n+\tgit config --global --get-all maintenance.repo >actual &&\n+\tcp before after &&\n+\tpwd >>after &&\n+\ttest_cmp after actual &&\n+\tgit maintenance unregister &&\n+\tgit config --global --get-all maintenance.repo >actual &&\n+\ttest_cmp before actual\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"407641","messageId":"ce0ced705fb48ef0afc62caa3c2325d3fd4c1d55.1602782524.git.gitgitgadget@gmail.com","threadId":"54189","inReplyTo":"pull.724.v4.git.1602782524.gitgitgadget@gmail.com","subject":"[PATCH v4 8/8] maintenance: add troubleshooting guide to docs","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-10-15T17:22:04Z","receivedAt":"2020-10-15T17:22:30Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe 'git maintenance run' subcommand takes a lock on the object database\nto prevent concurrent processes from competing for resources. This is an\nimportant safety measure to prevent possible repository corruption and\ndata loss.\n\nThis feature can lead to confusing behavior if a user is not aware of\nit. Add a TROUBLESHOOTING section to the 'git maintenance' builtin\ndocumentation that discusses these tradeoffs. The short version of this\nsection is that Git will not corrupt your repository, but if the list of\nscheduled tasks takes longer than an hour then some scheduled tasks may\nbe dropped due to this object database collision. For example, a\nlong-running \"daily\" task at midnight might prevent an \"hourly\" task\nfrom running at 1AM.\n\nThe opposite is also possible, but less likely as long as the \"hourly\"\ntasks are much faster than the \"daily\" and \"weekly\" tasks.\n\nHelped-by: Junio C Hamano \nSigned-off-by: Derrick Stolee \n---\n Documentation/git-maintenance.txt | 44 +++++++++++++++++++++++++++++++\n 1 file changed, 44 insertions(+)\n\ndiff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt\nindex b5944b4c51..6fec1eb8dc 100644\n--- a/Documentation/git-maintenance.txt\n+++ b/Documentation/git-maintenance.txt\n@@ -175,6 +175,50 @@ OPTIONS\n \t`maintenance..enabled` configured as `true` are considered.\n \tSee the 'TASKS' section for the list of accepted `` values.\n \n+\n+TROUBLESHOOTING\n+---------------\n+The `git maintenance` command is designed to simplify the repository\n+maintenance patterns while minimizing user wait time during Git commands.\n+A variety of configuration options are available to allow customizing this\n+process. The default maintenance options focus on operations that complete\n+quickly, even on large repositories.\n+\n+Users may find some cases where scheduled maintenance tasks do not run as\n+frequently as intended. Each `git maintenance run` command takes a lock on\n+the repository's object database, and this prevents other concurrent\n+`git maintenance run` commands from running on the same repository. Without\n+this safeguard, competing processes could leave the repository in an\n+unpredictable state.\n+\n+The background maintenance schedule runs `git maintenance run` processes\n+on an hourly basis. Each run executes the \"hourly\" tasks. At midnight,\n+that process also executes the \"daily\" tasks. At midnight on the first day\n+of the week, that process also executes the \"weekly\" tasks. A single\n+process iterates over each registered repository, performing the scheduled\n+tasks for that frequency. Depending on the number of registered\n+repositories and their sizes, this process may take longer than an hour.\n+In this case, multiple `git maintenance run` commands may run on the same\n+repository at the same time, colliding on the object database lock. This\n+results in one of the two tasks not running.\n+\n+If you find that some maintenance windows are taking longer than one hour\n+to complete, then consider reducing the complexity of your maintenance\n+tasks. For example, the `gc` task is much slower than the\n+`incremental-repack` task. However, this comes at a cost of a slightly\n+larger object database. Consider moving more expensive tasks to be run\n+less frequently.\n+\n+Expert users may consider scheduling their own maintenance tasks using a\n+different schedule than is available through `git maintenance start` and\n+Git configuration options. These users should be aware of the object\n+database lock and how concurrent `git maintenance run` commands behave.\n+Further, the `git gc` command should not be combined with\n+`git maintenance run` commands. `git gc` modifies the object database\n+but does not take the lock in the same way as `git maintenance run`. If\n+possible, use `git maintenance run --task=gc` instead of `git gc`.\n+\n+\n GIT\n ---\n Part of the linkgit:git[1] suite\n-- \ngitgitgadget\n"},{"id":"411893","messageId":"20201209185114.GN36751@google.com","threadId":"54189","inReplyTo":"5194f6b1facbd14cc17eea0337c0cc397a2a51fc.1602782524.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 5/8] maintenance: add start/stop subcommands","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2020-12-09T18:51:14Z","receivedAt":"2020-12-09T18:52:18Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"On 2020.10.15 17:22, Derrick Stolee via GitGitGadget wrote:\n> diff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\n> index 8f383d01d9..7715e40391 100755\n> --- a/t/t7900-maintenance.sh\n> +++ b/t/t7900-maintenance.sh\n> @@ -315,4 +315,32 @@ test_expect_success 'register and unregister' '\n> \ttest_cmp before actual\n> '\n> \n> +test_expect_success 'start from empty cron table' '\n> +\tGIT_TEST_CRONTAB=\"test-tool crontab cron.txt\" git maintenance start &&\n> +\n> +\t# start registers the repo\n> +\tgit config --get --global maintenance.repo \"$(pwd)\" &&\n> +\n> +\tgrep \"for-each-repo --config=maintenance.repo maintenance run --schedule=daily\" cron.txt &&\n> +\tgrep \"for-each-repo --config=maintenance.repo maintenance run --schedule=hourly\" cron.txt &&\n> +\tgrep \"for-each-repo --config=maintenance.repo maintenance run --schedule=weekly\" cron.txt\n> +'\n> +\n> +test_expect_success 'stop from existing schedule' '\n> +\tGIT_TEST_CRONTAB=\"test-tool crontab cron.txt\" git maintenance stop &&\n> +\n> +\t# stop does not unregister the repo\n> +\tgit config --get --global maintenance.repo \"$(pwd)\" &&\n> +\n> +\t# Operation is idempotent\n> +\tGIT_TEST_CRONTAB=\"test-tool crontab cron.txt\" git maintenance stop &&\n> +\ttest_must_be_empty cron.txt\n> +'\n> +\n> +test_expect_success 'start preserves existing schedule' '\n> +\techo \"Important information!\" >cron.txt &&\n> +\tGIT_TEST_CRONTAB=\"test-tool crontab cron.txt\" git maintenance start &&\n> +\tgrep \"Important information!\" cron.txt\n> +'\n> +\n> test_done\n\nThese two test cases fail when the paths passed to git-config contain\nERE metacharacters [similar to the issue addressed in 483a6d9b5da\n(maintenance: use 'git config --fixed-value', 2020-11-25)]. Since these\nare already in next, I'm providing a patch to add '--fixed-value' to the\ngit-config calls here as well.\n\n-- >8 --\nSubject: [PATCH] t7900: use --fixed-value in git-maintenance tests\n\nUse --fixed-value in git-config calls in the git-maintenance tests, so\nthat the tests will continue to work even if the repo path contains\nshell metacharacters.\n\nSigned-off-by: Josh Steadmon \n---\n t/t7900-maintenance.sh | 4 ++--\n 1 file changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\nindex fab0e01c39..41bf523953 100755\n--- a/t/t7900-maintenance.sh\n+++ b/t/t7900-maintenance.sh\n@@ -422,7 +422,7 @@ test_expect_success 'start from empty cron table' '\n \tGIT_TEST_CRONTAB=\"test-tool crontab cron.txt\" git maintenance start &&\n \n \t# start registers the repo\n-\tgit config --get --global maintenance.repo \"$(pwd)\" &&\n+\tgit config --get --global --fixed-value maintenance.repo \"$(pwd)\" &&\n \n \tgrep \"for-each-repo --config=maintenance.repo maintenance run --schedule=daily\" cron.txt &&\n \tgrep \"for-each-repo --config=maintenance.repo maintenance run --schedule=hourly\" cron.txt &&\n@@ -433,7 +433,7 @@ test_expect_success 'stop from existing schedule' '\n \tGIT_TEST_CRONTAB=\"test-tool crontab cron.txt\" git maintenance stop &&\n \n \t# stop does not unregister the repo\n-\tgit config --get --global maintenance.repo \"$(pwd)\" &&\n+\tgit config --get --global --fixed-value maintenance.repo \"$(pwd)\" &&\n \n \t# Operation is idempotent\n \tGIT_TEST_CRONTAB=\"test-tool crontab cron.txt\" git maintenance stop &&\n-- \n2.29.2.576.ga3fc446d84-goog\n\n"},{"id":"411896","messageId":"20201209191616.GO36751@google.com","threadId":"54189","inReplyTo":"20201209185114.GN36751@google.com","subject":"Re: [PATCH v4 5/8] maintenance: add start/stop subcommands","fromName":"Josh Steadmon","fromEmail":"steadmon@google.com","sentAt":"2020-12-09T19:16:16Z","receivedAt":"2020-12-09T19:17:20Z","isPatch":true,"sender":{"key":"steadmon@google.com","avatar":"https://avatars.githubusercontent.com/u/2654920?v=4"},"body":"Whoops, had a small think-o while writing the patch message. Fixed\nbelow.\n\n-- >8 --\nSubject: [PATCH] t7900: use --fixed-value in git-maintenance tests\n\nUse --fixed-value in git-config calls in the git-maintenance tests, so\nthat the tests will continue to work even if the repo path contains\nregexp metacharacters.\n\nSigned-off-by: Josh Steadmon \n---\n t/t7900-maintenance.sh | 4 ++--\n 1 file changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\nindex fab0e01c39..41bf523953 100755\n--- a/t/t7900-maintenance.sh\n+++ b/t/t7900-maintenance.sh\n@@ -422,7 +422,7 @@ test_expect_success 'start from empty cron table' '\n \tGIT_TEST_CRONTAB=\"test-tool crontab cron.txt\" git maintenance start &&\n \n \t# start registers the repo\n-\tgit config --get --global maintenance.repo \"$(pwd)\" &&\n+\tgit config --get --global --fixed-value maintenance.repo \"$(pwd)\" &&\n \n \tgrep \"for-each-repo --config=maintenance.repo maintenance run --schedule=daily\" cron.txt &&\n \tgrep \"for-each-repo --config=maintenance.repo maintenance run --schedule=hourly\" cron.txt &&\n@@ -433,7 +433,7 @@ test_expect_success 'stop from existing schedule' '\n \tGIT_TEST_CRONTAB=\"test-tool crontab cron.txt\" git maintenance stop &&\n \n \t# stop does not unregister the repo\n-\tgit config --get --global maintenance.repo \"$(pwd)\" &&\n+\tgit config --get --global --fixed-value maintenance.repo \"$(pwd)\" &&\n \n \t# Operation is idempotent\n \tGIT_TEST_CRONTAB=\"test-tool crontab cron.txt\" git maintenance stop &&\n-- \n2.29.2.576.ga3fc446d84-goog\n\n"},{"id":"411937","messageId":"19201a12-7df5-9b95-cd8d-ab2be90cbb40@gmail.com","threadId":"54189","inReplyTo":"20201209191616.GO36751@google.com","subject":"Re: [PATCH v4 5/8] maintenance: add start/stop subcommands","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-12-09T21:59:07Z","receivedAt":"2020-12-09T22:00:07Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 12/9/2020 2:16 PM, Josh Steadmon wrote:\n> Whoops, had a small think-o while writing the patch message. Fixed\n> below.\n> \n> -- >8 --\n> Subject: [PATCH] t7900: use --fixed-value in git-maintenance tests\n> \n> Use --fixed-value in git-config calls in the git-maintenance tests, so\n> that the tests will continue to work even if the repo path contains\n> regexp metacharacters.\n> \n> Signed-off-by: Josh Steadmon \n> ---\n> t/t7900-maintenance.sh | 4 ++--\n> 1 file changed, 2 insertions(+), 2 deletions(-)\n> \n> diff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\n> index fab0e01c39..41bf523953 100755\n> --- a/t/t7900-maintenance.sh\n> +++ b/t/t7900-maintenance.sh\n> @@ -422,7 +422,7 @@ test_expect_success 'start from empty cron table' '\n> \tGIT_TEST_CRONTAB=\"test-tool crontab cron.txt\" git maintenance start &&\n> \n> \t# start registers the repo\n> -\tgit config --get --global maintenance.repo \"$(pwd)\" &&\n> +\tgit config --get --global --fixed-value maintenance.repo \"$(pwd)\" &&\n> \n> \tgrep \"for-each-repo --config=maintenance.repo maintenance run --schedule=daily\" cron.txt &&\n> \tgrep \"for-each-repo --config=maintenance.repo maintenance run --schedule=hourly\" cron.txt &&\n> @@ -433,7 +433,7 @@ test_expect_success 'stop from existing schedule' '\n> \tGIT_TEST_CRONTAB=\"test-tool crontab cron.txt\" git maintenance stop &&\n> \n> \t# stop does not unregister the repo\n> -\tgit config --get --global maintenance.repo \"$(pwd)\" &&\n> +\tgit config --get --global --fixed-value maintenance.repo \"$(pwd)\" &&\n> \n> \t# Operation is idempotent\n> \tGIT_TEST_CRONTAB=\"test-tool crontab cron.txt\" git maintenance stop &&\n\nThank you for this. While I went to make sure the maintenance builtin worked\nproperly, I forgot to check the rest of the test script worked as well. This\nis a good way to fix that.\n\nThanks,\n-Stolee\n \n\n"},{"id":"411949","messageId":"xmqqtusuzffv.fsf@gitster.c.googlers.com","threadId":"54189","inReplyTo":"20201209191616.GO36751@google.com","subject":"Re: [PATCH v4 5/8] maintenance: add start/stop subcommands","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-12-10T00:13:08Z","receivedAt":"2020-12-10T00:13:52Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Josh Steadmon writes:\n\n> \t# start registers the repo\n> -\tgit config --get --global maintenance.repo \"$(pwd)\" &&\n> +\tgit config --get --global --fixed-value maintenance.repo \"$(pwd)\" &&\n\nThe rewrite makes it better than the original, but I wonder why the\noriginal did not do a more obvious\n\n\tgit config --get maintenance.repo >actual &&\n\tpwd >expect &&\n\ttest_cmp expect actual\n\n> \t# stop does not unregister the repo\n> -\tgit config --get --global maintenance.repo \"$(pwd)\" &&\n> +\tgit config --get --global --fixed-value maintenance.repo \"$(pwd)\" &&\n\nDitto.\n"},{"id":"411965","messageId":"995da636-94bc-378b-83d0-ecce9d802b81@gmail.com","threadId":"54189","inReplyTo":"xmqqtusuzffv.fsf@gitster.c.googlers.com","subject":"Re: [PATCH v4 5/8] maintenance: add start/stop subcommands","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-12-10T01:52:36Z","receivedAt":"2020-12-10T01:53:42Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 12/9/2020 7:13 PM, Junio C Hamano wrote:\n> Josh Steadmon writes:\n> \n>> \t# start registers the repo\n>> -\tgit config --get --global maintenance.repo \"$(pwd)\" &&\n>> +\tgit config --get --global --fixed-value maintenance.repo \"$(pwd)\" &&\n> \n> The rewrite makes it better than the original, but I wonder why the\n> original did not do a more obvious\n\nmaintenance.repo is a multi-valued config setting, so it is possible\nthat there are multiple existing values. Hence the reason for needing\nthe value filter.\n\nThanks,\n-Stolee\n"},{"id":"411982","messageId":"xmqqtusuw3qd.fsf@gitster.c.googlers.com","threadId":"54189","inReplyTo":"995da636-94bc-378b-83d0-ecce9d802b81@gmail.com","subject":"Re: [PATCH v4 5/8] maintenance: add start/stop subcommands","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-12-10T06:54:18Z","receivedAt":"2020-12-10T06:55:21Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n> On 12/9/2020 7:13 PM, Junio C Hamano wrote:\n>> Josh Steadmon writes:\n>> \n>>> \t# start registers the repo\n>>> -\tgit config --get --global maintenance.repo \"$(pwd)\" &&\n>>> +\tgit config --get --global --fixed-value maintenance.repo \"$(pwd)\" &&\n>> \n>> The rewrite makes it better than the original, but I wonder why the\n>> original did not do a more obvious\n>\n> maintenance.repo is a multi-valued config setting, so it is possible\n> that there are multiple existing values. Hence the reason for needing\n> the value filter.\n\nI do not quite get it. You mean as long as $(pwd) appears, you do\nnot care what other value appear on the variable? Aren't we control\nof what repositories have been registered to the system at this point\nin the test sequence?\n\nIt's not wrong per-se to use \"does this value exist for the key?\",\nespecially with the --fixed-value option. It somehow just felt a\nbit unusual to me.\n\nIn any case, thanks for the metacharacter fix. That is now on\n'master' so the previous breakages are all gone with Josh's fix.\n\nThanks, both.\n"},{"id":"416510","messageId":"87blctcp30.fsf@evledraar.gmail.com","threadId":"54189","inReplyTo":"dae8c04bb5523c9b63c770862a1104a0ff4aa6c4.1602782524.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 2/8] maintenance: add --schedule option and config","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-02-09T14:06:59Z","receivedAt":"2021-02-09T14:07:54Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Thu, Oct 15 2020, Derrick Stolee via GitGitGadget wrote:\n\n> +--schedule::\n> +\tWhen combined with the `run` subcommand, run maintenance tasks\n> +\tonly if certain time conditions are met, as specified by the\n> +\t`maintenance..schedule` config value for each ``.\n> +\tThis config value specifies a number of seconds since the last\n> +\ttime that task ran, according to the `maintenance..lastRun`\n> +\tconfig value. The tasks that are tested are those provided by\n> +\tthe `--task=` option(s) or those with\n> +\t`maintenance..enabled` set to true.\n\nI see from searching on list and from spying on your repo that patches\nfor this maintenance..lastRun feature exist, but there's no code\nfor it in git.git.\n\nSo we've got a 2.30.0 release with a mention of that, and it can't work,\nbecause it's only in the doc due to b08ff1fee00 (maintenance: add\n--schedule option and config, 2020-09-11).\n\n"},{"id":"416518","messageId":"1eff631f-b079-d097-b16d-36dc974efea9@gmail.com","threadId":"54189","inReplyTo":"87blctcp30.fsf@evledraar.gmail.com","subject":"Re: [PATCH v4 2/8] maintenance: add --schedule option and config","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-02-09T16:54:41Z","receivedAt":"2021-02-09T16:55:31Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 2/9/2021 9:06 AM, Ævar Arnfjörð Bjarmason wrote:\n> \n> On Thu, Oct 15 2020, Derrick Stolee via GitGitGadget wrote:\n> \n>> +--schedule::\n>> +\tWhen combined with the `run` subcommand, run maintenance tasks\n>> +\tonly if certain time conditions are met, as specified by the\n>> +\t`maintenance..schedule` config value for each ``.\n>> +\tThis config value specifies a number of seconds since the last\n>> +\ttime that task ran, according to the `maintenance..lastRun`\n>> +\tconfig value. The tasks that are tested are those provided by\n>> +\tthe `--task=` option(s) or those with\n>> +\t`maintenance..enabled` set to true.\n> \n> I see from searching on list and from spying on your repo that patches\n> for this maintenance..lastRun feature exist, but there's no code\n> for it in git.git.\n> \n> So we've got a 2.30.0 release with a mention of that, and it can't work,\n> because it's only in the doc due to b08ff1fee00 (maintenance: add\n> --schedule option and config, 2020-09-11).\n\nThank you for pointing out this docbug. This is based on an early\nversion of the patch series and should have been changed.\n\nPlease see this patch which attempts to do a better job. I can\ncreate a new thread with this submission if we need more edits.\n\nThanks,\n-Stolee\n\n--- >8 ---\n\nFrom 46436b06caf65ee824e781603a8108413bb87705 Mon Sep 17 00:00:00 2001\nFrom: Derrick Stolee \nDate: Tue, 9 Feb 2021 11:51:32 -0500\nSubject: [PATCH] maintenance: properly document --schedule\nMIME-Version: 1.0\nContent-Type: text/plain; charset=UTF-8\nContent-Transfer-Encoding: 8bit\n\nThe documentation for the '--schedule' option is incorrect and based on\nan early version of the background maintenance feature. Update the\ndocumentation to describe the actual use of the option.\n\nThe most important thing is that Git takes this option as a hint for\nwhich tasks it should run. Users should not run this command arbitrarily\nand expect that Git will enforce some timing restrictions.\n\nReported-by: Ævar Arnfjörð Bjarmason \nSigned-off-by: Derrick Stolee \n---\n Documentation/git-maintenance.txt | 16 ++++++++--------\n 1 file changed, 8 insertions(+), 8 deletions(-)\n\ndiff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt\nindex 6fec1eb8dc2..d4b5aea6760 100644\n--- a/Documentation/git-maintenance.txt\n+++ b/Documentation/git-maintenance.txt\n@@ -155,15 +155,15 @@ OPTIONS\n \texceeds the `gc.autoPackLimit` config setting. Not compatible with\n \tthe `--schedule` option.\n \n---schedule::\n+--schedule=::\n \tWhen combined with the `run` subcommand, run maintenance tasks\n-\tonly if certain time conditions are met, as specified by the\n-\t`maintenance..schedule` config value for each ``.\n-\tThis config value specifies a number of seconds since the last\n-\ttime that task ran, according to the `maintenance..lastRun`\n-\tconfig value. The tasks that are tested are those provided by\n-\tthe `--task=` option(s) or those with\n-\t`maintenance..enabled` set to true.\n+\twhose `maintenance..schedule` config value is equal to\n+\t``. There is no timing restriction imposed by this\n+\toption, but instead is used to inform the Git process which\n+\tfrequency to use. The command scheduler created by\n+\t`git maintenance start` runs this command with ``\n+\tequal to `hourly`, `daily`, and `weekly` at the appropriate\n+\tintervals.\n \n --quiet::\n \tDo not report progress or other information over `stderr`.\n-- \n2.30.0.vfs.0.0.exp\n\n\n\n"},{"id":"423507","messageId":"cda4f200-2400-e915-e995-36eea2a27c11@ahunt.org","threadId":"54189","inReplyTo":"dd9237927395ef69663ab376a2da74da4654c495.1602782524.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 3/8] for-each-repo: run subcommands on configured repos","fromName":"Andrzej Hunt","fromEmail":"andrzej@ahunt.org","sentAt":"2021-05-03T16:10:47Z","receivedAt":"2021-05-03T16:10:56Z","isPatch":true,"sender":{"key":"andrzej@ahunt.org","avatar":"https://avatars.githubusercontent.com/u/1546915?v=4"},"body":"\nOn 15/10/2020 19:21, Derrick Stolee via GitGitGadget wrote:\n> From: Derrick Stolee \n[... snip ...]\n> diff --git a/builtin/for-each-repo.c b/builtin/for-each-repo.c\n> new file mode 100644\n> index 0000000000..5bba623ff1\n> --- /dev/null\n> +++ b/builtin/for-each-repo.c\n> @@ -0,0 +1,58 @@\n> +#include \"cache.h\"\n> +#include \"config.h\"\n> +#include \"builtin.h\"\n> +#include \"parse-options.h\"\n> +#include \"run-command.h\"\n> +#include \"string-list.h\"\n> +\n> +static const char * const for_each_repo_usage[] = {\n> +\tN_(\"git for-each-repo --config= \"),\n> +\tNULL\n> +};\n> +\n> +static int run_command_on_repo(const char *path,\n> +\t\t\t void *cbdata)\n> +{\n> +\tint i;\n> +\tstruct child_process child = CHILD_PROCESS_INIT;\n> +\tstruct strvec *args = (struct strvec *)cbdata;\n\nI was curious there's a strong reason for declaring args as void * \nfollowed by this cast? The most obvious answer seems to be that this \nprobably evolved from a callback - and we could simplify it now?\n> +\n> +\tchild.git_cmd = 1;\n> +\tstrvec_pushl(&child.args, \"-C\", path, NULL);\n> +\n> +\tfor (i = 0; i < args->nr; i++)\n> +\t\tstrvec_push(&child.args, args->v[i]);\nSo here we're copying all of args - and I don't see any way of avoiding \nit since we're adding it to child's arg list.\n\n> +\n> +\treturn run_command(&child);\n> +}\n> +\n> +int cmd_for_each_repo(int argc, const char **argv, const char *prefix)\n> +{\n> +\tstatic const char *config_key = NULL;\n> +\tint i, result = 0;\n> +\tconst struct string_list *values;\n> +\tstruct strvec args = STRVEC_INIT;\n> +\n> +\tconst struct option options[] = {\n> +\t\tOPT_STRING(0, \"config\", &config_key, N_(\"config\"),\n> +\t\t\t N_(\"config key storing a list of repository paths\")),\n> +\t\tOPT_END()\n> +\t};\n> +\n> +\targc = parse_options(argc, argv, prefix, options, for_each_repo_usage,\n> +\t\t\t PARSE_OPT_STOP_AT_NON_OPTION);\n> +\n> +\tif (!config_key)\n> +\t\tdie(_(\"missing --config=\"));\n> +\n> +\tfor (i = 0; i < argc; i++)\n> +\t\tstrvec_push(&args, argv[i]);\n\nBut why do we have to copy all of argv 1:1 into args here, only to later \npass it to run_command_on_repo() which, as described above, copies the \nentire input again? I suspect this was done to comply with \nrun_command_on_repo()'s API (which takes strvec) - does that seem \nplausible, or did I miss something?\n\nWhich brings me to the real reason for my questions: I noticed we \"leak\" \nargs (this leak is of no significance since it happens in cmd_*, but \nLSAN still complains, and I'm trying to get tests running leak-free). My \ninitial inclination was to strvec_clear() or UNLEAK() args - but if we \ncan avoid creating args in the first place we also wouldn't need to \nclear it later.\n\nMy current proposal is therefore to completely remove args and pass \nargc+argv into run_command_on_repo() - but I wanted to be sure that I \ndidn't miss some important reason to stick with the current approach.\n\n> +\n> +\tvalues = repo_config_get_value_multi(the_repository,\n> +\t\t\t\t\t config_key);\n> +\n> +\tfor (i = 0; !result && i < values->nr; i++)\n> +\t\tresult = run_command_on_repo(values->items[i].string, &args);\n> +\n> +\treturn result;\n> +}\n[... snip ...]\n\n(I hope this doesn't come across as useless necroposting - I figured it \nwould be easier to clarify these questions on the original thread as \nopposed to potentially discussing it as part of my next leak-fixing \nseries :) .)\n\nATB,\n\n Andrzej\n"},{"id":"423510","messageId":"CAPig+cToeuq7S1176ze7kX9krjK1rGMOM-JDznF_Fr_qV09qWw@mail.gmail.com","threadId":"54189","inReplyTo":"cda4f200-2400-e915-e995-36eea2a27c11@ahunt.org","subject":"Re: [PATCH v4 3/8] for-each-repo: run subcommands on configured repos","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2021-05-03T17:01:42Z","receivedAt":"2021-05-03T17:07:55Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"I'm not Stolee and hadn't even looked at this code before, but I'll\njump in with a couple comments...\n\nOn Mon, May 3, 2021 at 12:11 PM Andrzej Hunt wrote:\n> On 15/10/2020 19:21, Derrick Stolee via GitGitGadget wrote:\n> > +static int run_command_on_repo(const char *path,\n> > + void *cbdata)\n> > +{\n> > + int i;\n> > + struct child_process child = CHILD_PROCESS_INIT;\n> > + struct strvec *args = (struct strvec *)cbdata;\n>\n> I was curious there's a strong reason for declaring args as void *\n> followed by this cast? The most obvious answer seems to be that this\n> probably evolved from a callback - and we could simplify it now?\n\nAgreed, the `void*` cbdata doesn't make sense here.\n\n> > + strvec_pushl(&child.args, \"-C\", path, NULL);\n> > +\n> > + for (i = 0; i < args->nr; i++)\n> > + strvec_push(&child.args, args->v[i]);\n>\n> So here we're copying all of args - and I don't see any way of avoiding\n> it since we're adding it to child's arg list.\n\n... (dot dot dot)\n\n> > +int cmd_for_each_repo(int argc, const char **argv, const char *prefix)\n> > +{\n> > + struct strvec args = STRVEC_INIT;\n> > + for (i = 0; i < argc; i++)\n> > + strvec_push(&args, argv[i]);\n>\n> But why do we have to copy all of argv 1:1 into args here, only to later\n> pass it to run_command_on_repo() which, as described above, copies the\n> entire input again? I suspect this was done to comply with\n> run_command_on_repo()'s API (which takes strvec) - does that seem\n> plausible, or did I miss something?\n>\n> Which brings me to the real reason for my questions: I noticed we \"leak\"\n> args (this leak is of no significance since it happens in cmd_*, but\n> LSAN still complains, and I'm trying to get tests running leak-free). My\n> initial inclination was to strvec_clear() or UNLEAK() args - but if we\n> can avoid creating args in the first place we also wouldn't need to\n> clear it later.\n>\n> My current proposal is therefore to completely remove args and pass\n> argc+argv into run_command_on_repo() - but I wanted to be sure that I\n> didn't miss some important reason to stick with the current approach.\n\nAn alternative to all this copying would be to take advantage of\nchild_process.argv which is owned by the caller, thus does not get\nfreed automatically by run_command(). This would allow you to re-use\nthe same argument vector for all calls. And you don't need\nrun_command_on_repo() at all. Something like this in\ncmd_for_each_repo(), untested and typed directly in email:\n\n struct child_process child = CHILD_PROCESS_INIT;\n\n for (i = 0; !result && i < values->nr; i++) {\n const char *d = chdir(values->items[i].string);\n if (chdir(d))\n die_errno(_(\"cannot chdir to '%s'\"), d);\n child.git_cmd = 1;\n child.argv = argv;\n result = run_command(&child);\n }\n\nThis assumes that argv[] is correctly NULL-terminated after\nparse_options() -- I didn't check, but expect it to be so. If not,\nit's easy enough to copy argv[] into `args` once and then\nstrvec_clear(&args) at the end of the function.\n\nThe one downside is that trace output wouldn't be as helpful (I think)\nbecause you wouldn't see an explicit \"-C \", but I suppose the\ntracing machinery can be invoked manually to address this.\n"},{"id":"423515","messageId":"CAPig+cT5m+DxcsNGkkVmYTaptt3-S4PV6MX7273kdXhM=_M4wg@mail.gmail.com","threadId":"54189","inReplyTo":"CAPig+cToeuq7S1176ze7kX9krjK1rGMOM-JDznF_Fr_qV09qWw@mail.gmail.com","subject":"Re: [PATCH v4 3/8] for-each-repo: run subcommands on configured repos","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2021-05-03T19:26:23Z","receivedAt":"2021-05-03T19:26:37Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Mon, May 3, 2021 at 1:01 PM Eric Sunshine wrote:\n> for (i = 0; !result && i < values->nr; i++) {\n> const char *d = chdir(values->items[i].string);\n> if (chdir(d))\n> die_errno(_(\"cannot chdir to '%s'\"), d);\n> child.git_cmd = 1;\n> child.argv = argv;\n> result = run_command(&child);\n> }\n\nWithout the copy/paste error, the declaration of `d` would, of course, be:\n\n const char *d = values->items[i].string;\n"},{"id":"423517","messageId":"68bbeca5-314b-08ee-ef36-040e3f3814e9@gmail.com","threadId":"54189","inReplyTo":"cda4f200-2400-e915-e995-36eea2a27c11@ahunt.org","subject":"Re: [PATCH v4 3/8] for-each-repo: run subcommands on configured repos","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-05-03T19:43:08Z","receivedAt":"2021-05-03T19:43:15Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 5/3/2021 12:10 PM, Andrzej Hunt wrote:\n> \n> On 15/10/2020 19:21, Derrick Stolee via GitGitGadget wrote:\n>> From: Derrick Stolee \n>> +static int run_command_on_repo(const char *path,\n>> +                   void *cbdata)\n>> +{\n>> +    int i;\n>> +    struct child_process child = CHILD_PROCESS_INIT;\n>> +    struct strvec *args = (struct strvec *)cbdata;\n> \n> I was curious there's a strong reason for declaring args as void * followed by this cast? The most obvious answer seems to be that this probably evolved from a callback - and we could simplify it now?\n\nYou are absolutely right that this evolved from a\ncallback. I look forward to reviewing your patch that\nupdates this. ;)\n\nThanks,\n-Stolee\n"},{"id":"424027","messageId":"87o8diu6jn.fsf@evledraar.gmail.com","threadId":"54189","inReplyTo":"1eff631f-b079-d097-b16d-36dc974efea9@gmail.com","subject":"Re: [PATCH v4 2/8] maintenance: add --schedule option and config","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2021-05-10T12:16:12Z","receivedAt":"2021-05-10T12:52:22Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Feb 09 2021, Derrick Stolee wrote:\n\n> On 2/9/2021 9:06 AM, Ævar Arnfjörð Bjarmason wrote:\n>> \n>> On Thu, Oct 15 2020, Derrick Stolee via GitGitGadget wrote:\n>> \n>>> +--schedule::\n>>> +\tWhen combined with the `run` subcommand, run maintenance tasks\n>>> +\tonly if certain time conditions are met, as specified by the\n>>> +\t`maintenance..schedule` config value for each ``.\n>>> +\tThis config value specifies a number of seconds since the last\n>>> +\ttime that task ran, according to the `maintenance..lastRun`\n>>> +\tconfig value. The tasks that are tested are those provided by\n>>> +\tthe `--task=` option(s) or those with\n>>> +\t`maintenance..enabled` set to true.\n>> \n>> I see from searching on list and from spying on your repo that patches\n>> for this maintenance..lastRun feature exist, but there's no code\n>> for it in git.git.\n>> \n>> So we've got a 2.30.0 release with a mention of that, and it can't work,\n>> because it's only in the doc due to b08ff1fee00 (maintenance: add\n>> --schedule option and config, 2020-09-11).\n>\n> Thank you for pointing out this docbug. This is based on an early\n> version of the patch series and should have been changed.\n>\n> Please see this patch which attempts to do a better job. I can\n> create a new thread with this submission if we need more edits.\n>\n> Thanks,\n> -Stolee\n>\n> --- >8 ---\n>\n> From 46436b06caf65ee824e781603a8108413bb87705 Mon Sep 17 00:00:00 2001\n> From: Derrick Stolee \n> Date: Tue, 9 Feb 2021 11:51:32 -0500\n> Subject: [PATCH] maintenance: properly document --schedule\n> MIME-Version: 1.0\n> Content-Type: text/plain; charset=UTF-8\n> Content-Transfer-Encoding: 8bit\n>\n> The documentation for the '--schedule' option is incorrect and based on\n> an early version of the background maintenance feature. Update the\n> documentation to describe the actual use of the option.\n>\n> The most important thing is that Git takes this option as a hint for\n> which tasks it should run. Users should not run this command arbitrarily\n> and expect that Git will enforce some timing restrictions.\n>\n> Reported-by: Ævar Arnfjörð Bjarmason \n> Signed-off-by: Derrick Stolee \n> ---\n> Documentation/git-maintenance.txt | 16 ++++++++--------\n> 1 file changed, 8 insertions(+), 8 deletions(-)\n>\n> diff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt\n> index 6fec1eb8dc2..d4b5aea6760 100644\n> --- a/Documentation/git-maintenance.txt\n> +++ b/Documentation/git-maintenance.txt\n> @@ -155,15 +155,15 @@ OPTIONS\n> \texceeds the `gc.autoPackLimit` config setting. Not compatible with\n> \tthe `--schedule` option.\n> \n> ---schedule::\n> +--schedule=::\n> \tWhen combined with the `run` subcommand, run maintenance tasks\n> -\tonly if certain time conditions are met, as specified by the\n> -\t`maintenance..schedule` config value for each ``.\n> -\tThis config value specifies a number of seconds since the last\n> -\ttime that task ran, according to the `maintenance..lastRun`\n> -\tconfig value. The tasks that are tested are those provided by\n> -\tthe `--task=` option(s) or those with\n> -\t`maintenance..enabled` set to true.\n> +\twhose `maintenance..schedule` config value is equal to\n> +\t``. There is no timing restriction imposed by this\n> +\toption, but instead is used to inform the Git process which\n> +\tfrequency to use. The command scheduler created by\n> +\t`git maintenance start` runs this command with ``\n> +\tequal to `hourly`, `daily`, and `weekly` at the appropriate\n> +\tintervals.\n> \n> --quiet::\n> \tDo not report progress or other information over `stderr`.\n\n+ CC Junio\n\nLate reply, I was reminded of this again. This patch looks good to me,\nbut I see it never got picked up, and our docs still mention an\nunsupported \"lastRun\". Junio: I think it makes sense to just pick this\nup...\n"},{"id":"424068","messageId":"xmqqpmxyph0o.fsf@gitster.g","threadId":"54189","inReplyTo":"87o8diu6jn.fsf@evledraar.gmail.com","subject":"Re: [PATCH v4 2/8] maintenance: add --schedule option and config","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-05-10T18:42:31Z","receivedAt":"2021-05-10T18:42:40Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ævar Arnfjörð Bjarmason writes:\n\n> On Tue, Feb 09 2021, Derrick Stolee wrote:\n>\n>> On 2/9/2021 9:06 AM, Ævar Arnfjörð Bjarmason wrote:\n>>> \n>>> On Thu, Oct 15 2020, Derrick Stolee via GitGitGadget wrote:\n>>> \n>>>> +--schedule::\n>>>> +\tWhen combined with the `run` subcommand, run maintenance tasks\n>>>> +\tonly if certain time conditions are met, as specified by the\n>>>> +\t`maintenance..schedule` config value for each ``.\n>>>> +\tThis config value specifies a number of seconds since the last\n>>>> +\ttime that task ran, according to the `maintenance..lastRun`\n>>>> +\tconfig value. The tasks that are tested are those provided by\n>>>> +\tthe `--task=` option(s) or those with\n>>>> +\t`maintenance..enabled` set to true.\n>>> \n>>> I see from searching on list and from spying on your repo that patches\n>>> for this maintenance..lastRun feature exist, but there's no code\n>>> for it in git.git.\n>>> \n>>> So we've got a 2.30.0 release with a mention of that, and it can't work,\n>>> because it's only in the doc due to b08ff1fee00 (maintenance: add\n>>> --schedule option and config, 2020-09-11).\n>>\n>> Thank you for pointing out this docbug. This is based on an early\n>> version of the patch series and should have been changed.\n>>\n>> Please see this patch which attempts to do a better job. I can\n>> create a new thread with this submission if we need more edits.\n\nOr resend it in a new thread for better visibility, with or without\nchange, with a mention of the original under the three-dash line.\nNobody can tell, with the above comment, if you abandoned the patch\nor if you thought it is good enough.\n\nThanks.\n"}]}