{"thread":{"id":"54070","subject":"[PATCH 0/7] [RFC] Maintenance III: background maintenance","startedAt":"2020-08-19T17:16:59Z","lastAt":"2020-08-28T15:46:05Z","messageCount":41,"participants":["Derrick Stolee via GitGitGadget","Đoàn Trần Công Danh","Derrick Stolee","Junio C Hamano","Michal Suchánek"],"isPatch":true,"patchVersion":1,"patchTotal":7},"messages":[{"id":"404037","messageId":"pull.680.git.1597857408.gitgitgadget@gmail.com","threadId":"54070","inReplyTo":null,"subject":"[PATCH 0/7] [RFC] Maintenance III: background maintenance","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-08-19T17:16:41Z","receivedAt":"2020-08-19T17:16:59Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"This is based on ds/maintenance-part-2, but with some local updates to\nreview feedback. It won't apply cleanly right now. This RFC is for early\nfeedback and not intended to make a new tracking branch until v2.\n\nThis RFC is intended to show how I hope to integrate true background\nmaintenance into Git. As opposed to my original RFC [1], this entirely\nintegrates with cron (through crontab [-e|-l]) to launch maintenance\ncommands in the background.\n\n[1] \nhttps://lore.kernel.org/git/pull.597.git.1585946894.gitgitgadget@gmail.com/\n\nSome preliminary work is done to allow a new --scheduled option that\ntriggers enabled tasks only if they have not been run in some amount of\ntime. The timestamp of the previous run is stored in the \nmaintenance..lastRun config value while the interval is stored in the \nmaintenance..schedule config value.\n\nA new for-each-repo builtin runs Git commands on every repo in a given list.\nCurrently, the list is stored as a config setting, allowing a new \nmaintenance.repos config list to store the repositories registered for\nbackground maintenance. Others may want to add a --file= option for\ntheir own workflows, but I focused on making this as simple as possible for\nnow.\n\nThe updates to the git maintenance builtin include new register/unregister \nsubcommands and start/stop subcommands. The register subcommand initializes\nthe config while the start subcommand does everything register does plus \nupdate the cron table. The unregister and stop commands reverse this\nprocess.\n\nThe very last patch is entirely optional. It sets a recommended schedule\nbased on my own experience with very large repositories. I'm open to other\nsuggestions, but these are ones that I think work well and don't cause a\n\"rewrite the world\" scenario like running nightly 'gc' would do.\n\nI've been testing this scenario on my macOS laptop for a while and my Linux\nmachine. I have modified my cron task to provide logging via trace2 so I can\nsee what's happening. A future direction here would be to add some\nmaintenance logs to the repository so we can track what is happening and\ndiagnose whether the maintenance strategy is working on real repos.\n\nIt could be helpful for contributors to suggest ways to configure certain\njobs to run \"nightly\" or \"overnight on a weekend\" instead of just \"whenever\nthe last run was long enough ago.\" One way to do this would be to set the \nlastRun config to be at the time of day that the job should run. Another\noption would be to make the cron table more complicated with multiple rows,\nbut I would prefer to avoid that option if there is a simpler mechanism.\n\nNote: git maintenance (start|stop) only works on machines with cron by\ndesign. The proper thing to do on Windows will come later. Perhaps this\ncommand should be marked as unavailable on Windows somehow, or at least a\nbetter error than \"cron may not be available on your system\". I did find\nthat that message is helpful sometimes: macOS worker agents for CI builds\ntypically do not have cron available.\n\nThanks, -Stolee\n\nDerrick Stolee (7):\n maintenance: optionally skip --auto process\n maintenance: store the \"last run\" time in config\n maintenance: add --scheduled option and config\n for-each-repo: run subcommands on configured repos\n maintenance: add [un]register subcommands\n maintenance: add start/stop subcommands\n maintenance: recommended schedule in register/start\n\n .gitignore | 1 +\n Documentation/config/maintenance.txt | 19 ++\n Documentation/git-for-each-repo.txt | 45 +++++\n Documentation/git-maintenance.txt | 44 ++++-\n Makefile | 2 +\n builtin.h | 1 +\n builtin/for-each-repo.c | 58 ++++++\n builtin/gc.c | 282 ++++++++++++++++++++++++++-\n git-gvfs-helper | Bin 0 -> 11171736 bytes\n git.c | 1 +\n run-command.c | 8 +\n t/helper/test-crontab.c | 35 ++++\n t/helper/test-gvfs-protocol | Bin 0 -> 10946928 bytes\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/t0068-for-each-repo.sh | 30 +++\n t/t7900-maintenance.sh | 95 ++++++++-\n t/test-lib.sh | 6 +\n 18 files changed, 625 insertions(+), 4 deletions(-)\n create mode 100644 Documentation/git-for-each-repo.txt\n create mode 100644 builtin/for-each-repo.c\n create mode 100755 git-gvfs-helper\n create mode 100644 t/helper/test-crontab.c\n create mode 100755 t/helper/test-gvfs-protocol\n create mode 100755 t/t0068-for-each-repo.sh\n\n\nbase-commit: 0c43c64dd2fb41ac14038f1c3143bddbc6c35585\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-680%2Fderrickstolee%2Fmaintenance%2Fscheduled-v1\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-680/derrickstolee/maintenance/scheduled-v1\nPull-Request: https://github.com/gitgitgadget/git/pull/680\n-- \ngitgitgadget\n"},{"id":"404038","messageId":"90de25d1287595fdedc9dcd194a2e0ac120d4aad.1597857409.git.gitgitgadget@gmail.com","threadId":"54070","inReplyTo":"pull.680.git.1597857408.gitgitgadget@gmail.com","subject":"[PATCH 1/7] maintenance: optionally skip --auto process","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-08-19T17:16:42Z","receivedAt":"2020-08-19T17:17:05Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nSome commands run 'git maintenance run --auto --[no-]quiet' after doing\ntheir normal work, as a way to keep repositories clean as they are used.\nCurrently, users who do not want this maintenance to occur would set the\n'gc.auto' config option to 0 to avoid the 'gc' task from running.\nHowever, this does not stop the extra process invocation. On Windows,\nthis extra process invocation can be more expensive than necessary.\n\nAllow users to drop this extra process by setting 'maintenance.auto' to\n'false'.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/config/maintenance.txt | 5 +++++\n run-command.c | 8 ++++++++\n t/t7900-maintenance.sh | 13 +++++++++++++\n 3 files changed, 26 insertions(+)\n\ndiff --git a/Documentation/config/maintenance.txt b/Documentation/config/maintenance.txt\nindex a0706d8f09..06db758172 100644\n--- a/Documentation/config/maintenance.txt\n+++ b/Documentation/config/maintenance.txt\n@@ -1,3 +1,8 @@\n+maintenance.auto::\n+\tThis boolean config option controls whether some commands run\n+\t`git maintenance run --auto` after doing their normal work. Defaults\n+\tto true.\n+\n maintenance..enabled::\n \tThis boolean config option controls whether the maintenance task\n \twith name `` is run when no `--task` option is specified to\ndiff --git a/run-command.c b/run-command.c\nindex 2ee59acdc8..9c9d5d7f98 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -7,6 +7,7 @@\n #include \"strbuf.h\"\n #include \"string-list.h\"\n #include \"quote.h\"\n+#include \"config.h\"\n \n void child_process_init(struct child_process *child)\n {\n@@ -1868,8 +1869,15 @@ int run_processes_parallel_tr2(int n, get_next_task_fn get_next_task,\n \n int run_auto_maintenance(int quiet)\n {\n+\tint enabled;\n \tstruct child_process maint = CHILD_PROCESS_INIT;\n \n+\tif (!git_config_get_bool(\"maintenance.auto\", &enabled) &&\n+\t !enabled) {\n+\t\t fprintf(stderr, \"enabled: %d\\n\", enabled);\n+\t\treturn 0;\n+\t }\n+\n \tmaint.git_cmd = 1;\n \tstrvec_pushl(&maint.args, \"maintenance\", \"run\", \"--auto\", NULL);\n \tstrvec_push(&maint.args, quiet ? \"--quiet\" : \"--no-quiet\");\ndiff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\nindex 6f878b0141..e0ba19e1ff 100755\n--- a/t/t7900-maintenance.sh\n+++ b/t/t7900-maintenance.sh\n@@ -26,6 +26,19 @@ test_expect_success 'run [--auto|--quiet]' '\n \ttest_subcommand git gc --no-quiet .enabled' '\n \tgit config maintenance.gc.enabled false &&\n \tgit config maintenance.commit-graph.enabled true &&\n-- \ngitgitgadget\n\n"},{"id":"404039","messageId":"ccb667dc6fa4704171da643158ecd79ed7dd2cd3.1597857412.git.gitgitgadget@gmail.com","threadId":"54070","inReplyTo":"pull.680.git.1597857408.gitgitgadget@gmail.com","subject":"[PATCH 4/7] for-each-repo: run subcommands on configured repos","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-08-19T17:16:45Z","receivedAt":"2020-08-19T17:17:25Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIt can be helpful to store a list of repositories in global or system\nconfig and then iterate Git commands on that list. Create a new builtin\nthat makes this process simple for experts. We will use this builtin to\nrun scheduled maintenance on all configured repositories in a future\nchange.\n\nThe test is very simple, but does highlight that the \"--\" argument is\noptional.\n\nSigned-off-by: Derrick Stolee \n---\n .gitignore | 1 +\n Documentation/git-for-each-repo.txt | 45 ++++++++++++++++++++++\n Makefile | 1 +\n builtin.h | 1 +\n builtin/for-each-repo.c | 58 +++++++++++++++++++++++++++++\n git.c | 1 +\n t/t0068-for-each-repo.sh | 30 +++++++++++++++\n 7 files changed, 137 insertions(+)\n create mode 100644 Documentation/git-for-each-repo.txt\n create mode 100644 builtin/for-each-repo.c\n create mode 100755 t/t0068-for-each-repo.sh\n\ndiff --git a/.gitignore b/.gitignore\nindex a5808fa30d..5eb2a2be71 100644\n--- a/.gitignore\n+++ b/.gitignore\n@@ -67,6 +67,7 @@\n /git-filter-branch\n /git-fmt-merge-msg\n /git-for-each-ref\n+/git-for-each-repo\n /git-format-patch\n /git-fsck\n /git-fsck-objects\ndiff --git a/Documentation/git-for-each-repo.txt b/Documentation/git-for-each-repo.txt\nnew file mode 100644\nindex 0000000000..83b06db410\n--- /dev/null\n+++ b/Documentation/git-for-each-repo.txt\n@@ -0,0 +1,45 @@\n+git-for-each-repo(1)\n+====================\n+\n+NAME\n+----\n+git-for-each-repo - Run a Git command on a list of repositories\n+\n+\n+SYNOPSIS\n+--------\n+[verse]\n+'git for-each-repo' --config= [--] \n+\n+\n+DESCRIPTION\n+-----------\n+Run a Git commands on a list of repositories. The arguments after the\n+known options or `--` indicator are used as the arguments for the Git\n+command.\n+\n+For example, we could run maintenance on each of a list of repositories\n+stored in a `maintenance.repo` config variable using\n+\n+-------------\n+git for-each-repo --config=maintenance.repo maintenance run\n+-------------\n+\n+This will run `git -C maintenance run` for each value ``\n+in the multi-valued config variable `maintenance.repo`.\n+\n+\n+OPTIONS\n+-------\n+--config=::\n+\tUse the given config variable as a multi-valued list storing\n+\tabsolute path names. Iterate on that list of paths to run\n+\tthe given arguments.\n++\n+These config values are loaded from system, global, and local Git config,\n+as available. If `git for-each-repo` is run in a directory that is not a\n+Git repository, then only the system and global config is used.\n+\n+GIT\n+---\n+Part of the linkgit:git[1] suite\ndiff --git a/Makefile b/Makefile\nindex 65f8cfb236..7c588ff036 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1071,6 +1071,7 @@ BUILTIN_OBJS += builtin/fetch-pack.o\n BUILTIN_OBJS += builtin/fetch.o\n BUILTIN_OBJS += builtin/fmt-merge-msg.o\n BUILTIN_OBJS += builtin/for-each-ref.o\n+BUILTIN_OBJS += builtin/for-each-repo.o\n BUILTIN_OBJS += builtin/fsck.o\n BUILTIN_OBJS += builtin/gc.o\n BUILTIN_OBJS += builtin/get-tar-commit-id.o\ndiff --git a/builtin.h b/builtin.h\nindex 17c1c0ce49..ff7c6e5aa9 100644\n--- a/builtin.h\n+++ b/builtin.h\n@@ -150,6 +150,7 @@ int cmd_fetch(int argc, const char **argv, const char *prefix);\n int cmd_fetch_pack(int argc, const char **argv, const char *prefix);\n int cmd_fmt_merge_msg(int argc, const char **argv, const char *prefix);\n int cmd_for_each_ref(int argc, const char **argv, const char *prefix);\n+int cmd_for_each_repo(int argc, const char **argv, const char *prefix);\n int cmd_format_patch(int argc, const char **argv, const char *prefix);\n int cmd_fsck(int argc, const char **argv, const char *prefix);\n int cmd_gc(int argc, const char **argv, const char *prefix);\ndiff --git a/builtin/for-each-repo.c b/builtin/for-each-repo.c\nnew file mode 100644\nindex 0000000000..5bba623ff1\n--- /dev/null\n+++ b/builtin/for-each-repo.c\n@@ -0,0 +1,58 @@\n+#include \"cache.h\"\n+#include \"config.h\"\n+#include \"builtin.h\"\n+#include \"parse-options.h\"\n+#include \"run-command.h\"\n+#include \"string-list.h\"\n+\n+static const char * const for_each_repo_usage[] = {\n+\tN_(\"git for-each-repo --config= \"),\n+\tNULL\n+};\n+\n+static int run_command_on_repo(const char *path,\n+\t\t\t void *cbdata)\n+{\n+\tint i;\n+\tstruct child_process child = CHILD_PROCESS_INIT;\n+\tstruct strvec *args = (struct strvec *)cbdata;\n+\n+\tchild.git_cmd = 1;\n+\tstrvec_pushl(&child.args, \"-C\", path, NULL);\n+\n+\tfor (i = 0; i < args->nr; i++)\n+\t\tstrvec_push(&child.args, args->v[i]);\n+\n+\treturn run_command(&child);\n+}\n+\n+int cmd_for_each_repo(int argc, const char **argv, const char *prefix)\n+{\n+\tstatic const char *config_key = NULL;\n+\tint i, result = 0;\n+\tconst struct string_list *values;\n+\tstruct strvec args = STRVEC_INIT;\n+\n+\tconst struct option options[] = {\n+\t\tOPT_STRING(0, \"config\", &config_key, N_(\"config\"),\n+\t\t\t N_(\"config key storing a list of repository paths\")),\n+\t\tOPT_END()\n+\t};\n+\n+\targc = parse_options(argc, argv, prefix, options, for_each_repo_usage,\n+\t\t\t PARSE_OPT_STOP_AT_NON_OPTION);\n+\n+\tif (!config_key)\n+\t\tdie(_(\"missing --config=\"));\n+\n+\tfor (i = 0; i < argc; i++)\n+\t\tstrvec_push(&args, argv[i]);\n+\n+\tvalues = repo_config_get_value_multi(the_repository,\n+\t\t\t\t\t config_key);\n+\n+\tfor (i = 0; !result && i < values->nr; i++)\n+\t\tresult = run_command_on_repo(values->items[i].string, &args);\n+\n+\treturn result;\n+}\ndiff --git a/git.c b/git.c\nindex 24f250d29a..1cab64b5d1 100644\n--- a/git.c\n+++ b/git.c\n@@ -511,6 +511,7 @@ static struct cmd_struct commands[] = {\n \t{ \"fetch-pack\", cmd_fetch_pack, RUN_SETUP | NO_PARSEOPT },\n \t{ \"fmt-merge-msg\", cmd_fmt_merge_msg, RUN_SETUP },\n \t{ \"for-each-ref\", cmd_for_each_ref, RUN_SETUP },\n+\t{ \"for-each-repo\", cmd_for_each_repo, RUN_SETUP_GENTLY },\n \t{ \"format-patch\", cmd_format_patch, RUN_SETUP },\n \t{ \"fsck\", cmd_fsck, RUN_SETUP },\n \t{ \"fsck-objects\", cmd_fsck, RUN_SETUP },\ndiff --git a/t/t0068-for-each-repo.sh b/t/t0068-for-each-repo.sh\nnew file mode 100755\nindex 0000000000..136b4ec839\n--- /dev/null\n+++ b/t/t0068-for-each-repo.sh\n@@ -0,0 +1,30 @@\n+#!/bin/sh\n+\n+test_description='git for-each-repo builtin'\n+\n+. ./test-lib.sh\n+\n+test_expect_success 'run based on configured value' '\n+\tgit init one &&\n+\tgit init two &&\n+\tgit init three &&\n+\tgit -C two commit --allow-empty -m \"DID NOT RUN\" &&\n+\tgit config run.key \"$TRASH_DIRECTORY/one\" &&\n+\tgit config --add run.key \"$TRASH_DIRECTORY/three\" &&\n+\tgit for-each-repo --config=run.key commit --allow-empty -m \"ran\" &&\n+\tgit -C one log -1 --pretty=format:%s >message &&\n+\tgrep ran message &&\n+\tgit -C two log -1 --pretty=format:%s >message &&\n+\t! grep ran message &&\n+\tgit -C three log -1 --pretty=format:%s >message &&\n+\tgrep ran message &&\n+\tgit for-each-repo --config=run.key -- commit --allow-empty -m \"ran again\" &&\n+\tgit -C one log -1 --pretty=format:%s >message &&\n+\tgrep again message &&\n+\tgit -C two log -1 --pretty=format:%s >message &&\n+\t! grep again message &&\n+\tgit -C three log -1 --pretty=format:%s >message &&\n+\tgrep again message\n+'\n+\n+test_done\n-- \ngitgitgadget\n\n"},{"id":"404040","messageId":"4473c93b118a0e0cdb205d1758aaaa2d8bf5139a.1597857412.git.gitgitgadget@gmail.com","threadId":"54070","inReplyTo":"pull.680.git.1597857408.gitgitgadget@gmail.com","subject":"[PATCH 3/7] maintenance: add --scheduled option and config","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-08-19T17:16:44Z","receivedAt":"2020-08-19T17:17:28Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nA user may want to run certain maintenance tasks based on frequency, not\nconditions given in the repository. For example, the user may want to\nperform a 'prefetch' task every hour, or 'gc' task every day. To assist,\nupdate the 'git maintenance run --scheduled' command to check the config\nfor the last run of that task and add a number of seconds. The task\nwould then run only if the current time is beyond that minimum\ntimestamp.\n\nAdd a '--scheduled' option to 'git maintenance run' to only run tasks\nthat have had enough time pass since their last run. This is done for\neach enabled task by checking if the current timestamp is at least as\nlarge as the sum of 'maintenance..lastRun' and\n'maintenance..schedule' in the Git config. This second value is\nnew to this commit, storing a number of seconds intended between runs.\n\nA user could then set up an hourly maintenance run with the following\ncron table:\n\n 0 * * * * git -C maintenance run --scheduled\n\nThen, the user could configure the repository with the following config\nvalues:\n\n maintenance.prefetch.schedule 3000\n maintenance.gc.schedule 86000\n\nThese numbers are slightly lower than one hour and one day (in seconds).\nThe cron schedule will enforce the hourly run rate, but we can use these\nschedules to ensure the 'gc' task runs once a day. The error is given\nbecause the *.lastRun config option is specified at the _start_ of the\ntask run. Otherwise, a slow task run could shift the \"daily\" job of 'gc'\nfrom a 10:00pm run to 11:00pm run, or later.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/config/maintenance.txt | 9 +++++\n Documentation/git-maintenance.txt | 13 +++++++-\n builtin/gc.c | 50 +++++++++++++++++++++++++++-\n t/t7900-maintenance.sh | 20 +++++++++++\n 4 files changed, 90 insertions(+), 2 deletions(-)\n\ndiff --git a/Documentation/config/maintenance.txt b/Documentation/config/maintenance.txt\nindex 8dd34169da..caacacd322 100644\n--- a/Documentation/config/maintenance.txt\n+++ b/Documentation/config/maintenance.txt\n@@ -15,6 +15,15 @@ maintenance..lastRun::\n \t`` is run. It stores a timestamp representing the most-recent\n \trun of the ``.\n \n+maintenance..schedule::\n+\tThis config option controls whether or not the given `` runs\n+\tduring a `git maintenance run --scheduled` command. If the option\n+\tis an integer value `S`, then the `` is run when the current\n+\ttime is `S` seconds after the timestamp stored in\n+\t`maintenance..lastRun`. If the option has no value or a\n+\tnon-integer value, then the task will never run with the `--scheduled`\n+\toption.\n+\n maintenance.commit-graph.auto::\n \tThis integer config option controls how often the `commit-graph` task\n \tshould be run as part of `git maintenance run --auto`. If zero, then\ndiff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt\nindex 95a333f000..e8004e7b11 100644\n--- a/Documentation/git-maintenance.txt\n+++ b/Documentation/git-maintenance.txt\n@@ -110,7 +110,18 @@ OPTIONS\n \tonly if certain thresholds are met. For example, the `gc` task\n \truns when the number of loose objects exceeds the number stored\n \tin the `gc.auto` config setting, or when the number of pack-files\n-\texceeds the `gc.autoPackLimit` config setting.\n+\texceeds the `gc.autoPackLimit` config setting. Not compatible with\n+\tthe `--scheduled` option.\n+\n+--scheduled::\n+\tWhen combined with the `run` subcommand, run maintenance tasks\n+\tonly if certain time conditions are met, as specified by the\n+\t`maintenance..schedule` config value for each ``.\n+\tThis config value specifies a number of seconds since the last\n+\ttime that task ran, according to the `maintenance..lastRun`\n+\tconfig value. The tasks that are tested are those provided by\n+\tthe `--task=` option(s) or those with\n+\t`maintenance..enabled` set to true.\n \n --quiet::\n \tDo not report progress or other information over `stderr`.\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 707c144fb9..352948529d 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -711,6 +711,7 @@ static const char * const builtin_maintenance_usage[] = {\n \n struct maintenance_opts {\n \tint auto_flag;\n+\tint scheduled;\n \tint quiet;\n };\n \n@@ -1226,7 +1227,8 @@ struct maintenance_task {\n \tconst char *name;\n \tmaintenance_task_fn *fn;\n \tmaintenance_auto_fn *auto_condition;\n-\tunsigned enabled:1;\n+\tunsigned enabled:1,\n+\t\t scheduled:1;\n \n \t/* -1 if not selected. */\n \tint selected_order;\n@@ -1337,6 +1339,9 @@ static int maintenance_run(struct maintenance_opts *opts)\n \t\t !tasks[i].auto_condition()))\n \t\t\tcontinue;\n \n+\t\tif (opts->scheduled && !tasks[i].scheduled)\n+\t\t\tcontinue;\n+\n \t\tupdate_last_run(&tasks[i]);\n \n \t\ttrace2_region_enter(\"maintenance\", tasks[i].name, r);\n@@ -1351,6 +1356,29 @@ static int maintenance_run(struct maintenance_opts *opts)\n \treturn result;\n }\n \n+static void fill_schedule_info(struct maintenance_task *task,\n+\t\t\t const char *config_name,\n+\t\t\t timestamp_t schedule_delay)\n+{\n+\ttimestamp_t now = approxidate(\"now\");\n+\tchar *value = NULL;\n+\tstruct strbuf last_run = STRBUF_INIT;\n+\tint64_t previous_run;\n+\n+\tstrbuf_addf(&last_run, \"maintenance.%s.lastrun\", task->name);\n+\n+\tif (git_config_get_string(last_run.buf, &value))\n+\t\ttask->scheduled = 1;\n+\telse {\n+\t\tprevious_run = git_config_int64(last_run.buf, value);\n+\t\tif (now >= previous_run + schedule_delay)\n+\t\t\ttask->scheduled = 1;\n+\t}\n+\n+\tfree(value);\n+\tstrbuf_release(&last_run);\n+}\n+\n static void initialize_task_config(void)\n {\n \tint i;\n@@ -1359,6 +1387,7 @@ static void initialize_task_config(void)\n \n \tfor (i = 0; i < TASK__COUNT; i++) {\n \t\tint config_value;\n+\t\tchar *config_str;\n \n \t\tstrbuf_setlen(&config_name, 0);\n \t\tstrbuf_addf(&config_name, \"maintenance.%s.enabled\",\n@@ -1366,6 +1395,20 @@ static void initialize_task_config(void)\n \n \t\tif (!git_config_get_bool(config_name.buf, &config_value))\n \t\t\ttasks[i].enabled = config_value;\n+\n+\t\tstrbuf_setlen(&config_name, 0);\n+\t\tstrbuf_addf(&config_name, \"maintenance.%s.schedule\",\n+\t\t\t tasks[i].name);\n+\n+\t\tif (!git_config_get_string(config_name.buf, &config_str)) {\n+\t\t\ttimestamp_t schedule_delay = git_config_int64(\n+\t\t\t\t\t\t\tconfig_name.buf,\n+\t\t\t\t\t\t\tconfig_str);\n+\t\t\tfill_schedule_info(&tasks[i],\n+\t\t\t\t\t\tconfig_name.buf,\n+\t\t\t\t\t\tschedule_delay);\n+\t\t\tfree(config_str);\n+\t\t}\n \t}\n \n \tstrbuf_release(&config_name);\n@@ -1409,6 +1452,8 @@ int cmd_maintenance(int argc, const char **argv, const char *prefix)\n \tstruct option builtin_maintenance_options[] = {\n \t\tOPT_BOOL(0, \"auto\", &opts.auto_flag,\n \t\t\t N_(\"run tasks based on the state of the repository\")),\n+\t\tOPT_BOOL(0, \"scheduled\", &opts.scheduled,\n+\t\t\t N_(\"run tasks based on time intervals\")),\n \t\tOPT_BOOL(0, \"quiet\", &opts.quiet,\n \t\t\t N_(\"do not report progress or other information over stderr\")),\n \t\tOPT_CALLBACK_F(0, \"task\", NULL, N_(\"task\"),\n@@ -1434,6 +1479,9 @@ int cmd_maintenance(int argc, const char **argv, const char *prefix)\n \t\t\t builtin_maintenance_usage,\n \t\t\t PARSE_OPT_KEEP_UNKNOWN);\n \n+\tif (opts.auto_flag + opts.scheduled > 1)\n+\t\tdie(_(\"use at most one of the --auto and --scheduled options\"));\n+\n \tif (argc != 1)\n \t\tusage_with_options(builtin_maintenance_usage,\n \t\t\t\t builtin_maintenance_options);\ndiff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\nindex a985ce3674..3e0c5f1ca8 100755\n--- a/t/t7900-maintenance.sh\n+++ b/t/t7900-maintenance.sh\n@@ -264,6 +264,11 @@ test_expect_success 'maintenance.incremental-repack.auto' '\n \tdone\n '\n \n+test_expect_success '--auto and --scheduled incompatible' '\n+\ttest_must_fail git maintenance run --auto --scheduled 2>err &&\n+\ttest_i18ngrep \"at most one\" err\n+'\n+\n test_expect_success 'tasks update maintenance..lastRun' '\n \tgit config --unset maintenance.commit-graph.lastrun &&\n \tGIT_TRACE2_EVENT=\"$(pwd)/run.txt\" \\\n@@ -274,4 +279,19 @@ test_expect_success 'tasks update maintenance..lastRun' '\n \ttest_cmp_config 1595000000 maintenance.commit-graph.lastrun\n '\n \n+test_expect_success '--scheduled with specific time' '\n+\tgit config maintenance.commit-graph.schedule 100 &&\n+\tGIT_TRACE2_EVENT=\"$(pwd)/too-soon.txt\" \\\n+\t\tGIT_TEST_DATE_NOW=1595000099 \\\n+\t\tgit maintenance run --scheduled 2>/dev/null &&\n+\ttest_subcommand ! git commit-graph write --split --reachable \\\n+\t\t--no-progress /dev/null &&\n+\ttest_subcommand git commit-graph write --split --reachable \\\n+\t\t--no-progress \n\nIn preparation for launching background maintenance from the 'git\nmaintenance' builtin, create register/unregister subcommands. These\ncommands update the new 'maintenance.repos' config option in the global\nconfig so the background maintenance job knows which repositories to\nmaintain.\n\nThese commands allow users to add a repository to the background\nmaintenance list without disrupting the actual maintenance mechanism.\n\nFor example, a user can run 'git maintenance register' when no\nbackground maintenance is running and it will not start the background\nmaintenance. A later update to start running background maintenance will\nthen pick up this repository automatically.\n\nThe opposite example is that a user can run 'git maintenance unregister'\nto remove the current repository from background maintenance without\nhalting maintenance for other repositories.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-maintenance.txt | 14 ++++++++\n builtin/gc.c | 55 ++++++++++++++++++++++++++++++-\n t/t7900-maintenance.sh | 17 +++++++++-\n 3 files changed, 84 insertions(+), 2 deletions(-)\n\ndiff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt\nindex e8004e7b11..ac6fcae678 100644\n--- a/Documentation/git-maintenance.txt\n+++ b/Documentation/git-maintenance.txt\n@@ -29,6 +29,15 @@ Git repository.\n SUBCOMMANDS\n -----------\n \n+register::\n+\tInitialize Git config values so any scheduled maintenance will\n+\tstart running on this repository. This adds the repository to the\n+\t`maintenance.repo` config variable in the current user's global\n+\tconfig and enables some recommended configuration values for\n+\t`maintenance..schedule`. The tasks that are enabled are safe\n+\tfor running in the background without disrupting foreground\n+\tprocesses.\n+\n run::\n \tRun one or more maintenance tasks. If one or more `--task` options\n \tare specified, then those tasks are run in that order. Otherwise,\n@@ -36,6 +45,11 @@ run::\n \tconfig options are true. By default, only `maintenance.gc.enabled`\n \tis true.\n \n+unregister::\n+\tRemove the current repository from background maintenance. This\n+\tonly removes the repository from the configured list. It does not\n+\tstop the background maintenance processes from running.\n+\n TASKS\n -----\n \ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 352948529d..eb8a0a52ab 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -705,7 +705,7 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n }\n \n static const char * const builtin_maintenance_usage[] = {\n-\tN_(\"git maintenance run []\"),\n+\tN_(\"git maintenance []\"),\n \tNULL\n };\n \n@@ -1445,6 +1445,55 @@ static int task_option_parse(const struct option *opt,\n \treturn 0;\n }\n \n+static int maintenance_register(void)\n+{\n+\tstruct child_process config_set = CHILD_PROCESS_INIT;\n+\tstruct child_process config_get = CHILD_PROCESS_INIT;\n+\n+\t/* There is no current repository, so skip registering it */\n+\tif (!the_repository || !the_repository->gitdir)\n+\t\treturn 0;\n+\n+\tconfig_get.git_cmd = 1;\n+\tstrvec_pushl(&config_get.args, \"config\", \"--global\", \"--get\", \"maintenance.repo\",\n+\t\t the_repository->worktree ? the_repository->worktree\n+\t\t\t\t\t : the_repository->gitdir,\n+\t\t\t NULL);\n+\tconfig_get.out = -1;\n+\n+\tif (start_command(&config_get))\n+\t\treturn error(_(\"failed to run 'git config'\"));\n+\n+\t/* We already have this value in our config! */\n+\tif (!finish_command(&config_get))\n+\t\treturn 0;\n+\n+\tconfig_set.git_cmd = 1;\n+\tstrvec_pushl(&config_set.args, \"config\", \"--add\", \"--global\", \"maintenance.repo\",\n+\t\t the_repository->worktree ? the_repository->worktree\n+\t\t\t\t\t : the_repository->gitdir,\n+\t\t NULL);\n+\n+\treturn run_command(&config_set);\n+}\n+\n+static int maintenance_unregister(void)\n+{\n+\tstruct child_process config_unset = CHILD_PROCESS_INIT;\n+\n+\tif (!the_repository || !the_repository->gitdir)\n+\t\treturn error(_(\"no current repository to unregister\"));\n+\n+\tconfig_unset.git_cmd = 1;\n+\tstrvec_pushl(&config_unset.args, \"config\", \"--global\", \"--unset\",\n+\t\t \"maintenance.repo\",\n+\t\t the_repository->worktree ? the_repository->worktree\n+\t\t\t\t\t : the_repository->gitdir,\n+\t\t NULL);\n+\n+\treturn run_command(&config_unset);\n+}\n+\n int cmd_maintenance(int argc, const char **argv, const char *prefix)\n {\n \tint i;\n@@ -1486,8 +1535,12 @@ int cmd_maintenance(int argc, const char **argv, const char *prefix)\n \t\tusage_with_options(builtin_maintenance_usage,\n \t\t\t\t builtin_maintenance_options);\n \n+\tif (!strcmp(argv[0], \"register\"))\n+\t\treturn maintenance_register();\n \tif (!strcmp(argv[0], \"run\"))\n \t\treturn maintenance_run(&opts);\n+\tif (!strcmp(argv[0], \"unregister\"))\n+\t\treturn maintenance_unregister();\n \n \tdie(_(\"invalid subcommand: %s\"), argv[0]);\n }\ndiff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\nindex 3e0c5f1ca8..b20ee2d542 100755\n--- a/t/t7900-maintenance.sh\n+++ b/t/t7900-maintenance.sh\n@@ -9,7 +9,7 @@ GIT_TEST_MULTI_PACK_INDEX=0\n \n test_expect_success 'help text' '\n \ttest_expect_code 129 git maintenance -h 2>err &&\n-\ttest_i18ngrep \"usage: git maintenance run\" err &&\n+\ttest_i18ngrep \"usage: git maintenance \" err &&\n \ttest_expect_code 128 git maintenance barf 2>err &&\n \ttest_i18ngrep \"invalid subcommand: barf\" err\n '\n@@ -294,4 +294,19 @@ test_expect_success '--scheduled with specific time' '\n \ttest_cmp_config 1595000100 maintenance.commit-graph.lastrun\n '\n \n+test_expect_success 'register and unregister' '\n+\ttest_when_finished git config --global --unset-all maintenance.repo &&\n+\tgit config --global --add maintenance.repo /existing1 &&\n+\tgit config --global --add maintenance.repo /existing2 &&\n+\tgit config --global --get-all maintenance.repo >before &&\n+\tgit maintenance register &&\n+\tgit config --global --get-all maintenance.repo >actual &&\n+\tcp before after &&\n+\tpwd >>after &&\n+\ttest_cmp after actual &&\n+\tgit maintenance unregister &&\n+\tgit config --global --get-all maintenance.repo >actual &&\n+\ttest_cmp before actual\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"404042","messageId":"2442071fd0581a221802e348182ed04267ef3044.1597857412.git.gitgitgadget@gmail.com","threadId":"54070","inReplyTo":"pull.680.git.1597857408.gitgitgadget@gmail.com","subject":"[PATCH 6/7] maintenance: add start/stop subcommands","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-08-19T17:16:47Z","receivedAt":"2020-08-19T17:17:43Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nAdd new subcommands to 'git maintenance' that start or stop background\nmaintenance using 'cron', when available. This integration is as simple\nas I could make it, barring some implementation complications.\n\nFor now, the background maintenance is scheduled to run hourly via the\nfollowing cron table row (ignore line breaks):\n\n\t0 * * * * $p/git --exec-path=$p\n\t\tfor-each-repo --config=maintenance.repo\n\t\tmaintenance run --scheduled\n\nFuture extensions may want to add more complex schedules or some form of\nlogging. For now, hourly runs seem frequent enough to satisfy the needs\nof tasks like 'prefetch' without being so frequent that users would\ncomplain about many no-op commands.\n\nHere, \"$p\" is a placeholder for the path to the current Git executable.\nThis is critical for systems with multiple versions of Git.\nSpecifically, macOS has a system version at '/usr/bin/git' while the\nversion that users can install resides at '/usr/local/bin/git' (symlinked\nto '/usr/local/libexec/git-core/git'). This will also use your\nlocally-built version if you build and run this in your development\nenvironment without installing first.\n\nThe GIT_TEST_CRONTAB environment variable is not intended for users to\nedit, but instead as a way to mock the 'crontab [-l]' command. This\nvariable is set in test-lib.sh to avoid a future test from accidentally\nrunning anything with the cron integration from modifying the user's\nschedule. We use GIT_TEST_CRONTAB='test-tool crontab ' in our\ntests to check how the schedule is modified in 'git maintenance\n(start|stop)' commands.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-maintenance.txt | 11 +++\n Makefile | 1 +\n builtin/gc.c | 117 ++++++++++++++++++++++++++++++\n t/helper/test-crontab.c | 35 +++++++++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/t7900-maintenance.sh | 30 ++++++++\n t/test-lib.sh | 6 ++\n 8 files changed, 202 insertions(+)\n create mode 100644 t/helper/test-crontab.c\n\ndiff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt\nindex ac6fcae678..600272caa5 100644\n--- a/Documentation/git-maintenance.txt\n+++ b/Documentation/git-maintenance.txt\n@@ -45,6 +45,17 @@ run::\n \tconfig options are true. By default, only `maintenance.gc.enabled`\n \tis true.\n \n+start::\n+\tStart running maintenance on the current repository. This performs\n+\tthe same config updates as the `register` subcommand, then updates\n+\tthe background scheduler to run `git maintenance run --scheduled`\n+\ton an hourly basis.\n+\n+stop::\n+\tHalt the background maintenance schedule. The current repository\n+\tis not removed from the list of maintained repositories, in case\n+\tthe background maintenance is restarted later.\n+\n unregister::\n \tRemove the current repository from background maintenance. This\n \tonly removes the repository from the configured list. It does not\ndiff --git a/Makefile b/Makefile\nindex 7c588ff036..c39b39bd7d 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -690,6 +690,7 @@ TEST_BUILTINS_OBJS += test-advise.o\n TEST_BUILTINS_OBJS += test-bloom.o\n TEST_BUILTINS_OBJS += test-chmtime.o\n TEST_BUILTINS_OBJS += test-config.o\n+TEST_BUILTINS_OBJS += test-crontab.o\n TEST_BUILTINS_OBJS += test-ctype.o\n TEST_BUILTINS_OBJS += test-date.o\n TEST_BUILTINS_OBJS += test-delta.o\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex eb8a0a52ab..b09287f0fc 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -32,6 +32,7 @@\n #include \"remote.h\"\n #include \"midx.h\"\n #include \"object-store.h\"\n+#include \"exec-cmd.h\"\n \n #define FAILED_RUN \"failed to run %s\"\n \n@@ -1494,6 +1495,118 @@ static int maintenance_unregister(void)\n \treturn run_command(&config_unset);\n }\n \n+#define BEGIN_LINE \"# BEGIN GIT MAINTENANCE SCHEDULE\"\n+#define END_LINE \"# END GIT MAINTENANCE SCHEDULE\"\n+\n+static int update_background_schedule(int run_maintenance)\n+{\n+\tint result = 0;\n+\tint in_old_region = 0;\n+\tstruct child_process crontab_list = CHILD_PROCESS_INIT;\n+\tstruct child_process crontab_edit = CHILD_PROCESS_INIT;\n+\tFILE *cron_list, *cron_in;\n+\tconst char *crontab_name;\n+\tstruct strbuf line = STRBUF_INIT;\n+\tstruct lock_file lk;\n+\tchar *lock_path = xstrfmt(\"%s/schedule\", the_repository->objects->odb->path);\n+\n+\tif (hold_lock_file_for_update(&lk, lock_path, LOCK_NO_DEREF) < 0)\n+\t\treturn error(_(\"another process is scheduling background maintenance\"));\n+\n+\tcrontab_name = getenv(\"GIT_TEST_CRONTAB\");\n+\tif (!crontab_name)\n+\t\tcrontab_name = \"crontab\";\n+\n+\tstrvec_split(&crontab_list.args, crontab_name);\n+\tstrvec_push(&crontab_list.args, \"-l\");\n+\tcrontab_list.in = -1;\n+\tcrontab_list.out = dup(lk.tempfile->fd);\n+\tcrontab_list.git_cmd = 0;\n+\n+\tif (start_command(&crontab_list)) {\n+\t\tresult = error(_(\"failed to run 'crontab -l'; your system might not support 'cron'\"));\n+\t\tgoto cleanup;\n+\t}\n+\n+\t/* Ignore exit code, as an empty crontab will return error. */\n+\tfinish_command(&crontab_list);\n+\n+\t/*\n+\t * Read from the .lock file, filtering out the old\n+\t * schedule while appending the new schedule.\n+\t */\n+\tcron_list = fdopen(lk.tempfile->fd, \"r\");\n+\trewind(cron_list);\n+\n+\tstrvec_split(&crontab_edit.args, crontab_name);\n+\tcrontab_edit.in = -1;\n+\tcrontab_edit.git_cmd = 0;\n+\n+\tif (start_command(&crontab_edit)) {\n+\t\tresult = error(_(\"failed to run 'crontab'; your system might not support 'cron'\"));\n+\t\tgoto cleanup;\n+\t}\n+\n+\tcron_in = fdopen(crontab_edit.in, \"w\");\n+\tif (!cron_in) {\n+\t\tresult = error(_(\"failed to open stdin of 'crontab'\"));\n+\t\tgoto done_editing;\n+\t}\n+\n+\twhile (!strbuf_getline_lf(&line, cron_list)) {\n+\t\tif (!in_old_region && !strcmp(line.buf, BEGIN_LINE))\n+\t\t\tin_old_region = 1;\n+\t\tif (in_old_region)\n+\t\t\tcontinue;\n+\t\tfprintf(cron_in, \"%s\\n\", line.buf);\n+\t\tif (in_old_region && !strcmp(line.buf, END_LINE))\n+\t\t\tin_old_region = 0;\n+\t}\n+\n+\tif (run_maintenance) {\n+\t\tconst char *exec_path = git_exec_path();\n+\n+\t\tfprintf(cron_in, \"\\n%s\\n\", BEGIN_LINE);\n+\t\tfprintf(cron_in, \"# The following schedule was created by Git\\n\");\n+\t\tfprintf(cron_in, \"# Any edits made in this region might be\\n\");\n+\t\tfprintf(cron_in, \"# replaced in the future by a Git command.\\n\\n\");\n+\n+\t\tfprintf(cron_in,\n+\t\t\t\"0 * * * * \\\"%s/git\\\" --exec-path=\\\"%s\\\" for-each-repo --config=maintenance.repo maintenance run --scheduled\\n\",\n+\t\t\texec_path, exec_path);\n+\n+\t\tfprintf(cron_in, \"\\n%s\\n\", END_LINE);\n+\t}\n+\n+\tfflush(cron_in);\n+\tfclose(cron_in);\n+\tclose(crontab_edit.in);\n+\n+done_editing:\n+\tif (finish_command(&crontab_edit)) {\n+\t\tresult = error(_(\"'crontab' died\"));\n+\t\tgoto cleanup;\n+\t}\n+\tfclose(cron_list);\n+\n+cleanup:\n+\trollback_lock_file(&lk);\n+\treturn result;\n+}\n+\n+static int maintenance_start(void)\n+{\n+\tif (maintenance_register())\n+\t\twarning(_(\"failed to add repo to global config\"));\n+\n+\treturn update_background_schedule(1);\n+}\n+\n+static int maintenance_stop(void)\n+{\n+\treturn update_background_schedule(0);\n+}\n+\n int cmd_maintenance(int argc, const char **argv, const char *prefix)\n {\n \tint i;\n@@ -1539,6 +1652,10 @@ int cmd_maintenance(int argc, const char **argv, const char *prefix)\n \t\treturn maintenance_register();\n \tif (!strcmp(argv[0], \"run\"))\n \t\treturn maintenance_run(&opts);\n+\tif (!strcmp(argv[0], \"start\"))\n+\t\treturn maintenance_start();\n+\tif (!strcmp(argv[0], \"stop\"))\n+\t\treturn maintenance_stop();\n \tif (!strcmp(argv[0], \"unregister\"))\n \t\treturn maintenance_unregister();\n \ndiff --git a/t/helper/test-crontab.c b/t/helper/test-crontab.c\nnew file mode 100644\nindex 0000000000..f5db6319c6\n--- /dev/null\n+++ b/t/helper/test-crontab.c\n@@ -0,0 +1,35 @@\n+#include \"test-tool.h\"\n+#include \"cache.h\"\n+\n+/*\n+ * Usage: test-tool cron [-l]\n+ *\n+ * If -l is specified, then write the contents of to stdou.\n+ * Otherwise, write from stdin into .\n+ */\n+int cmd__crontab(int argc, const char **argv)\n+{\n+\tchar a;\n+\tFILE *from, *to;\n+\n+\tif (argc == 3 && !strcmp(argv[2], \"-l\")) {\n+\t\tfrom = fopen(argv[1], \"r\");\n+\t\tif (!from)\n+\t\t\treturn 0;\n+\t\tto = stdout;\n+\t} else if (argc == 2) {\n+\t\tfrom = stdin;\n+\t\tto = fopen(argv[1], \"w\");\n+\t} else\n+\t\treturn error(\"unknown arguments\");\n+\n+\twhile ((a = fgetc(from)) != EOF)\n+\t\tfputc(a, to);\n+\n+\tif (argc == 3)\n+\t\tfclose(from);\n+\telse\n+\t\tfclose(to);\n+\n+\treturn 0;\n+}\ndiff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\nindex 590b2efca7..432b49d948 100644\n--- a/t/helper/test-tool.c\n+++ b/t/helper/test-tool.c\n@@ -18,6 +18,7 @@ static struct test_cmd cmds[] = {\n \t{ \"bloom\", cmd__bloom },\n \t{ \"chmtime\", cmd__chmtime },\n \t{ \"config\", cmd__config },\n+\t{ \"crontab\", cmd__crontab },\n \t{ \"ctype\", cmd__ctype },\n \t{ \"date\", cmd__date },\n \t{ \"delta\", cmd__delta },\ndiff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\nindex ddc8e990e9..7c3281e071 100644\n--- a/t/helper/test-tool.h\n+++ b/t/helper/test-tool.h\n@@ -8,6 +8,7 @@ int cmd__advise_if_enabled(int argc, const char **argv);\n int cmd__bloom(int argc, const char **argv);\n int cmd__chmtime(int argc, const char **argv);\n int cmd__config(int argc, const char **argv);\n+int cmd__crontab(int argc, const char **argv);\n int cmd__ctype(int argc, const char **argv);\n int cmd__date(int argc, const char **argv);\n int cmd__delta(int argc, const char **argv);\ndiff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\nindex b20ee2d542..6491031be8 100755\n--- a/t/t7900-maintenance.sh\n+++ b/t/t7900-maintenance.sh\n@@ -309,4 +309,34 @@ test_expect_success 'register and unregister' '\n \ttest_cmp before actual\n '\n \n+test_expect_success 'start from empty cron table' '\n+\tGIT_TEST_CRONTAB=\"test-tool crontab cron.txt\" git maintenance start &&\n+\n+\t# start registers the repo\n+\tgit config --get --global maintenance.repo \"$(pwd)\" &&\n+\n+\tgrep \"for-each-repo --config=maintenance.repo maintenance run --scheduled\" cron.txt\n+'\n+\n+test_expect_success 'stop from existing schedule' '\n+\tGIT_TEST_CRONTAB=\"test-tool crontab cron.txt\" git maintenance stop &&\n+\n+\t# stop does not unregister the repo\n+\tgit config --get --global maintenance.repo \"$(pwd)\" &&\n+\n+\t# The newline is preserved\n+\techo >empty &&\n+\ttest_cmp empty cron.txt &&\n+\n+\t# Operation is idempotent\n+\tGIT_TEST_CRONTAB=\"test-tool crontab cron.txt\" git maintenance stop &&\n+\ttest_cmp empty cron.txt\n+'\n+\n+test_expect_success 'start preserves existing schedule' '\n+\techo \"Important information!\" >cron.txt &&\n+\tGIT_TEST_CRONTAB=\"test-tool crontab cron.txt\" git maintenance start &&\n+\tgrep \"Important information!\" cron.txt\n+'\n+\n test_done\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex ef31f40037..4a60d1ed76 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -1702,3 +1702,9 @@ test_lazy_prereq SHA1 '\n test_lazy_prereq REBASE_P '\n \ttest -z \"$GIT_TEST_SKIP_REBASE_P\"\n '\n+\n+# Ensure that no test accidentally triggers a Git command\n+# that runs 'crontab', affecting a user's cron schedule.\n+# Tests that verify the cron integration must set this locally\n+# to avoid errors.\n+GIT_TEST_CRONTAB=\"exit 1\"\n-- \ngitgitgadget\n\n"},{"id":"404043","messageId":"40b1a0546ccea737df608aaffe90edf56377b4ce.1597857412.git.gitgitgadget@gmail.com","threadId":"54070","inReplyTo":"pull.680.git.1597857408.gitgitgadget@gmail.com","subject":"[PATCH 7/7] maintenance: recommended schedule in register/start","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-08-19T17:16:48Z","receivedAt":"2020-08-19T17:17:45Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe 'git maintenance (register|start)' subcommands add the current\nrepository to the global Git config so maintenance will operate on that\nrepository. It does not specify what maintenance should occur or how\noften.\n\nIf a user sets any 'maintenance..scheduled' config value, then\nthey have chosen a specific schedule for themselves and Git should\nrespect that.\n\nHowever, in an effort to recommend a good schedule for repositories of\nall sizes, set new config values for recommended tasks that are safe to\nrun in the background while users run foreground Git commands. These\ncommands are generally everything but the 'gc' task.\n\nAuthor's Note: I feel we should do _something_ to recommend a good\nschedule to users, but I'm not 100% set on this schedule. This is the\nschedule we use in Scalar and VFS for Git for very large repositories\nusing the GVFS protocol. While the schedule works in that environment,\nit is possible that \"normal\" Git repositories could benefit from\nsomething more obvious (such as running 'gc' once a day). However, this\npatch gives us a place to start a conversation on what we should\nrecommend. For my purposes, Scalar will set these config values so we\ncan always differ from core Git's recommendations.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-maintenance.txt | 6 +++++\n builtin/gc.c | 44 +++++++++++++++++++++++++++++++\n t/t7900-maintenance.sh | 5 ++++\n 3 files changed, 55 insertions(+)\n\ndiff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt\nindex 600272caa5..a4b46ea329 100644\n--- a/Documentation/git-maintenance.txt\n+++ b/Documentation/git-maintenance.txt\n@@ -37,6 +37,12 @@ register::\n \t`maintenance..schedule`. The tasks that are enabled are safe\n \tfor running in the background without disrupting foreground\n \tprocesses.\n++\n+If your repository has no 'maintenance..schedule' configuration\n+values set, then Git will set configuration values to some recommended\n+settings. These settings disable foreground maintenance while performing\n+maintenance tasks in the background that will not interrupt foreground Git\n+operations.\n \n run::\n \tRun one or more maintenance tasks. If one or more `--task` options\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex b09287f0fc..080d58735c 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -1446,6 +1446,47 @@ static int task_option_parse(const struct option *opt,\n \treturn 0;\n }\n \n+static int has_schedule_config(void)\n+{\n+\tint i, found = 0;\n+\tstruct strbuf config_name = STRBUF_INIT;\n+\tsize_t prefix;\n+\n+\tstrbuf_addstr(&config_name, \"maintenance.\");\n+\tprefix = config_name.len;\n+\n+\tfor (i = 0; !found && i < TASK__COUNT; i++) {\n+\t\tint value;\n+\n+\t\tstrbuf_setlen(&config_name, prefix);\n+\t\tstrbuf_addf(&config_name, \"%s.schedule\", tasks[i].name);\n+\n+\t\tif (!git_config_get_int(config_name.buf, &value))\n+\t\t\tfound = 1;\n+\t}\n+\n+\tstrbuf_release(&config_name);\n+\treturn found;\n+}\n+\n+static void set_recommended_schedule(void)\n+{\n+\tgit_config_set(\"maintenance.auto\", \"false\");\n+\tgit_config_set(\"maintenance.gc.enabled\", \"false\");\n+\n+\tgit_config_set(\"maintenance.prefetch.enabled\", \"true\");\n+\tgit_config_set(\"maintenance.prefetch.schedule\", \"3500\");\n+\n+\tgit_config_set(\"maintenance.commit-graph.enabled\", \"true\");\n+\tgit_config_set(\"maintenance.commit-graph.schedule\", \"3500\");\n+\n+\tgit_config_set(\"maintenance.loose-objects.enabled\", \"true\");\n+\tgit_config_set(\"maintenance.loose-objects.schedule\", \"86000\");\n+\n+\tgit_config_set(\"maintenance.incremental-repack.enabled\", \"true\");\n+\tgit_config_set(\"maintenance.incremental-repack.schedule\", \"86000\");\n+}\n+\n static int maintenance_register(void)\n {\n \tstruct child_process config_set = CHILD_PROCESS_INIT;\n@@ -1455,6 +1496,9 @@ static int maintenance_register(void)\n \tif (!the_repository || !the_repository->gitdir)\n \t\treturn 0;\n \n+\tif (has_schedule_config())\n+\t\tset_recommended_schedule();\n+\n \tconfig_get.git_cmd = 1;\n \tstrvec_pushl(&config_get.args, \"config\", \"--global\", \"--get\", \"maintenance.repo\",\n \t\t the_repository->worktree ? the_repository->worktree\ndiff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\nindex 6491031be8..7417e5858a 100755\n--- a/t/t7900-maintenance.sh\n+++ b/t/t7900-maintenance.sh\n@@ -300,6 +300,11 @@ test_expect_success 'register and unregister' '\n \tgit config --global --add maintenance.repo /existing2 &&\n \tgit config --global --get-all maintenance.repo >before &&\n \tgit maintenance register &&\n+\ttest_cmp_config false maintenance.auto &&\n+\ttest_cmp_config false maintenance.gc.enabled &&\n+\ttest_cmp_config true maintenance.prefetch.enabled &&\n+\ttest_cmp_config 3500 maintenance.commit-graph.schedule &&\n+\ttest_cmp_config 86000 maintenance.incremental-repack.schedule &&\n \tgit config --global --get-all maintenance.repo >actual &&\n \tcp before after &&\n \tpwd >>after &&\n-- \ngitgitgadget\n"},{"id":"404080","messageId":"20200820020658.GA31084@danh.dev","threadId":"54070","inReplyTo":"90de25d1287595fdedc9dcd194a2e0ac120d4aad.1597857409.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 1/7] maintenance: optionally skip --auto process","fromName":"Đoàn Trần Công Danh","fromEmail":"congdanhqx@gmail.com","sentAt":"2020-08-20T02:06:58Z","receivedAt":"2020-08-20T02:07:05Z","isPatch":true,"sender":{"key":"congdanhqx@gmail.com","avatar":"https://avatars.githubusercontent.com/u/42673067?v=4"},"body":"On 2020-08-19 17:16:42+0000, Derrick Stolee via GitGitGadget wrote:\n> From: Derrick Stolee \n> @@ -1868,8 +1869,15 @@ int run_processes_parallel_tr2(int n, get_next_task_fn get_next_task,\n> \n> int run_auto_maintenance(int quiet)\n> {\n> +\tint enabled;\n> \tstruct child_process maint = CHILD_PROCESS_INIT;\n> \n> +\tif (!git_config_get_bool(\"maintenance.auto\", &enabled) &&\n> +\t !enabled) {\n> +\t\t fprintf(stderr, \"enabled: %d\\n\", enabled);\n> +\t\treturn 0;\n> +\t }\n\nNit: This block of code is mis-indented (mixed space and tab).\n\nIf we're running into inner block, \"enabled\" will always be \"0\"\nWe can just write:\n\n\tfprintf(stderr, \"enabled: 0\\n\");\n\ninstead. Writing like that, we have one less thing to worry about\n(whether \"enabled\" is initialised in git_config_get_bool or not).\n\n\n-- \nDanh\n"},{"id":"404086","messageId":"629e3f77-a80f-2014-1756-dec8d8647326@gmail.com","threadId":"54070","inReplyTo":"20200820020658.GA31084@danh.dev","subject":"Re: [PATCH 1/7] maintenance: optionally skip --auto process","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-08-20T12:12:04Z","receivedAt":"2020-08-20T12:12:51Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 8/19/2020 10:06 PM, Đoàn Trần Công Danh wrote:\n> On 2020-08-19 17:16:42+0000, Derrick Stolee via GitGitGadget wrote:\n>> From: Derrick Stolee \n>> @@ -1868,8 +1869,15 @@ int run_processes_parallel_tr2(int n, get_next_task_fn get_next_task,\n>> \n>> int run_auto_maintenance(int quiet)\n>> {\n>> +\tint enabled;\n>> \tstruct child_process maint = CHILD_PROCESS_INIT;\n>> \n>> +\tif (!git_config_get_bool(\"maintenance.auto\", &enabled) &&\n>> +\t !enabled) {\n>> +\t\t fprintf(stderr, \"enabled: %d\\n\", enabled);\n>> +\t\treturn 0;\n>> +\t }\n> \n> Nit: This block of code is mis-indented (mixed space and tab).\n\nThanks.\n \n> If we're running into inner block, \"enabled\" will always be \"0\"\n> We can just write:\n> \n> \tfprintf(stderr, \"enabled: 0\\n\");\n> \n> instead. Writing like that, we have one less thing to worry about\n> (whether \"enabled\" is initialised in git_config_get_bool or not).\n\nWhoops! This fprintf shouldn't even be here, but got accidentally\nadded during a debugging session. Thanks for noticing!\n\n-Stolee\n\n"},{"id":"404095","messageId":"20200820143435.GB31084@danh.dev","threadId":"54070","inReplyTo":"bdc27fa28ee70222ed3c7c9863746ace8ea835e4.1597857409.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 2/7] maintenance: store the \"last run\" time in config","fromName":"Đoàn Trần Công Danh","fromEmail":"congdanhqx@gmail.com","sentAt":"2020-08-20T14:34:46Z","receivedAt":"2020-08-20T14:34:57Z","isPatch":true,"sender":{"key":"congdanhqx@gmail.com","avatar":"https://avatars.githubusercontent.com/u/42673067?v=4"},"body":"On 2020-08-19 17:16:43+0000, Derrick Stolee via GitGitGadget wrote:\n> From: Derrick Stolee \n> \n> Users may want to run certain maintenance tasks only so often. Update\n> the local config with a new 'maintenance..lastRun' config option\n> that stores the timestamp just before running the maintenance task.\n> \n> I selected the timestamp before the task, as opposed to after the task,\n> for a couple reasons:\n> \n> 1. The time the task takes to execute should not contribute to the\n> interval between running the tasks. If a daily task takes 10 minutes\n> to run, then every day the execution will drift by at least 10\n> minutes.\n> \n> 2. If the task fails for some unforseen reason, it would be good to\n> indicate that we _attempted_ the task at a certain timestamp. This\n> will avoid spamming a repository that is in a bad state.\n> \n> Signed-off-by: Derrick Stolee \n> ---\n> Documentation/config/maintenance.txt | 5 +++++\n> builtin/gc.c | 16 ++++++++++++++++\n> git-gvfs-helper | Bin 0 -> 11171736 bytes\n> t/helper/test-gvfs-protocol | Bin 0 -> 10946928 bytes\n\nLook like those 2 files should be added into .gitignore, no?\n\n-- \nDanh\n"},{"id":"404097","messageId":"20200820145127.GC31084@danh.dev","threadId":"54070","inReplyTo":"4473c93b118a0e0cdb205d1758aaaa2d8bf5139a.1597857412.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 3/7] maintenance: add --scheduled option and config","fromName":"Đoàn Trần Công Danh","fromEmail":"congdanhqx@gmail.com","sentAt":"2020-08-20T14:51:27Z","receivedAt":"2020-08-20T14:51:42Z","isPatch":true,"sender":{"key":"congdanhqx@gmail.com","avatar":"https://avatars.githubusercontent.com/u/42673067?v=4"},"body":"On 2020-08-19 17:16:44+0000, Derrick Stolee via GitGitGadget wrote:\n> +static void fill_schedule_info(struct maintenance_task *task,\n> +\t\t\t const char *config_name,\n> +\t\t\t timestamp_t schedule_delay)\n> +{\n> +\ttimestamp_t now = approxidate(\"now\");\n\nI see this pattern in both previous patch and this patch,\nshould we create a helper (if not exist) to get current timestamp\ninstead, parsing \"now\" every now and then is not a good idea, in my\nvery opinionated opinion.\n\n> +\tchar *value = NULL;\n> +\tstruct strbuf last_run = STRBUF_INIT;\n> +\tint64_t previous_run;\n> +\n> +\tstrbuf_addf(&last_run, \"maintenance.%s.lastrun\", task->name);\n> +\n> +\tif (git_config_get_string(last_run.buf, &value))\n> +\t\ttask->scheduled = 1;\n> +\telse {\n> +\t\tprevious_run = git_config_int64(last_run.buf, value);\n> +\t\tif (now >= previous_run + schedule_delay)\n> +\t\t\ttask->scheduled = 1;\n> +\t}\n> +\n> +\tfree(value);\n> +\tstrbuf_release(&last_run);\n> +}\n> +\n> static void initialize_task_config(void)\n> {\n> \tint i;\n> @@ -1359,6 +1387,7 @@ static void initialize_task_config(void)\n> \n> \tfor (i = 0; i < TASK__COUNT; i++) {\n> \t\tint config_value;\n> +\t\tchar *config_str;\n> \n> \t\tstrbuf_setlen(&config_name, 0);\n> \t\tstrbuf_addf(&config_name, \"maintenance.%s.enabled\",\n> @@ -1366,6 +1395,20 @@ static void initialize_task_config(void)\n> \n> \t\tif (!git_config_get_bool(config_name.buf, &config_value))\n> \t\t\ttasks[i].enabled = config_value;\n> +\n> +\t\tstrbuf_setlen(&config_name, 0);\n\nIt looks like we have a simple and better named alias for this:\n\n\tstrbuf_reset(&config_name)\n\n_reset has 400+ occurences in this code base, compare to 20 of _setlen\n\n\n-- \nDanh\n"},{"id":"404098","messageId":"20200820150048.GD31084@danh.dev","threadId":"54070","inReplyTo":"ccb667dc6fa4704171da643158ecd79ed7dd2cd3.1597857412.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 4/7] for-each-repo: run subcommands on configured repos","fromName":"Đoàn Trần Công Danh","fromEmail":"congdanhqx@gmail.com","sentAt":"2020-08-20T15:00:48Z","receivedAt":"2020-08-20T15:00:59Z","isPatch":true,"sender":{"key":"congdanhqx@gmail.com","avatar":"https://avatars.githubusercontent.com/u/42673067?v=4"},"body":"On 2020-08-19 17:16:45+0000, Derrick Stolee via GitGitGadget wrote:\n> From: Derrick Stolee \n> \n> It can be helpful to store a list of repositories in global or system\n> config and then iterate Git commands on that list. Create a new builtin\n> that makes this process simple for experts. We will use this builtin to\n> run scheduled maintenance on all configured repositories in a future\n> change.\n\nNice, I like this new command.\n\nHowever, I'm not sure if we could declare this command as plumbing or\nporcelain command.\n\nI guess this command is meant more for scripting purpose, hence, it\nshould be plumbing, thus we need to define clear protocol for this\ncommand, e.g, where it will redirect other command output, error to,\nwhere for-each-repo write its own output/error.\n\nAlso, I think it would be nice to declare this is experimental for now.\nLike we declared git-switch and git-restore.\n\n-- \nDanh\n\n> \n> The test is very simple, but does highlight that the \"--\" argument is\n> optional.\n> \n> Signed-off-by: Derrick Stolee \n> ---\n> .gitignore | 1 +\n> Documentation/git-for-each-repo.txt | 45 ++++++++++++++++++++++\n> Makefile | 1 +\n> builtin.h | 1 +\n> builtin/for-each-repo.c | 58 +++++++++++++++++++++++++++++\n> git.c | 1 +\n> t/t0068-for-each-repo.sh | 30 +++++++++++++++\n> 7 files changed, 137 insertions(+)\n> create mode 100644 Documentation/git-for-each-repo.txt\n> create mode 100644 builtin/for-each-repo.c\n> create mode 100755 t/t0068-for-each-repo.sh\n> \n> diff --git a/.gitignore b/.gitignore\n> index a5808fa30d..5eb2a2be71 100644\n> --- a/.gitignore\n> +++ b/.gitignore\n> @@ -67,6 +67,7 @@\n> /git-filter-branch\n> /git-fmt-merge-msg\n> /git-for-each-ref\n> +/git-for-each-repo\n> /git-format-patch\n> /git-fsck\n> /git-fsck-objects\n> diff --git a/Documentation/git-for-each-repo.txt b/Documentation/git-for-each-repo.txt\n> new file mode 100644\n> index 0000000000..83b06db410\n> --- /dev/null\n> +++ b/Documentation/git-for-each-repo.txt\n> @@ -0,0 +1,45 @@\n> +git-for-each-repo(1)\n> +====================\n> +\n> +NAME\n> +----\n> +git-for-each-repo - Run a Git command on a list of repositories\n> +\n> +\n> +SYNOPSIS\n> +--------\n> +[verse]\n> +'git for-each-repo' --config= [--] \n> +\n> +\n> +DESCRIPTION\n> +-----------\n> +Run a Git commands on a list of repositories. The arguments after the\n> +known options or `--` indicator are used as the arguments for the Git\n> +command.\n> +\n> +For example, we could run maintenance on each of a list of repositories\n> +stored in a `maintenance.repo` config variable using\n> +\n> +-------------\n> +git for-each-repo --config=maintenance.repo maintenance run\n> +-------------\n> +\n> +This will run `git -C maintenance run` for each value ``\n> +in the multi-valued config variable `maintenance.repo`.\n> +\n> +\n> +OPTIONS\n> +-------\n> +--config=::\n> +\tUse the given config variable as a multi-valued list storing\n> +\tabsolute path names. Iterate on that list of paths to run\n> +\tthe given arguments.\n> ++\n> +These config values are loaded from system, global, and local Git config,\n> +as available. If `git for-each-repo` is run in a directory that is not a\n> +Git repository, then only the system and global config is used.\n> +\n> +GIT\n> +---\n> +Part of the linkgit:git[1] suite\n> diff --git a/Makefile b/Makefile\n> index 65f8cfb236..7c588ff036 100644\n> --- a/Makefile\n> +++ b/Makefile\n> @@ -1071,6 +1071,7 @@ BUILTIN_OBJS += builtin/fetch-pack.o\n> BUILTIN_OBJS += builtin/fetch.o\n> BUILTIN_OBJS += builtin/fmt-merge-msg.o\n> BUILTIN_OBJS += builtin/for-each-ref.o\n> +BUILTIN_OBJS += builtin/for-each-repo.o\n> BUILTIN_OBJS += builtin/fsck.o\n> BUILTIN_OBJS += builtin/gc.o\n> BUILTIN_OBJS += builtin/get-tar-commit-id.o\n> diff --git a/builtin.h b/builtin.h\n> index 17c1c0ce49..ff7c6e5aa9 100644\n> --- a/builtin.h\n> +++ b/builtin.h\n> @@ -150,6 +150,7 @@ int cmd_fetch(int argc, const char **argv, const char *prefix);\n> int cmd_fetch_pack(int argc, const char **argv, const char *prefix);\n> int cmd_fmt_merge_msg(int argc, const char **argv, const char *prefix);\n> int cmd_for_each_ref(int argc, const char **argv, const char *prefix);\n> +int cmd_for_each_repo(int argc, const char **argv, const char *prefix);\n> int cmd_format_patch(int argc, const char **argv, const char *prefix);\n> int cmd_fsck(int argc, const char **argv, const char *prefix);\n> int cmd_gc(int argc, const char **argv, const char *prefix);\n> diff --git a/builtin/for-each-repo.c b/builtin/for-each-repo.c\n> new file mode 100644\n> index 0000000000..5bba623ff1\n> --- /dev/null\n> +++ b/builtin/for-each-repo.c\n> @@ -0,0 +1,58 @@\n> +#include \"cache.h\"\n> +#include \"config.h\"\n> +#include \"builtin.h\"\n> +#include \"parse-options.h\"\n> +#include \"run-command.h\"\n> +#include \"string-list.h\"\n> +\n> +static const char * const for_each_repo_usage[] = {\n> +\tN_(\"git for-each-repo --config= \"),\n> +\tNULL\n> +};\n> +\n> +static int run_command_on_repo(const char *path,\n> +\t\t\t void *cbdata)\n> +{\n> +\tint i;\n> +\tstruct child_process child = CHILD_PROCESS_INIT;\n> +\tstruct strvec *args = (struct strvec *)cbdata;\n> +\n> +\tchild.git_cmd = 1;\n> +\tstrvec_pushl(&child.args, \"-C\", path, NULL);\n> +\n> +\tfor (i = 0; i < args->nr; i++)\n> +\t\tstrvec_push(&child.args, args->v[i]);\n> +\n> +\treturn run_command(&child);\n> +}\n> +\n> +int cmd_for_each_repo(int argc, const char **argv, const char *prefix)\n> +{\n> +\tstatic const char *config_key = NULL;\n> +\tint i, result = 0;\n> +\tconst struct string_list *values;\n> +\tstruct strvec args = STRVEC_INIT;\n> +\n> +\tconst struct option options[] = {\n> +\t\tOPT_STRING(0, \"config\", &config_key, N_(\"config\"),\n> +\t\t\t N_(\"config key storing a list of repository paths\")),\n> +\t\tOPT_END()\n> +\t};\n> +\n> +\targc = parse_options(argc, argv, prefix, options, for_each_repo_usage,\n> +\t\t\t PARSE_OPT_STOP_AT_NON_OPTION);\n> +\n> +\tif (!config_key)\n> +\t\tdie(_(\"missing --config=\"));\n> +\n> +\tfor (i = 0; i < argc; i++)\n> +\t\tstrvec_push(&args, argv[i]);\n> +\n> +\tvalues = repo_config_get_value_multi(the_repository,\n> +\t\t\t\t\t config_key);\n> +\n> +\tfor (i = 0; !result && i < values->nr; i++)\n> +\t\tresult = run_command_on_repo(values->items[i].string, &args);\n> +\n> +\treturn result;\n> +}\n> diff --git a/git.c b/git.c\n> index 24f250d29a..1cab64b5d1 100644\n> --- a/git.c\n> +++ b/git.c\n> @@ -511,6 +511,7 @@ static struct cmd_struct commands[] = {\n> \t{ \"fetch-pack\", cmd_fetch_pack, RUN_SETUP | NO_PARSEOPT },\n> \t{ \"fmt-merge-msg\", cmd_fmt_merge_msg, RUN_SETUP },\n> \t{ \"for-each-ref\", cmd_for_each_ref, RUN_SETUP },\n> +\t{ \"for-each-repo\", cmd_for_each_repo, RUN_SETUP_GENTLY },\n> \t{ \"format-patch\", cmd_format_patch, RUN_SETUP },\n> \t{ \"fsck\", cmd_fsck, RUN_SETUP },\n> \t{ \"fsck-objects\", cmd_fsck, RUN_SETUP },\n> diff --git a/t/t0068-for-each-repo.sh b/t/t0068-for-each-repo.sh\n> new file mode 100755\n> index 0000000000..136b4ec839\n> --- /dev/null\n> +++ b/t/t0068-for-each-repo.sh\n> @@ -0,0 +1,30 @@\n> +#!/bin/sh\n> +\n> +test_description='git for-each-repo builtin'\n> +\n> +. ./test-lib.sh\n> +\n> +test_expect_success 'run based on configured value' '\n> +\tgit init one &&\n> +\tgit init two &&\n> +\tgit init three &&\n> +\tgit -C two commit --allow-empty -m \"DID NOT RUN\" &&\n> +\tgit config run.key \"$TRASH_DIRECTORY/one\" &&\n> +\tgit config --add run.key \"$TRASH_DIRECTORY/three\" &&\n> +\tgit for-each-repo --config=run.key commit --allow-empty -m \"ran\" &&\n> +\tgit -C one log -1 --pretty=format:%s >message &&\n> +\tgrep ran message &&\n> +\tgit -C two log -1 --pretty=format:%s >message &&\n> +\t! grep ran message &&\n> +\tgit -C three log -1 --pretty=format:%s >message &&\n> +\tgrep ran message &&\n> +\tgit for-each-repo --config=run.key -- commit --allow-empty -m \"ran again\" &&\n> +\tgit -C one log -1 --pretty=format:%s >message &&\n> +\tgrep again message &&\n> +\tgit -C two log -1 --pretty=format:%s >message &&\n> +\t! grep again message &&\n> +\tgit -C three log -1 --pretty=format:%s >message &&\n> +\tgrep again message\n> +'\n> +\n> +test_done\n> -- \n> gitgitgadget\n> \n\n-- \nDanh\n"},{"id":"404316","messageId":"83b90c15-6d19-3d6e-cefe-4d271aaaac3a@gmail.com","threadId":"54070","inReplyTo":"20200820145127.GC31084@danh.dev","subject":"Re: [PATCH 3/7] maintenance: add --scheduled option and config","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-08-24T14:03:51Z","receivedAt":"2020-08-24T14:04:03Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 8/20/2020 10:51 AM, Đoàn Trần Công Danh wrote:\n> On 2020-08-19 17:16:44+0000, Derrick Stolee via GitGitGadget wrote:\n>> +static void fill_schedule_info(struct maintenance_task *task,\n>> +\t\t\t const char *config_name,\n>> +\t\t\t timestamp_t schedule_delay)\n>> +{\n>> +\ttimestamp_t now = approxidate(\"now\");\n> \n> I see this pattern in both previous patch and this patch,\n> should we create a helper (if not exist) to get current timestamp\n> instead, parsing \"now\" every now and then is not a good idea, in my\n> very opinionated opinion.\n\nParsing \"now\" is not that much work, and it is done only once per\nmaintenance task. To make a helper that avoids these string comparisons\n(specifically to avoid iterating through the \"special\" array in date.c)\nis unlikely to be worth the effort and code duplication.\n\nIf you mean it would be good to use a macro here, then that would be\neasy:\n\n\t#define approxidate_now() approxidate(\"now\")\n\nOne important thing for using this over time(NULL) is that we really\nwant this to work with GIT_TEST_DATE_NOW.\n\n>> +\tchar *value = NULL;\n>> +\tstruct strbuf last_run = STRBUF_INIT;\n>> +\tint64_t previous_run;\n>> +\n>> +\tstrbuf_addf(&last_run, \"maintenance.%s.lastrun\", task->name);\n>> +\n>> +\tif (git_config_get_string(last_run.buf, &value))\n>> +\t\ttask->scheduled = 1;\n>> +\telse {\n>> +\t\tprevious_run = git_config_int64(last_run.buf, value);\n>> +\t\tif (now >= previous_run + schedule_delay)\n>> +\t\t\ttask->scheduled = 1;\n>> +\t}\n>> +\n>> +\tfree(value);\n>> +\tstrbuf_release(&last_run);\n>> +}\n>> +\n>> static void initialize_task_config(void)\n>> {\n>> \tint i;\n>> @@ -1359,6 +1387,7 @@ static void initialize_task_config(void)\n>> \n>> \tfor (i = 0; i < TASK__COUNT; i++) {\n>> \t\tint config_value;\n>> +\t\tchar *config_str;\n>> \n>> \t\tstrbuf_setlen(&config_name, 0);\n>> \t\tstrbuf_addf(&config_name, \"maintenance.%s.enabled\",\n>> @@ -1366,6 +1395,20 @@ static void initialize_task_config(void)\n>> \n>> \t\tif (!git_config_get_bool(config_name.buf, &config_value))\n>> \t\t\ttasks[i].enabled = config_value;\n>> +\n>> +\t\tstrbuf_setlen(&config_name, 0);\n> \n> It looks like we have a simple and better named alias for this:\n> \n> \tstrbuf_reset(&config_name)\n> \n> _reset has 400+ occurences in this code base, compare to 20 of _setlen\n\nMakes sense. Thanks.\n\n-Stolee\n\n"},{"id":"404463","messageId":"pull.680.v2.git.1598380805.gitgitgadget@gmail.com","threadId":"54070","inReplyTo":"pull.680.git.1597857408.gitgitgadget@gmail.com","subject":"[PATCH v2 0/7] [RFC] Maintenance III: background maintenance","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-08-25T18:39:57Z","receivedAt":"2020-08-25T18:40:12Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"This is based on v3 of Part II (ds/maintenance-part-2) [1].\n\n[1] \nhttps://lore.kernel.org/git/pull.696.v3.git.1598380599.gitgitgadget@gmail.com/\n\nThis RFC is intended to show how I hope to integrate true background\nmaintenance into Git. As opposed to my original RFC [2], this entirely\nintegrates with cron (through crontab [-e|-l]) to launch maintenance\ncommands in the background.\n\n[2] \nhttps://lore.kernel.org/git/pull.597.git.1585946894.gitgitgadget@gmail.com/\n\nSome preliminary work is done to allow a new --scheduled option that\ntriggers enabled tasks only if they have not been run in some amount of\ntime. The timestamp of the previous run is stored in the \nmaintenance..lastRun config value while the interval is stored in the \nmaintenance..schedule config value.\n\nA new for-each-repo builtin runs Git commands on every repo in a given list.\nCurrently, the list is stored as a config setting, allowing a new \nmaintenance.repos config list to store the repositories registered for\nbackground maintenance. Others may want to add a --file= option for\ntheir own workflows, but I focused on making this as simple as possible for\nnow.\n\nThe updates to the git maintenance builtin include new register/unregister \nsubcommands and start/stop subcommands. The register subcommand initializes\nthe config while the start subcommand does everything register does plus \nupdate the cron table. The unregister and stop commands reverse this\nprocess.\n\nThe very last patch is entirely optional. It sets a recommended schedule\nbased on my own experience with very large repositories. I'm open to other\nsuggestions, but these are ones that I think work well and don't cause a\n\"rewrite the world\" scenario like running nightly 'gc' would do.\n\nI've been testing this scenario on my macOS laptop for a while and my Linux\nmachine. I have modified my cron task to provide logging via trace2 so I can\nsee what's happening. A future direction here would be to add some\nmaintenance logs to the repository so we can track what is happening and\ndiagnose whether the maintenance strategy is working on real repos.\n\nIt could be helpful for contributors to suggest ways to configure certain\njobs to run \"nightly\" or \"overnight on a weekend\" instead of just \"whenever\nthe last run was long enough ago.\" One way to do this would be to set the \nlastRun config to be at the time of day that the job should run. Another\noption would be to make the cron table more complicated with multiple rows,\nbut I would prefer to avoid that option if there is a simpler mechanism.\n\nNote: git maintenance (start|stop) only works on machines with cron by\ndesign. The proper thing to do on Windows will come later. Perhaps this\ncommand should be marked as unavailable on Windows somehow, or at least a\nbetter error than \"cron may not be available on your system\". I did find\nthat that message is helpful sometimes: macOS worker agents for CI builds\ntypically do not have cron available.\n\nUpdates since RFC v1\n====================\n\n * Some fallout from rewriting the option parsing in \"Maintenance I\"\n \n \n * This applies cleanly on v3 of \"Maintenance II\"\n \n \n * Several helpful feedback items from Đoàn Trần Công Danh are applied.\n \n \n * There is an unresolved comment around the use of approxidate(\"now\").\n These calls are untouched from v1.\n \n \n\nThanks, -Stolee\n\nDerrick Stolee (7):\n maintenance: optionally skip --auto process\n maintenance: store the \"last run\" time in config\n maintenance: add --scheduled option and config\n for-each-repo: run subcommands on configured repos\n maintenance: add [un]register subcommands\n maintenance: add start/stop subcommands\n maintenance: recommended schedule in register/start\n\n .gitignore | 1 +\n Documentation/config/maintenance.txt | 19 ++\n Documentation/git-for-each-repo.txt | 59 ++++++\n Documentation/git-maintenance.txt | 44 ++++-\n Makefile | 2 +\n builtin.h | 1 +\n builtin/for-each-repo.c | 58 ++++++\n builtin/gc.c | 286 ++++++++++++++++++++++++++-\n command-list.txt | 1 +\n git.c | 1 +\n run-command.c | 6 +\n t/helper/test-crontab.c | 35 ++++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/t0068-for-each-repo.sh | 30 +++\n t/t7900-maintenance.sh | 95 ++++++++-\n t/test-lib.sh | 6 +\n 17 files changed, 640 insertions(+), 6 deletions(-)\n create mode 100644 Documentation/git-for-each-repo.txt\n create mode 100644 builtin/for-each-repo.c\n create mode 100644 t/helper/test-crontab.c\n create mode 100755 t/t0068-for-each-repo.sh\n\n\nbase-commit: e9bb32f53ade2067f773bfe6e5c13ed1a5d694a6\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-680%2Fderrickstolee%2Fmaintenance%2Fscheduled-v2\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-680/derrickstolee/maintenance/scheduled-v2\nPull-Request: https://github.com/gitgitgadget/git/pull/680\n\nRange-diff vs v1:\n\n 1: 90de25d128 ! 1: 5fdd8188b1 maintenance: optionally skip --auto process\n @@ run-command.c: int run_processes_parallel_tr2(int n, get_next_task_fn get_next_t\n \tstruct child_process maint = CHILD_PROCESS_INIT;\n \n +\tif (!git_config_get_bool(\"maintenance.auto\", &enabled) &&\n -+\t !enabled) {\n -+\t\t fprintf(stderr, \"enabled: %d\\n\", enabled);\n ++\t !enabled)\n +\t\treturn 0;\n -+\t }\n +\n \tmaint.git_cmd = 1;\n \tstrvec_pushl(&maint.args, \"maintenance\", \"run\", \"--auto\", NULL);\n 2: bdc27fa28e ! 2: e3ef0b9bea maintenance: store the \"last run\" time in config\n @@ builtin/gc.c: static int compare_tasks_by_selection(const void *a_, const void *\n +\tstrbuf_release(&value);\n +}\n +\n - static int maintenance_run(struct maintenance_opts *opts)\n + static int maintenance_run_tasks(struct maintenance_run_opts *opts)\n {\n \tint i, found_selected = 0;\n -@@ builtin/gc.c: static int maintenance_run(struct maintenance_opts *opts)\n +@@ builtin/gc.c: static int maintenance_run_tasks(struct maintenance_run_opts *opts)\n \t\t !tasks[i].auto_condition()))\n \t\t\tcontinue;\n \n @@ builtin/gc.c: static int maintenance_run(struct maintenance_opts *opts)\n \t\tif (tasks[i].fn(opts)) {\n \t\t\terror(_(\"task '%s' failed\"), tasks[i].name);\n \n - ## git-gvfs-helper (new) ##\n - Binary files /dev/null and git-gvfs-helper differ\n -\n - ## t/helper/test-gvfs-protocol (new) ##\n - Binary files /dev/null and t/helper/test-gvfs-protocol differ\n -\n ## t/t7900-maintenance.sh ##\n @@ t/t7900-maintenance.sh: test_expect_success 'maintenance.incremental-repack.auto' '\n \tdone\n 3: 4473c93b11 ! 3: c728c57d85 maintenance: add --scheduled option and config\n @@ Documentation/git-maintenance.txt: OPTIONS\n \tDo not report progress or other information over `stderr`.\n \n ## builtin/gc.c ##\n -@@ builtin/gc.c: static const char * const builtin_maintenance_usage[] = {\n +@@ builtin/gc.c: int cmd_gc(int argc, const char **argv, const char *prefix)\n + }\n + \n + static const char * const builtin_maintenance_run_usage[] = {\n +-\tN_(\"git maintenance run [--auto] [--[no-]quiet] [--task=]\"),\n ++\tN_(\"git maintenance run [--auto] [--[no-]quiet] [--task=] [--scheduled]\"),\n + \tNULL\n + };\n \n - struct maintenance_opts {\n + struct maintenance_run_opts {\n \tint auto_flag;\n +\tint scheduled;\n \tint quiet;\n @@ builtin/gc.c: struct maintenance_task {\n \n \t/* -1 if not selected. */\n \tint selected_order;\n -@@ builtin/gc.c: static int maintenance_run(struct maintenance_opts *opts)\n +@@ builtin/gc.c: static int maintenance_run_tasks(struct maintenance_run_opts *opts)\n \t\t !tasks[i].auto_condition()))\n \t\t\tcontinue;\n \n @@ builtin/gc.c: static int maintenance_run(struct maintenance_opts *opts)\n \t\tupdate_last_run(&tasks[i]);\n \n \t\ttrace2_region_enter(\"maintenance\", tasks[i].name, r);\n -@@ builtin/gc.c: static int maintenance_run(struct maintenance_opts *opts)\n +@@ builtin/gc.c: static int maintenance_run_tasks(struct maintenance_run_opts *opts)\n \treturn result;\n }\n \n @@ builtin/gc.c: static void initialize_task_config(void)\n \t\tint config_value;\n +\t\tchar *config_str;\n \n - \t\tstrbuf_setlen(&config_name, 0);\n +-\t\tstrbuf_setlen(&config_name, 0);\n ++\t\tstrbuf_reset(&config_name);\n \t\tstrbuf_addf(&config_name, \"maintenance.%s.enabled\",\n -@@ builtin/gc.c: static void initialize_task_config(void)\n + \t\t\t tasks[i].name);\n \n \t\tif (!git_config_get_bool(config_name.buf, &config_value))\n \t\t\ttasks[i].enabled = config_value;\n +\n -+\t\tstrbuf_setlen(&config_name, 0);\n ++\t\tstrbuf_reset(&config_name);\n +\t\tstrbuf_addf(&config_name, \"maintenance.%s.schedule\",\n +\t\t\t tasks[i].name);\n +\n @@ builtin/gc.c: static void initialize_task_config(void)\n \t}\n \n \tstrbuf_release(&config_name);\n -@@ builtin/gc.c: int cmd_maintenance(int argc, const char **argv, const char *prefix)\n - \tstruct option builtin_maintenance_options[] = {\n +@@ builtin/gc.c: static int maintenance_run(int argc, const char **argv, const char *prefix)\n + \tstruct option builtin_maintenance_run_options[] = {\n \t\tOPT_BOOL(0, \"auto\", &opts.auto_flag,\n \t\t\t N_(\"run tasks based on the state of the repository\")),\n +\t\tOPT_BOOL(0, \"scheduled\", &opts.scheduled,\n @@ builtin/gc.c: int cmd_maintenance(int argc, const char **argv, const char *prefi\n \t\tOPT_BOOL(0, \"quiet\", &opts.quiet,\n \t\t\t N_(\"do not report progress or other information over stderr\")),\n \t\tOPT_CALLBACK_F(0, \"task\", NULL, N_(\"task\"),\n -@@ builtin/gc.c: int cmd_maintenance(int argc, const char **argv, const char *prefix)\n - \t\t\t builtin_maintenance_usage,\n - \t\t\t PARSE_OPT_KEEP_UNKNOWN);\n +@@ builtin/gc.c: static int maintenance_run(int argc, const char **argv, const char *prefix)\n + \t\t\t builtin_maintenance_run_usage,\n + \t\t\t PARSE_OPT_STOP_AT_NON_OPTION);\n \n +\tif (opts.auto_flag + opts.scheduled > 1)\n +\t\tdie(_(\"use at most one of the --auto and --scheduled options\"));\n +\n - \tif (argc != 1)\n - \t\tusage_with_options(builtin_maintenance_usage,\n - \t\t\t\t builtin_maintenance_options);\n + \tif (argc != 0)\n + \t\tusage_with_options(builtin_maintenance_run_usage,\n + \t\t\t\t builtin_maintenance_run_options);\n \n ## t/t7900-maintenance.sh ##\n @@ t/t7900-maintenance.sh: test_expect_success 'maintenance.incremental-repack.auto' '\n 4: ccb667dc6f ! 4: 0314258c5c for-each-repo: run subcommands on configured repos\n @@ Documentation/git-for-each-repo.txt (new)\n +\n +DESCRIPTION\n +-----------\n -+Run a Git commands on a list of repositories. The arguments after the\n ++Run a Git command on a list of repositories. The arguments after the\n +known options or `--` indicator are used as the arguments for the Git\n -+command.\n ++subprocess.\n ++\n ++THIS COMMAND IS EXPERIMENTAL. THE BEHAVIOR MAY CHANGE.\n +\n +For example, we could run maintenance on each of a list of repositories\n +stored in a `maintenance.repo` config variable using\n @@ Documentation/git-for-each-repo.txt (new)\n +as available. If `git for-each-repo` is run in a directory that is not a\n +Git repository, then only the system and global config is used.\n +\n ++\n ++SUBPROCESS BEHAVIOR\n ++-------------------\n ++\n ++If any `git -C ` subprocess returns a non-zero exit code,\n ++then the `git for-each-repo` process returns that exit code without running\n ++more subprocesses.\n ++\n ++Each `git -C ` subprocess inherits the standard file\n ++descriptors `stdin`, `stdout`, and `stderr`.\n ++\n ++\n +GIT\n +---\n +Part of the linkgit:git[1] suite\n @@ builtin/for-each-repo.c (new)\n +\treturn result;\n +}\n \n + ## command-list.txt ##\n +@@ command-list.txt: git-fetch-pack synchingrepositories\n + git-filter-branch ancillarymanipulators\n + git-fmt-merge-msg purehelpers\n + git-for-each-ref plumbinginterrogators\n ++git-for-each-repo plumbinginterrogators\n + git-format-patch mainporcelain\n + git-fsck ancillaryinterrogators complete\n + git-gc mainporcelain\n +\n ## git.c ##\n @@ git.c: static struct cmd_struct commands[] = {\n \t{ \"fetch-pack\", cmd_fetch_pack, RUN_SETUP | NO_PARSEOPT },\n 5: f44c6a0f20 ! 5: c0ce1267a9 maintenance: add [un]register subcommands\n @@ Documentation/git-maintenance.txt: run::\n \n \n ## builtin/gc.c ##\n -@@ builtin/gc.c: int cmd_gc(int argc, const char **argv, const char *prefix)\n - }\n - \n - static const char * const builtin_maintenance_usage[] = {\n --\tN_(\"git maintenance run []\"),\n -+\tN_(\"git maintenance []\"),\n - \tNULL\n - };\n - \n -@@ builtin/gc.c: static int task_option_parse(const struct option *opt,\n - \treturn 0;\n +@@ builtin/gc.c: static int maintenance_run(int argc, const char **argv, const char *prefix)\n + \treturn maintenance_run_tasks(&opts);\n }\n \n +-static const char builtin_maintenance_usage[] = N_(\"git maintenance run []\");\n +static int maintenance_register(void)\n +{\n +\tstruct child_process config_set = CHILD_PROCESS_INIT;\n @@ builtin/gc.c: static int task_option_parse(const struct option *opt,\n +\treturn run_command(&config_unset);\n +}\n +\n ++static const char builtin_maintenance_usage[] =\tN_(\"git maintenance []\");\n + \n int cmd_maintenance(int argc, const char **argv, const char *prefix)\n {\n - \tint i;\n @@ builtin/gc.c: int cmd_maintenance(int argc, const char **argv, const char *prefix)\n - \t\tusage_with_options(builtin_maintenance_usage,\n - \t\t\t\t builtin_maintenance_options);\n \n -+\tif (!strcmp(argv[0], \"register\"))\n + \tif (!strcmp(argv[1], \"run\"))\n + \t\treturn maintenance_run(argc - 1, argv + 1, prefix);\n ++\tif (!strcmp(argv[1], \"register\"))\n +\t\treturn maintenance_register();\n - \tif (!strcmp(argv[0], \"run\"))\n - \t\treturn maintenance_run(&opts);\n -+\tif (!strcmp(argv[0], \"unregister\"))\n ++\tif (!strcmp(argv[1], \"unregister\"))\n +\t\treturn maintenance_unregister();\n \n - \tdie(_(\"invalid subcommand: %s\"), argv[0]);\n + \tdie(_(\"invalid subcommand: %s\"), argv[1]);\n }\n \n ## t/t7900-maintenance.sh ##\n 6: 2442071fd0 ! 6: 8a7c34035a maintenance: add start/stop subcommands\n @@ builtin/gc.c: static int maintenance_unregister(void)\n +\treturn update_background_schedule(0);\n +}\n +\n + static const char builtin_maintenance_usage[] =\tN_(\"git maintenance []\");\n + \n int cmd_maintenance(int argc, const char **argv, const char *prefix)\n - {\n - \tint i;\n @@ builtin/gc.c: int cmd_maintenance(int argc, const char **argv, const char *prefix)\n - \t\treturn maintenance_register();\n - \tif (!strcmp(argv[0], \"run\"))\n - \t\treturn maintenance_run(&opts);\n -+\tif (!strcmp(argv[0], \"start\"))\n + \n + \tif (!strcmp(argv[1], \"run\"))\n + \t\treturn maintenance_run(argc - 1, argv + 1, prefix);\n ++\tif (!strcmp(argv[1], \"start\"))\n +\t\treturn maintenance_start();\n -+\tif (!strcmp(argv[0], \"stop\"))\n ++\tif (!strcmp(argv[1], \"stop\"))\n +\t\treturn maintenance_stop();\n - \tif (!strcmp(argv[0], \"unregister\"))\n - \t\treturn maintenance_unregister();\n - \n + \tif (!strcmp(argv[1], \"register\"))\n + \t\treturn maintenance_register();\n + \tif (!strcmp(argv[1], \"unregister\"))\n \n ## t/helper/test-crontab.c (new) ##\n @@\n 7: 40b1a0546c ! 7: 9ecabeb055 maintenance: recommended schedule in register/start\n @@ Documentation/git-maintenance.txt: register::\n \tRun one or more maintenance tasks. If one or more `--task` options\n \n ## builtin/gc.c ##\n -@@ builtin/gc.c: static int task_option_parse(const struct option *opt,\n - \treturn 0;\n +@@ builtin/gc.c: static int maintenance_run(int argc, const char **argv, const char *prefix)\n + \treturn maintenance_run_tasks(&opts);\n }\n \n +static int has_schedule_config(void)\n\n-- \ngitgitgadget\n"},{"id":"404462","messageId":"5fdd8188b1d9b6efc2803b557b3ba344e184d22e.1598380805.git.gitgitgadget@gmail.com","threadId":"54070","inReplyTo":"pull.680.v2.git.1598380805.gitgitgadget@gmail.com","subject":"[PATCH v2 1/7] maintenance: optionally skip --auto process","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-08-25T18:39:58Z","receivedAt":"2020-08-25T18:40:13Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nSome commands run 'git maintenance run --auto --[no-]quiet' after doing\ntheir normal work, as a way to keep repositories clean as they are used.\nCurrently, users who do not want this maintenance to occur would set the\n'gc.auto' config option to 0 to avoid the 'gc' task from running.\nHowever, this does not stop the extra process invocation. On Windows,\nthis extra process invocation can be more expensive than necessary.\n\nAllow users to drop this extra process by setting 'maintenance.auto' to\n'false'.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/config/maintenance.txt | 5 +++++\n run-command.c | 6 ++++++\n t/t7900-maintenance.sh | 13 +++++++++++++\n 3 files changed, 24 insertions(+)\n\ndiff --git a/Documentation/config/maintenance.txt b/Documentation/config/maintenance.txt\nindex a0706d8f09..06db758172 100644\n--- a/Documentation/config/maintenance.txt\n+++ b/Documentation/config/maintenance.txt\n@@ -1,3 +1,8 @@\n+maintenance.auto::\n+\tThis boolean config option controls whether some commands run\n+\t`git maintenance run --auto` after doing their normal work. Defaults\n+\tto true.\n+\n maintenance..enabled::\n \tThis boolean config option controls whether the maintenance task\n \twith name `` is run when no `--task` option is specified to\ndiff --git a/run-command.c b/run-command.c\nindex 2ee59acdc8..ea4d0fb4b1 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -7,6 +7,7 @@\n #include \"strbuf.h\"\n #include \"string-list.h\"\n #include \"quote.h\"\n+#include \"config.h\"\n \n void child_process_init(struct child_process *child)\n {\n@@ -1868,8 +1869,13 @@ int run_processes_parallel_tr2(int n, get_next_task_fn get_next_task,\n \n int run_auto_maintenance(int quiet)\n {\n+\tint enabled;\n \tstruct child_process maint = CHILD_PROCESS_INIT;\n \n+\tif (!git_config_get_bool(\"maintenance.auto\", &enabled) &&\n+\t !enabled)\n+\t\treturn 0;\n+\n \tmaint.git_cmd = 1;\n \tstrvec_pushl(&maint.args, \"maintenance\", \"run\", \"--auto\", NULL);\n \tstrvec_push(&maint.args, quiet ? \"--quiet\" : \"--no-quiet\");\ndiff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\nindex 6f878b0141..e0ba19e1ff 100755\n--- a/t/t7900-maintenance.sh\n+++ b/t/t7900-maintenance.sh\n@@ -26,6 +26,19 @@ test_expect_success 'run [--auto|--quiet]' '\n \ttest_subcommand git gc --no-quiet .enabled' '\n \tgit config maintenance.gc.enabled false &&\n \tgit config maintenance.commit-graph.enabled true &&\n-- \ngitgitgadget\n\n"},{"id":"404464","messageId":"e3ef0b9bea4a31c72ce88841639e88355408f0c1.1598380805.git.gitgitgadget@gmail.com","threadId":"54070","inReplyTo":"pull.680.v2.git.1598380805.gitgitgadget@gmail.com","subject":"[PATCH v2 2/7] maintenance: store the \"last run\" time in config","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-08-25T18:39:59Z","receivedAt":"2020-08-25T18:40:18Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nUsers may want to run certain maintenance tasks only so often. Update\nthe local config with a new 'maintenance..lastRun' config option\nthat stores the timestamp just before running the maintenance task.\n\nI selected the timestamp before the task, as opposed to after the task,\nfor a couple reasons:\n\n 1. The time the task takes to execute should not contribute to the\n interval between running the tasks. If a daily task takes 10 minutes\n to run, then every day the execution will drift by at least 10\n minutes.\n\n 2. If the task fails for some unforseen reason, it would be good to\n indicate that we _attempted_ the task at a certain timestamp. This\n will avoid spamming a repository that is in a bad state.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/config/maintenance.txt | 5 +++++\n builtin/gc.c | 16 ++++++++++++++++\n t/t7900-maintenance.sh | 10 ++++++++++\n 3 files changed, 31 insertions(+)\n\ndiff --git a/Documentation/config/maintenance.txt b/Documentation/config/maintenance.txt\nindex 06db758172..8dd34169da 100644\n--- a/Documentation/config/maintenance.txt\n+++ b/Documentation/config/maintenance.txt\n@@ -10,6 +10,11 @@ maintenance..enabled::\n \t`--task` option exists. By default, only `maintenance.gc.enabled`\n \tis true.\n \n+maintenance..lastRun::\n+\tThis config value is automatically updated by Git when the task\n+\t`` is run. It stores a timestamp representing the most-recent\n+\trun of the ``.\n+\n maintenance.commit-graph.auto::\n \tThis integer config option controls how often the `commit-graph` task\n \tshould be run as part of `git maintenance run --auto`. If zero, then\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex f8459df04c..fb6f231a5c 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -1212,6 +1212,20 @@ static int compare_tasks_by_selection(const void *a_, const void *b_)\n \treturn b->selected_order - a->selected_order;\n }\n \n+static void update_last_run(struct maintenance_task *task)\n+{\n+\ttimestamp_t now = approxidate(\"now\");\n+\tstruct strbuf config = STRBUF_INIT;\n+\tstruct strbuf value = STRBUF_INIT;\n+\tstrbuf_addf(&config, \"maintenance.%s.lastrun\", task->name);\n+\tstrbuf_addf(&value, \"%\"PRItime\"\", now);\n+\n+\tgit_config_set(config.buf, value.buf);\n+\n+\tstrbuf_release(&config);\n+\tstrbuf_release(&value);\n+}\n+\n static int maintenance_run_tasks(struct maintenance_run_opts *opts)\n {\n \tint i, found_selected = 0;\n@@ -1254,6 +1268,8 @@ static int maintenance_run_tasks(struct maintenance_run_opts *opts)\n \t\t !tasks[i].auto_condition()))\n \t\t\tcontinue;\n \n+\t\tupdate_last_run(&tasks[i]);\n+\n \t\ttrace2_region_enter(\"maintenance\", tasks[i].name, r);\n \t\tif (tasks[i].fn(opts)) {\n \t\t\terror(_(\"task '%s' failed\"), tasks[i].name);\ndiff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\nindex e0ba19e1ff..a985ce3674 100755\n--- a/t/t7900-maintenance.sh\n+++ b/t/t7900-maintenance.sh\n@@ -264,4 +264,14 @@ test_expect_success 'maintenance.incremental-repack.auto' '\n \tdone\n '\n \n+test_expect_success 'tasks update maintenance..lastRun' '\n+\tgit config --unset maintenance.commit-graph.lastrun &&\n+\tGIT_TRACE2_EVENT=\"$(pwd)/run.txt\" \\\n+\t\tGIT_TEST_DATE_NOW=1595000000 \\\n+\t\tgit maintenance run --task=commit-graph 2>/dev/null &&\n+\ttest_subcommand git commit-graph write --split --reachable \\\n+\t\t--no-progress \n\nA user may want to run certain maintenance tasks based on frequency, not\nconditions given in the repository. For example, the user may want to\nperform a 'prefetch' task every hour, or 'gc' task every day. To assist,\nupdate the 'git maintenance run --scheduled' command to check the config\nfor the last run of that task and add a number of seconds. The task\nwould then run only if the current time is beyond that minimum\ntimestamp.\n\nAdd a '--scheduled' option to 'git maintenance run' to only run tasks\nthat have had enough time pass since their last run. This is done for\neach enabled task by checking if the current timestamp is at least as\nlarge as the sum of 'maintenance..lastRun' and\n'maintenance..schedule' in the Git config. This second value is\nnew to this commit, storing a number of seconds intended between runs.\n\nA user could then set up an hourly maintenance run with the following\ncron table:\n\n 0 * * * * git -C maintenance run --scheduled\n\nThen, the user could configure the repository with the following config\nvalues:\n\n maintenance.prefetch.schedule 3000\n maintenance.gc.schedule 86000\n\nThese numbers are slightly lower than one hour and one day (in seconds).\nThe cron schedule will enforce the hourly run rate, but we can use these\nschedules to ensure the 'gc' task runs once a day. The error is given\nbecause the *.lastRun config option is specified at the _start_ of the\ntask run. Otherwise, a slow task run could shift the \"daily\" job of 'gc'\nfrom a 10:00pm run to 11:00pm run, or later.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/config/maintenance.txt | 9 +++++\n Documentation/git-maintenance.txt | 13 ++++++-\n builtin/gc.c | 54 ++++++++++++++++++++++++++--\n t/t7900-maintenance.sh | 20 +++++++++++\n 4 files changed, 92 insertions(+), 4 deletions(-)\n\ndiff --git a/Documentation/config/maintenance.txt b/Documentation/config/maintenance.txt\nindex 8dd34169da..caacacd322 100644\n--- a/Documentation/config/maintenance.txt\n+++ b/Documentation/config/maintenance.txt\n@@ -15,6 +15,15 @@ maintenance..lastRun::\n \t`` is run. It stores a timestamp representing the most-recent\n \trun of the ``.\n \n+maintenance..schedule::\n+\tThis config option controls whether or not the given `` runs\n+\tduring a `git maintenance run --scheduled` command. If the option\n+\tis an integer value `S`, then the `` is run when the current\n+\ttime is `S` seconds after the timestamp stored in\n+\t`maintenance..lastRun`. If the option has no value or a\n+\tnon-integer value, then the task will never run with the `--scheduled`\n+\toption.\n+\n maintenance.commit-graph.auto::\n \tThis integer config option controls how often the `commit-graph` task\n \tshould be run as part of `git maintenance run --auto`. If zero, then\ndiff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt\nindex b44efb05a3..2bc02c65e4 100644\n--- a/Documentation/git-maintenance.txt\n+++ b/Documentation/git-maintenance.txt\n@@ -107,7 +107,18 @@ OPTIONS\n \tonly if certain thresholds are met. For example, the `gc` task\n \truns when the number of loose objects exceeds the number stored\n \tin the `gc.auto` config setting, or when the number of pack-files\n-\texceeds the `gc.autoPackLimit` config setting.\n+\texceeds the `gc.autoPackLimit` config setting. Not compatible with\n+\tthe `--scheduled` option.\n+\n+--scheduled::\n+\tWhen combined with the `run` subcommand, run maintenance tasks\n+\tonly if certain time conditions are met, as specified by the\n+\t`maintenance..schedule` config value for each ``.\n+\tThis config value specifies a number of seconds since the last\n+\ttime that task ran, according to the `maintenance..lastRun`\n+\tconfig value. The tasks that are tested are those provided by\n+\tthe `--task=` option(s) or those with\n+\t`maintenance..enabled` set to true.\n \n --quiet::\n \tDo not report progress or other information over `stderr`.\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex fb6f231a5c..5726a9a3b3 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -705,12 +705,13 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n }\n \n static const char * const builtin_maintenance_run_usage[] = {\n-\tN_(\"git maintenance run [--auto] [--[no-]quiet] [--task=]\"),\n+\tN_(\"git maintenance run [--auto] [--[no-]quiet] [--task=] [--scheduled]\"),\n \tNULL\n };\n \n struct maintenance_run_opts {\n \tint auto_flag;\n+\tint scheduled;\n \tint quiet;\n };\n \n@@ -1157,7 +1158,8 @@ struct maintenance_task {\n \tconst char *name;\n \tmaintenance_task_fn *fn;\n \tmaintenance_auto_fn *auto_condition;\n-\tunsigned enabled:1;\n+\tunsigned enabled:1,\n+\t\t scheduled:1;\n \n \t/* -1 if not selected. */\n \tint selected_order;\n@@ -1268,6 +1270,9 @@ static int maintenance_run_tasks(struct maintenance_run_opts *opts)\n \t\t !tasks[i].auto_condition()))\n \t\t\tcontinue;\n \n+\t\tif (opts->scheduled && !tasks[i].scheduled)\n+\t\t\tcontinue;\n+\n \t\tupdate_last_run(&tasks[i]);\n \n \t\ttrace2_region_enter(\"maintenance\", tasks[i].name, r);\n@@ -1282,6 +1287,29 @@ static int maintenance_run_tasks(struct maintenance_run_opts *opts)\n \treturn result;\n }\n \n+static void fill_schedule_info(struct maintenance_task *task,\n+\t\t\t const char *config_name,\n+\t\t\t timestamp_t schedule_delay)\n+{\n+\ttimestamp_t now = approxidate(\"now\");\n+\tchar *value = NULL;\n+\tstruct strbuf last_run = STRBUF_INIT;\n+\tint64_t previous_run;\n+\n+\tstrbuf_addf(&last_run, \"maintenance.%s.lastrun\", task->name);\n+\n+\tif (git_config_get_string(last_run.buf, &value))\n+\t\ttask->scheduled = 1;\n+\telse {\n+\t\tprevious_run = git_config_int64(last_run.buf, value);\n+\t\tif (now >= previous_run + schedule_delay)\n+\t\t\ttask->scheduled = 1;\n+\t}\n+\n+\tfree(value);\n+\tstrbuf_release(&last_run);\n+}\n+\n static void initialize_task_config(void)\n {\n \tint i;\n@@ -1290,13 +1318,28 @@ static void initialize_task_config(void)\n \n \tfor (i = 0; i < TASK__COUNT; i++) {\n \t\tint config_value;\n+\t\tchar *config_str;\n \n-\t\tstrbuf_setlen(&config_name, 0);\n+\t\tstrbuf_reset(&config_name);\n \t\tstrbuf_addf(&config_name, \"maintenance.%s.enabled\",\n \t\t\t tasks[i].name);\n \n \t\tif (!git_config_get_bool(config_name.buf, &config_value))\n \t\t\ttasks[i].enabled = config_value;\n+\n+\t\tstrbuf_reset(&config_name);\n+\t\tstrbuf_addf(&config_name, \"maintenance.%s.schedule\",\n+\t\t\t tasks[i].name);\n+\n+\t\tif (!git_config_get_string(config_name.buf, &config_str)) {\n+\t\t\ttimestamp_t schedule_delay = git_config_int64(\n+\t\t\t\t\t\t\tconfig_name.buf,\n+\t\t\t\t\t\t\tconfig_str);\n+\t\t\tfill_schedule_info(&tasks[i],\n+\t\t\t\t\t\tconfig_name.buf,\n+\t\t\t\t\t\tschedule_delay);\n+\t\t\tfree(config_str);\n+\t\t}\n \t}\n \n \tstrbuf_release(&config_name);\n@@ -1340,6 +1383,8 @@ static int maintenance_run(int argc, const char **argv, const char *prefix)\n \tstruct option builtin_maintenance_run_options[] = {\n \t\tOPT_BOOL(0, \"auto\", &opts.auto_flag,\n \t\t\t N_(\"run tasks based on the state of the repository\")),\n+\t\tOPT_BOOL(0, \"scheduled\", &opts.scheduled,\n+\t\t\t N_(\"run tasks based on time intervals\")),\n \t\tOPT_BOOL(0, \"quiet\", &opts.quiet,\n \t\t\t N_(\"do not report progress or other information over stderr\")),\n \t\tOPT_CALLBACK_F(0, \"task\", NULL, N_(\"task\"),\n@@ -1360,6 +1405,9 @@ static int maintenance_run(int argc, const char **argv, const char *prefix)\n \t\t\t builtin_maintenance_run_usage,\n \t\t\t PARSE_OPT_STOP_AT_NON_OPTION);\n \n+\tif (opts.auto_flag + opts.scheduled > 1)\n+\t\tdie(_(\"use at most one of the --auto and --scheduled options\"));\n+\n \tif (argc != 0)\n \t\tusage_with_options(builtin_maintenance_run_usage,\n \t\t\t\t builtin_maintenance_run_options);\ndiff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\nindex a985ce3674..3e0c5f1ca8 100755\n--- a/t/t7900-maintenance.sh\n+++ b/t/t7900-maintenance.sh\n@@ -264,6 +264,11 @@ test_expect_success 'maintenance.incremental-repack.auto' '\n \tdone\n '\n \n+test_expect_success '--auto and --scheduled incompatible' '\n+\ttest_must_fail git maintenance run --auto --scheduled 2>err &&\n+\ttest_i18ngrep \"at most one\" err\n+'\n+\n test_expect_success 'tasks update maintenance..lastRun' '\n \tgit config --unset maintenance.commit-graph.lastrun &&\n \tGIT_TRACE2_EVENT=\"$(pwd)/run.txt\" \\\n@@ -274,4 +279,19 @@ test_expect_success 'tasks update maintenance..lastRun' '\n \ttest_cmp_config 1595000000 maintenance.commit-graph.lastrun\n '\n \n+test_expect_success '--scheduled with specific time' '\n+\tgit config maintenance.commit-graph.schedule 100 &&\n+\tGIT_TRACE2_EVENT=\"$(pwd)/too-soon.txt\" \\\n+\t\tGIT_TEST_DATE_NOW=1595000099 \\\n+\t\tgit maintenance run --scheduled 2>/dev/null &&\n+\ttest_subcommand ! git commit-graph write --split --reachable \\\n+\t\t--no-progress /dev/null &&\n+\ttest_subcommand git commit-graph write --split --reachable \\\n+\t\t--no-progress \n\nIt can be helpful to store a list of repositories in global or system\nconfig and then iterate Git commands on that list. Create a new builtin\nthat makes this process simple for experts. We will use this builtin to\nrun scheduled maintenance on all configured repositories in a future\nchange.\n\nThe test is very simple, but does highlight that the \"--\" argument is\noptional.\n\nSigned-off-by: Derrick Stolee \n---\n .gitignore | 1 +\n Documentation/git-for-each-repo.txt | 59 +++++++++++++++++++++++++++++\n Makefile | 1 +\n builtin.h | 1 +\n builtin/for-each-repo.c | 58 ++++++++++++++++++++++++++++\n command-list.txt | 1 +\n git.c | 1 +\n t/t0068-for-each-repo.sh | 30 +++++++++++++++\n 8 files changed, 152 insertions(+)\n create mode 100644 Documentation/git-for-each-repo.txt\n create mode 100644 builtin/for-each-repo.c\n create mode 100755 t/t0068-for-each-repo.sh\n\ndiff --git a/.gitignore b/.gitignore\nindex a5808fa30d..5eb2a2be71 100644\n--- a/.gitignore\n+++ b/.gitignore\n@@ -67,6 +67,7 @@\n /git-filter-branch\n /git-fmt-merge-msg\n /git-for-each-ref\n+/git-for-each-repo\n /git-format-patch\n /git-fsck\n /git-fsck-objects\ndiff --git a/Documentation/git-for-each-repo.txt b/Documentation/git-for-each-repo.txt\nnew file mode 100644\nindex 0000000000..94bd19da26\n--- /dev/null\n+++ b/Documentation/git-for-each-repo.txt\n@@ -0,0 +1,59 @@\n+git-for-each-repo(1)\n+====================\n+\n+NAME\n+----\n+git-for-each-repo - Run a Git command on a list of repositories\n+\n+\n+SYNOPSIS\n+--------\n+[verse]\n+'git for-each-repo' --config= [--] \n+\n+\n+DESCRIPTION\n+-----------\n+Run a Git command on a list of repositories. The arguments after the\n+known options or `--` indicator are used as the arguments for the Git\n+subprocess.\n+\n+THIS COMMAND IS EXPERIMENTAL. THE BEHAVIOR MAY CHANGE.\n+\n+For example, we could run maintenance on each of a list of repositories\n+stored in a `maintenance.repo` config variable using\n+\n+-------------\n+git for-each-repo --config=maintenance.repo maintenance run\n+-------------\n+\n+This will run `git -C maintenance run` for each value ``\n+in the multi-valued config variable `maintenance.repo`.\n+\n+\n+OPTIONS\n+-------\n+--config=::\n+\tUse the given config variable as a multi-valued list storing\n+\tabsolute path names. Iterate on that list of paths to run\n+\tthe given arguments.\n++\n+These config values are loaded from system, global, and local Git config,\n+as available. If `git for-each-repo` is run in a directory that is not a\n+Git repository, then only the system and global config is used.\n+\n+\n+SUBPROCESS BEHAVIOR\n+-------------------\n+\n+If any `git -C ` subprocess returns a non-zero exit code,\n+then the `git for-each-repo` process returns that exit code without running\n+more subprocesses.\n+\n+Each `git -C ` subprocess inherits the standard file\n+descriptors `stdin`, `stdout`, and `stderr`.\n+\n+\n+GIT\n+---\n+Part of the linkgit:git[1] suite\ndiff --git a/Makefile b/Makefile\nindex 65f8cfb236..7c588ff036 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1071,6 +1071,7 @@ BUILTIN_OBJS += builtin/fetch-pack.o\n BUILTIN_OBJS += builtin/fetch.o\n BUILTIN_OBJS += builtin/fmt-merge-msg.o\n BUILTIN_OBJS += builtin/for-each-ref.o\n+BUILTIN_OBJS += builtin/for-each-repo.o\n BUILTIN_OBJS += builtin/fsck.o\n BUILTIN_OBJS += builtin/gc.o\n BUILTIN_OBJS += builtin/get-tar-commit-id.o\ndiff --git a/builtin.h b/builtin.h\nindex 17c1c0ce49..ff7c6e5aa9 100644\n--- a/builtin.h\n+++ b/builtin.h\n@@ -150,6 +150,7 @@ int cmd_fetch(int argc, const char **argv, const char *prefix);\n int cmd_fetch_pack(int argc, const char **argv, const char *prefix);\n int cmd_fmt_merge_msg(int argc, const char **argv, const char *prefix);\n int cmd_for_each_ref(int argc, const char **argv, const char *prefix);\n+int cmd_for_each_repo(int argc, const char **argv, const char *prefix);\n int cmd_format_patch(int argc, const char **argv, const char *prefix);\n int cmd_fsck(int argc, const char **argv, const char *prefix);\n int cmd_gc(int argc, const char **argv, const char *prefix);\ndiff --git a/builtin/for-each-repo.c b/builtin/for-each-repo.c\nnew file mode 100644\nindex 0000000000..5bba623ff1\n--- /dev/null\n+++ b/builtin/for-each-repo.c\n@@ -0,0 +1,58 @@\n+#include \"cache.h\"\n+#include \"config.h\"\n+#include \"builtin.h\"\n+#include \"parse-options.h\"\n+#include \"run-command.h\"\n+#include \"string-list.h\"\n+\n+static const char * const for_each_repo_usage[] = {\n+\tN_(\"git for-each-repo --config= \"),\n+\tNULL\n+};\n+\n+static int run_command_on_repo(const char *path,\n+\t\t\t void *cbdata)\n+{\n+\tint i;\n+\tstruct child_process child = CHILD_PROCESS_INIT;\n+\tstruct strvec *args = (struct strvec *)cbdata;\n+\n+\tchild.git_cmd = 1;\n+\tstrvec_pushl(&child.args, \"-C\", path, NULL);\n+\n+\tfor (i = 0; i < args->nr; i++)\n+\t\tstrvec_push(&child.args, args->v[i]);\n+\n+\treturn run_command(&child);\n+}\n+\n+int cmd_for_each_repo(int argc, const char **argv, const char *prefix)\n+{\n+\tstatic const char *config_key = NULL;\n+\tint i, result = 0;\n+\tconst struct string_list *values;\n+\tstruct strvec args = STRVEC_INIT;\n+\n+\tconst struct option options[] = {\n+\t\tOPT_STRING(0, \"config\", &config_key, N_(\"config\"),\n+\t\t\t N_(\"config key storing a list of repository paths\")),\n+\t\tOPT_END()\n+\t};\n+\n+\targc = parse_options(argc, argv, prefix, options, for_each_repo_usage,\n+\t\t\t PARSE_OPT_STOP_AT_NON_OPTION);\n+\n+\tif (!config_key)\n+\t\tdie(_(\"missing --config=\"));\n+\n+\tfor (i = 0; i < argc; i++)\n+\t\tstrvec_push(&args, argv[i]);\n+\n+\tvalues = repo_config_get_value_multi(the_repository,\n+\t\t\t\t\t config_key);\n+\n+\tfor (i = 0; !result && i < values->nr; i++)\n+\t\tresult = run_command_on_repo(values->items[i].string, &args);\n+\n+\treturn result;\n+}\ndiff --git a/command-list.txt b/command-list.txt\nindex 0e3204e7d1..581499be82 100644\n--- a/command-list.txt\n+++ b/command-list.txt\n@@ -94,6 +94,7 @@ git-fetch-pack synchingrepositories\n git-filter-branch ancillarymanipulators\n git-fmt-merge-msg purehelpers\n git-for-each-ref plumbinginterrogators\n+git-for-each-repo plumbinginterrogators\n git-format-patch mainporcelain\n git-fsck ancillaryinterrogators complete\n git-gc mainporcelain\ndiff --git a/git.c b/git.c\nindex 24f250d29a..1cab64b5d1 100644\n--- a/git.c\n+++ b/git.c\n@@ -511,6 +511,7 @@ static struct cmd_struct commands[] = {\n \t{ \"fetch-pack\", cmd_fetch_pack, RUN_SETUP | NO_PARSEOPT },\n \t{ \"fmt-merge-msg\", cmd_fmt_merge_msg, RUN_SETUP },\n \t{ \"for-each-ref\", cmd_for_each_ref, RUN_SETUP },\n+\t{ \"for-each-repo\", cmd_for_each_repo, RUN_SETUP_GENTLY },\n \t{ \"format-patch\", cmd_format_patch, RUN_SETUP },\n \t{ \"fsck\", cmd_fsck, RUN_SETUP },\n \t{ \"fsck-objects\", cmd_fsck, RUN_SETUP },\ndiff --git a/t/t0068-for-each-repo.sh b/t/t0068-for-each-repo.sh\nnew file mode 100755\nindex 0000000000..136b4ec839\n--- /dev/null\n+++ b/t/t0068-for-each-repo.sh\n@@ -0,0 +1,30 @@\n+#!/bin/sh\n+\n+test_description='git for-each-repo builtin'\n+\n+. ./test-lib.sh\n+\n+test_expect_success 'run based on configured value' '\n+\tgit init one &&\n+\tgit init two &&\n+\tgit init three &&\n+\tgit -C two commit --allow-empty -m \"DID NOT RUN\" &&\n+\tgit config run.key \"$TRASH_DIRECTORY/one\" &&\n+\tgit config --add run.key \"$TRASH_DIRECTORY/three\" &&\n+\tgit for-each-repo --config=run.key commit --allow-empty -m \"ran\" &&\n+\tgit -C one log -1 --pretty=format:%s >message &&\n+\tgrep ran message &&\n+\tgit -C two log -1 --pretty=format:%s >message &&\n+\t! grep ran message &&\n+\tgit -C three log -1 --pretty=format:%s >message &&\n+\tgrep ran message &&\n+\tgit for-each-repo --config=run.key -- commit --allow-empty -m \"ran again\" &&\n+\tgit -C one log -1 --pretty=format:%s >message &&\n+\tgrep again message &&\n+\tgit -C two log -1 --pretty=format:%s >message &&\n+\t! grep again message &&\n+\tgit -C three log -1 --pretty=format:%s >message &&\n+\tgrep again message\n+'\n+\n+test_done\n-- \ngitgitgadget\n\n"},{"id":"404467","messageId":"8a7c34035ac8a86c82dc2bdeee8a9a76fc5626ed.1598380805.git.gitgitgadget@gmail.com","threadId":"54070","inReplyTo":"pull.680.v2.git.1598380805.gitgitgadget@gmail.com","subject":"[PATCH v2 6/7] maintenance: add start/stop subcommands","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-08-25T18:40:03Z","receivedAt":"2020-08-25T18:40:34Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nAdd new subcommands to 'git maintenance' that start or stop background\nmaintenance using 'cron', when available. This integration is as simple\nas I could make it, barring some implementation complications.\n\nFor now, the background maintenance is scheduled to run hourly via the\nfollowing cron table row (ignore line breaks):\n\n\t0 * * * * $p/git --exec-path=$p\n\t\tfor-each-repo --config=maintenance.repo\n\t\tmaintenance run --scheduled\n\nFuture extensions may want to add more complex schedules or some form of\nlogging. For now, hourly runs seem frequent enough to satisfy the needs\nof tasks like 'prefetch' without being so frequent that users would\ncomplain about many no-op commands.\n\nHere, \"$p\" is a placeholder for the path to the current Git executable.\nThis is critical for systems with multiple versions of Git.\nSpecifically, macOS has a system version at '/usr/bin/git' while the\nversion that users can install resides at '/usr/local/bin/git' (symlinked\nto '/usr/local/libexec/git-core/git'). This will also use your\nlocally-built version if you build and run this in your development\nenvironment without installing first.\n\nThe GIT_TEST_CRONTAB environment variable is not intended for users to\nedit, but instead as a way to mock the 'crontab [-l]' command. This\nvariable is set in test-lib.sh to avoid a future test from accidentally\nrunning anything with the cron integration from modifying the user's\nschedule. We use GIT_TEST_CRONTAB='test-tool crontab ' in our\ntests to check how the schedule is modified in 'git maintenance\n(start|stop)' commands.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-maintenance.txt | 11 +++\n Makefile | 1 +\n builtin/gc.c | 117 ++++++++++++++++++++++++++++++\n t/helper/test-crontab.c | 35 +++++++++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/t7900-maintenance.sh | 30 ++++++++\n t/test-lib.sh | 6 ++\n 8 files changed, 202 insertions(+)\n create mode 100644 t/helper/test-crontab.c\n\ndiff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt\nindex c42a176a95..d0316db5ae 100644\n--- a/Documentation/git-maintenance.txt\n+++ b/Documentation/git-maintenance.txt\n@@ -45,6 +45,17 @@ run::\n \tconfig options are true. By default, only `maintenance.gc.enabled`\n \tis true.\n \n+start::\n+\tStart running maintenance on the current repository. This performs\n+\tthe same config updates as the `register` subcommand, then updates\n+\tthe background scheduler to run `git maintenance run --scheduled`\n+\ton an hourly basis.\n+\n+stop::\n+\tHalt the background maintenance schedule. The current repository\n+\tis not removed from the list of maintained repositories, in case\n+\tthe background maintenance is restarted later.\n+\n unregister::\n \tRemove the current repository from background maintenance. This\n \tonly removes the repository from the configured list. It does not\ndiff --git a/Makefile b/Makefile\nindex 7c588ff036..c39b39bd7d 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -690,6 +690,7 @@ TEST_BUILTINS_OBJS += test-advise.o\n TEST_BUILTINS_OBJS += test-bloom.o\n TEST_BUILTINS_OBJS += test-chmtime.o\n TEST_BUILTINS_OBJS += test-config.o\n+TEST_BUILTINS_OBJS += test-crontab.o\n TEST_BUILTINS_OBJS += test-ctype.o\n TEST_BUILTINS_OBJS += test-date.o\n TEST_BUILTINS_OBJS += test-delta.o\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 5218d52cb7..d97af4e546 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -32,6 +32,7 @@\n #include \"remote.h\"\n #include \"midx.h\"\n #include \"object-store.h\"\n+#include \"exec-cmd.h\"\n \n #define FAILED_RUN \"failed to run %s\"\n \n@@ -1463,6 +1464,118 @@ static int maintenance_unregister(void)\n \treturn run_command(&config_unset);\n }\n \n+#define BEGIN_LINE \"# BEGIN GIT MAINTENANCE SCHEDULE\"\n+#define END_LINE \"# END GIT MAINTENANCE SCHEDULE\"\n+\n+static int update_background_schedule(int run_maintenance)\n+{\n+\tint result = 0;\n+\tint in_old_region = 0;\n+\tstruct child_process crontab_list = CHILD_PROCESS_INIT;\n+\tstruct child_process crontab_edit = CHILD_PROCESS_INIT;\n+\tFILE *cron_list, *cron_in;\n+\tconst char *crontab_name;\n+\tstruct strbuf line = STRBUF_INIT;\n+\tstruct lock_file lk;\n+\tchar *lock_path = xstrfmt(\"%s/schedule\", the_repository->objects->odb->path);\n+\n+\tif (hold_lock_file_for_update(&lk, lock_path, LOCK_NO_DEREF) < 0)\n+\t\treturn error(_(\"another process is scheduling background maintenance\"));\n+\n+\tcrontab_name = getenv(\"GIT_TEST_CRONTAB\");\n+\tif (!crontab_name)\n+\t\tcrontab_name = \"crontab\";\n+\n+\tstrvec_split(&crontab_list.args, crontab_name);\n+\tstrvec_push(&crontab_list.args, \"-l\");\n+\tcrontab_list.in = -1;\n+\tcrontab_list.out = dup(lk.tempfile->fd);\n+\tcrontab_list.git_cmd = 0;\n+\n+\tif (start_command(&crontab_list)) {\n+\t\tresult = error(_(\"failed to run 'crontab -l'; your system might not support 'cron'\"));\n+\t\tgoto cleanup;\n+\t}\n+\n+\t/* Ignore exit code, as an empty crontab will return error. */\n+\tfinish_command(&crontab_list);\n+\n+\t/*\n+\t * Read from the .lock file, filtering out the old\n+\t * schedule while appending the new schedule.\n+\t */\n+\tcron_list = fdopen(lk.tempfile->fd, \"r\");\n+\trewind(cron_list);\n+\n+\tstrvec_split(&crontab_edit.args, crontab_name);\n+\tcrontab_edit.in = -1;\n+\tcrontab_edit.git_cmd = 0;\n+\n+\tif (start_command(&crontab_edit)) {\n+\t\tresult = error(_(\"failed to run 'crontab'; your system might not support 'cron'\"));\n+\t\tgoto cleanup;\n+\t}\n+\n+\tcron_in = fdopen(crontab_edit.in, \"w\");\n+\tif (!cron_in) {\n+\t\tresult = error(_(\"failed to open stdin of 'crontab'\"));\n+\t\tgoto done_editing;\n+\t}\n+\n+\twhile (!strbuf_getline_lf(&line, cron_list)) {\n+\t\tif (!in_old_region && !strcmp(line.buf, BEGIN_LINE))\n+\t\t\tin_old_region = 1;\n+\t\tif (in_old_region)\n+\t\t\tcontinue;\n+\t\tfprintf(cron_in, \"%s\\n\", line.buf);\n+\t\tif (in_old_region && !strcmp(line.buf, END_LINE))\n+\t\t\tin_old_region = 0;\n+\t}\n+\n+\tif (run_maintenance) {\n+\t\tconst char *exec_path = git_exec_path();\n+\n+\t\tfprintf(cron_in, \"\\n%s\\n\", BEGIN_LINE);\n+\t\tfprintf(cron_in, \"# The following schedule was created by Git\\n\");\n+\t\tfprintf(cron_in, \"# Any edits made in this region might be\\n\");\n+\t\tfprintf(cron_in, \"# replaced in the future by a Git command.\\n\\n\");\n+\n+\t\tfprintf(cron_in,\n+\t\t\t\"0 * * * * \\\"%s/git\\\" --exec-path=\\\"%s\\\" for-each-repo --config=maintenance.repo maintenance run --scheduled\\n\",\n+\t\t\texec_path, exec_path);\n+\n+\t\tfprintf(cron_in, \"\\n%s\\n\", END_LINE);\n+\t}\n+\n+\tfflush(cron_in);\n+\tfclose(cron_in);\n+\tclose(crontab_edit.in);\n+\n+done_editing:\n+\tif (finish_command(&crontab_edit)) {\n+\t\tresult = error(_(\"'crontab' died\"));\n+\t\tgoto cleanup;\n+\t}\n+\tfclose(cron_list);\n+\n+cleanup:\n+\trollback_lock_file(&lk);\n+\treturn result;\n+}\n+\n+static int maintenance_start(void)\n+{\n+\tif (maintenance_register())\n+\t\twarning(_(\"failed to add repo to global config\"));\n+\n+\treturn update_background_schedule(1);\n+}\n+\n+static int maintenance_stop(void)\n+{\n+\treturn update_background_schedule(0);\n+}\n+\n static const char builtin_maintenance_usage[] =\tN_(\"git maintenance []\");\n \n int cmd_maintenance(int argc, const char **argv, const char *prefix)\n@@ -1472,6 +1585,10 @@ int cmd_maintenance(int argc, const char **argv, const char *prefix)\n \n \tif (!strcmp(argv[1], \"run\"))\n \t\treturn maintenance_run(argc - 1, argv + 1, prefix);\n+\tif (!strcmp(argv[1], \"start\"))\n+\t\treturn maintenance_start();\n+\tif (!strcmp(argv[1], \"stop\"))\n+\t\treturn maintenance_stop();\n \tif (!strcmp(argv[1], \"register\"))\n \t\treturn maintenance_register();\n \tif (!strcmp(argv[1], \"unregister\"))\ndiff --git a/t/helper/test-crontab.c b/t/helper/test-crontab.c\nnew file mode 100644\nindex 0000000000..f5db6319c6\n--- /dev/null\n+++ b/t/helper/test-crontab.c\n@@ -0,0 +1,35 @@\n+#include \"test-tool.h\"\n+#include \"cache.h\"\n+\n+/*\n+ * Usage: test-tool cron [-l]\n+ *\n+ * If -l is specified, then write the contents of to stdou.\n+ * Otherwise, write from stdin into .\n+ */\n+int cmd__crontab(int argc, const char **argv)\n+{\n+\tchar a;\n+\tFILE *from, *to;\n+\n+\tif (argc == 3 && !strcmp(argv[2], \"-l\")) {\n+\t\tfrom = fopen(argv[1], \"r\");\n+\t\tif (!from)\n+\t\t\treturn 0;\n+\t\tto = stdout;\n+\t} else if (argc == 2) {\n+\t\tfrom = stdin;\n+\t\tto = fopen(argv[1], \"w\");\n+\t} else\n+\t\treturn error(\"unknown arguments\");\n+\n+\twhile ((a = fgetc(from)) != EOF)\n+\t\tfputc(a, to);\n+\n+\tif (argc == 3)\n+\t\tfclose(from);\n+\telse\n+\t\tfclose(to);\n+\n+\treturn 0;\n+}\ndiff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\nindex 590b2efca7..432b49d948 100644\n--- a/t/helper/test-tool.c\n+++ b/t/helper/test-tool.c\n@@ -18,6 +18,7 @@ static struct test_cmd cmds[] = {\n \t{ \"bloom\", cmd__bloom },\n \t{ \"chmtime\", cmd__chmtime },\n \t{ \"config\", cmd__config },\n+\t{ \"crontab\", cmd__crontab },\n \t{ \"ctype\", cmd__ctype },\n \t{ \"date\", cmd__date },\n \t{ \"delta\", cmd__delta },\ndiff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\nindex ddc8e990e9..7c3281e071 100644\n--- a/t/helper/test-tool.h\n+++ b/t/helper/test-tool.h\n@@ -8,6 +8,7 @@ int cmd__advise_if_enabled(int argc, const char **argv);\n int cmd__bloom(int argc, const char **argv);\n int cmd__chmtime(int argc, const char **argv);\n int cmd__config(int argc, const char **argv);\n+int cmd__crontab(int argc, const char **argv);\n int cmd__ctype(int argc, const char **argv);\n int cmd__date(int argc, const char **argv);\n int cmd__delta(int argc, const char **argv);\ndiff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\nindex b20ee2d542..6491031be8 100755\n--- a/t/t7900-maintenance.sh\n+++ b/t/t7900-maintenance.sh\n@@ -309,4 +309,34 @@ test_expect_success 'register and unregister' '\n \ttest_cmp before actual\n '\n \n+test_expect_success 'start from empty cron table' '\n+\tGIT_TEST_CRONTAB=\"test-tool crontab cron.txt\" git maintenance start &&\n+\n+\t# start registers the repo\n+\tgit config --get --global maintenance.repo \"$(pwd)\" &&\n+\n+\tgrep \"for-each-repo --config=maintenance.repo maintenance run --scheduled\" cron.txt\n+'\n+\n+test_expect_success 'stop from existing schedule' '\n+\tGIT_TEST_CRONTAB=\"test-tool crontab cron.txt\" git maintenance stop &&\n+\n+\t# stop does not unregister the repo\n+\tgit config --get --global maintenance.repo \"$(pwd)\" &&\n+\n+\t# The newline is preserved\n+\techo >empty &&\n+\ttest_cmp empty cron.txt &&\n+\n+\t# Operation is idempotent\n+\tGIT_TEST_CRONTAB=\"test-tool crontab cron.txt\" git maintenance stop &&\n+\ttest_cmp empty cron.txt\n+'\n+\n+test_expect_success 'start preserves existing schedule' '\n+\techo \"Important information!\" >cron.txt &&\n+\tGIT_TEST_CRONTAB=\"test-tool crontab cron.txt\" git maintenance start &&\n+\tgrep \"Important information!\" cron.txt\n+'\n+\n test_done\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex ef31f40037..4a60d1ed76 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -1702,3 +1702,9 @@ test_lazy_prereq SHA1 '\n test_lazy_prereq REBASE_P '\n \ttest -z \"$GIT_TEST_SKIP_REBASE_P\"\n '\n+\n+# Ensure that no test accidentally triggers a Git command\n+# that runs 'crontab', affecting a user's cron schedule.\n+# Tests that verify the cron integration must set this locally\n+# to avoid errors.\n+GIT_TEST_CRONTAB=\"exit 1\"\n-- \ngitgitgadget\n\n"},{"id":"404468","messageId":"9ecabeb0551653835d4992a8ac984b9995bd92f0.1598380805.git.gitgitgadget@gmail.com","threadId":"54070","inReplyTo":"pull.680.v2.git.1598380805.gitgitgadget@gmail.com","subject":"[PATCH v2 7/7] maintenance: recommended schedule in register/start","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-08-25T18:40:04Z","receivedAt":"2020-08-25T18:40:37Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe 'git maintenance (register|start)' subcommands add the current\nrepository to the global Git config so maintenance will operate on that\nrepository. It does not specify what maintenance should occur or how\noften.\n\nIf a user sets any 'maintenance..scheduled' config value, then\nthey have chosen a specific schedule for themselves and Git should\nrespect that.\n\nHowever, in an effort to recommend a good schedule for repositories of\nall sizes, set new config values for recommended tasks that are safe to\nrun in the background while users run foreground Git commands. These\ncommands are generally everything but the 'gc' task.\n\nAuthor's Note: I feel we should do _something_ to recommend a good\nschedule to users, but I'm not 100% set on this schedule. This is the\nschedule we use in Scalar and VFS for Git for very large repositories\nusing the GVFS protocol. While the schedule works in that environment,\nit is possible that \"normal\" Git repositories could benefit from\nsomething more obvious (such as running 'gc' once a day). However, this\npatch gives us a place to start a conversation on what we should\nrecommend. For my purposes, Scalar will set these config values so we\ncan always differ from core Git's recommendations.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-maintenance.txt | 6 +++++\n builtin/gc.c | 44 +++++++++++++++++++++++++++++++\n t/t7900-maintenance.sh | 5 ++++\n 3 files changed, 55 insertions(+)\n\ndiff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt\nindex d0316db5ae..bba76f0b0d 100644\n--- a/Documentation/git-maintenance.txt\n+++ b/Documentation/git-maintenance.txt\n@@ -37,6 +37,12 @@ register::\n \t`maintenance..schedule`. The tasks that are enabled are safe\n \tfor running in the background without disrupting foreground\n \tprocesses.\n++\n+If your repository has no 'maintenance..schedule' configuration\n+values set, then Git will set configuration values to some recommended\n+settings. These settings disable foreground maintenance while performing\n+maintenance tasks in the background that will not interrupt foreground Git\n+operations.\n \n run::\n \tRun one or more maintenance tasks. If one or more `--task` options\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex d97af4e546..037402e47f 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -1415,6 +1415,47 @@ static int maintenance_run(int argc, const char **argv, const char *prefix)\n \treturn maintenance_run_tasks(&opts);\n }\n \n+static int has_schedule_config(void)\n+{\n+\tint i, found = 0;\n+\tstruct strbuf config_name = STRBUF_INIT;\n+\tsize_t prefix;\n+\n+\tstrbuf_addstr(&config_name, \"maintenance.\");\n+\tprefix = config_name.len;\n+\n+\tfor (i = 0; !found && i < TASK__COUNT; i++) {\n+\t\tint value;\n+\n+\t\tstrbuf_setlen(&config_name, prefix);\n+\t\tstrbuf_addf(&config_name, \"%s.schedule\", tasks[i].name);\n+\n+\t\tif (!git_config_get_int(config_name.buf, &value))\n+\t\t\tfound = 1;\n+\t}\n+\n+\tstrbuf_release(&config_name);\n+\treturn found;\n+}\n+\n+static void set_recommended_schedule(void)\n+{\n+\tgit_config_set(\"maintenance.auto\", \"false\");\n+\tgit_config_set(\"maintenance.gc.enabled\", \"false\");\n+\n+\tgit_config_set(\"maintenance.prefetch.enabled\", \"true\");\n+\tgit_config_set(\"maintenance.prefetch.schedule\", \"3500\");\n+\n+\tgit_config_set(\"maintenance.commit-graph.enabled\", \"true\");\n+\tgit_config_set(\"maintenance.commit-graph.schedule\", \"3500\");\n+\n+\tgit_config_set(\"maintenance.loose-objects.enabled\", \"true\");\n+\tgit_config_set(\"maintenance.loose-objects.schedule\", \"86000\");\n+\n+\tgit_config_set(\"maintenance.incremental-repack.enabled\", \"true\");\n+\tgit_config_set(\"maintenance.incremental-repack.schedule\", \"86000\");\n+}\n+\n static int maintenance_register(void)\n {\n \tstruct child_process config_set = CHILD_PROCESS_INIT;\n@@ -1424,6 +1465,9 @@ static int maintenance_register(void)\n \tif (!the_repository || !the_repository->gitdir)\n \t\treturn 0;\n \n+\tif (has_schedule_config())\n+\t\tset_recommended_schedule();\n+\n \tconfig_get.git_cmd = 1;\n \tstrvec_pushl(&config_get.args, \"config\", \"--global\", \"--get\", \"maintenance.repo\",\n \t\t the_repository->worktree ? the_repository->worktree\ndiff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\nindex 6491031be8..7417e5858a 100755\n--- a/t/t7900-maintenance.sh\n+++ b/t/t7900-maintenance.sh\n@@ -300,6 +300,11 @@ test_expect_success 'register and unregister' '\n \tgit config --global --add maintenance.repo /existing2 &&\n \tgit config --global --get-all maintenance.repo >before &&\n \tgit maintenance register &&\n+\ttest_cmp_config false maintenance.auto &&\n+\ttest_cmp_config false maintenance.gc.enabled &&\n+\ttest_cmp_config true maintenance.prefetch.enabled &&\n+\ttest_cmp_config 3500 maintenance.commit-graph.schedule &&\n+\ttest_cmp_config 86000 maintenance.incremental-repack.schedule &&\n \tgit config --global --get-all maintenance.repo >actual &&\n \tcp before after &&\n \tpwd >>after &&\n-- \ngitgitgadget\n"},{"id":"404469","messageId":"c0ce1267a99606e7632c8930c3b4d6c33f901101.1598380805.git.gitgitgadget@gmail.com","threadId":"54070","inReplyTo":"pull.680.v2.git.1598380805.gitgitgadget@gmail.com","subject":"[PATCH v2 5/7] maintenance: add [un]register subcommands","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-08-25T18:40:02Z","receivedAt":"2020-08-25T18:40:40Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIn preparation for launching background maintenance from the 'git\nmaintenance' builtin, create register/unregister subcommands. These\ncommands update the new 'maintenance.repos' config option in the global\nconfig so the background maintenance job knows which repositories to\nmaintain.\n\nThese commands allow users to add a repository to the background\nmaintenance list without disrupting the actual maintenance mechanism.\n\nFor example, a user can run 'git maintenance register' when no\nbackground maintenance is running and it will not start the background\nmaintenance. A later update to start running background maintenance will\nthen pick up this repository automatically.\n\nThe opposite example is that a user can run 'git maintenance unregister'\nto remove the current repository from background maintenance without\nhalting maintenance for other repositories.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-maintenance.txt | 14 ++++++++\n builtin/gc.c | 55 ++++++++++++++++++++++++++++++-\n t/t7900-maintenance.sh | 17 +++++++++-\n 3 files changed, 84 insertions(+), 2 deletions(-)\n\ndiff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt\nindex 2bc02c65e4..c42a176a95 100644\n--- a/Documentation/git-maintenance.txt\n+++ b/Documentation/git-maintenance.txt\n@@ -29,6 +29,15 @@ Git repository.\n SUBCOMMANDS\n -----------\n \n+register::\n+\tInitialize Git config values so any scheduled maintenance will\n+\tstart running on this repository. This adds the repository to the\n+\t`maintenance.repo` config variable in the current user's global\n+\tconfig and enables some recommended configuration values for\n+\t`maintenance..schedule`. The tasks that are enabled are safe\n+\tfor running in the background without disrupting foreground\n+\tprocesses.\n+\n run::\n \tRun one or more maintenance tasks. If one or more `--task` options\n \tare specified, then those tasks are run in that order. Otherwise,\n@@ -36,6 +45,11 @@ run::\n \tconfig options are true. By default, only `maintenance.gc.enabled`\n \tis true.\n \n+unregister::\n+\tRemove the current repository from background maintenance. This\n+\tonly removes the repository from the configured list. It does not\n+\tstop the background maintenance processes from running.\n+\n TASKS\n -----\n \ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 5726a9a3b3..5218d52cb7 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -1414,7 +1414,56 @@ static int maintenance_run(int argc, const char **argv, const char *prefix)\n \treturn maintenance_run_tasks(&opts);\n }\n \n-static const char builtin_maintenance_usage[] = N_(\"git maintenance run []\");\n+static int maintenance_register(void)\n+{\n+\tstruct child_process config_set = CHILD_PROCESS_INIT;\n+\tstruct child_process config_get = CHILD_PROCESS_INIT;\n+\n+\t/* There is no current repository, so skip registering it */\n+\tif (!the_repository || !the_repository->gitdir)\n+\t\treturn 0;\n+\n+\tconfig_get.git_cmd = 1;\n+\tstrvec_pushl(&config_get.args, \"config\", \"--global\", \"--get\", \"maintenance.repo\",\n+\t\t the_repository->worktree ? the_repository->worktree\n+\t\t\t\t\t : the_repository->gitdir,\n+\t\t\t NULL);\n+\tconfig_get.out = -1;\n+\n+\tif (start_command(&config_get))\n+\t\treturn error(_(\"failed to run 'git config'\"));\n+\n+\t/* We already have this value in our config! */\n+\tif (!finish_command(&config_get))\n+\t\treturn 0;\n+\n+\tconfig_set.git_cmd = 1;\n+\tstrvec_pushl(&config_set.args, \"config\", \"--add\", \"--global\", \"maintenance.repo\",\n+\t\t the_repository->worktree ? the_repository->worktree\n+\t\t\t\t\t : the_repository->gitdir,\n+\t\t NULL);\n+\n+\treturn run_command(&config_set);\n+}\n+\n+static int maintenance_unregister(void)\n+{\n+\tstruct child_process config_unset = CHILD_PROCESS_INIT;\n+\n+\tif (!the_repository || !the_repository->gitdir)\n+\t\treturn error(_(\"no current repository to unregister\"));\n+\n+\tconfig_unset.git_cmd = 1;\n+\tstrvec_pushl(&config_unset.args, \"config\", \"--global\", \"--unset\",\n+\t\t \"maintenance.repo\",\n+\t\t the_repository->worktree ? the_repository->worktree\n+\t\t\t\t\t : the_repository->gitdir,\n+\t\t NULL);\n+\n+\treturn run_command(&config_unset);\n+}\n+\n+static const char builtin_maintenance_usage[] =\tN_(\"git maintenance []\");\n \n int cmd_maintenance(int argc, const char **argv, const char *prefix)\n {\n@@ -1423,6 +1472,10 @@ int cmd_maintenance(int argc, const char **argv, const char *prefix)\n \n \tif (!strcmp(argv[1], \"run\"))\n \t\treturn maintenance_run(argc - 1, argv + 1, prefix);\n+\tif (!strcmp(argv[1], \"register\"))\n+\t\treturn maintenance_register();\n+\tif (!strcmp(argv[1], \"unregister\"))\n+\t\treturn maintenance_unregister();\n \n \tdie(_(\"invalid subcommand: %s\"), argv[1]);\n }\ndiff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\nindex 3e0c5f1ca8..b20ee2d542 100755\n--- a/t/t7900-maintenance.sh\n+++ b/t/t7900-maintenance.sh\n@@ -9,7 +9,7 @@ GIT_TEST_MULTI_PACK_INDEX=0\n \n test_expect_success 'help text' '\n \ttest_expect_code 129 git maintenance -h 2>err &&\n-\ttest_i18ngrep \"usage: git maintenance run\" err &&\n+\ttest_i18ngrep \"usage: git maintenance \" err &&\n \ttest_expect_code 128 git maintenance barf 2>err &&\n \ttest_i18ngrep \"invalid subcommand: barf\" err\n '\n@@ -294,4 +294,19 @@ test_expect_success '--scheduled with specific time' '\n \ttest_cmp_config 1595000100 maintenance.commit-graph.lastrun\n '\n \n+test_expect_success 'register and unregister' '\n+\ttest_when_finished git config --global --unset-all maintenance.repo &&\n+\tgit config --global --add maintenance.repo /existing1 &&\n+\tgit config --global --add maintenance.repo /existing2 &&\n+\tgit config --global --get-all maintenance.repo >before &&\n+\tgit maintenance register &&\n+\tgit config --global --get-all maintenance.repo >actual &&\n+\tcp before after &&\n+\tpwd >>after &&\n+\ttest_cmp after actual &&\n+\tgit maintenance unregister &&\n+\tgit config --global --get-all maintenance.repo >actual &&\n+\ttest_cmp before actual\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"404482","messageId":"xmqqlfi2z9ht.fsf@gitster.c.googlers.com","threadId":"54070","inReplyTo":"5fdd8188b1d9b6efc2803b557b3ba344e184d22e.1598380805.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 1/7] maintenance: optionally skip --auto process","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-08-25T21:44:14Z","receivedAt":"2020-08-25T21:44:24Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" writes:\n\n> From: Derrick Stolee \n>\n> Some commands run 'git maintenance run --auto --[no-]quiet' after doing\n> their normal work, as a way to keep repositories clean as they are used.\n> Currently, users who do not want this maintenance to occur would set the\n> 'gc.auto' config option to 0 to avoid the 'gc' task from running.\n> However, this does not stop the extra process invocation.\n\nOK, that is because the configuration is checked on the other side,\nand the new check is implemented on this side before we decide to\nspawn the maintenance task.\n\nIt sounds like a change worth having without even waiting for the\n\"git maintenance\" to materialize ;-).\n\n> @@ -1868,8 +1869,13 @@ int run_processes_parallel_tr2(int n, get_next_task_fn get_next_task,\n> \n> int run_auto_maintenance(int quiet)\n> {\n> +\tint enabled;\n> \tstruct child_process maint = CHILD_PROCESS_INIT;\n> \n> +\tif (!git_config_get_bool(\"maintenance.auto\", &enabled) &&\n> +\t !enabled)\n> +\t\treturn 0;\n\nSo in a repository without this configuration, get_bool would fail\nand we do not short-circuit. Only if the value get_bool sees is\nfalse, we return without running the command. Makes sense.\n\n"},{"id":"404484","messageId":"xmqqh7sqz93k.fsf@gitster.c.googlers.com","threadId":"54070","inReplyTo":"e3ef0b9bea4a31c72ce88841639e88355408f0c1.1598380805.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 2/7] maintenance: store the \"last run\" time in config","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-08-25T21:52:47Z","receivedAt":"2020-08-25T21:52:57Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" writes:\n\n> I selected the timestamp before the task, as opposed to after the task,\n> for a couple reasons:\n>\n> 1. The time the task takes to execute should not contribute to the\n> interval between running the tasks.\n\n... as long as the run time is sufficiently shorter than the\ninterval, that is. If a task takes 10-30 minutes depending on how\ndirty the repository is, it does not make sense to even try to run\nit every 15 minutes.\n\n> If a daily task takes 10 minutes\n> to run, then every day the execution will drift by at least 10\n> minutes.\n\nThat is not incorrect per-se, but it does not tell us why drifting\nby 10 minutes is a bad thing.\n\n> 2. If the task fails for some unforseen reason, it would be good to\n> indicate that we _attempted_ the task at a certain timestamp. This\n> will avoid spamming a repository that is in a bad state.\n\nAbsolutely.\n\n> +static void update_last_run(struct maintenance_task *task)\n> +{\n> +\ttimestamp_t now = approxidate(\"now\");\n> +\tstruct strbuf config = STRBUF_INIT;\n> +\tstruct strbuf value = STRBUF_INIT;\n> +\tstrbuf_addf(&config, \"maintenance.%s.lastrun\", task->name);\n> +\tstrbuf_addf(&value, \"%\"PRItime\"\", now);\n\nSo is this essentially meant as a human-unreadable opaque value,\nlike we have in the commit object header lines? I do not have a\nstrong opinion, but it would be nice to allow curious to casually\nread it. Perhaps \"git config --type=timestamp maintenance.lastrun\"\ncan be taught to pretty print its value?\n\n"},{"id":"404485","messageId":"xmqqd03ez8pp.fsf@gitster.c.googlers.com","threadId":"54070","inReplyTo":"c728c57d85b17035d42313260620a7de5756b0c3.1598380805.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 3/7] maintenance: add --scheduled option and config","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-08-25T22:01:06Z","receivedAt":"2020-08-25T22:01:16Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" writes:\n\n> From: Derrick Stolee \n>\n> A user may want to run certain maintenance tasks based on frequency, not\n> conditions given in the repository. For example, the user may want to\n> perform a 'prefetch' task every hour, or 'gc' task every day. To assist,\n> update the 'git maintenance run --scheduled' command to check the config\n> for the last run of that task and add a number of seconds. The task\n> would then run only if the current time is beyond that minimum\n> timestamp.\n>\n> Add a '--scheduled' option to 'git maintenance run' to only run tasks\n> that have had enough time pass since their last run. This is done for\n> each enabled task by checking if the current timestamp is at least as\n> large as the sum of 'maintenance..lastRun' and\n> 'maintenance..schedule' in the Git config. This second value is\n> new to this commit, storing a number of seconds intended between runs.\n>\n> A user could then set up an hourly maintenance run with the following\n> cron table:\n>\n> 0 * * * * git -C maintenance run --scheduled\n\nThe scheme has one obvious drawback. An hourly crontab entry means\nyour maintenance.*.schedule that is finer grained than an hour\nincrement will not run as expected. You'd need to take all the\nschedule intervals and take their GCD to come up with the frequency\nof the single crontab entry. \n\nWouldn't it make more sense to have N crontab entries for N tasks\nyou want to run periodically, each with their own frequency\ncontrolled by crontab? That way, you do not need to maintain\nmaintenance.*.schedule configuration variables and the --scheduled\noption. It might make maintenance.*.lastrun timestamps unneeded,\nwhich would be an added plus to simplify the system quite\ndrastically. Most importantly, that would be the way crontab users\nare most used to in order to schedule their periodical jobs, so it\nis one less thing to learn.\n\n\n\n"},{"id":"404486","messageId":"xmqq8se2z7uh.fsf@gitster.c.googlers.com","threadId":"54070","inReplyTo":"0314258c5cbb8fd771c35e433bf6be95297c4597.1598380805.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 4/7] for-each-repo: run subcommands on configured repos","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-08-25T22:19:50Z","receivedAt":"2020-08-25T22:19:59Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" writes:\n\n> +SYNOPSIS\n> +--------\n> +[verse]\n> +'git for-each-repo' --config= [--] \n> + ...\n> +--config=::\n> +\tUse the given config variable as a multi-valued list storing\n> +\tabsolute path names.\n\nWould it make sense to allow this config to be read from the current\nrepository, I wonder. It is probably designed to be written to\neither ~/.gitconfig or /etc/gitconfig because it is probably a need\nthat is not per-repository to list repositories for various purposes\nspecified by the config key, but I suspect there _might_ be a good\nuse case for storing some custom list of repositories in the\nconfiguration file local to a repository, but it is not quite\nobvious what it is.\n\nIf we have a good example, we may want to spell it out---that would\nhelp future readers who wonder about this (just like I am doing now).\n\nAlso, if we do read from local config, should there be a way to say\n\"ah, you may have read values from /etc/gitconfig and ~/.gitconfig,\nbut please forget them---I have a full list I care when you are\nrunning in this repository\", i.e. clear the list. It is purely a\nconvention and there is no built-in mechanism for this in the config\nAPI, but often it is signalled by giving an empty string as a value.\n\nBy the way, I do not have a good concrete suggestion, but can we use\nsomething better than as the placeholder? I first thought\nthis was naming the name of a file that lists repositories, not the\nconfig variable name in our usual config namespace.\n\n> +static int run_command_on_repo(const char *path,\n> +\t\t\t void *cbdata)\n\nIs that on repo or in repo? When I saw \"-C\" on the command line, I\nimmediately thought of \"in repo\".\n\n> +{\n> +\tint i;\n> +\tstruct child_process child = CHILD_PROCESS_INIT;\n> +\tstruct strvec *args = (struct strvec *)cbdata;\n> +\n> +\tchild.git_cmd = 1;\n> +\tstrvec_pushl(&child.args, \"-C\", path, NULL);\n> +\n> +\tfor (i = 0; i < args->nr; i++)\n> +\t\tstrvec_push(&child.args, args->v[i]);\n\nWould strvec_pushv() work, or is args->v[] not NULL terminated?\n\n> +\treturn run_command(&child);\n> +}\n\n\n> +\tvalues = repo_config_get_value_multi(the_repository,\n> +\t\t\t\t\t config_key);\n\nNot your fault, but it is a bit unsatisfactory that we do not have\nspecial \"type\" meant for paths in the config API, unlike the\nparse-options API where there is a \"filename\" type that is a bit\nricher than a vanilla \"string\" type by allowing \"prefix\" handling.\nFor the purposes of this, as the values are limited to absolute/full\npathnames, it does not hurt as much, though.\n\nThanks.\n"},{"id":"404515","messageId":"1e107b2e-364a-4c17-90b8-0371e8e27878@gmail.com","threadId":"54070","inReplyTo":"xmqqlfi2z9ht.fsf@gitster.c.googlers.com","subject":"Re: [PATCH v2 1/7] maintenance: optionally skip --auto process","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-08-26T12:29:10Z","receivedAt":"2020-08-26T12:29:19Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 8/25/2020 5:44 PM, Junio C Hamano wrote:\n> \"Derrick Stolee via GitGitGadget\" writes:\n> \n>> From: Derrick Stolee \n>>\n>> Some commands run 'git maintenance run --auto --[no-]quiet' after doing\n>> their normal work, as a way to keep repositories clean as they are used.\n>> Currently, users who do not want this maintenance to occur would set the\n>> 'gc.auto' config option to 0 to avoid the 'gc' task from running.\n>> However, this does not stop the extra process invocation.\n> \n> OK, that is because the configuration is checked on the other side,\n> and the new check is implemented on this side before we decide to\n> spawn the maintenance task.\n> \n> It sounds like a change worth having without even waiting for the\n> \"git maintenance\" to materialize ;-).\n\nTrue. This one could be pulled out of Part III and placed any time\nafter Part I (outside of test script context conflicts). The only\nreason to wait until after Part I is that the name \"maintenance.auto\"\nimplies a connection to the maintenance builtin instead of gc.\n\nBefore the maintenance builtin, it would be natural to see \"gc.auto=0\"\nand do this same behavior, but that will not be general enough in the\nfuture.\n\nIf you prefer, I can pull this out into a series on its own to be\ntracked separately.\n\nThanks,\n-Stolee\n\n"},{"id":"404516","messageId":"20200826124240.GO21928@kitsune.suse.cz","threadId":"54070","inReplyTo":"pull.680.git.1597857408.gitgitgadget@gmail.com","subject":"Re: [PATCH 0/7] [RFC] Maintenance III: background maintenance","fromName":"Michal Suchánek","fromEmail":"msuchanek@suse.de","sentAt":"2020-08-26T12:42:40Z","receivedAt":"2020-08-26T12:42:44Z","isPatch":true,"sender":{"key":"msuchanek@suse.de","avatar":"https://avatars.githubusercontent.com/u/787652?v=4"},"body":"On Wed, Aug 19, 2020 at 05:16:41PM +0000, Derrick Stolee via GitGitGadget wrote:\n> This is based on ds/maintenance-part-2, but with some local updates to\n> review feedback. It won't apply cleanly right now. This RFC is for early\n> feedback and not intended to make a new tracking branch until v2.\n> \n> This RFC is intended to show how I hope to integrate true background\n> maintenance into Git. As opposed to my original RFC [1], this entirely\n> integrates with cron (through crontab [-e|-l]) to launch maintenance\n> commands in the background.\n> \n> [1] \n> https://lore.kernel.org/git/pull.597.git.1585946894.gitgitgadget@gmail.com/\n> \n> Some preliminary work is done to allow a new --scheduled option that\n> triggers enabled tasks only if they have not been run in some amount of\n> time. The timestamp of the previous run is stored in the \n> maintenance..lastRun config value while the interval is stored in the \n> maintenance..schedule config value.\nThis changes the config file from read-mostly to continuously updated. Is\nthat desirable?\nIn particular it significanly increases the risk of race with the user\nediting the file.\nI think timestamps are not configuration and should be written to some\nother file.\nOr is there already a core git feature that continuously updates the\nconfig file?\n\nThanks\n\nMichal\n"},{"id":"404517","messageId":"4fe41132-c8f5-6b85-414f-5a1e5fcccdc9@gmail.com","threadId":"54070","inReplyTo":"xmqqh7sqz93k.fsf@gitster.c.googlers.com","subject":"Re: [PATCH v2 2/7] maintenance: store the \"last run\" time in config","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-08-26T13:34:41Z","receivedAt":"2020-08-26T13:34:45Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 8/25/2020 5:52 PM, Junio C Hamano wrote:\n> \"Derrick Stolee via GitGitGadget\" writes:\n> \n>> I selected the timestamp before the task, as opposed to after the task,\n>> for a couple reasons:\n>>\n>> 1. The time the task takes to execute should not contribute to the\n>> interval between running the tasks.\n> \n> ... as long as the run time is sufficiently shorter than the\n> interval, that is. If a task takes 10-30 minutes depending on how\n> dirty the repository is, it does not make sense to even try to run\n> it every 15 minutes.\n\nDefinitely. The lock on the object database from earlier prevents these\nlonger-than-anticipated tasks from stacking.\n\n>> If a daily task takes 10 minutes\n>> to run, then every day the execution will drift by at least 10\n>> minutes.\n> \n> That is not incorrect per-se, but it does not tell us why drifting\n> by 10 minutes is a bad thing.\n\nTrue.\n\n>> 2. If the task fails for some unforseen reason, it would be good to\n>> indicate that we _attempted_ the task at a certain timestamp. This\n>> will avoid spamming a repository that is in a bad state.\n> \n> Absolutely.\n> \n>> +static void update_last_run(struct maintenance_task *task)\n>> +{\n>> +\ttimestamp_t now = approxidate(\"now\");\n>> +\tstruct strbuf config = STRBUF_INIT;\n>> +\tstruct strbuf value = STRBUF_INIT;\n>> +\tstrbuf_addf(&config, \"maintenance.%s.lastrun\", task->name);\n>> +\tstrbuf_addf(&value, \"%\"PRItime\"\", now);\n> \n> So is this essentially meant as a human-unreadable opaque value,\n> like we have in the commit object header lines? I do not have a\n> strong opinion, but it would be nice to allow curious to casually\n> read it. Perhaps \"git config --type=timestamp maintenance.lastrun\"\n> can be taught to pretty print its value?\n\nGood idea. I will think on this. Of course, we already have\n--type=expiry-date, which does the opposite. Perhaps this config\nvalue should be a human-readable date and then be parsed into a\ntimestamp in-process using git_config_expiry_date()?\n\nI have mixed feelings on using that format, because it can store\nboth a fixed or relative datetime. The *.lastRun config really\nwants a _fixed_ datetime, but the *.schedule config (in the next\npatch) would want a _relative_ datetime. This also allows things\nlike \"now\" or \"never\", so it presents a lot of flexibility for\nusers. A nightmare to test, but perhaps that flexibility is\nuseful.\n\n(Of course, in another thread you mentioned multiple `crontab`\nlines, which might make this entire discussion irrelevant. I'll\nfollow up there.)\n\nThanks,\n-Stolee\n"},{"id":"404526","messageId":"bd4e18b7-6265-73e7-bc1a-a7d647eafd0a@gmail.com","threadId":"54070","inReplyTo":"xmqqd03ez8pp.fsf@gitster.c.googlers.com","subject":"Re: [PATCH v2 3/7] maintenance: add --scheduled option and config","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-08-26T15:30:29Z","receivedAt":"2020-08-26T15:30:35Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 8/25/2020 6:01 PM, Junio C Hamano wrote:\n> \"Derrick Stolee via GitGitGadget\" writes:\n> \n>> From: Derrick Stolee \n>>\n>> A user may want to run certain maintenance tasks based on frequency, not\n>> conditions given in the repository. For example, the user may want to\n>> perform a 'prefetch' task every hour, or 'gc' task every day. To assist,\n>> update the 'git maintenance run --scheduled' command to check the config\n>> for the last run of that task and add a number of seconds. The task\n>> would then run only if the current time is beyond that minimum\n>> timestamp.\n>>\n>> Add a '--scheduled' option to 'git maintenance run' to only run tasks\n>> that have had enough time pass since their last run. This is done for\n>> each enabled task by checking if the current timestamp is at least as\n>> large as the sum of 'maintenance..lastRun' and\n>> 'maintenance..schedule' in the Git config. This second value is\n>> new to this commit, storing a number of seconds intended between runs.\n>>\n>> A user could then set up an hourly maintenance run with the following\n>> cron table:\n>>\n>> 0 * * * * git -C maintenance run --scheduled\n> \n> The scheme has one obvious drawback. An hourly crontab entry means\n> your maintenance.*.schedule that is finer grained than an hour\n> increment will not run as expected. You'd need to take all the\n> schedule intervals and take their GCD to come up with the frequency\n> of the single crontab entry. \n\nMy intention for the *.schedule is that it is not an _exact_ frequency,\nbut instead a lower bound on the frequency. That can be shelved for now\nas we discuss this setup:\n\n> Wouldn't it make more sense to have N crontab entries for N tasks\n> you want to run periodically, each with their own frequency\n> controlled by crontab? That way, you do not need to maintain\n> maintenance.*.schedule configuration variables and the --scheduled\n> option. It might make maintenance.*.lastrun timestamps unneeded,\n> which would be an added plus to simplify the system quite\n> drastically. Most importantly, that would be the way crontab users\n> are most used to in order to schedule their periodical jobs, so it\n> is one less thing to learn.\n\nI had briefly considered setting up crontab entries for each task\n(and possibly each repo) but ended up with these complications:\n\n 1. Maintenance frequency differs by task, so we need to split the\n crontab by task. But we can't just split everything because we\n do not want multiple tasks running at the same time on one\n repository. We would need to group the tasks and have one entry\n saying \"git maintenance run --task= --task= ...\"\n for all tasks in the group.\n\n 2. Different repositories might want different tasks at different\n frequencies, so we might need to split the crontab by repository.\n Again, we likely want to group repositories by these frequencies\n because a user could have 100 registered repositories and we don't\n really want to launch 100 parallel processes.\n\n 3. If we want to stop maintenance, then restart it, we need to\n clear the crontab and repopulate it, which would require iterating\n through all \"registered\" repositories to read their config for\n frequencies.\n\n 4. On macOS, editing the crontab doesn't require \"sudo\" but it _does_\n create a pop-up(!) to get permission from the user. It would be\n good to minimize how often we edit the crontab and instead use\n config edits to change frequencies.\n\nWith these things in mind, here is a suggested alternative design:\n\nLet users specify a schedule frequency among this list: hourly, daily,\nweekly, monthly. We then set the following* crontab:\n\n\t0 * * * * git for-each-repo --config=maintenance.repos maintenance run --scheduled=hourly\n\t0 0 * * * git for-each-repo --config=maintenance.repos maintenance run --scheduled=daily\n\t0 0 * * 0 git for-each-repo --config=maintenance.repos maintenance run --scheduled=weekly\n\t0 0 0 * * git for-each-repo --config=maintenance.repos maintenance run --scheduled=monthly\n\n*Of course, there is some care around \"$path/git --exec-path=$path\"\nthat I drop for ease here.\n\nThen, \"git maintenance (start|stop)\" can be just as simple as we have\nnow: write a fixed schedule every time.\n\nThe problem here is that cron will launch these processes in parallel,\nand then our object-database lock will cause some to fail! If anyone\nknows a simple way to tell cron \"run hourly _except_ not at midnight\"\nthen we could let the \"daily\" schedule also run the \"hourly\" jobs, for\ninstance. Hopefully that pattern could be extended to the weekly and\nmonthly collisions.\n\nAlternatively, we could run every hour and then interpret from config\nif the current \"hour\" matches one of the schedules ourselves. So, the\ncrontab would be this simple:\n\n\t0 * * * * git for-each-repo --config=maintenance.repos maintenance run --scheduled\n\nand then we would internally decide \"is this the midnight hour?\" and\n\"is this the first day of the week?\" and \"is this the first day of the\nmonth?\" to detect if we should run the daily/weekly/monthly tasks. While\nit adds more time-awareness into Git, it does avoid the parallel task\ncollisions. There are some concerns here related to long-running tasks\ndelaying sequential runs of \"git -C maintenance run --scheduled\"\ncausing the \"is this the midnight hour?\" queries to fail and having\nnightly/weekly/monthly maintenance be skipped accidentally. This\nmotivates the *.lastRun config giving us some guarantee of _eventually_\nrunning the tasks, just _not too frequently_.\n\nI hope this launches a good discussion to help us find a good cron\nschedule strategy. After we land on a suitable strategy, I'll summarize\nall of these subtleties in the commit message for posterity.\n\nHopefully, the current way that I integrate with crontab and test that\nintegration (in PATCH 6/7) could also be reviewed in parallel with this\ndiscussion. I'm very curious to see how that could be improved.\n\nThanks,\n-Stolee\n"},{"id":"404528","messageId":"30cadb74-924e-09bc-1ca4-352de3e629f3@gmail.com","threadId":"54070","inReplyTo":"xmqq8se2z7uh.fsf@gitster.c.googlers.com","subject":"Re: [PATCH v2 4/7] for-each-repo: run subcommands on configured repos","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-08-26T16:03:21Z","receivedAt":"2020-08-26T16:03:45Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 8/25/2020 6:19 PM, Junio C Hamano wrote:\n> \"Derrick Stolee via GitGitGadget\" writes:\n> \n>> +SYNOPSIS\n>> +--------\n>> +[verse]\n>> +'git for-each-repo' --config= [--] \n>> + ...\n>> +--config=::\n>> +\tUse the given config variable as a multi-valued list storing\n>> +\tabsolute path names.\n> \n> Would it make sense to allow this config to be read from the current\n> repository, I wonder. It is probably designed to be written to\n> either ~/.gitconfig or /etc/gitconfig because it is probably a need\n> that is not per-repository to list repositories for various purposes\n> specified by the config key, but I suspect there _might_ be a good\n> use case for storing some custom list of repositories in the\n> configuration file local to a repository, but it is not quite\n> obvious what it is.\n> \n> If we have a good example, we may want to spell it out---that would\n> help future readers who wonder about this (just like I am doing now).\n> \n> Also, if we do read from local config, should there be a way to say\n> \"ah, you may have read values from /etc/gitconfig and ~/.gitconfig,\n> but please forget them---I have a full list I care when you are\n> running in this repository\", i.e. clear the list. It is purely a\n> convention and there is no built-in mechanism for this in the config\n> API, but often it is signalled by giving an empty string as a value.\n\nI guess I should test this, but if I ask for a multi-valued config,\nwill I not get _all_ of the results from /etc/gitconfig, ~/.gitconfig,\nAND .git/config? That was my expectation, which is why I don't specify\n\"local\" or \"global\" config anywhere in the discussion.\n\n> By the way, I do not have a good concrete suggestion, but can we use\n> something better than as the placeholder? I first thought\n> this was naming the name of a file that lists repositories, not the\n> config variable name in our usual config namespace.\nSure. How about \"\"?\n\n>> +static int run_command_on_repo(const char *path,\n>> +\t\t\t void *cbdata)\n> \n> Is that on repo or in repo? When I saw \"-C\" on the command line, I\n> immediately thought of \"in repo\".\n\n\"in\" is better.\n\n>> +{\n>> +\tint i;\n>> +\tstruct child_process child = CHILD_PROCESS_INIT;\n>> +\tstruct strvec *args = (struct strvec *)cbdata;\n>> +\n>> +\tchild.git_cmd = 1;\n>> +\tstrvec_pushl(&child.args, \"-C\", path, NULL);\n>> +\n>> +\tfor (i = 0; i < args->nr; i++)\n>> +\t\tstrvec_push(&child.args, args->v[i]);\n> \n> Would strvec_pushv() work, or is args->v[] not NULL terminated?\n\nYeah, pushv should work.\n\n>> +\treturn run_command(&child);\n>> +}\n> \n> \n>> +\tvalues = repo_config_get_value_multi(the_repository,\n>> +\t\t\t\t\t config_key);\n> \n> Not your fault, but it is a bit unsatisfactory that we do not have\n> special \"type\" meant for paths in the config API, unlike the\n> parse-options API where there is a \"filename\" type that is a bit\n> richer than a vanilla \"string\" type by allowing \"prefix\" handling.\n> For the purposes of this, as the values are limited to absolute/full\n> pathnames, it does not hurt as much, though.\n\nInteresting. Noted.\n\nThanks,\n-Stolee\n"},{"id":"404539","messageId":"xmqq4kopwdjb.fsf@gitster.c.googlers.com","threadId":"54070","inReplyTo":"1e107b2e-364a-4c17-90b8-0371e8e27878@gmail.com","subject":"Re: [PATCH v2 1/7] maintenance: optionally skip --auto process","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-08-26T16:57:28Z","receivedAt":"2020-08-26T16:57:35Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n> Before the maintenance builtin, it would be natural to see \"gc.auto=0\"\n> and do this same behavior, but that will not be general enough in the\n> future.\n\nI do think it is an excellent change to move the check done in\nthe need_to_gc() to check the value of gc.auto to run_auto_gc()\nfor exactly the reason the log message of this patch gives. And\nafter the function is renamed to run_auto_maintenance(), at some\npoint the variable that gets checked would also be updated, and\nwe'd eventually reach the same state, I would think.\n\nBut it is so small a change that it probably is not worth the\nbook-keeping burden of remembering that the maintenance topic needs\nto build on the patch to update auto-gc. \n\n> If you prefer, I can pull this out into a series on its own to be\n> tracked separately.\n\nSo let's leave it as-is.\n\nThanks.\n"},{"id":"404540","messageId":"xmqqzh6huyoi.fsf@gitster.c.googlers.com","threadId":"54070","inReplyTo":"4fe41132-c8f5-6b85-414f-5a1e5fcccdc9@gmail.com","subject":"Re: [PATCH v2 2/7] maintenance: store the \"last run\" time in config","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-08-26T17:03:41Z","receivedAt":"2020-08-26T17:03:52Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n>>> 1. The time the task takes to execute should not contribute to the\n>>> interval between running the tasks.\n>> \n>> ... as long as the run time is sufficiently shorter than the\n>> interval, that is. If a task takes 10-30 minutes depending on how\n>> dirty the repository is, it does not make sense to even try to run\n>> it every 15 minutes.\n>\n> Definitely. The lock on the object database from earlier prevents these\n> longer-than-anticipated tasks from stacking.\n\nHmph, I actually was (anticipating|hoping) that you would give a\ngood argument for having maintenance subsystem in change of\nscheduling rather than cron, as it can monitor how the already\nrunning job is goind and skip one cycle if needed. The above is\ninstead a good argument that independent cron jobs can still\ncoordinate and there is no central and custom scheduler in the form\nof 'maintenance run'.\n\n>>> 2. If the task fails for some unforseen reason, it would be good to\n>>> indicate that we _attempted_ the task at a certain timestamp. This\n>>> will avoid spamming a repository that is in a bad state.\n>> \n>> Absolutely.\n\nSomebody already mentioned that using the configuration file for\nrecordkeeping may not be a good idea, and I tend to agree, by the\nway. I may want to periodically take a snapshot of my configuration\nto notice and remember changes I made myself intentionally\n(e.g. switched access method of a hosting site from ssh:// to\nhttps://, added a new branch that builds on something else, etc.) by\ncomparing the snapshot with previous ones (and might even put it\nunder version-control) and mechanical noise would interfere with it.\n\n"},{"id":"404606","messageId":"c30d745f-4d30-0404-420f-2009870da95b@gmail.com","threadId":"54070","inReplyTo":"xmqqzh6huyoi.fsf@gitster.c.googlers.com","subject":"Re: [PATCH v2 2/7] maintenance: store the \"last run\" time in config","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-08-27T13:02:57Z","receivedAt":"2020-08-27T14:54:31Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 8/26/2020 1:03 PM, Junio C Hamano wrote:\n> Derrick Stolee writes:\n> \n>>>> 1. The time the task takes to execute should not contribute to the\n>>>> interval between running the tasks.\n>>>\n>>> ... as long as the run time is sufficiently shorter than the\n>>> interval, that is. If a task takes 10-30 minutes depending on how\n>>> dirty the repository is, it does not make sense to even try to run\n>>> it every 15 minutes.\n>>\n>> Definitely. The lock on the object database from earlier prevents these\n>> longer-than-anticipated tasks from stacking.\n> \n> Hmph, I actually was (anticipating|hoping) that you would give a\n> good argument for having maintenance subsystem in change of\n> scheduling rather than cron, as it can monitor how the already\n> running job is goind and skip one cycle if needed. The above is\n> instead a good argument that independent cron jobs can still\n> coordinate and there is no central and custom scheduler in the form\n> of 'maintenance run'.\n\nWhile the lock does prevent concurrent 'maintenance run' commands\nfrom colliding and causing unpredictable behavior as they both\nmodify the object database, this does not help ensure that maintenance\ntasks actually happen if certain tasks are fired independently by\ncron and consistently collide.\n\nThis is the main motivation for me using a single crontab entry.\nMore discussion of all of the tradeoffs is in [1].\n\n[1] https://lore.kernel.org/git/bd4e18b7-6265-73e7-bc1a-a7d647eafd0a@gmail.com/\n\n>>>> 2. If the task fails for some unforseen reason, it would be good to\n>>>> indicate that we _attempted_ the task at a certain timestamp. This\n>>>> will avoid spamming a repository that is in a bad state.\n>>>\n>>> Absolutely.\n> \n> Somebody already mentioned that using the configuration file for\n> recordkeeping may not be a good idea, and I tend to agree, by the\n> way. I may want to periodically take a snapshot of my configuration\n> to notice and remember changes I made myself intentionally\n> (e.g. switched access method of a hosting site from ssh:// to\n> https://, added a new branch that builds on something else, etc.) by\n> comparing the snapshot with previous ones (and might even put it\n> under version-control) and mechanical noise would interfere with it.\n \nI will think of another way to handle this, then. If we cannot infer\nthat \"this task was launched, therefore it is due to run\" from an\noptimal cron schedule, then I'll probably create a new file in the\n.git repository that stores these values. That file would be in the\nconfig format to make parsing easy.\n\nThanks,\n-Stolee\n"},{"id":"404620","messageId":"4666c635-b808-a0a8-ab3a-0a07e222827b@gmail.com","threadId":"54070","inReplyTo":"bd4e18b7-6265-73e7-bc1a-a7d647eafd0a@gmail.com","subject":"Re: [PATCH v2 3/7] maintenance: add --scheduled option and config","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-08-27T15:47:03Z","receivedAt":"2020-08-27T15:47:07Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 8/26/2020 11:30 AM, Derrick Stolee wrote:\n> Let users specify a schedule frequency among this list: hourly, daily,\n> weekly, monthly. We then set the following* crontab:\n> \n> \t0 * * * * git for-each-repo --config=maintenance.repos maintenance run --scheduled=hourly\n> \t0 0 * * * git for-each-repo --config=maintenance.repos maintenance run --scheduled=daily\n> \t0 0 * * 0 git for-each-repo --config=maintenance.repos maintenance run --scheduled=weekly\n> \t0 0 0 * * git for-each-repo --config=maintenance.repos maintenance run --scheduled=monthly\n> \n> *Of course, there is some care around \"$path/git --exec-path=$path\"\n> that I drop for ease here.\n\nJeff Hostetler pointed out the following details in the crontab\ndocumentation [1]:\n\n Ranges of numbers are allowed. Ranges are two numbers separated with\n a hyphen. The specified range is inclusive. For example, 8-11 for\n an 'hours' entry specifies execution at hours 8, 9, 10, and 11. The\n first number must be less than or equal to the second one.\n\n[1] https://man7.org/linux/man-pages/man5/crontab.5.html\n\nThis means we could try this schedule:\n\n 0 1-23 * * * git for-each-repo --config=maintenance.repos maintenance run --scheduled=hourly\n 0 0 * * 1-6 git for-each-repo --config=maintenance.repos maintenance run --scheduled=daily\n 0 0 1-30 * 0 git for-each-repo --config=maintenance.repos maintenance run --scheduled=weekly\n 0 0 0 * * git for-each-repo --config=maintenance.repos maintenance run --scheduled=monthly\n\nAnd it should behave this way:\n\n Run --scheduled=hourly every hour, except at midnight. This runs\n all \"hourly\" tasks.\n\n Run --scheduled=daily at midnight, except on Sunday. This runs all\n \"hourly\" and \"daily\" tasks.\n\n Run --scheduled=weekly at midnight Sunday, except on the first day\n of the month. This runs all \"hourly\", \"daily\", and \"weekly\" tasks.\n\n Run --scheduled=monthly at midnight on the first day of the month.\n This runs all scheduled tasks.\n\nThere is some subtlety between whether the \"weekly\" runs should be a\nsubset of \"monthly\" and maybe the easiest way to handle that would\nbe to not support \"monthly\" and have only \"hourly\", \"daily\", and \"weekly\"\noptions for now.\n\nThis should get around all of the parallel issues and allow us to drop\nthe *.lastRun config option.\n\nThoughts?\n\nThanks,\n-Stolee\n"},{"id":"404703","messageId":"pull.680.v3.git.1598629517.gitgitgadget@gmail.com","threadId":"54070","inReplyTo":"pull.680.v2.git.1598380805.gitgitgadget@gmail.com","subject":"[PATCH v3 0/6] [RFC] Maintenance III: background maintenance","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-08-28T15:45:11Z","receivedAt":"2020-08-28T15:45:41Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"This is based on v3 of Part II (ds/maintenance-part-2) [1].\n\n[1] \nhttps://lore.kernel.org/git/pull.696.v3.git.1598380599.gitgitgadget@gmail.com/\n\nThis RFC is intended to show how I hope to integrate true background\nmaintenance into Git. As opposed to my original RFC [2], this entirely\nintegrates with cron (through crontab [-e|-l]) to launch maintenance\ncommands in the background.\n\n[2] \nhttps://lore.kernel.org/git/pull.597.git.1585946894.gitgitgadget@gmail.com/\n\nSome preliminary work is done to allow a new --schedule option that tells\nthe command which tasks to run based on a maintenance..schedule config\noption. The timing is not enforced by Git, but instead is expected to be\nprovided as a hint from a cron schedule.\n\nA new for-each-repo builtin runs Git commands on every repo in a given list.\nCurrently, the list is stored as a config setting, allowing a new \nmaintenance.repos config list to store the repositories registered for\nbackground maintenance. Others may want to add a --file= option for\ntheir own workflows, but I focused on making this as simple as possible for\nnow.\n\nThe updates to the git maintenance builtin include new register/unregister \nsubcommands and start/stop subcommands. The register subcommand initializes\nthe config while the start subcommand does everything register does plus \nupdate the cron table. The unregister and stop commands reverse this\nprocess.\n\nThe very last patch is entirely optional. It sets a recommended schedule\nbased on my own experience with very large repositories. I'm open to other\nsuggestions, but these are ones that I think work well and don't cause a\n\"rewrite the world\" scenario like running nightly 'gc' would do.\n\nI've been testing this scenario on my macOS laptop for a while and my Linux\nmachine. I have modified my cron task to provide logging via trace2 so I can\nsee what's happening. A future direction here would be to add some\nmaintenance logs to the repository so we can track what is happening and\ndiagnose whether the maintenance strategy is working on real repos.\n\nNote: git maintenance (start|stop) only works on machines with cron by\ndesign. The proper thing to do on Windows will come later. Perhaps this\ncommand should be marked as unavailable on Windows somehow, or at least a\nbetter error than \"cron may not be available on your system\". I did find\nthat that message is helpful sometimes: macOS worker agents for CI builds\ntypically do not have cron available.\n\nUpdates since RFC v2\n====================\n\n * Update the cron schedule with three lines saying \"run hourly except at\n midnight\", \"run daily except on first day of week\", and \"run weekly\".\n This avoids parallel processes competing for the object database lock.\n \n \n * Update the --schedule= and 'maintenance..schedule' config options. This\n is reflected in the recommended schedule at the end.\n \n \n * Drop the *.lastRun config option. It was going to trash config files but\n it is also not needed by the new cron schedule.\n \n \n\nI expect this to be my final RFC version before restarting the thread with a\nv1 next week. Please throw any and all critique at the plan here!\n\nUpdates since RFC v1\n====================\n\n * Some fallout from rewriting the option parsing in \"Maintenance I\"\n \n \n * This applies cleanly on v3 of \"Maintenance II\"\n \n \n * Several helpful feedback items from Đoàn Trần Công Danh are applied.\n \n \n * There is an unresolved comment around the use of approxidate(\"now\").\n These calls are untouched from v1.\n \n \n\nThanks, -Stolee\n\nCc: sandals@crustytoothpaste.net [sandals@crustytoothpaste.net], \nsteadmon@google.com [steadmon@google.com], jrnieder@gmail.com\n[jrnieder@gmail.com], peff@peff.net [peff@peff.net], congdanhqx@gmail.com\n[congdanhqx@gmail.com], phillip.wood123@gmail.com\n[phillip.wood123@gmail.com], emilyshaffer@google.com\n[emilyshaffer@google.com], sluongng@gmail.com [sluongng@gmail.com], \njonathantanmy@google.com [jonathantanmy@google.com]\n\nDerrick Stolee (6):\n maintenance: optionally skip --auto process\n maintenance: add --schedule option and config\n for-each-repo: run subcommands on configured repos\n maintenance: add [un]register subcommands\n maintenance: add start/stop subcommands\n maintenance: recommended schedule in register/start\n\n .gitignore | 1 +\n Documentation/config/maintenance.txt | 10 +\n Documentation/git-for-each-repo.txt | 59 ++++++\n Documentation/git-maintenance.txt | 44 +++-\n Makefile | 2 +\n builtin.h | 1 +\n builtin/for-each-repo.c | 58 ++++++\n builtin/gc.c | 292 ++++++++++++++++++++++++++-\n command-list.txt | 1 +\n git.c | 1 +\n run-command.c | 6 +\n t/helper/test-crontab.c | 35 ++++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/t0068-for-each-repo.sh | 30 +++\n t/t7900-maintenance.sh | 114 ++++++++++-\n t/test-lib.sh | 6 +\n 17 files changed, 654 insertions(+), 8 deletions(-)\n create mode 100644 Documentation/git-for-each-repo.txt\n create mode 100644 builtin/for-each-repo.c\n create mode 100644 t/helper/test-crontab.c\n create mode 100755 t/t0068-for-each-repo.sh\n\n\nbase-commit: e9bb32f53ade2067f773bfe6e5c13ed1a5d694a6\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-680%2Fderrickstolee%2Fmaintenance%2Fscheduled-v3\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-680/derrickstolee/maintenance/scheduled-v3\nPull-Request: https://github.com/gitgitgadget/git/pull/680\n\nRange-diff vs v2:\n\n 1: 5fdd8188b1 = 1: 5fdd8188b1 maintenance: optionally skip --auto process\n 2: e3ef0b9bea < -: ---------- maintenance: store the \"last run\" time in config\n 3: c728c57d85 ! 2: 41a067894d maintenance: add --scheduled option and config\n @@ Metadata\n Author: Derrick Stolee \n \n ## Commit message ##\n - maintenance: add --scheduled option and config\n + maintenance: add --schedule option and config\n \n A user may want to run certain maintenance tasks based on frequency, not\n conditions given in the repository. For example, the user may want to\n perform a 'prefetch' task every hour, or 'gc' task every day. To assist,\n - update the 'git maintenance run --scheduled' command to check the config\n - for the last run of that task and add a number of seconds. The task\n - would then run only if the current time is beyond that minimum\n - timestamp.\n + update the 'git maintenance run' command to include a\n + '--schedule=' option. The allowed frequencies are 'hourly',\n + 'daily', and 'weekly'. These values are also allowed in a new config\n + value 'maintenance..schedule'.\n \n - Add a '--scheduled' option to 'git maintenance run' to only run tasks\n - that have had enough time pass since their last run. This is done for\n - each enabled task by checking if the current timestamp is at least as\n - large as the sum of 'maintenance..lastRun' and\n - 'maintenance..schedule' in the Git config. This second value is\n - new to this commit, storing a number of seconds intended between runs.\n + The 'git maintenance run --schedule=' checks the '*.schedule'\n + config value for each enabled task to see if the configured frequency is\n + at least as frequent as the frequency from the '--schedule' argument. We\n + use the following order, for full clarity:\n \n - A user could then set up an hourly maintenance run with the following\n - cron table:\n + 'hourly' > 'daily' > 'weekly'\n \n - 0 * * * * git -C maintenance run --scheduled\n + Use new 'enum schedule_priority' to track these values numerically.\n \n - Then, the user could configure the repository with the following config\n - values:\n + The following cron table would run the scheduled tasks with the correct\n + frequencies:\n \n - maintenance.prefetch.schedule 3000\n - maintenance.gc.schedule 86000\n + 0 1-23 * * * git -C maintenance run --scheduled=hourly\n + 0 0 * * 1-6 git -C maintenance run --scheduled=daily\n + 0 0 * * 0 git -C maintenance run --scheduled=weekly\n \n - These numbers are slightly lower than one hour and one day (in seconds).\n - The cron schedule will enforce the hourly run rate, but we can use these\n - schedules to ensure the 'gc' task runs once a day. The error is given\n - because the *.lastRun config option is specified at the _start_ of the\n - task run. Otherwise, a slow task run could shift the \"daily\" job of 'gc'\n - from a 10:00pm run to 11:00pm run, or later.\n + This cron schedule will run --scheduled=hourly every hour except at\n + midnight. This avoids a concurrent run with the --scheduled=daily that\n + runs at midnight every day except the first day of the week. This avoids\n + a concurrent run with the --scheduled=weekly that runs at midnight on\n + the first day of the week. Since --scheduled=daily also runs the\n + 'hourly' tasks and --scheduled=weekly runs the 'hourly' and 'daily'\n + tasks, we will still see all tasks run with the proper frequencies.\n \n Signed-off-by: Derrick Stolee \n \n ## Documentation/config/maintenance.txt ##\n -@@ Documentation/config/maintenance.txt: maintenance..lastRun::\n - \t`` is run. It stores a timestamp representing the most-recent\n - \trun of the ``.\n +@@ Documentation/config/maintenance.txt: maintenance..enabled::\n + \t`--task` option exists. By default, only `maintenance.gc.enabled`\n + \tis true.\n \n +maintenance..schedule::\n +\tThis config option controls whether or not the given `` runs\n -+\tduring a `git maintenance run --scheduled` command. If the option\n -+\tis an integer value `S`, then the `` is run when the current\n -+\ttime is `S` seconds after the timestamp stored in\n -+\t`maintenance..lastRun`. If the option has no value or a\n -+\tnon-integer value, then the task will never run with the `--scheduled`\n -+\toption.\n ++\tduring a `git maintenance run --schedule=` command. The\n ++\tvalue must be one of \"hourly\", \"daily\", or \"weekly\".\n +\n maintenance.commit-graph.auto::\n \tThis integer config option controls how often the `commit-graph` task\n @@ Documentation/git-maintenance.txt: OPTIONS\n \tin the `gc.auto` config setting, or when the number of pack-files\n -\texceeds the `gc.autoPackLimit` config setting.\n +\texceeds the `gc.autoPackLimit` config setting. Not compatible with\n -+\tthe `--scheduled` option.\n ++\tthe `--schedule` option.\n +\n -+--scheduled::\n ++--schedule::\n +\tWhen combined with the `run` subcommand, run maintenance tasks\n +\tonly if certain time conditions are met, as specified by the\n +\t`maintenance..schedule` config value for each ``.\n @@ Documentation/git-maintenance.txt: OPTIONS\n \n ## builtin/gc.c ##\n @@ builtin/gc.c: int cmd_gc(int argc, const char **argv, const char *prefix)\n + \treturn 0;\n }\n \n - static const char * const builtin_maintenance_run_usage[] = {\n +-static const char * const builtin_maintenance_run_usage[] = {\n -\tN_(\"git maintenance run [--auto] [--[no-]quiet] [--task=]\"),\n -+\tN_(\"git maintenance run [--auto] [--[no-]quiet] [--task=] [--scheduled]\"),\n ++static const char *const builtin_maintenance_run_usage[] = {\n ++\tN_(\"git maintenance run [--auto] [--[no-]quiet] [--task=] [--schedule]\"),\n \tNULL\n };\n \n ++enum schedule_priority {\n ++\tSCHEDULE_NONE = 0,\n ++\tSCHEDULE_WEEKLY = 1,\n ++\tSCHEDULE_DAILY = 2,\n ++\tSCHEDULE_HOURLY = 3,\n ++};\n ++\n ++static enum schedule_priority parse_schedule(const char *value)\n ++{\n ++\tif (!value)\n ++\t\treturn SCHEDULE_NONE;\n ++\tif (!strcasecmp(value, \"hourly\"))\n ++\t\treturn SCHEDULE_HOURLY;\n ++\tif (!strcasecmp(value, \"daily\"))\n ++\t\treturn SCHEDULE_DAILY;\n ++\tif (!strcasecmp(value, \"weekly\"))\n ++\t\treturn SCHEDULE_WEEKLY;\n ++\treturn SCHEDULE_NONE;\n ++}\n ++\n ++static int maintenance_opt_schedule(const struct option *opt, const char *arg,\n ++\t\t\t\t int unset)\n ++{\n ++\tenum schedule_priority *priority = opt->value;\n ++\n ++\tif (unset)\n ++\t\tdie(_(\"--no-schedule is not allowed\"));\n ++\n ++\t*priority = parse_schedule(arg);\n ++\n ++\tif (!*priority)\n ++\t\tdie(_(\"unrecognized --schedule argument '%s'\"), arg);\n ++\n ++\treturn 0;\n ++}\n ++\n struct maintenance_run_opts {\n \tint auto_flag;\n -+\tint scheduled;\n \tint quiet;\n ++\tenum schedule_priority schedule;\n };\n \n + /* Remember to update object flag allocation in object.h */\n @@ builtin/gc.c: struct maintenance_task {\n - \tconst char *name;\n - \tmaintenance_task_fn *fn;\n \tmaintenance_auto_fn *auto_condition;\n --\tunsigned enabled:1;\n -+\tunsigned enabled:1,\n -+\t\t scheduled:1;\n + \tunsigned enabled:1;\n \n ++\tenum schedule_priority schedule;\n ++\n \t/* -1 if not selected. */\n \tint selected_order;\n + };\n @@ builtin/gc.c: static int maintenance_run_tasks(struct maintenance_run_opts *opts)\n - \t\t !tasks[i].auto_condition()))\n \t\t\tcontinue;\n \n -+\t\tif (opts->scheduled && !tasks[i].scheduled)\n + \t\tif (opts->auto_flag &&\n +-\t\t (!tasks[i].auto_condition ||\n +-\t\t !tasks[i].auto_condition()))\n ++\t\t (!tasks[i].auto_condition || !tasks[i].auto_condition()))\n +\t\t\tcontinue;\n +\n - \t\tupdate_last_run(&tasks[i]);\n ++\t\tif (opts->schedule && tasks[i].schedule < opts->schedule)\n + \t\t\tcontinue;\n \n \t\ttrace2_region_enter(\"maintenance\", tasks[i].name, r);\n -@@ builtin/gc.c: static int maintenance_run_tasks(struct maintenance_run_opts *opts)\n - \treturn result;\n - }\n - \n -+static void fill_schedule_info(struct maintenance_task *task,\n -+\t\t\t const char *config_name,\n -+\t\t\t timestamp_t schedule_delay)\n -+{\n -+\ttimestamp_t now = approxidate(\"now\");\n -+\tchar *value = NULL;\n -+\tstruct strbuf last_run = STRBUF_INIT;\n -+\tint64_t previous_run;\n -+\n -+\tstrbuf_addf(&last_run, \"maintenance.%s.lastrun\", task->name);\n -+\n -+\tif (git_config_get_string(last_run.buf, &value))\n -+\t\ttask->scheduled = 1;\n -+\telse {\n -+\t\tprevious_run = git_config_int64(last_run.buf, value);\n -+\t\tif (now >= previous_run + schedule_delay)\n -+\t\t\ttask->scheduled = 1;\n -+\t}\n -+\n -+\tfree(value);\n -+\tstrbuf_release(&last_run);\n -+}\n -+\n - static void initialize_task_config(void)\n - {\n - \tint i;\n @@ builtin/gc.c: static void initialize_task_config(void)\n \n \tfor (i = 0; i < TASK__COUNT; i++) {\n @@ builtin/gc.c: static void initialize_task_config(void)\n +\t\t\t tasks[i].name);\n +\n +\t\tif (!git_config_get_string(config_name.buf, &config_str)) {\n -+\t\t\ttimestamp_t schedule_delay = git_config_int64(\n -+\t\t\t\t\t\t\tconfig_name.buf,\n -+\t\t\t\t\t\t\tconfig_str);\n -+\t\t\tfill_schedule_info(&tasks[i],\n -+\t\t\t\t\t\tconfig_name.buf,\n -+\t\t\t\t\t\tschedule_delay);\n ++\t\t\ttasks[i].schedule = parse_schedule(config_str);\n +\t\t\tfree(config_str);\n +\t\t}\n \t}\n @@ builtin/gc.c: static int maintenance_run(int argc, const char **argv, const char\n \tstruct option builtin_maintenance_run_options[] = {\n \t\tOPT_BOOL(0, \"auto\", &opts.auto_flag,\n \t\t\t N_(\"run tasks based on the state of the repository\")),\n -+\t\tOPT_BOOL(0, \"scheduled\", &opts.scheduled,\n -+\t\t\t N_(\"run tasks based on time intervals\")),\n ++\t\tOPT_CALLBACK(0, \"schedule\", &opts.schedule, N_(\"frequency\"),\n ++\t\t\t N_(\"run tasks based on frequency\"),\n ++\t\t\t maintenance_opt_schedule),\n \t\tOPT_BOOL(0, \"quiet\", &opts.quiet,\n \t\t\t N_(\"do not report progress or other information over stderr\")),\n \t\tOPT_CALLBACK_F(0, \"task\", NULL, N_(\"task\"),\n @@ builtin/gc.c: static int maintenance_run(int argc, const char **argv, const char\n \t\t\t builtin_maintenance_run_usage,\n \t\t\t PARSE_OPT_STOP_AT_NON_OPTION);\n \n -+\tif (opts.auto_flag + opts.scheduled > 1)\n -+\t\tdie(_(\"use at most one of the --auto and --scheduled options\"));\n ++\tif (opts.auto_flag && opts.schedule)\n ++\t\tdie(_(\"use at most one of --auto and --schedule=\"));\n +\n \tif (argc != 0)\n \t\tusage_with_options(builtin_maintenance_run_usage,\n @@ t/t7900-maintenance.sh: test_expect_success 'maintenance.incremental-repack.auto\n \tdone\n '\n \n -+test_expect_success '--auto and --scheduled incompatible' '\n -+\ttest_must_fail git maintenance run --auto --scheduled 2>err &&\n ++test_expect_success '--auto and --schedule incompatible' '\n ++\ttest_must_fail git maintenance run --auto --schedule=daily 2>err &&\n +\ttest_i18ngrep \"at most one\" err\n +'\n +\n - test_expect_success 'tasks update maintenance..lastRun' '\n - \tgit config --unset maintenance.commit-graph.lastrun &&\n - \tGIT_TRACE2_EVENT=\"$(pwd)/run.txt\" \\\n -@@ t/t7900-maintenance.sh: test_expect_success 'tasks update maintenance..lastRun' '\n - \ttest_cmp_config 1595000000 maintenance.commit-graph.lastrun\n - '\n - \n -+test_expect_success '--scheduled with specific time' '\n -+\tgit config maintenance.commit-graph.schedule 100 &&\n -+\tGIT_TRACE2_EVENT=\"$(pwd)/too-soon.txt\" \\\n -+\t\tGIT_TEST_DATE_NOW=1595000099 \\\n -+\t\tgit maintenance run --scheduled 2>/dev/null &&\n ++test_expect_success 'invalid --schedule value' '\n ++\ttest_must_fail git maintenance run --schedule=annually 2>err &&\n ++\ttest_i18ngrep \"unrecognized --schedule\" err\n ++'\n ++\n ++test_expect_success '--schedule inheritance weekly -> daily -> hourly' '\n ++\tgit config maintenance.loose-objects.enabled true &&\n ++\tgit config maintenance.loose-objects.schedule hourly &&\n ++\tgit config maintenance.commit-graph.enabled true &&\n ++\tgit config maintenance.commit-graph.schedule daily &&\n ++\tgit config maintenance.incremental-repack.enabled true &&\n ++\tgit config maintenance.incremental-repack.schedule weekly &&\n ++\n ++\tGIT_TRACE2_EVENT=\"$(pwd)/hourly.txt\" \\\n ++\t\tgit maintenance run --schedule=hourly 2>/dev/null &&\n ++\ttest_subcommand git prune-packed --quiet /dev/null &&\n ++\t\t--no-progress /dev/null &&\n ++\ttest_subcommand git prune-packed --quiet /dev/null &&\n ++\ttest_subcommand git prune-packed --quiet err &&\n \ttest_i18ngrep \"invalid subcommand: barf\" err\n '\n -@@ t/t7900-maintenance.sh: test_expect_success '--scheduled with specific time' '\n - \ttest_cmp_config 1595000100 maintenance.commit-graph.lastrun\n +@@ t/t7900-maintenance.sh: test_expect_success '--schedule inheritance weekly -> daily -> hourly' '\n + \ttest_subcommand git multi-pack-index write --no-progress argument will run all tasks with _at least_\n + the given frequency, the daily runs will also run the hourly tasks.\n + Similarly, the weekly runs will also run the daily and hourly tasks.\n +\n The GIT_TEST_CRONTAB environment variable is not intended for users to\n edit, but instead as a way to mock the 'crontab [-l]' command. This\n variable is set in test-lib.sh to avoid a future test from accidentally\n @@ builtin/gc.c: static int maintenance_unregister(void)\n +\t}\n +\n +\tif (run_maintenance) {\n ++\t\tstruct strbuf line_format = STRBUF_INIT;\n +\t\tconst char *exec_path = git_exec_path();\n +\n -+\t\tfprintf(cron_in, \"\\n%s\\n\", BEGIN_LINE);\n -+\t\tfprintf(cron_in, \"# The following schedule was created by Git\\n\");\n ++\t\tfprintf(cron_in, \"%s\\n\", BEGIN_LINE);\n ++\t\tfprintf(cron_in,\n ++\t\t\t\"# The following schedule was created by Git\\n\");\n +\t\tfprintf(cron_in, \"# Any edits made in this region might be\\n\");\n -+\t\tfprintf(cron_in, \"# replaced in the future by a Git command.\\n\\n\");\n -+\n +\t\tfprintf(cron_in,\n -+\t\t\t\"0 * * * * \\\"%s/git\\\" --exec-path=\\\"%s\\\" for-each-repo --config=maintenance.repo maintenance run --scheduled\\n\",\n -+\t\t\texec_path, exec_path);\n ++\t\t\t\"# replaced in the future by a Git command.\\n\\n\");\n ++\n ++\t\tstrbuf_addf(&line_format,\n ++\t\t\t \"%%s %%s * * %%s \\\"%s/git\\\" --exec-path=\\\"%s\\\" for-each-repo --config=maintenance.repo maintenance run --schedule=%%s\\n\",\n ++\t\t\t exec_path, exec_path);\n ++\t\tfprintf(cron_in, line_format.buf, \"0\", \"1-23\", \"*\", \"hourly\");\n ++\t\tfprintf(cron_in, line_format.buf, \"0\", \"0\", \"1-6\", \"daily\");\n ++\t\tfprintf(cron_in, line_format.buf, \"0\", \"0\", \"0\", \"weekly\");\n ++\t\tstrbuf_release(&line_format);\n +\n +\t\tfprintf(cron_in, \"\\n%s\\n\", END_LINE);\n +\t}\n @@ t/t7900-maintenance.sh: test_expect_success 'register and unregister' '\n +\t# start registers the repo\n +\tgit config --get --global maintenance.repo \"$(pwd)\" &&\n +\n -+\tgrep \"for-each-repo --config=maintenance.repo maintenance run --scheduled\" cron.txt\n ++\tgrep \"for-each-repo --config=maintenance.repo maintenance run --schedule=daily\" cron.txt &&\n ++\tgrep \"for-each-repo --config=maintenance.repo maintenance run --schedule=hourly\" cron.txt &&\n ++\tgrep \"for-each-repo --config=maintenance.repo maintenance run --schedule=weekly\" cron.txt\n +'\n +\n +test_expect_success 'stop from existing schedule' '\n @@ t/t7900-maintenance.sh: test_expect_success 'register and unregister' '\n +\t# stop does not unregister the repo\n +\tgit config --get --global maintenance.repo \"$(pwd)\" &&\n +\n -+\t# The newline is preserved\n -+\techo >empty &&\n -+\ttest_cmp empty cron.txt &&\n -+\n +\t# Operation is idempotent\n +\tGIT_TEST_CRONTAB=\"test-tool crontab cron.txt\" git maintenance stop &&\n -+\ttest_cmp empty cron.txt\n ++\ttest_must_be_empty cron.txt\n +'\n +\n +test_expect_success 'start preserves existing schedule' '\n 7: 9ecabeb055 ! 6: 62e8db8b2a maintenance: recommended schedule in register/start\n @@ Commit message\n repository. It does not specify what maintenance should occur or how\n often.\n \n - If a user sets any 'maintenance..scheduled' config value, then\n + If a user sets any 'maintenance..schedule' config value, then\n they have chosen a specific schedule for themselves and Git should\n respect that.\n \n @@ Commit message\n schedule we use in Scalar and VFS for Git for very large repositories\n using the GVFS protocol. While the schedule works in that environment,\n it is possible that \"normal\" Git repositories could benefit from\n - something more obvious (such as running 'gc' once a day). However, this\n + something more obvious (such as running 'gc' weekly). However, this\n patch gives us a place to start a conversation on what we should\n recommend. For my purposes, Scalar will set these config values so we\n can always differ from core Git's recommendations.\n @@ builtin/gc.c: static int maintenance_run(int argc, const char **argv, const char\n +\tprefix = config_name.len;\n +\n +\tfor (i = 0; !found && i < TASK__COUNT; i++) {\n -+\t\tint value;\n ++\t\tchar *value;\n +\n +\t\tstrbuf_setlen(&config_name, prefix);\n +\t\tstrbuf_addf(&config_name, \"%s.schedule\", tasks[i].name);\n +\n -+\t\tif (!git_config_get_int(config_name.buf, &value))\n ++\t\tif (!git_config_get_string(config_name.buf, &value)) {\n +\t\t\tfound = 1;\n ++\t\t\tFREE_AND_NULL(value);\n ++\t\t}\n +\t}\n +\n +\tstrbuf_release(&config_name);\n @@ builtin/gc.c: static int maintenance_run(int argc, const char **argv, const char\n +\tgit_config_set(\"maintenance.gc.enabled\", \"false\");\n +\n +\tgit_config_set(\"maintenance.prefetch.enabled\", \"true\");\n -+\tgit_config_set(\"maintenance.prefetch.schedule\", \"3500\");\n ++\tgit_config_set(\"maintenance.prefetch.schedule\", \"hourly\");\n +\n +\tgit_config_set(\"maintenance.commit-graph.enabled\", \"true\");\n -+\tgit_config_set(\"maintenance.commit-graph.schedule\", \"3500\");\n ++\tgit_config_set(\"maintenance.commit-graph.schedule\", \"hourly\");\n +\n +\tgit_config_set(\"maintenance.loose-objects.enabled\", \"true\");\n -+\tgit_config_set(\"maintenance.loose-objects.schedule\", \"86000\");\n ++\tgit_config_set(\"maintenance.loose-objects.schedule\", \"daily\");\n +\n +\tgit_config_set(\"maintenance.incremental-repack.enabled\", \"true\");\n -+\tgit_config_set(\"maintenance.incremental-repack.schedule\", \"86000\");\n ++\tgit_config_set(\"maintenance.incremental-repack.schedule\", \"daily\");\n +}\n +\n static int maintenance_register(void)\n @@ builtin/gc.c: static int maintenance_register(void)\n \tif (!the_repository || !the_repository->gitdir)\n \t\treturn 0;\n \n -+\tif (has_schedule_config())\n ++\tif (!has_schedule_config())\n +\t\tset_recommended_schedule();\n +\n \tconfig_get.git_cmd = 1;\n @@ builtin/gc.c: static int maintenance_register(void)\n \n ## t/t7900-maintenance.sh ##\n @@ t/t7900-maintenance.sh: test_expect_success 'register and unregister' '\n + \tgit config --global --add maintenance.repo /existing1 &&\n \tgit config --global --add maintenance.repo /existing2 &&\n \tgit config --global --get-all maintenance.repo >before &&\n ++\n ++\t# We still have maintenance..schedule config set,\n ++\t# so this does not update the local schedule\n ++\tgit maintenance register &&\n ++\ttest_must_fail git config maintenance.auto &&\n ++\n ++\t# Clear previous maintenance..schedule values\n ++\tfor task in loose-objects commit-graph incremental-repack\n ++\tdo\n ++\t\tgit config --unset maintenance.$task.schedule || return 1\n ++\tdone &&\n \tgit maintenance register &&\n +\ttest_cmp_config false maintenance.auto &&\n +\ttest_cmp_config false maintenance.gc.enabled &&\n +\ttest_cmp_config true maintenance.prefetch.enabled &&\n -+\ttest_cmp_config 3500 maintenance.commit-graph.schedule &&\n -+\ttest_cmp_config 86000 maintenance.incremental-repack.schedule &&\n ++\ttest_cmp_config hourly maintenance.commit-graph.schedule &&\n ++\ttest_cmp_config daily maintenance.incremental-repack.schedule &&\n \tgit config --global --get-all maintenance.repo >actual &&\n \tcp before after &&\n \tpwd >>after &&\n\n-- \ngitgitgadget\n"},{"id":"404704","messageId":"41a067894d6fdcd966e4c376e9033d323af98ea7.1598629517.git.gitgitgadget@gmail.com","threadId":"54070","inReplyTo":"pull.680.v3.git.1598629517.gitgitgadget@gmail.com","subject":"[PATCH v3 2/6] maintenance: add --schedule option and config","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-08-28T15:45:13Z","receivedAt":"2020-08-28T15:45:42Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nA user may want to run certain maintenance tasks based on frequency, not\nconditions given in the repository. For example, the user may want to\nperform a 'prefetch' task every hour, or 'gc' task every day. To assist,\nupdate the 'git maintenance run' command to include a\n'--schedule=' option. The allowed frequencies are 'hourly',\n'daily', and 'weekly'. These values are also allowed in a new config\nvalue 'maintenance..schedule'.\n\nThe 'git maintenance run --schedule=' checks the '*.schedule'\nconfig value for each enabled task to see if the configured frequency is\nat least as frequent as the frequency from the '--schedule' argument. We\nuse the following order, for full clarity:\n\n\t'hourly' > 'daily' > 'weekly'\n\nUse new 'enum schedule_priority' to track these values numerically.\n\nThe following cron table would run the scheduled tasks with the correct\nfrequencies:\n\n 0 1-23 * * * git -C maintenance run --scheduled=hourly\n 0 0 * * 1-6 git -C maintenance run --scheduled=daily\n 0 0 * * 0 git -C maintenance run --scheduled=weekly\n\nThis cron schedule will run --scheduled=hourly every hour except at\nmidnight. This avoids a concurrent run with the --scheduled=daily that\nruns at midnight every day except the first day of the week. This avoids\na concurrent run with the --scheduled=weekly that runs at midnight on\nthe first day of the week. Since --scheduled=daily also runs the\n'hourly' tasks and --scheduled=weekly runs the 'hourly' and 'daily'\ntasks, we will still see all tasks run with the proper frequencies.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/config/maintenance.txt | 5 +++\n Documentation/git-maintenance.txt | 13 +++++-\n builtin/gc.c | 67 +++++++++++++++++++++++++---\n t/t7900-maintenance.sh | 40 +++++++++++++++++\n 4 files changed, 119 insertions(+), 6 deletions(-)\n\ndiff --git a/Documentation/config/maintenance.txt b/Documentation/config/maintenance.txt\nindex 06db758172..70585564fa 100644\n--- a/Documentation/config/maintenance.txt\n+++ b/Documentation/config/maintenance.txt\n@@ -10,6 +10,11 @@ maintenance..enabled::\n \t`--task` option exists. By default, only `maintenance.gc.enabled`\n \tis true.\n \n+maintenance..schedule::\n+\tThis config option controls whether or not the given `` runs\n+\tduring a `git maintenance run --schedule=` command. The\n+\tvalue must be one of \"hourly\", \"daily\", or \"weekly\".\n+\n maintenance.commit-graph.auto::\n \tThis integer config option controls how often the `commit-graph` task\n \tshould be run as part of `git maintenance run --auto`. If zero, then\ndiff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt\nindex b44efb05a3..3af5907b01 100644\n--- a/Documentation/git-maintenance.txt\n+++ b/Documentation/git-maintenance.txt\n@@ -107,7 +107,18 @@ OPTIONS\n \tonly if certain thresholds are met. For example, the `gc` task\n \truns when the number of loose objects exceeds the number stored\n \tin the `gc.auto` config setting, or when the number of pack-files\n-\texceeds the `gc.autoPackLimit` config setting.\n+\texceeds the `gc.autoPackLimit` config setting. Not compatible with\n+\tthe `--schedule` option.\n+\n+--schedule::\n+\tWhen combined with the `run` subcommand, run maintenance tasks\n+\tonly if certain time conditions are met, as specified by the\n+\t`maintenance..schedule` config value for each ``.\n+\tThis config value specifies a number of seconds since the last\n+\ttime that task ran, according to the `maintenance..lastRun`\n+\tconfig value. The tasks that are tested are those provided by\n+\tthe `--task=` option(s) or those with\n+\t`maintenance..enabled` set to true.\n \n --quiet::\n \tDo not report progress or other information over `stderr`.\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex f8459df04c..85a3370692 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -704,14 +704,51 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \treturn 0;\n }\n \n-static const char * const builtin_maintenance_run_usage[] = {\n-\tN_(\"git maintenance run [--auto] [--[no-]quiet] [--task=]\"),\n+static const char *const builtin_maintenance_run_usage[] = {\n+\tN_(\"git maintenance run [--auto] [--[no-]quiet] [--task=] [--schedule]\"),\n \tNULL\n };\n \n+enum schedule_priority {\n+\tSCHEDULE_NONE = 0,\n+\tSCHEDULE_WEEKLY = 1,\n+\tSCHEDULE_DAILY = 2,\n+\tSCHEDULE_HOURLY = 3,\n+};\n+\n+static enum schedule_priority parse_schedule(const char *value)\n+{\n+\tif (!value)\n+\t\treturn SCHEDULE_NONE;\n+\tif (!strcasecmp(value, \"hourly\"))\n+\t\treturn SCHEDULE_HOURLY;\n+\tif (!strcasecmp(value, \"daily\"))\n+\t\treturn SCHEDULE_DAILY;\n+\tif (!strcasecmp(value, \"weekly\"))\n+\t\treturn SCHEDULE_WEEKLY;\n+\treturn SCHEDULE_NONE;\n+}\n+\n+static int maintenance_opt_schedule(const struct option *opt, const char *arg,\n+\t\t\t\t int unset)\n+{\n+\tenum schedule_priority *priority = opt->value;\n+\n+\tif (unset)\n+\t\tdie(_(\"--no-schedule is not allowed\"));\n+\n+\t*priority = parse_schedule(arg);\n+\n+\tif (!*priority)\n+\t\tdie(_(\"unrecognized --schedule argument '%s'\"), arg);\n+\n+\treturn 0;\n+}\n+\n struct maintenance_run_opts {\n \tint auto_flag;\n \tint quiet;\n+\tenum schedule_priority schedule;\n };\n \n /* Remember to update object flag allocation in object.h */\n@@ -1159,6 +1196,8 @@ struct maintenance_task {\n \tmaintenance_auto_fn *auto_condition;\n \tunsigned enabled:1;\n \n+\tenum schedule_priority schedule;\n+\n \t/* -1 if not selected. */\n \tint selected_order;\n };\n@@ -1250,8 +1289,10 @@ static int maintenance_run_tasks(struct maintenance_run_opts *opts)\n \t\t\tcontinue;\n \n \t\tif (opts->auto_flag &&\n-\t\t (!tasks[i].auto_condition ||\n-\t\t !tasks[i].auto_condition()))\n+\t\t (!tasks[i].auto_condition || !tasks[i].auto_condition()))\n+\t\t\tcontinue;\n+\n+\t\tif (opts->schedule && tasks[i].schedule < opts->schedule)\n \t\t\tcontinue;\n \n \t\ttrace2_region_enter(\"maintenance\", tasks[i].name, r);\n@@ -1274,13 +1315,23 @@ static void initialize_task_config(void)\n \n \tfor (i = 0; i < TASK__COUNT; i++) {\n \t\tint config_value;\n+\t\tchar *config_str;\n \n-\t\tstrbuf_setlen(&config_name, 0);\n+\t\tstrbuf_reset(&config_name);\n \t\tstrbuf_addf(&config_name, \"maintenance.%s.enabled\",\n \t\t\t tasks[i].name);\n \n \t\tif (!git_config_get_bool(config_name.buf, &config_value))\n \t\t\ttasks[i].enabled = config_value;\n+\n+\t\tstrbuf_reset(&config_name);\n+\t\tstrbuf_addf(&config_name, \"maintenance.%s.schedule\",\n+\t\t\t tasks[i].name);\n+\n+\t\tif (!git_config_get_string(config_name.buf, &config_str)) {\n+\t\t\ttasks[i].schedule = parse_schedule(config_str);\n+\t\t\tfree(config_str);\n+\t\t}\n \t}\n \n \tstrbuf_release(&config_name);\n@@ -1324,6 +1375,9 @@ static int maintenance_run(int argc, const char **argv, const char *prefix)\n \tstruct option builtin_maintenance_run_options[] = {\n \t\tOPT_BOOL(0, \"auto\", &opts.auto_flag,\n \t\t\t N_(\"run tasks based on the state of the repository\")),\n+\t\tOPT_CALLBACK(0, \"schedule\", &opts.schedule, N_(\"frequency\"),\n+\t\t\t N_(\"run tasks based on frequency\"),\n+\t\t\t maintenance_opt_schedule),\n \t\tOPT_BOOL(0, \"quiet\", &opts.quiet,\n \t\t\t N_(\"do not report progress or other information over stderr\")),\n \t\tOPT_CALLBACK_F(0, \"task\", NULL, N_(\"task\"),\n@@ -1344,6 +1398,9 @@ static int maintenance_run(int argc, const char **argv, const char *prefix)\n \t\t\t builtin_maintenance_run_usage,\n \t\t\t PARSE_OPT_STOP_AT_NON_OPTION);\n \n+\tif (opts.auto_flag && opts.schedule)\n+\t\tdie(_(\"use at most one of --auto and --schedule=\"));\n+\n \tif (argc != 0)\n \t\tusage_with_options(builtin_maintenance_run_usage,\n \t\t\t\t builtin_maintenance_run_options);\ndiff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\nindex e0ba19e1ff..328bbaa830 100755\n--- a/t/t7900-maintenance.sh\n+++ b/t/t7900-maintenance.sh\n@@ -264,4 +264,44 @@ test_expect_success 'maintenance.incremental-repack.auto' '\n \tdone\n '\n \n+test_expect_success '--auto and --schedule incompatible' '\n+\ttest_must_fail git maintenance run --auto --schedule=daily 2>err &&\n+\ttest_i18ngrep \"at most one\" err\n+'\n+\n+test_expect_success 'invalid --schedule value' '\n+\ttest_must_fail git maintenance run --schedule=annually 2>err &&\n+\ttest_i18ngrep \"unrecognized --schedule\" err\n+'\n+\n+test_expect_success '--schedule inheritance weekly -> daily -> hourly' '\n+\tgit config maintenance.loose-objects.enabled true &&\n+\tgit config maintenance.loose-objects.schedule hourly &&\n+\tgit config maintenance.commit-graph.enabled true &&\n+\tgit config maintenance.commit-graph.schedule daily &&\n+\tgit config maintenance.incremental-repack.enabled true &&\n+\tgit config maintenance.incremental-repack.schedule weekly &&\n+\n+\tGIT_TRACE2_EVENT=\"$(pwd)/hourly.txt\" \\\n+\t\tgit maintenance run --schedule=hourly 2>/dev/null &&\n+\ttest_subcommand git prune-packed --quiet /dev/null &&\n+\ttest_subcommand git prune-packed --quiet /dev/null &&\n+\ttest_subcommand git prune-packed --quiet \n\nIn preparation for launching background maintenance from the 'git\nmaintenance' builtin, create register/unregister subcommands. These\ncommands update the new 'maintenance.repos' config option in the global\nconfig so the background maintenance job knows which repositories to\nmaintain.\n\nThese commands allow users to add a repository to the background\nmaintenance list without disrupting the actual maintenance mechanism.\n\nFor example, a user can run 'git maintenance register' when no\nbackground maintenance is running and it will not start the background\nmaintenance. A later update to start running background maintenance will\nthen pick up this repository automatically.\n\nThe opposite example is that a user can run 'git maintenance unregister'\nto remove the current repository from background maintenance without\nhalting maintenance for other repositories.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-maintenance.txt | 14 ++++++++\n builtin/gc.c | 55 ++++++++++++++++++++++++++++++-\n t/t7900-maintenance.sh | 17 +++++++++-\n 3 files changed, 84 insertions(+), 2 deletions(-)\n\ndiff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt\nindex 3af5907b01..78d0d8df91 100644\n--- a/Documentation/git-maintenance.txt\n+++ b/Documentation/git-maintenance.txt\n@@ -29,6 +29,15 @@ Git repository.\n SUBCOMMANDS\n -----------\n \n+register::\n+\tInitialize Git config values so any scheduled maintenance will\n+\tstart running on this repository. This adds the repository to the\n+\t`maintenance.repo` config variable in the current user's global\n+\tconfig and enables some recommended configuration values for\n+\t`maintenance..schedule`. The tasks that are enabled are safe\n+\tfor running in the background without disrupting foreground\n+\tprocesses.\n+\n run::\n \tRun one or more maintenance tasks. If one or more `--task` options\n \tare specified, then those tasks are run in that order. Otherwise,\n@@ -36,6 +45,11 @@ run::\n \tconfig options are true. By default, only `maintenance.gc.enabled`\n \tis true.\n \n+unregister::\n+\tRemove the current repository from background maintenance. This\n+\tonly removes the repository from the configured list. It does not\n+\tstop the background maintenance processes from running.\n+\n TASKS\n -----\n \ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 85a3370692..ec77e8d2fa 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -1407,7 +1407,56 @@ static int maintenance_run(int argc, const char **argv, const char *prefix)\n \treturn maintenance_run_tasks(&opts);\n }\n \n-static const char builtin_maintenance_usage[] = N_(\"git maintenance run []\");\n+static int maintenance_register(void)\n+{\n+\tstruct child_process config_set = CHILD_PROCESS_INIT;\n+\tstruct child_process config_get = CHILD_PROCESS_INIT;\n+\n+\t/* There is no current repository, so skip registering it */\n+\tif (!the_repository || !the_repository->gitdir)\n+\t\treturn 0;\n+\n+\tconfig_get.git_cmd = 1;\n+\tstrvec_pushl(&config_get.args, \"config\", \"--global\", \"--get\", \"maintenance.repo\",\n+\t\t the_repository->worktree ? the_repository->worktree\n+\t\t\t\t\t : the_repository->gitdir,\n+\t\t\t NULL);\n+\tconfig_get.out = -1;\n+\n+\tif (start_command(&config_get))\n+\t\treturn error(_(\"failed to run 'git config'\"));\n+\n+\t/* We already have this value in our config! */\n+\tif (!finish_command(&config_get))\n+\t\treturn 0;\n+\n+\tconfig_set.git_cmd = 1;\n+\tstrvec_pushl(&config_set.args, \"config\", \"--add\", \"--global\", \"maintenance.repo\",\n+\t\t the_repository->worktree ? the_repository->worktree\n+\t\t\t\t\t : the_repository->gitdir,\n+\t\t NULL);\n+\n+\treturn run_command(&config_set);\n+}\n+\n+static int maintenance_unregister(void)\n+{\n+\tstruct child_process config_unset = CHILD_PROCESS_INIT;\n+\n+\tif (!the_repository || !the_repository->gitdir)\n+\t\treturn error(_(\"no current repository to unregister\"));\n+\n+\tconfig_unset.git_cmd = 1;\n+\tstrvec_pushl(&config_unset.args, \"config\", \"--global\", \"--unset\",\n+\t\t \"maintenance.repo\",\n+\t\t the_repository->worktree ? the_repository->worktree\n+\t\t\t\t\t : the_repository->gitdir,\n+\t\t NULL);\n+\n+\treturn run_command(&config_unset);\n+}\n+\n+static const char builtin_maintenance_usage[] =\tN_(\"git maintenance []\");\n \n int cmd_maintenance(int argc, const char **argv, const char *prefix)\n {\n@@ -1416,6 +1465,10 @@ int cmd_maintenance(int argc, const char **argv, const char *prefix)\n \n \tif (!strcmp(argv[1], \"run\"))\n \t\treturn maintenance_run(argc - 1, argv + 1, prefix);\n+\tif (!strcmp(argv[1], \"register\"))\n+\t\treturn maintenance_register();\n+\tif (!strcmp(argv[1], \"unregister\"))\n+\t\treturn maintenance_unregister();\n \n \tdie(_(\"invalid subcommand: %s\"), argv[1]);\n }\ndiff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\nindex 328bbaa830..272d1605d2 100755\n--- a/t/t7900-maintenance.sh\n+++ b/t/t7900-maintenance.sh\n@@ -9,7 +9,7 @@ GIT_TEST_MULTI_PACK_INDEX=0\n \n test_expect_success 'help text' '\n \ttest_expect_code 129 git maintenance -h 2>err &&\n-\ttest_i18ngrep \"usage: git maintenance run\" err &&\n+\ttest_i18ngrep \"usage: git maintenance \" err &&\n \ttest_expect_code 128 git maintenance barf 2>err &&\n \ttest_i18ngrep \"invalid subcommand: barf\" err\n '\n@@ -304,4 +304,19 @@ test_expect_success '--schedule inheritance weekly -> daily -> hourly' '\n \ttest_subcommand git multi-pack-index write --no-progress before &&\n+\tgit maintenance register &&\n+\tgit config --global --get-all maintenance.repo >actual &&\n+\tcp before after &&\n+\tpwd >>after &&\n+\ttest_cmp after actual &&\n+\tgit maintenance unregister &&\n+\tgit config --global --get-all maintenance.repo >actual &&\n+\ttest_cmp before actual\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"404706","messageId":"5fdd8188b1d9b6efc2803b557b3ba344e184d22e.1598629517.git.gitgitgadget@gmail.com","threadId":"54070","inReplyTo":"pull.680.v3.git.1598629517.gitgitgadget@gmail.com","subject":"[PATCH v3 1/6] maintenance: optionally skip --auto process","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-08-28T15:45:12Z","receivedAt":"2020-08-28T15:45:47Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nSome commands run 'git maintenance run --auto --[no-]quiet' after doing\ntheir normal work, as a way to keep repositories clean as they are used.\nCurrently, users who do not want this maintenance to occur would set the\n'gc.auto' config option to 0 to avoid the 'gc' task from running.\nHowever, this does not stop the extra process invocation. On Windows,\nthis extra process invocation can be more expensive than necessary.\n\nAllow users to drop this extra process by setting 'maintenance.auto' to\n'false'.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/config/maintenance.txt | 5 +++++\n run-command.c | 6 ++++++\n t/t7900-maintenance.sh | 13 +++++++++++++\n 3 files changed, 24 insertions(+)\n\ndiff --git a/Documentation/config/maintenance.txt b/Documentation/config/maintenance.txt\nindex a0706d8f09..06db758172 100644\n--- a/Documentation/config/maintenance.txt\n+++ b/Documentation/config/maintenance.txt\n@@ -1,3 +1,8 @@\n+maintenance.auto::\n+\tThis boolean config option controls whether some commands run\n+\t`git maintenance run --auto` after doing their normal work. Defaults\n+\tto true.\n+\n maintenance..enabled::\n \tThis boolean config option controls whether the maintenance task\n \twith name `` is run when no `--task` option is specified to\ndiff --git a/run-command.c b/run-command.c\nindex 2ee59acdc8..ea4d0fb4b1 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -7,6 +7,7 @@\n #include \"strbuf.h\"\n #include \"string-list.h\"\n #include \"quote.h\"\n+#include \"config.h\"\n \n void child_process_init(struct child_process *child)\n {\n@@ -1868,8 +1869,13 @@ int run_processes_parallel_tr2(int n, get_next_task_fn get_next_task,\n \n int run_auto_maintenance(int quiet)\n {\n+\tint enabled;\n \tstruct child_process maint = CHILD_PROCESS_INIT;\n \n+\tif (!git_config_get_bool(\"maintenance.auto\", &enabled) &&\n+\t !enabled)\n+\t\treturn 0;\n+\n \tmaint.git_cmd = 1;\n \tstrvec_pushl(&maint.args, \"maintenance\", \"run\", \"--auto\", NULL);\n \tstrvec_push(&maint.args, quiet ? \"--quiet\" : \"--no-quiet\");\ndiff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\nindex 6f878b0141..e0ba19e1ff 100755\n--- a/t/t7900-maintenance.sh\n+++ b/t/t7900-maintenance.sh\n@@ -26,6 +26,19 @@ test_expect_success 'run [--auto|--quiet]' '\n \ttest_subcommand git gc --no-quiet .enabled' '\n \tgit config maintenance.gc.enabled false &&\n \tgit config maintenance.commit-graph.enabled true &&\n-- \ngitgitgadget\n\n"},{"id":"404707","messageId":"e9672c6a6c9ea2f9f42cf599f1726978d9629b71.1598629517.git.gitgitgadget@gmail.com","threadId":"54070","inReplyTo":"pull.680.v3.git.1598629517.gitgitgadget@gmail.com","subject":"[PATCH v3 5/6] maintenance: add start/stop subcommands","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-08-28T15:45:16Z","receivedAt":"2020-08-28T15:45:49Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nAdd new subcommands to 'git maintenance' that start or stop background\nmaintenance using 'cron', when available. This integration is as simple\nas I could make it, barring some implementation complications.\n\nThe schedule is laid out as follows:\n\n 0 1-23 * * * $cmd maintenance run --schedule=hourly\n 0 0 * * 1-6 $cmd maintenance run --schedule=daily\n 0 0 * * 0 $cmd maintenance run --schedule=weekly\n\nwhere $cmd is a properly-qualified 'git for-each-repo' execution:\n\n$cmd=$path/git --exec-path=$path for-each-repo --config=maintenance.repo\n\nwhere $path points to the location of the Git executable running 'git\nmaintenance start'. This is critical for systems with multiple versions\nof Git. Specifically, macOS has a system version at '/usr/bin/git' while\nthe version that users can install resides at '/usr/local/bin/git'\n(symlinked to '/usr/local/libexec/git-core/git'). This will also use\nyour locally-built version if you build and run this in your development\nenvironment without installing first.\n\nThis conditional schedule avoids having cron launch multiple 'git\nfor-each-repo' commands in parallel. Such parallel commands would likely\nlead to the 'hourly' and 'daily' tasks competing over the object\ndatabase lock. This could lead to to some tasks never being run! Since\nthe --schedule= argument will run all tasks with _at least_\nthe given frequency, the daily runs will also run the hourly tasks.\nSimilarly, the weekly runs will also run the daily and hourly tasks.\n\nThe GIT_TEST_CRONTAB environment variable is not intended for users to\nedit, but instead as a way to mock the 'crontab [-l]' command. This\nvariable is set in test-lib.sh to avoid a future test from accidentally\nrunning anything with the cron integration from modifying the user's\nschedule. We use GIT_TEST_CRONTAB='test-tool crontab ' in our\ntests to check how the schedule is modified in 'git maintenance\n(start|stop)' commands.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-maintenance.txt | 11 +++\n Makefile | 1 +\n builtin/gc.c | 124 ++++++++++++++++++++++++++++++\n t/helper/test-crontab.c | 35 +++++++++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/t7900-maintenance.sh | 28 +++++++\n t/test-lib.sh | 6 ++\n 8 files changed, 207 insertions(+)\n create mode 100644 t/helper/test-crontab.c\n\ndiff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt\nindex 78d0d8df91..7f8c279fe8 100644\n--- a/Documentation/git-maintenance.txt\n+++ b/Documentation/git-maintenance.txt\n@@ -45,6 +45,17 @@ run::\n \tconfig options are true. By default, only `maintenance.gc.enabled`\n \tis true.\n \n+start::\n+\tStart running maintenance on the current repository. This performs\n+\tthe same config updates as the `register` subcommand, then updates\n+\tthe background scheduler to run `git maintenance run --scheduled`\n+\ton an hourly basis.\n+\n+stop::\n+\tHalt the background maintenance schedule. The current repository\n+\tis not removed from the list of maintained repositories, in case\n+\tthe background maintenance is restarted later.\n+\n unregister::\n \tRemove the current repository from background maintenance. This\n \tonly removes the repository from the configured list. It does not\ndiff --git a/Makefile b/Makefile\nindex 7c588ff036..c39b39bd7d 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -690,6 +690,7 @@ TEST_BUILTINS_OBJS += test-advise.o\n TEST_BUILTINS_OBJS += test-bloom.o\n TEST_BUILTINS_OBJS += test-chmtime.o\n TEST_BUILTINS_OBJS += test-config.o\n+TEST_BUILTINS_OBJS += test-crontab.o\n TEST_BUILTINS_OBJS += test-ctype.o\n TEST_BUILTINS_OBJS += test-date.o\n TEST_BUILTINS_OBJS += test-delta.o\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex ec77e8d2fa..9914417e25 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -32,6 +32,7 @@\n #include \"remote.h\"\n #include \"midx.h\"\n #include \"object-store.h\"\n+#include \"exec-cmd.h\"\n \n #define FAILED_RUN \"failed to run %s\"\n \n@@ -1456,6 +1457,125 @@ static int maintenance_unregister(void)\n \treturn run_command(&config_unset);\n }\n \n+#define BEGIN_LINE \"# BEGIN GIT MAINTENANCE SCHEDULE\"\n+#define END_LINE \"# END GIT MAINTENANCE SCHEDULE\"\n+\n+static int update_background_schedule(int run_maintenance)\n+{\n+\tint result = 0;\n+\tint in_old_region = 0;\n+\tstruct child_process crontab_list = CHILD_PROCESS_INIT;\n+\tstruct child_process crontab_edit = CHILD_PROCESS_INIT;\n+\tFILE *cron_list, *cron_in;\n+\tconst char *crontab_name;\n+\tstruct strbuf line = STRBUF_INIT;\n+\tstruct lock_file lk;\n+\tchar *lock_path = xstrfmt(\"%s/schedule\", the_repository->objects->odb->path);\n+\n+\tif (hold_lock_file_for_update(&lk, lock_path, LOCK_NO_DEREF) < 0)\n+\t\treturn error(_(\"another process is scheduling background maintenance\"));\n+\n+\tcrontab_name = getenv(\"GIT_TEST_CRONTAB\");\n+\tif (!crontab_name)\n+\t\tcrontab_name = \"crontab\";\n+\n+\tstrvec_split(&crontab_list.args, crontab_name);\n+\tstrvec_push(&crontab_list.args, \"-l\");\n+\tcrontab_list.in = -1;\n+\tcrontab_list.out = dup(lk.tempfile->fd);\n+\tcrontab_list.git_cmd = 0;\n+\n+\tif (start_command(&crontab_list)) {\n+\t\tresult = error(_(\"failed to run 'crontab -l'; your system might not support 'cron'\"));\n+\t\tgoto cleanup;\n+\t}\n+\n+\t/* Ignore exit code, as an empty crontab will return error. */\n+\tfinish_command(&crontab_list);\n+\n+\t/*\n+\t * Read from the .lock file, filtering out the old\n+\t * schedule while appending the new schedule.\n+\t */\n+\tcron_list = fdopen(lk.tempfile->fd, \"r\");\n+\trewind(cron_list);\n+\n+\tstrvec_split(&crontab_edit.args, crontab_name);\n+\tcrontab_edit.in = -1;\n+\tcrontab_edit.git_cmd = 0;\n+\n+\tif (start_command(&crontab_edit)) {\n+\t\tresult = error(_(\"failed to run 'crontab'; your system might not support 'cron'\"));\n+\t\tgoto cleanup;\n+\t}\n+\n+\tcron_in = fdopen(crontab_edit.in, \"w\");\n+\tif (!cron_in) {\n+\t\tresult = error(_(\"failed to open stdin of 'crontab'\"));\n+\t\tgoto done_editing;\n+\t}\n+\n+\twhile (!strbuf_getline_lf(&line, cron_list)) {\n+\t\tif (!in_old_region && !strcmp(line.buf, BEGIN_LINE))\n+\t\t\tin_old_region = 1;\n+\t\tif (in_old_region)\n+\t\t\tcontinue;\n+\t\tfprintf(cron_in, \"%s\\n\", line.buf);\n+\t\tif (in_old_region && !strcmp(line.buf, END_LINE))\n+\t\t\tin_old_region = 0;\n+\t}\n+\n+\tif (run_maintenance) {\n+\t\tstruct strbuf line_format = STRBUF_INIT;\n+\t\tconst char *exec_path = git_exec_path();\n+\n+\t\tfprintf(cron_in, \"%s\\n\", BEGIN_LINE);\n+\t\tfprintf(cron_in,\n+\t\t\t\"# The following schedule was created by Git\\n\");\n+\t\tfprintf(cron_in, \"# Any edits made in this region might be\\n\");\n+\t\tfprintf(cron_in,\n+\t\t\t\"# replaced in the future by a Git command.\\n\\n\");\n+\n+\t\tstrbuf_addf(&line_format,\n+\t\t\t \"%%s %%s * * %%s \\\"%s/git\\\" --exec-path=\\\"%s\\\" for-each-repo --config=maintenance.repo maintenance run --schedule=%%s\\n\",\n+\t\t\t exec_path, exec_path);\n+\t\tfprintf(cron_in, line_format.buf, \"0\", \"1-23\", \"*\", \"hourly\");\n+\t\tfprintf(cron_in, line_format.buf, \"0\", \"0\", \"1-6\", \"daily\");\n+\t\tfprintf(cron_in, line_format.buf, \"0\", \"0\", \"0\", \"weekly\");\n+\t\tstrbuf_release(&line_format);\n+\n+\t\tfprintf(cron_in, \"\\n%s\\n\", END_LINE);\n+\t}\n+\n+\tfflush(cron_in);\n+\tfclose(cron_in);\n+\tclose(crontab_edit.in);\n+\n+done_editing:\n+\tif (finish_command(&crontab_edit)) {\n+\t\tresult = error(_(\"'crontab' died\"));\n+\t\tgoto cleanup;\n+\t}\n+\tfclose(cron_list);\n+\n+cleanup:\n+\trollback_lock_file(&lk);\n+\treturn result;\n+}\n+\n+static int maintenance_start(void)\n+{\n+\tif (maintenance_register())\n+\t\twarning(_(\"failed to add repo to global config\"));\n+\n+\treturn update_background_schedule(1);\n+}\n+\n+static int maintenance_stop(void)\n+{\n+\treturn update_background_schedule(0);\n+}\n+\n static const char builtin_maintenance_usage[] =\tN_(\"git maintenance []\");\n \n int cmd_maintenance(int argc, const char **argv, const char *prefix)\n@@ -1465,6 +1585,10 @@ int cmd_maintenance(int argc, const char **argv, const char *prefix)\n \n \tif (!strcmp(argv[1], \"run\"))\n \t\treturn maintenance_run(argc - 1, argv + 1, prefix);\n+\tif (!strcmp(argv[1], \"start\"))\n+\t\treturn maintenance_start();\n+\tif (!strcmp(argv[1], \"stop\"))\n+\t\treturn maintenance_stop();\n \tif (!strcmp(argv[1], \"register\"))\n \t\treturn maintenance_register();\n \tif (!strcmp(argv[1], \"unregister\"))\ndiff --git a/t/helper/test-crontab.c b/t/helper/test-crontab.c\nnew file mode 100644\nindex 0000000000..f5db6319c6\n--- /dev/null\n+++ b/t/helper/test-crontab.c\n@@ -0,0 +1,35 @@\n+#include \"test-tool.h\"\n+#include \"cache.h\"\n+\n+/*\n+ * Usage: test-tool cron [-l]\n+ *\n+ * If -l is specified, then write the contents of to stdou.\n+ * Otherwise, write from stdin into .\n+ */\n+int cmd__crontab(int argc, const char **argv)\n+{\n+\tchar a;\n+\tFILE *from, *to;\n+\n+\tif (argc == 3 && !strcmp(argv[2], \"-l\")) {\n+\t\tfrom = fopen(argv[1], \"r\");\n+\t\tif (!from)\n+\t\t\treturn 0;\n+\t\tto = stdout;\n+\t} else if (argc == 2) {\n+\t\tfrom = stdin;\n+\t\tto = fopen(argv[1], \"w\");\n+\t} else\n+\t\treturn error(\"unknown arguments\");\n+\n+\twhile ((a = fgetc(from)) != EOF)\n+\t\tfputc(a, to);\n+\n+\tif (argc == 3)\n+\t\tfclose(from);\n+\telse\n+\t\tfclose(to);\n+\n+\treturn 0;\n+}\ndiff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\nindex 590b2efca7..432b49d948 100644\n--- a/t/helper/test-tool.c\n+++ b/t/helper/test-tool.c\n@@ -18,6 +18,7 @@ static struct test_cmd cmds[] = {\n \t{ \"bloom\", cmd__bloom },\n \t{ \"chmtime\", cmd__chmtime },\n \t{ \"config\", cmd__config },\n+\t{ \"crontab\", cmd__crontab },\n \t{ \"ctype\", cmd__ctype },\n \t{ \"date\", cmd__date },\n \t{ \"delta\", cmd__delta },\ndiff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\nindex ddc8e990e9..7c3281e071 100644\n--- a/t/helper/test-tool.h\n+++ b/t/helper/test-tool.h\n@@ -8,6 +8,7 @@ int cmd__advise_if_enabled(int argc, const char **argv);\n int cmd__bloom(int argc, const char **argv);\n int cmd__chmtime(int argc, const char **argv);\n int cmd__config(int argc, const char **argv);\n+int cmd__crontab(int argc, const char **argv);\n int cmd__ctype(int argc, const char **argv);\n int cmd__date(int argc, const char **argv);\n int cmd__delta(int argc, const char **argv);\ndiff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\nindex 272d1605d2..8803fcf621 100755\n--- a/t/t7900-maintenance.sh\n+++ b/t/t7900-maintenance.sh\n@@ -319,4 +319,32 @@ test_expect_success 'register and unregister' '\n \ttest_cmp before actual\n '\n \n+test_expect_success 'start from empty cron table' '\n+\tGIT_TEST_CRONTAB=\"test-tool crontab cron.txt\" git maintenance start &&\n+\n+\t# start registers the repo\n+\tgit config --get --global maintenance.repo \"$(pwd)\" &&\n+\n+\tgrep \"for-each-repo --config=maintenance.repo maintenance run --schedule=daily\" cron.txt &&\n+\tgrep \"for-each-repo --config=maintenance.repo maintenance run --schedule=hourly\" cron.txt &&\n+\tgrep \"for-each-repo --config=maintenance.repo maintenance run --schedule=weekly\" cron.txt\n+'\n+\n+test_expect_success 'stop from existing schedule' '\n+\tGIT_TEST_CRONTAB=\"test-tool crontab cron.txt\" git maintenance stop &&\n+\n+\t# stop does not unregister the repo\n+\tgit config --get --global maintenance.repo \"$(pwd)\" &&\n+\n+\t# Operation is idempotent\n+\tGIT_TEST_CRONTAB=\"test-tool crontab cron.txt\" git maintenance stop &&\n+\ttest_must_be_empty cron.txt\n+'\n+\n+test_expect_success 'start preserves existing schedule' '\n+\techo \"Important information!\" >cron.txt &&\n+\tGIT_TEST_CRONTAB=\"test-tool crontab cron.txt\" git maintenance start &&\n+\tgrep \"Important information!\" cron.txt\n+'\n+\n test_done\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex ef31f40037..4a60d1ed76 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -1702,3 +1702,9 @@ test_lazy_prereq SHA1 '\n test_lazy_prereq REBASE_P '\n \ttest -z \"$GIT_TEST_SKIP_REBASE_P\"\n '\n+\n+# Ensure that no test accidentally triggers a Git command\n+# that runs 'crontab', affecting a user's cron schedule.\n+# Tests that verify the cron integration must set this locally\n+# to avoid errors.\n+GIT_TEST_CRONTAB=\"exit 1\"\n-- \ngitgitgadget\n\n"},{"id":"404708","messageId":"62e8db8b2a8674fb40681478cbae8bbbd6936661.1598629517.git.gitgitgadget@gmail.com","threadId":"54070","inReplyTo":"pull.680.v3.git.1598629517.gitgitgadget@gmail.com","subject":"[PATCH v3 6/6] maintenance: recommended schedule in register/start","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-08-28T15:45:17Z","receivedAt":"2020-08-28T15:45:54Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe 'git maintenance (register|start)' subcommands add the current\nrepository to the global Git config so maintenance will operate on that\nrepository. It does not specify what maintenance should occur or how\noften.\n\nIf a user sets any 'maintenance..schedule' config value, then\nthey have chosen a specific schedule for themselves and Git should\nrespect that.\n\nHowever, in an effort to recommend a good schedule for repositories of\nall sizes, set new config values for recommended tasks that are safe to\nrun in the background while users run foreground Git commands. These\ncommands are generally everything but the 'gc' task.\n\nAuthor's Note: I feel we should do _something_ to recommend a good\nschedule to users, but I'm not 100% set on this schedule. This is the\nschedule we use in Scalar and VFS for Git for very large repositories\nusing the GVFS protocol. While the schedule works in that environment,\nit is possible that \"normal\" Git repositories could benefit from\nsomething more obvious (such as running 'gc' weekly). However, this\npatch gives us a place to start a conversation on what we should\nrecommend. For my purposes, Scalar will set these config values so we\ncan always differ from core Git's recommendations.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-maintenance.txt | 6 ++++\n builtin/gc.c | 46 +++++++++++++++++++++++++++++++\n t/t7900-maintenance.sh | 16 +++++++++++\n 3 files changed, 68 insertions(+)\n\ndiff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt\nindex 7f8c279fe8..364b3e32bf 100644\n--- a/Documentation/git-maintenance.txt\n+++ b/Documentation/git-maintenance.txt\n@@ -37,6 +37,12 @@ register::\n \t`maintenance..schedule`. The tasks that are enabled are safe\n \tfor running in the background without disrupting foreground\n \tprocesses.\n++\n+If your repository has no 'maintenance..schedule' configuration\n+values set, then Git will set configuration values to some recommended\n+settings. These settings disable foreground maintenance while performing\n+maintenance tasks in the background that will not interrupt foreground Git\n+operations.\n \n run::\n \tRun one or more maintenance tasks. If one or more `--task` options\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 9914417e25..5f253d3458 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -1408,6 +1408,49 @@ static int maintenance_run(int argc, const char **argv, const char *prefix)\n \treturn maintenance_run_tasks(&opts);\n }\n \n+static int has_schedule_config(void)\n+{\n+\tint i, found = 0;\n+\tstruct strbuf config_name = STRBUF_INIT;\n+\tsize_t prefix;\n+\n+\tstrbuf_addstr(&config_name, \"maintenance.\");\n+\tprefix = config_name.len;\n+\n+\tfor (i = 0; !found && i < TASK__COUNT; i++) {\n+\t\tchar *value;\n+\n+\t\tstrbuf_setlen(&config_name, prefix);\n+\t\tstrbuf_addf(&config_name, \"%s.schedule\", tasks[i].name);\n+\n+\t\tif (!git_config_get_string(config_name.buf, &value)) {\n+\t\t\tfound = 1;\n+\t\t\tFREE_AND_NULL(value);\n+\t\t}\n+\t}\n+\n+\tstrbuf_release(&config_name);\n+\treturn found;\n+}\n+\n+static void set_recommended_schedule(void)\n+{\n+\tgit_config_set(\"maintenance.auto\", \"false\");\n+\tgit_config_set(\"maintenance.gc.enabled\", \"false\");\n+\n+\tgit_config_set(\"maintenance.prefetch.enabled\", \"true\");\n+\tgit_config_set(\"maintenance.prefetch.schedule\", \"hourly\");\n+\n+\tgit_config_set(\"maintenance.commit-graph.enabled\", \"true\");\n+\tgit_config_set(\"maintenance.commit-graph.schedule\", \"hourly\");\n+\n+\tgit_config_set(\"maintenance.loose-objects.enabled\", \"true\");\n+\tgit_config_set(\"maintenance.loose-objects.schedule\", \"daily\");\n+\n+\tgit_config_set(\"maintenance.incremental-repack.enabled\", \"true\");\n+\tgit_config_set(\"maintenance.incremental-repack.schedule\", \"daily\");\n+}\n+\n static int maintenance_register(void)\n {\n \tstruct child_process config_set = CHILD_PROCESS_INIT;\n@@ -1417,6 +1460,9 @@ static int maintenance_register(void)\n \tif (!the_repository || !the_repository->gitdir)\n \t\treturn 0;\n \n+\tif (!has_schedule_config())\n+\t\tset_recommended_schedule();\n+\n \tconfig_get.git_cmd = 1;\n \tstrvec_pushl(&config_get.args, \"config\", \"--global\", \"--get\", \"maintenance.repo\",\n \t\t the_repository->worktree ? the_repository->worktree\ndiff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\nindex 8803fcf621..5a31f3925b 100755\n--- a/t/t7900-maintenance.sh\n+++ b/t/t7900-maintenance.sh\n@@ -309,7 +309,23 @@ test_expect_success 'register and unregister' '\n \tgit config --global --add maintenance.repo /existing1 &&\n \tgit config --global --add maintenance.repo /existing2 &&\n \tgit config --global --get-all maintenance.repo >before &&\n+\n+\t# We still have maintenance..schedule config set,\n+\t# so this does not update the local schedule\n+\tgit maintenance register &&\n+\ttest_must_fail git config maintenance.auto &&\n+\n+\t# Clear previous maintenance..schedule values\n+\tfor task in loose-objects commit-graph incremental-repack\n+\tdo\n+\t\tgit config --unset maintenance.$task.schedule || return 1\n+\tdone &&\n \tgit maintenance register &&\n+\ttest_cmp_config false maintenance.auto &&\n+\ttest_cmp_config false maintenance.gc.enabled &&\n+\ttest_cmp_config true maintenance.prefetch.enabled &&\n+\ttest_cmp_config hourly maintenance.commit-graph.schedule &&\n+\ttest_cmp_config daily maintenance.incremental-repack.schedule &&\n \tgit config --global --get-all maintenance.repo >actual &&\n \tcp before after &&\n \tpwd >>after &&\n-- \ngitgitgadget\n"},{"id":"404709","messageId":"b29b68614b8b5d7b3c39bdb6db53dc7f3a1b95ff.1598629517.git.gitgitgadget@gmail.com","threadId":"54070","inReplyTo":"pull.680.v3.git.1598629517.gitgitgadget@gmail.com","subject":"[PATCH v3 3/6] for-each-repo: run subcommands on configured repos","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-08-28T15:45:14Z","receivedAt":"2020-08-28T15:46:05Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nIt can be helpful to store a list of repositories in global or system\nconfig and then iterate Git commands on that list. Create a new builtin\nthat makes this process simple for experts. We will use this builtin to\nrun scheduled maintenance on all configured repositories in a future\nchange.\n\nThe test is very simple, but does highlight that the \"--\" argument is\noptional.\n\nSigned-off-by: Derrick Stolee \n---\n .gitignore | 1 +\n Documentation/git-for-each-repo.txt | 59 +++++++++++++++++++++++++++++\n Makefile | 1 +\n builtin.h | 1 +\n builtin/for-each-repo.c | 58 ++++++++++++++++++++++++++++\n command-list.txt | 1 +\n git.c | 1 +\n t/t0068-for-each-repo.sh | 30 +++++++++++++++\n 8 files changed, 152 insertions(+)\n create mode 100644 Documentation/git-for-each-repo.txt\n create mode 100644 builtin/for-each-repo.c\n create mode 100755 t/t0068-for-each-repo.sh\n\ndiff --git a/.gitignore b/.gitignore\nindex a5808fa30d..5eb2a2be71 100644\n--- a/.gitignore\n+++ b/.gitignore\n@@ -67,6 +67,7 @@\n /git-filter-branch\n /git-fmt-merge-msg\n /git-for-each-ref\n+/git-for-each-repo\n /git-format-patch\n /git-fsck\n /git-fsck-objects\ndiff --git a/Documentation/git-for-each-repo.txt b/Documentation/git-for-each-repo.txt\nnew file mode 100644\nindex 0000000000..94bd19da26\n--- /dev/null\n+++ b/Documentation/git-for-each-repo.txt\n@@ -0,0 +1,59 @@\n+git-for-each-repo(1)\n+====================\n+\n+NAME\n+----\n+git-for-each-repo - Run a Git command on a list of repositories\n+\n+\n+SYNOPSIS\n+--------\n+[verse]\n+'git for-each-repo' --config= [--] \n+\n+\n+DESCRIPTION\n+-----------\n+Run a Git command on a list of repositories. The arguments after the\n+known options or `--` indicator are used as the arguments for the Git\n+subprocess.\n+\n+THIS COMMAND IS EXPERIMENTAL. THE BEHAVIOR MAY CHANGE.\n+\n+For example, we could run maintenance on each of a list of repositories\n+stored in a `maintenance.repo` config variable using\n+\n+-------------\n+git for-each-repo --config=maintenance.repo maintenance run\n+-------------\n+\n+This will run `git -C maintenance run` for each value ``\n+in the multi-valued config variable `maintenance.repo`.\n+\n+\n+OPTIONS\n+-------\n+--config=::\n+\tUse the given config variable as a multi-valued list storing\n+\tabsolute path names. Iterate on that list of paths to run\n+\tthe given arguments.\n++\n+These config values are loaded from system, global, and local Git config,\n+as available. If `git for-each-repo` is run in a directory that is not a\n+Git repository, then only the system and global config is used.\n+\n+\n+SUBPROCESS BEHAVIOR\n+-------------------\n+\n+If any `git -C ` subprocess returns a non-zero exit code,\n+then the `git for-each-repo` process returns that exit code without running\n+more subprocesses.\n+\n+Each `git -C ` subprocess inherits the standard file\n+descriptors `stdin`, `stdout`, and `stderr`.\n+\n+\n+GIT\n+---\n+Part of the linkgit:git[1] suite\ndiff --git a/Makefile b/Makefile\nindex 65f8cfb236..7c588ff036 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1071,6 +1071,7 @@ BUILTIN_OBJS += builtin/fetch-pack.o\n BUILTIN_OBJS += builtin/fetch.o\n BUILTIN_OBJS += builtin/fmt-merge-msg.o\n BUILTIN_OBJS += builtin/for-each-ref.o\n+BUILTIN_OBJS += builtin/for-each-repo.o\n BUILTIN_OBJS += builtin/fsck.o\n BUILTIN_OBJS += builtin/gc.o\n BUILTIN_OBJS += builtin/get-tar-commit-id.o\ndiff --git a/builtin.h b/builtin.h\nindex 17c1c0ce49..ff7c6e5aa9 100644\n--- a/builtin.h\n+++ b/builtin.h\n@@ -150,6 +150,7 @@ int cmd_fetch(int argc, const char **argv, const char *prefix);\n int cmd_fetch_pack(int argc, const char **argv, const char *prefix);\n int cmd_fmt_merge_msg(int argc, const char **argv, const char *prefix);\n int cmd_for_each_ref(int argc, const char **argv, const char *prefix);\n+int cmd_for_each_repo(int argc, const char **argv, const char *prefix);\n int cmd_format_patch(int argc, const char **argv, const char *prefix);\n int cmd_fsck(int argc, const char **argv, const char *prefix);\n int cmd_gc(int argc, const char **argv, const char *prefix);\ndiff --git a/builtin/for-each-repo.c b/builtin/for-each-repo.c\nnew file mode 100644\nindex 0000000000..5bba623ff1\n--- /dev/null\n+++ b/builtin/for-each-repo.c\n@@ -0,0 +1,58 @@\n+#include \"cache.h\"\n+#include \"config.h\"\n+#include \"builtin.h\"\n+#include \"parse-options.h\"\n+#include \"run-command.h\"\n+#include \"string-list.h\"\n+\n+static const char * const for_each_repo_usage[] = {\n+\tN_(\"git for-each-repo --config= \"),\n+\tNULL\n+};\n+\n+static int run_command_on_repo(const char *path,\n+\t\t\t void *cbdata)\n+{\n+\tint i;\n+\tstruct child_process child = CHILD_PROCESS_INIT;\n+\tstruct strvec *args = (struct strvec *)cbdata;\n+\n+\tchild.git_cmd = 1;\n+\tstrvec_pushl(&child.args, \"-C\", path, NULL);\n+\n+\tfor (i = 0; i < args->nr; i++)\n+\t\tstrvec_push(&child.args, args->v[i]);\n+\n+\treturn run_command(&child);\n+}\n+\n+int cmd_for_each_repo(int argc, const char **argv, const char *prefix)\n+{\n+\tstatic const char *config_key = NULL;\n+\tint i, result = 0;\n+\tconst struct string_list *values;\n+\tstruct strvec args = STRVEC_INIT;\n+\n+\tconst struct option options[] = {\n+\t\tOPT_STRING(0, \"config\", &config_key, N_(\"config\"),\n+\t\t\t N_(\"config key storing a list of repository paths\")),\n+\t\tOPT_END()\n+\t};\n+\n+\targc = parse_options(argc, argv, prefix, options, for_each_repo_usage,\n+\t\t\t PARSE_OPT_STOP_AT_NON_OPTION);\n+\n+\tif (!config_key)\n+\t\tdie(_(\"missing --config=\"));\n+\n+\tfor (i = 0; i < argc; i++)\n+\t\tstrvec_push(&args, argv[i]);\n+\n+\tvalues = repo_config_get_value_multi(the_repository,\n+\t\t\t\t\t config_key);\n+\n+\tfor (i = 0; !result && i < values->nr; i++)\n+\t\tresult = run_command_on_repo(values->items[i].string, &args);\n+\n+\treturn result;\n+}\ndiff --git a/command-list.txt b/command-list.txt\nindex 0e3204e7d1..581499be82 100644\n--- a/command-list.txt\n+++ b/command-list.txt\n@@ -94,6 +94,7 @@ git-fetch-pack synchingrepositories\n git-filter-branch ancillarymanipulators\n git-fmt-merge-msg purehelpers\n git-for-each-ref plumbinginterrogators\n+git-for-each-repo plumbinginterrogators\n git-format-patch mainporcelain\n git-fsck ancillaryinterrogators complete\n git-gc mainporcelain\ndiff --git a/git.c b/git.c\nindex 24f250d29a..1cab64b5d1 100644\n--- a/git.c\n+++ b/git.c\n@@ -511,6 +511,7 @@ static struct cmd_struct commands[] = {\n \t{ \"fetch-pack\", cmd_fetch_pack, RUN_SETUP | NO_PARSEOPT },\n \t{ \"fmt-merge-msg\", cmd_fmt_merge_msg, RUN_SETUP },\n \t{ \"for-each-ref\", cmd_for_each_ref, RUN_SETUP },\n+\t{ \"for-each-repo\", cmd_for_each_repo, RUN_SETUP_GENTLY },\n \t{ \"format-patch\", cmd_format_patch, RUN_SETUP },\n \t{ \"fsck\", cmd_fsck, RUN_SETUP },\n \t{ \"fsck-objects\", cmd_fsck, RUN_SETUP },\ndiff --git a/t/t0068-for-each-repo.sh b/t/t0068-for-each-repo.sh\nnew file mode 100755\nindex 0000000000..136b4ec839\n--- /dev/null\n+++ b/t/t0068-for-each-repo.sh\n@@ -0,0 +1,30 @@\n+#!/bin/sh\n+\n+test_description='git for-each-repo builtin'\n+\n+. ./test-lib.sh\n+\n+test_expect_success 'run based on configured value' '\n+\tgit init one &&\n+\tgit init two &&\n+\tgit init three &&\n+\tgit -C two commit --allow-empty -m \"DID NOT RUN\" &&\n+\tgit config run.key \"$TRASH_DIRECTORY/one\" &&\n+\tgit config --add run.key \"$TRASH_DIRECTORY/three\" &&\n+\tgit for-each-repo --config=run.key commit --allow-empty -m \"ran\" &&\n+\tgit -C one log -1 --pretty=format:%s >message &&\n+\tgrep ran message &&\n+\tgit -C two log -1 --pretty=format:%s >message &&\n+\t! grep ran message &&\n+\tgit -C three log -1 --pretty=format:%s >message &&\n+\tgrep ran message &&\n+\tgit for-each-repo --config=run.key -- commit --allow-empty -m \"ran again\" &&\n+\tgit -C one log -1 --pretty=format:%s >message &&\n+\tgrep again message &&\n+\tgit -C two log -1 --pretty=format:%s >message &&\n+\t! grep again message &&\n+\tgit -C three log -1 --pretty=format:%s >message &&\n+\tgrep again message\n+'\n+\n+test_done\n-- \ngitgitgadget\n\n"}]}