git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH 2/7] maintenance: add --schedule option and config

From
Đoàn Trần Công Danh <congdanhqx@gmail.com>
Date
Sep 8, 2020, 13:07 UTC
Message-ID
<20200908130738.GB25593@danh.dev>
In-Reply-To
<1783e80b8d3b8361d1d62947a49ba584685dacc4.1599234126.git.gitgitgadget@gmail.com>
On 2020-09-04 15:42:01+0000, Derrick Stolee via GitGitGadget <gitgitgadget@gmail.com> wrote:
> From: Derrick Stolee <dstolee@microsoft.com>
> 
> A user may want to run certain maintenance tasks based on frequency, not
> conditions given in the repository. For example, the user may want to
Hm, sorry but I couldn't decipher "not conditions" here. :|
> perform a 'prefetch' task every hour, or 'gc' task every day. To assist,

I think it's better to say: "To assist those users", at least it's easier to read for non-native English like me.

> update the 'git maintenance run' command to include a
> '--schedule=<frequency>' option. The allowed frequencies are 'hourly',
So, we have "--schedule=" here, ...
> 'daily', and 'weekly'. These values are also allowed in a new config
> value 'maintenance.<task>.schedule'.
> 
> The 'git maintenance run --schedule=<frequency>' checks the '*.schedule'
and here, ...
Show 14 quoted lines
> config value for each enabled task to see if the configured frequency is
> at least as frequent as the frequency from the '--schedule' argument. We
> use the following order, for full clarity:
> 
> 	'hourly' > 'daily' > 'weekly'
> 
> Use new 'enum schedule_priority' to track these values numerically.
> 
> The following cron table would run the scheduled tasks with the correct
> frequencies:
> 
>   0 1-23 * * *    git -C <repo> maintenance run --scheduled=hourly
>   0 0    * * 1-6  git -C <repo> maintenance run --scheduled=daily
>   0 0    * * 0    git -C <repo> maintenance run --scheduled=weekly
but it's spelt with "--scheduled=", here and below, mispell, I guess.
Reading the patch, it looks like "--scheduled=" is mispelt.
Show 131 quoted lines
> This cron schedule will run --scheduled=hourly every hour except at
> midnight. This avoids a concurrent run with the --scheduled=daily that
> runs at midnight every day except the first day of the week. This avoids
> a concurrent run with the --scheduled=weekly that runs at midnight on
> the first day of the week. Since --scheduled=daily also runs the
> 'hourly' tasks and --scheduled=weekly runs the 'hourly' and 'daily'
> tasks, we will still see all tasks run with the proper frequencies.
> 
> Signed-off-by: Derrick Stolee <dstolee@microsoft.com>
> ---
>  Documentation/config/maintenance.txt |  5 +++
>  Documentation/git-maintenance.txt    | 13 +++++-
>  builtin/gc.c                         | 67 +++++++++++++++++++++++++---
>  t/t7900-maintenance.sh               | 40 +++++++++++++++++
>  4 files changed, 119 insertions(+), 6 deletions(-)
> 
> diff --git a/Documentation/config/maintenance.txt b/Documentation/config/maintenance.txt
> index 06db758172..70585564fa 100644
> --- a/Documentation/config/maintenance.txt
> +++ b/Documentation/config/maintenance.txt
> @@ -10,6 +10,11 @@ maintenance.<task>.enabled::
>  	`--task` option exists. By default, only `maintenance.gc.enabled`
>  	is true.
>  
> +maintenance.<task>.schedule::
> +	This config option controls whether or not the given `<task>` runs
> +	during a `git maintenance run --schedule=<frequency>` command. The
> +	value must be one of "hourly", "daily", or "weekly".
> +
>  maintenance.commit-graph.auto::
>  	This integer config option controls how often the `commit-graph` task
>  	should be run as part of `git maintenance run --auto`. If zero, then
> diff --git a/Documentation/git-maintenance.txt b/Documentation/git-maintenance.txt
> index b44efb05a3..3af5907b01 100644
> --- a/Documentation/git-maintenance.txt
> +++ b/Documentation/git-maintenance.txt
> @@ -107,7 +107,18 @@ OPTIONS
>  	only if certain thresholds are met. For example, the `gc` task
>  	runs when the number of loose objects exceeds the number stored
>  	in the `gc.auto` config setting, or when the number of pack-files
> -	exceeds the `gc.autoPackLimit` config setting.
> +	exceeds the `gc.autoPackLimit` config setting. Not compatible with
> +	the `--schedule` option.
> +
> +--schedule::
> +	When combined with the `run` subcommand, run maintenance tasks
> +	only if certain time conditions are met, as specified by the
> +	`maintenance.<task>.schedule` config value for each `<task>`.
> +	This config value specifies a number of seconds since the last
> +	time that task ran, according to the `maintenance.<task>.lastRun`
> +	config value. The tasks that are tested are those provided by
> +	the `--task=<task>` option(s) or those with
> +	`maintenance.<task>.enabled` set to true.
>  
>  --quiet::
>  	Do not report progress or other information over `stderr`.
> diff --git a/builtin/gc.c b/builtin/gc.c
> index f8459df04c..85a3370692 100644
> --- a/builtin/gc.c
> +++ b/builtin/gc.c
> @@ -704,14 +704,51 @@ int cmd_gc(int argc, const char **argv, const char *prefix)
>  	return 0;
>  }
>  
> -static const char * const builtin_maintenance_run_usage[] = {
> -	N_("git maintenance run [--auto] [--[no-]quiet] [--task=<task>]"),
> +static const char *const builtin_maintenance_run_usage[] = {
> +	N_("git maintenance run [--auto] [--[no-]quiet] [--task=<task>] [--schedule]"),
>  	NULL
>  };
>  
> +enum schedule_priority {
> +	SCHEDULE_NONE = 0,
> +	SCHEDULE_WEEKLY = 1,
> +	SCHEDULE_DAILY = 2,
> +	SCHEDULE_HOURLY = 3,
> +};
> +
> +static enum schedule_priority parse_schedule(const char *value)
> +{
> +	if (!value)
> +		return SCHEDULE_NONE;
> +	if (!strcasecmp(value, "hourly"))
> +		return SCHEDULE_HOURLY;
> +	if (!strcasecmp(value, "daily"))
> +		return SCHEDULE_DAILY;
> +	if (!strcasecmp(value, "weekly"))
> +		return SCHEDULE_WEEKLY;
> +	return SCHEDULE_NONE;
> +}
> +
> +static int maintenance_opt_schedule(const struct option *opt, const char *arg,
> +				    int unset)
> +{
> +	enum schedule_priority *priority = opt->value;
> +
> +	if (unset)
> +		die(_("--no-schedule is not allowed"));
> +
> +	*priority = parse_schedule(arg);
> +
> +	if (!*priority)
> +		die(_("unrecognized --schedule argument '%s'"), arg);
> +
> +	return 0;
> +}
> +
>  struct maintenance_run_opts {
>  	int auto_flag;
>  	int quiet;
> +	enum schedule_priority schedule;
>  };
>  
>  /* Remember to update object flag allocation in object.h */
> @@ -1159,6 +1196,8 @@ struct maintenance_task {
>  	maintenance_auto_fn *auto_condition;
>  	unsigned enabled:1;
>  
> +	enum schedule_priority schedule;
> +
>  	/* -1 if not selected. */
>  	int selected_order;
>  };
> @@ -1250,8 +1289,10 @@ static int maintenance_run_tasks(struct maintenance_run_opts *opts)
>  			continue;
>  
>  		if (opts->auto_flag &&
> -		    (!tasks[i].auto_condition ||
> -		     !tasks[i].auto_condition()))
> +		    (!tasks[i].auto_condition || !tasks[i].auto_condition()))
> +			continue;
This line only add unnecessary noise to this patch.
-- 
Danh
> +
> +		if (opts->schedule && tasks[i].schedule < opts->schedule)
>  			continue;
>  
>  		trace2_region_enter("maintenance", tasks[i].name, r);
> @@ -1274,13 +1315,23 @@ static void initialize_task_config(void)
>  
>  	for (i = 0; i < TASK__COUNT; i++) {
>  		int config_value;
> +		char *config_str;
>  
> -		strbuf_setlen(&config_name, 0);
> +		strbuf_reset(&config_name);
>  		strbuf_addf(&config_name, "maintenance.%s.enabled",
>  			    tasks[i].name);
>  
>  		if (!git_config_get_bool(config_name.buf, &config_value))
>  			tasks[i].enabled = config_value;
> +
> +		strbuf_reset(&config_name);
> +		strbuf_addf(&config_name, "maintenance.%s.schedule",
> +			    tasks[i].name);
> +
> +		if (!git_config_get_string(config_name.buf, &config_str)) {
> +			tasks[i].schedule = parse_schedule(config_str);
> +			free(config_str);
> +		}
>  	}
>  
>  	strbuf_release(&config_name);
> @@ -1324,6 +1375,9 @@ static int maintenance_run(int argc, const char **argv, const char *prefix)
>  	struct option builtin_maintenance_run_options[] = {
>  		OPT_BOOL(0, "auto", &opts.auto_flag,
>  			 N_("run tasks based on the state of the repository")),
> +		OPT_CALLBACK(0, "schedule", &opts.schedule, N_("frequency"),
> +			     N_("run tasks based on frequency"),
> +			     maintenance_opt_schedule),
>  		OPT_BOOL(0, "quiet", &opts.quiet,
>  			 N_("do not report progress or other information over stderr")),
>  		OPT_CALLBACK_F(0, "task", NULL, N_("task"),
> @@ -1344,6 +1398,9 @@ static int maintenance_run(int argc, const char **argv, const char *prefix)
>  			     builtin_maintenance_run_usage,
>  			     PARSE_OPT_STOP_AT_NON_OPTION);
>  
> +	if (opts.auto_flag && opts.schedule)
> +		die(_("use at most one of --auto and --schedule=<frequency>"));
> +
>  	if (argc != 0)
>  		usage_with_options(builtin_maintenance_run_usage,
>  				   builtin_maintenance_run_options);
> diff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh
> index e0ba19e1ff..328bbaa830 100755
> --- a/t/t7900-maintenance.sh
> +++ b/t/t7900-maintenance.sh
> @@ -264,4 +264,44 @@ test_expect_success 'maintenance.incremental-repack.auto' '
>  	done
>  '
>  
> +test_expect_success '--auto and --schedule incompatible' '
> +	test_must_fail git maintenance run --auto --schedule=daily 2>err &&
> +	test_i18ngrep "at most one" err
> +'
> +
> +test_expect_success 'invalid --schedule value' '
> +	test_must_fail git maintenance run --schedule=annually 2>err &&
> +	test_i18ngrep "unrecognized --schedule" err
> +'
> +
> +test_expect_success '--schedule inheritance weekly -> daily -> hourly' '
> +	git config maintenance.loose-objects.enabled true &&
> +	git config maintenance.loose-objects.schedule hourly &&
> +	git config maintenance.commit-graph.enabled true &&
> +	git config maintenance.commit-graph.schedule daily &&
> +	git config maintenance.incremental-repack.enabled true &&
> +	git config maintenance.incremental-repack.schedule weekly &&
> +
> +	GIT_TRACE2_EVENT="$(pwd)/hourly.txt" \
> +		git maintenance run --schedule=hourly 2>/dev/null &&
> +	test_subcommand git prune-packed --quiet <hourly.txt &&
> +	test_subcommand ! git commit-graph write --split --reachable \
> +		--no-progress <hourly.txt &&
> +	test_subcommand ! git multi-pack-index write --no-progress <hourly.txt &&
> +
> +	GIT_TRACE2_EVENT="$(pwd)/daily.txt" \
> +		git maintenance run --schedule=daily 2>/dev/null &&
> +	test_subcommand git prune-packed --quiet <daily.txt &&
> +	test_subcommand git commit-graph write --split --reachable \
> +		--no-progress <daily.txt &&
> +	test_subcommand ! git multi-pack-index write --no-progress <daily.txt &&
> +
> +	GIT_TRACE2_EVENT="$(pwd)/weekly.txt" \
> +		git maintenance run --schedule=weekly 2>/dev/null &&
> +	test_subcommand git prune-packed --quiet <weekly.txt &&
> +	test_subcommand git commit-graph write --split --reachable \
> +		--no-progress <weekly.txt &&
> +	test_subcommand git multi-pack-index write --no-progress <weekly.txt
> +'
> +
>  test_done
> -- 
> gitgitgadget
> 

-- 
Danh
Previous: Derrick Stolee via GitGitGadgetNext: Derrick Stolee
Message 11 of 62 in “Maintenance III: Background maintenance”
  1. 0/7 Maintenance III: Background maintenanceDerrick Stolee via GitGitGadget, Sep 4, 2020
  2. 4/7 maintenance: add [un]register subcommandsDerrick Stolee via GitGitGadget, Sep 4, 2020
  3. 3/7 for-each-repo: run subcommands on configured reposDerrick Stolee via GitGitGadget, Sep 4, 2020
  4. 1/7 maintenance: optionally skip --auto processDerrick Stolee via GitGitGadget, Sep 4, 2020
  5. 7/7 maintenance: add troubleshooting guide to docsDerrick Stolee via GitGitGadget, Sep 4, 2020
  6. 5/7 maintenance: add start/stop subcommandsDerrick Stolee via GitGitGadget, Sep 4, 2020
  7. SZEDER GáborSep 8, 2020
  8. Derrick StoleeSep 8, 2020
  9. Junio C HamanoSep 8, 2020
  10. 2/7 maintenance: add --schedule option and configDerrick Stolee via GitGitGadget, Sep 4, 2020
  11. Đoàn Trần Công DanhSep 8, 2020
  12. Derrick StoleeSep 9, 2020
  13. 6/7 maintenance: recommended schedule in register/startDerrick Stolee via GitGitGadget, Sep 4, 2020
  14. 0/7 Maintenance III: Background maintenanceDerrick Stolee via GitGitGadget, Sep 11, 2020
  15. 1/7 maintenance: optionally skip --auto processDerrick Stolee via GitGitGadget, Sep 11, 2020
  16. 2/7 maintenance: add --schedule option and configDerrick Stolee via GitGitGadget, Sep 11, 2020
  17. 3/7 for-each-repo: run subcommands on configured reposDerrick Stolee via GitGitGadget, Sep 11, 2020
  18. 4/7 maintenance: add [un]register subcommandsDerrick Stolee via GitGitGadget, Sep 11, 2020
  19. Đoàn Trần Công DanhSep 17, 2020
  20. 5/7 maintenance: add start/stop subcommandsDerrick Stolee via GitGitGadget, Sep 11, 2020
  21. 7/7 maintenance: add troubleshooting guide to docsDerrick Stolee via GitGitGadget, Sep 11, 2020
  22. 6/7 maintenance: recommended schedule in register/startDerrick Stolee via GitGitGadget, Sep 11, 2020
  23. Martin ÅgrenSep 29, 2020
  24. Derrick StoleeSep 30, 2020
  25. Derrick StoleeOct 1, 2020
  26. Đoàn Trần Công DanhOct 2, 2020
  27. Derrick StoleeOct 2, 2020
  28. Đoàn Trần Công DanhOct 5, 2020
  29. Derrick StoleeOct 5, 2020
  30. 0/7 Maintenance III: Background maintenanceDerrick Stolee via GitGitGadget, Oct 5, 2020
  31. 3/7 for-each-repo: run subcommands on configured reposDerrick Stolee via GitGitGadget, Oct 5, 2020
  32. 2/7 maintenance: add --schedule option and configDerrick Stolee via GitGitGadget, Oct 5, 2020
  33. 6/7 maintenance: use default schedule if not configuredDerrick Stolee via GitGitGadget, Oct 5, 2020
  34. Martin ÅgrenOct 5, 2020
  35. Derrick StoleeOct 8, 2020
  36. 5/7 maintenance: add start/stop subcommandsDerrick Stolee via GitGitGadget, Oct 5, 2020
  37. 4/7 maintenance: add [un]register subcommandsDerrick Stolee via GitGitGadget, Oct 5, 2020
  38. 7/7 maintenance: add troubleshooting guide to docsDerrick Stolee via GitGitGadget, Oct 5, 2020
  39. 1/7 maintenance: optionally skip --auto processDerrick Stolee via GitGitGadget, Oct 5, 2020
  40. 0/8 Maintenance III: Background maintenanceDerrick Stolee via GitGitGadget, Oct 15, 2020
  41. 1/8 maintenance: optionally skip --auto processDerrick Stolee via GitGitGadget, Oct 15, 2020
  42. 3/8 for-each-repo: run subcommands on configured reposDerrick Stolee via GitGitGadget, Oct 15, 2020
  43. Andrzej HuntMay 3, 2021
  44. Eric SunshineMay 3, 2021
  45. Eric SunshineMay 3, 2021
  46. Derrick StoleeMay 3, 2021
  47. 5/8 maintenance: add start/stop subcommandsDerrick Stolee via GitGitGadget, Oct 15, 2020
  48. Josh SteadmonDec 9, 2020
  49. Josh SteadmonDec 9, 2020
  50. Derrick StoleeDec 9, 2020
  51. Junio C HamanoDec 10, 2020
  52. Derrick StoleeDec 10, 2020
  53. Junio C HamanoDec 10, 2020
  54. 7/8 maintenance: use 'incremental' strategy by defaultDerrick Stolee via GitGitGadget, Oct 15, 2020
  55. 2/8 maintenance: add --schedule option and configDerrick Stolee via GitGitGadget, Oct 15, 2020
  56. Ævar Arnfjörð BjarmasonFeb 9, 2021
  57. Derrick StoleeFeb 9, 2021
  58. Ævar Arnfjörð BjarmasonMay 10, 2021
  59. Junio C HamanoMay 10, 2021
  60. 6/8 maintenance: create maintenance.strategy configDerrick Stolee via GitGitGadget, Oct 15, 2020
  61. 4/8 maintenance: add [un]register subcommandsDerrick Stolee via GitGitGadget, Oct 15, 2020
  62. 8/8 maintenance: add troubleshooting guide to docsDerrick Stolee via GitGitGadget, Oct 15, 2020

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.