{"thread":{"id":"61940","subject":"[PATCH 0/7] builtin/maintenance: fix auto-detach with non-standard tasks","startedAt":"2024-08-13T07:17:42Z","lastAt":"2024-08-22T17:22:31Z","messageCount":79,"participants":["Patrick Steinhardt","Phillip Wood","James Liu","Derrick Stolee","Junio C Hamano","Jeff King"],"isPatch":true,"patchVersion":1,"patchTotal":7},"messages":[{"id":"500692","messageId":"cover.1723533091.git.ps@pks.im","threadId":"61940","inReplyTo":null,"subject":"[PATCH 0/7] builtin/maintenance: fix auto-detach with non-standard tasks","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-08-13T07:17:38Z","receivedAt":"2024-08-13T07:17:42Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Hi,\n\nI recently configured git-maintenance(1) to not use git-gc(1) anymore,\nbut instead to use git-multi-pack-index(1). I quickly noticed that the\nbehaviour here is somewhat broken because instead of auto-detaching when\n`git maintenance run --auto` executes, we wait for the process to run to\ncompletion.\n\nThe root cause is that git-maintenance(1), probably by accident,\ncontinues to rely on the auto-detaching mechanism in git-gc(1). So\ninstead of having git-maintenance(1) detach, it is git-gc(1) that\ndetaches and thus causes git-maintenance(1) to exit early. That of\ncourse falls flat once any maintenance task other than git-gc(1)\nexecutes, because these won't detach.\n\nDespite being a usability issue, this may also cause git-gc(1) to run\nconcurrently with any other enabled maintenance tasks. This shouldn't\nlead to data loss, but it can certainly lead to processes stomping on\neach others feet.\n\nThis patch series fixes this by wiring up new `--detach` flags for both\ngit-gc(1) and git-maintenance(1). Like this, git-maintenance(1) now\nknows to execute `git gc --auto --no-detach`, while our auto-maintenance\nwill execute `git mainteance run --auto --detach`.\n\nPatrick\n\nPatrick Steinhardt (7):\n config: fix constness of out parameter for `git_config_get_expiry()`\n builtin/gc: refactor to read config into structure\n builtin/gc: fix leaking config values\n builtin/gc: stop processing log file on signal\n builtin/gc: add a `--detach` flag\n builtin/maintenance: add a `--detach` flag\n builtin/maintenance: fix auto-detach with non-standard tasks\n\n Documentation/git-gc.txt | 5 +-\n builtin/gc.c | 384 ++++++++++++++++++++++++---------------\n config.c | 4 +-\n config.h | 2 +-\n read-cache.c | 12 +-\n run-command.c | 12 +-\n t/t5304-prune.sh | 1 +\n t/t5616-partial-clone.sh | 6 +-\n t/t6500-gc.sh | 39 ++++\n t/t7900-maintenance.sh | 82 ++++++++-\n 10 files changed, 381 insertions(+), 166 deletions(-)\n\n-- \n2.46.0.46.g406f326d27.dirty\n\n"},{"id":"500693","messageId":"040453f27f0b2755bb359fe07a703ce66e071ab6.1723533091.git.ps@pks.im","threadId":"61940","inReplyTo":"cover.1723533091.git.ps@pks.im","subject":"[PATCH 1/7] config: fix constness of out parameter for `git_config_get_expiry()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-08-13T07:17:40Z","receivedAt":"2024-08-13T07:17:44Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The type of the out parameter of `git_config_get_expiry()` is a pointer\nto a constant string, which creates the impression that ownership of the\nreturned data wasn't transferred to the caller. This isn't true though\nand thus quite misleading.\n\nAdapt the parameter to be of type `char **` and adjust callers\naccordingly. While at it, refactor `get_shared_index_expire_date()` to\ndrop the static `shared_index_expire` variable. It is only used in that\nfunction, and furthermore we would only hit the code where we parse the\nexpiry date a single time because we already use a static `prepared`\nvariable to track whether we did parse it.\n\nSigned-off-by: Patrick Steinhardt \n---\n builtin/gc.c | 6 +++---\n config.c | 4 ++--\n config.h | 2 +-\n read-cache.c | 12 +++++++++---\n 4 files changed, 15 insertions(+), 9 deletions(-)\n\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 72bac2554f..e7406bf667 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -167,9 +167,9 @@ static void gc_config(void)\n \tgit_config_get_bool(\"gc.autodetach\", &detach_auto);\n \tgit_config_get_bool(\"gc.cruftpacks\", &cruft_packs);\n \tgit_config_get_ulong(\"gc.maxcruftsize\", &max_cruft_size);\n-\tgit_config_get_expiry(\"gc.pruneexpire\", &prune_expire);\n-\tgit_config_get_expiry(\"gc.worktreepruneexpire\", &prune_worktrees_expire);\n-\tgit_config_get_expiry(\"gc.logexpiry\", &gc_log_expire);\n+\tgit_config_get_expiry(\"gc.pruneexpire\", (char **) &prune_expire);\n+\tgit_config_get_expiry(\"gc.worktreepruneexpire\", (char **) &prune_worktrees_expire);\n+\tgit_config_get_expiry(\"gc.logexpiry\", (char **) &gc_log_expire);\n \n \tgit_config_get_ulong(\"gc.bigpackthreshold\", &big_pack_threshold);\n \tgit_config_get_ulong(\"pack.deltacachesize\", &max_delta_cache_size);\ndiff --git a/config.c b/config.c\nindex 6421894614..dfa4df1417 100644\n--- a/config.c\n+++ b/config.c\n@@ -2766,9 +2766,9 @@ int git_config_get_pathname(const char *key, char **dest)\n \treturn repo_config_get_pathname(the_repository, key, dest);\n }\n \n-int git_config_get_expiry(const char *key, const char **output)\n+int git_config_get_expiry(const char *key, char **output)\n {\n-\tint ret = git_config_get_string(key, (char **)output);\n+\tint ret = git_config_get_string(key, output);\n \tif (ret)\n \t\treturn ret;\n \tif (strcmp(*output, \"now\")) {\ndiff --git a/config.h b/config.h\nindex 54b47dec9e..4801391c32 100644\n--- a/config.h\n+++ b/config.h\n@@ -701,7 +701,7 @@ int git_config_get_split_index(void);\n int git_config_get_max_percent_split_change(void);\n \n /* This dies if the configured or default date is in the future */\n-int git_config_get_expiry(const char *key, const char **output);\n+int git_config_get_expiry(const char *key, char **output);\n \n /* parse either \"this many days\" integer, or \"5.days.ago\" approxidate */\n int git_config_get_expiry_in_days(const char *key, timestamp_t *, timestamp_t now);\ndiff --git a/read-cache.c b/read-cache.c\nindex 48bf24f87c..7f393ee687 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -3176,18 +3176,24 @@ static int write_split_index(struct index_state *istate,\n \treturn ret;\n }\n \n-static const char *shared_index_expire = \"2.weeks.ago\";\n-\n static unsigned long get_shared_index_expire_date(void)\n {\n \tstatic unsigned long shared_index_expire_date;\n \tstatic int shared_index_expire_date_prepared;\n \n \tif (!shared_index_expire_date_prepared) {\n+\t\tconst char *shared_index_expire = \"2.weeks.ago\";\n+\t\tchar *value = NULL;\n+\n \t\tgit_config_get_expiry(\"splitindex.sharedindexexpire\",\n-\t\t\t\t &shared_index_expire);\n+\t\t\t\t &value);\n+\t\tif (value)\n+\t\t\tshared_index_expire = value;\n+\n \t\tshared_index_expire_date = approxidate(shared_index_expire);\n \t\tshared_index_expire_date_prepared = 1;\n+\n+\t\tfree(value);\n \t}\n \n \treturn shared_index_expire_date;\n-- \n2.46.0.46.g406f326d27.dirty\n\n"},{"id":"500694","messageId":"ff6aa9d7ba0bd67d3682ecb44c024a6d59a7decd.1723533091.git.ps@pks.im","threadId":"61940","inReplyTo":"cover.1723533091.git.ps@pks.im","subject":"[PATCH 2/7] builtin/gc: refactor to read config into structure","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-08-13T07:17:43Z","receivedAt":"2024-08-13T07:17:46Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The git-gc(1) command knows to read a bunch of config keys to tweak its\nown behaviour. The values are parsed into global variables, which makes\nit hard to correctly manage the lifecycle of values that may require a\nmemory allocation.\n\nRefactor the code to use a `struct gc_config` that gets populated and\npassed around. For one, this makes previously-implicit dependencies on\nthese config values clear. Second, it will allow us to properly manage\nthe lifecycle in the next commit.\n\nSigned-off-by: Patrick Steinhardt \n---\n builtin/gc.c | 255 +++++++++++++++++++++++++++++----------------------\n 1 file changed, 143 insertions(+), 112 deletions(-)\n\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex e7406bf667..eee7401647 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -49,23 +49,7 @@ static const char * const builtin_gc_usage[] = {\n \tNULL\n };\n \n-static int pack_refs = 1;\n-static int prune_reflogs = 1;\n-static int cruft_packs = 1;\n-static unsigned long max_cruft_size;\n-static int aggressive_depth = 50;\n-static int aggressive_window = 250;\n-static int gc_auto_threshold = 6700;\n-static int gc_auto_pack_limit = 50;\n-static int detach_auto = 1;\n static timestamp_t gc_log_expire_time;\n-static const char *gc_log_expire = \"1.day.ago\";\n-static const char *prune_expire = \"2.weeks.ago\";\n-static const char *prune_worktrees_expire = \"3.months.ago\";\n-static char *repack_filter;\n-static char *repack_filter_to;\n-static unsigned long big_pack_threshold;\n-static unsigned long max_delta_cache_size = DEFAULT_DELTA_CACHE_SIZE;\n \n static struct strvec reflog = STRVEC_INIT;\n static struct strvec repack = STRVEC_INIT;\n@@ -145,37 +129,71 @@ static int gc_config_is_timestamp_never(const char *var)\n \treturn 0;\n }\n \n-static void gc_config(void)\n+struct gc_config {\n+\tint pack_refs;\n+\tint prune_reflogs;\n+\tint cruft_packs;\n+\tunsigned long max_cruft_size;\n+\tint aggressive_depth;\n+\tint aggressive_window;\n+\tint gc_auto_threshold;\n+\tint gc_auto_pack_limit;\n+\tint detach_auto;\n+\tconst char *gc_log_expire;\n+\tconst char *prune_expire;\n+\tconst char *prune_worktrees_expire;\n+\tchar *repack_filter;\n+\tchar *repack_filter_to;\n+\tunsigned long big_pack_threshold;\n+\tunsigned long max_delta_cache_size;\n+};\n+\n+#define GC_CONFIG_INIT { \\\n+\t.pack_refs = 1, \\\n+\t.prune_reflogs = 1, \\\n+\t.cruft_packs = 1, \\\n+\t.aggressive_depth = 50, \\\n+\t.aggressive_window = 250, \\\n+\t.gc_auto_threshold = 6700, \\\n+\t.gc_auto_pack_limit = 50, \\\n+\t.detach_auto = 1, \\\n+\t.gc_log_expire = \"1.day.ago\", \\\n+\t.prune_expire = \"2.weeks.ago\", \\\n+\t.prune_worktrees_expire = \"3.months.ago\", \\\n+\t.max_delta_cache_size = DEFAULT_DELTA_CACHE_SIZE, \\\n+}\n+\n+static void gc_config(struct gc_config *cfg)\n {\n \tconst char *value;\n \n \tif (!git_config_get_value(\"gc.packrefs\", &value)) {\n \t\tif (value && !strcmp(value, \"notbare\"))\n-\t\t\tpack_refs = -1;\n+\t\t\tcfg->pack_refs = -1;\n \t\telse\n-\t\t\tpack_refs = git_config_bool(\"gc.packrefs\", value);\n+\t\t\tcfg->pack_refs = git_config_bool(\"gc.packrefs\", value);\n \t}\n \n \tif (gc_config_is_timestamp_never(\"gc.reflogexpire\") &&\n \t gc_config_is_timestamp_never(\"gc.reflogexpireunreachable\"))\n-\t\tprune_reflogs = 0;\n+\t\tcfg->prune_reflogs = 0;\n \n-\tgit_config_get_int(\"gc.aggressivewindow\", &aggressive_window);\n-\tgit_config_get_int(\"gc.aggressivedepth\", &aggressive_depth);\n-\tgit_config_get_int(\"gc.auto\", &gc_auto_threshold);\n-\tgit_config_get_int(\"gc.autopacklimit\", &gc_auto_pack_limit);\n-\tgit_config_get_bool(\"gc.autodetach\", &detach_auto);\n-\tgit_config_get_bool(\"gc.cruftpacks\", &cruft_packs);\n-\tgit_config_get_ulong(\"gc.maxcruftsize\", &max_cruft_size);\n-\tgit_config_get_expiry(\"gc.pruneexpire\", (char **) &prune_expire);\n-\tgit_config_get_expiry(\"gc.worktreepruneexpire\", (char **) &prune_worktrees_expire);\n-\tgit_config_get_expiry(\"gc.logexpiry\", (char **) &gc_log_expire);\n+\tgit_config_get_int(\"gc.aggressivewindow\", &cfg->aggressive_window);\n+\tgit_config_get_int(\"gc.aggressivedepth\", &cfg->aggressive_depth);\n+\tgit_config_get_int(\"gc.auto\", &cfg->gc_auto_threshold);\n+\tgit_config_get_int(\"gc.autopacklimit\", &cfg->gc_auto_pack_limit);\n+\tgit_config_get_bool(\"gc.autodetach\", &cfg->detach_auto);\n+\tgit_config_get_bool(\"gc.cruftpacks\", &cfg->cruft_packs);\n+\tgit_config_get_ulong(\"gc.maxcruftsize\", &cfg->max_cruft_size);\n+\tgit_config_get_expiry(\"gc.pruneexpire\", (char **) &cfg->prune_expire);\n+\tgit_config_get_expiry(\"gc.worktreepruneexpire\", (char **) &cfg->prune_worktrees_expire);\n+\tgit_config_get_expiry(\"gc.logexpiry\", (char **) &cfg->gc_log_expire);\n \n-\tgit_config_get_ulong(\"gc.bigpackthreshold\", &big_pack_threshold);\n-\tgit_config_get_ulong(\"pack.deltacachesize\", &max_delta_cache_size);\n+\tgit_config_get_ulong(\"gc.bigpackthreshold\", &cfg->big_pack_threshold);\n+\tgit_config_get_ulong(\"pack.deltacachesize\", &cfg->max_delta_cache_size);\n \n-\tgit_config_get_string(\"gc.repackfilter\", &repack_filter);\n-\tgit_config_get_string(\"gc.repackfilterto\", &repack_filter_to);\n+\tgit_config_get_string(\"gc.repackfilter\", &cfg->repack_filter);\n+\tgit_config_get_string(\"gc.repackfilterto\", &cfg->repack_filter_to);\n \n \tgit_config(git_default_config, NULL);\n }\n@@ -206,7 +224,7 @@ struct maintenance_run_opts {\n \tenum schedule_priority schedule;\n };\n \n-static int pack_refs_condition(void)\n+static int pack_refs_condition(UNUSED struct gc_config *cfg)\n {\n \t/*\n \t * The auto-repacking logic for refs is handled by the ref backends and\n@@ -216,7 +234,8 @@ static int pack_refs_condition(void)\n \treturn 1;\n }\n \n-static int maintenance_task_pack_refs(MAYBE_UNUSED struct maintenance_run_opts *opts)\n+static int maintenance_task_pack_refs(MAYBE_UNUSED struct maintenance_run_opts *opts,\n+\t\t\t\t UNUSED struct gc_config *cfg)\n {\n \tstruct child_process cmd = CHILD_PROCESS_INIT;\n \n@@ -228,7 +247,7 @@ static int maintenance_task_pack_refs(MAYBE_UNUSED struct maintenance_run_opts *\n \treturn run_command(&cmd);\n }\n \n-static int too_many_loose_objects(void)\n+static int too_many_loose_objects(struct gc_config *cfg)\n {\n \t/*\n \t * Quickly check if a \"gc\" is needed, by estimating how\n@@ -247,7 +266,7 @@ static int too_many_loose_objects(void)\n \tif (!dir)\n \t\treturn 0;\n \n-\tauto_threshold = DIV_ROUND_UP(gc_auto_threshold, 256);\n+\tauto_threshold = DIV_ROUND_UP(cfg->gc_auto_threshold, 256);\n \twhile ((ent = readdir(dir)) != NULL) {\n \t\tif (strspn(ent->d_name, \"0123456789abcdef\") != hexsz_loose ||\n \t\t ent->d_name[hexsz_loose] != '\\0')\n@@ -283,12 +302,12 @@ static struct packed_git *find_base_packs(struct string_list *packs,\n \treturn base;\n }\n \n-static int too_many_packs(void)\n+static int too_many_packs(struct gc_config *cfg)\n {\n \tstruct packed_git *p;\n \tint cnt;\n \n-\tif (gc_auto_pack_limit <= 0)\n+\tif (cfg->gc_auto_pack_limit <= 0)\n \t\treturn 0;\n \n \tfor (cnt = 0, p = get_all_packs(the_repository); p; p = p->next) {\n@@ -302,7 +321,7 @@ static int too_many_packs(void)\n \t\t */\n \t\tcnt++;\n \t}\n-\treturn gc_auto_pack_limit < cnt;\n+\treturn cfg->gc_auto_pack_limit < cnt;\n }\n \n static uint64_t total_ram(void)\n@@ -336,7 +355,8 @@ static uint64_t total_ram(void)\n \treturn 0;\n }\n \n-static uint64_t estimate_repack_memory(struct packed_git *pack)\n+static uint64_t estimate_repack_memory(struct gc_config *cfg,\n+\t\t\t\t struct packed_git *pack)\n {\n \tunsigned long nr_objects = repo_approximate_object_count(the_repository);\n \tsize_t os_cache, heap;\n@@ -373,7 +393,7 @@ static uint64_t estimate_repack_memory(struct packed_git *pack)\n \t */\n \theap += delta_base_cache_limit;\n \t/* and of course pack-objects has its own delta cache */\n-\theap += max_delta_cache_size;\n+\theap += cfg->max_delta_cache_size;\n \n \treturn os_cache + heap;\n }\n@@ -384,30 +404,31 @@ static int keep_one_pack(struct string_list_item *item, void *data UNUSED)\n \treturn 0;\n }\n \n-static void add_repack_all_option(struct string_list *keep_pack)\n+static void add_repack_all_option(struct gc_config *cfg,\n+\t\t\t\t struct string_list *keep_pack)\n {\n-\tif (prune_expire && !strcmp(prune_expire, \"now\"))\n+\tif (cfg->prune_expire && !strcmp(cfg->prune_expire, \"now\"))\n \t\tstrvec_push(&repack, \"-a\");\n-\telse if (cruft_packs) {\n+\telse if (cfg->cruft_packs) {\n \t\tstrvec_push(&repack, \"--cruft\");\n-\t\tif (prune_expire)\n-\t\t\tstrvec_pushf(&repack, \"--cruft-expiration=%s\", prune_expire);\n-\t\tif (max_cruft_size)\n+\t\tif (cfg->prune_expire)\n+\t\t\tstrvec_pushf(&repack, \"--cruft-expiration=%s\", cfg->prune_expire);\n+\t\tif (cfg->max_cruft_size)\n \t\t\tstrvec_pushf(&repack, \"--max-cruft-size=%lu\",\n-\t\t\t\t max_cruft_size);\n+\t\t\t\t cfg->max_cruft_size);\n \t} else {\n \t\tstrvec_push(&repack, \"-A\");\n-\t\tif (prune_expire)\n-\t\t\tstrvec_pushf(&repack, \"--unpack-unreachable=%s\", prune_expire);\n+\t\tif (cfg->prune_expire)\n+\t\t\tstrvec_pushf(&repack, \"--unpack-unreachable=%s\", cfg->prune_expire);\n \t}\n \n \tif (keep_pack)\n \t\tfor_each_string_list(keep_pack, keep_one_pack, NULL);\n \n-\tif (repack_filter && *repack_filter)\n-\t\tstrvec_pushf(&repack, \"--filter=%s\", repack_filter);\n-\tif (repack_filter_to && *repack_filter_to)\n-\t\tstrvec_pushf(&repack, \"--filter-to=%s\", repack_filter_to);\n+\tif (cfg->repack_filter && *cfg->repack_filter)\n+\t\tstrvec_pushf(&repack, \"--filter=%s\", cfg->repack_filter);\n+\tif (cfg->repack_filter_to && *cfg->repack_filter_to)\n+\t\tstrvec_pushf(&repack, \"--filter-to=%s\", cfg->repack_filter_to);\n }\n \n static void add_repack_incremental_option(void)\n@@ -415,13 +436,13 @@ static void add_repack_incremental_option(void)\n \tstrvec_push(&repack, \"--no-write-bitmap-index\");\n }\n \n-static int need_to_gc(void)\n+static int need_to_gc(struct gc_config *cfg)\n {\n \t/*\n \t * Setting gc.auto to 0 or negative can disable the\n \t * automatic gc.\n \t */\n-\tif (gc_auto_threshold <= 0)\n+\tif (cfg->gc_auto_threshold <= 0)\n \t\treturn 0;\n \n \t/*\n@@ -430,13 +451,13 @@ static int need_to_gc(void)\n \t * we run \"repack -A -d -l\". Otherwise we tell the caller\n \t * there is no need.\n \t */\n-\tif (too_many_packs()) {\n+\tif (too_many_packs(cfg)) {\n \t\tstruct string_list keep_pack = STRING_LIST_INIT_NODUP;\n \n-\t\tif (big_pack_threshold) {\n-\t\t\tfind_base_packs(&keep_pack, big_pack_threshold);\n-\t\t\tif (keep_pack.nr >= gc_auto_pack_limit) {\n-\t\t\t\tbig_pack_threshold = 0;\n+\t\tif (cfg->big_pack_threshold) {\n+\t\t\tfind_base_packs(&keep_pack, cfg->big_pack_threshold);\n+\t\t\tif (keep_pack.nr >= cfg->gc_auto_pack_limit) {\n+\t\t\t\tcfg->big_pack_threshold = 0;\n \t\t\t\tstring_list_clear(&keep_pack, 0);\n \t\t\t\tfind_base_packs(&keep_pack, 0);\n \t\t\t}\n@@ -445,7 +466,7 @@ static int need_to_gc(void)\n \t\t\tuint64_t mem_have, mem_want;\n \n \t\t\tmem_have = total_ram();\n-\t\t\tmem_want = estimate_repack_memory(p);\n+\t\t\tmem_want = estimate_repack_memory(cfg, p);\n \n \t\t\t/*\n \t\t\t * Only allow 1/2 of memory for pack-objects, leave\n@@ -456,9 +477,9 @@ static int need_to_gc(void)\n \t\t\t\tstring_list_clear(&keep_pack, 0);\n \t\t}\n \n-\t\tadd_repack_all_option(&keep_pack);\n+\t\tadd_repack_all_option(cfg, &keep_pack);\n \t\tstring_list_clear(&keep_pack, 0);\n-\t} else if (too_many_loose_objects())\n+\t} else if (too_many_loose_objects(cfg))\n \t\tadd_repack_incremental_option();\n \telse\n \t\treturn 0;\n@@ -585,7 +606,8 @@ static int report_last_gc_error(void)\n \treturn ret;\n }\n \n-static void gc_before_repack(struct maintenance_run_opts *opts)\n+static void gc_before_repack(struct maintenance_run_opts *opts,\n+\t\t\t struct gc_config *cfg)\n {\n \t/*\n \t * We may be called twice, as both the pre- and\n@@ -596,10 +618,10 @@ static void gc_before_repack(struct maintenance_run_opts *opts)\n \tif (done++)\n \t\treturn;\n \n-\tif (pack_refs && maintenance_task_pack_refs(opts))\n+\tif (cfg->pack_refs && maintenance_task_pack_refs(opts, cfg))\n \t\tdie(FAILED_RUN, \"pack-refs\");\n \n-\tif (prune_reflogs) {\n+\tif (cfg->prune_reflogs) {\n \t\tstruct child_process cmd = CHILD_PROCESS_INIT;\n \n \t\tcmd.git_cmd = 1;\n@@ -621,14 +643,15 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \ttimestamp_t dummy;\n \tstruct child_process rerere_cmd = CHILD_PROCESS_INIT;\n \tstruct maintenance_run_opts opts = {0};\n+\tstruct gc_config cfg = GC_CONFIG_INIT;\n \n \tstruct option builtin_gc_options[] = {\n \t\tOPT__QUIET(&quiet, N_(\"suppress progress reporting\")),\n-\t\t{ OPTION_STRING, 0, \"prune\", &prune_expire, N_(\"date\"),\n+\t\t{ OPTION_STRING, 0, \"prune\", &cfg.prune_expire, N_(\"date\"),\n \t\t\tN_(\"prune unreferenced objects\"),\n-\t\t\tPARSE_OPT_OPTARG, NULL, (intptr_t)prune_expire },\n-\t\tOPT_BOOL(0, \"cruft\", &cruft_packs, N_(\"pack unreferenced objects separately\")),\n-\t\tOPT_MAGNITUDE(0, \"max-cruft-size\", &max_cruft_size,\n+\t\t\tPARSE_OPT_OPTARG, NULL, (intptr_t)cfg.prune_expire },\n+\t\tOPT_BOOL(0, \"cruft\", &cfg.cruft_packs, N_(\"pack unreferenced objects separately\")),\n+\t\tOPT_MAGNITUDE(0, \"max-cruft-size\", &cfg.max_cruft_size,\n \t\t\t N_(\"with --cruft, limit the size of new cruft packs\")),\n \t\tOPT_BOOL(0, \"aggressive\", &aggressive, N_(\"be more thorough (increased runtime)\")),\n \t\tOPT_BOOL_F(0, \"auto\", &opts.auto_flag, N_(\"enable auto-gc mode\"),\n@@ -651,27 +674,27 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \tstrvec_pushl(&rerere, \"rerere\", \"gc\", NULL);\n \n \t/* default expiry time, overwritten in gc_config */\n-\tgc_config();\n-\tif (parse_expiry_date(gc_log_expire, &gc_log_expire_time))\n-\t\tdie(_(\"failed to parse gc.logExpiry value %s\"), gc_log_expire);\n+\tgc_config(&cfg);\n+\tif (parse_expiry_date(cfg.gc_log_expire, &gc_log_expire_time))\n+\t\tdie(_(\"failed to parse gc.logExpiry value %s\"), cfg.gc_log_expire);\n \n-\tif (pack_refs < 0)\n-\t\tpack_refs = !is_bare_repository();\n+\tif (cfg.pack_refs < 0)\n+\t\tcfg.pack_refs = !is_bare_repository();\n \n \targc = parse_options(argc, argv, prefix, builtin_gc_options,\n \t\t\t builtin_gc_usage, 0);\n \tif (argc > 0)\n \t\tusage_with_options(builtin_gc_usage, builtin_gc_options);\n \n-\tif (prune_expire && parse_expiry_date(prune_expire, &dummy))\n-\t\tdie(_(\"failed to parse prune expiry value %s\"), prune_expire);\n+\tif (cfg.prune_expire && parse_expiry_date(cfg.prune_expire, &dummy))\n+\t\tdie(_(\"failed to parse prune expiry value %s\"), cfg.prune_expire);\n \n \tif (aggressive) {\n \t\tstrvec_push(&repack, \"-f\");\n-\t\tif (aggressive_depth > 0)\n-\t\t\tstrvec_pushf(&repack, \"--depth=%d\", aggressive_depth);\n-\t\tif (aggressive_window > 0)\n-\t\t\tstrvec_pushf(&repack, \"--window=%d\", aggressive_window);\n+\t\tif (cfg.aggressive_depth > 0)\n+\t\t\tstrvec_pushf(&repack, \"--depth=%d\", cfg.aggressive_depth);\n+\t\tif (cfg.aggressive_window > 0)\n+\t\t\tstrvec_pushf(&repack, \"--window=%d\", cfg.aggressive_window);\n \t}\n \tif (quiet)\n \t\tstrvec_push(&repack, \"-q\");\n@@ -680,16 +703,16 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \t\t/*\n \t\t * Auto-gc should be least intrusive as possible.\n \t\t */\n-\t\tif (!need_to_gc())\n+\t\tif (!need_to_gc(&cfg))\n \t\t\treturn 0;\n \t\tif (!quiet) {\n-\t\t\tif (detach_auto)\n+\t\t\tif (cfg.detach_auto)\n \t\t\t\tfprintf(stderr, _(\"Auto packing the repository in background for optimum performance.\\n\"));\n \t\t\telse\n \t\t\t\tfprintf(stderr, _(\"Auto packing the repository for optimum performance.\\n\"));\n \t\t\tfprintf(stderr, _(\"See \\\"git help gc\\\" for manual housekeeping.\\n\"));\n \t\t}\n-\t\tif (detach_auto) {\n+\t\tif (cfg.detach_auto) {\n \t\t\tint ret = report_last_gc_error();\n \n \t\t\tif (ret == 1)\n@@ -701,7 +724,7 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \n \t\t\tif (lock_repo_for_gc(force, &pid))\n \t\t\t\treturn 0;\n-\t\t\tgc_before_repack(&opts); /* dies on failure */\n+\t\t\tgc_before_repack(&opts, &cfg); /* dies on failure */\n \t\t\tdelete_tempfile(&pidfile);\n \n \t\t\t/*\n@@ -716,11 +739,11 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \t\tif (keep_largest_pack != -1) {\n \t\t\tif (keep_largest_pack)\n \t\t\t\tfind_base_packs(&keep_pack, 0);\n-\t\t} else if (big_pack_threshold) {\n-\t\t\tfind_base_packs(&keep_pack, big_pack_threshold);\n+\t\t} else if (cfg.big_pack_threshold) {\n+\t\t\tfind_base_packs(&keep_pack, cfg.big_pack_threshold);\n \t\t}\n \n-\t\tadd_repack_all_option(&keep_pack);\n+\t\tadd_repack_all_option(&cfg, &keep_pack);\n \t\tstring_list_clear(&keep_pack, 0);\n \t}\n \n@@ -741,7 +764,7 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \t\tatexit(process_log_file_at_exit);\n \t}\n \n-\tgc_before_repack(&opts);\n+\tgc_before_repack(&opts, &cfg);\n \n \tif (!repository_format_precious_objects) {\n \t\tstruct child_process repack_cmd = CHILD_PROCESS_INIT;\n@@ -752,11 +775,11 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \t\tif (run_command(&repack_cmd))\n \t\t\tdie(FAILED_RUN, repack.v[0]);\n \n-\t\tif (prune_expire) {\n+\t\tif (cfg.prune_expire) {\n \t\t\tstruct child_process prune_cmd = CHILD_PROCESS_INIT;\n \n \t\t\t/* run `git prune` even if using cruft packs */\n-\t\t\tstrvec_push(&prune, prune_expire);\n+\t\t\tstrvec_push(&prune, cfg.prune_expire);\n \t\t\tif (quiet)\n \t\t\t\tstrvec_push(&prune, \"--no-progress\");\n \t\t\tif (repo_has_promisor_remote(the_repository))\n@@ -769,10 +792,10 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \t\t}\n \t}\n \n-\tif (prune_worktrees_expire) {\n+\tif (cfg.prune_worktrees_expire) {\n \t\tstruct child_process prune_worktrees_cmd = CHILD_PROCESS_INIT;\n \n-\t\tstrvec_push(&prune_worktrees, prune_worktrees_expire);\n+\t\tstrvec_push(&prune_worktrees, cfg.prune_worktrees_expire);\n \t\tprune_worktrees_cmd.git_cmd = 1;\n \t\tstrvec_pushv(&prune_worktrees_cmd.args, prune_worktrees.v);\n \t\tif (run_command(&prune_worktrees_cmd))\n@@ -796,7 +819,7 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \t\t\t\t\t !quiet && !daemonized ? COMMIT_GRAPH_WRITE_PROGRESS : 0,\n \t\t\t\t\t NULL);\n \n-\tif (opts.auto_flag && too_many_loose_objects())\n+\tif (opts.auto_flag && too_many_loose_objects(&cfg))\n \t\twarning(_(\"There are too many unreachable loose objects; \"\n \t\t\t\"run 'git prune' to remove them.\"));\n \n@@ -892,7 +915,7 @@ static int dfs_on_ref(const char *refname UNUSED,\n \treturn result;\n }\n \n-static int should_write_commit_graph(void)\n+static int should_write_commit_graph(struct gc_config *cfg)\n {\n \tint result;\n \tstruct cg_auto_data data;\n@@ -929,7 +952,8 @@ static int run_write_commit_graph(struct maintenance_run_opts *opts)\n \treturn !!run_command(&child);\n }\n \n-static int maintenance_task_commit_graph(struct maintenance_run_opts *opts)\n+static int maintenance_task_commit_graph(struct maintenance_run_opts *opts,\n+\t\t\t\t\t struct gc_config *cfg)\n {\n \tprepare_repo_settings(the_repository);\n \tif (!the_repository->settings.core_commit_graph)\n@@ -963,7 +987,8 @@ static int fetch_remote(struct remote *remote, void *cbdata)\n \treturn !!run_command(&child);\n }\n \n-static int maintenance_task_prefetch(struct maintenance_run_opts *opts)\n+static int maintenance_task_prefetch(struct maintenance_run_opts *opts,\n+\t\t\t\t struct gc_config *cfg)\n {\n \tif (for_each_remote(fetch_remote, opts)) {\n \t\terror(_(\"failed to prefetch remotes\"));\n@@ -973,7 +998,8 @@ static int maintenance_task_prefetch(struct maintenance_run_opts *opts)\n \treturn 0;\n }\n \n-static int maintenance_task_gc(struct maintenance_run_opts *opts)\n+static int maintenance_task_gc(struct maintenance_run_opts *opts,\n+\t\t\t struct gc_config *cfg)\n {\n \tstruct child_process child = CHILD_PROCESS_INIT;\n \n@@ -1021,7 +1047,7 @@ static int loose_object_count(const struct object_id *oid UNUSED,\n \treturn 0;\n }\n \n-static int loose_object_auto_condition(void)\n+static int loose_object_auto_condition(struct gc_config *cfg)\n {\n \tint count = 0;\n \n@@ -1106,12 +1132,13 @@ static int pack_loose(struct maintenance_run_opts *opts)\n \treturn result;\n }\n \n-static int maintenance_task_loose_objects(struct maintenance_run_opts *opts)\n+static int maintenance_task_loose_objects(struct maintenance_run_opts *opts,\n+\t\t\t\t\t struct gc_config *cfg)\n {\n \treturn prune_packed(opts) || pack_loose(opts);\n }\n \n-static int incremental_repack_auto_condition(void)\n+static int incremental_repack_auto_condition(struct gc_config *cfg)\n {\n \tstruct packed_git *p;\n \tint incremental_repack_auto_limit = 10;\n@@ -1230,7 +1257,8 @@ static int multi_pack_index_repack(struct maintenance_run_opts *opts)\n \treturn 0;\n }\n \n-static int maintenance_task_incremental_repack(struct maintenance_run_opts *opts)\n+static int maintenance_task_incremental_repack(struct maintenance_run_opts *opts,\n+\t\t\t\t\t struct gc_config *cfg)\n {\n \tprepare_repo_settings(the_repository);\n \tif (!the_repository->settings.core_multi_pack_index) {\n@@ -1247,14 +1275,15 @@ static int maintenance_task_incremental_repack(struct maintenance_run_opts *opts\n \treturn 0;\n }\n \n-typedef int maintenance_task_fn(struct maintenance_run_opts *opts);\n+typedef int maintenance_task_fn(struct maintenance_run_opts *opts,\n+\t\t\t\tstruct gc_config *cfg);\n \n /*\n * An auto condition function returns 1 if the task should run\n * and 0 if the task should NOT run. See needs_to_gc() for an\n * example.\n */\n-typedef int maintenance_auto_fn(void);\n+typedef int maintenance_auto_fn(struct gc_config *cfg);\n \n struct maintenance_task {\n \tconst char *name;\n@@ -1321,7 +1350,8 @@ static int compare_tasks_by_selection(const void *a_, const void *b_)\n \treturn b->selected_order - a->selected_order;\n }\n \n-static int maintenance_run_tasks(struct maintenance_run_opts *opts)\n+static int maintenance_run_tasks(struct maintenance_run_opts *opts,\n+\t\t\t\t struct gc_config *cfg)\n {\n \tint i, found_selected = 0;\n \tint result = 0;\n@@ -1360,14 +1390,14 @@ static int maintenance_run_tasks(struct maintenance_run_opts *opts)\n \n \t\tif (opts->auto_flag &&\n \t\t (!tasks[i].auto_condition ||\n-\t\t !tasks[i].auto_condition()))\n+\t\t !tasks[i].auto_condition(cfg)))\n \t\t\tcontinue;\n \n \t\tif (opts->schedule && tasks[i].schedule < opts->schedule)\n \t\t\tcontinue;\n \n \t\ttrace2_region_enter(\"maintenance\", tasks[i].name, r);\n-\t\tif (tasks[i].fn(opts)) {\n+\t\tif (tasks[i].fn(opts, cfg)) {\n \t\t\terror(_(\"task '%s' failed\"), tasks[i].name);\n \t\t\tresult = 1;\n \t\t}\n@@ -1404,7 +1434,6 @@ static void initialize_task_config(int schedule)\n {\n \tint i;\n \tstruct strbuf config_name = STRBUF_INIT;\n-\tgc_config();\n \n \tif (schedule)\n \t\tinitialize_maintenance_strategy();\n@@ -1468,6 +1497,7 @@ static int maintenance_run(int argc, const char **argv, const char *prefix)\n {\n \tint i;\n \tstruct maintenance_run_opts opts;\n+\tstruct gc_config cfg = GC_CONFIG_INIT;\n \tstruct option builtin_maintenance_run_options[] = {\n \t\tOPT_BOOL(0, \"auto\", &opts.auto_flag,\n \t\t\t N_(\"run tasks based on the state of the repository\")),\n@@ -1496,12 +1526,13 @@ static int maintenance_run(int argc, const char **argv, const char *prefix)\n \tif (opts.auto_flag && opts.schedule)\n \t\tdie(_(\"use at most one of --auto and --schedule=\"));\n \n+\tgc_config(&cfg);\n \tinitialize_task_config(opts.schedule);\n \n \tif (argc != 0)\n \t\tusage_with_options(builtin_maintenance_run_usage,\n \t\t\t\t builtin_maintenance_run_options);\n-\treturn maintenance_run_tasks(&opts);\n+\treturn maintenance_run_tasks(&opts, &cfg);\n }\n \n static char *get_maintpath(void)\n-- \n2.46.0.46.g406f326d27.dirty\n\n"},{"id":"500695","messageId":"310e361371efc156c3aaac94439bdbeaa965155f.1723533091.git.ps@pks.im","threadId":"61940","inReplyTo":"cover.1723533091.git.ps@pks.im","subject":"[PATCH 3/7] builtin/gc: fix leaking config values","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-08-13T07:17:46Z","receivedAt":"2024-08-13T07:17:49Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We're leaking config values in git-gc(1) when those values are tracked\nas strings. Introduce a new `gc_config_release()` function that releases\nthis memory to plug those leaks and release old values before populating\nthe config fields via `git_config_string()` et al.\n\nNote that there is one small gotcha here with the \"--prune\" option. Next\nto passing a string, this option also accepts the \"--no-prune\" option\nthat overrides the default or configured value. We thus need to discern\nbetween the option not having been passed by the user and the negative\nvariant of it. This is done by using a simple sentinel value that lets\nus discern these cases.\n\nSigned-off-by: Patrick Steinhardt \n---\n builtin/gc.c | 108 +++++++++++++++++++++++++++++++++++------------\n t/t5304-prune.sh | 1 +\n 2 files changed, 82 insertions(+), 27 deletions(-)\n\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex eee7401647..a93cfa147e 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -139,9 +139,9 @@ struct gc_config {\n \tint gc_auto_threshold;\n \tint gc_auto_pack_limit;\n \tint detach_auto;\n-\tconst char *gc_log_expire;\n-\tconst char *prune_expire;\n-\tconst char *prune_worktrees_expire;\n+\tchar *gc_log_expire;\n+\tchar *prune_expire;\n+\tchar *prune_worktrees_expire;\n \tchar *repack_filter;\n \tchar *repack_filter_to;\n \tunsigned long big_pack_threshold;\n@@ -157,15 +157,25 @@ struct gc_config {\n \t.gc_auto_threshold = 6700, \\\n \t.gc_auto_pack_limit = 50, \\\n \t.detach_auto = 1, \\\n-\t.gc_log_expire = \"1.day.ago\", \\\n-\t.prune_expire = \"2.weeks.ago\", \\\n-\t.prune_worktrees_expire = \"3.months.ago\", \\\n+\t.gc_log_expire = xstrdup(\"1.day.ago\"), \\\n+\t.prune_expire = xstrdup(\"2.weeks.ago\"), \\\n+\t.prune_worktrees_expire = xstrdup(\"3.months.ago\"), \\\n \t.max_delta_cache_size = DEFAULT_DELTA_CACHE_SIZE, \\\n }\n \n+static void gc_config_release(struct gc_config *cfg)\n+{\n+\tfree(cfg->gc_log_expire);\n+\tfree(cfg->prune_expire);\n+\tfree(cfg->prune_worktrees_expire);\n+\tfree(cfg->repack_filter);\n+\tfree(cfg->repack_filter_to);\n+}\n+\n static void gc_config(struct gc_config *cfg)\n {\n \tconst char *value;\n+\tchar *owned = NULL;\n \n \tif (!git_config_get_value(\"gc.packrefs\", &value)) {\n \t\tif (value && !strcmp(value, \"notbare\"))\n@@ -185,15 +195,34 @@ static void gc_config(struct gc_config *cfg)\n \tgit_config_get_bool(\"gc.autodetach\", &cfg->detach_auto);\n \tgit_config_get_bool(\"gc.cruftpacks\", &cfg->cruft_packs);\n \tgit_config_get_ulong(\"gc.maxcruftsize\", &cfg->max_cruft_size);\n-\tgit_config_get_expiry(\"gc.pruneexpire\", (char **) &cfg->prune_expire);\n-\tgit_config_get_expiry(\"gc.worktreepruneexpire\", (char **) &cfg->prune_worktrees_expire);\n-\tgit_config_get_expiry(\"gc.logexpiry\", (char **) &cfg->gc_log_expire);\n+\n+\tif (!git_config_get_expiry(\"gc.pruneexpire\", &owned)) {\n+\t\tfree(cfg->prune_expire);\n+\t\tcfg->prune_expire = owned;\n+\t}\n+\n+\tif (!git_config_get_expiry(\"gc.worktreepruneexpire\", &owned)) {\n+\t\tfree(cfg->prune_worktrees_expire);\n+\t\tcfg->prune_worktrees_expire = owned;\n+\t}\n+\n+\tif (!git_config_get_expiry(\"gc.logexpiry\", &owned)) {\n+\t\tfree(cfg->gc_log_expire);\n+\t\tcfg->gc_log_expire = owned;\n+\t}\n \n \tgit_config_get_ulong(\"gc.bigpackthreshold\", &cfg->big_pack_threshold);\n \tgit_config_get_ulong(\"pack.deltacachesize\", &cfg->max_delta_cache_size);\n \n-\tgit_config_get_string(\"gc.repackfilter\", &cfg->repack_filter);\n-\tgit_config_get_string(\"gc.repackfilterto\", &cfg->repack_filter_to);\n+\tif (!git_config_get_string(\"gc.repackfilter\", &owned)) {\n+\t\tfree(cfg->repack_filter);\n+\t\tcfg->repack_filter = owned;\n+\t}\n+\n+\tif (!git_config_get_string(\"gc.repackfilterto\", &owned)) {\n+\t\tfree(cfg->repack_filter_to);\n+\t\tcfg->repack_filter_to = owned;\n+\t}\n \n \tgit_config(git_default_config, NULL);\n }\n@@ -644,12 +673,15 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \tstruct child_process rerere_cmd = CHILD_PROCESS_INIT;\n \tstruct maintenance_run_opts opts = {0};\n \tstruct gc_config cfg = GC_CONFIG_INIT;\n+\tconst char *prune_expire_sentinel = \"sentinel\";\n+\tconst char *prune_expire_arg = prune_expire_sentinel;\n+\tint ret;\n \n \tstruct option builtin_gc_options[] = {\n \t\tOPT__QUIET(&quiet, N_(\"suppress progress reporting\")),\n-\t\t{ OPTION_STRING, 0, \"prune\", &cfg.prune_expire, N_(\"date\"),\n+\t\t{ OPTION_STRING, 0, \"prune\", &prune_expire_arg, N_(\"date\"),\n \t\t\tN_(\"prune unreferenced objects\"),\n-\t\t\tPARSE_OPT_OPTARG, NULL, (intptr_t)cfg.prune_expire },\n+\t\t\tPARSE_OPT_OPTARG, NULL, (intptr_t)prune_expire_arg },\n \t\tOPT_BOOL(0, \"cruft\", &cfg.cruft_packs, N_(\"pack unreferenced objects separately\")),\n \t\tOPT_MAGNITUDE(0, \"max-cruft-size\", &cfg.max_cruft_size,\n \t\t\t N_(\"with --cruft, limit the size of new cruft packs\")),\n@@ -673,8 +705,8 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \tstrvec_pushl(&prune_worktrees, \"worktree\", \"prune\", \"--expire\", NULL);\n \tstrvec_pushl(&rerere, \"rerere\", \"gc\", NULL);\n \n-\t/* default expiry time, overwritten in gc_config */\n \tgc_config(&cfg);\n+\n \tif (parse_expiry_date(cfg.gc_log_expire, &gc_log_expire_time))\n \t\tdie(_(\"failed to parse gc.logExpiry value %s\"), cfg.gc_log_expire);\n \n@@ -686,6 +718,10 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \tif (argc > 0)\n \t\tusage_with_options(builtin_gc_usage, builtin_gc_options);\n \n+\tif (prune_expire_arg != prune_expire_sentinel) {\n+\t\tfree(cfg.prune_expire);\n+\t\tcfg.prune_expire = xstrdup_or_null(prune_expire_arg);\n+\t}\n \tif (cfg.prune_expire && parse_expiry_date(cfg.prune_expire, &dummy))\n \t\tdie(_(\"failed to parse prune expiry value %s\"), cfg.prune_expire);\n \n@@ -703,8 +739,11 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \t\t/*\n \t\t * Auto-gc should be least intrusive as possible.\n \t\t */\n-\t\tif (!need_to_gc(&cfg))\n-\t\t\treturn 0;\n+\t\tif (!need_to_gc(&cfg)) {\n+\t\t\tret = 0;\n+\t\t\tgoto out;\n+\t\t}\n+\n \t\tif (!quiet) {\n \t\t\tif (cfg.detach_auto)\n \t\t\t\tfprintf(stderr, _(\"Auto packing the repository in background for optimum performance.\\n\"));\n@@ -713,17 +752,22 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \t\t\tfprintf(stderr, _(\"See \\\"git help gc\\\" for manual housekeeping.\\n\"));\n \t\t}\n \t\tif (cfg.detach_auto) {\n-\t\t\tint ret = report_last_gc_error();\n-\n-\t\t\tif (ret == 1)\n+\t\t\tret = report_last_gc_error();\n+\t\t\tif (ret == 1) {\n \t\t\t\t/* Last gc --auto failed. Skip this one. */\n-\t\t\t\treturn 0;\n-\t\t\telse if (ret)\n+\t\t\t\tret = 0;\n+\t\t\t\tgoto out;\n+\n+\t\t\t} else if (ret) {\n \t\t\t\t/* an I/O error occurred, already reported */\n-\t\t\t\treturn ret;\n+\t\t\t\tgoto out;\n+\t\t\t}\n+\n+\t\t\tif (lock_repo_for_gc(force, &pid)) {\n+\t\t\t\tret = 0;\n+\t\t\t\tgoto out;\n+\t\t\t}\n \n-\t\t\tif (lock_repo_for_gc(force, &pid))\n-\t\t\t\treturn 0;\n \t\t\tgc_before_repack(&opts, &cfg); /* dies on failure */\n \t\t\tdelete_tempfile(&pidfile);\n \n@@ -749,8 +793,11 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \n \tname = lock_repo_for_gc(force, &pid);\n \tif (name) {\n-\t\tif (opts.auto_flag)\n-\t\t\treturn 0; /* be quiet on --auto */\n+\t\tif (opts.auto_flag) {\n+\t\t\tret = 0;\n+\t\t\tgoto out; /* be quiet on --auto */\n+\t\t}\n+\n \t\tdie(_(\"gc is already running on machine '%s' pid %\"PRIuMAX\" (use --force if not)\"),\n \t\t name, (uintmax_t)pid);\n \t}\n@@ -826,6 +873,8 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \tif (!daemonized)\n \t\tunlink(git_path(\"gc.log\"));\n \n+out:\n+\tgc_config_release(&cfg);\n \treturn 0;\n }\n \n@@ -1511,6 +1560,8 @@ static int maintenance_run(int argc, const char **argv, const char *prefix)\n \t\t\tPARSE_OPT_NONEG, task_option_parse),\n \t\tOPT_END()\n \t};\n+\tint ret;\n+\n \tmemset(&opts, 0, sizeof(opts));\n \n \topts.quiet = !isatty(2);\n@@ -1532,7 +1583,10 @@ static int maintenance_run(int argc, const char **argv, const char *prefix)\n \tif (argc != 0)\n \t\tusage_with_options(builtin_maintenance_run_usage,\n \t\t\t\t builtin_maintenance_run_options);\n-\treturn maintenance_run_tasks(&opts, &cfg);\n+\n+\tret = maintenance_run_tasks(&opts, &cfg);\n+\tgc_config_release(&cfg);\n+\treturn ret;\n }\n \n static char *get_maintpath(void)\ndiff --git a/t/t5304-prune.sh b/t/t5304-prune.sh\nindex 1f1f664871..e641df0116 100755\n--- a/t/t5304-prune.sh\n+++ b/t/t5304-prune.sh\n@@ -7,6 +7,7 @@ test_description='prune'\n GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n \n+TEST_PASSES_SANITIZE_LEAK=true\n . ./test-lib.sh\n \n day=$((60*60*24))\n-- \n2.46.0.46.g406f326d27.dirty\n\n"},{"id":"500696","messageId":"812c61c9b66d7608e41c6c1d00a6e22e995cef06.1723533091.git.ps@pks.im","threadId":"61940","inReplyTo":"cover.1723533091.git.ps@pks.im","subject":"[PATCH 4/7] builtin/gc: stop processing log file on signal","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-08-13T07:17:49Z","receivedAt":"2024-08-13T07:17:55Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"When detaching, git-gc(1) will redirect its stderr to a \"gc.log\" log\nfile, which is then used to surface errors of a backgrounded process to\nthe user. To ensure that the file is properly managed on abnormal exit\npaths, we install both signal and exit handlers that try to either\ncommit the underlying lock file or roll it back in case there wasn't any\nerror.\n\nThis logic is severly broken when handling signals though, as we end up\ncalling all kinds of functions that are not signal safe. This includes\nmalloc(3P) via `git_path()`, fprintf(3P), fflush(3P) and many more\nfunctions. The consequence can be anything, from deadlocks to crashes.\nUnfortunately, we cannot really do much about this without a larger\nrefactoring.\n\nThe least-worst thing we can do is to not set up the signal handler in\nthe first place. This will still cause us to remove the lockfile, as the\nunderlying tempfile subsystem already knows to unlink locks when\nreceiving a signal. But it may cause us to remove the lock even in the\ncase where it would have contained actual errors, which is a change in\nbehaviour.\n\nThe consequence is that \"gc.log\" will not be committed, and thus\nsubsequent calls to `git gc --auto` won't bail out because of this.\nArguably though, it is better to retry garbage collection rather than\nhaving the process run into a potentially-corrupted state.\n\nSigned-off-by: Patrick Steinhardt \n---\n builtin/gc.c | 8 --------\n 1 file changed, 8 deletions(-)\n\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex a93cfa147e..f815557081 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -109,13 +109,6 @@ static void process_log_file_at_exit(void)\n \tprocess_log_file();\n }\n \n-static void process_log_file_on_signal(int signo)\n-{\n-\tprocess_log_file();\n-\tsigchain_pop(signo);\n-\traise(signo);\n-}\n-\n static int gc_config_is_timestamp_never(const char *var)\n {\n \tconst char *value;\n@@ -807,7 +800,6 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \t\t\t\t\t git_path(\"gc.log\"),\n \t\t\t\t\t LOCK_DIE_ON_ERROR);\n \t\tdup2(get_lock_file_fd(&log_lock), 2);\n-\t\tsigchain_push_common(process_log_file_on_signal);\n \t\tatexit(process_log_file_at_exit);\n \t}\n \n-- \n2.46.0.46.g406f326d27.dirty\n\n"},{"id":"500697","messageId":"ca78d3dc7c0270b434ee4ca4ef618212c7dc1d5b.1723533091.git.ps@pks.im","threadId":"61940","inReplyTo":"cover.1723533091.git.ps@pks.im","subject":"[PATCH 5/7] builtin/gc: add a `--detach` flag","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-08-13T07:17:54Z","receivedAt":"2024-08-13T07:17:57Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"When running `git gc --auto`, the command will by default detach and\ncontinue running in the background. This behaviour can be tweaked via\nthe `gc.autoDetach` config, but not via a command line switch. We need\nthat in a subsequent commit though, where git-maintenance(1) will want\nto ask its git-gc(1) child process to not detach anymore.\n\nAdd a `--[no-]detach` flag that does this for us.\n\nSigned-off-by: Patrick Steinhardt \n---\n Documentation/git-gc.txt | 5 ++-\n builtin/gc.c | 70 ++++++++++++++++++++++------------------\n t/t6500-gc.sh | 39 ++++++++++++++++++++++\n 3 files changed, 82 insertions(+), 32 deletions(-)\n\ndiff --git a/Documentation/git-gc.txt b/Documentation/git-gc.txt\nindex b5561c458a..370e22faae 100644\n--- a/Documentation/git-gc.txt\n+++ b/Documentation/git-gc.txt\n@@ -9,7 +9,7 @@ git-gc - Cleanup unnecessary files and optimize the local repository\n SYNOPSIS\n --------\n [verse]\n-'git gc' [--aggressive] [--auto] [--quiet] [--prune= | --no-prune] [--force] [--keep-largest-pack]\n+'git gc' [--aggressive] [--auto] [--[no-]detach] [--quiet] [--prune= | --no-prune] [--force] [--keep-largest-pack]\n \n DESCRIPTION\n -----------\n@@ -53,6 +53,9 @@ configuration options such as `gc.auto` and `gc.autoPackLimit`, all\n other housekeeping tasks (e.g. rerere, working trees, reflog...) will\n be performed as well.\n \n+--[no-]detach::\n+\tRun in the background if the system supports it. This option overrides\n+\tthe `gc.autoDetach` config.\n \n --[no-]cruft::\n \tWhen expiring unreachable objects, pack them separately into a\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex f815557081..269a77960f 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -242,9 +242,13 @@ static enum schedule_priority parse_schedule(const char *value)\n \n struct maintenance_run_opts {\n \tint auto_flag;\n+\tint detach;\n \tint quiet;\n \tenum schedule_priority schedule;\n };\n+#define MAINTENANCE_RUN_OPTS_INIT { \\\n+\t.detach = -1, \\\n+}\n \n static int pack_refs_condition(UNUSED struct gc_config *cfg)\n {\n@@ -664,7 +668,7 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \tint keep_largest_pack = -1;\n \ttimestamp_t dummy;\n \tstruct child_process rerere_cmd = CHILD_PROCESS_INIT;\n-\tstruct maintenance_run_opts opts = {0};\n+\tstruct maintenance_run_opts opts = MAINTENANCE_RUN_OPTS_INIT;\n \tstruct gc_config cfg = GC_CONFIG_INIT;\n \tconst char *prune_expire_sentinel = \"sentinel\";\n \tconst char *prune_expire_arg = prune_expire_sentinel;\n@@ -681,6 +685,8 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \t\tOPT_BOOL(0, \"aggressive\", &aggressive, N_(\"be more thorough (increased runtime)\")),\n \t\tOPT_BOOL_F(0, \"auto\", &opts.auto_flag, N_(\"enable auto-gc mode\"),\n \t\t\t PARSE_OPT_NOCOMPLETE),\n+\t\tOPT_BOOL(0, \"detach\", &opts.detach,\n+\t\t\t N_(\"perform garbage collection in the background\")),\n \t\tOPT_BOOL_F(0, \"force\", &force,\n \t\t\t N_(\"force running gc even if there may be another gc running\"),\n \t\t\t PARSE_OPT_NOCOMPLETE),\n@@ -729,6 +735,9 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \t\tstrvec_push(&repack, \"-q\");\n \n \tif (opts.auto_flag) {\n+\t\tif (cfg.detach_auto && opts.detach < 0)\n+\t\t\topts.detach = 1;\n+\n \t\t/*\n \t\t * Auto-gc should be least intrusive as possible.\n \t\t */\n@@ -738,38 +747,12 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \t\t}\n \n \t\tif (!quiet) {\n-\t\t\tif (cfg.detach_auto)\n+\t\t\tif (opts.detach > 0)\n \t\t\t\tfprintf(stderr, _(\"Auto packing the repository in background for optimum performance.\\n\"));\n \t\t\telse\n \t\t\t\tfprintf(stderr, _(\"Auto packing the repository for optimum performance.\\n\"));\n \t\t\tfprintf(stderr, _(\"See \\\"git help gc\\\" for manual housekeeping.\\n\"));\n \t\t}\n-\t\tif (cfg.detach_auto) {\n-\t\t\tret = report_last_gc_error();\n-\t\t\tif (ret == 1) {\n-\t\t\t\t/* Last gc --auto failed. Skip this one. */\n-\t\t\t\tret = 0;\n-\t\t\t\tgoto out;\n-\n-\t\t\t} else if (ret) {\n-\t\t\t\t/* an I/O error occurred, already reported */\n-\t\t\t\tgoto out;\n-\t\t\t}\n-\n-\t\t\tif (lock_repo_for_gc(force, &pid)) {\n-\t\t\t\tret = 0;\n-\t\t\t\tgoto out;\n-\t\t\t}\n-\n-\t\t\tgc_before_repack(&opts, &cfg); /* dies on failure */\n-\t\t\tdelete_tempfile(&pidfile);\n-\n-\t\t\t/*\n-\t\t\t * failure to daemonize is ok, we'll continue\n-\t\t\t * in foreground\n-\t\t\t */\n-\t\t\tdaemonized = !daemonize();\n-\t\t}\n \t} else {\n \t\tstruct string_list keep_pack = STRING_LIST_INIT_NODUP;\n \n@@ -784,6 +767,33 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \t\tstring_list_clear(&keep_pack, 0);\n \t}\n \n+\tif (opts.detach > 0) {\n+\t\tret = report_last_gc_error();\n+\t\tif (ret == 1) {\n+\t\t\t/* Last gc --auto failed. Skip this one. */\n+\t\t\tret = 0;\n+\t\t\tgoto out;\n+\n+\t\t} else if (ret) {\n+\t\t\t/* an I/O error occurred, already reported */\n+\t\t\tgoto out;\n+\t\t}\n+\n+\t\tif (lock_repo_for_gc(force, &pid)) {\n+\t\t\tret = 0;\n+\t\t\tgoto out;\n+\t\t}\n+\n+\t\tgc_before_repack(&opts, &cfg); /* dies on failure */\n+\t\tdelete_tempfile(&pidfile);\n+\n+\t\t/*\n+\t\t * failure to daemonize is ok, we'll continue\n+\t\t * in foreground\n+\t\t */\n+\t\tdaemonized = !daemonize();\n+\t}\n+\n \tname = lock_repo_for_gc(force, &pid);\n \tif (name) {\n \t\tif (opts.auto_flag) {\n@@ -1537,7 +1547,7 @@ static int task_option_parse(const struct option *opt UNUSED,\n static int maintenance_run(int argc, const char **argv, const char *prefix)\n {\n \tint i;\n-\tstruct maintenance_run_opts opts;\n+\tstruct maintenance_run_opts opts = MAINTENANCE_RUN_OPTS_INIT;\n \tstruct gc_config cfg = GC_CONFIG_INIT;\n \tstruct option builtin_maintenance_run_options[] = {\n \t\tOPT_BOOL(0, \"auto\", &opts.auto_flag,\n@@ -1554,8 +1564,6 @@ static int maintenance_run(int argc, const char **argv, const char *prefix)\n \t};\n \tint ret;\n \n-\tmemset(&opts, 0, sizeof(opts));\n-\n \topts.quiet = !isatty(2);\n \n \tfor (i = 0; i < TASK__COUNT; i++)\ndiff --git a/t/t6500-gc.sh b/t/t6500-gc.sh\nindex 1b5909d1b7..737c99e0f8 100755\n--- a/t/t6500-gc.sh\n+++ b/t/t6500-gc.sh\n@@ -396,6 +396,45 @@ test_expect_success 'background auto gc respects lock for all operations' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success '--detach overrides gc.autoDetach=false' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\n+\t\t# Prepare the repository such that git-gc(1) ends up repacking.\n+\t\ttest_commit \"$(test_oid blob17_1)\" &&\n+\t\ttest_commit \"$(test_oid blob17_2)\" &&\n+\t\tgit config gc.autodetach false &&\n+\t\tgit config gc.auto 2 &&\n+\n+\t\tcat >expect <<-EOF &&\n+\t\tAuto packing the repository in background for optimum performance.\n+\t\tSee \"git help gc\" for manual housekeeping.\n+\t\tEOF\n+\t\tGIT_PROGRESS_DELAY=0 git gc --auto --detach 2>actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success '--no-detach overrides gc.autoDetach=true' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\n+\t\t# Prepare the repository such that git-gc(1) ends up repacking.\n+\t\ttest_commit \"$(test_oid blob17_1)\" &&\n+\t\ttest_commit \"$(test_oid blob17_2)\" &&\n+\t\tgit config gc.autodetach true &&\n+\t\tgit config gc.auto 2 &&\n+\n+\t\tGIT_PROGRESS_DELAY=0 git gc --auto --no-detach 2>output &&\n+\t\ttest_grep \"Auto packing the repository for optimum performance.\" output &&\n+\t\ttest_grep \"Collecting referenced commits: 2, done.\" output\n+\t)\n+'\n+\n # DO NOT leave a detached auto gc process running near the end of the\n # test script: it can run long enough in the background to racily\n # interfere with the cleanup in 'test_done'.\n-- \n2.46.0.46.g406f326d27.dirty\n\n"},{"id":"500698","messageId":"06dbb73425f7df038dfdd9bc0d2b9a49edfe064e.1723533091.git.ps@pks.im","threadId":"61940","inReplyTo":"cover.1723533091.git.ps@pks.im","subject":"[PATCH 6/7] builtin/maintenance: add a `--detach` flag","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-08-13T07:17:57Z","receivedAt":"2024-08-13T07:17:59Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Same as the preceding commit, add a `--[no-]detach` flag to the\ngit-maintenance(1) command. This will be used in a subsequent commit to\nfix backgrounding of that command when configured with a non-standard\nset of tasks.\n\nSigned-off-by: Patrick Steinhardt \n---\n builtin/gc.c | 6 ++++++\n t/t7900-maintenance.sh | 39 +++++++++++++++++++++++++++++++++++++++\n 2 files changed, 45 insertions(+)\n\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 269a77960f..63106e2028 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -1426,6 +1426,10 @@ static int maintenance_run_tasks(struct maintenance_run_opts *opts,\n \t}\n \tfree(lock_path);\n \n+\t/* Failure to daemonize is ok, we'll continue in foreground. */\n+\tif (opts->detach > 0)\n+\t\tdaemonize();\n+\n \tfor (i = 0; !found_selected && i < TASK__COUNT; i++)\n \t\tfound_selected = tasks[i].selected_order >= 0;\n \n@@ -1552,6 +1556,8 @@ static int maintenance_run(int argc, const char **argv, const char *prefix)\n \tstruct option builtin_maintenance_run_options[] = {\n \t\tOPT_BOOL(0, \"auto\", &opts.auto_flag,\n \t\t\t N_(\"run tasks based on the state of the repository\")),\n+\t\tOPT_BOOL(0, \"detach\", &opts.detach,\n+\t\t\t N_(\"perform maintenance in the background\")),\n \t\tOPT_CALLBACK(0, \"schedule\", &opts.schedule, N_(\"frequency\"),\n \t\t\t N_(\"run tasks based on frequency\"),\n \t\t\t maintenance_opt_schedule),\ndiff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\nindex 8595489ceb..771525aa4b 100755\n--- a/t/t7900-maintenance.sh\n+++ b/t/t7900-maintenance.sh\n@@ -908,4 +908,43 @@ test_expect_success 'failed schedule prevents config change' '\n \tdone\n '\n \n+test_expect_success '--no-detach causes maintenance to not run in background' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\n+\t\t# Prepare the repository such that git-maintenance(1) ends up\n+\t\t# outputting something.\n+\t\ttest_commit something &&\n+\t\tgit config set maintenance.gc.enabled false &&\n+\t\tgit config set maintenance.loose-objects.enabled true &&\n+\t\tgit config set maintenance.loose-objects.auto 1 &&\n+\t\tgit config set maintenance.incremental-repack.enabled true &&\n+\n+\t\t# We have no better way to check whether or not the task ran in\n+\t\t# the background than to verify whether it output anything. The\n+\t\t# next testcase checks the reverse, making this somewhat safer.\n+\t\tgit maintenance run --no-detach >out 2>&1 &&\n+\t\ttest_line_count = 1 out\n+\t)\n+'\n+\n+test_expect_success '--detach causes maintenance to run in background' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\n+\t\ttest_commit something &&\n+\t\tgit config set maintenance.gc.enabled false &&\n+\t\tgit config set maintenance.loose-objects.enabled true &&\n+\t\tgit config set maintenance.loose-objects.auto 1 &&\n+\t\tgit config set maintenance.incremental-repack.enabled true &&\n+\n+\t\tgit maintenance run --detach >out 2>&1 &&\n+\t\ttest_must_be_empty out\n+\t)\n+'\n+\n test_done\n-- \n2.46.0.46.g406f326d27.dirty\n\n"},{"id":"500699","messageId":"8d6cbae951177718b49d5cfbbeca2d5b0073e266.1723533091.git.ps@pks.im","threadId":"61940","inReplyTo":"cover.1723533091.git.ps@pks.im","subject":"[PATCH 7/7] builtin/maintenance: fix auto-detach with non-standard tasks","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-08-13T07:18:00Z","receivedAt":"2024-08-13T07:18:02Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"In the past, we used to execute `git gc --auto` as part of our automatic\nhousekeeping routines. As git-gc(1) may require quite some time to\nperform the housekeeping, it knows to detach itself and run in the\nbackground so that the user can continue their work.\n\nEventually, we refactored our automatic housekeeping to instead use the\nmore flexible git-maintenance(1) command. The upside of this new infra\nis that the user can configure which maintenance tasks are performed, at\nleast to a certain degree. So while it continues to run git-gc(1) by\ndefault, it can also be adapted to e.g. use git-multi-pack-index(1) for\nmaintenance of the object database.\n\nThe auto-detach of the new infra is somewhat broken though once the user\nconfigures non-standard tasks. The problem is essentially that we detach\nat the wrong level in the process hierarchy: git-maintenance(1) never\ndetaches itself, but instead it continues to be git-gc(1) which does.\n\nWhen configured to only run the git-gc(1) maintenance task, then the\nresult is basically the same as before. But when configured to run other\ntasks, then git-maintenance(1) will wait for these to run to completion.\nEven worse, it may be that git-gc(1) runs concurrently with other\nhousekeeping tasks, stomping on each others feet.\n\nFix this bug by asking git-gc(1) to not detach when it is being invoked\nvia git-maintenance(1). Instead, the latter command now respects a new\nconfig \"maintenance.autoDetach\", the equivalent of \"gc.autoDetach\", and\ndetaches itself into the background if not told otherwise. This should\ncontinue to behave the same for all users which use the git-gc(1) task,\nonly. For others though, it means that we now properly perform all tasks\nin the background.\n\nSigned-off-by: Patrick Steinhardt \n---\n builtin/gc.c | 1 +\n run-command.c | 12 ++++++++++-\n t/t5616-partial-clone.sh | 6 +++---\n t/t7900-maintenance.sh | 43 +++++++++++++++++++++++++++++++---------\n 4 files changed, 49 insertions(+), 13 deletions(-)\n\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 63106e2028..bafee330a2 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -1063,6 +1063,7 @@ static int maintenance_task_gc(struct maintenance_run_opts *opts,\n \t\tstrvec_push(&child.args, \"--quiet\");\n \telse\n \t\tstrvec_push(&child.args, \"--no-quiet\");\n+\tstrvec_push(&child.args, \"--no-detach\");\n \n \treturn run_command(&child);\n }\ndiff --git a/run-command.c b/run-command.c\nindex 45ba544932..94f2f3079f 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -1808,16 +1808,26 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)\n \n int prepare_auto_maintenance(int quiet, struct child_process *maint)\n {\n-\tint enabled;\n+\tint enabled, auto_detach;\n \n \tif (!git_config_get_bool(\"maintenance.auto\", &enabled) &&\n \t !enabled)\n \t\treturn 0;\n \n+\t/*\n+\t * When `maintenance.autoDetach` isn't set, then we fall back to\n+\t * honoring `gc.autoDetach`. This is somewhat weird, but required to\n+\t * retain behaviour from when we used to run git-gc(1) here.\n+\t */\n+\tif (git_config_get_bool(\"maintenance.autodetach\", &auto_detach) &&\n+\t git_config_get_bool(\"gc.autodetach\", &auto_detach))\n+\t\tauto_detach = 1;\n+\n \tmaint->git_cmd = 1;\n \tmaint->close_object_store = 1;\n \tstrvec_pushl(&maint->args, \"maintenance\", \"run\", \"--auto\", NULL);\n \tstrvec_push(&maint->args, quiet ? \"--quiet\" : \"--no-quiet\");\n+\tstrvec_push(&maint->args, auto_detach ? \"--detach\" : \"--no-detach\");\n \n \treturn 1;\n }\ndiff --git a/t/t5616-partial-clone.sh b/t/t5616-partial-clone.sh\nindex 2da7291e37..8415884754 100755\n--- a/t/t5616-partial-clone.sh\n+++ b/t/t5616-partial-clone.sh\n@@ -229,7 +229,7 @@ test_expect_success 'fetch --refetch triggers repacking' '\n \n \tGIT_TRACE2_EVENT=\"$PWD/trace1.event\" \\\n \tgit -C pc1 fetch --refetch origin &&\n-\ttest_subcommand git maintenance run --auto --no-quiet /dev/null &&\n \tGIT_TRACE2_EVENT=\"$(pwd)/run-no-quiet.txt\" \\\n \t\tgit maintenance run --no-quiet 2>/dev/null &&\n-\ttest_subcommand git gc --quiet ' '\n \t\tgit maintenance run --task=commit-graph 2>/dev/null &&\n \tGIT_TRACE2_EVENT=\"$(pwd)/run-both.txt\" \\\n \t\tgit maintenance run --task=commit-graph --task=gc 2>/dev/null &&\n-\ttest_subcommand ! git gc --quiet \n> Fix this bug by asking git-gc(1) to not detach when it is being invoked\n> via git-maintenance(1). Instead, the latter command now respects a new\n> config \"maintenance.autoDetach\", the equivalent of \"gc.autoDetach\", and\n> detaches itself into the background if not told otherwise. This should\n> continue to behave the same for all users which use the git-gc(1) task,\n> only. For others though, it means that we now properly perform all tasks\n> in the background.\n\nI fear that users who are running \"git maintenance\" from a scheduler \nsuch as cron are likely to be surprised by this change in behavior. At \nthe very least \"git maintenance\" will no-longer return a meaningful exit \ncode. Perhaps we could switch the logic to be opt in and pass \"--detach\" \n(or \"-c maintenance.autoDetach=true\") when running \"git maintenance\" \nautomatically from \"git rebase\" etc.\n\nBest Wishes\n\nPhillip\n\n> Signed-off-by: Patrick Steinhardt \n> ---\n> builtin/gc.c | 1 +\n> run-command.c | 12 ++++++++++-\n> t/t5616-partial-clone.sh | 6 +++---\n> t/t7900-maintenance.sh | 43 +++++++++++++++++++++++++++++++---------\n> 4 files changed, 49 insertions(+), 13 deletions(-)\n> \n> diff --git a/builtin/gc.c b/builtin/gc.c\n> index 63106e2028..bafee330a2 100644\n> --- a/builtin/gc.c\n> +++ b/builtin/gc.c\n> @@ -1063,6 +1063,7 @@ static int maintenance_task_gc(struct maintenance_run_opts *opts,\n> \t\tstrvec_push(&child.args, \"--quiet\");\n> \telse\n> \t\tstrvec_push(&child.args, \"--no-quiet\");\n> +\tstrvec_push(&child.args, \"--no-detach\");\n> \n> \treturn run_command(&child);\n> }\n> diff --git a/run-command.c b/run-command.c\n> index 45ba544932..94f2f3079f 100644\n> --- a/run-command.c\n> +++ b/run-command.c\n> @@ -1808,16 +1808,26 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)\n> \n> int prepare_auto_maintenance(int quiet, struct child_process *maint)\n> {\n> -\tint enabled;\n> +\tint enabled, auto_detach;\n> \n> \tif (!git_config_get_bool(\"maintenance.auto\", &enabled) &&\n> \t !enabled)\n> \t\treturn 0;\n> \n> +\t/*\n> +\t * When `maintenance.autoDetach` isn't set, then we fall back to\n> +\t * honoring `gc.autoDetach`. This is somewhat weird, but required to\n> +\t * retain behaviour from when we used to run git-gc(1) here.\n> +\t */\n> +\tif (git_config_get_bool(\"maintenance.autodetach\", &auto_detach) &&\n> +\t git_config_get_bool(\"gc.autodetach\", &auto_detach))\n> +\t\tauto_detach = 1;\n> +\n> \tmaint->git_cmd = 1;\n> \tmaint->close_object_store = 1;\n> \tstrvec_pushl(&maint->args, \"maintenance\", \"run\", \"--auto\", NULL);\n> \tstrvec_push(&maint->args, quiet ? \"--quiet\" : \"--no-quiet\");\n> +\tstrvec_push(&maint->args, auto_detach ? \"--detach\" : \"--no-detach\");\n> \n> \treturn 1;\n> }\n> diff --git a/t/t5616-partial-clone.sh b/t/t5616-partial-clone.sh\n> index 2da7291e37..8415884754 100755\n> --- a/t/t5616-partial-clone.sh\n> +++ b/t/t5616-partial-clone.sh\n> @@ -229,7 +229,7 @@ test_expect_success 'fetch --refetch triggers repacking' '\n> \n> \tGIT_TRACE2_EVENT=\"$PWD/trace1.event\" \\\n> \tgit -C pc1 fetch --refetch origin &&\n> -\ttest_subcommand git maintenance run --auto --no-quiet +\ttest_subcommand git maintenance run --auto --no-quiet --detach \tgrep \\\"param\\\":\\\"gc.autopacklimit\\\",\\\"value\\\":\\\"1\\\" trace1.event &&\n> \tgrep \\\"param\\\":\\\"maintenance.incremental-repack.auto\\\",\\\"value\\\":\\\"-1\\\" trace1.event &&\n> \n> @@ -238,7 +238,7 @@ test_expect_success 'fetch --refetch triggers repacking' '\n> \t\t-c gc.autoPackLimit=0 \\\n> \t\t-c maintenance.incremental-repack.auto=1234 \\\n> \t\t-C pc1 fetch --refetch origin &&\n> -\ttest_subcommand git maintenance run --auto --no-quiet +\ttest_subcommand git maintenance run --auto --no-quiet --detach \tgrep \\\"param\\\":\\\"gc.autopacklimit\\\",\\\"value\\\":\\\"0\\\" trace2.event &&\n> \tgrep \\\"param\\\":\\\"maintenance.incremental-repack.auto\\\",\\\"value\\\":\\\"-1\\\" trace2.event &&\n> \n> @@ -247,7 +247,7 @@ test_expect_success 'fetch --refetch triggers repacking' '\n> \t\t-c gc.autoPackLimit=1234 \\\n> \t\t-c maintenance.incremental-repack.auto=0 \\\n> \t\t-C pc1 fetch --refetch origin &&\n> -\ttest_subcommand git maintenance run --auto --no-quiet +\ttest_subcommand git maintenance run --auto --no-quiet --detach \tgrep \\\"param\\\":\\\"gc.autopacklimit\\\",\\\"value\\\":\\\"1\\\" trace3.event &&\n> \tgrep \\\"param\\\":\\\"maintenance.incremental-repack.auto\\\",\\\"value\\\":\\\"0\\\" trace3.event\n> '\n> diff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\n> index 771525aa4b..06ab43cfb5 100755\n> --- a/t/t7900-maintenance.sh\n> +++ b/t/t7900-maintenance.sh\n> @@ -49,22 +49,47 @@ test_expect_success 'run [--auto|--quiet]' '\n> \t\tgit maintenance run --auto 2>/dev/null &&\n> \tGIT_TRACE2_EVENT=\"$(pwd)/run-no-quiet.txt\" \\\n> \t\tgit maintenance run --no-quiet 2>/dev/null &&\n> -\ttest_subcommand git gc --quiet -\ttest_subcommand ! git gc --auto --quiet -\ttest_subcommand git gc --no-quiet +\ttest_subcommand git gc --quiet --no-detach +\ttest_subcommand ! git gc --auto --quiet --no-detach +\ttest_subcommand git gc --no-quiet --no-detach '\n> \n> test_expect_success 'maintenance.auto config option' '\n> \tGIT_TRACE2_EVENT=\"$(pwd)/default\" git commit --quiet --allow-empty -m 1 &&\n> -\ttest_subcommand git maintenance run --auto --quiet +\ttest_subcommand git maintenance run --auto --quiet --detach \tGIT_TRACE2_EVENT=\"$(pwd)/true\" \\\n> \t\tgit -c maintenance.auto=true \\\n> \t\tcommit --quiet --allow-empty -m 2 &&\n> -\ttest_subcommand git maintenance run --auto --quiet +\ttest_subcommand git maintenance run --auto --quiet --detach \tGIT_TRACE2_EVENT=\"$(pwd)/false\" \\\n> \t\tgit -c maintenance.auto=false \\\n> \t\tcommit --quiet --allow-empty -m 3 &&\n> -\ttest_subcommand ! git maintenance run --auto --quiet +\ttest_subcommand ! git maintenance run --auto --quiet --detach +'\n> +\n> +for cfg in maintenance.autoDetach gc.autoDetach\n> +do\n> +\ttest_expect_success \"$cfg=true config option\" '\n> +\t\ttest_when_finished \"rm -f trace\" &&\n> +\t\ttest_config $cfg true &&\n> +\t\tGIT_TRACE2_EVENT=\"$(pwd)/trace\" git commit --quiet --allow-empty -m 1 &&\n> +\t\ttest_subcommand git maintenance run --auto --quiet --detach +\t'\n> +\n> +\ttest_expect_success \"$cfg=false config option\" '\n> +\t\ttest_when_finished \"rm -f trace\" &&\n> +\t\ttest_config $cfg false &&\n> +\t\tGIT_TRACE2_EVENT=\"$(pwd)/trace\" git commit --quiet --allow-empty -m 1 &&\n> +\t\ttest_subcommand git maintenance run --auto --quiet --no-detach +\t'\n> +done\n> +\n> +test_expect_success \"maintenance.autoDetach overrides gc.autoDetach\" '\n> +\ttest_when_finished \"rm -f trace\" &&\n> +\ttest_config maintenance.autoDetach false &&\n> +\ttest_config gc.autoDetach true &&\n> +\tGIT_TRACE2_EVENT=\"$(pwd)/trace\" git commit --quiet --allow-empty -m 1 &&\n> +\ttest_subcommand git maintenance run --auto --quiet --no-detach '\n> \n> test_expect_success 'register uses XDG_CONFIG_HOME config if it exists' '\n> @@ -129,9 +154,9 @@ test_expect_success 'run --task=' '\n> \t\tgit maintenance run --task=commit-graph 2>/dev/null &&\n> \tGIT_TRACE2_EVENT=\"$(pwd)/run-both.txt\" \\\n> \t\tgit maintenance run --task=commit-graph --task=gc 2>/dev/null &&\n> -\ttest_subcommand ! git gc --quiet -\ttest_subcommand git gc --quiet -\ttest_subcommand git gc --quiet +\ttest_subcommand ! git gc --quiet --no-detach +\ttest_subcommand git gc --quiet --no-detach +\ttest_subcommand git gc --quiet --no-detach \ttest_subcommand git commit-graph write --split --reachable --no-progress \ttest_subcommand ! git commit-graph write --split --reachable --no-progress \ttest_subcommand git commit-graph write --split --reachable --no-progress Hi Patrick\n> \n> On 13/08/2024 08:18, Patrick Steinhardt wrote:\n> > \n> > Fix this bug by asking git-gc(1) to not detach when it is being invoked\n> > via git-maintenance(1). Instead, the latter command now respects a new\n> > config \"maintenance.autoDetach\", the equivalent of \"gc.autoDetach\", and\n> > detaches itself into the background if not told otherwise. This should\n> > continue to behave the same for all users which use the git-gc(1) task,\n> > only. For others though, it means that we now properly perform all tasks\n> > in the background.\n> \n> I fear that users who are running \"git maintenance\" from a scheduler such as\n> cron are likely to be surprised by this change in behavior. At the very\n> least \"git maintenance\" will no-longer return a meaningful exit code.\n> Perhaps we could switch the logic to be opt in and pass \"--detach\" (or \"-c\n> maintenance.autoDetach=true\") when running \"git maintenance\" automatically\n> from \"git rebase\" etc.\n\nIt's actually the reverse: the old behaviour when run via a scheduler\nwas to detach by default, because git-gc(1) did. We now ask it to not\ndetach anymore, which fixes this. Furthermore, the default behaviour of\n`git maintenance run` did not change either: it stays in the foreground\nunless the `--detach` flag is given. So the thing you worry about is\nactually getting fixed by this series :)\n\nWhat _does_ change though is when we run `git maintenance` via our\nauto-maintenance framework. Here we now do detach the whole maintenance\nprocess, instead of only git-gc(1). This logic is only being executed by\nrandom commands (git-rebase, git-pull, git-commit etc), and I'd argue it\nis the expected behaviour.\n\nPatrick\n"},{"id":"500779","messageId":"a02e21f1-b2a4-499a-b767-3426876d31be@gmail.com","threadId":"61940","inReplyTo":"ZrtKoVFci6cdvMS_@tanuki","subject":"Re: [PATCH 7/7] builtin/maintenance: fix auto-detach with non-standard tasks","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2024-08-13T13:19:20Z","receivedAt":"2024-08-13T13:19:25Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"On 13/08/2024 12:59, Patrick Steinhardt wrote:\n> On Tue, Aug 13, 2024 at 12:29:47PM +0100, Phillip Wood wrote:\n>> Hi Patrick\n>>\n>> On 13/08/2024 08:18, Patrick Steinhardt wrote:\n>>>\n>>> Fix this bug by asking git-gc(1) to not detach when it is being invoked\n>>> via git-maintenance(1). Instead, the latter command now respects a new\n>>> config \"maintenance.autoDetach\", the equivalent of \"gc.autoDetach\", and\n>>> detaches itself into the background if not told otherwise. This should\n>>> continue to behave the same for all users which use the git-gc(1) task,\n>>> only. For others though, it means that we now properly perform all tasks\n>>> in the background.\n>>\n>> I fear that users who are running \"git maintenance\" from a scheduler such as\n>> cron are likely to be surprised by this change in behavior. At the very\n>> least \"git maintenance\" will no-longer return a meaningful exit code.\n>> Perhaps we could switch the logic to be opt in and pass \"--detach\" (or \"-c\n>> maintenance.autoDetach=true\") when running \"git maintenance\" automatically\n>> from \"git rebase\" etc.\n> \n> It's actually the reverse: the old behaviour when run via a scheduler\n> was to detach by default, because git-gc(1) did.\n\nOh, I misunderstood what this patch is changing. So despite being \ntagged builtin/maintenance and talking about \"git maintenance\" it does \nnot actually touch builtin/maintenance.c or change its behavior. What it \nis actually doing is changing how other git commands run \"git \nmaintenance --auto\" so that it is always run in the background unless \nthe user configures maintenance.autoDetach=false. That sounds like a \ngood change.\n\nThanks for clarifying\n\nPhillip\n\n> We now ask it to not\n> detach anymore, which fixes this. Furthermore, the default behaviour of\n> `git maintenance run` did not change either: it stays in the foreground\n> unless the `--detach` flag is given. So the thing you worry about is\n> actually getting fixed by this series :)\n> \n> What _does_ change though is when we run `git maintenance` via our\n> auto-maintenance framework. Here we now do detach the whole maintenance\n> process, instead of only git-gc(1). This logic is only being executed by\n> random commands (git-rebase, git-pull, git-commit etc), and I'd argue it\n> is the expected behaviour.\n> \n> Patrick\n> \n"},{"id":"500836","messageId":"ZrwvWIhjZuFkRNgl@tanuki","threadId":"61940","inReplyTo":"a02e21f1-b2a4-499a-b767-3426876d31be@gmail.com","subject":"Re: [PATCH 7/7] builtin/maintenance: fix auto-detach with non-standard tasks","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-08-14T04:15:29Z","receivedAt":"2024-08-14T04:15:34Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Aug 13, 2024 at 02:19:20PM +0100, Phillip Wood wrote:\n> On 13/08/2024 12:59, Patrick Steinhardt wrote:\n> > On Tue, Aug 13, 2024 at 12:29:47PM +0100, Phillip Wood wrote:\n> > > Hi Patrick\n> > > \n> > > On 13/08/2024 08:18, Patrick Steinhardt wrote:\n> > > > \n> > > > Fix this bug by asking git-gc(1) to not detach when it is being invoked\n> > > > via git-maintenance(1). Instead, the latter command now respects a new\n> > > > config \"maintenance.autoDetach\", the equivalent of \"gc.autoDetach\", and\n> > > > detaches itself into the background if not told otherwise. This should\n> > > > continue to behave the same for all users which use the git-gc(1) task,\n> > > > only. For others though, it means that we now properly perform all tasks\n> > > > in the background.\n> > > \n> > > I fear that users who are running \"git maintenance\" from a scheduler such as\n> > > cron are likely to be surprised by this change in behavior. At the very\n> > > least \"git maintenance\" will no-longer return a meaningful exit code.\n> > > Perhaps we could switch the logic to be opt in and pass \"--detach\" (or \"-c\n> > > maintenance.autoDetach=true\") when running \"git maintenance\" automatically\n> > > from \"git rebase\" etc.\n> > \n> > It's actually the reverse: the old behaviour when run via a scheduler\n> > was to detach by default, because git-gc(1) did.\n> \n> Oh, I misunderstood what this patch is changing. So despite being tagged\n> builtin/maintenance and talking about \"git maintenance\" it does not actually\n> touch builtin/maintenance.c or change its behavior. What it is actually\n> doing is changing how other git commands run \"git maintenance --auto\" so\n> that it is always run in the background unless the user configures\n> maintenance.autoDetach=false. That sounds like a good change.\n> \n> Thanks for clarifying\n\nYes. I should've probably prefixed this with \"run-command:\", not with\n\"builtin/maintenance\".\n\nPatrick\n"},{"id":"500914","messageId":"a3e9a4f9-baef-4c11-9f38-f30ff333e87a@gmail.com","threadId":"61940","inReplyTo":"ZrwvWIhjZuFkRNgl@tanuki","subject":"Re: [PATCH 7/7] builtin/maintenance: fix auto-detach with non-standard tasks","fromName":"Phillip Wood","fromEmail":"phillip.wood123@gmail.com","sentAt":"2024-08-14T15:13:05Z","receivedAt":"2024-08-14T15:13:07Z","isPatch":true,"sender":{"key":"phillip.wood@dunelm.org.uk","avatar":null},"body":"On 14/08/2024 05:15, Patrick Steinhardt wrote:\n> On Tue, Aug 13, 2024 at 02:19:20PM +0100, Phillip Wood wrote:\n>> On 13/08/2024 12:59, Patrick Steinhardt wrote:\n>>\n>> Oh, I misunderstood what this patch is changing. So despite being tagged\n>> builtin/maintenance and talking about \"git maintenance\" it does not actually\n>> touch builtin/maintenance.c or change its behavior. What it is actually\n>> doing is changing how other git commands run \"git maintenance --auto\" so\n>> that it is always run in the background unless the user configures\n>> maintenance.autoDetach=false. That sounds like a good change.\n>>\n>> Thanks for clarifying\n\nSorry my message sounds grumpier than I intended.\n\n> Yes. I should've probably prefixed this with \"run-command:\", not with\n> \"builtin/maintenance\".\n\nThat's a good idea, I think the important thing we want to convey is \nthat we're changing the way we run \"git maintenance --auto\", not the \nbehavior of \"git maintenance\" itself.\n\nThanks\n\nPhillip\n\n> Patrick\n> \n"},{"id":"500949","messageId":"D3G8EY3R09BC.2BKB6UND5J5MA@jamesliu.io","threadId":"61940","inReplyTo":"310e361371efc156c3aaac94439bdbeaa965155f.1723533091.git.ps@pks.im","subject":"Re: [PATCH 3/7] builtin/gc: fix leaking config values","fromName":"James Liu","fromEmail":"james@jamesliu.io","sentAt":"2024-08-15T05:22:04Z","receivedAt":"2024-08-15T05:22:08Z","isPatch":true,"sender":{"key":"james@jamesliu.io","avatar":"https://avatars.githubusercontent.com/u/4476568?v=4"},"body":"On Tue Aug 13, 2024 at 5:17 PM AEST, Patrick Steinhardt wrote:\n> Note that there is one small gotcha here with the \"--prune\" option. Next\n> to passing a string, this option also accepts the \"--no-prune\" option\n> that overrides the default or configured value. We thus need to discern\n> between the option not having been passed by the user and the negative\n> variant of it. This is done by using a simple sentinel value that lets\n> us discern these cases.\n>\n> @@ -644,12 +673,15 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n> \tstruct child_process rerere_cmd = CHILD_PROCESS_INIT;\n> \tstruct maintenance_run_opts opts = {0};\n> \tstruct gc_config cfg = GC_CONFIG_INIT;\n> +\tconst char *prune_expire_sentinel = \"sentinel\";\n> +\tconst char *prune_expire_arg = prune_expire_sentinel;\n> +\tint ret;\n> \n> \tstruct option builtin_gc_options[] = {\n> \t\tOPT__QUIET(&quiet, N_(\"suppress progress reporting\")),\n> -\t\t{ OPTION_STRING, 0, \"prune\", &cfg.prune_expire, N_(\"date\"),\n> +\t\t{ OPTION_STRING, 0, \"prune\", &prune_expire_arg, N_(\"date\"),\n> \t\t\tN_(\"prune unreferenced objects\"),\n> -\t\t\tPARSE_OPT_OPTARG, NULL, (intptr_t)cfg.prune_expire },\n> +\t\t\tPARSE_OPT_OPTARG, NULL, (intptr_t)prune_expire_arg },\n> \t\tOPT_BOOL(0, \"cruft\", &cfg.cruft_packs, N_(\"pack unreferenced objects separately\")),\n> \t\tOPT_MAGNITUDE(0, \"max-cruft-size\", &cfg.max_cruft_size,\n> \t\t\t N_(\"with --cruft, limit the size of new cruft packs\")),\n\nI was wondering how the `no-*` options worked since they're not\nexplicitly defined in the `builtin_gc_options` array. I guess\nthey're handled internally by `parse_options()`, and if the `no-` prefix\nwas present, it leaves that argument unset.\n\n"},{"id":"500950","messageId":"D3G8GP38BNG8.26Y1ZLIUPTWJB@jamesliu.io","threadId":"61940","inReplyTo":"ff6aa9d7ba0bd67d3682ecb44c024a6d59a7decd.1723533091.git.ps@pks.im","subject":"Re: [PATCH 2/7] builtin/gc: refactor to read config into structure","fromName":"James Liu","fromEmail":"james@jamesliu.io","sentAt":"2024-08-15T05:24:21Z","receivedAt":"2024-08-15T05:24:25Z","isPatch":true,"sender":{"key":"james@jamesliu.io","avatar":"https://avatars.githubusercontent.com/u/4476568?v=4"},"body":"On Tue Aug 13, 2024 at 5:17 PM AEST, Patrick Steinhardt wrote:\n> @@ -206,7 +224,7 @@ struct maintenance_run_opts {\n> \tenum schedule_priority schedule;\n> };\n> \n> -static int pack_refs_condition(void)\n> +static int pack_refs_condition(UNUSED struct gc_config *cfg)\n> {\n> \t/*\n> \t * The auto-repacking logic for refs is handled by the ref backends and\n> @@ -216,7 +234,8 @@ static int pack_refs_condition(void)\n> \treturn 1;\n> }\n> \n> -static int maintenance_task_pack_refs(MAYBE_UNUSED struct maintenance_run_opts *opts)\n> +static int maintenance_task_pack_refs(MAYBE_UNUSED struct maintenance_run_opts *opts,\n> +\t\t\t\t UNUSED struct gc_config *cfg)\n> {\n> \tstruct child_process cmd = CHILD_PROCESS_INIT;\n> \n\nAre we defining *cfg as an unused argument to conform to the\n`maintenance_task_fn` signature?\n\n"},{"id":"500953","messageId":"Zr2SiGOuPcBWS9Fh@tanuki","threadId":"61940","inReplyTo":"a3e9a4f9-baef-4c11-9f38-f30ff333e87a@gmail.com","subject":"Re: [PATCH 7/7] builtin/maintenance: fix auto-detach with non-standard tasks","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-08-15T05:30:48Z","receivedAt":"2024-08-15T05:30:53Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Wed, Aug 14, 2024 at 04:13:05PM +0100, Phillip Wood wrote:\n> On 14/08/2024 05:15, Patrick Steinhardt wrote:\n> > On Tue, Aug 13, 2024 at 02:19:20PM +0100, Phillip Wood wrote:\n> > > On 13/08/2024 12:59, Patrick Steinhardt wrote:\n> > > \n> > > Oh, I misunderstood what this patch is changing. So despite being tagged\n> > > builtin/maintenance and talking about \"git maintenance\" it does not actually\n> > > touch builtin/maintenance.c or change its behavior. What it is actually\n> > > doing is changing how other git commands run \"git maintenance --auto\" so\n> > > that it is always run in the background unless the user configures\n> > > maintenance.autoDetach=false. That sounds like a good change.\n> > > \n> > > Thanks for clarifying\n> \n> Sorry my message sounds grumpier than I intended.\n\nNo worries, I didn't receive it as grumpy.\n\n> > Yes. I should've probably prefixed this with \"run-command:\", not with\n> > \"builtin/maintenance\".\n> \n> That's a good idea, I think the important thing we want to convey is that\n> we're changing the way we run \"git maintenance --auto\", not the behavior of\n> \"git maintenance\" itself.\n\nI'll have a look in case I need to reroll this series.\n\nPatrick\n"},{"id":"500954","messageId":"D3G99ASPVAJC.BIWK2TH6EIQA@jamesliu.io","threadId":"61940","inReplyTo":"812c61c9b66d7608e41c6c1d00a6e22e995cef06.1723533091.git.ps@pks.im","subject":"Re: [PATCH 4/7] builtin/gc: stop processing log file on signal","fromName":"James Liu","fromEmail":"james@jamesliu.io","sentAt":"2024-08-15T06:01:43Z","receivedAt":"2024-08-15T06:01:47Z","isPatch":true,"sender":{"key":"james@jamesliu.io","avatar":"https://avatars.githubusercontent.com/u/4476568?v=4"},"body":"On Tue Aug 13, 2024 at 5:17 PM AEST, Patrick Steinhardt wrote:\n> The consequence is that \"gc.log\" will not be committed, and thus\n> subsequent calls to `git gc --auto` won't bail out because of this.\n> Arguably though, it is better to retry garbage collection rather than\n> having the process run into a potentially-corrupted state.\n\nAhh I see, because `report_last_gc_error()` won't have anything to read.\nI agree it's an appropriate tradeoff given that garbage collection is\nnot on the critical path, and it's not likely that GC will be\ninterrupted on every attempt.\n\n> @@ -807,7 +800,6 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n> \t\t\t\t\t git_path(\"gc.log\"),\n> \t\t\t\t\t LOCK_DIE_ON_ERROR);\n> \t\tdup2(get_lock_file_fd(&log_lock), 2);\n> -\t\tsigchain_push_common(process_log_file_on_signal);\n> \t\tatexit(process_log_file_at_exit);\n> \t}\n> \n\n"},{"id":"500955","messageId":"D3GA39J7D5UW.KCFXXN036OD@jamesliu.io","threadId":"61940","inReplyTo":"8d6cbae951177718b49d5cfbbeca2d5b0073e266.1723533091.git.ps@pks.im","subject":"Re: [PATCH 7/7] builtin/maintenance: fix auto-detach with non-standard tasks","fromName":"James Liu","fromEmail":"james@jamesliu.io","sentAt":"2024-08-15T06:40:51Z","receivedAt":"2024-08-15T06:40:55Z","isPatch":true,"sender":{"key":"james@jamesliu.io","avatar":"https://avatars.githubusercontent.com/u/4476568?v=4"},"body":"On Tue Aug 13, 2024 at 5:18 PM AEST, Patrick Steinhardt wrote:\n> diff --git a/run-command.c b/run-command.c\n> index 45ba544932..94f2f3079f 100644\n> --- a/run-command.c\n> +++ b/run-command.c\n> @@ -1808,16 +1808,26 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)\n> \n> int prepare_auto_maintenance(int quiet, struct child_process *maint)\n> {\n> -\tint enabled;\n> +\tint enabled, auto_detach;\n> \n> \tif (!git_config_get_bool(\"maintenance.auto\", &enabled) &&\n> \t !enabled)\n> \t\treturn 0;\n> \n> +\t/*\n> +\t * When `maintenance.autoDetach` isn't set, then we fall back to\n> +\t * honoring `gc.autoDetach`. This is somewhat weird, but required to\n> +\t * retain behaviour from when we used to run git-gc(1) here.\n> +\t */\n> +\tif (git_config_get_bool(\"maintenance.autodetach\", &auto_detach) &&\n> +\t git_config_get_bool(\"gc.autodetach\", &auto_detach))\n> +\t\tauto_detach = 1;\n> +\n\nDo the two `*.autodetach` values need to be camel-cased or does it not\nmatter? I've noticed a mix of both through the codebase so I suppose\nit's not case-sensitive.\n\n"},{"id":"500956","messageId":"D3GA4UVELOIG.VNY8MZEAEM5P@jamesliu.io","threadId":"61940","inReplyTo":"cover.1723533091.git.ps@pks.im","subject":"Re: [PATCH 0/7] builtin/maintenance: fix auto-detach with non-standard tasks","fromName":"James Liu","fromEmail":"james@jamesliu.io","sentAt":"2024-08-15T06:42:56Z","receivedAt":"2024-08-15T06:43:00Z","isPatch":true,"sender":{"key":"james@jamesliu.io","avatar":"https://avatars.githubusercontent.com/u/4476568?v=4"},"body":"On Tue Aug 13, 2024 at 5:17 PM AEST, Patrick Steinhardt wrote:\n> Hi,\n>\n> I recently configured git-maintenance(1) to not use git-gc(1) anymore,\n> but instead to use git-multi-pack-index(1). I quickly noticed that the\n> behaviour here is somewhat broken because instead of auto-detaching when\n> `git maintenance run --auto` executes, we wait for the process to run to\n> completion.\n>\n> The root cause is that git-maintenance(1), probably by accident,\n> continues to rely on the auto-detaching mechanism in git-gc(1). So\n> instead of having git-maintenance(1) detach, it is git-gc(1) that\n> detaches and thus causes git-maintenance(1) to exit early. That of\n> course falls flat once any maintenance task other than git-gc(1)\n> executes, because these won't detach.\n>\n> Despite being a usability issue, this may also cause git-gc(1) to run\n> concurrently with any other enabled maintenance tasks. This shouldn't\n> lead to data loss, but it can certainly lead to processes stomping on\n> each others feet.\n>\n> This patch series fixes this by wiring up new `--detach` flags for both\n> git-gc(1) and git-maintenance(1). Like this, git-maintenance(1) now\n> knows to execute `git gc --auto --no-detach`, while our auto-maintenance\n> will execute `git mainteance run --auto --detach`.\n>\n> Patrick\n>\n> Patrick Steinhardt (7):\n> config: fix constness of out parameter for `git_config_get_expiry()`\n> builtin/gc: refactor to read config into structure\n> builtin/gc: fix leaking config values\n> builtin/gc: stop processing log file on signal\n> builtin/gc: add a `--detach` flag\n> builtin/maintenance: add a `--detach` flag\n> builtin/maintenance: fix auto-detach with non-standard tasks\n>\n> Documentation/git-gc.txt | 5 +-\n> builtin/gc.c | 384 ++++++++++++++++++++++++---------------\n> config.c | 4 +-\n> config.h | 2 +-\n> read-cache.c | 12 +-\n> run-command.c | 12 +-\n> t/t5304-prune.sh | 1 +\n> t/t5616-partial-clone.sh | 6 +-\n> t/t6500-gc.sh | 39 ++++\n> t/t7900-maintenance.sh | 82 ++++++++-\n> 10 files changed, 381 insertions(+), 166 deletions(-)\n\nThanks Patrick! I've left a few replies, mostly non-blocking questions.\n\nCheers,\nJames\n"},{"id":"500970","messageId":"Zr25t1S-srZOShLC@tanuki","threadId":"61940","inReplyTo":"D3GA39J7D5UW.KCFXXN036OD@jamesliu.io","subject":"Re: [PATCH 7/7] builtin/maintenance: fix auto-detach with non-standard tasks","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-08-15T08:17:59Z","receivedAt":"2024-08-15T11:11:14Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Thu, Aug 15, 2024 at 04:40:51PM +1000, James Liu wrote:\n> On Tue Aug 13, 2024 at 5:18 PM AEST, Patrick Steinhardt wrote:\n> > diff --git a/run-command.c b/run-command.c\n> > index 45ba544932..94f2f3079f 100644\n> > --- a/run-command.c\n> > +++ b/run-command.c\n> > @@ -1808,16 +1808,26 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)\n> > \n> > int prepare_auto_maintenance(int quiet, struct child_process *maint)\n> > {\n> > -\tint enabled;\n> > +\tint enabled, auto_detach;\n> > \n> > \tif (!git_config_get_bool(\"maintenance.auto\", &enabled) &&\n> > \t !enabled)\n> > \t\treturn 0;\n> > \n> > +\t/*\n> > +\t * When `maintenance.autoDetach` isn't set, then we fall back to\n> > +\t * honoring `gc.autoDetach`. This is somewhat weird, but required to\n> > +\t * retain behaviour from when we used to run git-gc(1) here.\n> > +\t */\n> > +\tif (git_config_get_bool(\"maintenance.autodetach\", &auto_detach) &&\n> > +\t git_config_get_bool(\"gc.autodetach\", &auto_detach))\n> > +\t\tauto_detach = 1;\n> > +\n> \n> Do the two `*.autodetach` values need to be camel-cased or does it not\n> matter? I've noticed a mix of both through the codebase so I suppose\n> it's not case-sensitive.\n\nConfig keys are case-insensitive in general, and as far as I am aware we\ntypically use the all-lowercase variant when retrieving config keys. On\nthe other hand, in our docs we spell the config keys camel-cased to help\nthe user make sense of it.\n\nPatrick\n"},{"id":"500971","messageId":"Zr25uqmZIHgMBzWC@tanuki","threadId":"61940","inReplyTo":"D3G8GP38BNG8.26Y1ZLIUPTWJB@jamesliu.io","subject":"Re: [PATCH 2/7] builtin/gc: refactor to read config into structure","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-08-15T08:18:02Z","receivedAt":"2024-08-15T11:11:14Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Thu, Aug 15, 2024 at 03:24:21PM +1000, James Liu wrote:\n> On Tue Aug 13, 2024 at 5:17 PM AEST, Patrick Steinhardt wrote:\n> > @@ -206,7 +224,7 @@ struct maintenance_run_opts {\n> > \tenum schedule_priority schedule;\n> > };\n> > \n> > -static int pack_refs_condition(void)\n> > +static int pack_refs_condition(UNUSED struct gc_config *cfg)\n> > {\n> > \t/*\n> > \t * The auto-repacking logic for refs is handled by the ref backends and\n> > @@ -216,7 +234,8 @@ static int pack_refs_condition(void)\n> > \treturn 1;\n> > }\n> > \n> > -static int maintenance_task_pack_refs(MAYBE_UNUSED struct maintenance_run_opts *opts)\n> > +static int maintenance_task_pack_refs(MAYBE_UNUSED struct maintenance_run_opts *opts,\n> > +\t\t\t\t UNUSED struct gc_config *cfg)\n> > {\n> > \tstruct child_process cmd = CHILD_PROCESS_INIT;\n> > \n> \n> Are we defining *cfg as an unused argument to conform to the\n> `maintenance_task_fn` signature?\n\nYup, that's exactly it.\n\nPatrick\n"},{"id":"500973","messageId":"Zr25vm8cLhS0_sKV@tanuki","threadId":"61940","inReplyTo":"D3G8EY3R09BC.2BKB6UND5J5MA@jamesliu.io","subject":"Re: [PATCH 3/7] builtin/gc: fix leaking config values","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-08-15T08:18:06Z","receivedAt":"2024-08-15T11:11:14Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Thu, Aug 15, 2024 at 03:22:04PM +1000, James Liu wrote:\n> On Tue Aug 13, 2024 at 5:17 PM AEST, Patrick Steinhardt wrote:\n> > Note that there is one small gotcha here with the \"--prune\" option. Next\n> > to passing a string, this option also accepts the \"--no-prune\" option\n> > that overrides the default or configured value. We thus need to discern\n> > between the option not having been passed by the user and the negative\n> > variant of it. This is done by using a simple sentinel value that lets\n> > us discern these cases.\n> >\n> > @@ -644,12 +673,15 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n> > \tstruct child_process rerere_cmd = CHILD_PROCESS_INIT;\n> > \tstruct maintenance_run_opts opts = {0};\n> > \tstruct gc_config cfg = GC_CONFIG_INIT;\n> > +\tconst char *prune_expire_sentinel = \"sentinel\";\n> > +\tconst char *prune_expire_arg = prune_expire_sentinel;\n> > +\tint ret;\n> > \n> > \tstruct option builtin_gc_options[] = {\n> > \t\tOPT__QUIET(&quiet, N_(\"suppress progress reporting\")),\n> > -\t\t{ OPTION_STRING, 0, \"prune\", &cfg.prune_expire, N_(\"date\"),\n> > +\t\t{ OPTION_STRING, 0, \"prune\", &prune_expire_arg, N_(\"date\"),\n> > \t\t\tN_(\"prune unreferenced objects\"),\n> > -\t\t\tPARSE_OPT_OPTARG, NULL, (intptr_t)cfg.prune_expire },\n> > +\t\t\tPARSE_OPT_OPTARG, NULL, (intptr_t)prune_expire_arg },\n> > \t\tOPT_BOOL(0, \"cruft\", &cfg.cruft_packs, N_(\"pack unreferenced objects separately\")),\n> > \t\tOPT_MAGNITUDE(0, \"max-cruft-size\", &cfg.max_cruft_size,\n> > \t\t\t N_(\"with --cruft, limit the size of new cruft packs\")),\n> \n> I was wondering how the `no-*` options worked since they're not\n> explicitly defined in the `builtin_gc_options` array. I guess\n> they're handled internally by `parse_options()`, and if the `no-` prefix\n> was present, it leaves that argument unset.\n\nYes, this is being handled by \"parse-options.c\". But With the `no-`\nprefix it won't leave the argument unset, but will explicitly unset it.\nSo if the argument was already set to some value X, that value would be\nunset when parsing the `no-` variant.\n\nPatrick\n"},{"id":"500972","messageId":"ff6aa9d7ba0bd67d3682ecb44c024a6d59a7decd.1723712608.git.ps@pks.im","threadId":"61940","inReplyTo":"cover.1723712608.git.ps@pks.im","subject":"[PATCH v2 2/7] builtin/gc: refactor to read config into structure","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-08-15T09:12:31Z","receivedAt":"2024-08-15T11:11:15Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The git-gc(1) command knows to read a bunch of config keys to tweak its\nown behaviour. The values are parsed into global variables, which makes\nit hard to correctly manage the lifecycle of values that may require a\nmemory allocation.\n\nRefactor the code to use a `struct gc_config` that gets populated and\npassed around. For one, this makes previously-implicit dependencies on\nthese config values clear. Second, it will allow us to properly manage\nthe lifecycle in the next commit.\n\nSigned-off-by: Patrick Steinhardt \n---\n builtin/gc.c | 255 +++++++++++++++++++++++++++++----------------------\n 1 file changed, 143 insertions(+), 112 deletions(-)\n\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex e7406bf667..eee7401647 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -49,23 +49,7 @@ static const char * const builtin_gc_usage[] = {\n \tNULL\n };\n \n-static int pack_refs = 1;\n-static int prune_reflogs = 1;\n-static int cruft_packs = 1;\n-static unsigned long max_cruft_size;\n-static int aggressive_depth = 50;\n-static int aggressive_window = 250;\n-static int gc_auto_threshold = 6700;\n-static int gc_auto_pack_limit = 50;\n-static int detach_auto = 1;\n static timestamp_t gc_log_expire_time;\n-static const char *gc_log_expire = \"1.day.ago\";\n-static const char *prune_expire = \"2.weeks.ago\";\n-static const char *prune_worktrees_expire = \"3.months.ago\";\n-static char *repack_filter;\n-static char *repack_filter_to;\n-static unsigned long big_pack_threshold;\n-static unsigned long max_delta_cache_size = DEFAULT_DELTA_CACHE_SIZE;\n \n static struct strvec reflog = STRVEC_INIT;\n static struct strvec repack = STRVEC_INIT;\n@@ -145,37 +129,71 @@ static int gc_config_is_timestamp_never(const char *var)\n \treturn 0;\n }\n \n-static void gc_config(void)\n+struct gc_config {\n+\tint pack_refs;\n+\tint prune_reflogs;\n+\tint cruft_packs;\n+\tunsigned long max_cruft_size;\n+\tint aggressive_depth;\n+\tint aggressive_window;\n+\tint gc_auto_threshold;\n+\tint gc_auto_pack_limit;\n+\tint detach_auto;\n+\tconst char *gc_log_expire;\n+\tconst char *prune_expire;\n+\tconst char *prune_worktrees_expire;\n+\tchar *repack_filter;\n+\tchar *repack_filter_to;\n+\tunsigned long big_pack_threshold;\n+\tunsigned long max_delta_cache_size;\n+};\n+\n+#define GC_CONFIG_INIT { \\\n+\t.pack_refs = 1, \\\n+\t.prune_reflogs = 1, \\\n+\t.cruft_packs = 1, \\\n+\t.aggressive_depth = 50, \\\n+\t.aggressive_window = 250, \\\n+\t.gc_auto_threshold = 6700, \\\n+\t.gc_auto_pack_limit = 50, \\\n+\t.detach_auto = 1, \\\n+\t.gc_log_expire = \"1.day.ago\", \\\n+\t.prune_expire = \"2.weeks.ago\", \\\n+\t.prune_worktrees_expire = \"3.months.ago\", \\\n+\t.max_delta_cache_size = DEFAULT_DELTA_CACHE_SIZE, \\\n+}\n+\n+static void gc_config(struct gc_config *cfg)\n {\n \tconst char *value;\n \n \tif (!git_config_get_value(\"gc.packrefs\", &value)) {\n \t\tif (value && !strcmp(value, \"notbare\"))\n-\t\t\tpack_refs = -1;\n+\t\t\tcfg->pack_refs = -1;\n \t\telse\n-\t\t\tpack_refs = git_config_bool(\"gc.packrefs\", value);\n+\t\t\tcfg->pack_refs = git_config_bool(\"gc.packrefs\", value);\n \t}\n \n \tif (gc_config_is_timestamp_never(\"gc.reflogexpire\") &&\n \t gc_config_is_timestamp_never(\"gc.reflogexpireunreachable\"))\n-\t\tprune_reflogs = 0;\n+\t\tcfg->prune_reflogs = 0;\n \n-\tgit_config_get_int(\"gc.aggressivewindow\", &aggressive_window);\n-\tgit_config_get_int(\"gc.aggressivedepth\", &aggressive_depth);\n-\tgit_config_get_int(\"gc.auto\", &gc_auto_threshold);\n-\tgit_config_get_int(\"gc.autopacklimit\", &gc_auto_pack_limit);\n-\tgit_config_get_bool(\"gc.autodetach\", &detach_auto);\n-\tgit_config_get_bool(\"gc.cruftpacks\", &cruft_packs);\n-\tgit_config_get_ulong(\"gc.maxcruftsize\", &max_cruft_size);\n-\tgit_config_get_expiry(\"gc.pruneexpire\", (char **) &prune_expire);\n-\tgit_config_get_expiry(\"gc.worktreepruneexpire\", (char **) &prune_worktrees_expire);\n-\tgit_config_get_expiry(\"gc.logexpiry\", (char **) &gc_log_expire);\n+\tgit_config_get_int(\"gc.aggressivewindow\", &cfg->aggressive_window);\n+\tgit_config_get_int(\"gc.aggressivedepth\", &cfg->aggressive_depth);\n+\tgit_config_get_int(\"gc.auto\", &cfg->gc_auto_threshold);\n+\tgit_config_get_int(\"gc.autopacklimit\", &cfg->gc_auto_pack_limit);\n+\tgit_config_get_bool(\"gc.autodetach\", &cfg->detach_auto);\n+\tgit_config_get_bool(\"gc.cruftpacks\", &cfg->cruft_packs);\n+\tgit_config_get_ulong(\"gc.maxcruftsize\", &cfg->max_cruft_size);\n+\tgit_config_get_expiry(\"gc.pruneexpire\", (char **) &cfg->prune_expire);\n+\tgit_config_get_expiry(\"gc.worktreepruneexpire\", (char **) &cfg->prune_worktrees_expire);\n+\tgit_config_get_expiry(\"gc.logexpiry\", (char **) &cfg->gc_log_expire);\n \n-\tgit_config_get_ulong(\"gc.bigpackthreshold\", &big_pack_threshold);\n-\tgit_config_get_ulong(\"pack.deltacachesize\", &max_delta_cache_size);\n+\tgit_config_get_ulong(\"gc.bigpackthreshold\", &cfg->big_pack_threshold);\n+\tgit_config_get_ulong(\"pack.deltacachesize\", &cfg->max_delta_cache_size);\n \n-\tgit_config_get_string(\"gc.repackfilter\", &repack_filter);\n-\tgit_config_get_string(\"gc.repackfilterto\", &repack_filter_to);\n+\tgit_config_get_string(\"gc.repackfilter\", &cfg->repack_filter);\n+\tgit_config_get_string(\"gc.repackfilterto\", &cfg->repack_filter_to);\n \n \tgit_config(git_default_config, NULL);\n }\n@@ -206,7 +224,7 @@ struct maintenance_run_opts {\n \tenum schedule_priority schedule;\n };\n \n-static int pack_refs_condition(void)\n+static int pack_refs_condition(UNUSED struct gc_config *cfg)\n {\n \t/*\n \t * The auto-repacking logic for refs is handled by the ref backends and\n@@ -216,7 +234,8 @@ static int pack_refs_condition(void)\n \treturn 1;\n }\n \n-static int maintenance_task_pack_refs(MAYBE_UNUSED struct maintenance_run_opts *opts)\n+static int maintenance_task_pack_refs(MAYBE_UNUSED struct maintenance_run_opts *opts,\n+\t\t\t\t UNUSED struct gc_config *cfg)\n {\n \tstruct child_process cmd = CHILD_PROCESS_INIT;\n \n@@ -228,7 +247,7 @@ static int maintenance_task_pack_refs(MAYBE_UNUSED struct maintenance_run_opts *\n \treturn run_command(&cmd);\n }\n \n-static int too_many_loose_objects(void)\n+static int too_many_loose_objects(struct gc_config *cfg)\n {\n \t/*\n \t * Quickly check if a \"gc\" is needed, by estimating how\n@@ -247,7 +266,7 @@ static int too_many_loose_objects(void)\n \tif (!dir)\n \t\treturn 0;\n \n-\tauto_threshold = DIV_ROUND_UP(gc_auto_threshold, 256);\n+\tauto_threshold = DIV_ROUND_UP(cfg->gc_auto_threshold, 256);\n \twhile ((ent = readdir(dir)) != NULL) {\n \t\tif (strspn(ent->d_name, \"0123456789abcdef\") != hexsz_loose ||\n \t\t ent->d_name[hexsz_loose] != '\\0')\n@@ -283,12 +302,12 @@ static struct packed_git *find_base_packs(struct string_list *packs,\n \treturn base;\n }\n \n-static int too_many_packs(void)\n+static int too_many_packs(struct gc_config *cfg)\n {\n \tstruct packed_git *p;\n \tint cnt;\n \n-\tif (gc_auto_pack_limit <= 0)\n+\tif (cfg->gc_auto_pack_limit <= 0)\n \t\treturn 0;\n \n \tfor (cnt = 0, p = get_all_packs(the_repository); p; p = p->next) {\n@@ -302,7 +321,7 @@ static int too_many_packs(void)\n \t\t */\n \t\tcnt++;\n \t}\n-\treturn gc_auto_pack_limit < cnt;\n+\treturn cfg->gc_auto_pack_limit < cnt;\n }\n \n static uint64_t total_ram(void)\n@@ -336,7 +355,8 @@ static uint64_t total_ram(void)\n \treturn 0;\n }\n \n-static uint64_t estimate_repack_memory(struct packed_git *pack)\n+static uint64_t estimate_repack_memory(struct gc_config *cfg,\n+\t\t\t\t struct packed_git *pack)\n {\n \tunsigned long nr_objects = repo_approximate_object_count(the_repository);\n \tsize_t os_cache, heap;\n@@ -373,7 +393,7 @@ static uint64_t estimate_repack_memory(struct packed_git *pack)\n \t */\n \theap += delta_base_cache_limit;\n \t/* and of course pack-objects has its own delta cache */\n-\theap += max_delta_cache_size;\n+\theap += cfg->max_delta_cache_size;\n \n \treturn os_cache + heap;\n }\n@@ -384,30 +404,31 @@ static int keep_one_pack(struct string_list_item *item, void *data UNUSED)\n \treturn 0;\n }\n \n-static void add_repack_all_option(struct string_list *keep_pack)\n+static void add_repack_all_option(struct gc_config *cfg,\n+\t\t\t\t struct string_list *keep_pack)\n {\n-\tif (prune_expire && !strcmp(prune_expire, \"now\"))\n+\tif (cfg->prune_expire && !strcmp(cfg->prune_expire, \"now\"))\n \t\tstrvec_push(&repack, \"-a\");\n-\telse if (cruft_packs) {\n+\telse if (cfg->cruft_packs) {\n \t\tstrvec_push(&repack, \"--cruft\");\n-\t\tif (prune_expire)\n-\t\t\tstrvec_pushf(&repack, \"--cruft-expiration=%s\", prune_expire);\n-\t\tif (max_cruft_size)\n+\t\tif (cfg->prune_expire)\n+\t\t\tstrvec_pushf(&repack, \"--cruft-expiration=%s\", cfg->prune_expire);\n+\t\tif (cfg->max_cruft_size)\n \t\t\tstrvec_pushf(&repack, \"--max-cruft-size=%lu\",\n-\t\t\t\t max_cruft_size);\n+\t\t\t\t cfg->max_cruft_size);\n \t} else {\n \t\tstrvec_push(&repack, \"-A\");\n-\t\tif (prune_expire)\n-\t\t\tstrvec_pushf(&repack, \"--unpack-unreachable=%s\", prune_expire);\n+\t\tif (cfg->prune_expire)\n+\t\t\tstrvec_pushf(&repack, \"--unpack-unreachable=%s\", cfg->prune_expire);\n \t}\n \n \tif (keep_pack)\n \t\tfor_each_string_list(keep_pack, keep_one_pack, NULL);\n \n-\tif (repack_filter && *repack_filter)\n-\t\tstrvec_pushf(&repack, \"--filter=%s\", repack_filter);\n-\tif (repack_filter_to && *repack_filter_to)\n-\t\tstrvec_pushf(&repack, \"--filter-to=%s\", repack_filter_to);\n+\tif (cfg->repack_filter && *cfg->repack_filter)\n+\t\tstrvec_pushf(&repack, \"--filter=%s\", cfg->repack_filter);\n+\tif (cfg->repack_filter_to && *cfg->repack_filter_to)\n+\t\tstrvec_pushf(&repack, \"--filter-to=%s\", cfg->repack_filter_to);\n }\n \n static void add_repack_incremental_option(void)\n@@ -415,13 +436,13 @@ static void add_repack_incremental_option(void)\n \tstrvec_push(&repack, \"--no-write-bitmap-index\");\n }\n \n-static int need_to_gc(void)\n+static int need_to_gc(struct gc_config *cfg)\n {\n \t/*\n \t * Setting gc.auto to 0 or negative can disable the\n \t * automatic gc.\n \t */\n-\tif (gc_auto_threshold <= 0)\n+\tif (cfg->gc_auto_threshold <= 0)\n \t\treturn 0;\n \n \t/*\n@@ -430,13 +451,13 @@ static int need_to_gc(void)\n \t * we run \"repack -A -d -l\". Otherwise we tell the caller\n \t * there is no need.\n \t */\n-\tif (too_many_packs()) {\n+\tif (too_many_packs(cfg)) {\n \t\tstruct string_list keep_pack = STRING_LIST_INIT_NODUP;\n \n-\t\tif (big_pack_threshold) {\n-\t\t\tfind_base_packs(&keep_pack, big_pack_threshold);\n-\t\t\tif (keep_pack.nr >= gc_auto_pack_limit) {\n-\t\t\t\tbig_pack_threshold = 0;\n+\t\tif (cfg->big_pack_threshold) {\n+\t\t\tfind_base_packs(&keep_pack, cfg->big_pack_threshold);\n+\t\t\tif (keep_pack.nr >= cfg->gc_auto_pack_limit) {\n+\t\t\t\tcfg->big_pack_threshold = 0;\n \t\t\t\tstring_list_clear(&keep_pack, 0);\n \t\t\t\tfind_base_packs(&keep_pack, 0);\n \t\t\t}\n@@ -445,7 +466,7 @@ static int need_to_gc(void)\n \t\t\tuint64_t mem_have, mem_want;\n \n \t\t\tmem_have = total_ram();\n-\t\t\tmem_want = estimate_repack_memory(p);\n+\t\t\tmem_want = estimate_repack_memory(cfg, p);\n \n \t\t\t/*\n \t\t\t * Only allow 1/2 of memory for pack-objects, leave\n@@ -456,9 +477,9 @@ static int need_to_gc(void)\n \t\t\t\tstring_list_clear(&keep_pack, 0);\n \t\t}\n \n-\t\tadd_repack_all_option(&keep_pack);\n+\t\tadd_repack_all_option(cfg, &keep_pack);\n \t\tstring_list_clear(&keep_pack, 0);\n-\t} else if (too_many_loose_objects())\n+\t} else if (too_many_loose_objects(cfg))\n \t\tadd_repack_incremental_option();\n \telse\n \t\treturn 0;\n@@ -585,7 +606,8 @@ static int report_last_gc_error(void)\n \treturn ret;\n }\n \n-static void gc_before_repack(struct maintenance_run_opts *opts)\n+static void gc_before_repack(struct maintenance_run_opts *opts,\n+\t\t\t struct gc_config *cfg)\n {\n \t/*\n \t * We may be called twice, as both the pre- and\n@@ -596,10 +618,10 @@ static void gc_before_repack(struct maintenance_run_opts *opts)\n \tif (done++)\n \t\treturn;\n \n-\tif (pack_refs && maintenance_task_pack_refs(opts))\n+\tif (cfg->pack_refs && maintenance_task_pack_refs(opts, cfg))\n \t\tdie(FAILED_RUN, \"pack-refs\");\n \n-\tif (prune_reflogs) {\n+\tif (cfg->prune_reflogs) {\n \t\tstruct child_process cmd = CHILD_PROCESS_INIT;\n \n \t\tcmd.git_cmd = 1;\n@@ -621,14 +643,15 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \ttimestamp_t dummy;\n \tstruct child_process rerere_cmd = CHILD_PROCESS_INIT;\n \tstruct maintenance_run_opts opts = {0};\n+\tstruct gc_config cfg = GC_CONFIG_INIT;\n \n \tstruct option builtin_gc_options[] = {\n \t\tOPT__QUIET(&quiet, N_(\"suppress progress reporting\")),\n-\t\t{ OPTION_STRING, 0, \"prune\", &prune_expire, N_(\"date\"),\n+\t\t{ OPTION_STRING, 0, \"prune\", &cfg.prune_expire, N_(\"date\"),\n \t\t\tN_(\"prune unreferenced objects\"),\n-\t\t\tPARSE_OPT_OPTARG, NULL, (intptr_t)prune_expire },\n-\t\tOPT_BOOL(0, \"cruft\", &cruft_packs, N_(\"pack unreferenced objects separately\")),\n-\t\tOPT_MAGNITUDE(0, \"max-cruft-size\", &max_cruft_size,\n+\t\t\tPARSE_OPT_OPTARG, NULL, (intptr_t)cfg.prune_expire },\n+\t\tOPT_BOOL(0, \"cruft\", &cfg.cruft_packs, N_(\"pack unreferenced objects separately\")),\n+\t\tOPT_MAGNITUDE(0, \"max-cruft-size\", &cfg.max_cruft_size,\n \t\t\t N_(\"with --cruft, limit the size of new cruft packs\")),\n \t\tOPT_BOOL(0, \"aggressive\", &aggressive, N_(\"be more thorough (increased runtime)\")),\n \t\tOPT_BOOL_F(0, \"auto\", &opts.auto_flag, N_(\"enable auto-gc mode\"),\n@@ -651,27 +674,27 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \tstrvec_pushl(&rerere, \"rerere\", \"gc\", NULL);\n \n \t/* default expiry time, overwritten in gc_config */\n-\tgc_config();\n-\tif (parse_expiry_date(gc_log_expire, &gc_log_expire_time))\n-\t\tdie(_(\"failed to parse gc.logExpiry value %s\"), gc_log_expire);\n+\tgc_config(&cfg);\n+\tif (parse_expiry_date(cfg.gc_log_expire, &gc_log_expire_time))\n+\t\tdie(_(\"failed to parse gc.logExpiry value %s\"), cfg.gc_log_expire);\n \n-\tif (pack_refs < 0)\n-\t\tpack_refs = !is_bare_repository();\n+\tif (cfg.pack_refs < 0)\n+\t\tcfg.pack_refs = !is_bare_repository();\n \n \targc = parse_options(argc, argv, prefix, builtin_gc_options,\n \t\t\t builtin_gc_usage, 0);\n \tif (argc > 0)\n \t\tusage_with_options(builtin_gc_usage, builtin_gc_options);\n \n-\tif (prune_expire && parse_expiry_date(prune_expire, &dummy))\n-\t\tdie(_(\"failed to parse prune expiry value %s\"), prune_expire);\n+\tif (cfg.prune_expire && parse_expiry_date(cfg.prune_expire, &dummy))\n+\t\tdie(_(\"failed to parse prune expiry value %s\"), cfg.prune_expire);\n \n \tif (aggressive) {\n \t\tstrvec_push(&repack, \"-f\");\n-\t\tif (aggressive_depth > 0)\n-\t\t\tstrvec_pushf(&repack, \"--depth=%d\", aggressive_depth);\n-\t\tif (aggressive_window > 0)\n-\t\t\tstrvec_pushf(&repack, \"--window=%d\", aggressive_window);\n+\t\tif (cfg.aggressive_depth > 0)\n+\t\t\tstrvec_pushf(&repack, \"--depth=%d\", cfg.aggressive_depth);\n+\t\tif (cfg.aggressive_window > 0)\n+\t\t\tstrvec_pushf(&repack, \"--window=%d\", cfg.aggressive_window);\n \t}\n \tif (quiet)\n \t\tstrvec_push(&repack, \"-q\");\n@@ -680,16 +703,16 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \t\t/*\n \t\t * Auto-gc should be least intrusive as possible.\n \t\t */\n-\t\tif (!need_to_gc())\n+\t\tif (!need_to_gc(&cfg))\n \t\t\treturn 0;\n \t\tif (!quiet) {\n-\t\t\tif (detach_auto)\n+\t\t\tif (cfg.detach_auto)\n \t\t\t\tfprintf(stderr, _(\"Auto packing the repository in background for optimum performance.\\n\"));\n \t\t\telse\n \t\t\t\tfprintf(stderr, _(\"Auto packing the repository for optimum performance.\\n\"));\n \t\t\tfprintf(stderr, _(\"See \\\"git help gc\\\" for manual housekeeping.\\n\"));\n \t\t}\n-\t\tif (detach_auto) {\n+\t\tif (cfg.detach_auto) {\n \t\t\tint ret = report_last_gc_error();\n \n \t\t\tif (ret == 1)\n@@ -701,7 +724,7 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \n \t\t\tif (lock_repo_for_gc(force, &pid))\n \t\t\t\treturn 0;\n-\t\t\tgc_before_repack(&opts); /* dies on failure */\n+\t\t\tgc_before_repack(&opts, &cfg); /* dies on failure */\n \t\t\tdelete_tempfile(&pidfile);\n \n \t\t\t/*\n@@ -716,11 +739,11 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \t\tif (keep_largest_pack != -1) {\n \t\t\tif (keep_largest_pack)\n \t\t\t\tfind_base_packs(&keep_pack, 0);\n-\t\t} else if (big_pack_threshold) {\n-\t\t\tfind_base_packs(&keep_pack, big_pack_threshold);\n+\t\t} else if (cfg.big_pack_threshold) {\n+\t\t\tfind_base_packs(&keep_pack, cfg.big_pack_threshold);\n \t\t}\n \n-\t\tadd_repack_all_option(&keep_pack);\n+\t\tadd_repack_all_option(&cfg, &keep_pack);\n \t\tstring_list_clear(&keep_pack, 0);\n \t}\n \n@@ -741,7 +764,7 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \t\tatexit(process_log_file_at_exit);\n \t}\n \n-\tgc_before_repack(&opts);\n+\tgc_before_repack(&opts, &cfg);\n \n \tif (!repository_format_precious_objects) {\n \t\tstruct child_process repack_cmd = CHILD_PROCESS_INIT;\n@@ -752,11 +775,11 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \t\tif (run_command(&repack_cmd))\n \t\t\tdie(FAILED_RUN, repack.v[0]);\n \n-\t\tif (prune_expire) {\n+\t\tif (cfg.prune_expire) {\n \t\t\tstruct child_process prune_cmd = CHILD_PROCESS_INIT;\n \n \t\t\t/* run `git prune` even if using cruft packs */\n-\t\t\tstrvec_push(&prune, prune_expire);\n+\t\t\tstrvec_push(&prune, cfg.prune_expire);\n \t\t\tif (quiet)\n \t\t\t\tstrvec_push(&prune, \"--no-progress\");\n \t\t\tif (repo_has_promisor_remote(the_repository))\n@@ -769,10 +792,10 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \t\t}\n \t}\n \n-\tif (prune_worktrees_expire) {\n+\tif (cfg.prune_worktrees_expire) {\n \t\tstruct child_process prune_worktrees_cmd = CHILD_PROCESS_INIT;\n \n-\t\tstrvec_push(&prune_worktrees, prune_worktrees_expire);\n+\t\tstrvec_push(&prune_worktrees, cfg.prune_worktrees_expire);\n \t\tprune_worktrees_cmd.git_cmd = 1;\n \t\tstrvec_pushv(&prune_worktrees_cmd.args, prune_worktrees.v);\n \t\tif (run_command(&prune_worktrees_cmd))\n@@ -796,7 +819,7 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \t\t\t\t\t !quiet && !daemonized ? COMMIT_GRAPH_WRITE_PROGRESS : 0,\n \t\t\t\t\t NULL);\n \n-\tif (opts.auto_flag && too_many_loose_objects())\n+\tif (opts.auto_flag && too_many_loose_objects(&cfg))\n \t\twarning(_(\"There are too many unreachable loose objects; \"\n \t\t\t\"run 'git prune' to remove them.\"));\n \n@@ -892,7 +915,7 @@ static int dfs_on_ref(const char *refname UNUSED,\n \treturn result;\n }\n \n-static int should_write_commit_graph(void)\n+static int should_write_commit_graph(struct gc_config *cfg)\n {\n \tint result;\n \tstruct cg_auto_data data;\n@@ -929,7 +952,8 @@ static int run_write_commit_graph(struct maintenance_run_opts *opts)\n \treturn !!run_command(&child);\n }\n \n-static int maintenance_task_commit_graph(struct maintenance_run_opts *opts)\n+static int maintenance_task_commit_graph(struct maintenance_run_opts *opts,\n+\t\t\t\t\t struct gc_config *cfg)\n {\n \tprepare_repo_settings(the_repository);\n \tif (!the_repository->settings.core_commit_graph)\n@@ -963,7 +987,8 @@ static int fetch_remote(struct remote *remote, void *cbdata)\n \treturn !!run_command(&child);\n }\n \n-static int maintenance_task_prefetch(struct maintenance_run_opts *opts)\n+static int maintenance_task_prefetch(struct maintenance_run_opts *opts,\n+\t\t\t\t struct gc_config *cfg)\n {\n \tif (for_each_remote(fetch_remote, opts)) {\n \t\terror(_(\"failed to prefetch remotes\"));\n@@ -973,7 +998,8 @@ static int maintenance_task_prefetch(struct maintenance_run_opts *opts)\n \treturn 0;\n }\n \n-static int maintenance_task_gc(struct maintenance_run_opts *opts)\n+static int maintenance_task_gc(struct maintenance_run_opts *opts,\n+\t\t\t struct gc_config *cfg)\n {\n \tstruct child_process child = CHILD_PROCESS_INIT;\n \n@@ -1021,7 +1047,7 @@ static int loose_object_count(const struct object_id *oid UNUSED,\n \treturn 0;\n }\n \n-static int loose_object_auto_condition(void)\n+static int loose_object_auto_condition(struct gc_config *cfg)\n {\n \tint count = 0;\n \n@@ -1106,12 +1132,13 @@ static int pack_loose(struct maintenance_run_opts *opts)\n \treturn result;\n }\n \n-static int maintenance_task_loose_objects(struct maintenance_run_opts *opts)\n+static int maintenance_task_loose_objects(struct maintenance_run_opts *opts,\n+\t\t\t\t\t struct gc_config *cfg)\n {\n \treturn prune_packed(opts) || pack_loose(opts);\n }\n \n-static int incremental_repack_auto_condition(void)\n+static int incremental_repack_auto_condition(struct gc_config *cfg)\n {\n \tstruct packed_git *p;\n \tint incremental_repack_auto_limit = 10;\n@@ -1230,7 +1257,8 @@ static int multi_pack_index_repack(struct maintenance_run_opts *opts)\n \treturn 0;\n }\n \n-static int maintenance_task_incremental_repack(struct maintenance_run_opts *opts)\n+static int maintenance_task_incremental_repack(struct maintenance_run_opts *opts,\n+\t\t\t\t\t struct gc_config *cfg)\n {\n \tprepare_repo_settings(the_repository);\n \tif (!the_repository->settings.core_multi_pack_index) {\n@@ -1247,14 +1275,15 @@ static int maintenance_task_incremental_repack(struct maintenance_run_opts *opts\n \treturn 0;\n }\n \n-typedef int maintenance_task_fn(struct maintenance_run_opts *opts);\n+typedef int maintenance_task_fn(struct maintenance_run_opts *opts,\n+\t\t\t\tstruct gc_config *cfg);\n \n /*\n * An auto condition function returns 1 if the task should run\n * and 0 if the task should NOT run. See needs_to_gc() for an\n * example.\n */\n-typedef int maintenance_auto_fn(void);\n+typedef int maintenance_auto_fn(struct gc_config *cfg);\n \n struct maintenance_task {\n \tconst char *name;\n@@ -1321,7 +1350,8 @@ static int compare_tasks_by_selection(const void *a_, const void *b_)\n \treturn b->selected_order - a->selected_order;\n }\n \n-static int maintenance_run_tasks(struct maintenance_run_opts *opts)\n+static int maintenance_run_tasks(struct maintenance_run_opts *opts,\n+\t\t\t\t struct gc_config *cfg)\n {\n \tint i, found_selected = 0;\n \tint result = 0;\n@@ -1360,14 +1390,14 @@ static int maintenance_run_tasks(struct maintenance_run_opts *opts)\n \n \t\tif (opts->auto_flag &&\n \t\t (!tasks[i].auto_condition ||\n-\t\t !tasks[i].auto_condition()))\n+\t\t !tasks[i].auto_condition(cfg)))\n \t\t\tcontinue;\n \n \t\tif (opts->schedule && tasks[i].schedule < opts->schedule)\n \t\t\tcontinue;\n \n \t\ttrace2_region_enter(\"maintenance\", tasks[i].name, r);\n-\t\tif (tasks[i].fn(opts)) {\n+\t\tif (tasks[i].fn(opts, cfg)) {\n \t\t\terror(_(\"task '%s' failed\"), tasks[i].name);\n \t\t\tresult = 1;\n \t\t}\n@@ -1404,7 +1434,6 @@ static void initialize_task_config(int schedule)\n {\n \tint i;\n \tstruct strbuf config_name = STRBUF_INIT;\n-\tgc_config();\n \n \tif (schedule)\n \t\tinitialize_maintenance_strategy();\n@@ -1468,6 +1497,7 @@ static int maintenance_run(int argc, const char **argv, const char *prefix)\n {\n \tint i;\n \tstruct maintenance_run_opts opts;\n+\tstruct gc_config cfg = GC_CONFIG_INIT;\n \tstruct option builtin_maintenance_run_options[] = {\n \t\tOPT_BOOL(0, \"auto\", &opts.auto_flag,\n \t\t\t N_(\"run tasks based on the state of the repository\")),\n@@ -1496,12 +1526,13 @@ static int maintenance_run(int argc, const char **argv, const char *prefix)\n \tif (opts.auto_flag && opts.schedule)\n \t\tdie(_(\"use at most one of --auto and --schedule=\"));\n \n+\tgc_config(&cfg);\n \tinitialize_task_config(opts.schedule);\n \n \tif (argc != 0)\n \t\tusage_with_options(builtin_maintenance_run_usage,\n \t\t\t\t builtin_maintenance_run_options);\n-\treturn maintenance_run_tasks(&opts);\n+\treturn maintenance_run_tasks(&opts, &cfg);\n }\n \n static char *get_maintpath(void)\n-- \n2.46.0.46.g406f326d27.dirty\n\n"},{"id":"500980","messageId":"040453f27f0b2755bb359fe07a703ce66e071ab6.1723712608.git.ps@pks.im","threadId":"61940","inReplyTo":"cover.1723712608.git.ps@pks.im","subject":"[PATCH v2 1/7] config: fix constness of out parameter for `git_config_get_expiry()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-08-15T09:12:29Z","receivedAt":"2024-08-15T11:22:02Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The type of the out parameter of `git_config_get_expiry()` is a pointer\nto a constant string, which creates the impression that ownership of the\nreturned data wasn't transferred to the caller. This isn't true though\nand thus quite misleading.\n\nAdapt the parameter to be of type `char **` and adjust callers\naccordingly. While at it, refactor `get_shared_index_expire_date()` to\ndrop the static `shared_index_expire` variable. It is only used in that\nfunction, and furthermore we would only hit the code where we parse the\nexpiry date a single time because we already use a static `prepared`\nvariable to track whether we did parse it.\n\nSigned-off-by: Patrick Steinhardt \n---\n builtin/gc.c | 6 +++---\n config.c | 4 ++--\n config.h | 2 +-\n read-cache.c | 12 +++++++++---\n 4 files changed, 15 insertions(+), 9 deletions(-)\n\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 72bac2554f..e7406bf667 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -167,9 +167,9 @@ static void gc_config(void)\n \tgit_config_get_bool(\"gc.autodetach\", &detach_auto);\n \tgit_config_get_bool(\"gc.cruftpacks\", &cruft_packs);\n \tgit_config_get_ulong(\"gc.maxcruftsize\", &max_cruft_size);\n-\tgit_config_get_expiry(\"gc.pruneexpire\", &prune_expire);\n-\tgit_config_get_expiry(\"gc.worktreepruneexpire\", &prune_worktrees_expire);\n-\tgit_config_get_expiry(\"gc.logexpiry\", &gc_log_expire);\n+\tgit_config_get_expiry(\"gc.pruneexpire\", (char **) &prune_expire);\n+\tgit_config_get_expiry(\"gc.worktreepruneexpire\", (char **) &prune_worktrees_expire);\n+\tgit_config_get_expiry(\"gc.logexpiry\", (char **) &gc_log_expire);\n \n \tgit_config_get_ulong(\"gc.bigpackthreshold\", &big_pack_threshold);\n \tgit_config_get_ulong(\"pack.deltacachesize\", &max_delta_cache_size);\ndiff --git a/config.c b/config.c\nindex 6421894614..dfa4df1417 100644\n--- a/config.c\n+++ b/config.c\n@@ -2766,9 +2766,9 @@ int git_config_get_pathname(const char *key, char **dest)\n \treturn repo_config_get_pathname(the_repository, key, dest);\n }\n \n-int git_config_get_expiry(const char *key, const char **output)\n+int git_config_get_expiry(const char *key, char **output)\n {\n-\tint ret = git_config_get_string(key, (char **)output);\n+\tint ret = git_config_get_string(key, output);\n \tif (ret)\n \t\treturn ret;\n \tif (strcmp(*output, \"now\")) {\ndiff --git a/config.h b/config.h\nindex 54b47dec9e..4801391c32 100644\n--- a/config.h\n+++ b/config.h\n@@ -701,7 +701,7 @@ int git_config_get_split_index(void);\n int git_config_get_max_percent_split_change(void);\n \n /* This dies if the configured or default date is in the future */\n-int git_config_get_expiry(const char *key, const char **output);\n+int git_config_get_expiry(const char *key, char **output);\n \n /* parse either \"this many days\" integer, or \"5.days.ago\" approxidate */\n int git_config_get_expiry_in_days(const char *key, timestamp_t *, timestamp_t now);\ndiff --git a/read-cache.c b/read-cache.c\nindex 48bf24f87c..7f393ee687 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -3176,18 +3176,24 @@ static int write_split_index(struct index_state *istate,\n \treturn ret;\n }\n \n-static const char *shared_index_expire = \"2.weeks.ago\";\n-\n static unsigned long get_shared_index_expire_date(void)\n {\n \tstatic unsigned long shared_index_expire_date;\n \tstatic int shared_index_expire_date_prepared;\n \n \tif (!shared_index_expire_date_prepared) {\n+\t\tconst char *shared_index_expire = \"2.weeks.ago\";\n+\t\tchar *value = NULL;\n+\n \t\tgit_config_get_expiry(\"splitindex.sharedindexexpire\",\n-\t\t\t\t &shared_index_expire);\n+\t\t\t\t &value);\n+\t\tif (value)\n+\t\t\tshared_index_expire = value;\n+\n \t\tshared_index_expire_date = approxidate(shared_index_expire);\n \t\tshared_index_expire_date_prepared = 1;\n+\n+\t\tfree(value);\n \t}\n \n \treturn shared_index_expire_date;\n-- \n2.46.0.46.g406f326d27.dirty\n\n"},{"id":"500983","messageId":"cover.1723712608.git.ps@pks.im","threadId":"61940","inReplyTo":"cover.1723533091.git.ps@pks.im","subject":"[PATCH v2 0/7] builtin/maintenance: fix auto-detach with non-standard tasks","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-08-15T09:12:25Z","receivedAt":"2024-08-15T11:22:02Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Hi,\n\nthis is the second version of my patch series that fixes how\ngit-maintenance(1) detaches: instead of letting its child process\ngit-gc(1) detach, we now optionally ask git-maintenance(1) itself to\ndetach when running via our auto maintenance mechanism. This fixes\nbehaviour of git-maintenance(1) when configured to run non-standard\ntasks like the \"incremental\" task.\n\nThere is only a single change compared to v1, namely a rewording of the\nfinal commit message. It now clarifies that we really only fix the auto\nmaintenance behaviour, and the default behaviour of git-maintenance(1)\nremains the same when ran by the user.\n\nPatrick Steinhardt (7):\n config: fix constness of out parameter for `git_config_get_expiry()`\n builtin/gc: refactor to read config into structure\n builtin/gc: fix leaking config values\n builtin/gc: stop processing log file on signal\n builtin/gc: add a `--detach` flag\n builtin/maintenance: add a `--detach` flag\n run-command: fix detaching when running auto maintenance\n\n Documentation/git-gc.txt | 5 +-\n builtin/gc.c | 384 ++++++++++++++++++++++++---------------\n config.c | 4 +-\n config.h | 2 +-\n read-cache.c | 12 +-\n run-command.c | 12 +-\n t/t5304-prune.sh | 1 +\n t/t5616-partial-clone.sh | 6 +-\n t/t6500-gc.sh | 39 ++++\n t/t7900-maintenance.sh | 82 ++++++++-\n 10 files changed, 381 insertions(+), 166 deletions(-)\n\nRange-diff against v1:\n1: 040453f27f = 1: 040453f27f config: fix constness of out parameter for `git_config_get_expiry()`\n2: ff6aa9d7ba = 2: ff6aa9d7ba builtin/gc: refactor to read config into structure\n3: 310e361371 = 3: 310e361371 builtin/gc: fix leaking config values\n4: 812c61c9b6 = 4: 812c61c9b6 builtin/gc: stop processing log file on signal\n5: ca78d3dc7c = 5: ca78d3dc7c builtin/gc: add a `--detach` flag\n6: 06dbb73425 = 6: 06dbb73425 builtin/maintenance: add a `--detach` flag\n7: 8d6cbae951 ! 7: 6bc170ff05 builtin/maintenance: fix auto-detach with non-standard tasks\n @@ Metadata\n Author: Patrick Steinhardt \n \n ## Commit message ##\n - builtin/maintenance: fix auto-detach with non-standard tasks\n + run-command: fix detaching when running auto maintenance\n \n In the past, we used to execute `git gc --auto` as part of our automatic\n housekeeping routines. As git-gc(1) may require quite some time to\n @@ Commit message\n housekeeping tasks, stomping on each others feet.\n \n Fix this bug by asking git-gc(1) to not detach when it is being invoked\n - via git-maintenance(1). Instead, the latter command now respects a new\n + via git-maintenance(1). Instead, git-maintenance(1) now respects a new\n config \"maintenance.autoDetach\", the equivalent of \"gc.autoDetach\", and\n - detaches itself into the background if not told otherwise. This should\n - continue to behave the same for all users which use the git-gc(1) task,\n - only. For others though, it means that we now properly perform all tasks\n - in the background.\n + detaches itself into the background when running as part of our auto\n + maintenance. This should continue to behave the same for all users which\n + use the git-gc(1) task, only. For others though, it means that we now\n + properly perform all tasks in the background. The default behaviour of\n + git-maintenance(1) when executed by the user does not change, it will\n + remain in the foreground unless they pass the `--detach` option.\n \n Signed-off-by: Patrick Steinhardt \n \n-- \n2.46.0.46.g406f326d27.dirty\n\n"},{"id":"500985","messageId":"06dbb73425f7df038dfdd9bc0d2b9a49edfe064e.1723712608.git.ps@pks.im","threadId":"61940","inReplyTo":"cover.1723712608.git.ps@pks.im","subject":"[PATCH v2 6/7] builtin/maintenance: add a `--detach` flag","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-08-15T09:12:46Z","receivedAt":"2024-08-15T11:22:05Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Same as the preceding commit, add a `--[no-]detach` flag to the\ngit-maintenance(1) command. This will be used in a subsequent commit to\nfix backgrounding of that command when configured with a non-standard\nset of tasks.\n\nSigned-off-by: Patrick Steinhardt \n---\n builtin/gc.c | 6 ++++++\n t/t7900-maintenance.sh | 39 +++++++++++++++++++++++++++++++++++++++\n 2 files changed, 45 insertions(+)\n\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 269a77960f..63106e2028 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -1426,6 +1426,10 @@ static int maintenance_run_tasks(struct maintenance_run_opts *opts,\n \t}\n \tfree(lock_path);\n \n+\t/* Failure to daemonize is ok, we'll continue in foreground. */\n+\tif (opts->detach > 0)\n+\t\tdaemonize();\n+\n \tfor (i = 0; !found_selected && i < TASK__COUNT; i++)\n \t\tfound_selected = tasks[i].selected_order >= 0;\n \n@@ -1552,6 +1556,8 @@ static int maintenance_run(int argc, const char **argv, const char *prefix)\n \tstruct option builtin_maintenance_run_options[] = {\n \t\tOPT_BOOL(0, \"auto\", &opts.auto_flag,\n \t\t\t N_(\"run tasks based on the state of the repository\")),\n+\t\tOPT_BOOL(0, \"detach\", &opts.detach,\n+\t\t\t N_(\"perform maintenance in the background\")),\n \t\tOPT_CALLBACK(0, \"schedule\", &opts.schedule, N_(\"frequency\"),\n \t\t\t N_(\"run tasks based on frequency\"),\n \t\t\t maintenance_opt_schedule),\ndiff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\nindex 8595489ceb..771525aa4b 100755\n--- a/t/t7900-maintenance.sh\n+++ b/t/t7900-maintenance.sh\n@@ -908,4 +908,43 @@ test_expect_success 'failed schedule prevents config change' '\n \tdone\n '\n \n+test_expect_success '--no-detach causes maintenance to not run in background' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\n+\t\t# Prepare the repository such that git-maintenance(1) ends up\n+\t\t# outputting something.\n+\t\ttest_commit something &&\n+\t\tgit config set maintenance.gc.enabled false &&\n+\t\tgit config set maintenance.loose-objects.enabled true &&\n+\t\tgit config set maintenance.loose-objects.auto 1 &&\n+\t\tgit config set maintenance.incremental-repack.enabled true &&\n+\n+\t\t# We have no better way to check whether or not the task ran in\n+\t\t# the background than to verify whether it output anything. The\n+\t\t# next testcase checks the reverse, making this somewhat safer.\n+\t\tgit maintenance run --no-detach >out 2>&1 &&\n+\t\ttest_line_count = 1 out\n+\t)\n+'\n+\n+test_expect_success '--detach causes maintenance to run in background' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\n+\t\ttest_commit something &&\n+\t\tgit config set maintenance.gc.enabled false &&\n+\t\tgit config set maintenance.loose-objects.enabled true &&\n+\t\tgit config set maintenance.loose-objects.auto 1 &&\n+\t\tgit config set maintenance.incremental-repack.enabled true &&\n+\n+\t\tgit maintenance run --detach >out 2>&1 &&\n+\t\ttest_must_be_empty out\n+\t)\n+'\n+\n test_done\n-- \n2.46.0.46.g406f326d27.dirty\n\n"},{"id":"500986","messageId":"ca78d3dc7c0270b434ee4ca4ef618212c7dc1d5b.1723712608.git.ps@pks.im","threadId":"61940","inReplyTo":"cover.1723712608.git.ps@pks.im","subject":"[PATCH v2 5/7] builtin/gc: add a `--detach` flag","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-08-15T09:12:43Z","receivedAt":"2024-08-15T11:22:05Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"When running `git gc --auto`, the command will by default detach and\ncontinue running in the background. This behaviour can be tweaked via\nthe `gc.autoDetach` config, but not via a command line switch. We need\nthat in a subsequent commit though, where git-maintenance(1) will want\nto ask its git-gc(1) child process to not detach anymore.\n\nAdd a `--[no-]detach` flag that does this for us.\n\nSigned-off-by: Patrick Steinhardt \n---\n Documentation/git-gc.txt | 5 ++-\n builtin/gc.c | 70 ++++++++++++++++++++++------------------\n t/t6500-gc.sh | 39 ++++++++++++++++++++++\n 3 files changed, 82 insertions(+), 32 deletions(-)\n\ndiff --git a/Documentation/git-gc.txt b/Documentation/git-gc.txt\nindex b5561c458a..370e22faae 100644\n--- a/Documentation/git-gc.txt\n+++ b/Documentation/git-gc.txt\n@@ -9,7 +9,7 @@ git-gc - Cleanup unnecessary files and optimize the local repository\n SYNOPSIS\n --------\n [verse]\n-'git gc' [--aggressive] [--auto] [--quiet] [--prune= | --no-prune] [--force] [--keep-largest-pack]\n+'git gc' [--aggressive] [--auto] [--[no-]detach] [--quiet] [--prune= | --no-prune] [--force] [--keep-largest-pack]\n \n DESCRIPTION\n -----------\n@@ -53,6 +53,9 @@ configuration options such as `gc.auto` and `gc.autoPackLimit`, all\n other housekeeping tasks (e.g. rerere, working trees, reflog...) will\n be performed as well.\n \n+--[no-]detach::\n+\tRun in the background if the system supports it. This option overrides\n+\tthe `gc.autoDetach` config.\n \n --[no-]cruft::\n \tWhen expiring unreachable objects, pack them separately into a\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex f815557081..269a77960f 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -242,9 +242,13 @@ static enum schedule_priority parse_schedule(const char *value)\n \n struct maintenance_run_opts {\n \tint auto_flag;\n+\tint detach;\n \tint quiet;\n \tenum schedule_priority schedule;\n };\n+#define MAINTENANCE_RUN_OPTS_INIT { \\\n+\t.detach = -1, \\\n+}\n \n static int pack_refs_condition(UNUSED struct gc_config *cfg)\n {\n@@ -664,7 +668,7 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \tint keep_largest_pack = -1;\n \ttimestamp_t dummy;\n \tstruct child_process rerere_cmd = CHILD_PROCESS_INIT;\n-\tstruct maintenance_run_opts opts = {0};\n+\tstruct maintenance_run_opts opts = MAINTENANCE_RUN_OPTS_INIT;\n \tstruct gc_config cfg = GC_CONFIG_INIT;\n \tconst char *prune_expire_sentinel = \"sentinel\";\n \tconst char *prune_expire_arg = prune_expire_sentinel;\n@@ -681,6 +685,8 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \t\tOPT_BOOL(0, \"aggressive\", &aggressive, N_(\"be more thorough (increased runtime)\")),\n \t\tOPT_BOOL_F(0, \"auto\", &opts.auto_flag, N_(\"enable auto-gc mode\"),\n \t\t\t PARSE_OPT_NOCOMPLETE),\n+\t\tOPT_BOOL(0, \"detach\", &opts.detach,\n+\t\t\t N_(\"perform garbage collection in the background\")),\n \t\tOPT_BOOL_F(0, \"force\", &force,\n \t\t\t N_(\"force running gc even if there may be another gc running\"),\n \t\t\t PARSE_OPT_NOCOMPLETE),\n@@ -729,6 +735,9 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \t\tstrvec_push(&repack, \"-q\");\n \n \tif (opts.auto_flag) {\n+\t\tif (cfg.detach_auto && opts.detach < 0)\n+\t\t\topts.detach = 1;\n+\n \t\t/*\n \t\t * Auto-gc should be least intrusive as possible.\n \t\t */\n@@ -738,38 +747,12 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \t\t}\n \n \t\tif (!quiet) {\n-\t\t\tif (cfg.detach_auto)\n+\t\t\tif (opts.detach > 0)\n \t\t\t\tfprintf(stderr, _(\"Auto packing the repository in background for optimum performance.\\n\"));\n \t\t\telse\n \t\t\t\tfprintf(stderr, _(\"Auto packing the repository for optimum performance.\\n\"));\n \t\t\tfprintf(stderr, _(\"See \\\"git help gc\\\" for manual housekeeping.\\n\"));\n \t\t}\n-\t\tif (cfg.detach_auto) {\n-\t\t\tret = report_last_gc_error();\n-\t\t\tif (ret == 1) {\n-\t\t\t\t/* Last gc --auto failed. Skip this one. */\n-\t\t\t\tret = 0;\n-\t\t\t\tgoto out;\n-\n-\t\t\t} else if (ret) {\n-\t\t\t\t/* an I/O error occurred, already reported */\n-\t\t\t\tgoto out;\n-\t\t\t}\n-\n-\t\t\tif (lock_repo_for_gc(force, &pid)) {\n-\t\t\t\tret = 0;\n-\t\t\t\tgoto out;\n-\t\t\t}\n-\n-\t\t\tgc_before_repack(&opts, &cfg); /* dies on failure */\n-\t\t\tdelete_tempfile(&pidfile);\n-\n-\t\t\t/*\n-\t\t\t * failure to daemonize is ok, we'll continue\n-\t\t\t * in foreground\n-\t\t\t */\n-\t\t\tdaemonized = !daemonize();\n-\t\t}\n \t} else {\n \t\tstruct string_list keep_pack = STRING_LIST_INIT_NODUP;\n \n@@ -784,6 +767,33 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \t\tstring_list_clear(&keep_pack, 0);\n \t}\n \n+\tif (opts.detach > 0) {\n+\t\tret = report_last_gc_error();\n+\t\tif (ret == 1) {\n+\t\t\t/* Last gc --auto failed. Skip this one. */\n+\t\t\tret = 0;\n+\t\t\tgoto out;\n+\n+\t\t} else if (ret) {\n+\t\t\t/* an I/O error occurred, already reported */\n+\t\t\tgoto out;\n+\t\t}\n+\n+\t\tif (lock_repo_for_gc(force, &pid)) {\n+\t\t\tret = 0;\n+\t\t\tgoto out;\n+\t\t}\n+\n+\t\tgc_before_repack(&opts, &cfg); /* dies on failure */\n+\t\tdelete_tempfile(&pidfile);\n+\n+\t\t/*\n+\t\t * failure to daemonize is ok, we'll continue\n+\t\t * in foreground\n+\t\t */\n+\t\tdaemonized = !daemonize();\n+\t}\n+\n \tname = lock_repo_for_gc(force, &pid);\n \tif (name) {\n \t\tif (opts.auto_flag) {\n@@ -1537,7 +1547,7 @@ static int task_option_parse(const struct option *opt UNUSED,\n static int maintenance_run(int argc, const char **argv, const char *prefix)\n {\n \tint i;\n-\tstruct maintenance_run_opts opts;\n+\tstruct maintenance_run_opts opts = MAINTENANCE_RUN_OPTS_INIT;\n \tstruct gc_config cfg = GC_CONFIG_INIT;\n \tstruct option builtin_maintenance_run_options[] = {\n \t\tOPT_BOOL(0, \"auto\", &opts.auto_flag,\n@@ -1554,8 +1564,6 @@ static int maintenance_run(int argc, const char **argv, const char *prefix)\n \t};\n \tint ret;\n \n-\tmemset(&opts, 0, sizeof(opts));\n-\n \topts.quiet = !isatty(2);\n \n \tfor (i = 0; i < TASK__COUNT; i++)\ndiff --git a/t/t6500-gc.sh b/t/t6500-gc.sh\nindex 1b5909d1b7..737c99e0f8 100755\n--- a/t/t6500-gc.sh\n+++ b/t/t6500-gc.sh\n@@ -396,6 +396,45 @@ test_expect_success 'background auto gc respects lock for all operations' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success '--detach overrides gc.autoDetach=false' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\n+\t\t# Prepare the repository such that git-gc(1) ends up repacking.\n+\t\ttest_commit \"$(test_oid blob17_1)\" &&\n+\t\ttest_commit \"$(test_oid blob17_2)\" &&\n+\t\tgit config gc.autodetach false &&\n+\t\tgit config gc.auto 2 &&\n+\n+\t\tcat >expect <<-EOF &&\n+\t\tAuto packing the repository in background for optimum performance.\n+\t\tSee \"git help gc\" for manual housekeeping.\n+\t\tEOF\n+\t\tGIT_PROGRESS_DELAY=0 git gc --auto --detach 2>actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success '--no-detach overrides gc.autoDetach=true' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\n+\t\t# Prepare the repository such that git-gc(1) ends up repacking.\n+\t\ttest_commit \"$(test_oid blob17_1)\" &&\n+\t\ttest_commit \"$(test_oid blob17_2)\" &&\n+\t\tgit config gc.autodetach true &&\n+\t\tgit config gc.auto 2 &&\n+\n+\t\tGIT_PROGRESS_DELAY=0 git gc --auto --no-detach 2>output &&\n+\t\ttest_grep \"Auto packing the repository for optimum performance.\" output &&\n+\t\ttest_grep \"Collecting referenced commits: 2, done.\" output\n+\t)\n+'\n+\n # DO NOT leave a detached auto gc process running near the end of the\n # test script: it can run long enough in the background to racily\n # interfere with the cleanup in 'test_done'.\n-- \n2.46.0.46.g406f326d27.dirty\n\n"},{"id":"500987","messageId":"812c61c9b66d7608e41c6c1d00a6e22e995cef06.1723712608.git.ps@pks.im","threadId":"61940","inReplyTo":"cover.1723712608.git.ps@pks.im","subject":"[PATCH v2 4/7] builtin/gc: stop processing log file on signal","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-08-15T09:12:37Z","receivedAt":"2024-08-15T11:22:05Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"When detaching, git-gc(1) will redirect its stderr to a \"gc.log\" log\nfile, which is then used to surface errors of a backgrounded process to\nthe user. To ensure that the file is properly managed on abnormal exit\npaths, we install both signal and exit handlers that try to either\ncommit the underlying lock file or roll it back in case there wasn't any\nerror.\n\nThis logic is severly broken when handling signals though, as we end up\ncalling all kinds of functions that are not signal safe. This includes\nmalloc(3P) via `git_path()`, fprintf(3P), fflush(3P) and many more\nfunctions. The consequence can be anything, from deadlocks to crashes.\nUnfortunately, we cannot really do much about this without a larger\nrefactoring.\n\nThe least-worst thing we can do is to not set up the signal handler in\nthe first place. This will still cause us to remove the lockfile, as the\nunderlying tempfile subsystem already knows to unlink locks when\nreceiving a signal. But it may cause us to remove the lock even in the\ncase where it would have contained actual errors, which is a change in\nbehaviour.\n\nThe consequence is that \"gc.log\" will not be committed, and thus\nsubsequent calls to `git gc --auto` won't bail out because of this.\nArguably though, it is better to retry garbage collection rather than\nhaving the process run into a potentially-corrupted state.\n\nSigned-off-by: Patrick Steinhardt \n---\n builtin/gc.c | 8 --------\n 1 file changed, 8 deletions(-)\n\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex a93cfa147e..f815557081 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -109,13 +109,6 @@ static void process_log_file_at_exit(void)\n \tprocess_log_file();\n }\n \n-static void process_log_file_on_signal(int signo)\n-{\n-\tprocess_log_file();\n-\tsigchain_pop(signo);\n-\traise(signo);\n-}\n-\n static int gc_config_is_timestamp_never(const char *var)\n {\n \tconst char *value;\n@@ -807,7 +800,6 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \t\t\t\t\t git_path(\"gc.log\"),\n \t\t\t\t\t LOCK_DIE_ON_ERROR);\n \t\tdup2(get_lock_file_fd(&log_lock), 2);\n-\t\tsigchain_push_common(process_log_file_on_signal);\n \t\tatexit(process_log_file_at_exit);\n \t}\n \n-- \n2.46.0.46.g406f326d27.dirty\n\n"},{"id":"500988","messageId":"6bc170ff05d38845a012ce57e580a0ddd18f1143.1723712608.git.ps@pks.im","threadId":"61940","inReplyTo":"cover.1723712608.git.ps@pks.im","subject":"[PATCH v2 7/7] run-command: fix detaching when running auto maintenance","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-08-15T09:12:49Z","receivedAt":"2024-08-15T11:22:05Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"In the past, we used to execute `git gc --auto` as part of our automatic\nhousekeeping routines. As git-gc(1) may require quite some time to\nperform the housekeeping, it knows to detach itself and run in the\nbackground so that the user can continue their work.\n\nEventually, we refactored our automatic housekeeping to instead use the\nmore flexible git-maintenance(1) command. The upside of this new infra\nis that the user can configure which maintenance tasks are performed, at\nleast to a certain degree. So while it continues to run git-gc(1) by\ndefault, it can also be adapted to e.g. use git-multi-pack-index(1) for\nmaintenance of the object database.\n\nThe auto-detach of the new infra is somewhat broken though once the user\nconfigures non-standard tasks. The problem is essentially that we detach\nat the wrong level in the process hierarchy: git-maintenance(1) never\ndetaches itself, but instead it continues to be git-gc(1) which does.\n\nWhen configured to only run the git-gc(1) maintenance task, then the\nresult is basically the same as before. But when configured to run other\ntasks, then git-maintenance(1) will wait for these to run to completion.\nEven worse, it may be that git-gc(1) runs concurrently with other\nhousekeeping tasks, stomping on each others feet.\n\nFix this bug by asking git-gc(1) to not detach when it is being invoked\nvia git-maintenance(1). Instead, git-maintenance(1) now respects a new\nconfig \"maintenance.autoDetach\", the equivalent of \"gc.autoDetach\", and\ndetaches itself into the background when running as part of our auto\nmaintenance. This should continue to behave the same for all users which\nuse the git-gc(1) task, only. For others though, it means that we now\nproperly perform all tasks in the background. The default behaviour of\ngit-maintenance(1) when executed by the user does not change, it will\nremain in the foreground unless they pass the `--detach` option.\n\nSigned-off-by: Patrick Steinhardt \n---\n builtin/gc.c | 1 +\n run-command.c | 12 ++++++++++-\n t/t5616-partial-clone.sh | 6 +++---\n t/t7900-maintenance.sh | 43 +++++++++++++++++++++++++++++++---------\n 4 files changed, 49 insertions(+), 13 deletions(-)\n\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 63106e2028..bafee330a2 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -1063,6 +1063,7 @@ static int maintenance_task_gc(struct maintenance_run_opts *opts,\n \t\tstrvec_push(&child.args, \"--quiet\");\n \telse\n \t\tstrvec_push(&child.args, \"--no-quiet\");\n+\tstrvec_push(&child.args, \"--no-detach\");\n \n \treturn run_command(&child);\n }\ndiff --git a/run-command.c b/run-command.c\nindex 45ba544932..94f2f3079f 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -1808,16 +1808,26 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)\n \n int prepare_auto_maintenance(int quiet, struct child_process *maint)\n {\n-\tint enabled;\n+\tint enabled, auto_detach;\n \n \tif (!git_config_get_bool(\"maintenance.auto\", &enabled) &&\n \t !enabled)\n \t\treturn 0;\n \n+\t/*\n+\t * When `maintenance.autoDetach` isn't set, then we fall back to\n+\t * honoring `gc.autoDetach`. This is somewhat weird, but required to\n+\t * retain behaviour from when we used to run git-gc(1) here.\n+\t */\n+\tif (git_config_get_bool(\"maintenance.autodetach\", &auto_detach) &&\n+\t git_config_get_bool(\"gc.autodetach\", &auto_detach))\n+\t\tauto_detach = 1;\n+\n \tmaint->git_cmd = 1;\n \tmaint->close_object_store = 1;\n \tstrvec_pushl(&maint->args, \"maintenance\", \"run\", \"--auto\", NULL);\n \tstrvec_push(&maint->args, quiet ? \"--quiet\" : \"--no-quiet\");\n+\tstrvec_push(&maint->args, auto_detach ? \"--detach\" : \"--no-detach\");\n \n \treturn 1;\n }\ndiff --git a/t/t5616-partial-clone.sh b/t/t5616-partial-clone.sh\nindex 2da7291e37..8415884754 100755\n--- a/t/t5616-partial-clone.sh\n+++ b/t/t5616-partial-clone.sh\n@@ -229,7 +229,7 @@ test_expect_success 'fetch --refetch triggers repacking' '\n \n \tGIT_TRACE2_EVENT=\"$PWD/trace1.event\" \\\n \tgit -C pc1 fetch --refetch origin &&\n-\ttest_subcommand git maintenance run --auto --no-quiet /dev/null &&\n \tGIT_TRACE2_EVENT=\"$(pwd)/run-no-quiet.txt\" \\\n \t\tgit maintenance run --no-quiet 2>/dev/null &&\n-\ttest_subcommand git gc --quiet ' '\n \t\tgit maintenance run --task=commit-graph 2>/dev/null &&\n \tGIT_TRACE2_EVENT=\"$(pwd)/run-both.txt\" \\\n \t\tgit maintenance run --task=commit-graph --task=gc 2>/dev/null &&\n-\ttest_subcommand ! git gc --quiet \n---\n builtin/gc.c | 108 +++++++++++++++++++++++++++++++++++------------\n t/t5304-prune.sh | 1 +\n 2 files changed, 82 insertions(+), 27 deletions(-)\n\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex eee7401647..a93cfa147e 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -139,9 +139,9 @@ struct gc_config {\n \tint gc_auto_threshold;\n \tint gc_auto_pack_limit;\n \tint detach_auto;\n-\tconst char *gc_log_expire;\n-\tconst char *prune_expire;\n-\tconst char *prune_worktrees_expire;\n+\tchar *gc_log_expire;\n+\tchar *prune_expire;\n+\tchar *prune_worktrees_expire;\n \tchar *repack_filter;\n \tchar *repack_filter_to;\n \tunsigned long big_pack_threshold;\n@@ -157,15 +157,25 @@ struct gc_config {\n \t.gc_auto_threshold = 6700, \\\n \t.gc_auto_pack_limit = 50, \\\n \t.detach_auto = 1, \\\n-\t.gc_log_expire = \"1.day.ago\", \\\n-\t.prune_expire = \"2.weeks.ago\", \\\n-\t.prune_worktrees_expire = \"3.months.ago\", \\\n+\t.gc_log_expire = xstrdup(\"1.day.ago\"), \\\n+\t.prune_expire = xstrdup(\"2.weeks.ago\"), \\\n+\t.prune_worktrees_expire = xstrdup(\"3.months.ago\"), \\\n \t.max_delta_cache_size = DEFAULT_DELTA_CACHE_SIZE, \\\n }\n \n+static void gc_config_release(struct gc_config *cfg)\n+{\n+\tfree(cfg->gc_log_expire);\n+\tfree(cfg->prune_expire);\n+\tfree(cfg->prune_worktrees_expire);\n+\tfree(cfg->repack_filter);\n+\tfree(cfg->repack_filter_to);\n+}\n+\n static void gc_config(struct gc_config *cfg)\n {\n \tconst char *value;\n+\tchar *owned = NULL;\n \n \tif (!git_config_get_value(\"gc.packrefs\", &value)) {\n \t\tif (value && !strcmp(value, \"notbare\"))\n@@ -185,15 +195,34 @@ static void gc_config(struct gc_config *cfg)\n \tgit_config_get_bool(\"gc.autodetach\", &cfg->detach_auto);\n \tgit_config_get_bool(\"gc.cruftpacks\", &cfg->cruft_packs);\n \tgit_config_get_ulong(\"gc.maxcruftsize\", &cfg->max_cruft_size);\n-\tgit_config_get_expiry(\"gc.pruneexpire\", (char **) &cfg->prune_expire);\n-\tgit_config_get_expiry(\"gc.worktreepruneexpire\", (char **) &cfg->prune_worktrees_expire);\n-\tgit_config_get_expiry(\"gc.logexpiry\", (char **) &cfg->gc_log_expire);\n+\n+\tif (!git_config_get_expiry(\"gc.pruneexpire\", &owned)) {\n+\t\tfree(cfg->prune_expire);\n+\t\tcfg->prune_expire = owned;\n+\t}\n+\n+\tif (!git_config_get_expiry(\"gc.worktreepruneexpire\", &owned)) {\n+\t\tfree(cfg->prune_worktrees_expire);\n+\t\tcfg->prune_worktrees_expire = owned;\n+\t}\n+\n+\tif (!git_config_get_expiry(\"gc.logexpiry\", &owned)) {\n+\t\tfree(cfg->gc_log_expire);\n+\t\tcfg->gc_log_expire = owned;\n+\t}\n \n \tgit_config_get_ulong(\"gc.bigpackthreshold\", &cfg->big_pack_threshold);\n \tgit_config_get_ulong(\"pack.deltacachesize\", &cfg->max_delta_cache_size);\n \n-\tgit_config_get_string(\"gc.repackfilter\", &cfg->repack_filter);\n-\tgit_config_get_string(\"gc.repackfilterto\", &cfg->repack_filter_to);\n+\tif (!git_config_get_string(\"gc.repackfilter\", &owned)) {\n+\t\tfree(cfg->repack_filter);\n+\t\tcfg->repack_filter = owned;\n+\t}\n+\n+\tif (!git_config_get_string(\"gc.repackfilterto\", &owned)) {\n+\t\tfree(cfg->repack_filter_to);\n+\t\tcfg->repack_filter_to = owned;\n+\t}\n \n \tgit_config(git_default_config, NULL);\n }\n@@ -644,12 +673,15 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \tstruct child_process rerere_cmd = CHILD_PROCESS_INIT;\n \tstruct maintenance_run_opts opts = {0};\n \tstruct gc_config cfg = GC_CONFIG_INIT;\n+\tconst char *prune_expire_sentinel = \"sentinel\";\n+\tconst char *prune_expire_arg = prune_expire_sentinel;\n+\tint ret;\n \n \tstruct option builtin_gc_options[] = {\n \t\tOPT__QUIET(&quiet, N_(\"suppress progress reporting\")),\n-\t\t{ OPTION_STRING, 0, \"prune\", &cfg.prune_expire, N_(\"date\"),\n+\t\t{ OPTION_STRING, 0, \"prune\", &prune_expire_arg, N_(\"date\"),\n \t\t\tN_(\"prune unreferenced objects\"),\n-\t\t\tPARSE_OPT_OPTARG, NULL, (intptr_t)cfg.prune_expire },\n+\t\t\tPARSE_OPT_OPTARG, NULL, (intptr_t)prune_expire_arg },\n \t\tOPT_BOOL(0, \"cruft\", &cfg.cruft_packs, N_(\"pack unreferenced objects separately\")),\n \t\tOPT_MAGNITUDE(0, \"max-cruft-size\", &cfg.max_cruft_size,\n \t\t\t N_(\"with --cruft, limit the size of new cruft packs\")),\n@@ -673,8 +705,8 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \tstrvec_pushl(&prune_worktrees, \"worktree\", \"prune\", \"--expire\", NULL);\n \tstrvec_pushl(&rerere, \"rerere\", \"gc\", NULL);\n \n-\t/* default expiry time, overwritten in gc_config */\n \tgc_config(&cfg);\n+\n \tif (parse_expiry_date(cfg.gc_log_expire, &gc_log_expire_time))\n \t\tdie(_(\"failed to parse gc.logExpiry value %s\"), cfg.gc_log_expire);\n \n@@ -686,6 +718,10 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \tif (argc > 0)\n \t\tusage_with_options(builtin_gc_usage, builtin_gc_options);\n \n+\tif (prune_expire_arg != prune_expire_sentinel) {\n+\t\tfree(cfg.prune_expire);\n+\t\tcfg.prune_expire = xstrdup_or_null(prune_expire_arg);\n+\t}\n \tif (cfg.prune_expire && parse_expiry_date(cfg.prune_expire, &dummy))\n \t\tdie(_(\"failed to parse prune expiry value %s\"), cfg.prune_expire);\n \n@@ -703,8 +739,11 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \t\t/*\n \t\t * Auto-gc should be least intrusive as possible.\n \t\t */\n-\t\tif (!need_to_gc(&cfg))\n-\t\t\treturn 0;\n+\t\tif (!need_to_gc(&cfg)) {\n+\t\t\tret = 0;\n+\t\t\tgoto out;\n+\t\t}\n+\n \t\tif (!quiet) {\n \t\t\tif (cfg.detach_auto)\n \t\t\t\tfprintf(stderr, _(\"Auto packing the repository in background for optimum performance.\\n\"));\n@@ -713,17 +752,22 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \t\t\tfprintf(stderr, _(\"See \\\"git help gc\\\" for manual housekeeping.\\n\"));\n \t\t}\n \t\tif (cfg.detach_auto) {\n-\t\t\tint ret = report_last_gc_error();\n-\n-\t\t\tif (ret == 1)\n+\t\t\tret = report_last_gc_error();\n+\t\t\tif (ret == 1) {\n \t\t\t\t/* Last gc --auto failed. Skip this one. */\n-\t\t\t\treturn 0;\n-\t\t\telse if (ret)\n+\t\t\t\tret = 0;\n+\t\t\t\tgoto out;\n+\n+\t\t\t} else if (ret) {\n \t\t\t\t/* an I/O error occurred, already reported */\n-\t\t\t\treturn ret;\n+\t\t\t\tgoto out;\n+\t\t\t}\n+\n+\t\t\tif (lock_repo_for_gc(force, &pid)) {\n+\t\t\t\tret = 0;\n+\t\t\t\tgoto out;\n+\t\t\t}\n \n-\t\t\tif (lock_repo_for_gc(force, &pid))\n-\t\t\t\treturn 0;\n \t\t\tgc_before_repack(&opts, &cfg); /* dies on failure */\n \t\t\tdelete_tempfile(&pidfile);\n \n@@ -749,8 +793,11 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \n \tname = lock_repo_for_gc(force, &pid);\n \tif (name) {\n-\t\tif (opts.auto_flag)\n-\t\t\treturn 0; /* be quiet on --auto */\n+\t\tif (opts.auto_flag) {\n+\t\t\tret = 0;\n+\t\t\tgoto out; /* be quiet on --auto */\n+\t\t}\n+\n \t\tdie(_(\"gc is already running on machine '%s' pid %\"PRIuMAX\" (use --force if not)\"),\n \t\t name, (uintmax_t)pid);\n \t}\n@@ -826,6 +873,8 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \tif (!daemonized)\n \t\tunlink(git_path(\"gc.log\"));\n \n+out:\n+\tgc_config_release(&cfg);\n \treturn 0;\n }\n \n@@ -1511,6 +1560,8 @@ static int maintenance_run(int argc, const char **argv, const char *prefix)\n \t\t\tPARSE_OPT_NONEG, task_option_parse),\n \t\tOPT_END()\n \t};\n+\tint ret;\n+\n \tmemset(&opts, 0, sizeof(opts));\n \n \topts.quiet = !isatty(2);\n@@ -1532,7 +1583,10 @@ static int maintenance_run(int argc, const char **argv, const char *prefix)\n \tif (argc != 0)\n \t\tusage_with_options(builtin_maintenance_run_usage,\n \t\t\t\t builtin_maintenance_run_options);\n-\treturn maintenance_run_tasks(&opts, &cfg);\n+\n+\tret = maintenance_run_tasks(&opts, &cfg);\n+\tgc_config_release(&cfg);\n+\treturn ret;\n }\n \n static char *get_maintpath(void)\ndiff --git a/t/t5304-prune.sh b/t/t5304-prune.sh\nindex 1f1f664871..e641df0116 100755\n--- a/t/t5304-prune.sh\n+++ b/t/t5304-prune.sh\n@@ -7,6 +7,7 @@ test_description='prune'\n GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n \n+TEST_PASSES_SANITIZE_LEAK=true\n . ./test-lib.sh\n \n day=$((60*60*24))\n-- \n2.46.0.46.g406f326d27.dirty\n\n"},{"id":"501005","messageId":"6304df4c-705a-42f7-9e43-1d89404731c6@gmail.com","threadId":"61940","inReplyTo":"ff6aa9d7ba0bd67d3682ecb44c024a6d59a7decd.1723533091.git.ps@pks.im","subject":"Re: [PATCH 2/7] builtin/gc: refactor to read config into structure","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2024-08-15T13:46:47Z","receivedAt":"2024-08-15T13:46:50Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 8/13/24 3:17 AM, Patrick Steinhardt wrote:\n> The git-gc(1) command knows to read a bunch of config keys to tweak its\n> own behaviour. The values are parsed into global variables, which makes\n> it hard to correctly manage the lifecycle of values that may require a\n> memory allocation.\n> \n> Refactor the code to use a `struct gc_config` that gets populated and\n> passed around. For one, this makes previously-implicit dependencies on\n> these config values clear. Second, it will allow us to properly manage\n> the lifecycle in the next commit.\n\nI think this is a valuable goal.\n\n> -static const char *gc_log_expire = \"1.day.ago\";\n> -static const char *prune_expire = \"2.weeks.ago\";\n> -static const char *prune_worktrees_expire = \"3.months.ago\";\n\nI was going to mention this in the previous patch where you change how\nthese variables are cast into git_config_get_expiry(). They aren't\nchanging to non-const here, either.\n\n> +struct gc_config {\n...\n> +\tconst char *gc_log_expire;\n> +\tconst char *prune_expire;\n> +\tconst char *prune_worktrees_expire;\n\nThe fact that they are initialized to const strings makes it\ndifficult to know if they've been updated. I wonder if we need\nto change them to have a \"if NULL, then use a const default\"\nsomewhere. (And maybe you do this later in the series).\n\n> +static void gc_config(struct gc_config *cfg)\n\nI appreciate that you are taking the step to make the structure\na process parameter and not just another global.\n\nThanks,\n-Stolee\n\n"},{"id":"501006","messageId":"23188674-a465-4af8-ae5a-2404ab6d4c1e@gmail.com","threadId":"61940","inReplyTo":"310e361371efc156c3aaac94439bdbeaa965155f.1723533091.git.ps@pks.im","subject":"Re: [PATCH 3/7] builtin/gc: fix leaking config values","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2024-08-15T13:50:38Z","receivedAt":"2024-08-15T13:50:40Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 8/13/24 3:17 AM, Patrick Steinhardt wrote:\n> We're leaking config values in git-gc(1) when those values are tracked\n> as strings. Introduce a new `gc_config_release()` function that releases\n> this memory to plug those leaks and release old values before populating\n> the config fields via `git_config_string()` et al.\n> \n> Note that there is one small gotcha here with the \"--prune\" option. Next\n> to passing a string, this option also accepts the \"--no-prune\" option\n> that overrides the default or configured value. We thus need to discern\n> between the option not having been passed by the user and the negative\n> variant of it. This is done by using a simple sentinel value that lets\n> us discern these cases.\n\nI'm glad to see that you are correcting this in the very next patch\nwhere I pointed out my concern about it. Excellent.\n\n> -\tconst char *gc_log_expire;\n> -\tconst char *prune_expire;\n> -\tconst char *prune_worktrees_expire;\n> +\tchar *gc_log_expire;\n> +\tchar *prune_expire;\n> +\tchar *prune_worktrees_expire;\n\n> -\t.gc_log_expire = \"1.day.ago\", \\\n> -\t.prune_expire = \"2.weeks.ago\", \\\n> -\t.prune_worktrees_expire = \"3.months.ago\", \\\n> +\t.gc_log_expire = xstrdup(\"1.day.ago\"), \\\n> +\t.prune_expire = xstrdup(\"2.weeks.ago\"), \\\n> +\t.prune_worktrees_expire = xstrdup(\"3.months.ago\"), \\\n\nUsing xstrdup() is now possible that these aren't globals. Good.\n\n> static void gc_config(struct gc_config *cfg)\n> {\n> \tconst char *value;\n> +\tchar *owned = NULL;\n> \n> \tif (!git_config_get_value(\"gc.packrefs\", &value)) {\n> \t\tif (value && !strcmp(value, \"notbare\"))\n> @@ -185,15 +195,34 @@ static void gc_config(struct gc_config *cfg)\n> \tgit_config_get_bool(\"gc.autodetach\", &cfg->detach_auto);\n> \tgit_config_get_bool(\"gc.cruftpacks\", &cfg->cruft_packs);\n> \tgit_config_get_ulong(\"gc.maxcruftsize\", &cfg->max_cruft_size);\n> -\tgit_config_get_expiry(\"gc.pruneexpire\", (char **) &cfg->prune_expire);\n> -\tgit_config_get_expiry(\"gc.worktreepruneexpire\", (char **) &cfg->prune_worktrees_expire);\n> -\tgit_config_get_expiry(\"gc.logexpiry\", (char **) &cfg->gc_log_expire);\n> +\n> +\tif (!git_config_get_expiry(\"gc.pruneexpire\", &owned)) {\n> +\t\tfree(cfg->prune_expire);\n> +\t\tcfg->prune_expire = owned;\n> +\t}\n\nAh. Good logic of \"if I get a (possibly NULL) result, then free the old\nvalue before setting the new one\".\n\n> diff --git a/t/t5304-prune.sh b/t/t5304-prune.sh\n> index 1f1f664871..e641df0116 100755\n> --- a/t/t5304-prune.sh\n> +++ b/t/t5304-prune.sh\n> @@ -7,6 +7,7 @@ test_description='prune'\n> GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n> export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n> \n> +TEST_PASSES_SANITIZE_LEAK=true\n\nFantastic.\n\nThanks,\n-Stolee\n\n"},{"id":"501007","messageId":"eddcc0a0-a366-4627-8878-f1b5d0af2436@gmail.com","threadId":"61940","inReplyTo":"8d6cbae951177718b49d5cfbbeca2d5b0073e266.1723533091.git.ps@pks.im","subject":"Re: [PATCH 7/7] builtin/maintenance: fix auto-detach with non-standard tasks","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2024-08-15T14:00:13Z","receivedAt":"2024-08-15T14:00:15Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 8/13/24 3:18 AM, Patrick Steinhardt wrote:\n\n> @@ -1063,6 +1063,7 @@ static int maintenance_task_gc(struct maintenance_run_opts *opts,\n> \t\tstrvec_push(&child.args, \"--quiet\");\n> \telse\n> \t\tstrvec_push(&child.args, \"--no-quiet\");\n> +\tstrvec_push(&child.args, \"--no-detach\");\n> \n> \treturn run_command(&child);\n> }\n\nI was looking for this earlier, as it could have been placed in either of\nthe previous two patches. I can understand not putting it in Patch 5\nbecause then the default of 'git maintenance run --auto' would not\ndaemonize the gc subprocess. But it seems like patch 6 would also have\nbeen fine. Here is good, too.\n\n> +\t/*\n> +\t * When `maintenance.autoDetach` isn't set, then we fall back to\n> +\t * honoring `gc.autoDetach`. This is somewhat weird, but required to\n> +\t * retain behaviour from when we used to run git-gc(1) here.\n> +\t */\n> +\tif (git_config_get_bool(\"maintenance.autodetach\", &auto_detach) &&\n> +\t git_config_get_bool(\"gc.autodetach\", &auto_detach))\n> +\t\tauto_detach = 1;\n> +\n\nThis && caught me by surprise, but it's really \"if both of these config\noptions are unset, then set a default.\" Makes sense after thinking a bit\nharder.\n\n> \tmaint->git_cmd = 1;\n> \tmaint->close_object_store = 1;\n> \tstrvec_pushl(&maint->args, \"maintenance\", \"run\", \"--auto\", NULL);\n> \tstrvec_push(&maint->args, quiet ? \"--quiet\" : \"--no-quiet\");\n> +\tstrvec_push(&maint->args, auto_detach ? \"--detach\" : \"--no-detach\");\n> \n> \treturn 1;\n> }\n> diff --git a/t/t5616-partial-clone.sh b/t/t5616-partial-clone.sh\n> index 2da7291e37..8415884754 100755\n> --- a/t/t5616-partial-clone.sh\n> +++ b/t/t5616-partial-clone.sh\n> @@ -229,7 +229,7 @@ test_expect_success 'fetch --refetch triggers repacking' '\n> \n> \tGIT_TRACE2_EVENT=\"$PWD/trace1.event\" \\\n> \tgit -C pc1 fetch --refetch origin &&\n> -\ttest_subcommand git maintenance run --auto --no-quiet +\ttest_subcommand git maintenance run --auto --no-quiet --detach +for cfg in maintenance.autoDetach gc.autoDetach\n> +do\n> +\ttest_expect_success \"$cfg=true config option\" '\n> +\t\ttest_when_finished \"rm -f trace\" &&\n> +\t\ttest_config $cfg true &&\n> +\t\tGIT_TRACE2_EVENT=\"$(pwd)/trace\" git commit --quiet --allow-empty -m 1 &&\n> +\t\ttest_subcommand git maintenance run --auto --quiet --detach +\t'\n> +\n> +\ttest_expect_success \"$cfg=false config option\" '\n> +\t\ttest_when_finished \"rm -f trace\" &&\n> +\t\ttest_config $cfg false &&\n> +\t\tGIT_TRACE2_EVENT=\"$(pwd)/trace\" git commit --quiet --allow-empty -m 1 &&\n> +\t\ttest_subcommand git maintenance run --auto --quiet --no-detach +\t'\n> +done\n> +\n> +test_expect_success \"maintenance.autoDetach overrides gc.autoDetach\" '\n> +\ttest_when_finished \"rm -f trace\" &&\n> +\ttest_config maintenance.autoDetach false &&\n> +\ttest_config gc.autoDetach true &&\n> +\tGIT_TRACE2_EVENT=\"$(pwd)/trace\" git commit --quiet --allow-empty -m 1 &&\n> +\ttest_subcommand git maintenance run --auto --quiet --no-detach '\n\nI appreciate the care taken in these test cases. It verifies the logic around\nthe if statement that I had to read twice.\n\nThanks,\n-Stolee\n\n"},{"id":"501008","messageId":"de298972-a359-46dd-b808-4edaa8701d40@gmail.com","threadId":"61940","inReplyTo":"cover.1723533091.git.ps@pks.im","subject":"Re: [PATCH 0/7] builtin/maintenance: fix auto-detach with non-standard tasks","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2024-08-15T14:04:10Z","receivedAt":"2024-08-15T14:04:15Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 8/13/24 3:17 AM, Patrick Steinhardt wrote:\n\n> I recently configured git-maintenance(1) to not use git-gc(1) anymore,\n> but instead to use git-multi-pack-index(1). I quickly noticed that the\n> behaviour here is somewhat broken because instead of auto-detaching when\n> `git maintenance run --auto` executes, we wait for the process to run to\n> completion.\n> \n> The root cause is that git-maintenance(1), probably by accident,\n> continues to rely on the auto-detaching mechanism in git-gc(1). So\n> instead of having git-maintenance(1) detach, it is git-gc(1) that\n> detaches and thus causes git-maintenance(1) to exit early. That of\n> course falls flat once any maintenance task other than git-gc(1)\n> executes, because these won't detach.\n> \n> Despite being a usability issue, this may also cause git-gc(1) to run\n> concurrently with any other enabled maintenance tasks. This shouldn't\n> lead to data loss, but it can certainly lead to processes stomping on\n> each others feet.\n> \n> This patch series fixes this by wiring up new `--detach` flags for both\n> git-gc(1) and git-maintenance(1). Like this, git-maintenance(1) now\n> knows to execute `git gc --auto --no-detach`, while our auto-maintenance\n> will execute `git mainteance run --auto --detach`.\n\nThank you for noticing this behavior, which is essentially an unintended\nregression from when the maintenance command was first introduced. It\nworked for most users because of the accidental detachment of the GC\ntask, but now users can correctly customize their automatic maintenance\nto run in the background.\n\nThis was my oversight, as I was focused on scheduled maintenance as\nbeing the primary way that users would customize their maintenance tasks.\nThank you for unifying the concepts.\n\nI sprinkled in commentary, and most of it was just things I noticed\nwhile reading the series in order but then later patches or a careful\nread made my comments non-actionable.\n\nThis v1 looks good to me.\n\nThanks,\n-Stolee\n\n"},{"id":"501016","messageId":"xmqq8qwxvltp.fsf@gitster.g","threadId":"61940","inReplyTo":"de298972-a359-46dd-b808-4edaa8701d40@gmail.com","subject":"Re: [PATCH 0/7] builtin/maintenance: fix auto-detach with non-standard tasks","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-08-15T15:37:38Z","receivedAt":"2024-08-15T15:37:46Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n> I sprinkled in commentary, and most of it was just things I noticed\n> while reading the series in order but then later patches or a careful\n> read made my comments non-actionable.\n>\n> This v1 looks good to me.\n\nThanks for a \"think aloud\" review. Very much appreciated.\n\nI thought there was a minor reroll for phrasing planned, without any\nother change on significant part of the series? Let me mark the\ntopic for 'next' when it happens.\n\nThanks, both.\n\n"},{"id":"501017","messageId":"xmqq4j7lvk5z.fsf@gitster.g","threadId":"61940","inReplyTo":"6bc170ff05d38845a012ce57e580a0ddd18f1143.1723712608.git.ps@pks.im","subject":"Re: [PATCH v2 7/7] run-command: fix detaching when running auto maintenance","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-08-15T16:13:28Z","receivedAt":"2024-08-15T16:13:32Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n> diff --git a/run-command.c b/run-command.c\n> index 45ba544932..94f2f3079f 100644\n> --- a/run-command.c\n> +++ b/run-command.c\n> @@ -1808,16 +1808,26 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)\n> \n> int prepare_auto_maintenance(int quiet, struct child_process *maint)\n> {\n> -\tint enabled;\n> +\tint enabled, auto_detach;\n> \n> \tif (!git_config_get_bool(\"maintenance.auto\", &enabled) &&\n> \t !enabled)\n> \t\treturn 0;\n> \n> +\t/*\n> +\t * When `maintenance.autoDetach` isn't set, then we fall back to\n> +\t * honoring `gc.autoDetach`. This is somewhat weird, but required to\n> +\t * retain behaviour from when we used to run git-gc(1) here.\n> +\t */\n> +\tif (git_config_get_bool(\"maintenance.autodetach\", &auto_detach) &&\n> +\t git_config_get_bool(\"gc.autodetach\", &auto_detach))\n> +\t\tauto_detach = 1;\n\nI think this needs somehow documented. Something like this,\nperhaps?\n\n Documentation/config/gc.txt | 2 ++\n Documentation/config/maintenance.txt | 9 +++++++++\n 2 files changed, 11 insertions(+)\n\ndiff --git c/Documentation/config/gc.txt w/Documentation/config/gc.txt\nindex 664a3c2874..6506ccb87f 100644\n--- c/Documentation/config/gc.txt\n+++ w/Documentation/config/gc.txt\n@@ -41,6 +41,8 @@ use, it'll affect how the auto pack limit works.\n gc.autoDetach::\n \tMake `git gc --auto` return immediately and run in the background\n \tif the system supports it. Default is true.\n+\tIt also acts as a fallback setting for the `maintenance.autoDetach`\n+\tconfiguration variable.\n \n gc.bigPackThreshold::\n \tIf non-zero, all non-cruft packs larger than this limit are kept\ndiff --git c/Documentation/config/maintenance.txt w/Documentation/config/maintenance.txt\nindex 69a4f05153..7a481a494a 100644\n--- c/Documentation/config/maintenance.txt\n+++ w/Documentation/config/maintenance.txt\n@@ -3,6 +3,15 @@ maintenance.auto::\n \t`git maintenance run --auto` after doing their normal work. Defaults\n \tto true.\n \n+maintenance.autoDetach::\n+\tTasks that are run via `git maintenance run --auto` by\n+\tdefault runs in the background, if the system supports it.\n+\tSetting this configuration variable to `true` explicitly\n+\tasks them to run in the background, and setting it to\n+\t`false` forces them to run in the foreground. If this\n+\tvariable is not set, `gc.autoDetach` works as a fallback\n+\tvariable and behaves the same way.\n+\n maintenance.strategy::\n \tThis string config option provides a way to specify one of a few\n \trecommended schedules for background maintenance. This only affects\n"},{"id":"501038","messageId":"xmqq34n5txcj.fsf@gitster.g","threadId":"61940","inReplyTo":"ca78d3dc7c0270b434ee4ca4ef618212c7dc1d5b.1723712608.git.ps@pks.im","subject":"Re: [PATCH v2 5/7] builtin/gc: add a `--detach` flag","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-08-15T19:11:40Z","receivedAt":"2024-08-15T19:11:43Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n> +test_expect_success '--detach overrides gc.autoDetach=false' '\n> +\ttest_when_finished \"rm -rf repo\" &&\n> +\tgit init repo &&\n> +\t(\n> +\t\tcd repo &&\n> +\n> +\t\t# Prepare the repository such that git-gc(1) ends up repacking.\n> +\t\ttest_commit \"$(test_oid blob17_1)\" &&\n> +\t\ttest_commit \"$(test_oid blob17_2)\" &&\n> +\t\tgit config gc.autodetach false &&\n> +\t\tgit config gc.auto 2 &&\n> +\n> +\t\tcat >expect <<-EOF &&\n> +\t\tAuto packing the repository in background for optimum performance.\n> +\t\tSee \"git help gc\" for manual housekeeping.\n> +\t\tEOF\n> +\t\tGIT_PROGRESS_DELAY=0 git gc --auto --detach 2>actual &&\n> +\t\ttest_cmp expect actual\n> +\t)\n> +'\n\nIf the gc/maintenance is going to background itself, it is possible\nthat it still is running, possibly with files under repo/.git/ open\nand the process running in repo directory, when the test_when_finished\nclean-up trap goes in effect?\n\nI am wondering where this comes from:\n\n https://github.com/git/git/actions/runs/10408467351/job/28825980833#step:6:2000\n\nwhere \"rm -rf repo\" dies with an unusual\n\n rm: can't remove 'repo/.git': Directory not empty\n\nand my theory is that after \"rm -rf\" _thinks_ it removed everything\nunderneath, before it attempts to rmdir(\"repo/.git\"), the repack\nprocess in the background has created a new pack, and \"rm -rf\" does\nnot go back and try to create such a new cruft.\n\nThe most robust way to work around such a \"race\" is to wait for the\nbackgrounded process before cleaning up, or after seeing that the\nmessage we use as a signal that the \"gc\" has backgrounded itself,\nkill that backgrounded process before exiting the test and causing\nthe clean-up to trigger.\n\n"},{"id":"501075","messageId":"xmqqttflqv27.fsf@gitster.g","threadId":"61940","inReplyTo":"xmqq34n5txcj.fsf@gitster.g","subject":"Re: [PATCH v2 5/7] builtin/gc: add a `--detach` flag","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-08-15T22:29:20Z","receivedAt":"2024-08-15T22:29:23Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n> Patrick Steinhardt writes:\n>\n>> +test_expect_success '--detach overrides gc.autoDetach=false' '\n>> +\ttest_when_finished \"rm -rf repo\" &&\n>> +\tgit init repo &&\n>> +\t(\n>> +\t\tcd repo &&\n>> +\n>> +\t\t# Prepare the repository such that git-gc(1) ends up repacking.\n>> +\t\ttest_commit \"$(test_oid blob17_1)\" &&\n>> +\t\ttest_commit \"$(test_oid blob17_2)\" &&\n>> +\t\tgit config gc.autodetach false &&\n>> +\t\tgit config gc.auto 2 &&\n>> +\n>> +\t\tcat >expect <<-EOF &&\n>> +\t\tAuto packing the repository in background for optimum performance.\n>> +\t\tSee \"git help gc\" for manual housekeeping.\n>> +\t\tEOF\n>> +\t\tGIT_PROGRESS_DELAY=0 git gc --auto --detach 2>actual &&\n>> +\t\ttest_cmp expect actual\n>> +\t)\n>> +'\n>\n> If the gc/maintenance is going to background itself, it is possible\n> that it still is running, possibly with files under repo/.git/ open\n> and the process running in repo directory, when the test_when_finished\n> clean-up trap goes in effect?\n>\n> I am wondering where this comes from:\n>\n> https://github.com/git/git/actions/runs/10408467351/job/28825980833#step:6:2000\n>\n> where \"rm -rf repo\" dies with an unusual\n>\n> rm: can't remove 'repo/.git': Directory not empty\n>\n> and my theory is that after \"rm -rf\" _thinks_ it removed everything\n> underneath, before it attempts to rmdir(\"repo/.git\"), the repack\n> process in the background has created a new pack, and \"rm -rf\" does\n> not go back and try to create such a new cruft.\n>\n> The most robust way to work around such a \"race\" is to wait for the\n> backgrounded process before cleaning up, or after seeing that the\n> message we use as a signal that the \"gc\" has backgrounded itself,\n> kill that backgrounded process before exiting the test and causing\n> the clean-up to trigger.\n\nThere already is a clue left by those who worked on this test the\nlast time at the end of the script. It says:\n\n # DO NOT leave a detached auto gc process running near the end of the\n # test script: it can run long enough in the background to racily\n # interfere with the cleanup in 'test_done'.\n\nimmediately before \"test_done\".\n\nIn the meantime, I am wondering something simple and silly like the\nattached is sufficient. The idea is that we expect the \"oops we\ncouldn't clean\" code not to trigger most of the time, but if it\ndoes, we just wait (with back off) a bit and retry.\n\n\n t/t6500-gc.sh | 18 ++++++++++++++++--\n 1 file changed, 16 insertions(+), 2 deletions(-)\n\ndiff --git c/t/t6500-gc.sh w/t/t6500-gc.sh\nindex 737c99e0f8..4a991e087a 100755\n--- c/t/t6500-gc.sh\n+++ w/t/t6500-gc.sh\n@@ -396,8 +396,22 @@ test_expect_success 'background auto gc respects lock for all operations' '\n \ttest_cmp expect actual\n '\n \n+wait_to_clean () {\n+\tcount=10 sleep=1\n+\tuntil rm -rf \"$1\" && ! test -d \"$1\"\n+\tdo\n+\t\tif test $count = 0\n+\t\tthen\n+\t\t\treturn 1\n+\t\tfi\n+\t\tcount=$(( count - 1 ))\n+\t\tsleep=$(( sleep + sleep ))\n+\t\tsleep $sleep\n+\tdone\n+}\n+\n test_expect_success '--detach overrides gc.autoDetach=false' '\n-\ttest_when_finished \"rm -rf repo\" &&\n+\ttest_when_finished \"wait_to_clean repo\" &&\n \tgit init repo &&\n \t(\n \t\tcd repo &&\n@@ -418,7 +432,7 @@ test_expect_success '--detach overrides gc.autoDetach=false' '\n '\n \n test_expect_success '--no-detach overrides gc.autoDetach=true' '\n-\ttest_when_finished \"rm -rf repo\" &&\n+\ttest_when_finished \"wait_to_clean repo\" &&\n \tgit init repo &&\n \t(\n \t\tcd repo &&\n\n"},{"id":"501099","messageId":"Zr8IkziW-TfotARy@tanuki","threadId":"61940","inReplyTo":"de298972-a359-46dd-b808-4edaa8701d40@gmail.com","subject":"Re: [PATCH 0/7] builtin/maintenance: fix auto-detach with non-standard tasks","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-08-16T08:06:43Z","receivedAt":"2024-08-16T08:06:48Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Thu, Aug 15, 2024 at 10:04:10AM -0400, Derrick Stolee wrote:\n> On 8/13/24 3:17 AM, Patrick Steinhardt wrote:\n> \n> > I recently configured git-maintenance(1) to not use git-gc(1) anymore,\n> > but instead to use git-multi-pack-index(1). I quickly noticed that the\n> > behaviour here is somewhat broken because instead of auto-detaching when\n> > `git maintenance run --auto` executes, we wait for the process to run to\n> > completion.\n> > \n> > The root cause is that git-maintenance(1), probably by accident,\n> > continues to rely on the auto-detaching mechanism in git-gc(1). So\n> > instead of having git-maintenance(1) detach, it is git-gc(1) that\n> > detaches and thus causes git-maintenance(1) to exit early. That of\n> > course falls flat once any maintenance task other than git-gc(1)\n> > executes, because these won't detach.\n> > \n> > Despite being a usability issue, this may also cause git-gc(1) to run\n> > concurrently with any other enabled maintenance tasks. This shouldn't\n> > lead to data loss, but it can certainly lead to processes stomping on\n> > each others feet.\n> > \n> > This patch series fixes this by wiring up new `--detach` flags for both\n> > git-gc(1) and git-maintenance(1). Like this, git-maintenance(1) now\n> > knows to execute `git gc --auto --no-detach`, while our auto-maintenance\n> > will execute `git mainteance run --auto --detach`.\n> \n> Thank you for noticing this behavior, which is essentially an unintended\n> regression from when the maintenance command was first introduced. It\n> worked for most users because of the accidental detachment of the GC\n> task, but now users can correctly customize their automatic maintenance\n> to run in the background.\n> \n> This was my oversight, as I was focused on scheduled maintenance as\n> being the primary way that users would customize their maintenance tasks.\n> Thank you for unifying the concepts.\n> \n> I sprinkled in commentary, and most of it was just things I noticed\n> while reading the series in order but then later patches or a careful\n> read made my comments non-actionable.\n> \n> This v1 looks good to me.\n\nThanks for your thorough review!\n\nPatrick\n"},{"id":"501100","messageId":"Zr8ImP6knHO6-ZBb@tanuki","threadId":"61940","inReplyTo":"xmqqttflqv27.fsf@gitster.g","subject":"Re: [PATCH v2 5/7] builtin/gc: add a `--detach` flag","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-08-16T08:06:48Z","receivedAt":"2024-08-16T08:06:53Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Thu, Aug 15, 2024 at 03:29:20PM -0700, Junio C Hamano wrote:\n> Junio C Hamano writes:\n> \n> > Patrick Steinhardt writes:\n> >\n> >> +test_expect_success '--detach overrides gc.autoDetach=false' '\n> >> +\ttest_when_finished \"rm -rf repo\" &&\n> >> +\tgit init repo &&\n> >> +\t(\n> >> +\t\tcd repo &&\n> >> +\n> >> +\t\t# Prepare the repository such that git-gc(1) ends up repacking.\n> >> +\t\ttest_commit \"$(test_oid blob17_1)\" &&\n> >> +\t\ttest_commit \"$(test_oid blob17_2)\" &&\n> >> +\t\tgit config gc.autodetach false &&\n> >> +\t\tgit config gc.auto 2 &&\n> >> +\n> >> +\t\tcat >expect <<-EOF &&\n> >> +\t\tAuto packing the repository in background for optimum performance.\n> >> +\t\tSee \"git help gc\" for manual housekeeping.\n> >> +\t\tEOF\n> >> +\t\tGIT_PROGRESS_DELAY=0 git gc --auto --detach 2>actual &&\n> >> +\t\ttest_cmp expect actual\n> >> +\t)\n> >> +'\n> >\n> > If the gc/maintenance is going to background itself, it is possible\n> > that it still is running, possibly with files under repo/.git/ open\n> > and the process running in repo directory, when the test_when_finished\n> > clean-up trap goes in effect?\n> >\n> > I am wondering where this comes from:\n> >\n> > https://github.com/git/git/actions/runs/10408467351/job/28825980833#step:6:2000\n> >\n> > where \"rm -rf repo\" dies with an unusual\n> >\n> > rm: can't remove 'repo/.git': Directory not empty\n> >\n> > and my theory is that after \"rm -rf\" _thinks_ it removed everything\n> > underneath, before it attempts to rmdir(\"repo/.git\"), the repack\n> > process in the background has created a new pack, and \"rm -rf\" does\n> > not go back and try to create such a new cruft.\n> >\n> > The most robust way to work around such a \"race\" is to wait for the\n> > backgrounded process before cleaning up, or after seeing that the\n> > message we use as a signal that the \"gc\" has backgrounded itself,\n> > kill that backgrounded process before exiting the test and causing\n> > the clean-up to trigger.\n> \n> There already is a clue left by those who worked on this test the\n> last time at the end of the script. It says:\n> \n> # DO NOT leave a detached auto gc process running near the end of the\n> # test script: it can run long enough in the background to racily\n> # interfere with the cleanup in 'test_done'.\n> \n> immediately before \"test_done\".\n> \n> In the meantime, I am wondering something simple and silly like the\n> attached is sufficient. The idea is that we expect the \"oops we\n> couldn't clean\" code not to trigger most of the time, but if it\n> does, we just wait (with back off) a bit and retry.\n\nAh, indeed, that is a problem. We already have a better tool to fix this\nwith `run_and_wait_for_auto_gc()`. It creates a separate file descriptor\nand waits for it to close via some shell trickery, which will only\nhappen once the child process of git-gc(1) has exited.\n\nWill fix, thanks!\n\nPatrick\n"},{"id":"501101","messageId":"Zr8InHskJNixuSWW@tanuki","threadId":"61940","inReplyTo":"xmqq4j7lvk5z.fsf@gitster.g","subject":"Re: [PATCH v2 7/7] run-command: fix detaching when running auto maintenance","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-08-16T08:06:52Z","receivedAt":"2024-08-16T08:06:56Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Thu, Aug 15, 2024 at 09:13:28AM -0700, Junio C Hamano wrote:\n> Patrick Steinhardt writes:\n> \n> > diff --git a/run-command.c b/run-command.c\n> > index 45ba544932..94f2f3079f 100644\n> > --- a/run-command.c\n> > +++ b/run-command.c\n> > @@ -1808,16 +1808,26 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)\n> > \n> > int prepare_auto_maintenance(int quiet, struct child_process *maint)\n> > {\n> > -\tint enabled;\n> > +\tint enabled, auto_detach;\n> > \n> > \tif (!git_config_get_bool(\"maintenance.auto\", &enabled) &&\n> > \t !enabled)\n> > \t\treturn 0;\n> > \n> > +\t/*\n> > +\t * When `maintenance.autoDetach` isn't set, then we fall back to\n> > +\t * honoring `gc.autoDetach`. This is somewhat weird, but required to\n> > +\t * retain behaviour from when we used to run git-gc(1) here.\n> > +\t */\n> > +\tif (git_config_get_bool(\"maintenance.autodetach\", &auto_detach) &&\n> > +\t git_config_get_bool(\"gc.autodetach\", &auto_detach))\n> > +\t\tauto_detach = 1;\n> \n> I think this needs somehow documented. Something like this,\n> perhaps?\n\nIndeed, I totally forgot doing that.\n\n> --- c/Documentation/config/maintenance.txt\n> +++ w/Documentation/config/maintenance.txt\n> @@ -3,6 +3,15 @@ maintenance.auto::\n> \t`git maintenance run --auto` after doing their normal work. Defaults\n> \tto true.\n> \n> +maintenance.autoDetach::\n> +\tTasks that are run via `git maintenance run --auto` by\n> +\tdefault runs in the background, if the system supports it.\n> +\tSetting this configuration variable to `true` explicitly\n> +\tasks them to run in the background, and setting it to\n> +\t`false` forces them to run in the foreground. If this\n> +\tvariable is not set, `gc.autoDetach` works as a fallback\n> +\tvariable and behaves the same way.\n\nThis isn't entirely true. `git maintenance run --auto` will not\nbackground, because that'd change preexisting behaviour. It also would\nnot make a lot of sense, because here the `--auto` trigger tells the\ncommand to do maintenance as-needed. Coupling that with whether or not\nto detach was a misdesign of git-gc(1), I think. What it does control is\nwhether we detach or not when automatically executing maintenance via\n`prepare_auto_maintenance()`.\n\nAnyway, my fault for not documenting it, not yours for getting it\nslightly wrong.\n\nPatrick\n"},{"id":"501122","messageId":"cover.1723804990.git.ps@pks.im","threadId":"61940","inReplyTo":"cover.1723533091.git.ps@pks.im","subject":"[PATCH v3 0/7] builtin/maintenance: fix auto-detach with non-standard tasks","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-08-16T10:44:54Z","receivedAt":"2024-08-16T10:44:59Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Hi,\n\nthis is the second version of my patch series that fixes how\ngit-maintenance(1) detaches: instead of letting its child process\ngit-gc(1) detach, we now optionally ask git-maintenance(1) itself to\ndetach when running via our auto maintenance mechanism. This fixes\nbehaviour of git-maintenance(1) when configured to run non-standard\ntasks like the \"incremental\" task.\n\nChanges compared to v2:\n\n - Fix leaking git-gc(1) process in t6500.\n\n - Add missing documentation for `maintenance.autoDetach`.\n\nThanks!\n\nPatrick\n\nPatrick Steinhardt (7):\n config: fix constness of out parameter for `git_config_get_expiry()`\n builtin/gc: refactor to read config into structure\n builtin/gc: fix leaking config values\n builtin/gc: stop processing log file on signal\n builtin/gc: add a `--detach` flag\n builtin/maintenance: add a `--detach` flag\n run-command: fix detaching when running auto maintenance\n\n Documentation/config/gc.txt | 3 +-\n Documentation/config/maintenance.txt | 11 +\n Documentation/git-gc.txt | 5 +-\n builtin/gc.c | 384 +++++++++++++++++----------\n config.c | 4 +-\n config.h | 2 +-\n read-cache.c | 12 +-\n run-command.c | 12 +-\n t/t5304-prune.sh | 1 +\n t/t5616-partial-clone.sh | 6 +-\n t/t6500-gc.sh | 45 +++-\n t/t7900-maintenance.sh | 82 +++++-\n 12 files changed, 396 insertions(+), 171 deletions(-)\n\nRange-diff against v2:\n1: 040453f27f = 1: 040453f27f config: fix constness of out parameter for `git_config_get_expiry()`\n2: ff6aa9d7ba = 2: ff6aa9d7ba builtin/gc: refactor to read config into structure\n3: 310e361371 = 3: 310e361371 builtin/gc: fix leaking config values\n4: 812c61c9b6 = 4: 812c61c9b6 builtin/gc: stop processing log file on signal\n5: ca78d3dc7c ! 5: b934b23889 builtin/gc: add a `--detach` flag\n @@ builtin/gc.c: static int maintenance_run(int argc, const char **argv, const char\n \tfor (i = 0; i < TASK__COUNT; i++)\n \n ## t/t6500-gc.sh ##\n +@@ t/t6500-gc.sh: test_expect_success 'gc.maxCruftSize sets appropriate repack options' '\n + \ttest_subcommand $cruft_max_size_opts --max-cruft-size=3145728 &1)\n ++\tdoesnt_matter=$(git gc \"$@\" 9>&1)\n + }\n + \n + test_expect_success 'background auto gc does not run if gc.log is present and recent but does if it is old' '\n +@@ t/t6500-gc.sh: test_expect_success 'background auto gc does not run if gc.log is present and re\n + \ttest-tool chmtime =-345600 .git/gc.log &&\n + \tgit gc --auto &&\n + \ttest_config gc.logexpiry 2.days &&\n +-\trun_and_wait_for_auto_gc &&\n ++\trun_and_wait_for_gc --auto &&\n + \tls .git/objects/pack/pack-*.pack >packs &&\n + \ttest_line_count = 1 packs\n + '\n @@ t/t6500-gc.sh: test_expect_success 'background auto gc respects lock for all operations' '\n + \tprintf \"%d %s\" \"$shell_pid\" \"$hostname\" >.git/gc.pid &&\n + \n + \t# our gc should exit zero without doing anything\n +-\trun_and_wait_for_auto_gc &&\n ++\trun_and_wait_for_gc --auto &&\n + \t(ls -1 .git/refs/heads .git/reftable >actual || true) &&\n \ttest_cmp expect actual\n '\n \n @@ t/t6500-gc.sh: test_expect_success 'background auto gc respects lock for all ope\n +\t\tgit config gc.autodetach false &&\n +\t\tgit config gc.auto 2 &&\n +\n -+\t\tcat >expect <<-EOF &&\n -+\t\tAuto packing the repository in background for optimum performance.\n -+\t\tSee \"git help gc\" for manual housekeeping.\n -+\t\tEOF\n -+\t\tGIT_PROGRESS_DELAY=0 git gc --auto --detach 2>actual &&\n -+\t\ttest_cmp expect actual\n ++\t\t# Note that we cannot use `test_cmp` here to compare stderr\n ++\t\t# because it may contain output from `set -x`.\n ++\t\trun_and_wait_for_gc --auto --detach 2>actual &&\n ++\t\ttest_grep \"Auto packing the repository in background for optimum performance.\" actual\n +\t)\n +'\n +\n6: 06dbb73425 = 6: 347d0a2002 builtin/maintenance: add a `--detach` flag\n7: 6bc170ff05 ! 7: 9befef7c1f run-command: fix detaching when running auto maintenance\n @@ Commit message\n \n Signed-off-by: Patrick Steinhardt \n \n + ## Documentation/config/gc.txt ##\n +@@ Documentation/config/gc.txt: use, it'll affect how the auto pack limit works.\n + \n + gc.autoDetach::\n + \tMake `git gc --auto` return immediately and run in the background\n +-\tif the system supports it. Default is true.\n ++\tif the system supports it. Default is true. This config variable acts\n ++\tas a fallback in case `maintenance.autoDetach` is not set.\n + \n + gc.bigPackThreshold::\n + \tIf non-zero, all non-cruft packs larger than this limit are kept\n +\n + ## Documentation/config/maintenance.txt ##\n +@@ Documentation/config/maintenance.txt: maintenance.auto::\n + \t`git maintenance run --auto` after doing their normal work. Defaults\n + \tto true.\n + \n ++maintenance.autoDetach::\n ++\tMany Git commands trigger automatic maintenance after they have\n ++\twritten data into the repository. This boolean config option\n ++\tcontrols whether this automatic maintenance shall happen in the\n ++\tforeground or whether the maintenance process shall detach and\n ++\tcontinue to run in the background.\n +++\n ++If unset, the value of `gc.autoDetach` is used as a fallback. Defaults\n ++to true if both are unset, meaning that the maintenance process will\n ++detach.\n ++\n + maintenance.strategy::\n + \tThis string config option provides a way to specify one of a few\n + \trecommended schedules for background maintenance. This only affects\n +\n ## builtin/gc.c ##\n @@ builtin/gc.c: static int maintenance_task_gc(struct maintenance_run_opts *opts,\n \t\tstrvec_push(&child.args, \"--quiet\");\n-- \n2.46.0.46.g406f326d27.dirty\n\n"},{"id":"501123","messageId":"040453f27f0b2755bb359fe07a703ce66e071ab6.1723804990.git.ps@pks.im","threadId":"61940","inReplyTo":"cover.1723804990.git.ps@pks.im","subject":"[PATCH v3 1/7] config: fix constness of out parameter for `git_config_get_expiry()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-08-16T10:44:57Z","receivedAt":"2024-08-16T10:45:01Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The type of the out parameter of `git_config_get_expiry()` is a pointer\nto a constant string, which creates the impression that ownership of the\nreturned data wasn't transferred to the caller. This isn't true though\nand thus quite misleading.\n\nAdapt the parameter to be of type `char **` and adjust callers\naccordingly. While at it, refactor `get_shared_index_expire_date()` to\ndrop the static `shared_index_expire` variable. It is only used in that\nfunction, and furthermore we would only hit the code where we parse the\nexpiry date a single time because we already use a static `prepared`\nvariable to track whether we did parse it.\n\nSigned-off-by: Patrick Steinhardt \n---\n builtin/gc.c | 6 +++---\n config.c | 4 ++--\n config.h | 2 +-\n read-cache.c | 12 +++++++++---\n 4 files changed, 15 insertions(+), 9 deletions(-)\n\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 72bac2554f..e7406bf667 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -167,9 +167,9 @@ static void gc_config(void)\n \tgit_config_get_bool(\"gc.autodetach\", &detach_auto);\n \tgit_config_get_bool(\"gc.cruftpacks\", &cruft_packs);\n \tgit_config_get_ulong(\"gc.maxcruftsize\", &max_cruft_size);\n-\tgit_config_get_expiry(\"gc.pruneexpire\", &prune_expire);\n-\tgit_config_get_expiry(\"gc.worktreepruneexpire\", &prune_worktrees_expire);\n-\tgit_config_get_expiry(\"gc.logexpiry\", &gc_log_expire);\n+\tgit_config_get_expiry(\"gc.pruneexpire\", (char **) &prune_expire);\n+\tgit_config_get_expiry(\"gc.worktreepruneexpire\", (char **) &prune_worktrees_expire);\n+\tgit_config_get_expiry(\"gc.logexpiry\", (char **) &gc_log_expire);\n \n \tgit_config_get_ulong(\"gc.bigpackthreshold\", &big_pack_threshold);\n \tgit_config_get_ulong(\"pack.deltacachesize\", &max_delta_cache_size);\ndiff --git a/config.c b/config.c\nindex 6421894614..dfa4df1417 100644\n--- a/config.c\n+++ b/config.c\n@@ -2766,9 +2766,9 @@ int git_config_get_pathname(const char *key, char **dest)\n \treturn repo_config_get_pathname(the_repository, key, dest);\n }\n \n-int git_config_get_expiry(const char *key, const char **output)\n+int git_config_get_expiry(const char *key, char **output)\n {\n-\tint ret = git_config_get_string(key, (char **)output);\n+\tint ret = git_config_get_string(key, output);\n \tif (ret)\n \t\treturn ret;\n \tif (strcmp(*output, \"now\")) {\ndiff --git a/config.h b/config.h\nindex 54b47dec9e..4801391c32 100644\n--- a/config.h\n+++ b/config.h\n@@ -701,7 +701,7 @@ int git_config_get_split_index(void);\n int git_config_get_max_percent_split_change(void);\n \n /* This dies if the configured or default date is in the future */\n-int git_config_get_expiry(const char *key, const char **output);\n+int git_config_get_expiry(const char *key, char **output);\n \n /* parse either \"this many days\" integer, or \"5.days.ago\" approxidate */\n int git_config_get_expiry_in_days(const char *key, timestamp_t *, timestamp_t now);\ndiff --git a/read-cache.c b/read-cache.c\nindex 48bf24f87c..7f393ee687 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -3176,18 +3176,24 @@ static int write_split_index(struct index_state *istate,\n \treturn ret;\n }\n \n-static const char *shared_index_expire = \"2.weeks.ago\";\n-\n static unsigned long get_shared_index_expire_date(void)\n {\n \tstatic unsigned long shared_index_expire_date;\n \tstatic int shared_index_expire_date_prepared;\n \n \tif (!shared_index_expire_date_prepared) {\n+\t\tconst char *shared_index_expire = \"2.weeks.ago\";\n+\t\tchar *value = NULL;\n+\n \t\tgit_config_get_expiry(\"splitindex.sharedindexexpire\",\n-\t\t\t\t &shared_index_expire);\n+\t\t\t\t &value);\n+\t\tif (value)\n+\t\t\tshared_index_expire = value;\n+\n \t\tshared_index_expire_date = approxidate(shared_index_expire);\n \t\tshared_index_expire_date_prepared = 1;\n+\n+\t\tfree(value);\n \t}\n \n \treturn shared_index_expire_date;\n-- \n2.46.0.46.g406f326d27.dirty\n\n"},{"id":"501124","messageId":"ff6aa9d7ba0bd67d3682ecb44c024a6d59a7decd.1723804990.git.ps@pks.im","threadId":"61940","inReplyTo":"cover.1723804990.git.ps@pks.im","subject":"[PATCH v3 2/7] builtin/gc: refactor to read config into structure","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-08-16T10:45:00Z","receivedAt":"2024-08-16T10:45:04Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The git-gc(1) command knows to read a bunch of config keys to tweak its\nown behaviour. The values are parsed into global variables, which makes\nit hard to correctly manage the lifecycle of values that may require a\nmemory allocation.\n\nRefactor the code to use a `struct gc_config` that gets populated and\npassed around. For one, this makes previously-implicit dependencies on\nthese config values clear. Second, it will allow us to properly manage\nthe lifecycle in the next commit.\n\nSigned-off-by: Patrick Steinhardt \n---\n builtin/gc.c | 255 +++++++++++++++++++++++++++++----------------------\n 1 file changed, 143 insertions(+), 112 deletions(-)\n\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex e7406bf667..eee7401647 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -49,23 +49,7 @@ static const char * const builtin_gc_usage[] = {\n \tNULL\n };\n \n-static int pack_refs = 1;\n-static int prune_reflogs = 1;\n-static int cruft_packs = 1;\n-static unsigned long max_cruft_size;\n-static int aggressive_depth = 50;\n-static int aggressive_window = 250;\n-static int gc_auto_threshold = 6700;\n-static int gc_auto_pack_limit = 50;\n-static int detach_auto = 1;\n static timestamp_t gc_log_expire_time;\n-static const char *gc_log_expire = \"1.day.ago\";\n-static const char *prune_expire = \"2.weeks.ago\";\n-static const char *prune_worktrees_expire = \"3.months.ago\";\n-static char *repack_filter;\n-static char *repack_filter_to;\n-static unsigned long big_pack_threshold;\n-static unsigned long max_delta_cache_size = DEFAULT_DELTA_CACHE_SIZE;\n \n static struct strvec reflog = STRVEC_INIT;\n static struct strvec repack = STRVEC_INIT;\n@@ -145,37 +129,71 @@ static int gc_config_is_timestamp_never(const char *var)\n \treturn 0;\n }\n \n-static void gc_config(void)\n+struct gc_config {\n+\tint pack_refs;\n+\tint prune_reflogs;\n+\tint cruft_packs;\n+\tunsigned long max_cruft_size;\n+\tint aggressive_depth;\n+\tint aggressive_window;\n+\tint gc_auto_threshold;\n+\tint gc_auto_pack_limit;\n+\tint detach_auto;\n+\tconst char *gc_log_expire;\n+\tconst char *prune_expire;\n+\tconst char *prune_worktrees_expire;\n+\tchar *repack_filter;\n+\tchar *repack_filter_to;\n+\tunsigned long big_pack_threshold;\n+\tunsigned long max_delta_cache_size;\n+};\n+\n+#define GC_CONFIG_INIT { \\\n+\t.pack_refs = 1, \\\n+\t.prune_reflogs = 1, \\\n+\t.cruft_packs = 1, \\\n+\t.aggressive_depth = 50, \\\n+\t.aggressive_window = 250, \\\n+\t.gc_auto_threshold = 6700, \\\n+\t.gc_auto_pack_limit = 50, \\\n+\t.detach_auto = 1, \\\n+\t.gc_log_expire = \"1.day.ago\", \\\n+\t.prune_expire = \"2.weeks.ago\", \\\n+\t.prune_worktrees_expire = \"3.months.ago\", \\\n+\t.max_delta_cache_size = DEFAULT_DELTA_CACHE_SIZE, \\\n+}\n+\n+static void gc_config(struct gc_config *cfg)\n {\n \tconst char *value;\n \n \tif (!git_config_get_value(\"gc.packrefs\", &value)) {\n \t\tif (value && !strcmp(value, \"notbare\"))\n-\t\t\tpack_refs = -1;\n+\t\t\tcfg->pack_refs = -1;\n \t\telse\n-\t\t\tpack_refs = git_config_bool(\"gc.packrefs\", value);\n+\t\t\tcfg->pack_refs = git_config_bool(\"gc.packrefs\", value);\n \t}\n \n \tif (gc_config_is_timestamp_never(\"gc.reflogexpire\") &&\n \t gc_config_is_timestamp_never(\"gc.reflogexpireunreachable\"))\n-\t\tprune_reflogs = 0;\n+\t\tcfg->prune_reflogs = 0;\n \n-\tgit_config_get_int(\"gc.aggressivewindow\", &aggressive_window);\n-\tgit_config_get_int(\"gc.aggressivedepth\", &aggressive_depth);\n-\tgit_config_get_int(\"gc.auto\", &gc_auto_threshold);\n-\tgit_config_get_int(\"gc.autopacklimit\", &gc_auto_pack_limit);\n-\tgit_config_get_bool(\"gc.autodetach\", &detach_auto);\n-\tgit_config_get_bool(\"gc.cruftpacks\", &cruft_packs);\n-\tgit_config_get_ulong(\"gc.maxcruftsize\", &max_cruft_size);\n-\tgit_config_get_expiry(\"gc.pruneexpire\", (char **) &prune_expire);\n-\tgit_config_get_expiry(\"gc.worktreepruneexpire\", (char **) &prune_worktrees_expire);\n-\tgit_config_get_expiry(\"gc.logexpiry\", (char **) &gc_log_expire);\n+\tgit_config_get_int(\"gc.aggressivewindow\", &cfg->aggressive_window);\n+\tgit_config_get_int(\"gc.aggressivedepth\", &cfg->aggressive_depth);\n+\tgit_config_get_int(\"gc.auto\", &cfg->gc_auto_threshold);\n+\tgit_config_get_int(\"gc.autopacklimit\", &cfg->gc_auto_pack_limit);\n+\tgit_config_get_bool(\"gc.autodetach\", &cfg->detach_auto);\n+\tgit_config_get_bool(\"gc.cruftpacks\", &cfg->cruft_packs);\n+\tgit_config_get_ulong(\"gc.maxcruftsize\", &cfg->max_cruft_size);\n+\tgit_config_get_expiry(\"gc.pruneexpire\", (char **) &cfg->prune_expire);\n+\tgit_config_get_expiry(\"gc.worktreepruneexpire\", (char **) &cfg->prune_worktrees_expire);\n+\tgit_config_get_expiry(\"gc.logexpiry\", (char **) &cfg->gc_log_expire);\n \n-\tgit_config_get_ulong(\"gc.bigpackthreshold\", &big_pack_threshold);\n-\tgit_config_get_ulong(\"pack.deltacachesize\", &max_delta_cache_size);\n+\tgit_config_get_ulong(\"gc.bigpackthreshold\", &cfg->big_pack_threshold);\n+\tgit_config_get_ulong(\"pack.deltacachesize\", &cfg->max_delta_cache_size);\n \n-\tgit_config_get_string(\"gc.repackfilter\", &repack_filter);\n-\tgit_config_get_string(\"gc.repackfilterto\", &repack_filter_to);\n+\tgit_config_get_string(\"gc.repackfilter\", &cfg->repack_filter);\n+\tgit_config_get_string(\"gc.repackfilterto\", &cfg->repack_filter_to);\n \n \tgit_config(git_default_config, NULL);\n }\n@@ -206,7 +224,7 @@ struct maintenance_run_opts {\n \tenum schedule_priority schedule;\n };\n \n-static int pack_refs_condition(void)\n+static int pack_refs_condition(UNUSED struct gc_config *cfg)\n {\n \t/*\n \t * The auto-repacking logic for refs is handled by the ref backends and\n@@ -216,7 +234,8 @@ static int pack_refs_condition(void)\n \treturn 1;\n }\n \n-static int maintenance_task_pack_refs(MAYBE_UNUSED struct maintenance_run_opts *opts)\n+static int maintenance_task_pack_refs(MAYBE_UNUSED struct maintenance_run_opts *opts,\n+\t\t\t\t UNUSED struct gc_config *cfg)\n {\n \tstruct child_process cmd = CHILD_PROCESS_INIT;\n \n@@ -228,7 +247,7 @@ static int maintenance_task_pack_refs(MAYBE_UNUSED struct maintenance_run_opts *\n \treturn run_command(&cmd);\n }\n \n-static int too_many_loose_objects(void)\n+static int too_many_loose_objects(struct gc_config *cfg)\n {\n \t/*\n \t * Quickly check if a \"gc\" is needed, by estimating how\n@@ -247,7 +266,7 @@ static int too_many_loose_objects(void)\n \tif (!dir)\n \t\treturn 0;\n \n-\tauto_threshold = DIV_ROUND_UP(gc_auto_threshold, 256);\n+\tauto_threshold = DIV_ROUND_UP(cfg->gc_auto_threshold, 256);\n \twhile ((ent = readdir(dir)) != NULL) {\n \t\tif (strspn(ent->d_name, \"0123456789abcdef\") != hexsz_loose ||\n \t\t ent->d_name[hexsz_loose] != '\\0')\n@@ -283,12 +302,12 @@ static struct packed_git *find_base_packs(struct string_list *packs,\n \treturn base;\n }\n \n-static int too_many_packs(void)\n+static int too_many_packs(struct gc_config *cfg)\n {\n \tstruct packed_git *p;\n \tint cnt;\n \n-\tif (gc_auto_pack_limit <= 0)\n+\tif (cfg->gc_auto_pack_limit <= 0)\n \t\treturn 0;\n \n \tfor (cnt = 0, p = get_all_packs(the_repository); p; p = p->next) {\n@@ -302,7 +321,7 @@ static int too_many_packs(void)\n \t\t */\n \t\tcnt++;\n \t}\n-\treturn gc_auto_pack_limit < cnt;\n+\treturn cfg->gc_auto_pack_limit < cnt;\n }\n \n static uint64_t total_ram(void)\n@@ -336,7 +355,8 @@ static uint64_t total_ram(void)\n \treturn 0;\n }\n \n-static uint64_t estimate_repack_memory(struct packed_git *pack)\n+static uint64_t estimate_repack_memory(struct gc_config *cfg,\n+\t\t\t\t struct packed_git *pack)\n {\n \tunsigned long nr_objects = repo_approximate_object_count(the_repository);\n \tsize_t os_cache, heap;\n@@ -373,7 +393,7 @@ static uint64_t estimate_repack_memory(struct packed_git *pack)\n \t */\n \theap += delta_base_cache_limit;\n \t/* and of course pack-objects has its own delta cache */\n-\theap += max_delta_cache_size;\n+\theap += cfg->max_delta_cache_size;\n \n \treturn os_cache + heap;\n }\n@@ -384,30 +404,31 @@ static int keep_one_pack(struct string_list_item *item, void *data UNUSED)\n \treturn 0;\n }\n \n-static void add_repack_all_option(struct string_list *keep_pack)\n+static void add_repack_all_option(struct gc_config *cfg,\n+\t\t\t\t struct string_list *keep_pack)\n {\n-\tif (prune_expire && !strcmp(prune_expire, \"now\"))\n+\tif (cfg->prune_expire && !strcmp(cfg->prune_expire, \"now\"))\n \t\tstrvec_push(&repack, \"-a\");\n-\telse if (cruft_packs) {\n+\telse if (cfg->cruft_packs) {\n \t\tstrvec_push(&repack, \"--cruft\");\n-\t\tif (prune_expire)\n-\t\t\tstrvec_pushf(&repack, \"--cruft-expiration=%s\", prune_expire);\n-\t\tif (max_cruft_size)\n+\t\tif (cfg->prune_expire)\n+\t\t\tstrvec_pushf(&repack, \"--cruft-expiration=%s\", cfg->prune_expire);\n+\t\tif (cfg->max_cruft_size)\n \t\t\tstrvec_pushf(&repack, \"--max-cruft-size=%lu\",\n-\t\t\t\t max_cruft_size);\n+\t\t\t\t cfg->max_cruft_size);\n \t} else {\n \t\tstrvec_push(&repack, \"-A\");\n-\t\tif (prune_expire)\n-\t\t\tstrvec_pushf(&repack, \"--unpack-unreachable=%s\", prune_expire);\n+\t\tif (cfg->prune_expire)\n+\t\t\tstrvec_pushf(&repack, \"--unpack-unreachable=%s\", cfg->prune_expire);\n \t}\n \n \tif (keep_pack)\n \t\tfor_each_string_list(keep_pack, keep_one_pack, NULL);\n \n-\tif (repack_filter && *repack_filter)\n-\t\tstrvec_pushf(&repack, \"--filter=%s\", repack_filter);\n-\tif (repack_filter_to && *repack_filter_to)\n-\t\tstrvec_pushf(&repack, \"--filter-to=%s\", repack_filter_to);\n+\tif (cfg->repack_filter && *cfg->repack_filter)\n+\t\tstrvec_pushf(&repack, \"--filter=%s\", cfg->repack_filter);\n+\tif (cfg->repack_filter_to && *cfg->repack_filter_to)\n+\t\tstrvec_pushf(&repack, \"--filter-to=%s\", cfg->repack_filter_to);\n }\n \n static void add_repack_incremental_option(void)\n@@ -415,13 +436,13 @@ static void add_repack_incremental_option(void)\n \tstrvec_push(&repack, \"--no-write-bitmap-index\");\n }\n \n-static int need_to_gc(void)\n+static int need_to_gc(struct gc_config *cfg)\n {\n \t/*\n \t * Setting gc.auto to 0 or negative can disable the\n \t * automatic gc.\n \t */\n-\tif (gc_auto_threshold <= 0)\n+\tif (cfg->gc_auto_threshold <= 0)\n \t\treturn 0;\n \n \t/*\n@@ -430,13 +451,13 @@ static int need_to_gc(void)\n \t * we run \"repack -A -d -l\". Otherwise we tell the caller\n \t * there is no need.\n \t */\n-\tif (too_many_packs()) {\n+\tif (too_many_packs(cfg)) {\n \t\tstruct string_list keep_pack = STRING_LIST_INIT_NODUP;\n \n-\t\tif (big_pack_threshold) {\n-\t\t\tfind_base_packs(&keep_pack, big_pack_threshold);\n-\t\t\tif (keep_pack.nr >= gc_auto_pack_limit) {\n-\t\t\t\tbig_pack_threshold = 0;\n+\t\tif (cfg->big_pack_threshold) {\n+\t\t\tfind_base_packs(&keep_pack, cfg->big_pack_threshold);\n+\t\t\tif (keep_pack.nr >= cfg->gc_auto_pack_limit) {\n+\t\t\t\tcfg->big_pack_threshold = 0;\n \t\t\t\tstring_list_clear(&keep_pack, 0);\n \t\t\t\tfind_base_packs(&keep_pack, 0);\n \t\t\t}\n@@ -445,7 +466,7 @@ static int need_to_gc(void)\n \t\t\tuint64_t mem_have, mem_want;\n \n \t\t\tmem_have = total_ram();\n-\t\t\tmem_want = estimate_repack_memory(p);\n+\t\t\tmem_want = estimate_repack_memory(cfg, p);\n \n \t\t\t/*\n \t\t\t * Only allow 1/2 of memory for pack-objects, leave\n@@ -456,9 +477,9 @@ static int need_to_gc(void)\n \t\t\t\tstring_list_clear(&keep_pack, 0);\n \t\t}\n \n-\t\tadd_repack_all_option(&keep_pack);\n+\t\tadd_repack_all_option(cfg, &keep_pack);\n \t\tstring_list_clear(&keep_pack, 0);\n-\t} else if (too_many_loose_objects())\n+\t} else if (too_many_loose_objects(cfg))\n \t\tadd_repack_incremental_option();\n \telse\n \t\treturn 0;\n@@ -585,7 +606,8 @@ static int report_last_gc_error(void)\n \treturn ret;\n }\n \n-static void gc_before_repack(struct maintenance_run_opts *opts)\n+static void gc_before_repack(struct maintenance_run_opts *opts,\n+\t\t\t struct gc_config *cfg)\n {\n \t/*\n \t * We may be called twice, as both the pre- and\n@@ -596,10 +618,10 @@ static void gc_before_repack(struct maintenance_run_opts *opts)\n \tif (done++)\n \t\treturn;\n \n-\tif (pack_refs && maintenance_task_pack_refs(opts))\n+\tif (cfg->pack_refs && maintenance_task_pack_refs(opts, cfg))\n \t\tdie(FAILED_RUN, \"pack-refs\");\n \n-\tif (prune_reflogs) {\n+\tif (cfg->prune_reflogs) {\n \t\tstruct child_process cmd = CHILD_PROCESS_INIT;\n \n \t\tcmd.git_cmd = 1;\n@@ -621,14 +643,15 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \ttimestamp_t dummy;\n \tstruct child_process rerere_cmd = CHILD_PROCESS_INIT;\n \tstruct maintenance_run_opts opts = {0};\n+\tstruct gc_config cfg = GC_CONFIG_INIT;\n \n \tstruct option builtin_gc_options[] = {\n \t\tOPT__QUIET(&quiet, N_(\"suppress progress reporting\")),\n-\t\t{ OPTION_STRING, 0, \"prune\", &prune_expire, N_(\"date\"),\n+\t\t{ OPTION_STRING, 0, \"prune\", &cfg.prune_expire, N_(\"date\"),\n \t\t\tN_(\"prune unreferenced objects\"),\n-\t\t\tPARSE_OPT_OPTARG, NULL, (intptr_t)prune_expire },\n-\t\tOPT_BOOL(0, \"cruft\", &cruft_packs, N_(\"pack unreferenced objects separately\")),\n-\t\tOPT_MAGNITUDE(0, \"max-cruft-size\", &max_cruft_size,\n+\t\t\tPARSE_OPT_OPTARG, NULL, (intptr_t)cfg.prune_expire },\n+\t\tOPT_BOOL(0, \"cruft\", &cfg.cruft_packs, N_(\"pack unreferenced objects separately\")),\n+\t\tOPT_MAGNITUDE(0, \"max-cruft-size\", &cfg.max_cruft_size,\n \t\t\t N_(\"with --cruft, limit the size of new cruft packs\")),\n \t\tOPT_BOOL(0, \"aggressive\", &aggressive, N_(\"be more thorough (increased runtime)\")),\n \t\tOPT_BOOL_F(0, \"auto\", &opts.auto_flag, N_(\"enable auto-gc mode\"),\n@@ -651,27 +674,27 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \tstrvec_pushl(&rerere, \"rerere\", \"gc\", NULL);\n \n \t/* default expiry time, overwritten in gc_config */\n-\tgc_config();\n-\tif (parse_expiry_date(gc_log_expire, &gc_log_expire_time))\n-\t\tdie(_(\"failed to parse gc.logExpiry value %s\"), gc_log_expire);\n+\tgc_config(&cfg);\n+\tif (parse_expiry_date(cfg.gc_log_expire, &gc_log_expire_time))\n+\t\tdie(_(\"failed to parse gc.logExpiry value %s\"), cfg.gc_log_expire);\n \n-\tif (pack_refs < 0)\n-\t\tpack_refs = !is_bare_repository();\n+\tif (cfg.pack_refs < 0)\n+\t\tcfg.pack_refs = !is_bare_repository();\n \n \targc = parse_options(argc, argv, prefix, builtin_gc_options,\n \t\t\t builtin_gc_usage, 0);\n \tif (argc > 0)\n \t\tusage_with_options(builtin_gc_usage, builtin_gc_options);\n \n-\tif (prune_expire && parse_expiry_date(prune_expire, &dummy))\n-\t\tdie(_(\"failed to parse prune expiry value %s\"), prune_expire);\n+\tif (cfg.prune_expire && parse_expiry_date(cfg.prune_expire, &dummy))\n+\t\tdie(_(\"failed to parse prune expiry value %s\"), cfg.prune_expire);\n \n \tif (aggressive) {\n \t\tstrvec_push(&repack, \"-f\");\n-\t\tif (aggressive_depth > 0)\n-\t\t\tstrvec_pushf(&repack, \"--depth=%d\", aggressive_depth);\n-\t\tif (aggressive_window > 0)\n-\t\t\tstrvec_pushf(&repack, \"--window=%d\", aggressive_window);\n+\t\tif (cfg.aggressive_depth > 0)\n+\t\t\tstrvec_pushf(&repack, \"--depth=%d\", cfg.aggressive_depth);\n+\t\tif (cfg.aggressive_window > 0)\n+\t\t\tstrvec_pushf(&repack, \"--window=%d\", cfg.aggressive_window);\n \t}\n \tif (quiet)\n \t\tstrvec_push(&repack, \"-q\");\n@@ -680,16 +703,16 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \t\t/*\n \t\t * Auto-gc should be least intrusive as possible.\n \t\t */\n-\t\tif (!need_to_gc())\n+\t\tif (!need_to_gc(&cfg))\n \t\t\treturn 0;\n \t\tif (!quiet) {\n-\t\t\tif (detach_auto)\n+\t\t\tif (cfg.detach_auto)\n \t\t\t\tfprintf(stderr, _(\"Auto packing the repository in background for optimum performance.\\n\"));\n \t\t\telse\n \t\t\t\tfprintf(stderr, _(\"Auto packing the repository for optimum performance.\\n\"));\n \t\t\tfprintf(stderr, _(\"See \\\"git help gc\\\" for manual housekeeping.\\n\"));\n \t\t}\n-\t\tif (detach_auto) {\n+\t\tif (cfg.detach_auto) {\n \t\t\tint ret = report_last_gc_error();\n \n \t\t\tif (ret == 1)\n@@ -701,7 +724,7 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \n \t\t\tif (lock_repo_for_gc(force, &pid))\n \t\t\t\treturn 0;\n-\t\t\tgc_before_repack(&opts); /* dies on failure */\n+\t\t\tgc_before_repack(&opts, &cfg); /* dies on failure */\n \t\t\tdelete_tempfile(&pidfile);\n \n \t\t\t/*\n@@ -716,11 +739,11 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \t\tif (keep_largest_pack != -1) {\n \t\t\tif (keep_largest_pack)\n \t\t\t\tfind_base_packs(&keep_pack, 0);\n-\t\t} else if (big_pack_threshold) {\n-\t\t\tfind_base_packs(&keep_pack, big_pack_threshold);\n+\t\t} else if (cfg.big_pack_threshold) {\n+\t\t\tfind_base_packs(&keep_pack, cfg.big_pack_threshold);\n \t\t}\n \n-\t\tadd_repack_all_option(&keep_pack);\n+\t\tadd_repack_all_option(&cfg, &keep_pack);\n \t\tstring_list_clear(&keep_pack, 0);\n \t}\n \n@@ -741,7 +764,7 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \t\tatexit(process_log_file_at_exit);\n \t}\n \n-\tgc_before_repack(&opts);\n+\tgc_before_repack(&opts, &cfg);\n \n \tif (!repository_format_precious_objects) {\n \t\tstruct child_process repack_cmd = CHILD_PROCESS_INIT;\n@@ -752,11 +775,11 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \t\tif (run_command(&repack_cmd))\n \t\t\tdie(FAILED_RUN, repack.v[0]);\n \n-\t\tif (prune_expire) {\n+\t\tif (cfg.prune_expire) {\n \t\t\tstruct child_process prune_cmd = CHILD_PROCESS_INIT;\n \n \t\t\t/* run `git prune` even if using cruft packs */\n-\t\t\tstrvec_push(&prune, prune_expire);\n+\t\t\tstrvec_push(&prune, cfg.prune_expire);\n \t\t\tif (quiet)\n \t\t\t\tstrvec_push(&prune, \"--no-progress\");\n \t\t\tif (repo_has_promisor_remote(the_repository))\n@@ -769,10 +792,10 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \t\t}\n \t}\n \n-\tif (prune_worktrees_expire) {\n+\tif (cfg.prune_worktrees_expire) {\n \t\tstruct child_process prune_worktrees_cmd = CHILD_PROCESS_INIT;\n \n-\t\tstrvec_push(&prune_worktrees, prune_worktrees_expire);\n+\t\tstrvec_push(&prune_worktrees, cfg.prune_worktrees_expire);\n \t\tprune_worktrees_cmd.git_cmd = 1;\n \t\tstrvec_pushv(&prune_worktrees_cmd.args, prune_worktrees.v);\n \t\tif (run_command(&prune_worktrees_cmd))\n@@ -796,7 +819,7 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \t\t\t\t\t !quiet && !daemonized ? COMMIT_GRAPH_WRITE_PROGRESS : 0,\n \t\t\t\t\t NULL);\n \n-\tif (opts.auto_flag && too_many_loose_objects())\n+\tif (opts.auto_flag && too_many_loose_objects(&cfg))\n \t\twarning(_(\"There are too many unreachable loose objects; \"\n \t\t\t\"run 'git prune' to remove them.\"));\n \n@@ -892,7 +915,7 @@ static int dfs_on_ref(const char *refname UNUSED,\n \treturn result;\n }\n \n-static int should_write_commit_graph(void)\n+static int should_write_commit_graph(struct gc_config *cfg)\n {\n \tint result;\n \tstruct cg_auto_data data;\n@@ -929,7 +952,8 @@ static int run_write_commit_graph(struct maintenance_run_opts *opts)\n \treturn !!run_command(&child);\n }\n \n-static int maintenance_task_commit_graph(struct maintenance_run_opts *opts)\n+static int maintenance_task_commit_graph(struct maintenance_run_opts *opts,\n+\t\t\t\t\t struct gc_config *cfg)\n {\n \tprepare_repo_settings(the_repository);\n \tif (!the_repository->settings.core_commit_graph)\n@@ -963,7 +987,8 @@ static int fetch_remote(struct remote *remote, void *cbdata)\n \treturn !!run_command(&child);\n }\n \n-static int maintenance_task_prefetch(struct maintenance_run_opts *opts)\n+static int maintenance_task_prefetch(struct maintenance_run_opts *opts,\n+\t\t\t\t struct gc_config *cfg)\n {\n \tif (for_each_remote(fetch_remote, opts)) {\n \t\terror(_(\"failed to prefetch remotes\"));\n@@ -973,7 +998,8 @@ static int maintenance_task_prefetch(struct maintenance_run_opts *opts)\n \treturn 0;\n }\n \n-static int maintenance_task_gc(struct maintenance_run_opts *opts)\n+static int maintenance_task_gc(struct maintenance_run_opts *opts,\n+\t\t\t struct gc_config *cfg)\n {\n \tstruct child_process child = CHILD_PROCESS_INIT;\n \n@@ -1021,7 +1047,7 @@ static int loose_object_count(const struct object_id *oid UNUSED,\n \treturn 0;\n }\n \n-static int loose_object_auto_condition(void)\n+static int loose_object_auto_condition(struct gc_config *cfg)\n {\n \tint count = 0;\n \n@@ -1106,12 +1132,13 @@ static int pack_loose(struct maintenance_run_opts *opts)\n \treturn result;\n }\n \n-static int maintenance_task_loose_objects(struct maintenance_run_opts *opts)\n+static int maintenance_task_loose_objects(struct maintenance_run_opts *opts,\n+\t\t\t\t\t struct gc_config *cfg)\n {\n \treturn prune_packed(opts) || pack_loose(opts);\n }\n \n-static int incremental_repack_auto_condition(void)\n+static int incremental_repack_auto_condition(struct gc_config *cfg)\n {\n \tstruct packed_git *p;\n \tint incremental_repack_auto_limit = 10;\n@@ -1230,7 +1257,8 @@ static int multi_pack_index_repack(struct maintenance_run_opts *opts)\n \treturn 0;\n }\n \n-static int maintenance_task_incremental_repack(struct maintenance_run_opts *opts)\n+static int maintenance_task_incremental_repack(struct maintenance_run_opts *opts,\n+\t\t\t\t\t struct gc_config *cfg)\n {\n \tprepare_repo_settings(the_repository);\n \tif (!the_repository->settings.core_multi_pack_index) {\n@@ -1247,14 +1275,15 @@ static int maintenance_task_incremental_repack(struct maintenance_run_opts *opts\n \treturn 0;\n }\n \n-typedef int maintenance_task_fn(struct maintenance_run_opts *opts);\n+typedef int maintenance_task_fn(struct maintenance_run_opts *opts,\n+\t\t\t\tstruct gc_config *cfg);\n \n /*\n * An auto condition function returns 1 if the task should run\n * and 0 if the task should NOT run. See needs_to_gc() for an\n * example.\n */\n-typedef int maintenance_auto_fn(void);\n+typedef int maintenance_auto_fn(struct gc_config *cfg);\n \n struct maintenance_task {\n \tconst char *name;\n@@ -1321,7 +1350,8 @@ static int compare_tasks_by_selection(const void *a_, const void *b_)\n \treturn b->selected_order - a->selected_order;\n }\n \n-static int maintenance_run_tasks(struct maintenance_run_opts *opts)\n+static int maintenance_run_tasks(struct maintenance_run_opts *opts,\n+\t\t\t\t struct gc_config *cfg)\n {\n \tint i, found_selected = 0;\n \tint result = 0;\n@@ -1360,14 +1390,14 @@ static int maintenance_run_tasks(struct maintenance_run_opts *opts)\n \n \t\tif (opts->auto_flag &&\n \t\t (!tasks[i].auto_condition ||\n-\t\t !tasks[i].auto_condition()))\n+\t\t !tasks[i].auto_condition(cfg)))\n \t\t\tcontinue;\n \n \t\tif (opts->schedule && tasks[i].schedule < opts->schedule)\n \t\t\tcontinue;\n \n \t\ttrace2_region_enter(\"maintenance\", tasks[i].name, r);\n-\t\tif (tasks[i].fn(opts)) {\n+\t\tif (tasks[i].fn(opts, cfg)) {\n \t\t\terror(_(\"task '%s' failed\"), tasks[i].name);\n \t\t\tresult = 1;\n \t\t}\n@@ -1404,7 +1434,6 @@ static void initialize_task_config(int schedule)\n {\n \tint i;\n \tstruct strbuf config_name = STRBUF_INIT;\n-\tgc_config();\n \n \tif (schedule)\n \t\tinitialize_maintenance_strategy();\n@@ -1468,6 +1497,7 @@ static int maintenance_run(int argc, const char **argv, const char *prefix)\n {\n \tint i;\n \tstruct maintenance_run_opts opts;\n+\tstruct gc_config cfg = GC_CONFIG_INIT;\n \tstruct option builtin_maintenance_run_options[] = {\n \t\tOPT_BOOL(0, \"auto\", &opts.auto_flag,\n \t\t\t N_(\"run tasks based on the state of the repository\")),\n@@ -1496,12 +1526,13 @@ static int maintenance_run(int argc, const char **argv, const char *prefix)\n \tif (opts.auto_flag && opts.schedule)\n \t\tdie(_(\"use at most one of --auto and --schedule=\"));\n \n+\tgc_config(&cfg);\n \tinitialize_task_config(opts.schedule);\n \n \tif (argc != 0)\n \t\tusage_with_options(builtin_maintenance_run_usage,\n \t\t\t\t builtin_maintenance_run_options);\n-\treturn maintenance_run_tasks(&opts);\n+\treturn maintenance_run_tasks(&opts, &cfg);\n }\n \n static char *get_maintpath(void)\n-- \n2.46.0.46.g406f326d27.dirty\n\n"},{"id":"501125","messageId":"310e361371efc156c3aaac94439bdbeaa965155f.1723804990.git.ps@pks.im","threadId":"61940","inReplyTo":"cover.1723804990.git.ps@pks.im","subject":"[PATCH v3 3/7] builtin/gc: fix leaking config values","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-08-16T10:45:03Z","receivedAt":"2024-08-16T10:45:07Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"We're leaking config values in git-gc(1) when those values are tracked\nas strings. Introduce a new `gc_config_release()` function that releases\nthis memory to plug those leaks and release old values before populating\nthe config fields via `git_config_string()` et al.\n\nNote that there is one small gotcha here with the \"--prune\" option. Next\nto passing a string, this option also accepts the \"--no-prune\" option\nthat overrides the default or configured value. We thus need to discern\nbetween the option not having been passed by the user and the negative\nvariant of it. This is done by using a simple sentinel value that lets\nus discern these cases.\n\nSigned-off-by: Patrick Steinhardt \n---\n builtin/gc.c | 108 +++++++++++++++++++++++++++++++++++------------\n t/t5304-prune.sh | 1 +\n 2 files changed, 82 insertions(+), 27 deletions(-)\n\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex eee7401647..a93cfa147e 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -139,9 +139,9 @@ struct gc_config {\n \tint gc_auto_threshold;\n \tint gc_auto_pack_limit;\n \tint detach_auto;\n-\tconst char *gc_log_expire;\n-\tconst char *prune_expire;\n-\tconst char *prune_worktrees_expire;\n+\tchar *gc_log_expire;\n+\tchar *prune_expire;\n+\tchar *prune_worktrees_expire;\n \tchar *repack_filter;\n \tchar *repack_filter_to;\n \tunsigned long big_pack_threshold;\n@@ -157,15 +157,25 @@ struct gc_config {\n \t.gc_auto_threshold = 6700, \\\n \t.gc_auto_pack_limit = 50, \\\n \t.detach_auto = 1, \\\n-\t.gc_log_expire = \"1.day.ago\", \\\n-\t.prune_expire = \"2.weeks.ago\", \\\n-\t.prune_worktrees_expire = \"3.months.ago\", \\\n+\t.gc_log_expire = xstrdup(\"1.day.ago\"), \\\n+\t.prune_expire = xstrdup(\"2.weeks.ago\"), \\\n+\t.prune_worktrees_expire = xstrdup(\"3.months.ago\"), \\\n \t.max_delta_cache_size = DEFAULT_DELTA_CACHE_SIZE, \\\n }\n \n+static void gc_config_release(struct gc_config *cfg)\n+{\n+\tfree(cfg->gc_log_expire);\n+\tfree(cfg->prune_expire);\n+\tfree(cfg->prune_worktrees_expire);\n+\tfree(cfg->repack_filter);\n+\tfree(cfg->repack_filter_to);\n+}\n+\n static void gc_config(struct gc_config *cfg)\n {\n \tconst char *value;\n+\tchar *owned = NULL;\n \n \tif (!git_config_get_value(\"gc.packrefs\", &value)) {\n \t\tif (value && !strcmp(value, \"notbare\"))\n@@ -185,15 +195,34 @@ static void gc_config(struct gc_config *cfg)\n \tgit_config_get_bool(\"gc.autodetach\", &cfg->detach_auto);\n \tgit_config_get_bool(\"gc.cruftpacks\", &cfg->cruft_packs);\n \tgit_config_get_ulong(\"gc.maxcruftsize\", &cfg->max_cruft_size);\n-\tgit_config_get_expiry(\"gc.pruneexpire\", (char **) &cfg->prune_expire);\n-\tgit_config_get_expiry(\"gc.worktreepruneexpire\", (char **) &cfg->prune_worktrees_expire);\n-\tgit_config_get_expiry(\"gc.logexpiry\", (char **) &cfg->gc_log_expire);\n+\n+\tif (!git_config_get_expiry(\"gc.pruneexpire\", &owned)) {\n+\t\tfree(cfg->prune_expire);\n+\t\tcfg->prune_expire = owned;\n+\t}\n+\n+\tif (!git_config_get_expiry(\"gc.worktreepruneexpire\", &owned)) {\n+\t\tfree(cfg->prune_worktrees_expire);\n+\t\tcfg->prune_worktrees_expire = owned;\n+\t}\n+\n+\tif (!git_config_get_expiry(\"gc.logexpiry\", &owned)) {\n+\t\tfree(cfg->gc_log_expire);\n+\t\tcfg->gc_log_expire = owned;\n+\t}\n \n \tgit_config_get_ulong(\"gc.bigpackthreshold\", &cfg->big_pack_threshold);\n \tgit_config_get_ulong(\"pack.deltacachesize\", &cfg->max_delta_cache_size);\n \n-\tgit_config_get_string(\"gc.repackfilter\", &cfg->repack_filter);\n-\tgit_config_get_string(\"gc.repackfilterto\", &cfg->repack_filter_to);\n+\tif (!git_config_get_string(\"gc.repackfilter\", &owned)) {\n+\t\tfree(cfg->repack_filter);\n+\t\tcfg->repack_filter = owned;\n+\t}\n+\n+\tif (!git_config_get_string(\"gc.repackfilterto\", &owned)) {\n+\t\tfree(cfg->repack_filter_to);\n+\t\tcfg->repack_filter_to = owned;\n+\t}\n \n \tgit_config(git_default_config, NULL);\n }\n@@ -644,12 +673,15 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \tstruct child_process rerere_cmd = CHILD_PROCESS_INIT;\n \tstruct maintenance_run_opts opts = {0};\n \tstruct gc_config cfg = GC_CONFIG_INIT;\n+\tconst char *prune_expire_sentinel = \"sentinel\";\n+\tconst char *prune_expire_arg = prune_expire_sentinel;\n+\tint ret;\n \n \tstruct option builtin_gc_options[] = {\n \t\tOPT__QUIET(&quiet, N_(\"suppress progress reporting\")),\n-\t\t{ OPTION_STRING, 0, \"prune\", &cfg.prune_expire, N_(\"date\"),\n+\t\t{ OPTION_STRING, 0, \"prune\", &prune_expire_arg, N_(\"date\"),\n \t\t\tN_(\"prune unreferenced objects\"),\n-\t\t\tPARSE_OPT_OPTARG, NULL, (intptr_t)cfg.prune_expire },\n+\t\t\tPARSE_OPT_OPTARG, NULL, (intptr_t)prune_expire_arg },\n \t\tOPT_BOOL(0, \"cruft\", &cfg.cruft_packs, N_(\"pack unreferenced objects separately\")),\n \t\tOPT_MAGNITUDE(0, \"max-cruft-size\", &cfg.max_cruft_size,\n \t\t\t N_(\"with --cruft, limit the size of new cruft packs\")),\n@@ -673,8 +705,8 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \tstrvec_pushl(&prune_worktrees, \"worktree\", \"prune\", \"--expire\", NULL);\n \tstrvec_pushl(&rerere, \"rerere\", \"gc\", NULL);\n \n-\t/* default expiry time, overwritten in gc_config */\n \tgc_config(&cfg);\n+\n \tif (parse_expiry_date(cfg.gc_log_expire, &gc_log_expire_time))\n \t\tdie(_(\"failed to parse gc.logExpiry value %s\"), cfg.gc_log_expire);\n \n@@ -686,6 +718,10 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \tif (argc > 0)\n \t\tusage_with_options(builtin_gc_usage, builtin_gc_options);\n \n+\tif (prune_expire_arg != prune_expire_sentinel) {\n+\t\tfree(cfg.prune_expire);\n+\t\tcfg.prune_expire = xstrdup_or_null(prune_expire_arg);\n+\t}\n \tif (cfg.prune_expire && parse_expiry_date(cfg.prune_expire, &dummy))\n \t\tdie(_(\"failed to parse prune expiry value %s\"), cfg.prune_expire);\n \n@@ -703,8 +739,11 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \t\t/*\n \t\t * Auto-gc should be least intrusive as possible.\n \t\t */\n-\t\tif (!need_to_gc(&cfg))\n-\t\t\treturn 0;\n+\t\tif (!need_to_gc(&cfg)) {\n+\t\t\tret = 0;\n+\t\t\tgoto out;\n+\t\t}\n+\n \t\tif (!quiet) {\n \t\t\tif (cfg.detach_auto)\n \t\t\t\tfprintf(stderr, _(\"Auto packing the repository in background for optimum performance.\\n\"));\n@@ -713,17 +752,22 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \t\t\tfprintf(stderr, _(\"See \\\"git help gc\\\" for manual housekeeping.\\n\"));\n \t\t}\n \t\tif (cfg.detach_auto) {\n-\t\t\tint ret = report_last_gc_error();\n-\n-\t\t\tif (ret == 1)\n+\t\t\tret = report_last_gc_error();\n+\t\t\tif (ret == 1) {\n \t\t\t\t/* Last gc --auto failed. Skip this one. */\n-\t\t\t\treturn 0;\n-\t\t\telse if (ret)\n+\t\t\t\tret = 0;\n+\t\t\t\tgoto out;\n+\n+\t\t\t} else if (ret) {\n \t\t\t\t/* an I/O error occurred, already reported */\n-\t\t\t\treturn ret;\n+\t\t\t\tgoto out;\n+\t\t\t}\n+\n+\t\t\tif (lock_repo_for_gc(force, &pid)) {\n+\t\t\t\tret = 0;\n+\t\t\t\tgoto out;\n+\t\t\t}\n \n-\t\t\tif (lock_repo_for_gc(force, &pid))\n-\t\t\t\treturn 0;\n \t\t\tgc_before_repack(&opts, &cfg); /* dies on failure */\n \t\t\tdelete_tempfile(&pidfile);\n \n@@ -749,8 +793,11 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \n \tname = lock_repo_for_gc(force, &pid);\n \tif (name) {\n-\t\tif (opts.auto_flag)\n-\t\t\treturn 0; /* be quiet on --auto */\n+\t\tif (opts.auto_flag) {\n+\t\t\tret = 0;\n+\t\t\tgoto out; /* be quiet on --auto */\n+\t\t}\n+\n \t\tdie(_(\"gc is already running on machine '%s' pid %\"PRIuMAX\" (use --force if not)\"),\n \t\t name, (uintmax_t)pid);\n \t}\n@@ -826,6 +873,8 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \tif (!daemonized)\n \t\tunlink(git_path(\"gc.log\"));\n \n+out:\n+\tgc_config_release(&cfg);\n \treturn 0;\n }\n \n@@ -1511,6 +1560,8 @@ static int maintenance_run(int argc, const char **argv, const char *prefix)\n \t\t\tPARSE_OPT_NONEG, task_option_parse),\n \t\tOPT_END()\n \t};\n+\tint ret;\n+\n \tmemset(&opts, 0, sizeof(opts));\n \n \topts.quiet = !isatty(2);\n@@ -1532,7 +1583,10 @@ static int maintenance_run(int argc, const char **argv, const char *prefix)\n \tif (argc != 0)\n \t\tusage_with_options(builtin_maintenance_run_usage,\n \t\t\t\t builtin_maintenance_run_options);\n-\treturn maintenance_run_tasks(&opts, &cfg);\n+\n+\tret = maintenance_run_tasks(&opts, &cfg);\n+\tgc_config_release(&cfg);\n+\treturn ret;\n }\n \n static char *get_maintpath(void)\ndiff --git a/t/t5304-prune.sh b/t/t5304-prune.sh\nindex 1f1f664871..e641df0116 100755\n--- a/t/t5304-prune.sh\n+++ b/t/t5304-prune.sh\n@@ -7,6 +7,7 @@ test_description='prune'\n GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main\n export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME\n \n+TEST_PASSES_SANITIZE_LEAK=true\n . ./test-lib.sh\n \n day=$((60*60*24))\n-- \n2.46.0.46.g406f326d27.dirty\n\n"},{"id":"501126","messageId":"812c61c9b66d7608e41c6c1d00a6e22e995cef06.1723804990.git.ps@pks.im","threadId":"61940","inReplyTo":"cover.1723804990.git.ps@pks.im","subject":"[PATCH v3 4/7] builtin/gc: stop processing log file on signal","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-08-16T10:45:06Z","receivedAt":"2024-08-16T10:45:12Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"When detaching, git-gc(1) will redirect its stderr to a \"gc.log\" log\nfile, which is then used to surface errors of a backgrounded process to\nthe user. To ensure that the file is properly managed on abnormal exit\npaths, we install both signal and exit handlers that try to either\ncommit the underlying lock file or roll it back in case there wasn't any\nerror.\n\nThis logic is severly broken when handling signals though, as we end up\ncalling all kinds of functions that are not signal safe. This includes\nmalloc(3P) via `git_path()`, fprintf(3P), fflush(3P) and many more\nfunctions. The consequence can be anything, from deadlocks to crashes.\nUnfortunately, we cannot really do much about this without a larger\nrefactoring.\n\nThe least-worst thing we can do is to not set up the signal handler in\nthe first place. This will still cause us to remove the lockfile, as the\nunderlying tempfile subsystem already knows to unlink locks when\nreceiving a signal. But it may cause us to remove the lock even in the\ncase where it would have contained actual errors, which is a change in\nbehaviour.\n\nThe consequence is that \"gc.log\" will not be committed, and thus\nsubsequent calls to `git gc --auto` won't bail out because of this.\nArguably though, it is better to retry garbage collection rather than\nhaving the process run into a potentially-corrupted state.\n\nSigned-off-by: Patrick Steinhardt \n---\n builtin/gc.c | 8 --------\n 1 file changed, 8 deletions(-)\n\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex a93cfa147e..f815557081 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -109,13 +109,6 @@ static void process_log_file_at_exit(void)\n \tprocess_log_file();\n }\n \n-static void process_log_file_on_signal(int signo)\n-{\n-\tprocess_log_file();\n-\tsigchain_pop(signo);\n-\traise(signo);\n-}\n-\n static int gc_config_is_timestamp_never(const char *var)\n {\n \tconst char *value;\n@@ -807,7 +800,6 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \t\t\t\t\t git_path(\"gc.log\"),\n \t\t\t\t\t LOCK_DIE_ON_ERROR);\n \t\tdup2(get_lock_file_fd(&log_lock), 2);\n-\t\tsigchain_push_common(process_log_file_on_signal);\n \t\tatexit(process_log_file_at_exit);\n \t}\n \n-- \n2.46.0.46.g406f326d27.dirty\n\n"},{"id":"501127","messageId":"b934b238897c55595de4735dcfec5425b6ab22af.1723804990.git.ps@pks.im","threadId":"61940","inReplyTo":"cover.1723804990.git.ps@pks.im","subject":"[PATCH v3 5/7] builtin/gc: add a `--detach` flag","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-08-16T10:45:11Z","receivedAt":"2024-08-16T10:45:15Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"When running `git gc --auto`, the command will by default detach and\ncontinue running in the background. This behaviour can be tweaked via\nthe `gc.autoDetach` config, but not via a command line switch. We need\nthat in a subsequent commit though, where git-maintenance(1) will want\nto ask its git-gc(1) child process to not detach anymore.\n\nAdd a `--[no-]detach` flag that does this for us.\n\nSigned-off-by: Patrick Steinhardt \n---\n Documentation/git-gc.txt | 5 ++-\n builtin/gc.c | 70 ++++++++++++++++++++++------------------\n t/t6500-gc.sh | 45 +++++++++++++++++++++++---\n 3 files changed, 84 insertions(+), 36 deletions(-)\n\ndiff --git a/Documentation/git-gc.txt b/Documentation/git-gc.txt\nindex b5561c458a..370e22faae 100644\n--- a/Documentation/git-gc.txt\n+++ b/Documentation/git-gc.txt\n@@ -9,7 +9,7 @@ git-gc - Cleanup unnecessary files and optimize the local repository\n SYNOPSIS\n --------\n [verse]\n-'git gc' [--aggressive] [--auto] [--quiet] [--prune= | --no-prune] [--force] [--keep-largest-pack]\n+'git gc' [--aggressive] [--auto] [--[no-]detach] [--quiet] [--prune= | --no-prune] [--force] [--keep-largest-pack]\n \n DESCRIPTION\n -----------\n@@ -53,6 +53,9 @@ configuration options such as `gc.auto` and `gc.autoPackLimit`, all\n other housekeeping tasks (e.g. rerere, working trees, reflog...) will\n be performed as well.\n \n+--[no-]detach::\n+\tRun in the background if the system supports it. This option overrides\n+\tthe `gc.autoDetach` config.\n \n --[no-]cruft::\n \tWhen expiring unreachable objects, pack them separately into a\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex f815557081..269a77960f 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -242,9 +242,13 @@ static enum schedule_priority parse_schedule(const char *value)\n \n struct maintenance_run_opts {\n \tint auto_flag;\n+\tint detach;\n \tint quiet;\n \tenum schedule_priority schedule;\n };\n+#define MAINTENANCE_RUN_OPTS_INIT { \\\n+\t.detach = -1, \\\n+}\n \n static int pack_refs_condition(UNUSED struct gc_config *cfg)\n {\n@@ -664,7 +668,7 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \tint keep_largest_pack = -1;\n \ttimestamp_t dummy;\n \tstruct child_process rerere_cmd = CHILD_PROCESS_INIT;\n-\tstruct maintenance_run_opts opts = {0};\n+\tstruct maintenance_run_opts opts = MAINTENANCE_RUN_OPTS_INIT;\n \tstruct gc_config cfg = GC_CONFIG_INIT;\n \tconst char *prune_expire_sentinel = \"sentinel\";\n \tconst char *prune_expire_arg = prune_expire_sentinel;\n@@ -681,6 +685,8 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \t\tOPT_BOOL(0, \"aggressive\", &aggressive, N_(\"be more thorough (increased runtime)\")),\n \t\tOPT_BOOL_F(0, \"auto\", &opts.auto_flag, N_(\"enable auto-gc mode\"),\n \t\t\t PARSE_OPT_NOCOMPLETE),\n+\t\tOPT_BOOL(0, \"detach\", &opts.detach,\n+\t\t\t N_(\"perform garbage collection in the background\")),\n \t\tOPT_BOOL_F(0, \"force\", &force,\n \t\t\t N_(\"force running gc even if there may be another gc running\"),\n \t\t\t PARSE_OPT_NOCOMPLETE),\n@@ -729,6 +735,9 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \t\tstrvec_push(&repack, \"-q\");\n \n \tif (opts.auto_flag) {\n+\t\tif (cfg.detach_auto && opts.detach < 0)\n+\t\t\topts.detach = 1;\n+\n \t\t/*\n \t\t * Auto-gc should be least intrusive as possible.\n \t\t */\n@@ -738,38 +747,12 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \t\t}\n \n \t\tif (!quiet) {\n-\t\t\tif (cfg.detach_auto)\n+\t\t\tif (opts.detach > 0)\n \t\t\t\tfprintf(stderr, _(\"Auto packing the repository in background for optimum performance.\\n\"));\n \t\t\telse\n \t\t\t\tfprintf(stderr, _(\"Auto packing the repository for optimum performance.\\n\"));\n \t\t\tfprintf(stderr, _(\"See \\\"git help gc\\\" for manual housekeeping.\\n\"));\n \t\t}\n-\t\tif (cfg.detach_auto) {\n-\t\t\tret = report_last_gc_error();\n-\t\t\tif (ret == 1) {\n-\t\t\t\t/* Last gc --auto failed. Skip this one. */\n-\t\t\t\tret = 0;\n-\t\t\t\tgoto out;\n-\n-\t\t\t} else if (ret) {\n-\t\t\t\t/* an I/O error occurred, already reported */\n-\t\t\t\tgoto out;\n-\t\t\t}\n-\n-\t\t\tif (lock_repo_for_gc(force, &pid)) {\n-\t\t\t\tret = 0;\n-\t\t\t\tgoto out;\n-\t\t\t}\n-\n-\t\t\tgc_before_repack(&opts, &cfg); /* dies on failure */\n-\t\t\tdelete_tempfile(&pidfile);\n-\n-\t\t\t/*\n-\t\t\t * failure to daemonize is ok, we'll continue\n-\t\t\t * in foreground\n-\t\t\t */\n-\t\t\tdaemonized = !daemonize();\n-\t\t}\n \t} else {\n \t\tstruct string_list keep_pack = STRING_LIST_INIT_NODUP;\n \n@@ -784,6 +767,33 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \t\tstring_list_clear(&keep_pack, 0);\n \t}\n \n+\tif (opts.detach > 0) {\n+\t\tret = report_last_gc_error();\n+\t\tif (ret == 1) {\n+\t\t\t/* Last gc --auto failed. Skip this one. */\n+\t\t\tret = 0;\n+\t\t\tgoto out;\n+\n+\t\t} else if (ret) {\n+\t\t\t/* an I/O error occurred, already reported */\n+\t\t\tgoto out;\n+\t\t}\n+\n+\t\tif (lock_repo_for_gc(force, &pid)) {\n+\t\t\tret = 0;\n+\t\t\tgoto out;\n+\t\t}\n+\n+\t\tgc_before_repack(&opts, &cfg); /* dies on failure */\n+\t\tdelete_tempfile(&pidfile);\n+\n+\t\t/*\n+\t\t * failure to daemonize is ok, we'll continue\n+\t\t * in foreground\n+\t\t */\n+\t\tdaemonized = !daemonize();\n+\t}\n+\n \tname = lock_repo_for_gc(force, &pid);\n \tif (name) {\n \t\tif (opts.auto_flag) {\n@@ -1537,7 +1547,7 @@ static int task_option_parse(const struct option *opt UNUSED,\n static int maintenance_run(int argc, const char **argv, const char *prefix)\n {\n \tint i;\n-\tstruct maintenance_run_opts opts;\n+\tstruct maintenance_run_opts opts = MAINTENANCE_RUN_OPTS_INIT;\n \tstruct gc_config cfg = GC_CONFIG_INIT;\n \tstruct option builtin_maintenance_run_options[] = {\n \t\tOPT_BOOL(0, \"auto\", &opts.auto_flag,\n@@ -1554,8 +1564,6 @@ static int maintenance_run(int argc, const char **argv, const char *prefix)\n \t};\n \tint ret;\n \n-\tmemset(&opts, 0, sizeof(opts));\n-\n \topts.quiet = !isatty(2);\n \n \tfor (i = 0; i < TASK__COUNT; i++)\ndiff --git a/t/t6500-gc.sh b/t/t6500-gc.sh\nindex 1b5909d1b7..5378455968 100755\n--- a/t/t6500-gc.sh\n+++ b/t/t6500-gc.sh\n@@ -338,14 +338,14 @@ test_expect_success 'gc.maxCruftSize sets appropriate repack options' '\n \ttest_subcommand $cruft_max_size_opts --max-cruft-size=3145728 &1)\n+\tdoesnt_matter=$(git gc \"$@\" 9>&1)\n }\n \n test_expect_success 'background auto gc does not run if gc.log is present and recent but does if it is old' '\n@@ -361,7 +361,7 @@ test_expect_success 'background auto gc does not run if gc.log is present and re\n \ttest-tool chmtime =-345600 .git/gc.log &&\n \tgit gc --auto &&\n \ttest_config gc.logexpiry 2.days &&\n-\trun_and_wait_for_auto_gc &&\n+\trun_and_wait_for_gc --auto &&\n \tls .git/objects/pack/pack-*.pack >packs &&\n \ttest_line_count = 1 packs\n '\n@@ -391,11 +391,48 @@ test_expect_success 'background auto gc respects lock for all operations' '\n \tprintf \"%d %s\" \"$shell_pid\" \"$hostname\" >.git/gc.pid &&\n \n \t# our gc should exit zero without doing anything\n-\trun_and_wait_for_auto_gc &&\n+\trun_and_wait_for_gc --auto &&\n \t(ls -1 .git/refs/heads .git/reftable >actual || true) &&\n \ttest_cmp expect actual\n '\n \n+test_expect_success '--detach overrides gc.autoDetach=false' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\n+\t\t# Prepare the repository such that git-gc(1) ends up repacking.\n+\t\ttest_commit \"$(test_oid blob17_1)\" &&\n+\t\ttest_commit \"$(test_oid blob17_2)\" &&\n+\t\tgit config gc.autodetach false &&\n+\t\tgit config gc.auto 2 &&\n+\n+\t\t# Note that we cannot use `test_cmp` here to compare stderr\n+\t\t# because it may contain output from `set -x`.\n+\t\trun_and_wait_for_gc --auto --detach 2>actual &&\n+\t\ttest_grep \"Auto packing the repository in background for optimum performance.\" actual\n+\t)\n+'\n+\n+test_expect_success '--no-detach overrides gc.autoDetach=true' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\n+\t\t# Prepare the repository such that git-gc(1) ends up repacking.\n+\t\ttest_commit \"$(test_oid blob17_1)\" &&\n+\t\ttest_commit \"$(test_oid blob17_2)\" &&\n+\t\tgit config gc.autodetach true &&\n+\t\tgit config gc.auto 2 &&\n+\n+\t\tGIT_PROGRESS_DELAY=0 git gc --auto --no-detach 2>output &&\n+\t\ttest_grep \"Auto packing the repository for optimum performance.\" output &&\n+\t\ttest_grep \"Collecting referenced commits: 2, done.\" output\n+\t)\n+'\n+\n # DO NOT leave a detached auto gc process running near the end of the\n # test script: it can run long enough in the background to racily\n # interfere with the cleanup in 'test_done'.\n-- \n2.46.0.46.g406f326d27.dirty\n\n"},{"id":"501128","messageId":"347d0a200201ce215f5b2c46d23de0cdd0181956.1723804990.git.ps@pks.im","threadId":"61940","inReplyTo":"cover.1723804990.git.ps@pks.im","subject":"[PATCH v3 6/7] builtin/maintenance: add a `--detach` flag","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-08-16T10:45:15Z","receivedAt":"2024-08-16T10:45:18Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Same as the preceding commit, add a `--[no-]detach` flag to the\ngit-maintenance(1) command. This will be used in a subsequent commit to\nfix backgrounding of that command when configured with a non-standard\nset of tasks.\n\nSigned-off-by: Patrick Steinhardt \n---\n builtin/gc.c | 6 ++++++\n t/t7900-maintenance.sh | 39 +++++++++++++++++++++++++++++++++++++++\n 2 files changed, 45 insertions(+)\n\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 269a77960f..63106e2028 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -1426,6 +1426,10 @@ static int maintenance_run_tasks(struct maintenance_run_opts *opts,\n \t}\n \tfree(lock_path);\n \n+\t/* Failure to daemonize is ok, we'll continue in foreground. */\n+\tif (opts->detach > 0)\n+\t\tdaemonize();\n+\n \tfor (i = 0; !found_selected && i < TASK__COUNT; i++)\n \t\tfound_selected = tasks[i].selected_order >= 0;\n \n@@ -1552,6 +1556,8 @@ static int maintenance_run(int argc, const char **argv, const char *prefix)\n \tstruct option builtin_maintenance_run_options[] = {\n \t\tOPT_BOOL(0, \"auto\", &opts.auto_flag,\n \t\t\t N_(\"run tasks based on the state of the repository\")),\n+\t\tOPT_BOOL(0, \"detach\", &opts.detach,\n+\t\t\t N_(\"perform maintenance in the background\")),\n \t\tOPT_CALLBACK(0, \"schedule\", &opts.schedule, N_(\"frequency\"),\n \t\t\t N_(\"run tasks based on frequency\"),\n \t\t\t maintenance_opt_schedule),\ndiff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\nindex 8595489ceb..771525aa4b 100755\n--- a/t/t7900-maintenance.sh\n+++ b/t/t7900-maintenance.sh\n@@ -908,4 +908,43 @@ test_expect_success 'failed schedule prevents config change' '\n \tdone\n '\n \n+test_expect_success '--no-detach causes maintenance to not run in background' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\n+\t\t# Prepare the repository such that git-maintenance(1) ends up\n+\t\t# outputting something.\n+\t\ttest_commit something &&\n+\t\tgit config set maintenance.gc.enabled false &&\n+\t\tgit config set maintenance.loose-objects.enabled true &&\n+\t\tgit config set maintenance.loose-objects.auto 1 &&\n+\t\tgit config set maintenance.incremental-repack.enabled true &&\n+\n+\t\t# We have no better way to check whether or not the task ran in\n+\t\t# the background than to verify whether it output anything. The\n+\t\t# next testcase checks the reverse, making this somewhat safer.\n+\t\tgit maintenance run --no-detach >out 2>&1 &&\n+\t\ttest_line_count = 1 out\n+\t)\n+'\n+\n+test_expect_success '--detach causes maintenance to run in background' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\n+\t\ttest_commit something &&\n+\t\tgit config set maintenance.gc.enabled false &&\n+\t\tgit config set maintenance.loose-objects.enabled true &&\n+\t\tgit config set maintenance.loose-objects.auto 1 &&\n+\t\tgit config set maintenance.incremental-repack.enabled true &&\n+\n+\t\tgit maintenance run --detach >out 2>&1 &&\n+\t\ttest_must_be_empty out\n+\t)\n+'\n+\n test_done\n-- \n2.46.0.46.g406f326d27.dirty\n\n"},{"id":"501129","messageId":"9befef7c1f7520d58af2b2db17174b8dbc493d56.1723804990.git.ps@pks.im","threadId":"61940","inReplyTo":"cover.1723804990.git.ps@pks.im","subject":"[PATCH v3 7/7] run-command: fix detaching when running auto maintenance","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-08-16T10:45:17Z","receivedAt":"2024-08-16T10:45:21Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"In the past, we used to execute `git gc --auto` as part of our automatic\nhousekeeping routines. As git-gc(1) may require quite some time to\nperform the housekeeping, it knows to detach itself and run in the\nbackground so that the user can continue their work.\n\nEventually, we refactored our automatic housekeeping to instead use the\nmore flexible git-maintenance(1) command. The upside of this new infra\nis that the user can configure which maintenance tasks are performed, at\nleast to a certain degree. So while it continues to run git-gc(1) by\ndefault, it can also be adapted to e.g. use git-multi-pack-index(1) for\nmaintenance of the object database.\n\nThe auto-detach of the new infra is somewhat broken though once the user\nconfigures non-standard tasks. The problem is essentially that we detach\nat the wrong level in the process hierarchy: git-maintenance(1) never\ndetaches itself, but instead it continues to be git-gc(1) which does.\n\nWhen configured to only run the git-gc(1) maintenance task, then the\nresult is basically the same as before. But when configured to run other\ntasks, then git-maintenance(1) will wait for these to run to completion.\nEven worse, it may be that git-gc(1) runs concurrently with other\nhousekeeping tasks, stomping on each others feet.\n\nFix this bug by asking git-gc(1) to not detach when it is being invoked\nvia git-maintenance(1). Instead, git-maintenance(1) now respects a new\nconfig \"maintenance.autoDetach\", the equivalent of \"gc.autoDetach\", and\ndetaches itself into the background when running as part of our auto\nmaintenance. This should continue to behave the same for all users which\nuse the git-gc(1) task, only. For others though, it means that we now\nproperly perform all tasks in the background. The default behaviour of\ngit-maintenance(1) when executed by the user does not change, it will\nremain in the foreground unless they pass the `--detach` option.\n\nSigned-off-by: Patrick Steinhardt \n---\n Documentation/config/gc.txt | 3 +-\n Documentation/config/maintenance.txt | 11 +++++++\n builtin/gc.c | 1 +\n run-command.c | 12 +++++++-\n t/t5616-partial-clone.sh | 6 ++--\n t/t7900-maintenance.sh | 43 ++++++++++++++++++++++------\n 6 files changed, 62 insertions(+), 14 deletions(-)\n\ndiff --git a/Documentation/config/gc.txt b/Documentation/config/gc.txt\nindex 664a3c2874..1d4f9470ea 100644\n--- a/Documentation/config/gc.txt\n+++ b/Documentation/config/gc.txt\n@@ -40,7 +40,8 @@ use, it'll affect how the auto pack limit works.\n \n gc.autoDetach::\n \tMake `git gc --auto` return immediately and run in the background\n-\tif the system supports it. Default is true.\n+\tif the system supports it. Default is true. This config variable acts\n+\tas a fallback in case `maintenance.autoDetach` is not set.\n \n gc.bigPackThreshold::\n \tIf non-zero, all non-cruft packs larger than this limit are kept\ndiff --git a/Documentation/config/maintenance.txt b/Documentation/config/maintenance.txt\nindex 69a4f05153..72a9d6cf81 100644\n--- a/Documentation/config/maintenance.txt\n+++ b/Documentation/config/maintenance.txt\n@@ -3,6 +3,17 @@ maintenance.auto::\n \t`git maintenance run --auto` after doing their normal work. Defaults\n \tto true.\n \n+maintenance.autoDetach::\n+\tMany Git commands trigger automatic maintenance after they have\n+\twritten data into the repository. This boolean config option\n+\tcontrols whether this automatic maintenance shall happen in the\n+\tforeground or whether the maintenance process shall detach and\n+\tcontinue to run in the background.\n++\n+If unset, the value of `gc.autoDetach` is used as a fallback. Defaults\n+to true if both are unset, meaning that the maintenance process will\n+detach.\n+\n maintenance.strategy::\n \tThis string config option provides a way to specify one of a few\n \trecommended schedules for background maintenance. This only affects\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 63106e2028..bafee330a2 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -1063,6 +1063,7 @@ static int maintenance_task_gc(struct maintenance_run_opts *opts,\n \t\tstrvec_push(&child.args, \"--quiet\");\n \telse\n \t\tstrvec_push(&child.args, \"--no-quiet\");\n+\tstrvec_push(&child.args, \"--no-detach\");\n \n \treturn run_command(&child);\n }\ndiff --git a/run-command.c b/run-command.c\nindex 45ba544932..94f2f3079f 100644\n--- a/run-command.c\n+++ b/run-command.c\n@@ -1808,16 +1808,26 @@ void run_processes_parallel(const struct run_process_parallel_opts *opts)\n \n int prepare_auto_maintenance(int quiet, struct child_process *maint)\n {\n-\tint enabled;\n+\tint enabled, auto_detach;\n \n \tif (!git_config_get_bool(\"maintenance.auto\", &enabled) &&\n \t !enabled)\n \t\treturn 0;\n \n+\t/*\n+\t * When `maintenance.autoDetach` isn't set, then we fall back to\n+\t * honoring `gc.autoDetach`. This is somewhat weird, but required to\n+\t * retain behaviour from when we used to run git-gc(1) here.\n+\t */\n+\tif (git_config_get_bool(\"maintenance.autodetach\", &auto_detach) &&\n+\t git_config_get_bool(\"gc.autodetach\", &auto_detach))\n+\t\tauto_detach = 1;\n+\n \tmaint->git_cmd = 1;\n \tmaint->close_object_store = 1;\n \tstrvec_pushl(&maint->args, \"maintenance\", \"run\", \"--auto\", NULL);\n \tstrvec_push(&maint->args, quiet ? \"--quiet\" : \"--no-quiet\");\n+\tstrvec_push(&maint->args, auto_detach ? \"--detach\" : \"--no-detach\");\n \n \treturn 1;\n }\ndiff --git a/t/t5616-partial-clone.sh b/t/t5616-partial-clone.sh\nindex 2da7291e37..8415884754 100755\n--- a/t/t5616-partial-clone.sh\n+++ b/t/t5616-partial-clone.sh\n@@ -229,7 +229,7 @@ test_expect_success 'fetch --refetch triggers repacking' '\n \n \tGIT_TRACE2_EVENT=\"$PWD/trace1.event\" \\\n \tgit -C pc1 fetch --refetch origin &&\n-\ttest_subcommand git maintenance run --auto --no-quiet /dev/null &&\n \tGIT_TRACE2_EVENT=\"$(pwd)/run-no-quiet.txt\" \\\n \t\tgit maintenance run --no-quiet 2>/dev/null &&\n-\ttest_subcommand git gc --quiet ' '\n \t\tgit maintenance run --task=commit-graph 2>/dev/null &&\n \tGIT_TRACE2_EVENT=\"$(pwd)/run-both.txt\" \\\n \t\tgit maintenance run --task=commit-graph --task=gc 2>/dev/null &&\n-\ttest_subcommand ! git gc --quiet +test_expect_success '--no-detach causes maintenance to not run in background' '\n> [...]\n> +\t\t# We have no better way to check whether or not the task ran in\n> +\t\t# the background than to verify whether it output anything. The\n> +\t\t# next testcase checks the reverse, making this somewhat safer.\n> +\t\tgit maintenance run --no-detach >out 2>&1 &&\n> +\t\ttest_line_count = 1 out\n> [...]\n> +test_expect_success '--detach causes maintenance to run in background' '\n> +\ttest_when_finished \"rm -rf repo\" &&\n> +\tgit init repo &&\n> +\t(\n> +\t\tcd repo &&\n> +\n> +\t\ttest_commit something &&\n> +\t\tgit config set maintenance.gc.enabled false &&\n> +\t\tgit config set maintenance.loose-objects.enabled true &&\n> +\t\tgit config set maintenance.loose-objects.auto 1 &&\n> +\t\tgit config set maintenance.incremental-repack.enabled true &&\n> +\n> +\t\tgit maintenance run --detach >out 2>&1 &&\n> +\t\ttest_must_be_empty out\n> +\t)\n> +'\n\nThis second test seems to fail racily (or maybe always? see below). In\nCI on Windows, I saw:\n\n 'out' is not empty, it contains:\n fc9fea69579f349e3b02e3264cffbef03e4b1852\n\nThat would make sense to me if the detached process still held the\noriginal stdout/stderr channel open (in which case we'd racily see the\nsame line as in the no-detach case). But we do appear to call\ndaemonize(), which closes both.\n\nCuriously, the code in gc.c does this:\n\n /* Failure to daemonize is ok, we'll continue in foreground. */\n if (opts->detach > 0)\n daemonize();\n\nand the only way for daemonize to fail is if NO_POSIX_GOODIES is set.\nWhich I'd expect on Windows. But then I'd expect this test to _always_\nfail on Windows. Does it? If so, should it be marked with !MINGW?\n\nWhile investigating that, I ran it with --stress locally (on Linux) and\ngot some odd (and definitely racy) results. The test itself passes, but\nthe \"rm -rf repo\" in the test_when_finished sometimes fails with:\n\n rm: cannot remove 'repo/.git/objects': Directory not empty\n\nor similar (sometimes it's another directory like 'repo/.git'). My guess\nis that the background process is still running and creating files in\nthe repository, racing with rm's call to rmdir().\n\nEven if we remove the test_when_finished, it would mean that the final\ncleanup after test_done might similarly fail, leaving a crufty trash\ndirectory. I think to make this robust, we'd need some way of detecting\nwhen the background process has finished. I don't think we report the\npid anywhere, and the daemonize() call means it won't even be in the\nsame process group. Maybe we could spin looking for the incremental pack\nit will create (and timeout after N seconds)? That feels pretty hacky,\nbut I can't think of anything better.\n\n-Peff\n"},{"id":"501179","messageId":"20240817071435.GA1531456@coredump.intra.peff.net","threadId":"61940","inReplyTo":"20240817070924.GA1438563@coredump.intra.peff.net","subject":"Re: [PATCH v3 6/7] builtin/maintenance: add a `--detach` flag","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2024-08-17T07:14:35Z","receivedAt":"2024-08-17T07:14:37Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sat, Aug 17, 2024 at 03:09:25AM -0400, Jeff King wrote:\n\n> While investigating that, I ran it with --stress locally (on Linux) and\n> got some odd (and definitely racy) results. The test itself passes, but\n> the \"rm -rf repo\" in the test_when_finished sometimes fails with:\n> \n> rm: cannot remove 'repo/.git/objects': Directory not empty\n> \n> or similar (sometimes it's another directory like 'repo/.git'). My guess\n> is that the background process is still running and creating files in\n> the repository, racing with rm's call to rmdir().\n> \n> Even if we remove the test_when_finished, it would mean that the final\n> cleanup after test_done might similarly fail, leaving a crufty trash\n> directory. I think to make this robust, we'd need some way of detecting\n> when the background process has finished. I don't think we report the\n> pid anywhere, and the daemonize() call means it won't even be in the\n> same process group. Maybe we could spin looking for the incremental pack\n> it will create (and timeout after N seconds)? That feels pretty hacky,\n> but I can't think of anything better.\n\nAh, I just noticed that a similar problem happened in t6500, discussed\nduring v2 of the series (I only looked at the latest version). I think\nt7900 needs a similar run_and_wait_for_auto_gc() solution.\n\nI suspect the Windows \"out is not empty\" issue is separate, though.\n\n-Peff\n"},{"id":"501222","messageId":"20240817121424.GA2439299@coredump.intra.peff.net","threadId":"61940","inReplyTo":"9befef7c1f7520d58af2b2db17174b8dbc493d56.1723804990.git.ps@pks.im","subject":"Re: [PATCH v3 7/7] run-command: fix detaching when running auto maintenance","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2024-08-17T12:14:24Z","receivedAt":"2024-08-17T12:14:25Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Aug 16, 2024 at 12:45:17PM +0200, Patrick Steinhardt wrote:\n\n> Fix this bug by asking git-gc(1) to not detach when it is being invoked\n> via git-maintenance(1). Instead, git-maintenance(1) now respects a new\n> config \"maintenance.autoDetach\", the equivalent of \"gc.autoDetach\", and\n> detaches itself into the background when running as part of our auto\n> maintenance. This should continue to behave the same for all users which\n> use the git-gc(1) task, only. For others though, it means that we now\n> properly perform all tasks in the background. The default behaviour of\n> git-maintenance(1) when executed by the user does not change, it will\n> remain in the foreground unless they pass the `--detach` option.\n\nThis patch seems to cause segfaults in t5616 when combined with the\nreftable backend. Try this:\n\n GIT_TEST_DEFAULT_REF_FORMAT=reftable ./t5616-partial-clone.sh --run=1-16 --stress\n\nwhich fails for me within a few runs. Bisecting leads to 98077d06b2\n(run-command: fix detaching when running auto maintenance, 2024-08-16).\nIt doesn't trigger with the files ref backend.\n\nCompiling with ASan gets me a stack trace like this:\n\n + git -c protocol.version=0 -C pc1 fetch --filter=blob:limit=29999 --refetch origin\n AddressSanitizer:DEADLYSIGNAL\n =================================================================\n ==657994==ERROR: AddressSanitizer: SEGV on unknown address 0x7fa0f0ec6089 (pc 0x55f23e52ddf9 bp 0x7ffe7bfa1700 sp 0x7ffe7bfa1700 T0)\n ==657994==The signal is caused by a READ memory access.\n #0 0x55f23e52ddf9 in get_var_int reftable/record.c:29\n #1 0x55f23e53295e in reftable_decode_keylen reftable/record.c:170\n #2 0x55f23e532cc0 in reftable_decode_key reftable/record.c:194\n #3 0x55f23e54e72e in block_iter_next reftable/block.c:398\n #4 0x55f23e5573dc in table_iter_next_in_block reftable/reader.c:240\n #5 0x55f23e5573dc in table_iter_next reftable/reader.c:355\n #6 0x55f23e5573dc in table_iter_next reftable/reader.c:339\n #7 0x55f23e551283 in merged_iter_advance_subiter reftable/merged.c:69\n #8 0x55f23e55169e in merged_iter_next_entry reftable/merged.c:123\n #9 0x55f23e55169e in merged_iter_next_void reftable/merged.c:172\n #10 0x55f23e537625 in reftable_iterator_next_ref reftable/generic.c:175\n #11 0x55f23e2cf9c6 in reftable_ref_iterator_advance refs/reftable-backend.c:464\n #12 0x55f23e2d996e in ref_iterator_advance refs/iterator.c:13\n #13 0x55f23e2d996e in do_for_each_ref_iterator refs/iterator.c:452\n #14 0x55f23dca6767 in get_ref_map builtin/fetch.c:623\n #15 0x55f23dca6767 in do_fetch builtin/fetch.c:1659\n #16 0x55f23dca6767 in fetch_one builtin/fetch.c:2133\n #17 0x55f23dca6767 in cmd_fetch builtin/fetch.c:2432\n #18 0x55f23dba7764 in run_builtin git.c:484\n #19 0x55f23dba7764 in handle_builtin git.c:741\n #20 0x55f23dbab61e in run_argv git.c:805\n #21 0x55f23dbab61e in cmd_main git.c:1000\n #22 0x55f23dba4781 in main common-main.c:64\n #23 0x7fa0f063fc89 in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58\n #24 0x7fa0f063fd44 in __libc_start_main_impl ../csu/libc-start.c:360\n #25 0x55f23dba6ad0 in _start (git+0xadfad0) (BuildId: 803b2b7f59beb03d7849fb8294a8e2145dd4aa27)\n\nMy guess based on what I'm seeing and what the patch does is that now\nmaintenance from a previous command is running in the background while\nour foreground git-fetch runs, and it somehow confuses things (perhaps\nby trying to compact reftables or something?). So I think there are two\nproblems:\n\n 1. The reftable code needs to be more robust against whatever race is\n happening. I didn't dig further, but I'm sure it would be possible\n to produce a coredump.\n\n 2. Having racy background maintenance doesn't seem great for test\n robustness. At the very least, it might subject us to the \"rm\"\n problems mentioned elsewhere, where we fail to clean up. Annotating\n individual \"git gc\" or \"git maintenance\" calls with an extra\n descriptor isn't too bad, but in this case it's all happening under\n the hood via fetch. Is it a potential problem for every script,\n then? If so, should we disable background detaching for all test\n repos, and then let the few that want to test it turn it back on?\n\n-Peff\n"},{"id":"501244","messageId":"ZsLjcjhgI8Wk2tIV@tanuki","threadId":"61940","inReplyTo":"20240817121424.GA2439299@coredump.intra.peff.net","subject":"Re: [PATCH v3 7/7] run-command: fix detaching when running auto maintenance","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-08-19T06:17:22Z","receivedAt":"2024-08-19T06:17:29Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Sat, Aug 17, 2024 at 08:14:24AM -0400, Jeff King wrote:\n> On Fri, Aug 16, 2024 at 12:45:17PM +0200, Patrick Steinhardt wrote:\n> \n> > Fix this bug by asking git-gc(1) to not detach when it is being invoked\n> > via git-maintenance(1). Instead, git-maintenance(1) now respects a new\n> > config \"maintenance.autoDetach\", the equivalent of \"gc.autoDetach\", and\n> > detaches itself into the background when running as part of our auto\n> > maintenance. This should continue to behave the same for all users which\n> > use the git-gc(1) task, only. For others though, it means that we now\n> > properly perform all tasks in the background. The default behaviour of\n> > git-maintenance(1) when executed by the user does not change, it will\n> > remain in the foreground unless they pass the `--detach` option.\n> \n> This patch seems to cause segfaults in t5616 when combined with the\n> reftable backend. Try this:\n> \n> GIT_TEST_DEFAULT_REF_FORMAT=reftable ./t5616-partial-clone.sh --run=1-16 --stress\n> \n> which fails for me within a few runs. Bisecting leads to 98077d06b2\n> (run-command: fix detaching when running auto maintenance, 2024-08-16).\n> It doesn't trigger with the files ref backend.\n> \n> Compiling with ASan gets me a stack trace like this:\n> \n> + git -c protocol.version=0 -C pc1 fetch --filter=blob:limit=29999 --refetch origin\n> AddressSanitizer:DEADLYSIGNAL\n> =================================================================\n> ==657994==ERROR: AddressSanitizer: SEGV on unknown address 0x7fa0f0ec6089 (pc 0x55f23e52ddf9 bp 0x7ffe7bfa1700 sp 0x7ffe7bfa1700 T0)\n> ==657994==The signal is caused by a READ memory access.\n> #0 0x55f23e52ddf9 in get_var_int reftable/record.c:29\n> #1 0x55f23e53295e in reftable_decode_keylen reftable/record.c:170\n> #2 0x55f23e532cc0 in reftable_decode_key reftable/record.c:194\n> #3 0x55f23e54e72e in block_iter_next reftable/block.c:398\n> #4 0x55f23e5573dc in table_iter_next_in_block reftable/reader.c:240\n> #5 0x55f23e5573dc in table_iter_next reftable/reader.c:355\n> #6 0x55f23e5573dc in table_iter_next reftable/reader.c:339\n> #7 0x55f23e551283 in merged_iter_advance_subiter reftable/merged.c:69\n> #8 0x55f23e55169e in merged_iter_next_entry reftable/merged.c:123\n> #9 0x55f23e55169e in merged_iter_next_void reftable/merged.c:172\n> #10 0x55f23e537625 in reftable_iterator_next_ref reftable/generic.c:175\n> #11 0x55f23e2cf9c6 in reftable_ref_iterator_advance refs/reftable-backend.c:464\n> #12 0x55f23e2d996e in ref_iterator_advance refs/iterator.c:13\n> #13 0x55f23e2d996e in do_for_each_ref_iterator refs/iterator.c:452\n> #14 0x55f23dca6767 in get_ref_map builtin/fetch.c:623\n> #15 0x55f23dca6767 in do_fetch builtin/fetch.c:1659\n> #16 0x55f23dca6767 in fetch_one builtin/fetch.c:2133\n> #17 0x55f23dca6767 in cmd_fetch builtin/fetch.c:2432\n> #18 0x55f23dba7764 in run_builtin git.c:484\n> #19 0x55f23dba7764 in handle_builtin git.c:741\n> #20 0x55f23dbab61e in run_argv git.c:805\n> #21 0x55f23dbab61e in cmd_main git.c:1000\n> #22 0x55f23dba4781 in main common-main.c:64\n> #23 0x7fa0f063fc89 in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58\n> #24 0x7fa0f063fd44 in __libc_start_main_impl ../csu/libc-start.c:360\n> #25 0x55f23dba6ad0 in _start (git+0xadfad0) (BuildId: 803b2b7f59beb03d7849fb8294a8e2145dd4aa27)\n> \n> My guess based on what I'm seeing and what the patch does is that now\n> maintenance from a previous command is running in the background while\n> our foreground git-fetch runs, and it somehow confuses things (perhaps\n> by trying to compact reftables or something?). So I think there are two\n> problems:\n> \n> 1. The reftable code needs to be more robust against whatever race is\n> happening. I didn't dig further, but I'm sure it would be possible\n> to produce a coredump.\n\nYes, it certainly has to be robust against this. It's also where the\nrecent reftable compaction fixes [1] came from. In theory, it should\nwork alright with a concurrent process compacting the stack at the same\ntime where another process is reading. In practice the backend is still\nin its infancy, so I'm not entirely surprised that there are concurrency\nbugs.\n\nI will investigate this issue, thanks a lot for the backtrace.\n\n[1]: https://lore.kernel.org/git/cover.1722435214.git.ps@pks.im/\n\n> 2. Having racy background maintenance doesn't seem great for test\n> robustness. At the very least, it might subject us to the \"rm\"\n> problems mentioned elsewhere, where we fail to clean up. Annotating\n> individual \"git gc\" or \"git maintenance\" calls with an extra\n> descriptor isn't too bad, but in this case it's all happening under\n> the hood via fetch. Is it a potential problem for every script,\n> then? If so, should we disable background detaching for all test\n> repos, and then let the few that want to test it turn it back on?\n\nMight be a good idea to set `maintenance.autoDetach=false` globally,\nyes. The only downside is of course that it wouldn't cause us to detect\nfailures like the above, where the concurrency itself causes failure.\n\nAnyway, for now I'll:\n\n - Send a patch to fix the race in t7900.\n\n - Investigate the reftable concurrency issue.\n\n - _Not_ send a patch that sets `maintenance.autoDetach=false`.\n\nThe last one requires a bit more discussion first, and we have been\nrunning with `gc.autoDetach=true` implicitly in the past. Thinking a bit\nmore about it, the reason why the above bug triggers now is that\ngit-gc(1) itself runs git-pack-refs(1), but does that _synchronously_\nbefore detaching itself. Now we detach at a higher level in the\nhierarchy, which means that the previously-synchronous part now runs\nasynchronously, as well.\n\nI cannot think of a reason why we shouldn't do this, as the ref backends\nshould handle this gracefully. The fact that the reftable backend\ndoesn't is a separate, preexisting bug.\n\nPatrick\n"},{"id":"501245","messageId":"ZsLjeDtV9Gt3kbhH@tanuki","threadId":"61940","inReplyTo":"20240817070924.GA1438563@coredump.intra.peff.net","subject":"Re: [PATCH v3 6/7] builtin/maintenance: add a `--detach` flag","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-08-19T06:17:28Z","receivedAt":"2024-08-19T06:17:32Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Sat, Aug 17, 2024 at 03:09:24AM -0400, Jeff King wrote:\n> On Fri, Aug 16, 2024 at 12:45:15PM +0200, Patrick Steinhardt wrote:\n> \n> > +test_expect_success '--no-detach causes maintenance to not run in background' '\n> > [...]\n> > +\t\t# We have no better way to check whether or not the task ran in\n> > +\t\t# the background than to verify whether it output anything. The\n> > +\t\t# next testcase checks the reverse, making this somewhat safer.\n> > +\t\tgit maintenance run --no-detach >out 2>&1 &&\n> > +\t\ttest_line_count = 1 out\n> > [...]\n> > +test_expect_success '--detach causes maintenance to run in background' '\n> > +\ttest_when_finished \"rm -rf repo\" &&\n> > +\tgit init repo &&\n> > +\t(\n> > +\t\tcd repo &&\n> > +\n> > +\t\ttest_commit something &&\n> > +\t\tgit config set maintenance.gc.enabled false &&\n> > +\t\tgit config set maintenance.loose-objects.enabled true &&\n> > +\t\tgit config set maintenance.loose-objects.auto 1 &&\n> > +\t\tgit config set maintenance.incremental-repack.enabled true &&\n> > +\n> > +\t\tgit maintenance run --detach >out 2>&1 &&\n> > +\t\ttest_must_be_empty out\n> > +\t)\n> > +'\n> \n> This second test seems to fail racily (or maybe always? see below). In\n> CI on Windows, I saw:\n> \n> 'out' is not empty, it contains:\n> fc9fea69579f349e3b02e3264cffbef03e4b1852\n> \n> That would make sense to me if the detached process still held the\n> original stdout/stderr channel open (in which case we'd racily see the\n> same line as in the no-detach case). But we do appear to call\n> daemonize(), which closes both.\n> \n> Curiously, the code in gc.c does this:\n> \n> /* Failure to daemonize is ok, we'll continue in foreground. */\n> if (opts->detach > 0)\n> daemonize();\n> \n> and the only way for daemonize to fail is if NO_POSIX_GOODIES is set.\n> Which I'd expect on Windows. But then I'd expect this test to _always_\n> fail on Windows. Does it? If so, should it be marked with !MINGW?\n> \n> While investigating that, I ran it with --stress locally (on Linux) and\n> got some odd (and definitely racy) results. The test itself passes, but\n> the \"rm -rf repo\" in the test_when_finished sometimes fails with:\n> \n> rm: cannot remove 'repo/.git/objects': Directory not empty\n> \n> or similar (sometimes it's another directory like 'repo/.git'). My guess\n> is that the background process is still running and creating files in\n> the repository, racing with rm's call to rmdir().\n> \n> Even if we remove the test_when_finished, it would mean that the final\n> cleanup after test_done might similarly fail, leaving a crufty trash\n> directory. I think to make this robust, we'd need some way of detecting\n> when the background process has finished. I don't think we report the\n> pid anywhere, and the daemonize() call means it won't even be in the\n> same process group. Maybe we could spin looking for the incremental pack\n> it will create (and timeout after N seconds)? That feels pretty hacky,\n> but I can't think of anything better.\n\nOh, good catch indeed, I can reproduce this on Linux with `--stress`.\n\nI think we can use the same workaround as we do in t6500, namely open a\nnew file descriptor, inherit it and then wait for the child process to\nclose it.\n\nThanks!\n\nPatrick\n"},{"id":"501246","messageId":"cover.1724053639.git.ps@pks.im","threadId":"61940","inReplyTo":"ZsLjcjhgI8Wk2tIV@tanuki","subject":"[PATCH 0/3] Fixups for git-maintenance(1) tests","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-08-19T07:47:56Z","receivedAt":"2024-08-19T07:48:02Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Hi,\n\nthis small patch series fixes up the test flakes and issues on Windows\nas reported by Peff:\n\n - We now wait for git-maintenance(1) to run to completion.\n\n - Instead of checking for the detach logic via the output, we now have\n a new trace2 region that allows us to check whether the detaching\n logic was executed.\n\n - Fix another bug that caused the \"loose-objects\" task to emit the\n packfile hash. This is a preexisting issue, but Peff made me have a\n deeper look at it.\n\nPatrick\n\nPatrick Steinhardt (3):\n t7900: fix flaky test due to leaking background job\n t7900: exercise detaching via trace2 regions\n builtin/maintenance: fix loose objects task emitting pack hash\n\n builtin/gc.c | 11 ++++++++++-\n t/t7900-maintenance.sh | 28 +++++++++++++++++++++++++---\n 2 files changed, 35 insertions(+), 4 deletions(-)\n\n\nbase-commit: 98077d06b28b97d508c389886ee5014056707a5e\n-- \n2.46.0.164.g477ce5ccd6.dirty\n\n"},{"id":"501247","messageId":"4805bb6f6c2c96a2c40d1d8359b63b8c7045e0b6.1724053639.git.ps@pks.im","threadId":"61940","inReplyTo":"cover.1724053639.git.ps@pks.im","subject":"[PATCH 1/3] t7900: fix flaky test due to leaking background job","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-08-19T07:47:59Z","receivedAt":"2024-08-19T07:48:03Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"One of the recently-added tests in t7900 exercises git-maintanance(1)\nwith the `--detach` flag, which causes it to perform maintenance in the\nbackground. We do not wait for the backgrounded process to exit though,\nwhich causes the process to leak outside of the test, leading to racy\nbehaviour.\n\nFix this by synchronizing with the process via a separate file\ndescriptor. This is the same workaround as we use in t6500, see the\nfunction `run_and_wait_for_auto_gc ()`.\n\nSigned-off-by: Patrick Steinhardt \n---\n t/t7900-maintenance.sh | 9 +++++++--\n 1 file changed, 7 insertions(+), 2 deletions(-)\n\ndiff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\nindex 06ab43cfb5..074eadcd1c 100755\n--- a/t/t7900-maintenance.sh\n+++ b/t/t7900-maintenance.sh\n@@ -967,8 +967,13 @@ test_expect_success '--detach causes maintenance to run in background' '\n \t\tgit config set maintenance.loose-objects.auto 1 &&\n \t\tgit config set maintenance.incremental-repack.enabled true &&\n \n-\t\tgit maintenance run --detach >out 2>&1 &&\n-\t\ttest_must_be_empty out\n+\t\t# The extra file descriptor gets inherited to the child\n+\t\t# process, and by reading stdout we thus essentially wait for\n+\t\t# that descriptor to get closed, which indicates that the child\n+\t\t# is done, too.\n+\t\toutput=$(git maintenance run --detach 2>&1 9>&1) &&\n+\t\tprintf \"%s\" \"$output\" >output &&\n+\t\ttest_must_be_empty output\n \t)\n '\n \n-- \n2.46.0.164.g477ce5ccd6.dirty\n\n"},{"id":"501248","messageId":"9712aae82bcb51dd94fdc10f4156e9c78e4b6d8c.1724053639.git.ps@pks.im","threadId":"61940","inReplyTo":"cover.1724053639.git.ps@pks.im","subject":"[PATCH 2/3] t7900: exercise detaching via trace2 regions","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-08-19T07:48:02Z","receivedAt":"2024-08-19T07:48:06Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"In t7900, we exercise the `--detach` logic by checking whether the\ncommand ended up writing anything to its output or not. This supposedly\nworks because we close stdin, stdout and stderr when daemonizing. But\none, it breaks on platforms where daemonize is a no-op, like Windows.\nAnd second, that git-maintenance(1) outputs anything at all in these\ntests is a bug in the first place that we'll fix in a subsequent commit.\n\nIntroduce a new trace2 region around the detach which allows us to more\nexplicitly check whether the detaching logic was executed. This is a\nmuch more direct way to exercise the logic, provides a potentially\nuseful signal to tracing logs and also works alright on platforms which\ndo not have the ability to daemonize.\n\nSigned-off-by: Patrick Steinhardt \n---\n builtin/gc.c | 5 ++++-\n t/t7900-maintenance.sh | 11 ++++++-----\n 2 files changed, 10 insertions(+), 6 deletions(-)\n\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex bafee330a2..13bc0572a3 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -1428,8 +1428,11 @@ static int maintenance_run_tasks(struct maintenance_run_opts *opts,\n \tfree(lock_path);\n \n \t/* Failure to daemonize is ok, we'll continue in foreground. */\n-\tif (opts->detach > 0)\n+\tif (opts->detach > 0) {\n+\t\ttrace2_region_enter(\"maintenance\", \"detach\", the_repository);\n \t\tdaemonize();\n+\t\ttrace2_region_leave(\"maintenance\", \"detach\", the_repository);\n+\t}\n \n \tfor (i = 0; !found_selected && i < TASK__COUNT; i++)\n \t\tfound_selected = tasks[i].selected_order >= 0;\ndiff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\nindex 074eadcd1c..46a61d66fb 100755\n--- a/t/t7900-maintenance.sh\n+++ b/t/t7900-maintenance.sh\n@@ -950,8 +950,9 @@ test_expect_success '--no-detach causes maintenance to not run in background' '\n \t\t# We have no better way to check whether or not the task ran in\n \t\t# the background than to verify whether it output anything. The\n \t\t# next testcase checks the reverse, making this somewhat safer.\n-\t\tgit maintenance run --no-detach >out 2>&1 &&\n-\t\ttest_line_count = 1 out\n+\t\tGIT_TRACE2_EVENT=\"$(pwd)/trace.txt\" \\\n+\t\t\tgit maintenance run --no-detach >out 2>&1 &&\n+\t\t! test_region maintenance detach trace.txt\n \t)\n '\n \n@@ -971,9 +972,9 @@ test_expect_success '--detach causes maintenance to run in background' '\n \t\t# process, and by reading stdout we thus essentially wait for\n \t\t# that descriptor to get closed, which indicates that the child\n \t\t# is done, too.\n-\t\toutput=$(git maintenance run --detach 2>&1 9>&1) &&\n-\t\tprintf \"%s\" \"$output\" >output &&\n-\t\ttest_must_be_empty output\n+\t\tdoes_not_matter=$(GIT_TRACE2_EVENT=\"$(pwd)/trace.txt\" \\\n+\t\t\tgit maintenance run --detach 9>&1) &&\n+\t\ttest_region maintenance detach trace.txt\n \t)\n '\n \n-- \n2.46.0.164.g477ce5ccd6.dirty\n\n"},{"id":"501249","messageId":"c25b5333f60a5920c1fade06532e3379c6686908.1724053639.git.ps@pks.im","threadId":"61940","inReplyTo":"cover.1724053639.git.ps@pks.im","subject":"[PATCH 3/3] builtin/maintenance: fix loose objects task emitting pack hash","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-08-19T07:48:05Z","receivedAt":"2024-08-19T07:48:09Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"The \"loose-objects\" maintenance tasks executes git-pack-objects(1) to\npack all loose objects into a new packfile. This command ends up\nprinting the hash of the packfile to stdout though, which clutters the\noutput of `git maintenance run`.\n\nFix this issue by disabling stdout of the child process.\n\nSigned-off-by: Patrick Steinhardt \n---\n builtin/gc.c | 6 ++++++\n t/t7900-maintenance.sh | 16 ++++++++++++++++\n 2 files changed, 22 insertions(+)\n\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 13bc0572a3..be75efa17a 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -1159,6 +1159,12 @@ static int pack_loose(struct maintenance_run_opts *opts)\n \n \tpack_proc.in = -1;\n \n+\t/*\n+\t * git-pack-objects(1) ends up writing the pack hash to stdout, which\n+\t * we do not care for.\n+\t */\n+\tpack_proc.out = -1;\n+\n \tif (start_command(&pack_proc)) {\n \t\terror(_(\"failed to start 'git pack-objects' process\"));\n \t\treturn 1;\ndiff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\nindex 46a61d66fb..7cc4eb262c 100755\n--- a/t/t7900-maintenance.sh\n+++ b/t/t7900-maintenance.sh\n@@ -978,4 +978,20 @@ test_expect_success '--detach causes maintenance to run in background' '\n \t)\n '\n \n+test_expect_success 'repacking loose objects is quiet' '\n+\ttest_when_finished \"rm -rf repo\" &&\n+\tgit init repo &&\n+\t(\n+\t\tcd repo &&\n+\n+\t\ttest_commit something &&\n+\t\tgit config set maintenance.gc.enabled false &&\n+\t\tgit config set maintenance.loose-objects.enabled true &&\n+\t\tgit config set maintenance.loose-objects.auto 1 &&\n+\n+\t\tgit maintenance run --quiet >out 2>&1 &&\n+\t\ttest_must_be_empty out\n+\t)\n+'\n+\n test_done\n-- \n2.46.0.164.g477ce5ccd6.dirty\n\n"},{"id":"501250","messageId":"20240819084614.GA2955268@coredump.intra.peff.net","threadId":"61940","inReplyTo":"ZsLjcjhgI8Wk2tIV@tanuki","subject":"Re: [PATCH v3 7/7] run-command: fix detaching when running auto maintenance","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2024-08-19T08:46:14Z","receivedAt":"2024-08-19T08:46:23Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Aug 19, 2024 at 08:17:22AM +0200, Patrick Steinhardt wrote:\n\n> > 2. Having racy background maintenance doesn't seem great for test\n> > robustness. At the very least, it might subject us to the \"rm\"\n> > problems mentioned elsewhere, where we fail to clean up. Annotating\n> > individual \"git gc\" or \"git maintenance\" calls with an extra\n> > descriptor isn't too bad, but in this case it's all happening under\n> > the hood via fetch. Is it a potential problem for every script,\n> > then? If so, should we disable background detaching for all test\n> > repos, and then let the few that want to test it turn it back on?\n> \n> Might be a good idea to set `maintenance.autoDetach=false` globally,\n> yes. The only downside is of course that it wouldn't cause us to detect\n> failures like the above, where the concurrency itself causes failure.\n> \n> Anyway, for now I'll:\n> \n> - Send a patch to fix the race in t7900.\n> \n> - Investigate the reftable concurrency issue.\n> \n> - _Not_ send a patch that sets `maintenance.autoDetach=false`.\n\nThat sounds like a good direction. I do suspect there are at least _two_\nraces in t7900:\n\n 1. the detached maintenance that we run explicitly, which causes the\n \"rm\" cleanup to fail\n\n 2. whatever earlier test is kicking off detached maintenance via \"git\n fetch\", which is causing the reftable concurrency issue.\n\nFixing (1) should be easy (and it looks like you've already sent a\nseries). Fixing the reftable code will stop us from segfaulting for (2),\nbut I wonder if that detached maintenance might cause similar \"rm\" style\nproblems elsewhere.\n\n> The last one requires a bit more discussion first, and we have been\n> running with `gc.autoDetach=true` implicitly in the past. Thinking a bit\n> more about it, the reason why the above bug triggers now is that\n> git-gc(1) itself runs git-pack-refs(1), but does that _synchronously_\n> before detaching itself. Now we detach at a higher level in the\n> hierarchy, which means that the previously-synchronous part now runs\n> asynchronously, as well.\n\nThat makes sense. I guess we've perhaps been doing background gc for a\nlong time, then, just not in the refs? In practice most repos in the\ntest suite aren't big enough to trigger auto-gc anyway, so it may only\naffect a handful of scripts.\n\nOnce the reftable issue is fixed, it's possible that the lingering\ndetached processes don't cause a problem in practice (because they're\nnot really writing much, and/or have finished by the time anybody else\ngets to cleanup), and we can just live with them. But I'm worried that\nsounds like wishful thinking. ;)\n\n-Peff\n"},{"id":"501251","messageId":"20240819084943.GB2955268@coredump.intra.peff.net","threadId":"61940","inReplyTo":"4805bb6f6c2c96a2c40d1d8359b63b8c7045e0b6.1724053639.git.ps@pks.im","subject":"Re: [PATCH 1/3] t7900: fix flaky test due to leaking background job","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2024-08-19T08:49:43Z","receivedAt":"2024-08-19T08:49:45Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Aug 19, 2024 at 09:47:59AM +0200, Patrick Steinhardt wrote:\n\n> diff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\n> index 06ab43cfb5..074eadcd1c 100755\n> --- a/t/t7900-maintenance.sh\n> +++ b/t/t7900-maintenance.sh\n> @@ -967,8 +967,13 @@ test_expect_success '--detach causes maintenance to run in background' '\n> \t\tgit config set maintenance.loose-objects.auto 1 &&\n> \t\tgit config set maintenance.incremental-repack.enabled true &&\n> \n> -\t\tgit maintenance run --detach >out 2>&1 &&\n> -\t\ttest_must_be_empty out\n> +\t\t# The extra file descriptor gets inherited to the child\n> +\t\t# process, and by reading stdout we thus essentially wait for\n> +\t\t# that descriptor to get closed, which indicates that the child\n> +\t\t# is done, too.\n> +\t\toutput=$(git maintenance run --detach 2>&1 9>&1) &&\n> +\t\tprintf \"%s\" \"$output\" >output &&\n> +\t\ttest_must_be_empty output\n> \t)\n> '\n\nThis looks correct, but should we be doing it for all of the \"git\nmaintenance\" runs in that script? They're all going to kick off detached\ngc jobs, I think.\n\n-Peff\n"},{"id":"501252","messageId":"20240819085105.GC2955268@coredump.intra.peff.net","threadId":"61940","inReplyTo":"9712aae82bcb51dd94fdc10f4156e9c78e4b6d8c.1724053639.git.ps@pks.im","subject":"Re: [PATCH 2/3] t7900: exercise detaching via trace2 regions","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2024-08-19T08:51:05Z","receivedAt":"2024-08-19T08:51:07Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Aug 19, 2024 at 09:48:02AM +0200, Patrick Steinhardt wrote:\n\n> In t7900, we exercise the `--detach` logic by checking whether the\n> command ended up writing anything to its output or not. This supposedly\n> works because we close stdin, stdout and stderr when daemonizing. But\n> one, it breaks on platforms where daemonize is a no-op, like Windows.\n> And second, that git-maintenance(1) outputs anything at all in these\n> tests is a bug in the first place that we'll fix in a subsequent commit.\n> \n> Introduce a new trace2 region around the detach which allows us to more\n> explicitly check whether the detaching logic was executed. This is a\n> much more direct way to exercise the logic, provides a potentially\n> useful signal to tracing logs and also works alright on platforms which\n> do not have the ability to daemonize.\n\nNice, this is so much cleaner than the way the existing test worked. The\ncode looks good, but...\n\n> diff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\n> index 074eadcd1c..46a61d66fb 100755\n> --- a/t/t7900-maintenance.sh\n> +++ b/t/t7900-maintenance.sh\n> @@ -950,8 +950,9 @@ test_expect_success '--no-detach causes maintenance to not run in background' '\n> \t\t# We have no better way to check whether or not the task ran in\n> \t\t# the background than to verify whether it output anything. The\n> \t\t# next testcase checks the reverse, making this somewhat safer.\n> -\t\tgit maintenance run --no-detach >out 2>&1 &&\n> -\t\ttest_line_count = 1 out\n> +\t\tGIT_TRACE2_EVENT=\"$(pwd)/trace.txt\" \\\n> +\t\t\tgit maintenance run --no-detach >out 2>&1 &&\n> +\t\t! test_region maintenance detach trace.txt\n> \t)\n> '\n\n...I think this \"we have no better way...\" comment is now out of date\n(and can probably just be dropped).\n\n-Peff\n"},{"id":"501253","messageId":"ZsMIa_5jCFs4OWYx@tanuki","threadId":"61940","inReplyTo":"20240819084943.GB2955268@coredump.intra.peff.net","subject":"Re: [PATCH 1/3] t7900: fix flaky test due to leaking background job","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-08-19T08:55:07Z","receivedAt":"2024-08-19T08:55:17Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, Aug 19, 2024 at 04:49:43AM -0400, Jeff King wrote:\n> On Mon, Aug 19, 2024 at 09:47:59AM +0200, Patrick Steinhardt wrote:\n> \n> > diff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\n> > index 06ab43cfb5..074eadcd1c 100755\n> > --- a/t/t7900-maintenance.sh\n> > +++ b/t/t7900-maintenance.sh\n> > @@ -967,8 +967,13 @@ test_expect_success '--detach causes maintenance to run in background' '\n> > \t\tgit config set maintenance.loose-objects.auto 1 &&\n> > \t\tgit config set maintenance.incremental-repack.enabled true &&\n> > \n> > -\t\tgit maintenance run --detach >out 2>&1 &&\n> > -\t\ttest_must_be_empty out\n> > +\t\t# The extra file descriptor gets inherited to the child\n> > +\t\t# process, and by reading stdout we thus essentially wait for\n> > +\t\t# that descriptor to get closed, which indicates that the child\n> > +\t\t# is done, too.\n> > +\t\toutput=$(git maintenance run --detach 2>&1 9>&1) &&\n> > +\t\tprintf \"%s\" \"$output\" >output &&\n> > +\t\ttest_must_be_empty output\n> > \t)\n> > '\n> \n> This looks correct, but should we be doing it for all of the \"git\n> maintenance\" runs in that script? They're all going to kick off detached\n> gc jobs, I think.\n\nOnly those that use `--detach` run in the background.\n\nPatrick\n"},{"id":"501254","messageId":"20240819085522.GD2955268@coredump.intra.peff.net","threadId":"61940","inReplyTo":"c25b5333f60a5920c1fade06532e3379c6686908.1724053639.git.ps@pks.im","subject":"Re: [PATCH 3/3] builtin/maintenance: fix loose objects task emitting pack hash","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2024-08-19T08:55:22Z","receivedAt":"2024-08-19T08:55:23Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Aug 19, 2024 at 09:48:05AM +0200, Patrick Steinhardt wrote:\n\n> The \"loose-objects\" maintenance tasks executes git-pack-objects(1) to\n> pack all loose objects into a new packfile. This command ends up\n> printing the hash of the packfile to stdout though, which clutters the\n> output of `git maintenance run`.\n> \n> Fix this issue by disabling stdout of the child process.\n\nAh, I wondered where that output was coming from.\n\n> diff --git a/builtin/gc.c b/builtin/gc.c\n> index 13bc0572a3..be75efa17a 100644\n> --- a/builtin/gc.c\n> +++ b/builtin/gc.c\n> @@ -1159,6 +1159,12 @@ static int pack_loose(struct maintenance_run_opts *opts)\n> \n> \tpack_proc.in = -1;\n> \n> +\t/*\n> +\t * git-pack-objects(1) ends up writing the pack hash to stdout, which\n> +\t * we do not care for.\n> +\t */\n> +\tpack_proc.out = -1;\n> +\n> \tif (start_command(&pack_proc)) {\n> \t\terror(_(\"failed to start 'git pack-objects' process\"));\n> \t\treturn 1;\n\nI have not paid much attention to the \"maintenance\" stuff. It is a\nlittle weird to me that it is not building on \"git repack\", which\nalready handles this, but perhaps there are reasons. Anyway, totally\nunrelated to your patch (which looks good to me).\n\n> +++ b/t/t7900-maintenance.sh\n> @@ -978,4 +978,20 @@ test_expect_success '--detach causes maintenance to run in background' '\n> \t)\n> '\n> \n> +test_expect_success 'repacking loose objects is quiet' '\n> +\ttest_when_finished \"rm -rf repo\" &&\n> +\tgit init repo &&\n> +\t(\n> +\t\tcd repo &&\n> +\n> +\t\ttest_commit something &&\n> +\t\tgit config set maintenance.gc.enabled false &&\n> +\t\tgit config set maintenance.loose-objects.enabled true &&\n> +\t\tgit config set maintenance.loose-objects.auto 1 &&\n> +\n> +\t\tgit maintenance run --quiet >out 2>&1 &&\n> +\t\ttest_must_be_empty out\n> +\t)\n> +'\n\nI wondered if you needed --no-detach here to avoid a race, but I guess\nas a non-auto run, it would never background?\n\n-Peff\n"},{"id":"501255","messageId":"ZsMIxmX2Ash9YtEU@tanuki","threadId":"61940","inReplyTo":"20240819085105.GC2955268@coredump.intra.peff.net","subject":"Re: [PATCH 2/3] t7900: exercise detaching via trace2 regions","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-08-19T08:56:38Z","receivedAt":"2024-08-19T08:56:44Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, Aug 19, 2024 at 04:51:05AM -0400, Jeff King wrote:\n> On Mon, Aug 19, 2024 at 09:48:02AM +0200, Patrick Steinhardt wrote:\n> \n> > In t7900, we exercise the `--detach` logic by checking whether the\n> > command ended up writing anything to its output or not. This supposedly\n> > works because we close stdin, stdout and stderr when daemonizing. But\n> > one, it breaks on platforms where daemonize is a no-op, like Windows.\n> > And second, that git-maintenance(1) outputs anything at all in these\n> > tests is a bug in the first place that we'll fix in a subsequent commit.\n> > \n> > Introduce a new trace2 region around the detach which allows us to more\n> > explicitly check whether the detaching logic was executed. This is a\n> > much more direct way to exercise the logic, provides a potentially\n> > useful signal to tracing logs and also works alright on platforms which\n> > do not have the ability to daemonize.\n> \n> Nice, this is so much cleaner than the way the existing test worked. The\n> code looks good, but...\n> \n> > diff --git a/t/t7900-maintenance.sh b/t/t7900-maintenance.sh\n> > index 074eadcd1c..46a61d66fb 100755\n> > --- a/t/t7900-maintenance.sh\n> > +++ b/t/t7900-maintenance.sh\n> > @@ -950,8 +950,9 @@ test_expect_success '--no-detach causes maintenance to not run in background' '\n> > \t\t# We have no better way to check whether or not the task ran in\n> > \t\t# the background than to verify whether it output anything. The\n> > \t\t# next testcase checks the reverse, making this somewhat safer.\n> > -\t\tgit maintenance run --no-detach >out 2>&1 &&\n> > -\t\ttest_line_count = 1 out\n> > +\t\tGIT_TRACE2_EVENT=\"$(pwd)/trace.txt\" \\\n> > +\t\t\tgit maintenance run --no-detach >out 2>&1 &&\n> > +\t\t! test_region maintenance detach trace.txt\n> > \t)\n> > '\n> \n> ...I think this \"we have no better way...\" comment is now out of date\n> (and can probably just be dropped).\n\nOops, yes, that one is definitely stale. I'll drop it in the next\nversion of this patch series.\n\nPatrick\n"},{"id":"501256","messageId":"ZsMKpNMDTysNcreR@tanuki","threadId":"61940","inReplyTo":"20240819084614.GA2955268@coredump.intra.peff.net","subject":"Re: [PATCH v3 7/7] run-command: fix detaching when running auto maintenance","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-08-19T09:04:42Z","receivedAt":"2024-08-19T09:04:47Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, Aug 19, 2024 at 04:46:14AM -0400, Jeff King wrote:\n> On Mon, Aug 19, 2024 at 08:17:22AM +0200, Patrick Steinhardt wrote:\n> \n> > > 2. Having racy background maintenance doesn't seem great for test\n> > > robustness. At the very least, it might subject us to the \"rm\"\n> > > problems mentioned elsewhere, where we fail to clean up. Annotating\n> > > individual \"git gc\" or \"git maintenance\" calls with an extra\n> > > descriptor isn't too bad, but in this case it's all happening under\n> > > the hood via fetch. Is it a potential problem for every script,\n> > > then? If so, should we disable background detaching for all test\n> > > repos, and then let the few that want to test it turn it back on?\n> > \n> > Might be a good idea to set `maintenance.autoDetach=false` globally,\n> > yes. The only downside is of course that it wouldn't cause us to detect\n> > failures like the above, where the concurrency itself causes failure.\n> > \n> > Anyway, for now I'll:\n> > \n> > - Send a patch to fix the race in t7900.\n> > \n> > - Investigate the reftable concurrency issue.\n> > \n> > - _Not_ send a patch that sets `maintenance.autoDetach=false`.\n> \n> That sounds like a good direction. I do suspect there are at least _two_\n> races in t7900:\n> \n> 1. the detached maintenance that we run explicitly, which causes the\n> \"rm\" cleanup to fail\n> \n> 2. whatever earlier test is kicking off detached maintenance via \"git\n> fetch\", which is causing the reftable concurrency issue.\n> \n> Fixing (1) should be easy (and it looks like you've already sent a\n> series). Fixing the reftable code will stop us from segfaulting for (2),\n> but I wonder if that detached maintenance might cause similar \"rm\" style\n> problems elsewhere.\n\nIt certainly might. The only reason why I don't want to send that patch\nnow is that it feels a bit too reactionary. We haven't had issues in the\npast with it to the best of my knowledge, even though it is an issue in\ntheory.\n\nWe can certainly revisit that though if we see that it indeed is a more\nwidespread issue.\n\n> > The last one requires a bit more discussion first, and we have been\n> > running with `gc.autoDetach=true` implicitly in the past. Thinking a bit\n> > more about it, the reason why the above bug triggers now is that\n> > git-gc(1) itself runs git-pack-refs(1), but does that _synchronously_\n> > before detaching itself. Now we detach at a higher level in the\n> > hierarchy, which means that the previously-synchronous part now runs\n> > asynchronously, as well.\n> \n> That makes sense. I guess we've perhaps been doing background gc for a\n> long time, then, just not in the refs? In practice most repos in the\n> test suite aren't big enough to trigger auto-gc anyway, so it may only\n> affect a handful of scripts.\n\nYup. We should expect this to work just fine, because it can trigger\nregardless of whether or not we run auto-compaction concurrently or not.\nAfter all, the reftable backend even performs auto-compaction after\nevery write to the table, so any two concurrent writes may hit the bug\nwithout even invoking git-maintenance(1) at all.\n\n> Once the reftable issue is fixed, it's possible that the lingering\n> detached processes don't cause a problem in practice (because they're\n> not really writing much, and/or have finished by the time anybody else\n> gets to cleanup), and we can just live with them. But I'm worried that\n> sounds like wishful thinking. ;)\n\nCould certainly be, yeah. I just want to focus on fixing the immediate\nissues before we jump to conclusions too fast.\n\nPatrick\n"},{"id":"501257","messageId":"ZsMLZ3Tlhxsg6Qdr@tanuki","threadId":"61940","inReplyTo":"20240819085522.GD2955268@coredump.intra.peff.net","subject":"Re: [PATCH 3/3] builtin/maintenance: fix loose objects task emitting pack hash","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-08-19T09:07:51Z","receivedAt":"2024-08-19T09:07:57Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, Aug 19, 2024 at 04:55:22AM -0400, Jeff King wrote:\n> On Mon, Aug 19, 2024 at 09:48:05AM +0200, Patrick Steinhardt wrote:\n> \n> > The \"loose-objects\" maintenance tasks executes git-pack-objects(1) to\n> > pack all loose objects into a new packfile. This command ends up\n> > printing the hash of the packfile to stdout though, which clutters the\n> > output of `git maintenance run`.\n> > \n> > Fix this issue by disabling stdout of the child process.\n> \n> Ah, I wondered where that output was coming from.\n> \n> > diff --git a/builtin/gc.c b/builtin/gc.c\n> > index 13bc0572a3..be75efa17a 100644\n> > --- a/builtin/gc.c\n> > +++ b/builtin/gc.c\n> > @@ -1159,6 +1159,12 @@ static int pack_loose(struct maintenance_run_opts *opts)\n> > \n> > \tpack_proc.in = -1;\n> > \n> > +\t/*\n> > +\t * git-pack-objects(1) ends up writing the pack hash to stdout, which\n> > +\t * we do not care for.\n> > +\t */\n> > +\tpack_proc.out = -1;\n> > +\n> > \tif (start_command(&pack_proc)) {\n> > \t\terror(_(\"failed to start 'git pack-objects' process\"));\n> > \t\treturn 1;\n> \n> I have not paid much attention to the \"maintenance\" stuff. It is a\n> little weird to me that it is not building on \"git repack\", which\n> already handles this, but perhaps there are reasons. Anyway, totally\n> unrelated to your patch (which looks good to me).\n\ngit-repack(1) is way less efficient than running git-pack-objects(1)\ndirectly. I've also noticed that at one point in time when revamping how\nwe do housekeeping in Git.\n\nIt mostly boils down to git-repack(1) doing a connectivity check,\nwhereas git-pack-objects(1) doesn't. We just soak up every single loose\nobject, and then eventually we expire them via git-multi-pack-index(1)'s\n\"expire\" subcommand.\n\n> > +++ b/t/t7900-maintenance.sh\n> > @@ -978,4 +978,20 @@ test_expect_success '--detach causes maintenance to run in background' '\n> > \t)\n> > '\n> > \n> > +test_expect_success 'repacking loose objects is quiet' '\n> > +\ttest_when_finished \"rm -rf repo\" &&\n> > +\tgit init repo &&\n> > +\t(\n> > +\t\tcd repo &&\n> > +\n> > +\t\ttest_commit something &&\n> > +\t\tgit config set maintenance.gc.enabled false &&\n> > +\t\tgit config set maintenance.loose-objects.enabled true &&\n> > +\t\tgit config set maintenance.loose-objects.auto 1 &&\n> > +\n> > +\t\tgit maintenance run --quiet >out 2>&1 &&\n> > +\t\ttest_must_be_empty out\n> > +\t)\n> > +'\n> \n> I wondered if you needed --no-detach here to avoid a race, but I guess\n> as a non-auto run, it would never background?\n\nEven the `--auto` run does not background. That was the case for\ngit-gc(1), but is not the case for git-maintenance(1). You now have to\npass `--detach` explicitly to cause it to background, which I think is\nthe saner way to do this anyway.\n\nPatrick\n"},{"id":"501258","messageId":"20240819091241.GA2958552@coredump.intra.peff.net","threadId":"61940","inReplyTo":"ZsMIa_5jCFs4OWYx@tanuki","subject":"Re: [PATCH 1/3] t7900: fix flaky test due to leaking background job","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2024-08-19T09:12:41Z","receivedAt":"2024-08-19T09:12:43Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Aug 19, 2024 at 10:55:07AM +0200, Patrick Steinhardt wrote:\n\n> > This looks correct, but should we be doing it for all of the \"git\n> > maintenance\" runs in that script? They're all going to kick off detached\n> > gc jobs, I think.\n> \n> Only those that use `--detach` run in the background.\n\nI thought since the default for maintenance.autoDetach was true, all of\nthe \"--auto\" ones would need something similar. I notice a lot of those\nuse \"--task\", though, so maybe that doesn't count. I'm not clear on all\nof the rules.\n\n-Peff\n"},{"id":"501259","messageId":"20240819091715.GB2958552@coredump.intra.peff.net","threadId":"61940","inReplyTo":"ZsMLZ3Tlhxsg6Qdr@tanuki","subject":"Re: [PATCH 3/3] builtin/maintenance: fix loose objects task emitting pack hash","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2024-08-19T09:17:15Z","receivedAt":"2024-08-19T09:17:17Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Aug 19, 2024 at 11:07:51AM +0200, Patrick Steinhardt wrote:\n\n> > I have not paid much attention to the \"maintenance\" stuff. It is a\n> > little weird to me that it is not building on \"git repack\", which\n> > already handles this, but perhaps there are reasons. Anyway, totally\n> > unrelated to your patch (which looks good to me).\n> \n> git-repack(1) is way less efficient than running git-pack-objects(1)\n> directly. I've also noticed that at one point in time when revamping how\n> we do housekeeping in Git.\n> \n> It mostly boils down to git-repack(1) doing a connectivity check,\n> whereas git-pack-objects(1) doesn't. We just soak up every single loose\n> object, and then eventually we expire them via git-multi-pack-index(1)'s\n> \"expire\" subcommand.\n\nHmph. I'd have suggested that we should teach git-repack to do the more\nefficient thing. I'm a bit worried about having parallel universes of\nhow maintenance works making it harder to reason about when or how\nthings happen, and how various concurrent / racy behaviors work.\n\nBut it's probably a bit late to re-open that (and certainly it's not\npart of your series).\n\n> > I wondered if you needed --no-detach here to avoid a race, but I guess\n> > as a non-auto run, it would never background?\n> \n> Even the `--auto` run does not background. That was the case for\n> git-gc(1), but is not the case for git-maintenance(1). You now have to\n> pass `--detach` explicitly to cause it to background, which I think is\n> the saner way to do this anyway.\n\nAm I misreading the documentation? The entry for maintenance.autoDetach\non 'next' says:\n\n If unset, the value of `gc.autoDetach` is used as a fallback. Defaults\n to true if both are unset, meaning that the maintenance process will\n detach.\n\n-Peff\n"},{"id":"501260","messageId":"ZsMNndPUOgKrqI4t@tanuki","threadId":"61940","inReplyTo":"20240819091241.GA2958552@coredump.intra.peff.net","subject":"Re: [PATCH 1/3] t7900: fix flaky test due to leaking background job","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-08-19T09:17:17Z","receivedAt":"2024-08-19T09:17:23Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, Aug 19, 2024 at 05:12:41AM -0400, Jeff King wrote:\n> On Mon, Aug 19, 2024 at 10:55:07AM +0200, Patrick Steinhardt wrote:\n> \n> > > This looks correct, but should we be doing it for all of the \"git\n> > > maintenance\" runs in that script? They're all going to kick off detached\n> > > gc jobs, I think.\n> > \n> > Only those that use `--detach` run in the background.\n> \n> I thought since the default for maintenance.autoDetach was true, all of\n> the \"--auto\" ones would need something similar. I notice a lot of those\n> use \"--task\", though, so maybe that doesn't count. I'm not clear on all\n> of the rules.\n\n`maintenance.autoDetach` only influences the auto-maintenance as\nexecuted by `run_auto_maintenance()`.\n\nPatrick\n"},{"id":"501261","messageId":"ZsMPqEWVOSLOi39o@tanuki","threadId":"61940","inReplyTo":"20240819091715.GB2958552@coredump.intra.peff.net","subject":"Re: [PATCH 3/3] builtin/maintenance: fix loose objects task emitting pack hash","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-08-19T09:26:06Z","receivedAt":"2024-08-19T09:26:12Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, Aug 19, 2024 at 05:17:15AM -0400, Jeff King wrote:\n> On Mon, Aug 19, 2024 at 11:07:51AM +0200, Patrick Steinhardt wrote:\n> \n> > > I have not paid much attention to the \"maintenance\" stuff. It is a\n> > > little weird to me that it is not building on \"git repack\", which\n> > > already handles this, but perhaps there are reasons. Anyway, totally\n> > > unrelated to your patch (which looks good to me).\n> > \n> > git-repack(1) is way less efficient than running git-pack-objects(1)\n> > directly. I've also noticed that at one point in time when revamping how\n> > we do housekeeping in Git.\n> > \n> > It mostly boils down to git-repack(1) doing a connectivity check,\n> > whereas git-pack-objects(1) doesn't. We just soak up every single loose\n> > object, and then eventually we expire them via git-multi-pack-index(1)'s\n> > \"expire\" subcommand.\n> \n> Hmph. I'd have suggested that we should teach git-repack to do the more\n> efficient thing. I'm a bit worried about having parallel universes of\n> how maintenance works making it harder to reason about when or how\n> things happen, and how various concurrent / racy behaviors work.\n> \n> But it's probably a bit late to re-open that (and certainly it's not\n> part of your series).\n> \n> > > I wondered if you needed --no-detach here to avoid a race, but I guess\n> > > as a non-auto run, it would never background?\n> > \n> > Even the `--auto` run does not background. That was the case for\n> > git-gc(1), but is not the case for git-maintenance(1). You now have to\n> > pass `--detach` explicitly to cause it to background, which I think is\n> > the saner way to do this anyway.\n> \n> Am I misreading the documentation? The entry for maintenance.autoDetach\n> on 'next' says:\n> \n> If unset, the value of `gc.autoDetach` is used as a fallback. Defaults\n> to true if both are unset, meaning that the maintenance process will\n> detach.\n\nYou've omitted the important part:\n\n\tMany Git commands trigger automatic maintenance after they have\n\twritten data into the repository. This boolean config option\n\tcontrols whether this automatic maintenance shall happen in the\n\tforeground or whether the maintenance process shall detach and\n\tcontinue to run in the background.\n\nThe `maintenance.autoDetach` setting only impacts auto-maintentance as\nrun via `run_auto_maintenance()`. The `--auto` flag is somewhat\northogonal: it asks the git-maintenance(1) job to do nothing in case the\nrepository is already optimal.\n\nFor git-gc(1) we indeed did tie the `--auto` flag to backgrounding,\nwhich is somewhat nonsensical. There are usecases where you may want to\npass `--auto`, but still have it run in the foreground. That's why we\nhandle this differently for git-maintenance(1), which requires you to\npass an explicit `--detach` flag.\n\nAlso, we cannot change the behaviour of git-maintenance(1) retroactively\nto make `--auto` detach. While it already essentially did detach for\ngit-gc(1), that was a bug. E.g. when running as part of the scheduler,\nwe'd always have detached and thus ended up with a bunch of concurrent\ngit-gc(1) processes. So even though it does make sense for the scheduler\nto use `--auto`, it wouldn't want the process to detach.\n\nPatrick\n"},{"id":"501262","messageId":"20240819102602.GA2961332@coredump.intra.peff.net","threadId":"61940","inReplyTo":"ZsMPqEWVOSLOi39o@tanuki","subject":"Re: [PATCH 3/3] builtin/maintenance: fix loose objects task emitting pack hash","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2024-08-19T10:26:02Z","receivedAt":"2024-08-19T10:26:06Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Aug 19, 2024 at 11:26:06AM +0200, Patrick Steinhardt wrote:\n\n> > Am I misreading the documentation? The entry for maintenance.autoDetach\n> > on 'next' says:\n> > \n> > If unset, the value of `gc.autoDetach` is used as a fallback. Defaults\n> > to true if both are unset, meaning that the maintenance process will\n> > detach.\n> \n> You've omitted the important part:\n> \n> \tMany Git commands trigger automatic maintenance after they have\n> \twritten data into the repository. This boolean config option\n> \tcontrols whether this automatic maintenance shall happen in the\n> \tforeground or whether the maintenance process shall detach and\n> \tcontinue to run in the background.\n> \n> The `maintenance.autoDetach` setting only impacts auto-maintentance as\n> run via `run_auto_maintenance()`. The `--auto` flag is somewhat\n> orthogonal: it asks the git-maintenance(1) job to do nothing in case the\n> repository is already optimal.\n\nAh. I naively assumed that they did so by passing the \"--auto\" flag. But\nI see now that the caller actually checks the config and passes\n\"--detach\" or not.\n\nThat seems kind of unfriendly to scripted porcelains which want to\ninvoke it, since they have to reimplement that logic. The idea of \"git\ngc --auto\" was that it provided a single API for scripts to invoke,\nincluding respecting the user's config. Now that \"maintenance --auto\"\nhas taken that over, I'd have expected it to do the same.\n\nTo be clear, I don't feel all that strongly about it, but I'm not sure I\nbuy the argument that it is orthogonal, or that here:\n\n> For git-gc(1) we indeed did tie the `--auto` flag to backgrounding,\n> which is somewhat nonsensical. There are usecases where you may want to\n> pass `--auto`, but still have it run in the foreground. That's why we\n> handle this differently for git-maintenance(1), which requires you to\n> pass an explicit `--detach` flag.\n\nwe couldn't just patch \"--no-detach\" for cases where you want to be sure\nit is in the foreground.\n\n> Also, we cannot change the behaviour of git-maintenance(1) retroactively\n> to make `--auto` detach. While it already essentially did detach for\n> git-gc(1), that was a bug. E.g. when running as part of the scheduler,\n> we'd always have detached and thus ended up with a bunch of concurrent\n> git-gc(1) processes. So even though it does make sense for the scheduler\n> to use `--auto`, it wouldn't want the process to detach.\n\nBackwards compatibility is a more compelling argument here, if we've had\n\"maintenance --auto\" that didn't ever detach (though it sounds like it\ndid, via gc, anyway). But yes, one kicked off from a scheduler should be\nusing --no-detach, I'd think.\n\nLike I said, I don't feel strongly enough to work on any changes here.\nI'd hoped to never think about repository maintenance ever again. So you\ncan take these as just impressions of a (relatively) clueful user seeing\nit for the first time. ;)\n\n-Peff\n"},{"id":"501263","messageId":"ZsMjSi8AqA55Rp_X@tanuki","threadId":"61940","inReplyTo":"20240817121424.GA2439299@coredump.intra.peff.net","subject":"Re: [PATCH v3 7/7] run-command: fix detaching when running auto maintenance","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-08-19T10:49:52Z","receivedAt":"2024-08-19T10:49:58Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Sat, Aug 17, 2024 at 08:14:24AM -0400, Jeff King wrote:\n> On Fri, Aug 16, 2024 at 12:45:17PM +0200, Patrick Steinhardt wrote:\n> \n> > Fix this bug by asking git-gc(1) to not detach when it is being invoked\n> > via git-maintenance(1). Instead, git-maintenance(1) now respects a new\n> > config \"maintenance.autoDetach\", the equivalent of \"gc.autoDetach\", and\n> > detaches itself into the background when running as part of our auto\n> > maintenance. This should continue to behave the same for all users which\n> > use the git-gc(1) task, only. For others though, it means that we now\n> > properly perform all tasks in the background. The default behaviour of\n> > git-maintenance(1) when executed by the user does not change, it will\n> > remain in the foreground unless they pass the `--detach` option.\n> \n> This patch seems to cause segfaults in t5616 when combined with the\n> reftable backend. Try this:\n> \n> GIT_TEST_DEFAULT_REF_FORMAT=reftable ./t5616-partial-clone.sh --run=1-16 --stress\n> \n> which fails for me within a few runs. Bisecting leads to 98077d06b2\n> (run-command: fix detaching when running auto maintenance, 2024-08-16).\n> It doesn't trigger with the files ref backend.\n> \n> Compiling with ASan gets me a stack trace like this:\n> \n> + git -c protocol.version=0 -C pc1 fetch --filter=blob:limit=29999 --refetch origin\n> AddressSanitizer:DEADLYSIGNAL\n> =================================================================\n> ==657994==ERROR: AddressSanitizer: SEGV on unknown address 0x7fa0f0ec6089 (pc 0x55f23e52ddf9 bp 0x7ffe7bfa1700 sp 0x7ffe7bfa1700 T0)\n> ==657994==The signal is caused by a READ memory access.\n> #0 0x55f23e52ddf9 in get_var_int reftable/record.c:29\n> #1 0x55f23e53295e in reftable_decode_keylen reftable/record.c:170\n> #2 0x55f23e532cc0 in reftable_decode_key reftable/record.c:194\n> #3 0x55f23e54e72e in block_iter_next reftable/block.c:398\n> #4 0x55f23e5573dc in table_iter_next_in_block reftable/reader.c:240\n> #5 0x55f23e5573dc in table_iter_next reftable/reader.c:355\n> #6 0x55f23e5573dc in table_iter_next reftable/reader.c:339\n> #7 0x55f23e551283 in merged_iter_advance_subiter reftable/merged.c:69\n> #8 0x55f23e55169e in merged_iter_next_entry reftable/merged.c:123\n> #9 0x55f23e55169e in merged_iter_next_void reftable/merged.c:172\n> #10 0x55f23e537625 in reftable_iterator_next_ref reftable/generic.c:175\n> #11 0x55f23e2cf9c6 in reftable_ref_iterator_advance refs/reftable-backend.c:464\n> #12 0x55f23e2d996e in ref_iterator_advance refs/iterator.c:13\n> #13 0x55f23e2d996e in do_for_each_ref_iterator refs/iterator.c:452\n> #14 0x55f23dca6767 in get_ref_map builtin/fetch.c:623\n> #15 0x55f23dca6767 in do_fetch builtin/fetch.c:1659\n> #16 0x55f23dca6767 in fetch_one builtin/fetch.c:2133\n> #17 0x55f23dca6767 in cmd_fetch builtin/fetch.c:2432\n> #18 0x55f23dba7764 in run_builtin git.c:484\n> #19 0x55f23dba7764 in handle_builtin git.c:741\n> #20 0x55f23dbab61e in run_argv git.c:805\n> #21 0x55f23dbab61e in cmd_main git.c:1000\n> #22 0x55f23dba4781 in main common-main.c:64\n> #23 0x7fa0f063fc89 in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58\n> #24 0x7fa0f063fd44 in __libc_start_main_impl ../csu/libc-start.c:360\n> #25 0x55f23dba6ad0 in _start (git+0xadfad0) (BuildId: 803b2b7f59beb03d7849fb8294a8e2145dd4aa27)\n\nI haven't yet been able to definitely tell, but I think this is a\nlifetime issue. We create an iterator, eventually notice that the\nreftable stack has been rewritten, and reload the stack. But the\nblock sources used for the old tables are still referenced by the\niterator, even though it was closed. As such, the mmapped memory of the\ntable has been unmapped and is now invalid, which causes the above\ninvalid reads.\n\nI'll work on a patch series that introduces refcounting for block\nsources, but guess that'll take a bit.\n\nPatrick\n"},{"id":"501282","messageId":"ZsNnsx_J4Te9cxCu@tanuki","threadId":"61940","inReplyTo":"ZsMjSi8AqA55Rp_X@tanuki","subject":"Re: [PATCH v3 7/7] run-command: fix detaching when running auto maintenance","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-08-19T15:41:39Z","receivedAt":"2024-08-19T15:41:45Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, Aug 19, 2024 at 12:49:52PM +0200, Patrick Steinhardt wrote:\n> On Sat, Aug 17, 2024 at 08:14:24AM -0400, Jeff King wrote:\n> > On Fri, Aug 16, 2024 at 12:45:17PM +0200, Patrick Steinhardt wrote:\n> > \n> > > Fix this bug by asking git-gc(1) to not detach when it is being invoked\n> > > via git-maintenance(1). Instead, git-maintenance(1) now respects a new\n> > > config \"maintenance.autoDetach\", the equivalent of \"gc.autoDetach\", and\n> > > detaches itself into the background when running as part of our auto\n> > > maintenance. This should continue to behave the same for all users which\n> > > use the git-gc(1) task, only. For others though, it means that we now\n> > > properly perform all tasks in the background. The default behaviour of\n> > > git-maintenance(1) when executed by the user does not change, it will\n> > > remain in the foreground unless they pass the `--detach` option.\n> > \n> > This patch seems to cause segfaults in t5616 when combined with the\n> > reftable backend. Try this:\n> > \n> > GIT_TEST_DEFAULT_REF_FORMAT=reftable ./t5616-partial-clone.sh --run=1-16 --stress\n> > \n> > which fails for me within a few runs. Bisecting leads to 98077d06b2\n> > (run-command: fix detaching when running auto maintenance, 2024-08-16).\n> > It doesn't trigger with the files ref backend.\n> > \n> > Compiling with ASan gets me a stack trace like this:\n> > \n> > + git -c protocol.version=0 -C pc1 fetch --filter=blob:limit=29999 --refetch origin\n> > AddressSanitizer:DEADLYSIGNAL\n> > =================================================================\n> > ==657994==ERROR: AddressSanitizer: SEGV on unknown address 0x7fa0f0ec6089 (pc 0x55f23e52ddf9 bp 0x7ffe7bfa1700 sp 0x7ffe7bfa1700 T0)\n> > ==657994==The signal is caused by a READ memory access.\n> > #0 0x55f23e52ddf9 in get_var_int reftable/record.c:29\n> > #1 0x55f23e53295e in reftable_decode_keylen reftable/record.c:170\n> > #2 0x55f23e532cc0 in reftable_decode_key reftable/record.c:194\n> > #3 0x55f23e54e72e in block_iter_next reftable/block.c:398\n> > #4 0x55f23e5573dc in table_iter_next_in_block reftable/reader.c:240\n> > #5 0x55f23e5573dc in table_iter_next reftable/reader.c:355\n> > #6 0x55f23e5573dc in table_iter_next reftable/reader.c:339\n> > #7 0x55f23e551283 in merged_iter_advance_subiter reftable/merged.c:69\n> > #8 0x55f23e55169e in merged_iter_next_entry reftable/merged.c:123\n> > #9 0x55f23e55169e in merged_iter_next_void reftable/merged.c:172\n> > #10 0x55f23e537625 in reftable_iterator_next_ref reftable/generic.c:175\n> > #11 0x55f23e2cf9c6 in reftable_ref_iterator_advance refs/reftable-backend.c:464\n> > #12 0x55f23e2d996e in ref_iterator_advance refs/iterator.c:13\n> > #13 0x55f23e2d996e in do_for_each_ref_iterator refs/iterator.c:452\n> > #14 0x55f23dca6767 in get_ref_map builtin/fetch.c:623\n> > #15 0x55f23dca6767 in do_fetch builtin/fetch.c:1659\n> > #16 0x55f23dca6767 in fetch_one builtin/fetch.c:2133\n> > #17 0x55f23dca6767 in cmd_fetch builtin/fetch.c:2432\n> > #18 0x55f23dba7764 in run_builtin git.c:484\n> > #19 0x55f23dba7764 in handle_builtin git.c:741\n> > #20 0x55f23dbab61e in run_argv git.c:805\n> > #21 0x55f23dbab61e in cmd_main git.c:1000\n> > #22 0x55f23dba4781 in main common-main.c:64\n> > #23 0x7fa0f063fc89 in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58\n> > #24 0x7fa0f063fd44 in __libc_start_main_impl ../csu/libc-start.c:360\n> > #25 0x55f23dba6ad0 in _start (git+0xadfad0) (BuildId: 803b2b7f59beb03d7849fb8294a8e2145dd4aa27)\n> \n> I haven't yet been able to definitely tell, but I think this is a\n> lifetime issue. We create an iterator, eventually notice that the\n> reftable stack has been rewritten, and reload the stack. But the\n> block sources used for the old tables are still referenced by the\n> iterator, even though it was closed. As such, the mmapped memory of the\n> table has been unmapped and is now invalid, which causes the above\n> invalid reads.\n> \n> I'll work on a patch series that introduces refcounting for block\n> sources, but guess that'll take a bit.\n\nThis is being handled via\nhttps://lore.kernel.org/git/cover.1724080006.git.ps@pks.im/.\n\nPatrick\n"},{"id":"501293","messageId":"xmqqzfp8e94s.fsf@gitster.g","threadId":"61940","inReplyTo":"20240819091715.GB2958552@coredump.intra.peff.net","subject":"Re: [PATCH 3/3] builtin/maintenance: fix loose objects task emitting pack hash","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-08-19T17:05:07Z","receivedAt":"2024-08-19T17:10:26Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King writes:\n\n> On Mon, Aug 19, 2024 at 11:07:51AM +0200, Patrick Steinhardt wrote:\n>\n>> It mostly boils down to git-repack(1) doing a connectivity check,\n>> whereas git-pack-objects(1) doesn't. We just soak up every single loose\n>> object, and then eventually we expire them via git-multi-pack-index(1)'s\n>> \"expire\" subcommand.\n>\n> Hmph. I'd have suggested that we should teach git-repack to do the more\n> efficient thing. I'm a bit worried about having parallel universes of\n> how maintenance works making it harder to reason about when or how\n> things happen, and how various concurrent / racy behaviors work.\n\nI'd suggest being careful before going there.\n\nThe above only explains why it is OK not to exclude unreachable\ncruft, but does not address another thing we should be worried\nabout, which is the quality of the resulting pack.\n\nThrowing a random set of object names at pack-objects in the order\nthat they are discovered by for_each_loose_file_in_objdir(), which\nis what gc.c:pack_loose() does, would give no locality benefit that\nwalking the commits would. If we assume that we will pack_loose()\noften enough that we won't have huge number of objects in the\nresulting pack, packing objects that are close in the history may\nnot matter much, but on the other hand, if we run pack_loose() too\noften to produce a small pack, you would not have a great delta base\nselection.\n\nSo we should probably monitor how much \"badness\" the pack_loose()\nis causing, and if it turns out to be too much, we may need to\nreconsider its design. Being able to produce ultra-quickly a pack\nwhose layout and delta base choice would hurt runtime performance is\nnot a feature.\n\n> But it's probably a bit late to re-open that (and certainly it's not\n> part of your series).\n\nTrue.\n\nThanks.\n"},{"id":"501326","messageId":"ZsRIHS0mFZaRHFVc@tanuki","threadId":"61940","inReplyTo":"20240819102602.GA2961332@coredump.intra.peff.net","subject":"Re: [PATCH 3/3] builtin/maintenance: fix loose objects task emitting pack hash","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-08-20T07:39:09Z","receivedAt":"2024-08-20T07:39:16Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, Aug 19, 2024 at 06:26:02AM -0400, Jeff King wrote:\n> On Mon, Aug 19, 2024 at 11:26:06AM +0200, Patrick Steinhardt wrote:\n> \n> > > Am I misreading the documentation? The entry for maintenance.autoDetach\n> > > on 'next' says:\n> > > \n> > > If unset, the value of `gc.autoDetach` is used as a fallback. Defaults\n> > > to true if both are unset, meaning that the maintenance process will\n> > > detach.\n> > \n> > You've omitted the important part:\n> > \n> > \tMany Git commands trigger automatic maintenance after they have\n> > \twritten data into the repository. This boolean config option\n> > \tcontrols whether this automatic maintenance shall happen in the\n> > \tforeground or whether the maintenance process shall detach and\n> > \tcontinue to run in the background.\n> > \n> > The `maintenance.autoDetach` setting only impacts auto-maintentance as\n> > run via `run_auto_maintenance()`. The `--auto` flag is somewhat\n> > orthogonal: it asks the git-maintenance(1) job to do nothing in case the\n> > repository is already optimal.\n> \n> Ah. I naively assumed that they did so by passing the \"--auto\" flag. But\n> I see now that the caller actually checks the config and passes\n> \"--detach\" or not.\n> \n> That seems kind of unfriendly to scripted porcelains which want to\n> invoke it, since they have to reimplement that logic. The idea of \"git\n> gc --auto\" was that it provided a single API for scripts to invoke,\n> including respecting the user's config. Now that \"maintenance --auto\"\n> has taken that over, I'd have expected it to do the same.\n> \n> To be clear, I don't feel all that strongly about it, but I'm not sure I\n> buy the argument that it is orthogonal, or that here:\n> \n> > For git-gc(1) we indeed did tie the `--auto` flag to backgrounding,\n> > which is somewhat nonsensical. There are usecases where you may want to\n> > pass `--auto`, but still have it run in the foreground. That's why we\n> > handle this differently for git-maintenance(1), which requires you to\n> > pass an explicit `--detach` flag.\n> \n> we couldn't just patch \"--no-detach\" for cases where you want to be sure\n> it is in the foreground.\n\nWe certainly could. But honestly, your scripted use case you mention\nabove is even more of an argument why we shouldn't do it, in my opinion.\nWe have long had the stance that the behaviour of plumbing tools should\n_not_ be impacted by the user configuration. And detaching based on some\nconfig to me very much sounds like the exact opposite.\n\nMind you, we are all quite used to `git gc --auto` detaching. But if I\nwere new to the project, I'd find it quite surprising that it may or may\nnot detach if all I want it to do is to decide for itself whether it\nneeds to garbage collect or not. It is much more straight forward and\nway less surprising for a script writer to use `--detach` if they want\nthe script to detach, because now the command does what they want\nwithout them having to worry about the user's config.\n\n> > Also, we cannot change the behaviour of git-maintenance(1) retroactively\n> > to make `--auto` detach. While it already essentially did detach for\n> > git-gc(1), that was a bug. E.g. when running as part of the scheduler,\n> > we'd always have detached and thus ended up with a bunch of concurrent\n> > git-gc(1) processes. So even though it does make sense for the scheduler\n> > to use `--auto`, it wouldn't want the process to detach.\n> \n> Backwards compatibility is a more compelling argument here, if we've had\n> \"maintenance --auto\" that didn't ever detach (though it sounds like it\n> did, via gc, anyway). But yes, one kicked off from a scheduler should be\n> using --no-detach, I'd think.\n\nYes, we did, but as mentioned it was buggy. Once the scheduler kicks\noff, you'd now have N git-gc(1) processes all running in parallel to\neach other. With N being large you will certainly face some issues. You\nalso lose the exit code, which is another issue.\n\nBut as you said, you could make the scheduler pass `--no-detach`. In\nfact, the first versions of this patch series were using your approach,\nwhere I changed `git maintenance run --auto` to detach based on the\nconfig. But after some thought (and after seeing the negative fallout\nthat this had on our test suite) I decided to throw this approach away\nbecause it just didn't feel right to me.\n\n> Like I said, I don't feel strongly enough to work on any changes here.\n> I'd hoped to never think about repository maintenance ever again. So you\n> can take these as just impressions of a (relatively) clueful user seeing\n> it for the first time. ;)\n\nI certainly appreciate the discussion, thanks for chiming in! I'm still\nnot convinced that we should continue to couple auto-maintenance and\nbackgrounding to each other. In my opinion, this behaviour was a mistake\nin the past and continues to surprise now, too. Making it an explicit\noption feels more natural to me.\n\nThat being said, when others feel strongly about this, as well, then I'm\nof course happy to adapt.\n\nPatrick\n"},{"id":"501383","messageId":"xmqqle0rb2zy.fsf@gitster.g","threadId":"61940","inReplyTo":"ZsRIHS0mFZaRHFVc@tanuki","subject":"Re: [PATCH 3/3] builtin/maintenance: fix loose objects task emitting pack hash","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-08-20T15:58:09Z","receivedAt":"2024-08-20T15:58:14Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n> I certainly appreciate the discussion, thanks for chiming in! I'm still\n> not convinced that we should continue to couple auto-maintenance and\n> backgrounding to each other. In my opinion, this behaviour was a mistake\n> in the past and continues to surprise now, too. Making it an explicit\n> option feels more natural to me.\n>\n> That being said, when others feel strongly about this, as well, then I'm\n> of course happy to adapt.\n\nFWIW, I find it is a sensible approach to have a separate \"run in\nthe background\" that is not strongly tied to \"do your thing if you\nthink the repository really needs it\".\n\nThanks.\n\n"},{"id":"501454","messageId":"xmqqfrqxyb5h.fsf@gitster.g","threadId":"61940","inReplyTo":"ZsMIxmX2Ash9YtEU@tanuki","subject":"Re: [PATCH 2/3] t7900: exercise detaching via trace2 regions","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-08-21T18:38:02Z","receivedAt":"2024-08-21T18:38:08Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n>> ...I think this \"we have no better way...\" comment is now out of date\n>> (and can probably just be dropped).\n>\n> Oops, yes, that one is definitely stale. I'll drop it in the next\n> version of this patch series.\n\nI am not sure if there is a need for \"the next version\"; in the\nmeantime, let me do this. I'd prefer to merge the main topic down\nto 'master' soonish.\n\nThanks.\n\n1: 759b453f9f = 1: 759b453f9f t7900: fix flaky test due to leaking background job\n2: b64db3e437 ! 2: 51a0b8a2a7 t7900: exercise detaching via trace2 regions\n @@ Commit message\n do not have the ability to daemonize.\n \n Signed-off-by: Patrick Steinhardt \n + [jc: dropped a stale in-code comment from a test]\n Signed-off-by: Junio C Hamano \n \n ## builtin/gc.c ##\n @@ builtin/gc.c: static int maintenance_run_tasks(struct maintenance_run_opts *opts\n \n ## t/t7900-maintenance.sh ##\n @@ t/t7900-maintenance.sh: test_expect_success '--no-detach causes maintenance to not run in background' '\n - \t\t# We have no better way to check whether or not the task ran in\n - \t\t# the background than to verify whether it output anything. The\n - \t\t# next testcase checks the reverse, making this somewhat safer.\n + \t\tgit config set maintenance.loose-objects.auto 1 &&\n + \t\tgit config set maintenance.incremental-repack.enabled true &&\n + \n +-\t\t# We have no better way to check whether or not the task ran in\n +-\t\t# the background than to verify whether it output anything. The\n +-\t\t# next testcase checks the reverse, making this somewhat safer.\n -\t\tgit maintenance run --no-detach >out 2>&1 &&\n -\t\ttest_line_count = 1 out\n +\t\tGIT_TRACE2_EVENT=\"$(pwd)/trace.txt\" \\\n3: eac44feff3 = 3: 8311e3b551 builtin/maintenance: fix loose objects task emitting pack hash\n"},{"id":"501468","messageId":"ZsbPiGJlgxWiM-t9@tanuki","threadId":"61940","inReplyTo":"xmqqfrqxyb5h.fsf@gitster.g","subject":"Re: [PATCH 2/3] t7900: exercise detaching via trace2 regions","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-08-22T05:41:34Z","receivedAt":"2024-08-22T05:41:41Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Wed, Aug 21, 2024 at 11:38:02AM -0700, Junio C Hamano wrote:\n> Patrick Steinhardt writes:\n> \n> >> ...I think this \"we have no better way...\" comment is now out of date\n> >> (and can probably just be dropped).\n> >\n> > Oops, yes, that one is definitely stale. I'll drop it in the next\n> > version of this patch series.\n> \n> I am not sure if there is a need for \"the next version\"; in the\n> meantime, let me do this. I'd prefer to merge the main topic down\n> to 'master' soonish.\n> \n> Thanks.\n\nThanks, this looks good to me.\n\nPatrick\n"},{"id":"501551","messageId":"xmqq34mwsca5.fsf@gitster.g","threadId":"61940","inReplyTo":"ZsbPiGJlgxWiM-t9@tanuki","subject":"Re: [PATCH 2/3] t7900: exercise detaching via trace2 regions","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-08-22T17:22:26Z","receivedAt":"2024-08-22T17:22:31Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n> On Wed, Aug 21, 2024 at 11:38:02AM -0700, Junio C Hamano wrote:\n>> Patrick Steinhardt writes:\n>> \n>> >> ...I think this \"we have no better way...\" comment is now out of date\n>> >> (and can probably just be dropped).\n>> >\n>> > Oops, yes, that one is definitely stale. I'll drop it in the next\n>> > version of this patch series.\n>> \n>> I am not sure if there is a need for \"the next version\"; in the\n>> meantime, let me do this. I'd prefer to merge the main topic down\n>> to 'master' soonish.\n>> \n>> Thanks.\n>\n> Thanks, this looks good to me.\n\nOK. Let me merge the whole thing to 'next', cook it for a few days\nand then merge it together with the base topic down to 'master',\nthen.\n\nThanks.\n"}]}